`, `# Heading`, or heading nodes
+- `'code'` — from ``/``, fenced/indented code blocks, or code nodes
+- `'prose'` — regular paragraph text
+- `'label'` — labels, captions, metadata-like text
+
+**Custom Content Extractors**
+
+New `contentExtractor` parameter lets developers plug in custom parsers:
+
+```typescript
+const highlights = await brain.highlight({
+ query: "function",
+ text: sourceCode,
+ contentExtractor: (text) => treeSitterParse(text) // Custom parser
+})
+```
+
+**Content Type Hints**
+
+New `contentType` parameter to skip auto-detection:
+
+```typescript
+await brain.highlight({ query: "test", text: input, contentType: 'html' })
+```
+
+### New Types
+
+- `ContentType`: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`
+- `ContentCategory`: `'prose' | 'heading' | 'code' | 'label'`
+- `ExtractedSegment`: `{ text: string, contentCategory: ContentCategory }`
+- `HighlightParams.contentType?` — optional content type hint
+- `HighlightParams.contentExtractor?` — optional custom parser callback
+- `Highlight.contentCategory?` — content role annotation
+
+## [7.7.0](https://github.com/soulcraftlabs/brainy/compare/v7.6.1...v7.7.0) (2026-01-26)
+
+### Features
+
+**Match Visibility in Search Results**
+
+Search results now include detailed match information:
+- `textMatches: string[]` - Query words found in entity
+- `textScore: number` - Text match quality (0-1)
+- `semanticScore: number` - Semantic similarity (0-1)
+- `matchSource: 'text' | 'semantic' | 'both'` - Where result came from
+
+```typescript
+const results = await brain.find({ query: 'david the warrior' })
+results[0].textMatches // ["david", "warrior"]
+results[0].semanticScore // 0.87
+results[0].matchSource // "both"
+```
+
+**Semantic Highlighting API**
+
+New `highlight()` method shows which concepts matched:
+
+```typescript
+const highlights = await brain.highlight({
+ query: "david the warrior",
+ text: "David Smith is a brave fighter who battles dragons"
+})
+// Returns both exact matches and semantic concepts:
+// [
+// { text: "David", score: 1.0, matchType: "text" },
+// { text: "fighter", score: 0.78, matchType: "semantic" },
+// { text: "battles", score: 0.72, matchType: "semantic" }
+// ]
+```
+
+**Scalable Word Indexing**
+
+- Increased word limit from 50 to 5000 words per entity
+- Supports articles, chapters, and large documents
+- Roaring Bitmaps provide efficient compression at scale
+
+### Performance
+
+- O(1) fast path in `findMatchingWords()` for text results
+- 500 chunk limit in `highlight()` for memory safety
+- Stopword filtering reduces embedding overhead
+
+### [7.6.1](https://github.com/soulcraftlabs/brainy/compare/v7.6.0...v7.6.1) (2026-01-26)
+
+- docs: add link to hosted API documentation at soulcraft.com/docs
+
+## [7.6.0](https://github.com/soulcraftlabs/brainy/compare/v7.5.0...v7.6.0) (2026-01-26)
+
+- chore: republish (npm ghost versions in 7.5.x range)
### [7.5.0](https://github.com/soulcraftlabs/brainy/compare/v7.4.1...v7.5.0) (2026-01-26)
@@ -260,7 +1096,7 @@ v7.0.0 introduced **breaking changes** to the embedding system:
| copy 15 files | ~120s | ~20-40s | 3-6x faster |
| move 15 files | ~240s | ~40-60s | 4-6x faster |
-Requested by: Soulcraft Workshop team (BRAINY-VFS-RMDIR-PERFORMANCE)
+Requested by: a consumer team (BRAINY-VFS-RMDIR-PERFORMANCE)
### [6.3.2](https://github.com/soulcraftlabs/brainy/compare/v6.3.1...v6.3.2) (2025-12-09)
@@ -303,7 +1139,7 @@ Requested by: Soulcraft Workshop team (BRAINY-VFS-RMDIR-PERFORMANCE)
**Fixed VFS tree operations on cloud storage (GCS, S3, Azure, R2, OPFS)**
**Issue:** Despite v6.1.0's PathResolver optimization, `vfs.getTreeStructure()` remained critically slow on cloud storage:
-- **Workshop Production (GCS):** 5,304ms for tree with maxDepth=2
+- **Production (GCS) deployment:** 5,304ms for tree with maxDepth=2
- **Root Cause:** Tree traversal made 111+ separate storage calls (one per directory)
- **Why v6.1.0 didn't help:** v6.1.0 optimized path→ID resolution, but tree traversal still called `getChildren()` 111+ times
@@ -331,7 +1167,7 @@ Result: 111 storage calls → 1 storage call
- `src/vfs/VirtualFileSystem.ts:730-762` - Updated `getDescendants()` to use batch fetch
**Impact:**
-- ✅ Workshop file explorer now loads instantly on GCS
+- ✅ Consumer file explorer now loads instantly on GCS
- ✅ Clean architecture: one code path, no fallbacks
- ✅ Production-scale: uses in-memory graph + single batch fetch
- ✅ Works for ALL storage adapters (GCS, S3, Azure, R2, OPFS, FileSystem)
@@ -670,7 +1506,7 @@ for (const id of pageIds) {
**Fix:** Set `isInitialized = true` at the START of `BaseStorage.init()` (before any initialization work) to prevent recursive calls. Flag is reset to `false` on error to allow retries.
**Impact:**
-- ✅ Fixes production blocker reported by Workshop team
+- ✅ Fixes production blocker reported by a consumer team
- ✅ All 8 storage adapters fixed (FileSystem, Memory, S3, R2, GCS, Azure, OPFS, Historical)
- ✅ Init completes in ~1 second on fresh installation (was hanging indefinitely)
- ✅ No new test failures introduced (1178 tests passing)
@@ -950,7 +1786,7 @@ See comprehensive guides:
v5.10.0 reintroduced a critical bug where `BlobStorage.read()` was hashing wrapped binary data instead of unwrapped content, causing all blob integrity checks to fail:
- **Symptom**: `Blob integrity check failed: ` errors on every VFS file read
- **Root Cause**: Missing defense-in-depth unwrap verification in `BlobStorage.read()`
-- **Impact**: 100% failure rate for VFS file operations in Workshop application
+- **Impact**: 100% failure rate for VFS file operations in A consumer application
### The Fix (v5.10.1)
1. **Defense-in-Depth Unwrapping**: Added unwrap verification in `BlobStorage.read()` before hash check
@@ -1098,7 +1934,7 @@ Reverted storage internals to v5.6.3 implementation:
- ✅ No 12+ second delays per entity
### Verification
-Workshop team (production users) should upgrade immediately:
+a consumer team (production users) should upgrade immediately:
```bash
npm install @soulcraft/brainy@5.7.1
```
@@ -1135,7 +1971,7 @@ Expected behavior after upgrade:
### 🐛 Bug Fixes
-* **storage**: Fix `clear()` not deleting COW version control data ([#workshop-bug-report](https://github.com/soulcraftlabs/brainy/issues))
+* **storage**: Fix `clear()` not deleting COW version control data (consumer-reported)
- Fixed all storage adapters to properly delete `_cow/` directory on clear()
- Fixed in-memory entity counters not being reset after clear()
- Prevents COW reinitialization after clear() by setting `cowEnabled = false`
@@ -1233,10 +2069,10 @@ Expected behavior after upgrade:
- Impact: All relationship queries via `getRelations()`, `getRelationships()`
- Reference: `src/storage/baseStorage.ts:2030-2040`, `src/storage/baseStorage.ts:2081-2091`
-* **Workshop blob integrity**: Verified v5.4.0 lazy-loading asOf() prevents corruption
+* **Consumer blob integrity**: Verified v5.4.0 lazy-loading asOf() prevents corruption
- HistoricalStorageAdapter eliminates race conditions
- Snapshots created on-demand (no commit-time snapshot)
- - Verified with 570-entity test matching Workshop production scale
+ - Verified with 570-entity test matching consumer production scale
### ⚡ Performance Adjustments
@@ -1529,7 +2365,7 @@ v5.1.0 delivers a significantly improved developer experience:
- **Problem**: In v5.0.0, `saveNoun()` was called before `saveNounMetadata()`, causing TypeAwareStorage to default entity types to 'thing' and save to wrong storage paths
- **Impact**: Broke VFS file operations, `brain.get()`, `brain.relate()`, and all features depending on entity metadata
- **Solution**: Reversed save order - now saves metadata FIRST, then noun vector
- - **Fixes**: [Workshop Bug Report](https://github.com/soulcraftlabs/brain-cloud/issues/VFS-METADATA-MISSING)
+ - **Fixes**: VFS metadata-missing regression (internal tracker)
**Fork API: Lazy COW Initialization**
@@ -1556,7 +2392,7 @@ v5.1.0 delivers a significantly improved developer experience:
### 📊 Impact
-* **Unblocks**: Workshop team and all VFS users
+* **Unblocks**: a consumer team and all VFS users
* **Fixes**: All metadata-dependent features (get, relate, find, VFS)
* **Maintains**: Full backward compatibility with v4.x data
@@ -1904,7 +2740,7 @@ This release transforms Brainy imports from entity extractors into true knowledg
- **Relationship Type Metadata**: All relationships tagged as `vfs`, `semantic`, or `provenance` for filtering
- **Enhanced Column Detection**: 7 relationship types (vs 1 previously) - Location, Owner, Creator, Uses, Member, Friend, Related
- **Type-Based Inference**: Smart relationship classification based on entity types and context analysis
- - **Impact**: Workshop import now creates ~3,900 relationships (vs 581), with 5-20+ connections per entity
+ - **Impact**: A consumer import now creates ~3,900 relationships (vs 581), with 5-20+ connections per entity
* **import**: New configuration option `createProvenanceLinks` (defaults to `true`)
- Enables/disables provenance relationship creation
@@ -1946,7 +2782,7 @@ Graph: Rich network, 5-20+ connections per entity
### [4.7.4](https://github.com/soulcraftlabs/brainy/compare/v4.7.3...v4.7.4) (2025-10-27)
-**CRITICAL SYSTEMIC VFS BUG FIX - Workshop Team Unblocked!**
+**CRITICAL SYSTEMIC VFS BUG FIX - A consumer team Unblocked!**
This hotfix resolves a systemic bug affecting ALL storage adapters that caused VFS queries to return empty results even when data existed.
@@ -1958,7 +2794,7 @@ This hotfix resolves a systemic bug affecting ALL storage adapters that caused V
- **Bug Pattern**: `if (!metadata) continue` in getNouns()/getVerbs() methods
- **Fixed Locations**: 12 bug sites across 7 adapters (TypeAware, Memory, FileSystem, GCS, S3, R2, OPFS, Azure)
- **Solution**: Allow optional metadata with `metadata: (metadata || {}) as NounMetadata`
- - **Result**: Workshop team UNBLOCKED - VFS entities now queryable
+ - **Result**: a consumer team UNBLOCKED - VFS entities now queryable
* **neural**: Fix SmartExtractor weighted score threshold bug (28 test failures → 4)
- **Root Cause**: Single signal with 0.8 confidence × 0.2 weight = 0.16 < 0.60 threshold
@@ -2026,7 +2862,7 @@ Clean separation between VFS (Virtual File System) entities and knowledge graph
### 🐛 Critical Bug Fixes
* **vfs.initializeRoot()**: add includeVFS to prevent duplicate root creation
- - **Critical Fix**: VFS init was creating ~10 duplicate root entities (Workshop team issue)
+ - **Critical Fix**: VFS init was creating ~10 duplicate root entities (a consumer team issue)
- **Root Cause**: `initializeRoot()` called `brain.find()` without `includeVFS: true`, never found existing VFS root
- **Impact**: Every `vfs.init()` created a new root, causing empty `readdir('/')` results
- **Solution**: Added `includeVFS: true` to root entity lookup (line 171)
@@ -2120,7 +2956,7 @@ const files = await vfs.search('documentation')
- **Root Cause**: HNSW `rebuild()` and Graph `rebuild()` methods still called `getNounsWithPagination()`/`getVerbsWithPagination()` repeatedly
- Each pagination call triggered `getAllShardedFiles()` reading all 256 shard directories
- For 1,157 entities: MetadataIndex (2-3s) + HNSW (~20s) + Graph (~10s) = **30-35 seconds total**
- - Workshop team reported: "v4.2.3 is at batch 7 after ~60 seconds" - still far from claimed 100x improvement
+ - a consumer team reported: "v4.2.3 is at batch 7 after ~60 seconds" - still far from claimed 100x improvement
- **Solution**: Apply v4.2.3 adaptive loading pattern to ALL 3 indexes
- **FileSystemStorage/MemoryStorage/OPFSStorage**: Load all entities at once (limit: 10000000)
- **Cloud storage (GCS/S3/R2/Azure)**: Keep pagination (native APIs are efficient)
@@ -2141,7 +2977,7 @@ const files = await vfs.search('documentation')
- Graph Index: Loads all verbs at once for local, paginated for cloud (lines 300-361)
- Pattern matches v4.2.3 MetadataIndex implementation exactly
- Zero config: Completely automatic based on storage adapter type
- - **Resolution**: Fully resolves Workshop team's v4.2.x performance regression
+ - **Resolution**: Fully resolves a consumer team's v4.2.x performance regression
- **Files Changed**:
- `src/hnsw/hnswIndex.ts` (updated rebuild() with adaptive loading)
- `src/graph/graphAdjacencyIndex.ts` (updated rebuild() with adaptive loading)
@@ -2166,7 +3002,7 @@ const files = await vfs.search('documentation')
- Pagination was designed for cloud storage socket exhaustion
- FileSystem doesn't need pagination - can handle loading thousands of entities at once
- Eliminates repeated directory scans: 3 batches × 256 dirs → 1 batch × 256 dirs
- - **Workshop Team**: This resolves the v4.2.2 stalling issue - rebuild will now complete in seconds
+ - **A consumer team**: This resolves the v4.2.2 stalling issue - rebuild will now complete in seconds
- **Files Changed**: `src/utils/metadataIndex.ts` (rebuilt() method with adaptive loading strategy)
### [4.2.2](https://github.com/soulcraftlabs/brainy/compare/v4.2.1...v4.2.2) (2025-10-23)
@@ -2208,7 +3044,7 @@ const files = await vfs.search('documentation')
- Universal: All storage adapters (FileSystem, GCS, S3, R2, Memory, OPFS)
- **Zero Config**: Completely automatic, no configuration needed
- **Self-Healing**: Gracefully handles missing/corrupt registry (rebuilds once)
- - **Impact**: Fixes Workshop team bug report - production-ready at billion scale
+ - **Impact**: Fixes a consumer team bug report - production-ready at billion scale
- **Files Changed**: `src/utils/metadataIndex.ts` (added saveFieldRegistry/loadFieldRegistry methods, updated init/flush)
### [4.2.0](https://github.com/soulcraftlabs/brainy/compare/v4.1.4...v4.2.0) (2025-10-23)
@@ -2252,7 +3088,7 @@ const files = await vfs.search('documentation')
- Fixed property access bugs: `verb.target` → `verb.to`, `verb.verb` → `verb.type`
- Added comprehensive integration tests (14 tests covering all query patterns)
- Updated JSDoc documentation with usage examples
- - **Impact**: Resolves Workshop team bug where 524 imported relationships were inaccessible
+ - **Impact**: Resolves a consumer team bug where 524 imported relationships were inaccessible
- **Breaking**: None - fully backward compatible
### [4.1.2](https://github.com/soulcraftlabs/brainy/compare/v4.1.1...v4.1.2) (2025-10-21)
diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 00000000..568c10db
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,208 @@
+# Brainy - Claude Code Project Guide
+
+This file provides guidance for Claude Code (and human contributors) when working on the Brainy codebase.
+
+## Cross-Project Coordination
+
+Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
+
+**At session START:** Read the handoff. Find rows where Owner = Brainy. Act on those first.
+
+**At session END:** Mark completed actions ✅, delete rows you finished, delete threads with zero remaining actions. File must not grow. **If you shipped anything consumers need to know about, update `RELEASES.md` before closing.**
+
+**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
+
+**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`)
+
+---
+
+## Project Overview
+
+Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraft/brainy` on npm under the MIT license.
+
+## Getting Started
+
+```bash
+npm install # Install dependencies
+npm run build # Build the project
+npm test # Run test suite (Vitest)
+```
+
+## Architecture
+
+Full architecture reference: `.claude/skills/architecture.md`
+
+### Core Systems
+- **Storage** (`src/storage/`): Pluggable storage backends via StorageAdapter interface (`src/coreTypes.ts`)
+- **Vector Search** (`src/hnsw/`): HNSW approximate nearest neighbor search
+- **Graph Engine** (`src/graph/`): Relationship traversal with adjacency index and pathfinding
+- **Metadata Index** (`src/utils/metadataIndex.ts`): O(1) exact match, O(log n) range queries
+- **Triple Intelligence** (`src/triple/`): Unified query combining all three intelligence types
+- **Aggregation Engine** (`src/aggregation/`): Write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows
+- **Virtual Filesystem** (`src/vfs/`): Full VFS with semantic search
+
+### Type System
+- **NounType** (42 types): Entity classification -- Person, Concept, Collection, Document, Task, etc.
+- **VerbType** (127 types): Relationship types -- Contains, RelatedTo, PartOf, Creates, DependsOn, etc.
+- Defined in `src/types/graphTypes.ts`
+
+## Code Standards
+
+### TypeScript
+- Strict mode enabled
+- Target: ES2020, NodeNext module resolution
+- All new code must be TypeScript
+- Follow existing patterns -- read related code before writing
+
+### Quality
+- All code must compile without errors
+- All code must have working tests that exercise real behavior
+- No stub returns (`return {} as any`)
+- No incomplete implementations with TODO comments
+- If something can't be fully implemented, throw an explicit error rather than faking it
+
+### Verification Before Code Changes
+1. Check that interfaces and methods actually exist before using them
+2. Check that type properties are in the type definitions
+3. Run `npm test` -- tests must pass
+4. Run `npm run build` -- build must succeed
+
+### Testing
+- Framework: Vitest
+- Tests in `tests/` (unit, integration, benchmarks, comprehensive)
+- Use in-memory storage for speed where possible
+- Tests must exercise real behavior, not mock it
+- Benchmarks are in `tests/benchmarks/` (not tests/performance/)
+
+## Commit Conventions
+
+Use [Conventional Commits](https://www.conventionalcommits.org/):
+
+```
+feat: add new feature (minor version bump)
+fix: resolve bug (patch version bump)
+docs: update documentation (patch version bump)
+perf: improve performance (patch version bump)
+refactor: restructure code (patch version bump)
+test: add/update tests (patch version bump)
+```
+
+**Important:** Never use `BREAKING CHANGE` in commit messages. Major version bumps are manual decisions only (`npm run release:major`).
+
+## Docs Pipeline — soulcraft.com/docs
+
+Docs in `docs/**/*.md` are published with the npm package (included in `files`) and synced to soulcraft.com/docs on every portal deploy. Frontmatter controls what appears publicly.
+
+### Docs check triggers
+
+Run the docs check whenever the user says ANY of:
+- "commit, publish, release" / "release" / "publish"
+- "update the docs" / "make sure docs are accurate" / "check the docs"
+- "review docs" / "clean up docs"
+
+### Pre-release docs check (MANDATORY before every release)
+
+When the user says "commit, publish, release" or any variation, **before committing**:
+
+1. **Scan all files changed in this session** (and any recently added `docs/*.md` files)
+2. For each changed/new doc, decide: is this useful to external users?
+ - **Yes** → ensure it has complete frontmatter (add or update it)
+ - **No** (internal, migration, dev-only) → ensure it has no frontmatter or `public: false`
+3. For docs that already have frontmatter, verify:
+ - `description` still matches the actual content
+ - `next` links still exist and are still the right follow-up pages
+ - `title` matches the doc's h1
+4. Include frontmatter changes in the commit
+
+### Frontmatter format
+
+```yaml
+---
+title: Human-readable title
+slug: category/page-name # URL: soulcraft.com/docs/category/page-name
+public: true # false or absent = not published
+category: getting-started | concepts | guides | api
+template: guide | concept | api # controls layout on soulcraft.com
+order: 1 # sidebar position within category (lower = first)
+description: One sentence. What this doc covers and why it matters.
+next: # "Next steps" links shown at bottom of page
+ - category/other-slug
+---
+```
+
+### Category guide
+
+| category | use for |
+|----------|---------|
+| `getting-started` | installation, quick start, first steps |
+| `concepts` | how the system works, mental models |
+| `guides` | how to do specific things, recipes |
+| `api` | method reference, signatures, parameters |
+
+### What stays internal (no frontmatter / `public: false`)
+
+- Release guides, developer learning paths
+- Migration guides for old versions (v3→v4, v5.11)
+- Architecture analysis docs (clustering algorithms, etc.)
+- Anything in `docs/internal/`
+- Deployment/ops/cost docs (cloud-run, kubernetes, cost-optimization)
+
+## Release Process
+
+Fully automated via `scripts/release.sh`:
+
+```bash
+npm run release:dry # Preview (no changes)
+npm run release:patch # Bug fixes
+npm run release:minor # New features
+npm run release:major # Breaking changes (rare, manual decision)
+```
+
+The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release.
+
+After a successful release, remind the user:
+> "Published. Deploy portal to pick up the new docs → go to the portal project and deploy."
+
+Do NOT deploy portal from here. Portal is always deployed separately from within the portal project.
+
+## Closed-Source Product Names — HARD RULE
+
+Brainy is the only Soulcraft open-source project. Nothing in this repo — code, JSDoc, tests,
+docs, RELEASES.md, CHANGELOG.md, commit messages — may reference closed-source Soulcraft
+products by name (Workshop, Venue, Memory, Muse, Hall, Forge, Academy, Pulse, Heart,
+Collective, SDK) or by their specific class/method names (`BookingDraftService`,
+`getDemandHeatmap`, `systemKind`, etc.).
+
+When recording a consumer-reported bug, regression scenario, or release note:
+- Refer to "a consumer", "a downstream application", "a production deployment", or "an
+ internal report" — never name the product.
+- All doc examples must use generic domain values (`'employee'`, `'customer'`, `'invoice'`,
+ `'milestone'`, `OrderService`, `/orders/...`), not product-specific schemas.
+- Internal session artifacts (`.strategy/`, `~/.claude/plans/`, handoff files outside the
+ repo) MAY name products — those are not public.
+
+If you catch yourself typing a product name into a tracked file, stop and rephrase.
+
+## Performance Claims
+
+When documenting performance characteristics:
+- **MEASURED**: Cite the test file and line number
+- **PROJECTED**: Clearly label as extrapolated from tested scale
+- Never claim a performance figure without context or evidence
+
+## Debugging
+
+When a bug persists through 2+ fix attempts, switch to systematic debugging:
+1. Add comprehensive logging at every step
+2. Test with production-like data
+3. Trace the complete execution path
+4. Check both library code and consumer code
+5. Verify with actual test execution before declaring fixed
+
+## Key Paths
+
+- Main class: `src/brainy.ts`
+- Public API: `src/index.ts` (38+ exports)
+- Storage interface: `src/coreTypes.ts`
+- Type definitions: `src/types/`
+- Strategy/planning docs: `.strategy/` (gitignored, not public)
diff --git a/README.md b/README.md
index e0f9e855..d1342f52 100644
--- a/README.md
+++ b/README.md
@@ -1,745 +1,217 @@
-# Brainy
-
- 
+
-[](https://www.npmjs.com/package/@soulcraft/brainy)
-[](https://www.npmjs.com/package/@soulcraft/brainy)
-[](LICENSE)
-[](https://www.typescriptlang.org/)
+Brainy
-## The Knowledge Operating System
+
+ Three database paradigms. One API. Zero configuration.
+ The in-process knowledge database for TypeScript — vector search, graph traversal,
+ and metadata filtering unified in a single query.
+
-**Every piece of knowledge in your application — living, connected, and intelligent.**
+
+ 
+ 
+ 
+ 
+ 
+ 
+
-Stop fighting with vector databases, graph databases, and document stores. Stop stitching together Pinecone + Neo4j + MongoDB. **Brainy does all three, in one elegant API, from prototype to planet-scale.**
-
-```javascript
-const brain = new Brainy()
-await brain.init()
-
-// That's it. You now have semantic search, graph relationships,
-// and document filtering. Zero configuration. Just works.
-```
-
-**Built by developers who were tired of:**
-- Spending weeks configuring embeddings, indexes, and schemas
-- Choosing between vector similarity OR graph relationships OR metadata filtering
-- Rewriting everything when you need to scale from 1,000 to 1,000,000,000 entities
-
-**Brainy makes the impossible simple: All three paradigms. One API. Any scale.**
+
+ Quick start ·
+ One query ·
+ Features ·
+ Scale with Cor ·
+ Docs
+
---
-## 👉 Choose Your Path
+Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact **three ways at once** and lets one call query them together:
-**New to Brainy? Pick your starting point:**
+| You write | Brainy indexes it as | You query it with |
+|---|---|---|
+| `data: 'Ada wrote the first program'` | a **384-dim vector** (local embedding — no API key) | `find({ query: 'computing pioneers' })` |
+| `metadata: { field: 'CS', year: 1843 }` | **structured fields** (O(1) exact, O(log n) range) | `find({ where: { year: { lessThan: 1900 } } })` |
+| `relate({ from: ada, to: babbage })` | a **typed, directed graph edge** | `find({ connected: { to: babbage, depth: 2 } })` |
-### 🚀 Path 1: I want to build something NOW
-**→ [Complete API Reference](docs/api/README.md)** ⭐ **Most developers start here** ⭐
-- Every method documented with examples
-- Quick start in 60 seconds
-- 1,870 lines of copy-paste ready code
-- **This is your primary resource**
+It runs **inside your process** — no server, no Docker, nothing to operate — and persists to plain files you can snapshot with a hard link.
-### 🧠 Path 2: I want to understand the big picture first
-**→ Keep reading below** for demos, architecture, and use cases
+**New here?** → **[What is Brainy? — plain-language overview, no jargon](docs/eli5.md)**
-### 📊 Path 3: I'm evaluating database options
-**→ Jump to [Why Revolutionary](#why-brainy-is-revolutionary)** or **[Benchmarks](#benchmarks)**
-
----
-
-## See It In Action
-
-**30 seconds to understand why Brainy is different:**
-
-```javascript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// Add knowledge with context
-const reactId = await brain.add({
- data: "React is a JavaScript library for building user interfaces",
- type: NounType.Concept,
- metadata: { category: "frontend", year: 2013 }
-})
-
-const nextId = await brain.add({
- data: "Next.js framework for React with server-side rendering",
- type: NounType.Concept,
- metadata: { category: "framework", year: 2016 }
-})
-
-// Create relationships
-await brain.relate({ from: nextId, to: reactId, type: VerbType.BuiltOn })
-
-// NOW THE MAGIC: Query with natural language
-const results = await brain.find({
- query: "modern frontend frameworks", // 🔍 Vector similarity
- where: { year: { greaterThan: 2015 } }, // 📊 Document filtering
- connected: { to: reactId, depth: 2 } // 🕸️ Graph traversal
-})
-
-// ALL THREE PARADIGMS. ONE QUERY. 10ms response time.
-```
-
-**This is impossible with traditional databases.** Brainy makes it trivial.
-
----
-
-## Quick Start
+## Quick start
```bash
-npm install @soulcraft/brainy
+bun add @soulcraft/brainy # Bun ≥ 1.1 — recommended
+npm install @soulcraft/brainy # Node.js ≥ 22
```
-### Your First Knowledge Graph (60 seconds)
-
```javascript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-const brain = new Brainy()
+const brain = new Brainy() // in-memory; one line swaps to disk
await brain.init()
-// Add knowledge
-const jsId = await brain.add({
- data: "JavaScript is a programming language",
- type: NounType.Concept,
- metadata: { category: "language", year: 1995 }
+// Text auto-embeds locally; metadata auto-indexes
+const react = await brain.add({
+ data: 'React is a JavaScript library for building user interfaces',
+ type: NounType.Concept,
+ subtype: 'library',
+ metadata: { category: 'frontend', year: 2013 }
})
-const nodeId = await brain.add({
- data: "Node.js runtime environment",
- type: NounType.Concept,
- metadata: { category: "runtime", year: 2009 }
+const next = await brain.add({
+ data: 'Next.js is a React framework with server-side rendering',
+ type: NounType.Concept,
+ subtype: 'framework',
+ metadata: { category: 'frontend', year: 2016 }
})
-// Create relationships
-await brain.relate({ from: nodeId, to: jsId, type: VerbType.Executes })
+await brain.relate({ from: next, to: react, type: VerbType.DependsOn, subtype: 'runtime' })
+```
-// Query with Triple Intelligence
+## One query, three engines
+
+```javascript
const results = await brain.find({
- query: "JavaScript", // 🔍 Vector
- where: { category: "language" }, // 📊 Document
- connected: { from: nodeId, depth: 1 } // 🕸️ Graph
+ query: 'modern frontend frameworks', // vector — what it means
+ where: { year: { greaterThan: 2015 } }, // metadata — what it is
+ connected: { to: react, depth: 2 } // graph — what it touches
})
```
-**Done.** No configuration. No complexity. Production-ready from day one.
+Every clause is optional; any combination composes. Under the hood Brainy plans the query across an HNSW vector index, a roaring-bitmap field index, and an adjacency graph index — and re-validates every result against your predicate before returning it, so a corrupt index can never hand you a wrong answer.
-**→ Ready to dive deeper? [Complete API Documentation](docs/api/README.md)** has every method with examples.
+## Feature tour
----
+### The database is a value
-## Entity Extraction (NEW in v5.7.6)
-
-**Extract entities from text with AI-powered classification:**
+Pin it, rewind it, fork it. Snapshot isolation without a server.
```javascript
-import { Brainy, NounType } from '@soulcraft/brainy'
+const db = brain.now() // pin current state — O(1)
-const brain = new Brainy()
-await brain.init()
+await brain.transact([ // atomic all-or-nothing, CAS-guarded
+ { op: 'update', id: order, metadata: { status: 'paid' } },
+ { op: 'relate', from: invoice, to: order, type: VerbType.References, subtype: 'billing' }
+], { ifAtGeneration: db.generation })
-// Extract all entities
-const entities = await brain.extractEntities('John Smith founded Acme Corp in New York')
-// Returns:
-// [
-// { text: 'John Smith', type: NounType.Person, confidence: 0.95 },
-// { text: 'Acme Corp', type: NounType.Organization, confidence: 0.92 },
-// { text: 'New York', type: NounType.Location, confidence: 0.88 }
-// ]
+await db.get(order) // still 'pending' — pinned forever
+await brain.get(order) // 'paid' — live
-// Extract with filters
-const people = await brain.extractEntities(resume, {
- types: [NounType.Person],
- confidence: 0.8
-})
-
-// Advanced: Direct access to extractors
-import { SmartExtractor } from '@soulcraft/brainy'
-
-const extractor = new SmartExtractor(brain, { minConfidence: 0.7 })
-const result = await extractor.extract('CEO', {
- formatContext: { format: 'excel', columnHeader: 'Title' }
-})
+const lastWeek = await brain.asOf(Date.now() - 7 * 86_400_000) // full query surface, past state
+const whatIf = await db.with([{ op: 'remove', id: order }]) // speculative — never touches disk
+await brain.now().persist('/backups/today') // instant hard-link snapshot
```
-**Features:**
-- 🎯 **4-Signal Ensemble** - ExactMatch (40%) + Embedding (35%) + Pattern (20%) + Context (5%)
-- 📊 **Format Intelligence** - Adapts to Excel, CSV, PDF, YAML, DOCX, JSON, Markdown
-- ⚡ **Fast** - ~15-20ms per extraction with LRU caching
-- 🌍 **42 Types** - Person, Organization, Location, Document, and 38 more
+**[Consistency model](docs/concepts/consistency-model.md)** · **[Snapshots & time travel](docs/guides/snapshots-and-time-travel.md)**
-**→ [Neural Extraction Guide](docs/neural-extraction.md)** | **[Import Preview Mode](docs/neural-extraction.md#import-preview-mode)**
+### Local embeddings — no API keys
----
-
-## From Prototype to Planet Scale
-
-**The same API. Zero rewrites. Any scale.**
-
-### 👤 Individual Developer → Weekend Prototype
-```javascript
-const brain = new Brainy() // Zero config, starts in memory
-await brain.init()
-```
-**Perfect for:** Hackathons, side projects, prototyping, learning
-
-### 👥 Small Team → Production MVP
-```javascript
-const brain = new Brainy({
- storage: { type: 'filesystem', path: './data', compression: true }
-})
-```
-**Scale:** Thousands to hundreds of thousands • **Performance:** <5ms queries
-**→ [Production Service Architecture](docs/PRODUCTION_SERVICE_ARCHITECTURE.md)** — Singleton patterns, caching, and scaling for Bun/Node.js services
-
-### 🏢 Growing Company → Multi-Million Scale
-```javascript
-const brain = new Brainy({
- storage: { type: 's3', s3Storage: { bucketName: 'my-kb', region: 'us-east-1' } },
- hnsw: { typeAware: true } // 87% memory reduction
-})
-```
-**Scale:** Millions of entities • **Performance:** <10ms queries, 12GB @ 10M entities
-
-### 🌍 Enterprise → Billion+ Scale
-```javascript
-const brain = new Brainy({
- storage: { type: 'gcs', gcsStorage: { bucketName: 'global-kb' } },
- hnsw: { typeAware: true, M: 32, efConstruction: 400 }
-})
-```
-**Scale:** Billions (tested @ 1B+) • **Performance:** 18ms queries, 50GB memory
-**Cost:** $138k/year → $6k/year with intelligent tiering (96% savings)
-
-**→ [Capacity Planning Guide](docs/operations/capacity-planning.md)** | **[Cost Optimization](docs/operations/)**
-
-### 🎯 The Point
-
-**Start simple. Scale infinitely. Never rewrite.**
-
-Most systems make you choose: Simple (SQLite) OR Scalable (Kubernetes + 7 databases).
-**Brainy gives you both.** Starts simple as SQLite. Scales like Google.
-
----
-
-## Why Brainy Is Revolutionary
-
-### 🧠 **Triple Intelligence™** — The Impossible Made Possible
-
-**The world's first to unify three database paradigms in ONE API:**
-
-| What You Get | Like Having | But Unified |
-|-------------|-------------|-------------|
-| 🔍 **Vector Search** | Pinecone, Weaviate | Find by meaning |
-| 🕸️ **Graph Relationships** | Neo4j, ArangoDB | Navigate connections |
-| 📊 **Document Filtering** | MongoDB, Elasticsearch | Query metadata |
-
-**Every other system makes you choose.** Brainy does all three together.
-
-**Why this matters:** Your data isn't just vectors or just documents or just graphs. It's all three at once. A research paper is semantically similar to other papers (vector), written by an author (graph), and published in 2023 (document). **Brainy is the only system that understands this.**
-
-### 🎯 **42 Noun Types × 127 Verb Types = Universal Protocol**
-
-Model **any domain** with mathematical completeness:
-
-```
-42 Nouns × 127 Verbs × ∞ Metadata = 5,334+ base combinations
-Stage 3 CANONICAL: 96-97% coverage of all human knowledge
-```
-
-**Real-world expressiveness:**
-- Healthcare: `Patient → diagnoses → Condition`
-- Finance: `Account → transfers → Transaction`
-- Manufacturing: `Product → assembles → Component`
-- Education: `Student → completes → Course`
-- **YOUR domain** → Your types + relationships = Your knowledge graph
-
-[→ See the Mathematical Proof](docs/architecture/noun-verb-taxonomy.md)
-
-### ⚡ **Zero Configuration Philosophy**
-
-**We hate configuration files. So we eliminated them.**
+Strings embed on-device with a bundled MiniLM model (WASM). Semantic search works offline, in CI, and on air-gapped machines, at zero cost per call. Hybrid keyword + semantic ranking is the default:
```javascript
-const brain = new Brainy() // Auto-detects everything
-await brain.init() // Optimizes for your environment
+await brain.find({ query: 'David Smith' }) // auto: text + semantic
+await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // semantic only
```
-Brainy automatically:
-- Detects optimal storage (memory/filesystem/cloud)
-- Configures memory based on available RAM
-- Optimizes for containers (Docker/K8s)
-- Tunes indexes for your data patterns
-- Manages embedding models and caching
+### A typed graph, not a bag of edges
-**You write business logic. Brainy handles infrastructure.**
-
-### 🚀 **Git-Style Version Control** — Database & Entity Level (v5.0.0+)
-
-**Clone your entire database in <100ms. Track every entity change. Full Git-style workflow.**
+42 entity types × 127 relationship types form a shared vocabulary for any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), yours. Your own taxonomy layers on with `subtype`, enforced at write time:
```javascript
-// Fork instantly - Snowflake-style copy-on-write
-const experiment = await brain.fork('test-migration')
+await brain.add({ data: 'Avery Brooks', type: NounType.Person, subtype: 'employee' })
-// Make changes safely in isolation
-await experiment.add({ type: 'user', data: { name: 'Test User' } })
-await experiment.updateAll({ /* migration logic */ })
-
-// Commit your work
-await experiment.commit({ message: 'Add test user', author: 'dev@example.com' })
-
-// Switch to experimental branch to make it active
-await brain.checkout('test-migration')
-
-// Time-travel: Query database at any past commit (read-only)
-const commits = await brain.getHistory({ limit: 10 })
-const snapshot = await brain.asOf(commits[5].id)
-const pastResults = await snapshot.find({ query: 'historical data' })
-await snapshot.close()
-
-// Entity versioning: Track changes to individual entities (v5.3.0+)
-const userId = await brain.add({ type: 'user', data: { name: 'Alice' } })
-await brain.versions.save(userId, { tag: 'v1.0', description: 'Initial profile' })
-
-await brain.update(userId, { data: { name: 'Alice Smith', role: 'admin' } })
-await brain.versions.save(userId, { tag: 'v2.0', description: 'Added role' })
-
-// Compare versions or restore previous state
-const diff = await brain.versions.compare(userId, 1, 2) // See what changed
-await brain.versions.restore(userId, 1) // Restore v1.0
+brain.counts.bySubtype(NounType.Person) // O(1) — { employee: 12, customer: 847 }
+brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
```
-**Database-level version control (v5.0.0):**
-- ✅ `fork()` - Instant clone in <100ms
-- ✅ `merge()` - Merge with conflict resolution
-- ✅ `commit()` - Snapshot state
-- ✅ `asOf()` - Time-travel queries (query at any commit)
-- ✅ `getHistory()` - View commit history
-- ✅ `checkout()`, `listBranches()` - Full branch management
-- ✅ CLI support for all features
+**[Type system](docs/architecture/noun-verb-taxonomy.md)** · **[Subtypes & facets](docs/guides/subtypes-and-facets.md)**
-**Entity-level version control (v5.3.0):**
-- ✅ `versions.save()` - Save entity snapshots with tags
-- ✅ `versions.restore()` - Restore previous versions
-- ✅ `versions.compare()` - Diff between versions
-- ✅ `versions.list()` - View version history
-- ✅ Automatic deduplication (content-addressable storage)
-
-**How it works:** Snowflake-style COW shares HNSW index structures, copying only modified nodes (10-20% memory overhead).
-
-**Perfect for:** Safe migrations, A/B testing, feature branches, distributed development, time-travel debugging, audit trails, document versioning, compliance tracking
-
-[→ See Full Documentation](docs/features/instant-fork.md)
-
----
-
-## What Can You Build?
-
-**If your app needs to remember, understand, or connect information — Brainy makes it trivial.**
-
-### 🤖 **AI Agents with Perfect Memory**
-Give your AI unlimited context that persists forever. Not just chat history — true understanding of relationships, evolution, and meaning over time.
-
-**Examples:** Personal assistants, code assistants, conversational AI, research agents
-
-### 📚 **Living Documentation & Knowledge Bases**
-Documentation that understands itself. Auto-links related concepts, detects outdated information, finds connections across your entire knowledge base.
-
-**Examples:** Internal wikis, research platforms, smart documentation, learning systems
-
-### 🔍 **Semantic Search at Any Scale**
-Find by meaning, not keywords. Search codebases, research papers, customer data, or media libraries with natural language.
-
-**Examples:** Code search, research platforms, content discovery, recommendation engines
-
-### 🏢 **Enterprise Knowledge Management**
-Corporate memory that never forgets. Track every customer interaction, product evolution, and business relationship.
-
-**Examples:** CRM systems, product catalogs, customer intelligence, institutional knowledge
-
-### 🎮 **Rich Interactive Experiences**
-NPCs that remember. Characters that persist across stories. Worlds that evolve based on real relationships.
-
-**Examples:** Game worlds, interactive fiction, educational platforms, creative tools
-
-### 🎨 **Content & Media Platforms**
-Every asset knows its relationships. Intelligent tagging, similarity-based discovery, and relationship-aware management.
-
-**Examples:** DAM systems, media libraries, writing assistants, content management
-
-**The pattern:** Knowledge that needs to live, connect, and evolve. That's what Brainy was built for.
-
----
-
-## Core Features
-
-### 🧠 **Natural Language Queries**
+### Graph analytics built in
```javascript
-// Ask naturally - Brainy understands
-await brain.find("recent React components with tests")
-await brain.find("JavaScript libraries similar to Vue")
-
-// Or use structured Triple Intelligence queries
-await brain.find({
- query: "React",
- where: { type: "library", year: { greaterThan: 2020 } },
- connected: { to: "JavaScript", depth: 2 }
-})
+await brain.graph.rank() // which entities matter most (centrality)
+await brain.graph.communities() // natural clusters
+await brain.graph.path(a, b) // how two things connect
+await brain.graph.subgraph([seed], { depth: 2 }) // bounded neighborhood → { nodes, edges }
+await brain.graph.export() // whole graph, one O(N+E) streaming pass
```
-**→ [See all query methods in API Reference](docs/api/README.md#search--query)**
+### Write-time aggregations
-### 🌐 **Virtual Filesystem** — Intelligent File Management
+`SUM` / `COUNT` / `AVG` / `MIN` / `MAX` with `GROUP BY` and time windows, maintained incrementally on every write — reads are O(1) lookups, not scans. **[Aggregation guide](docs/guides/aggregation.md)**
-Build file explorers and IDEs that never crash:
+### Import anything
```javascript
-const vfs = brain.vfs()
-
-// Tree-aware operations prevent infinite recursion
-const tree = await vfs.getTreeStructure('/projects', { maxDepth: 3 })
-
-// Semantic file search
-const reactFiles = await vfs.search('React components with hooks')
-```
-
-**[📖 VFS Quick Start →](docs/vfs/QUICK_START.md)** | **[Common Patterns →](docs/vfs/COMMON_PATTERNS.md)** | **[Neural Extraction →](docs/vfs/NEURAL_EXTRACTION.md)**
-
-### 🚀 **Import Anything** — CSV, Excel, PDF, URLs
-
-```javascript
-await brain.import('customers.csv') // Auto-detects everything
-await brain.import('sales-data.xlsx', { excelSheets: ['Q1', 'Q2'] })
-await brain.import('research-paper.pdf', { pdfExtractTables: true })
+await brain.import('customers.csv')
+await brain.import('sales.xlsx') // every sheet
+await brain.import('research-paper.pdf') // tables extracted
await brain.import('https://api.example.com/data.json')
```
-**[📖 Complete Import Guide →](docs/guides/import-anything.md)**
+Entities auto-classify on the way in; `brain.extractEntities(text)` exposes the same NER ensemble directly. **[Import guide](docs/guides/import-anything.md)**
-### 🧠 **Neural API** — Advanced Semantic Analysis
+### A filesystem that understands content
```javascript
-// Clustering, similarity, outlier detection, visualization
-const clusters = await brain.neural.clusters({ algorithm: 'kmeans' })
-const similarity = await brain.neural.similar('item1', 'item2')
-const outliers = await brain.neural.outliers(0.3)
-const vizData = await brain.neural.visualize({ maxNodes: 100 })
+await brain.vfs.writeFile('/docs/readme.md', 'Project documentation')
+await brain.vfs.search('React components with hooks') // semantic file search
```
----
+**[VFS quick start](docs/vfs/QUICK_START.md)**
-## Framework Integration
+### Operations-grade by default
-**Works with any modern framework.** React, Vue, Angular, Svelte, Solid.js — your choice.
+- **Single-writer, many-reader** — an exclusive lock protects the data directory; `Brainy.openReadOnly()` and the `brainy inspect` CLI examine a live brain from another process, safely.
+- **Self-upgrading data files** — a 7.x brain opens under 8.x and migrates itself behind an observable lock (`getIndexStatus().migration`), with an automatic pre-upgrade backup. No migration scripts.
+- **No silent wrong answers** — cold-open guards self-heal or throw typed errors (`MetadataIndexNotReadyError`, `GraphIndexNotReadyError`); they never return `[]` for data that exists.
-```javascript
-// React
-const [brain] = useState(() => new Brainy())
-useEffect(() => { brain.init() }, [])
+**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
-// Vue
-async mounted() { this.brain = await new Brainy().init() }
+## From laptop to hundreds of millions
-// Angular
-@Injectable() export class BrainyService { brain = new Brainy() }
-```
-
-**Supports:** All bundlers (Webpack, Vite, Rollup) • SSR/SSG • Edge runtimes • Browser/Node.js
-
-**[📖 Framework Integration Guide →](docs/guides/framework-integration.md)** | **[Next.js →](docs/guides/nextjs-integration.md)** | **[Vue →](docs/guides/vue-integration.md)**
-
----
-
-## Storage — From Memory to Planet-Scale
-
-### Development → Just Works
-```javascript
-const brain = new Brainy() // Memory storage, zero config
-```
-
-### Production → Persistence with Compression
-```javascript
-const brain = new Brainy({
- storage: { type: 'filesystem', path: './data', compression: true }
-})
-// 60-80% space savings with gzip
-```
-
-### Cloud → AWS, GCS, Azure, Cloudflare R2
-```javascript
-// AWS S3 / Cloudflare R2
-const brain = new Brainy({
- storage: {
- type: 's3',
- s3Storage: {
- bucketName: 'my-knowledge-base',
- region: 'us-east-1'
- }
- }
-})
-
-// Enable Intelligent-Tiering: 96% cost savings
-await brain.storage.enableIntelligentTiering('entities/', 'auto-tier')
-```
-
-**Cost optimization at scale:**
-
-| Scale | Standard | With Intelligent Tiering | Annual Savings |
-|-------|----------|--------------------------|----------------|
-| 5TB | $1,380 | $59 | $1,321 (96%) |
-| 50TB | $13,800 | $594 | $13,206 (96%) |
-| 500TB | $138,000 | $5,940 | $132,060 (96%) |
-
-**[📖 Cloud Storage Guide →](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[AWS Cost Optimization →](docs/operations/cost-optimization-aws-s3.md)** | **[GCS →](docs/operations/cost-optimization-gcs.md)** | **[Azure →](docs/operations/cost-optimization-azure.md)**
-
----
-
-## Production Features
-
-### 🎯 Type-Aware HNSW Indexing
-
-Efficient type-based organization for large-scale deployments:
-
-- **Type-based queries:** Faster via directory structure (measured at 1K-1M scale)
-- **Type count tracking:** 284 bytes (Uint32Array, measured)
-- **Billion-scale projections:** NOT tested at 1B entities (extrapolated from 1M)
-
-```javascript
-const brain = new Brainy({ hnsw: { typeAware: true } })
-```
-
-**[📖 How Type-Aware Indexing Works →](docs/architecture/data-storage-architecture.md)**
-
-### ⚡ Enterprise-Ready Operations (v4.0.0)
-
-- **Batch operations** with retry logic (1000x faster deletes)
-- **Gzip compression** (60-80% space savings)
-- **OPFS quota monitoring** (browser storage)
-- **Metadata/Vector separation** (billion-entity scalability)
-- **Circuit breakers & backpressure** (enterprise reliability)
-
-```javascript
-// Batch operations
-await brain.storage.batchDelete(keys, { maxRetries: 3 })
-
-// Monitor storage
-const status = await brain.storage.getStorageStatus()
-```
-
-### 📊 Adaptive Memory Management
-
-Auto-scales 2GB → 128GB+ based on environment:
-
-- Container-aware (Docker/K8s cgroups)
-- Environment-optimized (dev/staging/production)
-- Built-in cache monitoring with tuning recommendations
-
-```javascript
-const stats = brain.getCacheStats() // Performance insights
-```
-
-**[📖 Capacity Planning Guide →](docs/operations/capacity-planning.md)**
-
----
-
-## Benchmarks
-
-| Operation | Performance | Memory |
-|-----------|-------------|--------|
-| Initialize | 450ms | 24MB |
-| Add entity | 12ms | +0.1MB |
-| Vector search (1K) | 3ms | - |
-| Metadata filter (10K) | 0.8ms | - |
-| Bulk import (1K) | 2.3s | +8MB |
-| **10M entities** | **5.8ms** | **12GB** |
-| **1B entities** | **18ms** | **50GB** |
-
----
-
-## 🧠 Deep Dive: How Brainy Actually Works
-
-**Want to understand the magic under the hood?**
-
-### 🔍 Triple Intelligence & find() API
-Understand how vector search, graph relationships, and document filtering work together in one unified query:
-
-**[📖 Triple Intelligence Architecture →](docs/architecture/triple-intelligence.md)**
-**[📖 Natural Language Guide →](docs/guides/natural-language.md)**
-**[📖 API Reference: find() →](docs/api/README.md)**
-
-### 🗂️ Type-Aware Indexing & HNSW
-Learn about our indexing architecture with measured performance optimizations:
-
-**[📖 Data Storage Architecture →](docs/architecture/data-storage-architecture.md)**
-**[📖 Architecture Overview →](docs/architecture/overview.md)**
-
-### 📈 Scaling: Individual → Planet
-Understand how the same code scales from prototype to billions of entities:
-
-**[📖 Capacity Planning →](docs/operations/capacity-planning.md)**
-**[📖 Cloud Deployment Guide →](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)**
-
-### 🎯 The Universal Type System
-Explore the mathematical foundation: 42 nouns × 127 verbs = Stage 3 CANONICAL taxonomy:
-
-**[📖 Noun-Verb Taxonomy →](docs/architecture/noun-verb-taxonomy.md)**
-
----
-
-## CLI Tools
+Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**:
```bash
-npm install -g brainy
-
-brainy add "JavaScript is awesome" --metadata '{"type":"opinion"}'
-brainy find "awesome programming languages"
-brainy search "programming"
+npm install @soulcraft/cor
```
-47 commands available, including storage management, imports, and neural operations.
+```javascript
+const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } })
+await brain.init() // @soulcraft/cor detected — same code, native engines underneath
+```
----
+Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
-## 📖 Complete Documentation
+Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both.
-**For most developers:** Start with the **[Complete API Reference](docs/api/README.md)** ⭐
+## Performance
-This comprehensive guide includes:
-- ✅ Every method with parameters, returns, and examples
-- ✅ Quick start in 60 seconds
-- ✅ Core CRUD → Advanced features (branching, versioning, time-travel)
-- ✅ TypeScript types and patterns
-- ✅ 1,870 lines of copy-paste ready code
+- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
+- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
+- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
----
+## Use cases
-### 🎯 Essential Reading (Start Here)
+**AI agent memory** — persistent semantic recall with relationship tracking · **Knowledge bases** — auto-linking and meaning-aware navigation · **Semantic search** over codebases, documents, media · **Enterprise data** — CRM, catalogs, institutional memory · **Games & simulations** — worlds and characters that remember.
-1. **[📖 Complete API Reference](docs/api/README.md)** ⭐ **START HERE** ⭐
- - Your primary resource for building with Brainy
- - Every method documented with working examples
+## Documentation
-2. **[Filter & Query Syntax Guide](docs/FIND_SYSTEM.md)**
- - Complete reference for operators, compound filters, and optimization tips
-
-3. **[Natural Language Queries](docs/guides/natural-language.md)**
- - Master the `find()` method and Triple Intelligence queries
-
-4. **[v4.0.0 Migration Guide](docs/MIGRATION-V3-TO-V4.md)**
- - Upgrading from v3 (100% backward compatible)
-
-### 🧠 Core Concepts & Architecture
-
-- **[Triple Intelligence Architecture](docs/architecture/triple-intelligence.md)** — How vector + graph + document work together
-- **[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** — The universal type system (42 nouns × 127 verbs)
-- **[Transactions](docs/transactions.md)** — Atomic operations with automatic rollback
-- **[Architecture Overview](docs/architecture/overview.md)** — System design and components
-- **[Data Storage Architecture](docs/architecture/data-storage-architecture.md)** — Type-aware indexing and HNSW
-
-### ☁️ Production & Operations
-
-- **[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** — Deploy to AWS, GCS, Azure
-- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling to billions
-- **Cost Optimization:** **[AWS S3](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[Cloudflare R2](docs/operations/cost-optimization-cloudflare-r2.md)**
-
-### 🌐 Framework Integration
-
-- **[Framework Integration Guide](docs/guides/framework-integration.md)** — React, Vue, Angular, Svelte
-- **[Next.js Integration](docs/guides/nextjs-integration.md)**
-- **[Vue.js Integration](docs/guides/vue-integration.md)**
-
-### 🌳 Virtual Filesystem (VFS)
-
-- **[VFS Quick Start](docs/vfs/QUICK_START.md)** — Build file explorers that never crash
-- **[VFS Core Documentation](docs/vfs/VFS_CORE.md)**
-- **[Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md)** — AI-powered file navigation
-- **[Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md)**
-
-### 📦 Data Import & Processing
-
-- **[Import Anything Guide](docs/guides/import-anything.md)** — CSV, Excel, PDF, URLs with auto-detection
-
----
-
-## What's New in v4.0.0
-
-**Enterprise-scale cost optimization and performance improvements:**
-
-- 🎯 **96% cloud storage cost savings** with intelligent tiering (AWS, GCS, Azure)
-- ⚡ **1000x faster batch deletions** (533 entities/sec vs 0.5/sec)
-- 📦 **60-80% compression** with gzip (FileSystem storage)
-- 🔄 **Enhanced metadata/vector separation** for billion-scale deployments
-
-**[📖 Full v4.0.0 Changelog →](CHANGELOG.md)** | **[Migration Guide →](docs/MIGRATION-V3-TO-V4.md)** (100% backward compatible)
-
----
+| Start | Core | Going deeper |
+|---|---|---|
+| [Brainy explained simply](docs/eli5.md) | [API reference](docs/api/README.md) | [Architecture overview](docs/architecture/overview.md) |
+| [Installation](docs/guides/installation.md) | [Data model](docs/DATA_MODEL.md) | [Consistency model](docs/concepts/consistency-model.md) |
+| [Natural-language queries](docs/guides/natural-language.md) | [Query operators](docs/QUERY_OPERATORS.md) | [Multi-process model](docs/concepts/multi-process.md) |
+| | [Find system](docs/FIND_SYSTEM.md) | [Scaling](docs/SCALING.md) |
## Requirements
-**Bun 1.0+** (recommended) or **Node.js 22 LTS**
+**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
-```bash
-# Bun (recommended - best performance, single-binary deployment)
-bun install @soulcraft/brainy
+## Contributing & license
-# Node.js (fully supported)
-npm install @soulcraft/brainy
-```
-
-> **Why Bun?** Brainy's Candle WASM engine works seamlessly with `bun --compile` for standalone binary deployment. No external model files, no runtime downloads.
-
----
-
-## Why Brainy Exists
-
-**The Vision:** Traditional systems force you to choose between vector databases, graph databases, and document stores. You need all three, but combining them is complex and fragile.
-
-**Brainy solved the impossible:** One API. All three paradigms. Any scale.
-
-Like HTTP standardized web communication, **Brainy standardizes knowledge representation.** One protocol that any AI model understands. One system that scales from prototype to planet.
-
-**[📖 Read the Mathematical Proof →](docs/architecture/noun-verb-taxonomy.md)**
-
----
-
-## Enterprise & Support
-
-**🏢 Brain Cloud** — Managed Brainy with team sync, persistent memory, and enterprise connectors.
-Visit **[soulcraft.com](https://soulcraft.com)** for more information.
-
-**💖 Support Development:**
-- ⭐ Star us on **[GitHub](https://github.com/soulcraftlabs/brainy)**
-- 💝 Sponsor via **[GitHub Sponsors](https://github.com/sponsors/soulcraftlabs)**
-- 🐛 Report issues and contribute code
-- 📣 Share with your team and community
-
-**Brainy is 100% free and open source.** No paywalls, no premium tiers, no feature gates.
-
----
-
-## Contributing
-
-We welcome contributions! See **[CONTRIBUTING.md](CONTRIBUTING.md)** for guidelines.
-
----
-
-## License
-
-MIT © Brainy Contributors
-
----
-
-
- Built with ❤️ by the Brainy community
- The Knowledge Operating System
- From prototype to planet-scale • Zero configuration • Triple Intelligence™
-
+Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors.
diff --git a/RELEASES.md b/RELEASES.md
new file mode 100644
index 00000000..7799c6f6
--- /dev/null
+++ b/RELEASES.md
@@ -0,0 +1,2901 @@
+# @soulcraft/brainy — Release Notes for Consumers
+
+This file is the **quick reference for downstream sessions** tracking Brainy changes.
+Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/soulcraftlabs/brainy/releases
+
+**How to use:** Brainy is the underlying data engine for downstream applications. Read this when:
+- Upgrading `@soulcraft/brainy` in your application
+- Debugging data, query, or storage behaviour
+- A new Brainy feature is available that you want to adopt
+
+## Removed APIs — 7.x → 8.x (the complete ledger)
+
+Every public API removed at the 8.0 major, with its sanctioned replacement. If your code
+still calls a left-column name on 8.x it throws (or the config key is rejected) — the
+replacement is always a one-line change. (Standing contract from 8.9.0 forward: removals
+happen only at majors, after ≥1 minor of loud runtime deprecation naming the replacement.)
+
+| Removed (7.x) | Replacement (8.x) |
+|---|---|
+| `brain.search(query, k)` | `find({ query })` — semantic; `find({ query, searchMode })` for hybrid |
+| `brain.getRelations({...})` | `related(id, opts)` for adjacency; `find({ connected: {...} })` for scoped traversal |
+| `brain.neural()` clustering | `find({ vector })` + aggregation `GROUP BY` |
+| `Db.search()` | `db.find({ vector })` |
+| Pre-8.0 storage path aliases (`directory`, `basePath`, …) | one `storage.path` key (old aliases throw) |
+| Reserved keys inside `metadata` bags (silently remapped in 7.x) | top-level params (`subtype`, `visibility`, `confidence`, `weight`, …) — reserved-in-bag throws |
+| 7.x COW branches layout (`branches/main/`) | generational MVCC (`asOf()`, `now()`, `db.persist(path)`) — on-disk migration is automatic at first 8.x open |
+
+The fork/snapshot family (`brain.snapshot()`, `createSnapshot()`, `restoreSnapshot()`)
+is sometimes cited as a 7.x removal — those methods never existed on 7.x; the 8.0 Db API
+(`asOf`/`persist`/`restore({confirm})`) is their first real implementation.
+
+---
+
+## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
+
+The write path stops paying maintenance costs — the last structural piece of the
+flush-storm class (a production deployment measured single writes blocked 25–191s behind
+history reclaim running inline on flush under memory pressure):
+
+- **`flush()` never compacts history.** It persists the current window's deltas and
+ nothing else — its cost no longer depends on history backlog or retention mode, in any
+ configuration. **`close()` is the auto-compaction site** (time-bounded per pass, ~5s;
+ an early stop is a consistent prefix and the next pass resumes).
+- **`compactHistory()` gains `timeBudgetMs`** — bound your own maintenance windows; the
+ same resumable-prefix guarantee applies.
+- **The documented trade**: a long-lived writer that never closes accumulates history
+ until its next explicit `compactHistory()`. Predictable writes, explicit maintenance.
+ If you run bounded retention on an always-on service, schedule a periodic
+ `compactHistory({ ...caps, timeBudgetMs })` in your maintenance window.
+- **New public doc: `docs/performance-envelopes.md`** — measured per-op envelopes
+ (p50/p95 at stated scales, hardware, and backend, with the measuring script cited).
+ Refresh rule going forward: any release touching a measured path re-runs that op's
+ benchmark and updates the envelope in the same release.
+- **New in this file: the Removed APIs 7.x→8.x table** (top of this document) — every
+ removal with its sanctioned replacement, one place, per the engine-currency contract.
+ Standing from here: removals only at majors, after ≥1 minor of loud runtime deprecation.
+
+## v8.8.2 — 2026-07-19 (one field-resolution law: reserved-field aggregates stop drifting)
+
+Four fixes from a consumer conformance audit, all rooted in the same disease — two field-resolution
+regimes where there must be one:
+
+- **Aggregates grouped by a RESERVED field (`subtype`, `visibility`, …) now decrement on
+ delete.** The delete/update hooks fed the aggregation engine a partial entity view (type,
+ service, data, metadata only), so a reserved-field `groupBy` resolved to a nonexistent group
+ on the way DOWN — counts drifted upward forever after any delete, and updates that moved an
+ entity between reserved-field groups double-counted it. The hooks now pass the full-fidelity
+ entity view (every reserved field top-level, the same shape the add path uses). If your
+ deployment derives stats from reserved-field aggregates, re-define those aggregates once
+ after upgrading (a changed definition triggers one rescan) or run them fresh — the drifted
+ persisted counts do not self-heal retroactively.
+- **Aggregation `source.where` on reserved fields now filters** instead of silently matching
+ nothing: the matcher resolves fields through the same resolver `groupBy` uses (top-level
+ standard fields + custom metadata), so `where: { subtype: 'note' }` means what it says.
+- **`removeMany()` refuses empty/invalid selectors loudly.** A bare array passed positionally
+ (`removeMany([id])` instead of `removeMany({ ids: [id] })`), an empty params object, or
+ `ids: []` used to resolve successfully having deleted nothing. All three now throw.
+- **`find()` accepts both where-key spellings.** Metadata is flattened at index time
+ (`metadata.entry.title` indexes as `entry.title`); a `metadata.`-prefixed where key now
+ falls back to its flattened spelling when the prefixed one isn't indexed — the
+ "unindexed field(s), returning []" confusion for storage-shaped spellings is gone. (A
+ literal nested custom key named `metadata` still wins when indexed as spelled.)
+
+## v8.8.1 — 2026-07-18 (flush no longer walks the whole generation history + the import dedup off-switch is now honest)
+
+### The flush-storm fix (production incident, reported by a long-running deployment)
+
+Under the default adaptive retention, **every `flush()` re-walked the entire committed
+generation history** to compute total history bytes for the budget check — O(all
+generations) with disk re-reads past the 4,096-entry delta-cache bound. On a brain with
+70,000+ accumulated generations that turned every write into a full-tail scan (60-100s
+writes), even though the budget (free-RAM-based) never tripped and nothing was ever
+reclaimed. Fixed:
+
+- `historyBytes()` now maintains a **running total**: seeded by one walk on first use,
+ then updated incrementally at every commit and reclaim — the adaptive retention check
+ on every flush is O(1). Invariant regression-pinned (running total ≡ fresh walk through
+ both commit paths and compaction).
+- New **`brain.historyStats()`** (read-only, exported `HistoryStats`): generation count,
+ total on-disk bytes, generation/timestamp range, compaction horizon, retention mode,
+ and the effective adaptive budget — the one-call fleet-audit for sizing retention
+ exposure per brain.
+- Interim guidance for keep-everything deployments already affected: `retention: 'all'`
+ skips the adaptive accounting entirely (and is the correct policy if you never want
+ history reclaimed). The accumulated files are harmless at rest; this release removes
+ the per-write cost of their existence.
+
+### The import dedup off-switch (lifecycle honesty)
+
+The post-import background deduplication pass (a merge-DELETE writer that runs ~5 minutes
+after an import, merging entities judged duplicates by id / name / vector similarity) had
+three lifecycle defects, all fixed:
+
+- **`enableDeduplication: false` now actually disables it.** The background pass was
+ scheduled unconditionally — an import that explicitly opted out could still have
+ entities auto-removed 5 minutes later. The flag now gates BOTH the inline merge and
+ the background pass (regression-pinned).
+- **One deduplicator per brain, owned by the brain.** Each `import()` call constructed its
+ own coordinator + deduplicator, so the "debounced" timer never actually debounced across
+ imports (N imports = N delete timers). The brain now owns a single instance — the
+ debounce genuinely spans imports — and `close()` cancels pending work, so a delete pass
+ can never fire against a closed brain.
+- **The 5-minute timer is unref'd** — a pending pass no longer holds the process open
+ (the exit-hang class; this timer had escaped the earlier sweep).
+
+Retention note for keep-everything deployments: with `enableDeduplication: false` on
+import calls and `retention: 'all'` in config, no engine path removes records
+automatically.
+
+## v8.8.0 — 2026-07-17 (OS-limit detection for pool-scale deployments)
+
+Small minor: brains now detect the two OS limits that bite at pool scale and warn **before**
+the incident instead of during it.
+
+- At open (once per process, Linux-only, measurement-only), Brainy reads `RLIMIT_NOFILE`
+ (soft/hard, from `/proc/self/limits`) and `vm.max_map_count`, and warns loudly when either
+ sits below the pool-scale floors (soft NOFILE < 65 536; max_map_count < 262 144) — with the
+ exact raise commands (`ulimit -n` / `LimitNOFILE=` / `sysctl vm.max_map_count`). On stock
+ defaults the failure otherwise arrives as `EMFILE` or a failed mmap deep inside an index
+ open, long after the real cause stopped being visible. An unreadable limit produces **no**
+ warning — no measurement, no claim (non-Linux platforms stay silent).
+- Exported for ops doors: `checkOsLimits()` returns the full `OsLimitsReport`
+ (values + warnings) programmatically, with the floors exported as constants.
+
+## v8.7.1 — 2026-07-17 (writer-lock acquisition is race-proof + machine-readable through init)
+
+Two hardenings of the multi-process writer lock (the `locks/_writer.lock` lease that makes a
+second writer on the same brain directory fail loudly):
+
+- **Lock acquisition claims atomically.** The acquire path used read-then-write, leaving a
+ window where two processes racing an *absent* lock could both "succeed" — and the loser
+ kept running unlocked, silently. The claim is now an atomic create-exclusive write
+ (`O_EXCL`): exactly one racer wins; the loser re-evaluates and either fails loudly with
+ the winner's details or performs a verified stale-takeover. Bounded retries; contention
+ beyond them fails loudly rather than degrading into a lockless open.
+- **`BRAINY_WRITER_LOCKED` survives `init()`.** The conflict error documents a
+ machine-readable contract (`err.code`, `err.lockInfo` with the holder's pid/host/
+ heartbeat), but init's error wrapping silently stripped both, leaving consumers a message
+ to regex against. The error now passes through unwrapped.
+
+Measured while verifying (for operators sizing audits): `brain.auditGraph()` at a
+production-consumer scale of ~2,600 relationships / 800 entities costs ~0.1 s warm and
+~0.5 s cold, with exact scar counting across reopen.
+
+## v8.7.0 — 2026-07-17 (bulk-transact ergonomics: scaled budgets + timeout telemetry)
+
+The bulk-import ergonomics release, from a consumer's measured production incident (a serial
+import on network-attached storage at ~2 s/op met a flat 30 s transaction budget):
+
+- **The transact apply budget now scales with the batch** — `max(30 s, opCount × 2 s)` — or
+ is exactly what you pass as the new `TransactOptions.timeoutMs`. A flat 30 s cap silently
+ limited honest bulk work to ~15 operations on slow disks while looking generous for small
+ batches. Internal batch paths (e.g. `removeMany` chunks) get the same scaling.
+- **`TransactionTimeoutError` is a diagnosis, not just a failure**: it now reports the
+ operation it stopped at as `i/N` with the operation's name, elapsed vs budgeted time, and
+ states the batch rolled back atomically and is retryable. Its `context` carries the same
+ fields programmatically.
+- **The transact envelope is documented** — batch sizing, budget math, chunking with
+ `ifAbsent` idempotency, and the precompute pattern (`embedBatch` + per-op `vector`) that
+ keeps model inference out of the commit path. Guide: `docs/guides/optimistic-concurrency.md`.
+- Note: `brain.embed()` / `brain.embedBatch()` (the precompute APIs) already ship — public,
+ with native-provider passthrough, verified end-to-end (batch and single paths produce
+ bit-identical vectors; a vector-supplied `add` is fully searchable). Honest measurement:
+ on the default WASM engine, batch throughput ≈ sequential (~160 ms/text) — the precompute
+ win is keeping inference out of the budgeted commit path, not raw embedding speed.
+
+## v8.6.0 — 2026-07-17 (brain.auditGraph — the graph-truth verification instrument)
+
+A minor release adding one new public API, from the fleet's graph-trust program: a read-only
+audit that **proves whether relationship reads return stored truth** on a given brain.
+
+- **`brain.auditGraph(options?)`** walks every canonical relationship record, queries the same
+ read path applications use (`related()` / VFS `readdir`) with all visibility tiers included,
+ and classifies every discrepancy into its failure family: `missingFromReads` (records the
+ read path omits — a stale adjacency index), `danglingEndpoints` (relationships whose endpoint
+ entity no longer exists — the historical partial-delete scar class), and `readOnlyVerbIds`
+ (read-path edges with no stored record — ghosts). Design-hidden internal/system edges are
+ counted separately so intentional hiding is never misclassified as loss. Counts are exact;
+ example lists cap at `maxExamples` with an explicit `truncatedExamples` flag; the result is
+ narrated loudly on incoherence. Mutates nothing — safe on a live brain.
+- The operational pairing: audit → if incoherent, `repairIndex()` → audit again. A `coherent`
+ report after the repair is the verified all-clear. Run it after any engine upgrade, restore,
+ or migration. Guide: `docs/guides/inspection.md`.
+- Also: `getNounIds` pagination now refuses an undecodable resume cursor loudly (the same
+ contract `getNouns`/`getVerbs` gained in 8.5.2 — the third and final walk brought under it).
+- Types exported: `GraphAuditReport`, `GraphAuditDiscrepancy`.
+
+## v8.5.2 — 2026-07-17 (aggregation backfill: exception-safe, generation-verified, and loud)
+
+Hardening patch from a migration incident (a byte-copied store on new hardware; the service
+entered a silent full-CPU loop at boot). Four changes, all in the aggregation engine's
+backfill/adoption path:
+
+- **Backfill walks are exception-safe and non-destructive.** A rescan now builds into a
+ staging map and swaps in atomically on completion; a mid-walk failure drops the staging map,
+ keeps the previous live state serving, and surfaces the storage error to the failing query.
+ Previously the walk wiped live state *before* a scan that could throw, never cleared the
+ pending flag on failure, and re-ran a full walk on every subsequent query — a silent
+ wipe/walk/throw loop at the caller's retry rate.
+- **Failed walks are latched.** After a walk fails, retries within a 30-second cooldown rethrow
+ the recorded error instantly instead of re-walking — a tight caller-side retry loop now costs
+ one loud error per query, never a full store walk per query.
+- **Adoption is generation-verified.** Persisted aggregation state is stamped with the store's
+ committed generation at flush; reopen adoption requires the stamp to equal the current
+ watermark. Stale state (unclean shutdown) or over-counting state (a fact-log truncation on a
+ copied store pulled the watermark back) triggers exactly one loud rescan — never a silent
+ adopt. Pre-8.5.2 state on generation-aware stores rescans once after upgrade, then is stamped.
+- **The path narrates.** Adoption decisions, no-adoptable-state outcomes, walk start/finish
+ (entity count + duration), and walk failures all log by default; a non-advancing storage
+ pagination cursor aborts the walk loudly instead of looping forever.
+
+Plus three guards from a full audit of every loop in the open/init path:
+
+- **Invalid pagination cursors fail loudly.** A supplied-but-undecodable resume token to
+ `getNouns`/`getVerbs` used to silently restart the walk at offset 0 — to a `while(hasMore)`
+ caller that re-serves page 1 forever (an unbounded silent CPU loop). It now throws with a
+ clear message instead.
+- **The graph cold-load verb walk has a stall guard.** `hasMore=true` with a missing or
+ non-advancing cursor aborts loudly instead of re-reading the same page forever.
+- **A derived index AHEAD of the store is named at open.** Brainy already surfaced a provider
+ generation *behind* the committed watermark; the *ahead* direction (the signature of a
+ byte-copy of a live service, or a log truncation during crash recovery) now logs a loud
+ warning explaining what happened and that `brain.repairIndex()` forces a heal — instead of
+ passing unnamed into whatever the derived index does next.
+
+## v8.5.1 — 2026-07-17 (aggregation state survives restarts + the query-cap ratchet removed)
+
+Patch release from a production incident (aggregate/count paths taking 40–90 s on an idle box
+while vector search stayed fast, and every `find({ limit: 5000 })` suddenly failing against an
+"auto-configured query limit of 1000"). Three fixes, one cosmetic:
+
+- **Aggregation state is actually adopted on reopen.** The boot pattern `defineAggregate()` →
+ query raced the engine's async state load: the synchronous define always won, flagged a
+ backfill, and the first query then wiped the just-loaded persisted state and re-walked the
+ entire store — every restart, forever. Reopening with an unchanged definition now adopts the
+ persisted state directly (zero scans); a backfill runs only on a real definition change, a
+ missing/failed state load, or a write that landed before adoption (exactness wins). Apps that
+ rely on persisted definitions without re-defining at boot also no longer race a spurious
+ "Aggregate not defined".
+- **Backfills are single-flight and batched.** Concurrent queries on a cold aggregate used to
+ each wipe the others' partial state and start their own full store walk — under steady query
+ arrival the store never converged (the 40–90 s loop). Now all concurrent queries share one
+ walk, and one walk fills every aggregate pending backfill (M aggregates ≠ M scans).
+- **The query-cap "learning" ratchet is removed.** `maxLimit` is a memory-protection bound, but
+ a hidden tuner shrank it 20 % per recorded query while the lifetime-average query time
+ exceeded 1 s — down to a floor of 1000, below the documented 10 000 auto floor, with no
+ practical recovery, and the resulting error blamed "available free memory" (stale basis
+ label). The cap now comes from its construction-time basis (or your explicit
+ `maxQueryLimit` / `reservedQueryMemory`) alone and never changes at runtime; query timing is
+ recorded for diagnostics only.
+- **Cosmetic:** the engine's own persistence keys (`__aggregation_*`, `brainy:entityIdMapper`)
+ no longer log `[Storage] Unknown key format` at boot — they were always routed correctly;
+ they're now recognized before the warning fires.
+
+Operationally: if a host was bitten, upgrading and restarting is the whole fix — no repair
+ritual needed. Setting `maxQueryLimit` explicitly remains the valve that bypasses auto-detection
+entirely.
+
+## v8.5.0 — 2026-07-15 (provider access to the fact log + the shared stamp verifier)
+
+Small additive follow-up to 8.4.0, from the native accelerator's first consumption pass:
+
+- **Index providers can now reach the fact log through the storage adapter** — new optional
+ capability `storage.scanFacts()` / `storage.factLogHeadGeneration()` / `storage.factSegmentPaths()`,
+ wired by the host brain at init as a closure over its live log. Providers hold only `storage` and
+ must never construct their own fact-log reader (the log's open path is writer-side); `null` means
+ "no fact log here — use the enumeration walk."
+- **The family-stamp verifier is shared** via `@soulcraft/brainy/internals`
+ (`readFamilyStamp` / `writeFamilyStamp` / `verifyFamilyStamp` + types) so first-party native
+ providers run literally the same verification function, never a synchronized copy.
+- **Rollup stamp invariants accept strings** (`number | string`) — content fingerprints such as a
+ per-tree SHA-256 are valid invariant values; a type mismatch reads as incoherence, never a pass.
+- **`storage.committedGeneration()`** — the committed watermark as a capability, so a provider
+ compares its stamp's `sourceGeneration` against the store's truth without parsing the private
+ manifest format.
+- **Two durability/stability contracts pinned in the suite:** fsync-before-ack (holds for
+ `transact()` today; the single-op path is pinned as the documented future target — group commit
+ becomes latency batching, never durability skipping) and scan-stability-under-rotation (a scan
+ snapshot yields exactly its facts — no gaps, duplicates, or bleed-in — while segments rotate
+ beneath it).
+
+No behavior change for applications; all additions.
+
+## v8.4.0 — 2026-07-15 (the generation fact log — a sequential, self-verifying commit stream)
+
+A minor release, fully backward-compatible (all additions; no behavior change for existing APIs).
+This is infrastructure: it changes nothing about how you query today, and lays the substrate that
+makes index heals and incremental catch-up sequential-read problems instead of directory walks.
+
+- **Every committed write now also appends a "fact" — an after-image commit record.** Alongside the
+ existing before-image history, each committed generation (single-op and `transact()` alike) appends
+ what each touched entity/relationship *became* — or a body-less tombstone for a removal — to an
+ append-only, checksummed segment log under `_generations/facts/`. Crash-safe by construction: a
+ torn tail is detected and ignored; on open the log is reconciled to committed truth, so an absent
+ generation always means "never committed." `transact()` facts are durable when `transact()`
+ returns; single-op facts ride the same group-commit flush as their history.
+
+- **New: `brain.scanFacts()`** — stream committed facts in commit order, in batches, with heal-grade
+ telemetry (total scope up front; per-batch generation range, byte size, and segment; a summary
+ cross-check at the end; loud abort on any gap — never a silent skip). **`brain.factSegmentPaths()`**
+ hands zero-copy consumers the immutable sealed segment files directly. New exported types:
+ `CommitFact`, `FactOp`, `FactScanBatch`, `FactScanHandle`. Facts accumulate from the first write
+ after upgrading — pre-existing history is not retroactively converted (enumeration remains the
+ fallback for old data).
+
+- **New: the entity-tree family stamp.** At every flush/close, brainy stamps which committed
+ generation the canonical entity files reflect plus the rollup invariants (entity/relationship
+ counts) that verify the tree whole. At open, coherence is a comparison — a genuine divergence is
+ loud and names the failing invariant; `repairIndex()` recounts from canonical and re-stamps. New
+ exports: `readFamilyStamp`, `verifyFamilyStamp`, `ENTITY_TREE_STAMP_PATH`, `FamilyStamp` types.
+
+- **Storage adapters** gain optional binary raw-byte primitives (`appendRawBytes`, `readRawBytes`,
+ `writeRawBytes`, `rawByteSize`) — feature-detected; the filesystem and memory adapters implement
+ them; an adapter without them simply hosts no fact log. The fact-log namespace is registered as a
+ protected family: no sweeper or GC can delete under it.
+
+No API breaks. 24 new tests; the full commit-path regression suite is green.
+
+## v8.3.3 — 2026-07-15 (rename moves the containment edge — no ghost in the old directory)
+
+One production-reported fix plus a repair path, completing the delete/move hygiene arc (8.3.1 fixed
+deletes, 8.3.2 fixed counters, this fixes moves).
+
+- **A cross-directory `vfs.rename()` now MOVES the containment edge instead of accumulating one per
+ parent.** The old parent's `Contains` edge was never removed on a move, leaving the entity a child
+ of **both** directories: `readdir(oldDir)` kept listing it after the move, re-creating the old path
+ showed the same name twice, and any tree-walking consumer (sync engines, file browsers) saw the
+ file in two places. The old edge is now removed by edge id, resolved from the graph's own adjacency
+ — a removal never requires reading the thing being removed. Bonus fix in the same seam: a move **to
+ the root** now gets its containment edge (it was previously skipped, orphaning the file out of
+ `readdir('/')`).
+
+- **`repairIndex()` now also reconciles VFS containment** (new `vfs.repairContainment()`): every VFS
+ entity's containment edges are checked against its canonical `metadata.path` — stale old-parent
+ ghosts and duplicate edges are removed, a missing expected edge is restored, and user
+ knowledge-graph edges are never touched (only `vfs-contains` edges are candidates). Loud per
+ repair. Stores that performed cross-directory renames under ≤8.3.2 should run `brain.repairIndex()`
+ once after upgrading — the same single ritual now heals orphan directories, counters, **and**
+ containment edges.
+
+- Also ships a permanent lens-consistency regression suite (combined type+subtype vs subtype-only vs
+ canonical ground truth, id-for-id, warm and after a cold reopen), ported from the field
+ investigation that closed the historical lens-drop report.
+
+No API changes beyond the new optional `vfs.repairContainment()` (also invoked by `repairIndex()`).
+
+## v8.3.2 — 2026-07-14 (honest counters — the recount + removal-without-re-reading)
+
+Completes 8.3.1's delete-hygiene story at the counter layer, from a production proof chain reported
+by a downstream deployment: persisted entity totals were permanently **inflated** — deletes whose
+count decrement was silently skipped — and because paginated `totalCount` serves
+`Math.max(persistedTotal, scanned)`, the inflated number always won and **no disk cleanup could ever
+lower it**.
+
+- **A removal's count decrement no longer requires re-reading the record being removed.** The
+ decrement was sourced from re-reading the entity's metadata inside the delete; if that read
+ returned `null` (a replace race, or a ghost left by a pre-8.3.1 partial delete) the decrement was
+ silently skipped while the paired add had counted — minting drift on every write→delete→re-create
+ cycle. The caller's pre-delete read now rides through the whole delete path
+ (`remove()`/`removeMany()` → the delete operation → `deleteNoun`/`deleteVerb` →
+ `deleteNounMetadata`/`deleteVerbMetadata`, both sides symmetric): a null internal read falls back
+ to the known prior record instead of skipping. The `StorageAdapter` signatures gain an optional
+ `priorMetadata` parameter (additive; existing adapters unaffected).
+
+- **`repairIndex()` is the sanctioned counter recount — unconditional, and it actually persists.**
+ `rebuildTypeCounts()` previously rebuilt only the type-statistics arrays and computed the total
+ *just to log it* — the persisted scalar (`counts.json`) survived every "rebuild" untouched, so an
+ already-inflated brain could never be corrected. One canonical walk now rebuilds **every** counter
+ rollup — scalar totals, per-type maps, and type statistics — and persists them, and `repairIndex()`
+ runs it unconditionally (not only when orphan directories are found: counters can be inflated over
+ perfectly clean shelves). Brains with delete history should run `brain.repairIndex()` once after
+ upgrading; the correction survives reopen.
+
+No API breaks (optional-parameter additions only). Regression tests cover the drift cycle, the
+null-read decrement fallback, and the persisted recount across reopen.
+
+## v8.3.1 — 2026-07-14 (full-removal deletes + family-scoped migration gate)
+
+Two production-reported fixes in the write/index spine, plus an operator repair path. No API changes;
+all behavior changes make previously-wrong states honest.
+
+- **Deleting an entity now removes it completely — no more "ghost" leftovers on disk.** A canonical
+ noun delete removed the metadata (content) leg but left the entity's `vectors.json` and its `/`
+ directory behind. Consequences observed in a production deployment: deleted rows were
+ indistinguishable on disk from damage scars, enumerated counts inflated monotonically with every
+ delete (the leftovers were counted forever), and locator-style reads hit unreadable ghost rows.
+ `remove()`/`removeMany()`/`deleteNoun`/`deleteVerb` now remove **both legs and the entity
+ container**, with a full two-leg before-image rollback inside the transaction. The generation log
+ still holds the delete's before-image, so `asOf()` time-travel reconstructs deleted entities exactly
+ as before — this is live-HEAD hygiene, not a history change. This also fixes **duplicate `readdir`
+ entries for re-created VFS paths** at the root: with no ghost state, a delete always unposts its
+ index rows, so a delete→recreate cycle lists the path exactly once (regression-tested across
+ repeated cycles).
+
+- **`repairIndex()` prunes ghost/scar directories left by earlier versions.** Stores that deleted
+ entities under ≤8.3.0 may hold orphaned entity directories (a vector-only leg, or an empty dir).
+ `brain.repairIndex()` now sweeps them: it removes only containers with **no metadata content leg**
+ (never a directory that still holds content), logs every removal, and recomputes type/subtype
+ counts afterward so totals stop counting ghosts.
+
+- **Reads no longer hang behind an unrelated index migration (family-scoped gate).** During a native
+ provider's one-time background migration, *every* read — including plain `get()`, VFS
+ `readdir`/`readFile`, and metadata-only `find({ where })` — blocked on the whole-brain migration
+ lock until timeout, even when the migrating index was irrelevant to the read. The gate is now
+ scoped to the index families a read actually consults: canonical reads (`get`, `batchGet`, VFS
+ content) never wait; a `find` waits only on the families its query shape needs (vector for
+ semantic, metadata for `where`/type, graph for `connected`); graph traversals wait only on the
+ graph family. Writes and unclassified operations keep the conservative whole-brain wait. A read
+ that *does* need the migrating family still blocks (bounded by `migrationWaitTimeoutMs`) and
+ surfaces the retryable `MigrationInProgressError` — never a partial result.
+
+No breaking API change. Each fix ships with regression tests.
+
+## v8.3.0 — 2026-07-13 (faster index heals + the cross-layer integrity contract)
+
+Three additive changes. The first is an immediate, standalone performance win; the other two are the
+brainy side of the write/index-spine integrity contract, inert until a native accelerator
+that implements the matching hooks is present — so this release changes nothing for a JS-only brain
+beyond the speedup.
+
+- **Canonical enumeration is up to ~16× faster — the dominant term in an index heal.** The paginated
+ entity walk (`getNounsWithPagination`) hydrated each entity's vector + metadata one-at-a-time; since
+ every index rebuild enumerates canonical storage, that serial per-item latency dominated multi-minute
+ heals. Hydration is now 16-way bounded-concurrency, with the pagination contract (order, cursor
+ resume, filters, totalCount) byte-identical to before. New **`getNounIdsWithPagination()`** returns
+ ids without hydrating anything (zero per-entity reads when unfiltered) for callers that own their own
+ IO schedule.
+
+- **Cross-layer integrity — `validateIndexConsistency()` is no longer blind to native providers.** It
+ only ran the JS metadata index's own check, so a native provider whose manifest/segments/counts had
+ diverged still read as "healthy". It now feature-detects and aggregates each provider's optional
+ `validateInvariants()` self-report, names every failing invariant with its numbers, and exposes the
+ per-provider reports. `repairIndex()` now reconciles native derived state from canonical too
+ (rebuilding any provider whose failing invariant asks for it). New exported types
+ `ProviderInvariantReport` / `InvariantResult` / `InvariantHeal`.
+
+- **Registered-blob families — declared index files are undeletable through the storage layer.** A
+ provider can declare a derived-index blob *family* (a set of members that are load-bearing together);
+ once declared, `deleteBinaryBlob` / `removeRawPrefix` refuse to remove a member (new exported
+ `ProtectedArtifactError`), so a stray in-process sweeper cannot delete a load-bearing index file. The
+ declaration persists across reopen; `checkDerivedFamiliesPresent()` names any member missing on open.
+ New optional `StorageAdapter` surface (`registerDerivedFamily` / `unregisterDerivedFamily` /
+ `listDerivedFamilies` + `DerivedFamilyDeclaration`) and exported `DerivedArtifactMissingError`.
+
+No breaking API change (all additions are optional/new). Each change ships with regression tests.
+
+## v8.2.8 — 2026-07-13 (honest index readiness — no more silently-empty queries on a cold index)
+
+Closes the last of the three spine anti-patterns: the "dishonest readiness proxy," where `size() > 0`
+was treated as "this index actually serves queries." On a cold open (fresh boot, restart, crash
+recovery) a native index can load its **count** before its **serving structure** — so for a brief
+window it has data but cannot answer, and a query returned a silent empty result indistinguishable
+from "no such data." Every fix here asks the index whether it can *actually serve*, and if not,
+self-heals or fails loudly instead of returning `[]`.
+
+- **Semantic search no longer returns a silent `[]` on a cold vector index.** A pure semantic
+ `find({ query })` has no filter, so nothing previously guarded the vector index. A new one-shot
+ guard verifies the vector index serves a known persisted vector on the first semantic/proximity
+ search — preferring the provider's honest `isReady()` signal, else a known-vector self-match probe.
+ It rebuilds from canonical records if the serving structure did not load, and throws the new
+ **`VectorIndexNotReadyError`** only if a rebuild still cannot serve — never a silent empty result.
+
+- **Relationship reads fall back to the canonical scan instead of an empty result on a cold graph
+ index.** `getVerbsBySource`/`getVerbsByTarget` (used by relationship queries and virtual-filesystem
+ traversal) skipped the fast path only on `isInitialized` — which reads true once the manifest loaded
+ even if the source→target adjacency did not. They now consult the honest readiness signal and, when
+ the adjacency is not serving, take the correct-but-slower canonical shard scan. A one-shot self-heal
+ probe covers providers that expose no readiness signal.
+
+- **`getIndexStatus()` tells the truth for readiness probes.** It reported `populated: size>0` only, so
+ a Kubernetes readiness check could route traffic to a brain still warming up. It now folds in the
+ honest per-index `ready` signal (making `populated` honest), plus the degraded states already
+ surfaced by `checkHealth()`/`validateIndexConsistency()` (`rebuildFailed`/`rebuildError` and a
+ `degradedIds` count) — so a probe never reports 200-ready over a known-degraded index.
+
+This completes the write/index-spine hardening end to end. New export: **`VectorIndexNotReadyError`**;
+`getIndexStatus()` gains additive fields (`rebuildFailed`, `rebuildError?`, `degradedIds`, per-index
+`ready?`). No breaking API change. Each fix ships with a dedicated regression test.
+
+## v8.2.7 — 2026-07-13 (loadBinaryBlob fault-propagation — the lockstep completion of 8.2.6)
+
+Completes the Pass-1 spine hardening. `loadBinaryBlob` (the raw-blob read a native accelerator mmaps
+for its index files) previously returned `null` on ANY read error, so a real IO fault
+(EIO/EACCES/EMFILE) on a present-but-unreadable index blob masqueraded as "the blob is absent" —
+driving a needless full rebuild or an empty read. It now distinguishes genuine absence (ENOENT →
+`null`, the documented contract) from a real fault (→ throw), so a transient disk fault surfaces
+loudly instead of silently degrading the index.
+
+This one change was deliberately held out of 8.2.6 and ships now, in lockstep with the native
+accelerator release that hardened its two column-store read sites to handle the throw (a faulted
+segment read marks the field unavailable and throws a named error, instead of relying on
+null-on-error). A consumer on new brainy + an older accelerator was never at risk: 8.2.6 kept the
+prior swallow-on-fault behavior for this method until the accelerator was ready.
+
+No API change. Regression: `tests/unit/storage/blob-save-durability.test.ts` gains the loadBinaryBlob
+leg (absent → null; present → bytes; real fault → throws).
+
+## v8.2.6 — 2026-07-13 (write/index-spine hardening — loud errors, never quiet losses)
+
+Durability + integrity hardening across the write and index paths. Every fix converts a place that
+could *silently* lose data, serve a partial result, or acknowledge a write that did not land into a
+loud, observable failure. No breaking API changes; one new exported error.
+
+- **A durable blob write that stored nothing is no longer acknowledged.** The atomic blob write
+ (`saveBinaryBlob`, used for vector/graph index segments and the native index files) uses a unique
+ per-writer temp file, so a rename `ENOENT` can only mean *our* temp vanished before the rename —
+ the bytes never landed. It previously returned success on that path; it now retries once with a
+ fresh temp and, if the temp vanishes again, throws instead of acknowledging a write that persisted
+ nothing. A downstream deployment that mmaps these files no longer finds a "successfully written"
+ blob missing on the next open.
+
+- **Single-op history durability is enforced, not assumed.** The asynchronous group-commit flush
+ that persists single-op generation history previously swallowed a persist failure as a warning
+ while writes kept succeeding and their history piled up, undurable, in memory. It now tolerates a
+ transient blip (retry with capped backoff) and, after repeated failures, **refuses further writes**
+ with the new **`PendingFlushDurabilityError`** rather than promise a durability it cannot deliver.
+ Live canonical data is untouched; the latch self-heals the moment a flush succeeds. Callers needing
+ hard per-write durability should keep using `transact()` (or `flush()` after a single-op).
+
+- **A degraded derived index is surfaced on reads, not served silently.** When a non-fatal index
+ rebuild fails at open, or a write commits via adopt-forward recovery with an incomplete derived
+ index, `find()` and `get()` now emit one loud warning per degraded window (reads still return —
+ canonical is the source of truth), the state folds into `checkHealth()` /
+ `validateIndexConsistency()`, and `repairIndex()` reconciles and clears it.
+
+- **`clear()` removes the full derived footprint.** It previously left raw index blobs, the native
+ id-mapper, and column-index manifests on disk, so a cleared brain could re-read stale native state
+ on reopen. It now wipes them together as a set.
+
+- **Counts stay honest across deletes.** A delete decremented the per-type breakdown but not the
+ scalar total, so the total inflated permanently (and won pagination). Delete now decrements both;
+ the invariant `total === Σ per-type` holds across any interleaving of add / visibility-flip /
+ delete and across a reopen.
+
+- **Index-maintenance and aggregation failures are loud.** A partial LSM/segment load no longer
+ publishes the manifest's full count as if healthy; an HNSW flush that can't persist a node throws
+ (`HnswFlushError`) instead of returning a lying count and dropping the node; a corrupt or missing
+ manifest-listed column segment throws (`ColumnSegmentLoadError`) instead of silently dropping its
+ entities from every query; and aggregation materialization / state-load failures now warn instead
+ of vanishing into an empty catch.
+
+New export: **`PendingFlushDurabilityError`** (with `.cause` and `.failedAttempts`). No other public
+API change. Each fix ships with a dedicated regression test.
+
+## v8.2.5 — 2026-07-12 (honest response when a transaction rollback can't complete)
+
+Data-integrity fix. When a transaction failed and its rollback then *also* failed to undo a
+canonical write (retries exhausted), the old behavior logged, continued, and threw
+`TransactionRollbackError` — while the record it couldn't undo stayed durable on disk, and the
+transaction even reported its state as `'rolled_back'`. The caller got an error implying the write
+was undone; a read-back showed the record. A failed rollback had no truthful response.
+
+Rollback now tells the truth, with a two-branch contract:
+
+- **Adopt-forward (safe case).** A single-op write (`add`/`update`/…) whose only damage is a
+ durably-present record — the write the caller asked for — is **adopted**: its generation is
+ committed, the write returns success, and a loud warning records that the derived index may be
+ incomplete for that id until the next rebuild/`repairIndex()`. No error, no double-write; the
+ record is immediately durable and retrievable by `get()`.
+- **Fail loud + quarantine (unsafe case).** A multi-operation batch, or *any* case where a record
+ was lost (a remove/update whose restore-undo failed), throws the new **`StoreInconsistentError`**
+ naming every unreconciled record and its disposition (`orphan` vs `loss`), and puts the brain into
+ **write-quarantine**: reads keep working, but writes are refused until `repairIndex()` reconciles
+ the derived indexes against canonical storage and lifts the quarantine. The generation counter is
+ not advanced, and a transaction whose rollback failed is now `'inconsistent'`, never the
+ `'rolled_back'` lie.
+
+The decision is made by *observation*, not guesswork: both commit paths already hold byte-identical
+before-images, so after a failed rollback the store compares current canonical state to them to
+classify exactly which records are orphaned or lost.
+
+New export: `StoreInconsistentError` (with `.records` and `.cause`) and the `UnreconciledRecord` type.
+No other API change; `repairIndex()` gains the quarantine-lift behavior. Regression
+(`tests/integration/rollback-trapdoor.test.ts`) injects the exact failure (index add throws →
+canonical delete-undo fails) and pins adopt-forward (durable, get-able, not quarantined), fail-loud
+(`StoreInconsistentError` + quarantine + reads work + `repairIndex()` lifts it), and the error's
+record naming.
+
+## v8.2.4 — 2026-07-12 (restore can no longer destroy the store it's recovering)
+
+Recovery-safety fix. `restore()` removed the entire live brain directory and THEN copied the
+snapshot in — so any copy failure left the store destroyed with only a partial copy. The sharpest
+edge: the copy (`fs.cp`) materialized the holes of sparse mmap blob files, so a snapshot that fits
+on disk could balloon and `ENOSPC` mid-copy, and the recovery tool would have just destroyed the
+brain it was asked to recover. Reported from a downstream incident's recovery forensics.
+
+Restore is now **non-destructive and crash-resumable**:
+
+- The snapshot is copied into a staging area (`_restore_staging/`) **before any live data is
+ touched**. The copy is **sparse-aware** — all-zero regions are left as holes, so a store of
+ mostly-hole blobs restores at its true allocated size instead of its apparent size.
+- A copy failure (including `ENOSPC`) removes only the half-written staging area and throws; the
+ live store is left **exactly as it was**.
+- Only after the copy succeeds and a completion marker is `fsync`'d does an **atomic per-entry
+ swap** move the staged data into place — same-filesystem renames that cannot fail for disk space.
+- A crash mid-swap is finished **forward** on the next open: startup resumes a committed-but-
+ incomplete swap, or discards an uncommitted staging area (live data still authoritative).
+
+No API change — `restore(path, { confirm: true })` is unchanged. `persist()` was already safe
+(hard-link snapshot). Regression (`tests/integration/restore-nondestructive.test.ts`): a forced
+copy failure leaves live data fully intact, a normal restore round-trips, an interrupted-but-
+committed restore completes on reopen, an uncommitted staging area is discarded, and the sparse
+copy is byte-identical with allocation far below apparent size.
+
+## v8.2.3 — 2026-07-12 (a committed transaction is durable on return)
+
+Durability fix. A `transact()` reported "committed" while its canonical entity writes were still
+only in the OS page cache (written via tmp+rename, not yet `fsync`'d), even though the generation
+counter and manifest WERE fsync'd. A hard kill (power loss, SIGKILL) in that window could leave the
+durable generation counter **ahead of** the persisted entity bytes — so a consumer resuming from the
+counter would see "phantom progress": a generation that claims writes the disk never kept. Reported
+from a downstream migration's crash-lifecycle forensics.
+
+`commitTransaction` now runs a **durability barrier**: it records every canonical write and delete
+the batch's operations make, then `fsync`s that entire footprint (file contents, the rename
+directory entries, and the parent directories of any deletes) **before** advancing the generation
+counter and manifest. A committed transaction is therefore durable the moment `transact()` returns —
+the counter can never outrun the entity bytes.
+
+**Durability contract, now explicit.** `transact()` is durable-on-return (above). A **single-op**
+write (`add`/`update`/`remove`/`relate`/…) is Model-B group-commit: its live bytes are written but
+become durable at the next `flush()` or `close()`, not the instant the call resolves — the counter
+is buffered alongside the data, so a crash loses both together (never a torn counter-ahead-of-state
+store). Need per-write durability? Use `transact()` (even for one op), or `flush()` after the write.
+This deliberately trades single-op fsync latency (a 3-5x write regression) for throughput.
+
+No API change; no accelerator involvement (the barrier is in the filesystem storage adapter). In-memory
+and durable-per-call (cloud object-PUT) adapters treat the barrier as a no-op. Regression:
+`tests/integration/transact-durability-barrier.test.ts` proves entity writes fsync in an earlier
+batch than the manifest, for single-op and multi-op (add+relate) transactions, and that a
+precommit-rejected batch opens no barrier and advances nothing.
+
+## v8.2.2 — 2026-07-11 (P0: a timed-out transaction now rolls back — no torn state)
+
+Data-integrity fix. A transaction that exceeded its time budget **mid-flight** (e.g. a bulk
+`transact()` on slower hardware crossing the 30s ceiling) threw a timeout error WITHOUT rolling
+back the operations it had already applied. The budget check sat outside the per-operation
+rollback path, so only per-operation *failures* rolled back — a timeout stranded the partial
+writes in canonical storage while the generation was never stamped, leaving torn, generation-less
+state. This was caught by a downstream migration that crossed the ceiling on a large batch.
+
+`Transaction.execute()` now has a **single rollback point**: any error that escapes the operation
+loop — an operation failure OR a mid-flight timeout — rolls back every applied operation in
+reverse order, then surfaces the original error (a rollback failure supersedes it, loudly, as
+before). This restores the invariant the generation-store commit path already depended on: a throw
+from execute means the applied operations were undone byte-identically, and the aborted
+transaction leaves `generation()` unchanged and storage byte-identical to its pre-transaction
+state.
+
+No API change. Regression pins the reported requirement — after a mid-flight timeout,
+storage is byte-identical and the transaction ends in the `rolled_back` terminal state
+(`tests/unit/transaction/timeout-rollback.test.ts`), plus the operation-failure and
+rollback-failure paths through the same single rollback point.
+
+**Note on the 30s ceiling itself** (configurable/scaled timeout, batched embedding precompute,
+timeout telemetry) — that ergonomics work is tracked separately; this release fixes only the
+correctness bug (a timeout must never leave partial state), independent of where the ceiling sits.
+
+## v8.2.1 — 2026-07-11 (transact forward references work on the native accelerator)
+
+Parity fix. `transact()` has always promised atomic forward references — `add` an entity and
+`relate` to it in one batch — but the transaction planner resolved relationship endpoint ids at
+**plan** time, before the batch's adds had applied. Asking the id mapper about an entity that
+doesn't exist yet made the accelerator's (correctly strict) native mapper throw in
+`EntityIdMapper.getOrAssign`, so `transact([{op:'add', id:X}, {op:'relate', to:X}])` failed on
+native deployments while the permissive JS mapper masked the bug — and silently leaked an id
+assignment whenever a batch was later rejected by a commit precondition.
+
+Endpoint resolution is now **lazy** — evaluated when the graph operation executes inside the
+commit, after the batch's adds have applied (the same lazy pattern the operation's generation
+stamp already used). Fixed across all transact-planned graph operations: relate (including the
+bidirectional reverse edge), remove's relationship cascade, and unrelate. Single-operation writes
+are unchanged. Also fixed as a byproduct: a rejected batch no longer pollutes the id mapper.
+
+Regression suite covers the exact reported shape, both-endpoints-in-batch, bidirectional,
+add+relate+remove in one batch, and the mapper-cleanliness proof on rejected batches
+(`tests/integration/transact-forward-ref-graph.test.ts`). No API changes; no accelerator version
+pairing required.
+
+## v8.2.0 — 2026-07-10 (temporal VFS — file content joins the immutability model)
+
+Time travel now covers Virtual Filesystem **content**. Previously the temporal model had a hole
+exactly where files were concerned: every entity write was an immutable generation, but the
+content *bytes* lived under an eager reference-count GC — deleting a file could physically destroy
+bytes that in-window history still referenced, while overwriting never released the old content at
+all (an unbounded silent leak that only *accidentally* preserved history). A production consumer's
+recovery from a bad deploy succeeded only because a stale field happened to hold the good value —
+luck, not a guarantee. Both directions are now fixed by making blob reclamation a **history**
+decision instead of a **liveness** decision:
+
+- **Content blobs are retention-protected.** Each blob tracks live references AND history
+ references (one per persisted generation record that carries its hash). Deleting/overwriting a
+ file drops only the live reference; bytes are physically reclaimed in exactly one place —
+ history compaction — once no live reference and no retained generation references the hash.
+ Pinned views are exempt automatically (compaction already respects pins). Crash ordering is
+ over-count-only (never under-count), so a crash can leak until the built-in scrub recounts, but
+ can never reclaim bytes history still needs. Existing stores get a one-time, marker-gated
+ backfill on open; cross-file content dedup is handled exactly.
+- **`vfs.readFile(path, { asOf })`** — the file's exact bytes as of a generation or `Date`,
+ guaranteed present within the retention window.
+- **`vfs.history(path)`** — the file's versions (`{ generation, timestamp, hash, size,
+ mimeType? }`, ascending). Restore = read the old bytes `asOf` and write them back (a new write;
+ history is never rewritten).
+- **Overwrites now refresh the file entity's `data`/embedding text** — previously semantic search
+ and the `data` field served the FIRST version's text forever.
+
+Lifecycle note: deleting a file no longer frees its bytes immediately — old content lives until
+compaction reclaims its generations, under the same `retention` budget as all Model-B history
+(`retention: 'all'` keeps every version forever). Guide: the new "Time travel for files" section in
+`docs/guides/snapshots-and-time-travel.md`. Native accelerator: unaffected (canonical write path
+only) — no version pairing required.
+
+## v8.1.0 — 2026-07-10 (`brain.onChange` — the in-process change feed)
+
+New public API: subscribe to every committed mutation with
+`brain.onChange(cb) → unsubscribe`. One event per affected record, for **every**
+canonical write regardless of origin — direct calls, batch methods, `transact()`,
+imports, and Virtual Filesystem writes all funnel through the same commit point the
+feed is emitted from. This is the authoritative in-process signal for live UIs,
+cache invalidation, and realtime sync layers (downstream SDKs can forward it over
+their own transports to make local and remote brains uniform).
+
+Event shape (`BrainyChangeEvent`, exported): `kind` (`entity`/`relation`/`store`),
+`op` (`add`/`update`/`remove`/`relate`/`unrelate`/`updateRelation`/`clear`/`restore`),
+the post-commit `entity` or `relation` view (type + full custom metadata), the
+committed `generation`, and `timestamp`. Notable properties:
+
+- **Post-commit only** — an aborted write (a losing `ifRev` CAS, a rejected
+ transaction) never emits. If you got the event, the write is durable.
+- **Deletes fully described** — `remove`/`unrelate` carry the record's last
+ committed state (from the commit's own history record), not just an id; batch
+ deletes included.
+- **Batches emit per item**; a `transact()` batch's events share its single
+ generation. Entity deletes also emit `unrelate` for each cascaded relationship.
+- **Commit-ordered, asynchronous, isolated** — events arrive in commit order,
+ dispatched in a microtask so a slow listener never delays a write and a throwing
+ listener never affects the write or other listeners. Zero overhead with no
+ subscribers.
+- **`kind: 'store'`** events for `clear()`/`restore()` mean "refetch everything".
+- Fire-and-forget by design; each event's `generation` composes with
+ `asOf()`/`transactionLog()` for catch-up after a gap.
+
+Guide: `docs/guides/reacting-to-changes.md`. No behavior changes for existing code.
+
+## v8.0.17 — 2026-07-08 (count recovery scans the real layout · ~1,100 lines of dead 7.x machinery removed)
+
+A cleanup of the vestigial 7.x "hnsw sharding" machinery that turned up two real fixes.
+
+**1 — Count recovery now scans the layout the database actually uses.** When `counts.json` is
+lost or corrupted (container restarts, partial copies), the recovery scan rebuilt the entity/
+relationship counters by counting files in `entities/*/hnsw/` — directories the 8.0 write path
+never populates — so an established store recovered to **zero counts** on real data: wrong
+`getNounCount()`/stats, a "New installation"-style boot log, and a mis-sized rebuild-strategy
+decision at open. The scan now counts the canonical `entities////` tree.
+Regression-tested: delete `counts.json` from a populated store → reopen → exact counts.
+
+**2 — A latent init-time deadlock removed.** The recovery scan's type-distribution sampler called
+a guarded accessor (`getNounMetadata` → `ensureInitialized`) from **inside** `init()`, which
+re-enters `init()` and hangs the open. It was unreachable before only because the old scan never
+found anything to sample; the fix reads the sampled metadata files directly.
+
+**3 — The dead machinery itself is gone (−1,138 lines).** Twenty-three methods with zero callers:
+the 7.x hnsw-layout entity/edge CRUD, the sharding-depth prober + depth-migration engine, and an
+orphaned streaming paginator. Verified by reachability analysis before deletion; no public API
+touched; the full suite is green. New stores no longer carry the always-empty legacy directories'
+scan cost, and the boot log keeps the truthful new-vs-established wording from v8.0.13.
+
+## v8.0.16 — 2026-07-08 (concurrency fast-follow: atomic `ifAbsent`/`upsert` + exact blob reference counts)
+
+Closes the two remaining check-then-act races found in the sweep that followed v8.0.15's CAS fix
+(disclosed in that release's notes). Same bug class — a check performed before the serialization
+point that guards the apply — same cure.
+
+**1 — `add({ ifAbsent })` and `add({ upsert })` are now atomic.** The absence check ran before the
+commit mutex, so N concurrent same-id creates could all pass it and all write — the second
+silently overwriting the first, violating ifAbsent's "returns the existing id **without writing**"
+contract and upsert's "merge, never clobber" contract. The insert leg now carries a must-be-absent
+precondition (the same conditional-commit primitive as `ifRev`), verified under the commit mutex:
+exactly one concurrent create wins; every other caller takes its documented resolution — ifAbsent
+returns the existing id with zero writes, upsert merges into the now-existing entity (with a
+bounded retry if a concurrent delete intervenes). Verified: 8 concurrent `ifAbsent` creates
+advance the store by exactly ONE generation; 8 concurrent upserts on an absent id produce one
+create + seven merges (`_rev` lands at exactly 8). Note: `transact()`'s batch-level
+ifAbsent/upsert keep planning-time semantics (the batch converges to a valid entity; the window is
+documented, not silent).
+
+**2 — Blob reference counts are exact under concurrency.** The content-addressed blob store's
+`write()` (dedup: exists → add a reference, absent → create) and `delete()` (decrement → remove at
+zero) were unserialized read-modify-writes over the blob's metadata. Concurrent writes of
+identical content could lose references — making a later delete remove bytes **another file still
+referenced** (data loss), or leak unreferenced blobs. All reference-count-bearing mutations are
+now serialized per content hash (distinct content never contends): N concurrent identical writes
+yield exactly N references, and a blob is physically removed only when the true last reference
+drops. Verified with concurrent write/delete storms.
+
+Both fixes are in-process complete by construction — storage enforces single-writer-per-directory,
+so the process is the whole concurrency domain. No API changes.
+
+## v8.0.15 — 2026-07-08 (`ifRev` CAS is now atomic — exactly one winner under concurrency)
+
+Correctness fix for optimistic concurrency, reported from a production cutover rehearsal. N
+**concurrent** `update({ ifRev })` calls carrying the same expected revision ALL succeeded — zero
+`RevisionConflictError`s, last-writer-wins, the other N−1 writes silently lost. (Sequential calls
+conflicted correctly.) The revision check ran before the commit mutex, so interleaved callers all
+passed it before any apply landed — breaking the exactly-one-winner semantics the
+[optimistic-concurrency guide](docs/guides/optimistic-concurrency.md) promises, which advisory
+locks and per-entity ledgers/counters build on.
+
+The fix makes the check-and-apply a **conditional commit**: the generation store's commit paths
+accept a precondition that runs under the commit mutex against the just-read authoritative
+before-images — the per-record analogue of `ifAtGeneration`, which always ran there. `update()`
+and `transact()` per-op `ifRev` both re-verify at that point (the earlier check remains as a cheap
+fast-fail). A conflict aborts atomically: nothing staged, nothing applied, the same
+`RevisionConflictError` as before. Two adjacent behaviors also became honest:
+
+- **`_rev` is now monotonic under concurrency.** The winner's stamp derives from the
+ authoritative before-image, so N concurrent plain updates advance `_rev` by N (previously they
+ could all stamp the same stale value).
+- **CAS against a concurrently-deleted entity** now throws `EntityNotFoundError` instead of
+ silently resurrecting it (plain updates keep their last-writer-wins re-create semantics).
+
+Verified with a concurrency regression suite: 8 parallel same-rev updates → exactly 1 winner + 7
+conflicts; the documented read→CAS→retry ledger loop converges exactly (8 workers, 8 decrements,
+0 lost) — `tests/integration/ifrev-concurrent-cas.test.ts`. If you serialized writes in your own
+adapter as a workaround, it can come out after this upgrade.
+
+## v8.0.14 — 2026-07-07 (7→8 migration preserves branch-scoped non-entity state instead of deleting it)
+
+Defense-in-depth for the one-time 7→8 layout migration. The migration rescues the head branch's
+entities (`branches//entities/*` → `entities/*`) and then drained the whole `branches//`
+directory. If a 7.x engine had written durable state under the head branch *outside* `entities/`
+(a branch-scoped index, blob area, or field registry), that drain would have silently deleted it —
+the same failure class as the VFS content blobs a 7.x store kept in `_cow/`. The migration now drains
+the branch only when nothing but the moved entities remains; if any non-entity object survives, it
+**preserves the branch** (no delete) and logs a loud warning naming the leftover keys, so the state is
+recoverable rather than lost. No effect on a normal migration (a clean branch still drains); purely a
+guard against silent data loss.
+
+Cosmetic boot-log fix. Every persisted 8.0 store logged `📁 New installation: using depth 1
+sharding` on **every** open — even established brains holding thousands of entities — which is
+alarming to read during a restart or incident. Cause: 8.0 stores nouns in the canonical
+`entities/nouns///vectors.json` layout, but the legacy sharding probe inspected the
+`entities/nouns/hnsw/` directory, which the 8.0 write path never populates, so it always concluded
+"new". The new-vs-existing decision now consults the layout the database actually reads and writes
+(plus the known noun count), so an established store logs `📁 Using depth 1 sharding (N entities)`
+and only a genuinely empty store reports a new installation. No behavior change beyond the log line —
+it never triggered a rebuild or migration; entities were always read correctly.
+
+## v8.0.12 — 2026-07-07 (7→8 VFS-content recovery · zero-rebuild cold open · strict query operators)
+
+Three consumer-facing fixes.
+
+**1 — A 7→8 upgrade recovers Virtual Filesystem content automatically (data-integrity).**
+7.x stored VFS file content as blobs in the branch system's copy-on-write area (`_cow/`). 8.0
+removed that system and stores content blobs in the content-addressed store (`_cas/`), but the
+one-time layout migration only moved entities — it never adopted the `_cow/` content blobs. On a
+store that used the VFS (for example, a CMS with published pages), that left every VFS-backed read
+throwing `Blob metadata not found` and the pages 500ing. **8.0.12 adds an on-open recovery pass**
+that adopts every orphaned `_cow/` blob into `_cas/` — copying both the bytes and the metadata,
+idempotently and **non-destructively** (the `_cow/` originals are never deleted). It heals both a
+fresh 7→8 upgrade and a store **already** upgraded by an earlier 8.0.x that stranded them, with no
+operator action: just open the store under 8.0.12. An explicit force path is exposed as
+`brain.vfs.adoptOrphanedBlobs()` (returns `{ cowBlobs, adopted, alreadyPresent, incomplete }`). The
+automatic pre-upgrade backup is now **retained** if any blob can't be fully adopted
+(`incomplete > 0`) instead of being removed on entity-migration "success" alone. Affects any 7.x
+store that used the VFS; native-8.0 and non-VFS stores no-op on a cheap existence check. Full guide:
+`docs/guides/upgrading-7-to-8.md`.
+
+**2 — A cold open no longer re-derives durable indexes it can just load.**
+Opening a persisted store rebuilt the vector, graph, and metadata indexes from the canonical
+records even when the durable index state was present and loadable — an O(N) cost paid on every
+boot (measured at ~48 s on an ~11k-entity store). The rebuild gate now consults an honest
+per-provider durability signal (`init()` / `isReady()`) instead of an in-memory size heuristic, so
+a provider that has loaded (or can cheaply demand-load) its persisted index is not rebuilt. The
+built-in JS indexes keep today's behavior (a rebuild *is* their load path); the live query-time
+guards that self-heal a genuinely lost index are unchanged. **The zero-rebuild boot activates when
+the accelerator exposes the durability signal (`@soulcraft/cor@3.0.5`);** on earlier accelerator
+versions 8.0.12 falls back to the previous behavior safely — no regression, just no speedup yet.
+
+**3 — Unknown query operators now throw instead of silently returning nothing.**
+`find({ where })` had two gaps: the in-memory matcher (used for egress re-validation and historical
+reads) was missing several documented operators (`in`, `greaterThanOrEqual`, `lessThanOrEqual`), so
+it silently disagreed with the index path; and an unknown operator key (a typo, or `notIn` written
+where `not: { in }` was meant) was treated as a nested-object field and silently matched nothing.
+8.0.12 aligns the matcher to the full documented operator set and **validates the `where` filter
+up front**, throwing a typed `BrainyError('INVALID_QUERY')` naming the bad operator. Dotted paths
+remain the supported form for nested fields. If you relied on an unknown key silently returning `[]`,
+it now throws — the fix is to use the documented operator or dot-notation.
+
+## v8.0.11 — 2026-07-02 (no script shape can hang on brainy's internals)
+
+Completes v8.0.10's exit fix for every operation class. Two further mechanisms found and fixed:
+the `beforeExit` auto-flush hook looped forever on any script that never reaches `close()` (Node
+re-emits `beforeExit` after each event-loop drain and the async flush schedules new work — it now
+self-deregisters before its single flush, which still lands your buffered data before exit), and
+every background-maintenance interval (graph auto-flush, LSM compaction, metadata write-buffer,
+VFS/path-cache maintenance, statistics debounce) is now unref'd at creation. Verified against the
+published package: an `add + relate` script exits cleanly both with `close()` (~0.5 s) and with no
+teardown at all — with the data confirmed durable on reopen. A per-operation-class sweep test now
+asserts no ref'd timer survives `close()`, so this bug class stays closed.
+
+## v8.0.10 — 2026-07-02 (a bare script exits cleanly after `close()`)
+
+A minimal `init → add → close` script used to hang forever after `close()` returned — brainy held
+four process keep-alives it never released: the global SIGTERM/SIGINT shutdown hooks (never
+removed; now deregistered when the last live instance closes), an internal cache-fairness interval
+(unclearable by construction; now unref'd), and the VFS/PathResolver maintenance intervals
+(`close()` never shut the VFS down; now wired). Verified against the published package: the same
+script now exits ~1 ms after `close()` resolves. No API change; servers and long-running processes
+are unaffected.
+
+## v8.0.9 — 2026-07-02 (guarded plugin auto-detection — the "install it and it's on" contract)
+
+With the default config (`plugins` unset), brainy now **auto-detects the first-party accelerator**:
+installing `@soulcraft/cor` is the opt-in. The detection is guarded — everything except "not
+installed" fails loud:
+
+- Not installed → plain brainy, silently (the free path — zero noise, zero cost).
+- Installed and healthy → it loads and announces itself (`[brainy] Plugin activated`).
+- Installed but broken (unresolvable, invalid shape, failed activation, version mismatch) →
+ **`init()` throws.** An installed accelerator never silently vanishes behind the JS engines.
+- `plugins: []` / `false` = explicit opt-out; `plugins: ['@soulcraft/cor']` pins the exact list
+ (unchanged semantics).
+
+Also new: if a plugin activates but registers **zero** native providers (e.g. a licensing gate
+declining to engage), brainy warns loudly instead of leaving you to discover every query is running
+on the JS engines.
+
+> This supersedes v8.0.8's "plugins are explicit opt-in" wording, which documented the pre-GA
+> loader accurately but contradicted the published product contract ("add the package and it
+> activates"). 8.0.9 makes the contract true — with the loud-failure guarantees intact.
+
+## v8.0.8 — 2026-07-02 (docs-only patch on the GA)
+
+Corrects the README's scale-up section: the native provider is **not** auto-detected — plugins are
+**explicit opt-in** (`new Brainy({ plugins: ['@soulcraft/cor'] })`; brainy never auto-imports a
+package you didn't list, and a listed plugin that fails to load throws rather than silently falling
+back to the JS engines). Also fixes the stale `plugins` config comment in the public types and one
+phrase in the plugin-author guide. No code change.
+
+## v8.0.7 — 2026-07-02 (GA, npm tag `latest`)
+
+> **Why 8.0.7:** npm permanently retires unpublished version numbers, and `8.0.0`–`8.0.6` were
+> consumed by a January development cycle (published and immediately unpublished). `8.0.7` is
+> the first stable release of the 8.x line; there are no earlier stable 8.0.x releases.
+
+**8.0.7 is the first stable major on the u64-id core.** It ships in lockstep with the optional
+native provider's `3.0` (the billion-scale path). Everything below works standalone on the open-core
+JS engine — a native provider is feature-detected and only changes the scale ceiling, never the API.
+
+**Upgrading from 7.x: nothing to script.** The first time 8.0 opens a 7.x brain it upgrades itself —
+an automatic, observable, **coordinated migration lock** rebuilds every derived index from your
+canonical records while Brainy **blocks and queues** reads and writes, so no operation ever touches a
+half-built index and no write is ever lost. Budget seconds-to-minutes per brain at large sizes; small
+brains rebuild inline on open. A pre-upgrade hard-link **backup** is taken automatically (removed on
+success, kept on failure for rollback). Observe it via `getIndexStatus().migration`; a caller that
+hits the window gets a typed, retryable **`MigrationInProgressError`** (catch → HTTP 503 +
+`Retry-After`). Bounded by `migrationWaitTimeoutMs` (default 30 s — it bounds *your wait*, not the
+rebuild). Opt out of the backup with `migrationBackup: false`.
+
+### Headline changes
+
+- **The cold-open silent-`[]` class is gone.** On a cold reopen, filtered finds (`find({ where })`)
+ and graph reads (`find({ connected })` / `neighbors()` / `related()`) never silently return `[]`
+ for data that is actually present. With a native provider the durable indexes cold-serve every
+ filter and edge with zero rebuild; the open-core engine adds belt-and-suspenders guards that
+ self-heal from canonical data or throw a loud, typed error — never a silent empty result. Two new
+ exported errors: **`MetadataIndexNotReadyError`**, **`GraphIndexNotReadyError`**.
+
+- **Faster vector search (open-core).** The JS distance functions are allocation-free loops instead
+ of `reduce`: **~6× cosine, ~1.4× euclidean** (MEASURED — `tests/benchmarks/distance-microbench.mjs`,
+ 384-dim, median of 41). Numerically identical, so recall is unchanged. Exact metadata-filter
+ pushdown and scale-aware search width keep `find()` recall-correct as data grows.
+
+- **Per-write immutable history.** Every write is generation-stamped; `now()` / `asOf()` /
+ `transact()` read a consistent point in time, and a retention knob bounds history growth. Single
+ writes participate in the same immutable timeline as transactions.
+
+- **Billion-scale resident memory.** Nothing is O(N)-resident on the write or time-travel path —
+ per-id caches removed, the committed-generation ledger is an interval set, per-id history chains
+ are bounded (hot-window + LRU). Resident memory is independent of entity count.
+
+- **Deterministic, round-trip-free writes.** Supply your own ids, forward-reference not-yet-written
+ entities inside a `transact()`, `ifAbsent` upsert, and `brain.newId()` (uuidv7) — write graphs
+ without a write→wait→get round-trip.
+
+### Breaking changes (summary — the full migration guide follows below)
+
+- **Runtime floor: Node ≥ 22 / Bun ≥ 1.1** (Bun recommended). Compiler target ES2023; the DOM lib is
+ dropped — 8.0 is a Node/Bun/Deno engine with no browser path.
+- **`neural()` removed** and **`Db.search` removed** — use `find()` on the brain.
+- **Storage config is one `path` key.** Removed aliases now throw with a message pointing at `path`.
+- **`get()` omits the vector by default** — pass `{ includeVectors: true }` when you need it.
+- **Export/import type is `PortableGraph`** (was `BackupData`; wire tag `brainy-backup` →
+ `brainy-portable-graph`). Re-export any snapshots you version outside Brainy.
+- **Reserved `visibility` field** (`public` / `internal` / `system`) on nouns and verbs; `internal`
+ and `system` are excluded from normal reads by default.
+- **4 deprecated query operators removed** (`is`/`isNot`/`greaterEqual`/`lessEqual`) and reserved
+ keys in a `metadata` bag now **throw** by default — details in the rc notes below.
+
+> Post-rc.9 hardening folded into GA (no on-disk or provider-contract change vs `8.0.0-rc.9`):
+> the metadata cold-read guard, the auto pre-upgrade backup (with an `_id_mapper/*` mmap byte-copy
+> correctness fix), and a CI fix so a fresh clone builds green.
+
+### RC history (rc.1 → rc.9, npm tag `rc`)
+
+> **rc.9 changes (2026-07-01):**
+> - **Whole-brain auto-upgrade is now a coordinated, observable LOCK — supersedes rc.8's no-freeze
+> approach.** When a large brain's derived-index format changes (7.x→8.0), the native provider
+> rebuilds the indexes from the canonical records **in place** while Brainy **blocks and queues**
+> reads and writes — so no operation ever touches a half-built index. This is a clean *blocking
+> upgrade to a known-good state* (vs rc.8's online background swap): unknown/halfway states are
+> more dangerous than a bounded wait. Small brains still rebuild inline on open. New surface, all
+> additive: a typed, exported, retryable **`MigrationInProgressError`** (catch it → HTTP 503 +
+> `Retry-After`); `getIndexStatus()` gains **`migrating` + `migration`** progress (never gated —
+> the readiness-probe signal); a **`migrationWaitTimeoutMs`** config (default 30 s — it bounds the
+> *caller's wait*, NOT the rebuild, which is unbounded); `health()` / `checkHealth()` report the
+> upgrade without blocking. No effect without a native provider.
+> - **Faster vector search (open-core).** The JS distance functions are rewritten from `reduce` to
+> allocation-free loops — **~6× cosine, ~1.4× euclidean** (MEASURED, `tests/benchmarks/distance-microbench.mjs`,
+> 384-dim, median of 41). Numerically identical → recall unchanged.
+> - **Runtime floor + Bun.** Engines are now **Node ≥22 / Bun ≥1.1** (fixes a Node-24 `EBADENGINE`);
+> compiler target ES2023 with DOM dropped from the type lib (8.0 is Node/Bun/Deno-only, no browser
+> path). **Bun is recommended as a runtime** (`bun add` / `bun run`); the single-binary
+> `bun build --compile` is not a supported target (native addon can't embed + a Bun 1.3.10 codegen
+> regression). `.d.ts` stays TS-5.x-parseable.
+
+> **rc.8 additions (2026-06-30) — additive, no breaking change:**
+> - **No-freeze (online) whole-brain auto-upgrade.** When a derived-index format changes, a large
+> brain upgrades **without blocking** — the native provider rebuilds the new indexes in the
+> background and serves correct reads from the canonical records meanwhile, then atomically swaps;
+> no minutes-long freeze on the first open/query. (Small brains still auto-rebuild inline on open.)
+> New surface: an optional provider `isMigrating()` deference signal, a public `brain.stampBrainFormat()`
+> the provider calls once its swap verifies, and a `@soulcraft/brainy/brain-format` export so the
+> provider shares the `indexEpoch` constant. No effect without a native provider.
+
+> **rc.7 additions (2026-06-30) — additive, no breaking change:**
+> - **8.0 cold-graph self-heal.** `find({ connected })` / `neighbors()` / `related()` never
+> silently return `[]` on a cold open where the graph adjacency loaded its membership but not
+> its source→target edges — it self-heals (rebuild from storage) or throws a loud
+> `GraphIndexNotReadyError`, never empty-for-persisted-data. (The 8.0 equivalent of the 7.33.4
+> fix 7.x consumers already have; gates on the native provider's honest `isReady()` edge-readiness
+> signal.)
+> - **Billion-scale RAM — nothing O(N)-resident on the write/time-travel path.** Eliminated the
+> per-id storage caches (per-type counts now sourced from the record), made the
+> committed-generation ledger an interval set, and bounded the per-id time-travel history chains
+> (hot-window + LRU, with lock-light reconstruction so historical reads never stall writers).
+> Resident memory is now independent of entity count.
+> - **Whole-brain version handshake.** A `_system/brain-format.json` marker + `brain.formatInfo()`
+> + a shared, lockstep `indexEpoch`: a future derived-index format change auto-rebuilds the
+> indexes from the canonical records on open (non-destructively) — the foundation for
+> whole-brain auto-upgrade with the native provider. No effect on an unchanged brain.
+
+> **rc.6 additions (2026-06-29) — additive, no breaking change:**
+> - **Open-core perf**: HNSW delete is now O(in-degree) (was O(N²) for bulk delete) via a
+> reverse-adjacency index; the negation/absence operators (`ne`/`exists:false`/`missing:true`)
+> are served as a roaring-bitmap difference instead of materializing the whole corpus.
+> - **Native provider contract** (cor lockstep, optional/feature-detected — no effect without a
+> native provider): a cold-open `probeConsistency()` self-heal hook, and a `getIdsForFilter`
+> page bound so the native index can early-stop the unsorted `find({ type, where, limit })` path.
+> - **Test hygiene**: re-homed previously-unrun test suites into CI + a guard so a test file can
+> never silently fall outside every config again. No source/API change from these.
+
+
+**Affected products:** every consumer — this is a major release with removed
+surfaces, hard renames (no aliases, no deprecation period), and one flipped
+default. This entry **is** the migration guide: the find/replace pairs, the
+sed snippet, and the upgrade checklist below are the complete story — there is
+no separate migration doc. **`8.0.7` is GA on `latest`** — install with
+`npm i @soulcraft/brainy@latest`.
+
+> **rc.3–rc.5 additions (2026-06-24):**
+> - **Showcase-quality GA hardening (rc.5).** A full readiness audit closed a cold-init
+> version-coupling bug (a matched native provider could be rejected on first open), turned
+> silent storage-read failures into named `BrainyError`s (no more empty-result-on-error),
+> stopped the JS graph LSM from orphaning compacted SSTables, corrected the public docs to
+> the real API, and documented the two headline methods. Plus a dead/deprecated-code sweep.
+> - **BREAKING — 4 deprecated query operators removed.** `is`→`eq`, `isNot`→`ne`,
+> `greaterEqual`→`gte`, `lessEqual`→`lte`. The canonical operators and their clean long-form
+> aliases (`equals`/`notEquals`/`greaterThan`/`greaterThanOrEqual`/`lessThan`/`lessThanOrEqual`)
+> are unchanged — only the four redundant spellings are gone. Find/replace if you used them.
+> - **`find()` search mode** is now the single `searchMode: SearchMode` option (the redundant,
+> silently-ignored `mode` alias and the unwired `explain` flag were removed from `FindParams`).
+> - The phantom index-integrity guard (find() re-validates every result against its predicate)
+> shipped in rc.3; rc.4 added the `accel.isInitialized` gate + #35 at-gen candidate vectors.
+
+> **rc.1 additions (this entry predates them — full writeup lands at GA):** entity-id
+> normalization — `brain.newId()` (UUID v7) + v7 default ids, and non-UUID string ids are
+> transparently normalized to a stable UUID v5 (original key preserved under `_originalId`);
+> `brain.neural()` clustering and `Db.search()` removed (use `find({ vector })` /
+> `find()` + aggregation `GROUP BY`); storage config collapsed to one `path` key (the
+> pre-8.0 aliases now throw); and `queryAggregate` no longer hangs/staled min-max after a delete.
+> **Reserved fields in a `metadata` bag now throw by default** — passing a Brainy-reserved key
+> (`confidence`, `weight`, `subtype`, `visibility`, `service`, `createdBy`, `noun`/`verb`, `data`,
+> `createdAt`, `updatedAt`, `_rev`) inside `metadata` on `add()`/`update()`/`relate()`/`updateRelation()`
+> (untyped/JS callers — TypeScript already blocks it) is rejected with an Error naming the correct
+> param, instead of the old silent remap-or-drop. Pass reserved values as their dedicated top-level
+> params. Opt back into the legacy behavior with `new Brainy({ reservedFieldPolicy: 'warn' | 'remap' })`.
+
+> **rc.2 additions (2026-06-21):**
+> - **Graph performance + the `brain.graph` namespace.** New `brain.graph.subgraph(seeds, opts)`
+> (bounded multi-hop neighborhood → `{ nodes, edges, truncated }`) and `brain.graph.export(opts)`
+> (stream the whole graph in one O(N+E) pass — the right primitive for visualizing all data
+> instead of paging per node). `related({ node })` returns every edge incident to an entity in
+> **both directions** in one O(degree) call. Under the hood: `related({ from/to })` now stays
+> O(degree) under the default visibility filter (was a full scan), and the verb **and** noun
+> pagination walks are cursor-based, so a full edge/node walk is **O(N), not O(N²)** (a consumer
+> measured a 19k-edge whole-graph read drop from ~27s toward a single scan). These transparently
+> use a native graph engine when present (the `@soulcraft/cor` 3.0 acceleration layer) and fall
+> back to pure-TS adjacency otherwise.
+> - **Additive ergonomics:** `add({ upsert: true })` (create-or-update in one call — merges into an
+> existing id instead of overwriting), `find({ includeVectors: true })`, and `removeMany`'s batch
+> size is now storage-adaptive (was a hardcoded 10).
+> - **Correctness fixes (apply to all consumers, not just graph users):** `related()` results now
+> carry `visibility`; whole-graph/`getNouns` streaming no longer leaks `system`/`internal` entities;
+> and small-page cursor walks over nouns no longer loop. No API change — just correct behavior.
+
+> **rc.3 additions (2026-06-23):**
+> - **Graph analytics on the `brain.graph` namespace.** Three intent-level reads that answer
+> whole-graph questions in one call:
+> - **`brain.graph.rank(opts?)`** → `{ id, score }[]` descending — "which entities matter most"
+> (importance / centrality). `topK` to cap.
+> - **`brain.graph.communities(opts?)`** → `{ groups: string[][], count }` — "which things group
+> together". Weakly-connected components by default; `{ directed: true }` returns
+> strongly-connected components.
+> - **`brain.graph.path(from, to, opts?)`** → `{ nodes, relationships, cost } | null` — the best
+> route between two entities. Fewest hops by default; `{ by: 'weight' }` minimizes summed edge
+> weight (the 0–1 connection strength); `direction`, `type`, and `maxDepth` filters apply.
+> These are **intent contracts, not algorithm contracts** — the question is the promise, the
+> algorithm is the engine's choice. They transparently use the native `@soulcraft/cor` 3.0 graph
+> engine when present, and fall back to pure-TS kernels (PageRank, connected components / Tarjan
+> SCC, BFS / Dijkstra) otherwise. Both paths return identical shapes and respect the default
+> visibility filter (opt in with `includeInternal` / `includeSystem`).
+> - **Filtered vector search keeps its recall (`allowedIds` pushdown).** A `find({ query, where })`
+> that combines semantic search with a metadata filter now restricts the vector walk to the
+> matching candidates *inside* the search (walk-all, collect-allowed) instead of filtering the
+> top-k afterward — so a query whose nearest vectors are all filtered out still returns the best
+> matches that DO pass the filter, rather than coming back empty. No API change. With the native
+> `@soulcraft/cor` 3.0 stack the matched universe is forwarded as an opaque roaring buffer with
+> zero id materialization in TypeScript; the pure-JS path restricts the beam walk with a string set.
+> - **Historical (`asOf`) semantic search can skip the rebuild.** A filtered semantic query at a
+> past generation — `db.asOf(g).find({ query, where })` — no longer always rebuilds an ephemeral
+> in-memory vector index over every at-`g` vector (O(n@G)). When a native versioned vector engine
+> is registered and can serve the pinned generation, Brainy resolves the at-`g` filter universe from
+> its record layer (no rebuild) and routes the vector leg to the engine with the matched ids + the
+> generation; without one it falls back to the materialization (unchanged). The `VectorIndexProvider.search`
+> options gained an optional `generation?: bigint` ("omitted = now") to carry this — additive, the
+> built-in index ignores it. No public API change.
+> - **`find({ where, orderBy })` bounds the sort to the page.** A broad filter + `orderBy`
+> returning one page no longer materializes every matching sorted id (it produced the full sorted
+> match set, then sliced) — the page bound (`offset + limit`) is threaded into the column store's
+> top-K sort, so returning 20 rows from a million matches stays a bounded-K heap. No API change;
+> ordering and pagination are unchanged.
+> - **`brain.graph.subgraph()` accepts a query (query→expand).** The seed selector now takes not
+> just entity id(s) but a `find()` result or a `FindParams` query — `brain.graph.subgraph({ where:
+> { team: 'platform' } }, { depth: 1 })` runs the query and expands the neighborhood of every
+> match in one call. With the native engine a metadata-only query's matched universe is handed to
+> the traversal as an opaque set with no id materialization in TypeScript (the query→expand
+> fusion); the pure-JS path materializes the matched ids and expands from them. Existing
+> id-seeded calls are unchanged.
+
+### Headline: Database as a Value
+
+8.0 replaces Brainy's two overlapping version-control subsystems (copy-on-write
+branching and per-entity versioning, ~5,100 LOC combined) with **one
+mechanism**: generational MVCC over immutable, generation-stamped records,
+exposed through a Datomic-style immutable database value — the **`Db`**.
+
+```ts
+const db = brain.now() // pin the current state — O(1), no I/O
+
+await brain.transact([
+ { op: 'update', id: invoiceId, metadata: { status: 'paid' } }
+], { meta: { author: 'billing-service', reason: 'PO-7741' } })
+
+await db.get(invoiceId) // still 'pending' — pinned, forever
+await brain.get(invoiceId) // 'paid' — live
+await db.release() // unpin when done
+```
+
+What you get:
+
+- **`brain.now()`** — pins the current generation as an immutable `Db` view.
+ True snapshot isolation: the view reads exactly its pinned state no matter
+ what commits afterwards, including deletes. Readers never block writers and
+ writers never block readers.
+- **`brain.transact(ops, { meta, ifAtGeneration })`** — atomic multi-write
+ batches (`add` / `update` / `remove` / `relate` / `unrelate`) committed as
+ exactly one generation. Either every operation applies or none do.
+ `ifAtGeneration` is whole-store compare-and-swap (`GenerationConflictError`
+ on conflict) — the big sibling of the per-entity `ifRev` CAS from 7.31.
+ `meta` is reified Datomic-style into an append-only transaction log,
+ readable via `brain.transactionLog()`.
+- **`brain.asOf(generation | Date | snapshotPath, { exclusive? })`** — time
+ travel with the **full query surface**: `get()`, `find()` in every mode,
+ semantic search, graph traversal, cursors, aggregation, all at the pinned past
+ state. `exclusive: true` pins the generation immediately before the target
+ (strict-before).
+- **`db.with(ops)`** — speculative writes applied in memory on top of a view.
+ Nothing touches disk, the generation counter, or the indexes. What-if
+ analysis, then `transact()` the same ops for real.
+- **`db.persist(path)`** — instant self-contained snapshots. On filesystem
+ storage they are built from hard links (no entity data is copied; later
+ writes to the source can never alter the snapshot because rewrites swap
+ inodes). `brain.restore(path, { confirm: true })` replaces the whole store
+ from one; `Brainy.load(path)` opens one read-only with the full query
+ surface.
+- **Range verbs over history** — answer "what happened BETWEEN two points":
+ - **`db.since(Db | generation | Date)`** — the entity and relationship ids
+ committed transactions touched after an **exclusive** lower bound (now
+ accepts a generation or `Date`, not just a prior `Db`).
+ - **`brain.diff(a, b)`** — the touched ids CLASSIFIED as `{ added, removed,
+ modified }` (split by nouns/verbs) by resolving each at both endpoints; a
+ touched-but-reverted id lands in none of the buckets. Endpoints are a
+ generation, `Date`, or `Db`, in either order.
+ - **`brain.history(id, { from?, to? })`** — every distinct version of ONE
+ entity/relationship over a range, oldest first (`value: null` marks a
+ removal); each version equals `asOf(version.generation).get(id)`.
+ - **`brain.transactionLog({ from?, to?, limit? })`** — the commit log over an
+ **inclusive** generation/`Date` window (contrast `since`'s exclusive lower
+ bound); `limit` applies last.
+- **Every write is versioned (Model-B).** History granularity is **per-write**:
+ EVERY write — `transact()` AND a single-operation `add`/`update`/`remove`/
+ `relate` — is its own immutable generation. A `now()` pin always freezes
+ against later writes, and `asOf`/`since`/`diff`/`history`/`transactionLog`
+ reflect single-ops exactly like transacts. `transact()` groups several
+ operations into ONE atomic generation. Single-op history durability is **async
+ group-commit** (the live write is acknowledged immediately; its before-image
+ is batched to disk in one fsync) — a hard crash can lose only the last
+ un-flushed window's *history*, never live data, and a crash mid-flush is
+ recovered by drop-without-restore. A freshly-initialized brain is
+ `generation() === 0` with an empty `transactionLog()` (init-time
+ infrastructure is the un-versioned baseline); the first user write is gen 1.
+- **`new Brainy({ retention })`** — the retention knob governs auto-compaction on
+ `flush()`/`close()`: **unset → ADAPTIVE** (disk/RAM-pressure byte budget,
+ zero-config; a coordinator can drive it via `brain.setRetentionBudget(bytes)`)
+ · **`'all'`** → unbounded · **`{ maxGenerations?, maxAge?, maxBytes? }`** →
+ explicit caps (reclaim oldest-unpinned while ANY cap is exceeded). Live pins
+ are ALWAYS exempt.
+- **`brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? })`** — manual
+ reclaim on the same caps. Compaction never breaks a pinned read. `diff`/`since`
+ throw `GenerationCompactedError` below the horizon; `history` truncates to it.
+
+The precise guarantees are documented in:
+
+- [docs/concepts/consistency-model.md](docs/concepts/consistency-model.md) —
+ the guarantees, each proven by a dedicated test in
+ `tests/integration/db-mvcc.test.ts`
+- [docs/guides/snapshots-and-time-travel.md](docs/guides/snapshots-and-time-travel.md)
+ — the recipes: backup, restore, time-travel debugging, what-if, audit trails
+- [docs/ADR-001-generational-mvcc.md](docs/ADR-001-generational-mvcc.md) — the
+ design record: persisted layout, commit protocol, crash recovery, proof table
+
+### Portable export & import (PortableGraph v1)
+
+A portable, versioned graph format that serializes part or all of a brain to a
+single JSON document and restores it — the cross-environment, cross-version
+(7.x↔8.0), partial-or-whole companion to the native `persist()` snapshot.
+
+```ts
+// Export is a method on the immutable Db, so it composes with now()/asOf()/with()
+const graph = await brain.export({ collection: id }, { includeVectors: true })
+await otherBrain.import(graph, { onConflict: 'merge' }) // dedup-by-id
+
+;(await brain.asOf(gen)).export(sel) // time-travel export
+brain.now().with(ops).export(sel) // what-if export
+```
+
+- **`brain.export(selector?, options?)` / `db.export(...)`** → a versioned
+ `PortableGraph` document. Selectors reuse `find()`'s grammar: `{ ids }`,
+ `{ collection }` (alias `memberOf`, transitive `Contains`),
+ `{ connected: { from, depth } }`, `{ vfsPath }`, predicate
+ (`{ type, subtype, where, service }`), or the whole brain (omit) — and they
+ compose. Options: `includeVectors`, `includeContent` (VFS file bytes),
+ `includeSystem`, `edges: 'induced' | 'incident' | 'none'`.
+- **`brain.import(graph, options?)`** is polymorphic: a `PortableGraph` document is
+ restored as **one atomic transaction** (`onConflict: 'merge' | 'replace' | 'skip'`,
+ `reembed: 'auto' | 'never'`, `remapIds` for cloning); a file/buffer routes to the
+ existing CSV/PDF/Excel/JSON ingestion. No migration for ingestion callers.
+- **`validatePortableGraph(data)`** — dry-run structural/version/endpoint check before import.
+- **Format** (`format:'brainy-portable-graph'`, `formatVersion: 1`) is identical on
+ 7.x and 8.0; standard fields (`subtype`/`visibility`/`data`/…) are top-level,
+ `metadata` is custom-only. Current-state (no generation history — that lives in
+ `persist()`). Types exported from the package root.
+- **Naming:** the type is `PortableGraph` (with `PortableGraphEntity` /
+ `PortableGraphRelation`) — it is the portable interchange form of a graph, not a
+ backup (that role is `persist()`/`load()`). Renamed from the short-lived
+ `BackupData`/`'brainy-backup'` (introduced in 7.32.0, never adopted) with no
+ compatibility shim.
+- Guide: [docs/guides/export-and-import.md](docs/guides/export-and-import.md).
+
+### Aggregation: distinctCount over any value type
+
+`distinctCount` now counts distinct values of **any** type (strings, booleans,
+numbers) — previously it silently returned `0` for non-numeric fields, missing its
+primary use (distinct categories / users / tags). `percentile` (with `p`; median =
+`p: 0.5`), `stddev`, and `variance` are exact and delete-safe alongside the
+write-time `sum`/`count`/`avg`/`min`/`max`.
+
+### Removed surfaces and their replacements
+
+| Removed in 8.0 | Replacement |
+|---|---|
+| `brain.fork(name)` | Speculation: `db.with(ops)` (in-memory). Long-lived writable copy: `brain.restore()` a persisted snapshot into a fresh data directory. |
+| `brain.checkout(branch)` | Open the snapshot you want — `Brainy.load(path)` read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store. |
+| `brain.listBranches()` / `brain.getCurrentBranch()` / `brain.deleteBranch()` | A "branch" is now a name → snapshot-path mapping owned by your application. |
+| `brain.commit({ message })` | `brain.transact(ops, { meta })` — every batch is an atomic, logged, time-travelable commit; audit fields live in the database, not in commit messages. |
+| `brain.getHistory()` / `brain.streamHistory()` | `brain.transactionLog({ limit })` + `db.since(olderDb)`. |
+| `brain.versions.*` (`save` / `list` / `compare` / `restore` / `prune`) | A pinned `Db` or persisted snapshot captures *every* entity at that moment; `asOf()` reads any entity's past state; `restore()` for whole-store rollback. |
+| `brain.data()` (DataAPI: `clear` / `import` / `export` / `getStats`) | `brain.clear()`, `brain.import()`, `db.persist(path)` (the full-fidelity export format), `brain.restore(path, { confirm: true })`, `brain.stats()`. |
+| Cloud + browser storage adapters (`s3` / `gcs` / `r2` / `opfs` storage types, Azure Blob, plus the `storage.branch` option) | `filesystem` and `memory` only. Cloud backup is now an operator concern: `db.persist()` produces a plain directory — sync it with `gsutil` / `aws s3 cp` / `rclone` / `azcopy`. Passing a removed storage type throws at construction with this exact guidance. |
+| Browser support | Node.js 22 LTS or Bun ≥ 1.0 (`engines` enforces `node: 22.x`); Deno works through its Node compatibility layer. |
+| `brain.migrateToDiskAnn()` / `brain.migrateToHnsw()` | Gone — there is one canonical `'vector'` provider slot. A registered native vector provider selects its own operating mode; there is nothing to call. |
+| Distributed-clustering subsystem — `config.distributed`, the `DistributedRole` enum, the 13 `BRAINY_*` cluster env vars (`BRAINY_DISTRIBUTED`, `BRAINY_ROLE`, `BRAINY_HTTP_PORT`, `BRAINY_WS_PORT`, `BRAINY_DNS`, `BRAINY_SERVICE`, `BRAINY_NAMESPACE`, `BRAINY_CONSENSUS`, `BRAINY_COORDINATOR`, `BRAINY_NODES`, `BRAINY_REPLICAS`, `BRAINY_SHARDS`, `BRAINY_TRANSPORT`), the storage `setDistributedComponents` hook, and the `@soulcraft/brainy/config` preset/augmentation registry | Removed. Brainy 8.0 is a single-process library — there is no coordinator, peer discovery, or consensus to operate. Scale via: the optional native provider (`@soulcraft/cor` — on-disk DiskANN to 10B+ vectors on one machine); per-tenant pools (one instance + storage dir per tenant); and horizontal read scaling (many reader processes against one shared store, single writer). The `mode: 'reader' \| 'writer'` multi-process roles are unchanged. |
+| CLI `fork` / `branch` / `checkout` / `migrate` | CLI `snapshot ` / `restore ` / `history` / `generation`. |
+| `BrainyZeroConfig` type | Removed. 8.0 zero-config is automatic and internal — `new Brainy()` auto-detects storage, derives HNSW quality from `config.vector.recall`, picks `persistMode` from the adapter, and sizes caches to the detected container-memory limit. There is no config-generation type to import. |
+| `brain.isFullyInitialized()` / `brain.awaitBackgroundInit()` | `await brain.ready`. The built-in filesystem/memory adapters finish initialization synchronously inside `init()`, so a single readiness promise is the whole story — these methods were no-ops once cloud adapters were removed. |
+
+**Opening a 7.x store auto-migrates it.** 8.0 stores entities at the root; 7.x
+stored them branch-scoped under `branches//`. On first open, 8.0 collapses
+the **HEAD branch** (`config.storage.branch`, default `main`) to the 8.0 layout in
+place and rebuilds all derived state — no action required (`autoMigrate` defaults
+to `true`; set it to `false` to make 8.0 refuse a legacy layout with an explicit
+error instead). Two caveats:
+
+- **Non-HEAD branches are not imported.** 8.0 has no COW branches; only the head
+ branch's data is migrated. If you care about other branches, **export each one
+ while still on 7.x** (`fork → export`, or copy its store).
+- **Back up first for rollback.** The migration mutates the directory in place and
+ 8.0 does not keep the old layout — copy the data directory before the first 8.0
+ open if you need to roll back.
+
+### Renames — find/replace pairs
+
+Hard renames, no aliases. Every pair below is verified against the 8.0 source.
+
+| Brainy 7.x | Brainy 8.0 |
+|---|---|
+| `brain.delete(id)` | `brain.remove(id)` — matches the transact op vocabulary (`add` / `update` / `remove` / `relate` / `unrelate`) |
+| `brain.deleteMany(params)` | `brain.removeMany(params)` |
+| `DeleteManyParams` (type) | `RemoveManyParams` |
+| `brain.getRelations(paramsOrId)` | `brain.related(paramsOrId)` — the same name a pinned `Db` exposes (`db.related()`) |
+| `GetRelationsParams` (type) | `RelatedParams` |
+| CLI `delete ` | CLI `remove ` (JSON output `{ id, removed: true }`, matching `unrelate`) |
+| `HnswProvider` (from `@soulcraft/brainy/plugin`) | `VectorIndexProvider` |
+| `HNSWIndex` (exported class) | `JsHnswVectorIndex` |
+| Provider keys `'hnsw'` and `'diskann'` | `'vector'` (the only key Brainy consults for the vector index) |
+| `config.hnsw = { quantization, vectorStorage }` | `config.vector = { recall?, persistMode? }` (shape change — see below) |
+| `config.vector.quantization` (and 7.x `config.hnsw.quantization`) | **Removed.** The JS vector path is full-precision (exact float32 distances) only — quantization at scale is the native provider's job (e.g. DiskANN PQ). |
+| `hnswPersistMode: 'immediate' \| 'deferred'` (top level) | `config.vector.persistMode` (auto-selected from the storage adapter when omitted: `'immediate'` on filesystem, `'deferred'` on memory) |
+| `config.storage.type: 's3' \| 'gcs' \| 'r2' \| 'opfs'` | `'filesystem'` (or `'memory'` / `'auto'`) |
+| `config.storage.branch` | Removed (no COW branches) |
+| `brain.stats().indexHealth.hnsw` | `brain.stats().indexHealth.vector` |
+| Storage adapter contract `saveHNSWData()` / `getHNSWData()` | `saveVectorIndexData()` / `getVectorIndexData()` |
+| Cache category `'hnsw'` (`CacheProvider` union) | `'vectors'` |
+| `config.history = { retainGenerations, retainMs, autoCompact }` | `config.retention` — `'all'` \| `'adaptive'` \| `{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }`; **unset → adaptive** (was keep-100/7-days). The fields are now **caps**, not floors. |
+| `compactHistory({ retainGenerations, retainMs })` | `compactHistory({ maxGenerations, maxAge, maxBytes })` — `retainGenerations`→`maxGenerations`, `retainMs`→`maxAge` (now upper-bound CAPS), plus new `maxBytes`. |
+| `CompactHistoryOptions.retainGenerations` / `.retainMs` | `.maxGenerations` / `.maxAge` (+ `.maxBytes`) |
+
+Mechanical renames as one command (run from your repo root, review the diff):
+
+```bash
+find . -name '*.ts' -not -path '*/node_modules/*' -print0 | xargs -0 sed -i \
+ -e 's/\bHnswProvider\b/VectorIndexProvider/g' \
+ -e 's/\bHNSWIndex\b/JsHnswVectorIndex/g' \
+ -e 's/\bsaveHNSWData\b/saveVectorIndexData/g' \
+ -e 's/\bgetHNSWData\b/getVectorIndexData/g' \
+ -e 's/indexHealth\.hnsw\b/indexHealth.vector/g' \
+ -e "s/registerProvider('hnsw'/registerProvider('vector'/g" \
+ -e "s/registerProvider('diskann'/registerProvider('vector'/g" \
+ -e "s/getProvider('hnsw'/getProvider('vector'/g" \
+ -e "s/getProvider('diskann'/getProvider('vector'/g" \
+ -e 's/\.getRelations(/.related(/g' \
+ -e 's/\bGetRelationsParams\b/RelatedParams/g' \
+ -e 's/\.deleteMany(/.removeMany(/g' \
+ -e 's/\bDeleteManyParams\b/RemoveManyParams/g'
+```
+
+`delete()` → `remove()` is deliberately **not** in the snippet: `.delete(` is
+too common (`Map`/`Set`/storage adapters) for a blind sed. Find your
+`brain.delete(...)` call sites and rename them by hand.
+
+The config shape changes need a human (they are not 1:1 textual):
+
+```ts
+// 7.x
+new Brainy({
+ hnswPersistMode: 'deferred',
+ hnsw: { quantization: { enabled: true, bits: 8, rerankMultiplier: 3 } }
+})
+
+// 8.0 — config.vector is { recall?, persistMode? }
+new Brainy({
+ vector: {
+ recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
+ persistMode: 'deferred' // optional; auto-selected from storage when omitted
+ }
+})
+```
+
+`config.vector.recall` replaces the algorithm-internal HNSW knobs: `'balanced'`
+(the default) maps to exactly the 7.x default parameters, so an upgrade without
+explicit knobs changes nothing about search behaviour. `config.vector.quantization`
+and `config.hnsw.vectorStorage` are removed: the JS vector path now computes exact
+float32 distances throughout (no rerank/approximate branch), which is what made
+its quantization a memory *increase* — it stored both the full and the quantized
+vectors in RAM. Quantization at scale belongs to the native provider's own PQ.
+The on-disk vector index file names are **unchanged** (e.g. `hnsw-system.json`) —
+existing filesystem stores need no data migration for this rename; only the API
+surface moved.
+
+### Behavior changes
+
+- **`subtype` is required by default.** 7.30's opt-in strict mode is now the
+ default: every `add()` / `addMany()` / `update()` / `relate()` /
+ `relateMany()` / `updateRelation()` rejects writes whose type carries no
+ non-empty `subtype` (`src/brainy.ts:9759` — `requireSubtype ?? true`).
+ Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy
+ data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type
+ allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)`
+ still compose. `brain.audit()` reports entries missing a subtype and the new
+ `brain.fillSubtypes(rules)` migration helper backfills them — one rule per
+ NounType/VerbType (literal default or per-entry function), applied only to
+ entries still missing a subtype, returning
+ `{ scanned, filled, skipped, errors, byType }`. Idempotent: re-running fills
+ nothing, so a crashed run is resumed by running it again.
+- **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws
+ a teaching error if a caller passes an `id` field
+ (`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently
+ ignored (a generated UUID was used anyway); 8.0 says so out loud, because
+ graph-index providers key verb-int interning on the raw UUID bytes.
+- **`update({ vector })` is honored on its own.** 7.x only applied a supplied
+ pre-computed vector when `data` was also passed (it was silently dropped
+ otherwise). 8.0 applies an explicit `params.vector` with dimension
+ validation (`src/brainy.ts:1702-1713`; proven by
+ `tests/unit/brainy/update.test.ts:256`).
+- **`brain.clear()` re-resolves all indexes exactly as `init()` does** —
+ including plugin-provided vector/metadata/entityIdMapper factories and VFS
+ root re-creation. In 7.x, clearing a plugin-accelerated brain could leave
+ the metadata index rebuilt without its native id-mapper wiring.
+- **`find({ connected: { …, subtype } })` with `depth > 1` now works.** 7.30
+ threw `NOT_SUPPORTED` for multi-hop subtype-filtered traversal; 8.0
+ implements the BFS in the JS graph index and routes to a native provider
+ when one is registered.
+- **Every write advances the generation clock; only `transact()` writes
+ history.** Single-operation writes (`add` / `update` / `remove` / `relate`
+ outside `transact()`) bump `brain.generation()` so watermarks and CAS stay
+ sound, but they do not stage historical records — they remain visible
+ through earlier pins and are not reported by `db.since()`. Writes you want
+ to travel back through go through `transact()`. This is the documented
+ contract, stated rather than papered over.
+- **Reserved fields have one canonical location — enforced at every layer.**
+ The Brainy-owned field names (`RESERVED_ENTITY_FIELDS`:
+ `noun`/`subtype`/`createdAt`/`updatedAt`/`confidence`/`weight`/`service`/
+ `data`/`createdBy`/`_rev`; verb mirror `RESERVED_RELATION_FIELDS` with
+ `verb` for the type key) are now (1) a **compile error** inside any
+ `metadata` param — `add`/`update`/`relate`/`updateRelation` and the
+ matching `transact()` ops; (2) **normalized at write time** for untyped
+ callers — user-settable fields remap to their dedicated top-level param
+ (top-level wins; `update({metadata:{confidence}})` no longer silently
+ no-ops, closing a 7.x trap), system-managed fields drop with a one-shot
+ warning naming the right path; (3) **split at read time** through one
+ canonical helper, so `entity.metadata` / `relation.metadata` contain ONLY
+ custom fields on every read path — `get`, `find`, `related` (several
+ paginated/by-source/by-target paths previously echoed the full stored
+ record, including the `verb` type key, inside `metadata`), batch reads,
+ and historical `asOf()` reads. `related()` results now also surface
+ `confidence`/`updatedAt` top-level, and `updateRelation()` no longer
+ erases a relationship's `service`/`createdBy`. See "Reserved fields" in
+ `docs/concepts/consistency-model.md`.
+- **Named errors for not-found contract failures.** `update()`, `relate()`,
+ `updateRelation()`, `similar({ to: id })`, `transact()` planning, and
+ speculative `db.with()` planning now throw `EntityNotFoundError` /
+ `RelationNotFoundError` (both exported from the package root, both carrying
+ the missing `id` as a field) instead of a generic `Error`. Message texts are
+ unchanged, so existing `/not found/` matching keeps working — `instanceof`
+ is now the supported way to branch.
+- **Unchanged from 7.31:** per-entity `_rev`, `update({ ifRev })`
+ (`RevisionConflictError`), and `add({ ifAbsent })` work exactly as before,
+ and `ifRev` is also accepted on `transact()` update operations (a conflict
+ rejects the whole batch).
+
+### For provider and plugin authors
+
+The provider contracts in `@soulcraft/brainy/plugin` (`src/plugin.ts`) changed
+shape:
+
+- **`GraphIndexProvider` speaks BigInt at the boundary.** Reads take entity
+ ints and return entity/verb ints as `bigint[]`; the coordinator owns all
+ UUID ↔ int conversion. `addVerb(verb, sourceInt, targetInt)` receives the
+ resolved endpoint ints (also mirrored on the new optional
+ `GraphVerb.sourceInt` / `GraphVerb.targetInt` fields) and returns the
+ interned verb int. The batch reverse resolver
+ `verbIntsToIds(verbInts: bigint[]): Promise<(string | null)[]>` is
+ **required** — the provider owns durable verb-int interning; Brainy keeps
+ only a bounded in-memory warm cache. Implementations may stay u32 internally
+ (`Number(bigint)` narrowing is lossless under the `EntityIdSpaceExceeded`
+ guard) but must speak the bigint contract.
+- **`VersionedIndexProvider`** is a new optional 4-method capability —
+ `generation()`, `isGenerationVisible(g)`, `pin(g)`, `release(g)`, BigInt
+ generations — feature-detected on every registered index provider. Providers
+ are *post-commit appliers*: the storage-record commit is the source of
+ truth; on open, a provider behind the committed watermark replays the gap or
+ requests a rebuild. Explicit pins override any time-based snapshot retention
+ the provider has. Speculative `db.with()` overlays never reach providers. A
+ provider implementing this serves historical reads with **no rebuild** —
+ the open-core materializer is the correctness baseline, the provider is the
+ accelerator.
+- **The vector index registers under `'vector'`** and implements
+ `VectorIndexProvider`. The `'hnsw'` and `'diskann'` keys are retired and
+ never looked up. `CacheProvider`'s category union uses `'vectors'` in place
+ of `'hnsw'`.
+
+### Scale and cost — how the mechanisms behave
+
+Mechanism descriptions, not benchmark numbers (none are published for 8.0 yet):
+
+- `brain.now()` pins in O(1) and adds **zero read overhead until history
+ actually moves** — while nothing has committed past the pin, every read
+ delegates to the live fast paths.
+- A `transact()` commit pays O(ids touched) extra writes (before-images +
+ delta + manifest) — never O(store size). Single-operation writes pay an
+ in-memory counter bump with coalesced persistence.
+- `db.persist()` on filesystem storage is a hard-link farm: snapshot creation
+ copies no entity data and the snapshot shares disk space with the source.
+ Cross-device targets fall back to per-file byte copies; persisting an
+ in-memory brain serializes a real, durable, loadable store.
+- Historical **index-accelerated** queries (semantic search, traversal,
+ cursors, aggregation) on the open-core path pay a one-time O(n at the
+ pinned generation) in-memory index materialization per `Db`, cached until
+ `release()`. Record-path reads (`get`, metadata `find`, filter `related`)
+ at any reachable generation are served directly from the immutable record
+ layer. A native `VersionedIndexProvider` serves the same reads rebuild-free.
+
+The full cost model is in
+[docs/concepts/consistency-model.md](docs/concepts/consistency-model.md).
+
+### Upgrade checklist (from 7.x)
+
+1. **While still on 7.x:** back up your data directory (a plain file copy is
+ fine). If you used COW branches, materialize each branch you need — 8.0
+ does not read branch state. If you ran a cloud storage adapter, export your
+ data with 7.x tooling (`(await brain.data()).export()`) — 8.0 opens
+ `filesystem` and `memory` stores only.
+2. **Runtime:** Node.js 22 LTS or Bun ≥ 1.0.
+3. **Config:** change removed storage types to `'filesystem'`; delete
+ `storage.branch`; move `config.hnsw` / `hnswPersistMode` to
+ `config.vector` per the shape example above.
+4. **Renames:** run the sed snippet, then fix the manual shape changes.
+5. **Removed APIs:** replace `fork` / `checkout` / branch methods / `commit` /
+ `getHistory` / `streamHistory` / `versions` / `data()` per the table above.
+6. **Subtypes:** if your data predates subtype discipline, start with
+ `requireSubtype: false`, run `brain.audit()`, backfill with
+ `brain.fillSubtypes(rules)` (one rule per type — a literal default or a
+ function deriving the subtype from each entry), re-run `audit()` until
+ `total === 0`, then remove the opt-out so the 8.0 default enforcement
+ protects you going forward.
+7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` →
+ `snapshot ` / `restore ` / `history` / `generation`.
+8. **Plugin authors:** apply the contract renames and the BigInt graph
+ contract; optionally implement `VersionedIndexProvider`.
+9. **Verify, then take your first snapshot:** run your test suite, then
+ `const db = brain.now(); await db.persist('/backups/post-8.0-upgrade');
+ await db.release()`.
+
+What did **not** change: the core query surface (`find` / `search` / `get` /
+`add` / `update` / `relate` and friends), the aggregation engine, the VFS,
+neural extraction, and the on-disk entity/relationship layout — 8.0 creates
+its MVCC bookkeeping (`_system/generation.json`, `_system/manifest.json`,
+`_system/tx-log.jsonl`, `_generations/`) alongside the existing files on
+first use.
+
+### Native-provider (Cor) compatibility
+
+Brainy 8.0 pairs with `@soulcraft/cor` 3.0 — the renamed successor to the
+`@soulcraft/cortex` 2.x native provider — shipping in lockstep. The old
+`@soulcraft/cortex` 2.x cannot accelerate Brainy 8.0: 8.0 never consults the
+`'hnsw'` / `'diskann'` provider keys, and the graph/column provider contracts
+are BigInt at the boundary. Upgrade both packages together (`@soulcraft/brainy@8`
++ `@soulcraft/cor@3`).
+
+---
+
+## v7.31.2 — 2026-06-09
+
+**Affected products:** none in behaviour; affects anyone reading Brainy's TypeScript
+type definitions or coreTypes for the `hnsw.quantization.bits` option. Drop-in from 7.31.1.
+
+### Fix: stale comment claimed SQ4 vector quantization required a paid native provider
+
+Brainy ships a pure-JS SQ4 distance implementation (`distanceSQ4Js` in
+`src/utils/vectorQuantization.ts`) that's been part of the open-core path the whole
+time. Two type-definition comments incorrectly stated SQ4 required a native (paid)
+provider:
+
+```ts
+// coreTypes.ts:472 + types/brainy.types.ts:1123 — before
+bits?: 8 | 4 // default: 8 (SQ8). SQ4 requires cortex native.
+```
+
+The comment was simply wrong — open-core users can use SQ4 today, and the native SIMD
+acceleration is a drop-in via `setSQ4DistanceImplementation`, not a hard requirement.
+Corrected to:
+
+```ts
+bits?: 8 | 4 // default: 8 (SQ8). SQ4 has a pure-JS implementation;
+ // cortex's distance:sq4 SIMD provider accelerates it.
+```
+
+Both locations updated. Pure documentation; no behaviour change. Caught during an
+upstream open-core-boundary audit — thanks to whoever read the types carefully enough
+to spot the misleading comment.
+
+### Cortex compatibility
+
+Zero changes required. The fix is documentation only; runtime behaviour was already
+correct (`vectorQuantization.ts:412` ships `distanceSQ4Js`).
+
+---
+
+## v7.31.1 — 2026-06-09
+
+**Affected products:** any consumer running `@soulcraft/brainy` against `FileSystemStorage`
+that triggers concurrent flush / compaction (e.g. explicit `brain.flush()` calls overlapping
+with periodic background compaction, or a backup job + a flush job racing). Production-impacting
+when present: `brain.flush()` throws `ENOENT` on rename, downstream jobs that rely on flush
+(GCS / S3 / Azure backups, snapshot exports) fail continuously. Drop-in from 7.31.0.
+
+### Fixes a same-key rename race in `saveBinaryBlob`
+
+`FileSystemStorage.saveBinaryBlob` used a bare `${filePath}.tmp` suffix for its atomic-write
+temp file. Two concurrent same-key calls computed the **same** temp path; both `writeFile`d,
+the first `rename` succeeded, the second `rename` fired against a missing temp and threw
+`ENOENT`. The throw propagated up through `brain.flush()` and broke any downstream job that
+called it.
+
+```
+[job-queue] gcs-backup: failed — ENOENT: no such file or directory,
+ rename '/data/brainy-data/.../_column_index/owner/DELETED.bin.tmp'
+ -> '/data/brainy-data/.../_column_index/owner/DELETED.bin'
+```
+
+**Patch:** unique per-writer temp suffix (matches the pattern at six other atomic-write
+sites in the same file) + ENOENT swallow on rename (defensive: if the temp is gone, the
+work has already landed; `saveBinaryBlob` is idempotent for a given key) + temp cleanup
+on any other rename failure (no orphan `.tmp.*` files).
+
+```ts
+// before (7.31.0)
+const tmpPath = filePath + '.tmp'
+await fs.promises.writeFile(tmpPath, data)
+await fs.promises.rename(tmpPath, filePath)
+
+// after (7.31.1)
+const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
+await fs.promises.writeFile(tmpPath, data)
+try {
+ await fs.promises.rename(tmpPath, filePath)
+} catch (err) {
+ if ((err as NodeJS.ErrnoException).code === 'ENOENT') return
+ await fs.promises.unlink(tmpPath).catch(() => {})
+ throw err
+}
+```
+
+### Scope
+
+The fix is one site — `src/storage/adapters/fileSystemStorage.ts:992-999`. Audit results:
+
+- **`FileSystemStorage`** — six sibling atomic-write sites already used unique suffixes;
+ only `saveBinaryBlob` was the outlier. All six were rechecked. Clean.
+- **`OPFSStorage`** — writes via WritableStream (no tmp+rename). Not affected.
+- **`GCSStorage` / `R2Storage` / `AzureBlobStorage` / `S3CompatibleStorage`** — use
+ object-store `PUT` (atomic at the API). Not affected.
+- **`MemoryStorage`** — in-memory. Not affected.
+- **`HistoricalStorageAdapter`** — read-only. Not affected.
+- **COW / versioning / snapshot / HNSW / aggregation** — all delegate to storage adapters
+ via `saveBinaryBlob` / `writeObjectToPath`. They get the fix automatically by virtue of
+ using the patched primitive.
+
+### Beneficiary surfaces (broader than the reported bug)
+
+Column-store compaction (`_column_index//DELETED.bin`, segment files) is what the
+production report named, but `saveBinaryBlob` is also used by HNSW connection persistence
+(`_hnsw_conn/` — `src/hnsw/hnswIndex.ts:252`). If two flush paths ever raced on
+HNSW persistence for the same node, they would have hit the same ENOENT. No production
+reports of HNSW-side failures, but the fix removes the latent race.
+
+### Tests
+
+- **New `tests/integration/savebinaryblob-concurrent-rename.test.ts`** (4 tests):
+ - 20 concurrent `saveBinaryBlob(sameKey, ...)` calls all resolve without throwing
+ - No orphan `.tmp.*` siblings remain in the blob directory after concurrent writes
+ - Production-shape: two concurrent compactor passes over 10 column-index fields
+ (`owner`, `path`, `permissions`, `vfsType`, `modified`, `createdAt`, `accessed`,
+ `updatedAt`, `mimeType`, `size`)
+ - Single-writer path still produces correct bytes (no regression)
+- The first three tests reproduce the production `ENOENT: rename '...DELETED.bin.tmp' ->
+ '...DELETED.bin'` error verbatim on the pre-7.31.1 code path. Verified by stashing the
+ fix and watching the tests fail with the exact production error.
+- Existing suites unchanged: 1468/1468 unit. All integration suites pass.
+
+### Cortex compatibility
+
+Zero changes required. The bug and the fix are pure JS in the filesystem storage adapter;
+the Cortex mmap-filesystem adapter wraps `FileSystemStorage` and inherits the patch
+automatically.
+
+### Forward-compat
+
+The bare-`.tmp` pattern is gone repo-wide. 8.0's `Db.persist()` will route through the
+same patched primitive; no additional work needed when the immutable Db API ships.
+
+---
+
+## v7.31.0 — 2026-06-09
+
+**Affected products:** consumers that need multi-writer coordination (job
+schedulers, idempotent state machines, distributed locks, optimistic UI updates)
+or idempotent bootstrap of well-known singletons. Additive; drop-in from 7.30.2.
+
+### Per-entity `_rev` + `update({ ifRev })` + `add({ ifAbsent })`
+
+CouchDB / PouchDB / ETag-style optimistic concurrency. Every entity now carries a
+monotonic `_rev: number` that Brainy auto-bumps on every successful `update()`.
+Pass it back as `ifRev` to make read-modify-write race-safe without an external
+lock service. `ifAbsent` adds by-ID idempotent insert for singletons / config rows
+/ deterministic-ID bootstraps.
+
+**What's new:**
+
+- **`entity._rev: number`** — initialized to `1` on `add()`, bumped by `1` on every
+ successful `update()` that reaches storage. Surfaced on `get()` (both fast metadata-
+ only and full-vector paths), `find()`, `search()`, and on the `Result` flatten layer
+ for backward compatibility. Pre-7.31.0 entities without `_rev` are read as `1`.
+- **`update({ id, ..., ifRev: number })`** — optimistic-concurrency check. When
+ provided, throws `RevisionConflictError` if the persisted `_rev` no longer matches.
+ Carries `{ id, expected, actual }` for principled recovery. Omitting `ifRev` keeps
+ the prior unconditional-update behavior.
+- **`add({ id, ifAbsent: true })`** — by-ID idempotent insert. Returns the existing
+ `id` without writing if the entity already exists. No throw, no overwrite. Ignored
+ when `id` is omitted (a fresh UUID can never collide).
+- **`addMany({ items, ifAbsent: true })`** — applies the flag to every item; per-item
+ `ifAbsent` overrides the batch flag.
+
+### The lock recipe (single-process or distributed)
+
+```ts
+import { Brainy, RevisionConflictError } from '@soulcraft/brainy'
+
+const LOCK_ID = '...uuid...'
+
+await brain.add({
+ id: LOCK_ID,
+ type: NounType.Document,
+ data: { owner: null, expiresAt: 0 },
+ ifAbsent: true
+})
+
+async function tryAcquireLock(workerId: string, ttlMs: number) {
+ const lock = await brain.get(LOCK_ID)
+ if (!lock) throw new Error('lock document missing')
+ const state = lock.data as { owner: string | null; expiresAt: number }
+ if (state.owner && state.expiresAt > Date.now()) return false
+
+ try {
+ await brain.update({
+ id: LOCK_ID,
+ data: { owner: workerId, expiresAt: Date.now() + ttlMs },
+ ifRev: lock._rev
+ })
+ return true
+ } catch (err) {
+ if (err instanceof RevisionConflictError) return false
+ throw err
+ }
+}
+```
+
+### Why this is scoped to `_rev` + `ifAbsent`
+
+A public `brain.transaction(fn)` wrapper was on the table but was cut. The internal
+`TransactionManager` exposes raw `Operation` classes that take a `StorageAdapter`
+constructor argument; a high-level facade in 7.31.0 would have meant either
+(a) delegating to `brain.add()` / `update()` / `relate()` — each of which commits
+its own internal transaction before the closure returns, so multi-write rollback
+wouldn't actually work (a footgun), or (b) threading `tx?` through every internal
+write path — ~2 days of refactor with regression surface, and the API shape changes
+again in 8.0 anyway.
+
+The SDK-scheduler use case that drove the thread (read-then-CAS lock pattern) is
+fully solved by `_rev` + `ifRev`. Multi-write atomicity is the 8.0 `brain.transact()`
+use case, where the Datomic-style immutable Db makes it atomic by construction. We
+chose not to ship a worse version in the interim.
+
+### How `_rev` interacts with branches and the snapshot API
+
+| System | What it tracks | When it advances |
+|---|---|---|
+| `_rev` (NEW) | Per-entity write counter | Auto, on every successful `update()` |
+| `brain.versions.save()` | Named snapshots per entity | Explicit — you call `save()` |
+| `brain.fork()` / branches | Whole-brain copy-on-write | Explicit — you call `fork(name)` |
+| VFS file versioning | Per-VFS-file snapshots | Same as `brain.versions.save()` |
+
+`_rev` is independent. Each branch has its own copy of every entity, so each
+branch has its own `_rev` per entity (same as every other field under COW).
+Snapshots taken via `brain.versions.save()` capture the entity at that moment
+including its `_rev` at that time; the snapshot's own `version: number` is the
+snapshot index, distinct from `_rev`.
+
+### Docs
+
+- **New `docs/guides/optimistic-concurrency.md`** (public, indexed at
+ soulcraft.com/docs after portal deploy) — full reference: the lock pattern,
+ read-modify-write retry, idempotent bootstrap, how `_rev` interacts with branches /
+ snapshots / VFS, what's coming in 8.0.
+- `docs/api/README.md` `add()` and `update()` entries get the new params + tips
+ pointing at the guide.
+
+### Tests
+
+- **New `tests/integration/rev-and-ifabsent.test.ts`** (18 tests): rev initialization,
+ rev bump on update, ifRev pass / fail / omit / legacy-entity-fallback, ifAbsent
+ with custom id / no id / batch propagation / per-item override, plus a full
+ SDK-scheduler-style two-concurrent-CAS-updaters scenario.
+- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-enforcement
+ 30/30, strict-mode-self-test 13/13, find-limits 9/9. Unit 1468/1468.
+
+### Cortex compatibility
+
+Zero changes required. `_rev` is a metadata column already supported by
+`NativeColumnStore`; the auto-bump runs in Brainy JS before any storage call.
+Cortex never sees the per-entity counter — it's a single integer field updated
+alongside every other metadata field on the existing write paths.
+
+### 8.0 forward-compat
+
+`_rev`, `ifRev`, `RevisionConflictError`, and `ifAbsent` all survive the 8.0 Db
+redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for
+whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record
+patterns, generation-based for "did the world move under me."
+
+---
+
+## v7.30.2 — 2026-06-08
+
+**Affected products:** any consumer with `find({ limit })` call sites that pass values
+≥ ~9000 — a common safety-cap pattern in production code (`limit: 10_000` against
+type-filtered queries that typically return 10–500 entities). Additive; drop-in
+from 7.30.1.
+
+### Why
+
+Brainy 7.30 introduced a memory-derived cap on `find({ limit })` to prevent OOM
+from runaway queries. The cap was sound in intent but **~4× too conservative in
+calibration**: the formula assumed 100 KB per result while typical entity footprint
+is 7-10 KB (384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
+free-memory box this capped `limit` at 9000, breaking production dashboards where
+cascading 500s from queries with `limit: 10_000` silently degraded the `Promise.all`
+loading the canvas.
+
+7.30.2 recalibrates the formula and changes the enforcement from synchronous-throw
+to two-tier (warn-then-throw) so existing pre-7.30 code doesn't break on upgrade.
+
+### Recalibrated formula — `25 KB` per result (was `100 KB`)
+
+The auto-configured cap now uses a realistic per-result size:
+
+| Memory source | 7.30.0–7.30.1 cap | 7.30.2 cap |
+|---|---|---|
+| 4 GB Cloud Run container | 10 000 | 40 000 |
+| 2 GB container | 5 000 | 20 000 |
+| 900 MB free system memory | 9 000 | ~36 000 |
+| `reservedQueryMemory: 1 GB` | 10 000 | 40 000 |
+
+Hard ceiling stays at 100 000. Existing `BrainyConfig.maxQueryLimit` /
+`reservedQueryMemory` overrides continue to work unchanged.
+
+### Two-tier enforcement — warn, then throw
+
+**Below cap (`limit ≤ maxLimit`):** silent pass. No signal, no friction. Unchanged.
+
+**Soft tier (`maxLimit < limit ≤ 2 × maxLimit`)** *NEW*: one-time warning per call
+site (dedup keyed on stack location + limit value), query proceeds. Pre-7.30.2 code
+that relied on the cap silently allowing safety-cap limits keeps working — the
+warning teaches the recipe so consumers can fix it intentionally.
+
+**Hard tier (`limit > 2 × maxLimit`)** *NEW*: throw with the same message format the
+warning uses. Real OOM territory; the cap stops being a recommendation and becomes
+a guardrail.
+
+The 2× soft margin is chosen to absorb existing safety-cap patterns (`limit: 10_000`
+against a 9 K-cap box) without disabling OOM protection. Real OOM territory on a JS
+in-memory brain is hundreds of thousands of results, not 10× the safety cap.
+
+### Improved error / warning message (mirrors the 7.30.1 enforcement-error format)
+
+Before (7.30 → 7.30.1):
+```
+limit exceeds auto-configured maximum of 9000 (based on available memory)
+```
+
+After (7.30.2):
+```
+find({ limit: 10000 }) exceeds the auto-configured query limit of 9000 (basis:
+available free memory). Choose one:
+ • Increase the cap: new Brainy({ maxQueryLimit: 10000 })
+ • Reserve more memory: new Brainy({ reservedQueryMemory: 256000000 })
+ • Paginate: split the query with { limit, offset } pages
+ at OrderService.loadDashboard (/app/src/orders/dashboard.ts:142:18)
+Docs: https://soulcraft.com/docs/guides/find-limits
+```
+
+Caller location comes from the same `findCallerLocation()` helper introduced in
+7.30.1 (now extracted to `src/utils/callerLocation.ts` so both subtype enforcement
+and limit enforcement share it).
+
+### New docs
+
+New `docs/guides/subtypes-and-facets.md`-style guide at
+`docs/guides/find-limits.md` (public, indexed) — explains the cap, the four memory
+sources the auto-config considers, the three escape valves (`maxQueryLimit`,
+`reservedQueryMemory`, pagination), and when to use which. Explicit "pagination is
+the future-proof pattern" callout — the cap can get tighter in 8.0 (Datomic-style
+`Db.find()` may make per-call limits stricter to keep snapshot semantics cheap),
+but pagination keeps working unchanged.
+
+`docs/api/README.md` `find()` entry gets a one-paragraph `limit` tip + pointer
+to the new guide.
+
+### Tests
+
+- **New `tests/integration/find-limits.test.ts`** (9 tests): below-cap silent
+ pass; soft-tier warns once per call site (dedup verified by exercising same vs.
+ different source lines); soft-tier message format (names all three escape
+ valves + docs link); soft-tier message includes caller location; hard-tier
+ throws; hard-tier message format same as soft-tier; consumer `maxQueryLimit`
+ override raises the cap and shifts both tiers accordingly; **the pre-7.30.2
+ regression scenario** explicitly covered — `limit: 10_000` against a
+ `maxQueryLimit: 9000` cap now warns and passes instead of throwing.
+- Existing unit tests in `tests/unit/utils/memoryLimits.test.ts` updated to
+ reflect the recalibrated formula (5 tests: hardcoded values for
+ container / reserved / free-memory paths bumped 4×).
+- Existing `paramValidation.test.ts` "should auto-limit based on system memory"
+ test extended to cover the new three-tier semantics (below-cap pass /
+ soft-tier silent / hard-tier throw).
+- All prior suites still green: subtype-and-facets 26/26, verb-subtype-and-
+ enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
+
+### Cortex compatibility
+
+**No Cortex changes required for 7.30.2.** Every change is JS-side: formula
+recalibration runs in `ValidationConfig.constructor()`, two-tier enforcement runs
+in `validateFindParams()`, both fire before any storage/index/Cortex call. Cortex
+2.x and the pending Cortex 3.0 work both stay compatible without code changes.
+
+The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()`
+may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point
+whose rollout staging now also covers query limits.
+
+### What consumers should do
+
+- **If you've been carrying a `limit: 9000` workaround for the 7.30.0–7.30.1
+ cap:** drop it on bump to 7.30.2 — existing `limit: 10_000` patterns now
+ warn (teaching the recipe) but no longer throw. The warning is
+ one-time-per-call-site, so production logs don't spam.
+- **All other consumers:** no action required if no enforcement-error was being
+ hit. If you see new `[Brainy]` warnings in logs after upgrade, follow the
+ recipe in the warning or read `docs/guides/find-limits.md`.
+- **SDK wrappers:** wrapper-level pools that auto-construct `Brainy` instances
+ should consider surfacing `maxQueryLimit` / `reservedQueryMemory` as
+ consumer-facing config; Brainy now documents the knob explicitly.
+
+---
+
+## v7.30.1 — 2026-06-08
+
+**Affected products:** anyone running a brain where a platform layer (SDK, framework wrapper)
+has registered `brain.requireSubtype()` rules on common NounTypes, AND anyone preparing for
+the upcoming Brainy 8.0 default-on strict mode. Additive; drop-in from 7.30.0. No behavior
+change for consumers not using strict-mode enforcement.
+
+### Why
+
+Production incident 2026-06-08: a consumer using SDK 3.20.0 (which registers
+`requireSubtype()` rules on `NounType.{Event, Collection, Message, Contract, Media, Document}`)
+saw their booking flow start returning 500s because `brain.add({ type: NounType.Event, ... })`
+calls in their codebase lacked `subtype`. An audit of Brainy's OWN source revealed 14 HIGH-risk
+internal write paths that also omit subtype — VFS move/copy/symlink edges, aggregation
+materializer, neural extraction, importers, integrations (Sheets/OData), MCP client, CLI.
+Any consumer running `requireSubtype()` rules on those NounTypes was one step away from breaking
+Brainy's own infrastructure paths, not just their own code. 7.30.1 closes both gaps before
+8.0 ships and makes strict mode the default.
+
+### New — `brain.audit()` diagnostic
+
+Find entities and relationships missing a `subtype` value, grouped by type. The companion to
+`migrateField()` (and to 8.0's `fillSubtypes()`): answers "what would break if I enabled
+strict subtype enforcement?".
+
+```typescript
+const report = await brain.audit()
+// {
+// entitiesWithoutSubtype: { event: 24, document: 3 },
+// relationshipsWithoutSubtype: { relatedTo: 1402 },
+// total: 1429,
+// scanned: 8400,
+// recommendation: 'Found 1429 entries without subtype. Migrate via `brain.migrateField()`
+// (7.x) — or wait for `brain.fillSubtypes()` (8.0) which closes the same
+// gap with caller-supplied rules.'
+// }
+```
+
+VFS infrastructure entities are excluded by default (they bypass enforcement via
+`metadata.isVFSEntity` markers). Pass `{ includeVFS: true }` to surface them.
+
+### New — Improved enforcement error messages
+
+The error fired when subtype enforcement rejects a write now includes:
+
+1. **The caller's source location** — extracted from the JavaScript stack so you see your own
+ call site, not a Brainy internal frame. Eliminates the "grep your repo for `brain.add`" step.
+2. **Specific guidance** — points at the registered vocabulary when one exists; mentions
+ brain-wide strict mode and the `except` escape valve when not; otherwise the
+ `brain.requireSubtype()` registration recipe.
+3. **A documentation link** — `https://soulcraft.com/docs/guides/subtypes-and-facets#strict-mode`
+ for the canonical migration recipe.
+
+Before:
+```
+add(): NounType.Event requires subtype but got undefined. Register vocabulary via brain.requireSubtype().
+```
+
+After:
+```
+add(): NounType.event requires subtype but got undefined.
+ at OrderService.getOrCreateByToken (/app/src/orders/draft.ts:42:23)
+ Pass one of: standard, recurring, milestone.
+ Migration recipe: https://soulcraft.com/docs/guides/subtypes-and-facets#strict-mode
+```
+
+### Internal subtype labels — Brainy's own infrastructure paths
+
+Every internal Brainy write path now sets a stable, queryable `subtype`. Consumers don't need
+to do anything for these — they're documented here so you can query Brainy-managed data:
+
+| Code path | NounType / VerbType | Subtype label |
+|---|---|---|
+| VFS root directory `/` | `Collection` | `'vfs-root'` |
+| VFS subdirectories | `Collection` | `'vfs-directory'` |
+| VFS files | mime-driven (e.g. `Document`/`Code`/`Image`) | `'vfs-file'` |
+| VFS symlinks | `File` | `'vfs-symlink'` (NEW — distinct from `'vfs-file'`) |
+| VFS Contains edges (create + move/copy/symlink/batch) | `Contains` | `'vfs-contains'` |
+| Aggregation materialized output | `Measurement` | `'materialized-aggregate'` |
+| Import-source provenance entity | `Document` | `'import-source'` |
+| Importer-extracted entities | extractor-driven type | `'imported'` |
+| Importer placeholder targets | `Thing` | `'import-placeholder'` |
+| Neural extraction | extractor-driven type | `'extracted'` |
+| GoogleSheets API entity writes | request-driven | `'imported-from-sheets'` |
+| OData API entity writes | request-driven | `'imported-from-odata'` |
+| MCP message storage | `Message` | `'mcp-message'` |
+| `brainy add` CLI default | user-supplied type | `'cli-add'` |
+| `brainy relate` CLI default | user-supplied verb | `'cli-relate'` |
+
+Query examples:
+
+```typescript
+// Every VFS-managed file in your brain
+await brain.find({ subtype: 'vfs-file' })
+
+// Document breakdown — distinguishes import-source from extracted/imported/user content
+brain.counts.bySubtype(NounType.Document)
+// → { 'import-source': 12, 'imported': 847, 'extracted': 34, 'vfs-file': 102, ... }
+```
+
+### Caller-supplied `defaultSubtype` on importers and extraction
+
+Importer + extraction paths now accept a caller-supplied `defaultSubtype` config so consumers
+can tag a whole batch with their own provenance label instead of the Brainy default:
+
+```typescript
+// SmartImportOrchestrator / ImportCoordinator / NeuralImport all accept this
+await brain.importer.import(file, {
+ defaultSubtype: 'customer-upload-2026q2', // your batch label
+ // ...
+})
+```
+
+Precedence: extractor-set subtype (highest) → caller's `defaultSubtype` → Brainy default
+(`'imported'` for importers, `'extracted'` for extractors).
+
+### CLI `--subtype` flag
+
+`brainy add` and `brainy relate` gain a `--subtype ` (`-s`) flag for use with
+strict-mode brains:
+
+```bash
+brainy add "Avery Brooks — runs the AI lab" --type person --subtype employee
+brainy relate alice manages bob --subtype direct
+```
+
+When the flag isn't supplied, the CLI uses `'cli-add'` / `'cli-relate'` as defaults so
+ad-hoc CLI usage still works against strict-mode brains.
+
+### Cortex compatibility
+
+**No Cortex changes required for 7.30.1.** Every change is JS-side: internal subtype labels are
+arbitrary strings stored transparently by Cortex; `brain.audit()` runs purely on JS via existing
+`storage.getNouns()` / `getVerbs()` pagination; the CLI is JS-only; error messages fire in
+Brainy before any native call.
+
+**For Cortex 3.0 (forward-looking, not blocking):**
+
+- **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)).
+ A native implementation reading from a "null-subtype" bitmap in the column store would be
+ O(buckets).
+- **Strict-mode parity test.** Cortex should mirror Brainy's new
+ `tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any
+ latent bug where native writes bypass JS validation.
+- **Reserved-label awareness (optional).** Brainy's internal labels (`'vfs-*'`,
+ `'materialized-aggregate'`, `'imported'`, `'extracted'`, `'mcp-message'`, `'cli-*'`) become
+ a documented part of the 8.0 contract; useful for telemetry that surfaces "X% of entities are
+ Brainy-managed infrastructure".
+
+### Docs
+
+Updated `docs/guides/subtypes-and-facets.md` with a new "Strict mode in practice" section
+covering the SDK_CORE_VOCABULARY pattern, a 4-step migration recipe, the Brainy-internal label
+reference table, and an 8.0 forward-look. `docs/api/README.md` documents `brain.audit()` and
+adds a strict-mode tip to the `add()` / `relate()` reference entries.
+
+### Tests
+
+- **New `tests/integration/strict-mode-self-test.test.ts`** (13 tests): creates a brain under
+ a realistic consumer vocabulary shape + brain-wide strict mode, then exercises every
+ internal Brainy path (VFS root/mkdir/writeFile/cp/mv/ln, aggregation engine, audit
+ diagnostic, error-message UX). Zero rejections expected.
+- Existing 7.30.0 + 7.29.0 integration suites unchanged: 26/26 + 30/30.
+- Unit suite unchanged: 1468/1468.
+
+---
+
+## v7.30.0 — 2026-06-05
+
+**Affected products:** consumers modeling typed relationships with sub-classification
+(direct vs dotted-line management; spouse / sibling / colleague; collaborator vs competitor;
+etc.), and anyone wanting to enforce the pairing of `type` + `subtype` on every write.
+Additive; drop-in from 7.29.x. No deprecations.
+
+### Symmetric — `subtype` on relationships (parity with 7.29.0 nouns)
+
+`subtype?: string` is now a first-class standard field on every relationship, mirroring the
+noun-side work shipped in 7.29.0. Verbs and nouns are now first-class peers — every API
+available on the noun side has a verb-side mirror.
+
+```typescript
+const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' })
+const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' })
+
+await brain.relate({
+ from: ceoId,
+ to: vpId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct' // sub-classification on the edge
+})
+```
+
+**Read & filter:**
+
+```typescript
+// Fast-path filter — column-store hit, not metadata fallback
+const direct = await brain.getRelations({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct'
+})
+
+// Set membership
+const all = await brain.getRelations({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: ['direct', 'dotted-line']
+})
+
+// Traversal filter (depth-1 in JS; multi-hop lands on Cortex native)
+const reports = await brain.find({
+ connected: { from: ceoId, via: VerbType.ReportsTo, subtype: 'direct', depth: 1 }
+})
+```
+
+### New — `updateRelation()` closes a long-standing gap
+
+Verbs previously had no update method — the only way to change a relationship was
+delete-then-recreate, which lost the relation id. 7.30 ships `brain.updateRelation()`:
+
+```typescript
+await brain.updateRelation({ id: relId, subtype: 'dotted-line' })
+await brain.updateRelation({ id: relId, weight: 0.5, confidence: 0.9 })
+
+// Change verb type — re-indexes in graph adjacency, id preserved
+await brain.updateRelation({ id: relId, type: VerbType.WorksWith })
+```
+
+### New — O(1) verb subtype counts via the persisted rollup
+
+`_system/verb-subtype-statistics.json` mirrors the noun-side rollup shipped in 7.29.0.
+Per-VerbType-per-subtype counts are maintained incrementally and persisted; reads are O(1):
+
+```typescript
+brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
+// → { direct: 12, 'dotted-line': 3 }
+
+brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct') // O(1) point
+// → 12
+
+brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
+// → [['direct', 12], ['dotted-line', 3]]
+
+brain.relationshipSubtypesOf(VerbType.ReportsTo)
+// → ['direct', 'dotted-line']
+```
+
+### New — `brain.requireSubtype(type, options)` per-type enforcement
+
+Unified API for noun OR verb types — register specific types as requiring a subtype,
+optionally with a fixed vocabulary:
+
+```typescript
+brain.requireSubtype(NounType.Person, {
+ values: ['employee', 'customer', 'vendor'],
+ required: true
+})
+
+brain.requireSubtype(VerbType.ReportsTo, {
+ values: ['direct', 'dotted-line'],
+ required: true
+})
+
+// Now this throws — Person requires subtype:
+await brain.add({ type: NounType.Person, data: 'no subtype' })
+
+// And this throws — 'matrix' isn't in the registered vocabulary:
+await brain.relate({ from: a, to: b, type: VerbType.ReportsTo, subtype: 'matrix' })
+```
+
+### New — Brain-wide strict mode
+
+`new Brainy({ requireSubtype: true })` enforces subtype on every public write across the
+whole brain. Composes with per-type rules; per-type rules win when both apply.
+
+```typescript
+// Every write must include subtype
+const brain = new Brainy({ requireSubtype: true })
+
+// Exempt specific types (e.g. catch-all Thing)
+const brain2 = new Brainy({
+ requireSubtype: { except: [NounType.Thing, NounType.Custom] }
+})
+```
+
+When strict mode is on:
+- Every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()`
+ rejects writes missing a subtype on a non-exempt type.
+- `addMany()` and `relateMany()` validate every item BEFORE any storage write —
+ atomic-fail semantics, no partial writes.
+- Brainy's own infrastructure writes (VFS root, directories, files) bypass via the
+ `metadata.isVFSEntity: true` marker so existing consumers' VFS usage continues to work.
+
+The brain-wide flag becomes the default in 8.0.0; the type-level `subtype` field becomes
+required at the type system level. The full 8.0 contract upgrade is coordinated through the
+internal platform handoff (`CTX-SUBTYPE-8.0-CONTRACT`).
+
+### `migrateField()` extended to verbs
+
+The migration helper shipped in 7.29.0 now walks verbs too via the new `entityKind` option:
+
+```typescript
+// Migrate verb-side metadata.kind → top-level subtype
+await brain.migrateField({
+ from: 'metadata.kind',
+ to: 'subtype',
+ entityKind: 'verb'
+})
+
+// Or walk nouns and verbs in one pass
+await brain.migrateField({
+ from: 'metadata.kind',
+ to: 'subtype',
+ entityKind: 'both'
+})
+```
+
+Default is `entityKind: 'noun'` (backward-compatible).
+
+### Symmetry — noun + verb capability matrix
+
+7.30 closes every gap between nouns and verbs. Every capability available on the noun side
+has a verb-side mirror:
+
+| Capability | Nouns | Verbs |
+|---|---|---|
+| `subtype` top-level field | ✓ (7.29) | ✓ (7.30 new) |
+| Standard-field set | `STANDARD_ENTITY_FIELDS` | `STANDARD_VERB_FIELDS` (new) |
+| Field resolver helper | `resolveEntityField` | `resolveVerbField` (new) |
+| Statistics rollup | `_system/subtype-statistics.json` | `_system/verb-subtype-statistics.json` (new) |
+| Counts breakdown | `counts.bySubtype` / `topSubtypes` / `subtypesOf` | `counts.byRelationshipSubtype` / `topRelationshipSubtypes` / `relationshipSubtypesOf` (new) |
+| Fast-path filter | `find({type, subtype})` | `getRelations({type, subtype})` + `find({connected, subtype})` (new) |
+| Update method | `update()` | `updateRelation()` (new — closed pre-7.30 gap) |
+| Migration helper | `migrateField()` | `migrateField({entityKind: 'verb'\|'both'})` (new) |
+| Enforcement | `requireSubtype(NounType, ...)` | `requireSubtype(VerbType, ...)` — one unified API |
+| Brain-wide strict mode | `new Brainy({ requireSubtype })` covers both | same |
+
+### Cortex compatibility
+
+Verb subtype works under Cortex out of the box via auto-field-indexing. Cortex parity items
+for the verb side (native query planner recognition, `verbSubtypeCountsByType` native rollup,
+per-edge subtype on native graph adjacency for multi-hop traversal filtering) ship in the
+next Cortex release. Not a Brainy blocker.
+
+### Docs
+
+Full guide: `docs/guides/subtypes-and-facets.md` (extended with Layer V + Enforcement
+sections). The verb subtype + enforcement APIs are documented in `docs/api/README.md`.
+
+---
+
+## v7.29.0 — 2026-06-04
+
+**Affected products:** anyone modeling entities with per-product sub-classification — every
+consumer that's been reaching for `metadata.kind`, a custom `metadata.subtype` field, a
+`data.kind` shape inside the payload, or similar ad-hoc conventions. Additive; drop-in from
+7.28.x. No deprecations.
+
+### New — `subtype` promoted to a top-level standard field
+
+`subtype?: string` is now a first-class standard field on every entity, alongside `type` /
+`confidence` / `weight`. Use it as the platform-standard primitive for sub-classifying entities
+within a NounType (`Person` → `'employee'` / `'customer'`; `Document` → `'invoice'` / `'contract'`;
+`Event` → `'milestone'` / `'meeting'`):
+
+```typescript
+await brain.add({
+ data: 'Avery Brooks — runs the AI lab',
+ type: NounType.Person,
+ subtype: 'employee', // top-level write param
+ metadata: { department: 'ai-lab' }
+})
+
+// Fast-path filter — column-store hit, not metadata fallback:
+const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
+
+// Set membership:
+const internal = await brain.find({
+ type: NounType.Person,
+ subtype: ['employee', 'contractor']
+})
+```
+
+Flat string, no hierarchy — your vocabulary, your choice. Brainy stores and counts, never validates.
+
+### New — O(1) subtype counts via the persisted rollup
+
+A new `_system/subtype-statistics.json` rollup is maintained incrementally as entities are added,
+updated, and deleted — mirroring the existing `nounCountsByType` machinery with the same self-heal
+behavior on poison detection. Counts are O(1) at billion scale:
+
+```typescript
+brain.counts.bySubtype(NounType.Person)
+// → { employee: 12, customer: 847, vendor: 34 }
+
+brain.counts.bySubtype(NounType.Person, 'employee') // O(1) point count
+// → 12
+
+brain.counts.topSubtypes(NounType.Person, 3)
+// → [['customer', 847], ['employee', 12], ['vendor', 34]]
+
+brain.subtypesOf(NounType.Person)
+// → ['customer', 'employee', 'vendor']
+```
+
+### New — `brain.trackField()` for other metadata facets
+
+For facets that aren't the *primary* sub-classification (`status`, `source`, `role`, `paradigm`),
+`trackField` registers a field for cardinality + per-NounType breakdown stats without promoting
+each one to a top-level slot. Piggybacks on the existing aggregation engine, so backfill-on-define
+applies — registering on a populated brain scans existing entities on the first query:
+
+```typescript
+brain.trackField('status', { perType: true })
+
+await brain.counts.byField('status')
+// → { todo: 12, doing: 3, done: 47 }
+
+await brain.counts.byField('status', { type: NounType.Task })
+// → { todo: 8, doing: 2, done: 30 }
+
+// Opt-in vocabulary validation:
+brain.trackField('priority', { values: ['low', 'medium', 'high'] })
+// brain.add({ ..., metadata: { priority: 'urgent' } }) → throws
+```
+
+### New — generic `brain.migrateField()` for one-shot rewrites
+
+Streams every entity and copies a value from one field path to another. Supports top-level
+standard fields, `metadata.*`, and `data.*` paths; idempotent; with optional `readBoth: true`
+deprecation window that preserves the source field alongside the new one:
+
+```typescript
+// Phase 1: dual-populate (legacy readers still work)
+await brain.migrateField({
+ from: 'metadata.kind',
+ to: 'subtype',
+ readBoth: true
+})
+
+// ...readers migrate to subtype at their own pace...
+
+// Phase 2: clear the source field
+await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
+// → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] }
+```
+
+Supports `batchSize` and `onProgress` for large brains.
+
+### Aggregation composition
+
+`subtype` is a standard-field group-by dimension out of the box — no `where:` wrapper, no
+metadata-fallback overhead:
+
+```typescript
+await brain.find({
+ aggregate: {
+ groupBy: ['type', 'subtype'],
+ metrics: { count: { op: 'COUNT' } }
+ }
+})
+// [
+// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 },
+// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 },
+// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 },
+// ...
+// ]
+```
+
+### Cortex compatibility
+
+Subtype works under Cortex out of the box via auto-field-indexing. Cortex will land its own
+native-side query-planner recognition + `subtypeCountsByType` native rollup in the next Cortex
+release for full parity with `type`'s fast path. Not a Brainy blocker — subtype reads/writes
+function correctly with current Cortex.
+
+### Docs
+
+Full guide: `docs/guides/subtypes-and-facets.md`. Subtype is documented as a core primitive
+throughout `README.md`, `docs/DATA_MODEL.md`, `docs/architecture/finite-type-system.md`,
+`docs/api/README.md`, and `docs/QUERY_OPERATORS.md`.
+
+---
+
+## v7.24.0 — 2026-05-26
+
+**Affected products:** aggregation/reporting users + anyone calling `extractEntities` on
+multi-entity text. Additive; drop-in from 7.23.x.
+
+### New — array-unnest `groupBy` (tag frequency / faceted counts)
+
+A `groupBy` dimension can now be `{ field, unnest: true }`: the field holds an array and the
+entity contributes once per **distinct** element. Enables tag-frequency / label-count style
+aggregates:
+
+```typescript
+brain.defineAggregate({
+ name: 'tag_frequency',
+ source: { type: NounType.Document },
+ groupBy: [{ field: 'tags', unnest: true }],
+ metrics: { count: { op: 'count' } }
+})
+// queryAggregate('tag_frequency', { orderBy: 'count', order: 'desc' })
+```
+
+Duplicate elements on one entity count once; an entity with an empty/missing array joins no group.
+
+### Perf — entity extraction batch-embeds candidates
+
+`extractEntities` / `extractConcepts` now embed all unique candidate spans in a single
+`embedBatch` call instead of one `embed()` per candidate (N sequential model calls before). No
+behavior change — and with vectors reliably available, the embedding signal more consistently
+reinforces correct types (e.g. clearer Organization confidence). Falls back to per-candidate
+embedding if a batch call fails.
+
+---
+
+## v7.23.0 — 2026-05-26
+
+**Affected products:** anyone using aggregation (stats/dashboards), graph traversal, or entity
+extraction. Adds two report APIs and fixes three correctness gaps surfaced by BR-ADV-FEATURES-BUN.
+Drop-in upgrade from 7.22.x.
+
+### New — `brain.queryAggregate(name, params)`
+
+A first-class report API returning the clean `AggregateResult[]` shape
+(`{ groupKey, metrics, count }[]`) directly, instead of the search-`Result` wrapper that
+`find({ aggregate })` returns. Accepts `where` / `having` / `orderBy` / `order` / `limit` / `offset`.
+
+### New — HAVING (filter groups by metric value)
+
+`find({ aggregate, having: { revenue: { greaterThan: 1000 } } })` (and the same on
+`queryAggregate`) filters groups by their computed metrics — the analytics equivalent of SQL
+`HAVING`, complementing `where` (which filters group keys). Evaluated per group: **O(groups),
+independent of entity count.**
+
+### Fix — aggregates now backfill when defined over existing data (R1)
+
+Defining an aggregate on a store that already holds matching entities returned `[]` because
+write-time hooks only saw *future* writes. It now backfills from existing entities on first query
+(one-time scan via `getNouns`, then incremental) — storage-agnostic, so it works under durable
+backends (Cortex) where a brain reopens pre-populated. Also: `groupBy:['noun']` now resolves to
+the entity type instead of a single null group. `find({ aggregate })` rows now expose
+`groupKey`/`metrics`/`count` at the top level (previously only under `.metadata`).
+
+### Fix — multi-hop `find({ connected: { depth, via } })` (traversal)
+
+`depth` and `via` are now honored at **every hop** (previously only the immediate neighbour was
+returned, and verb filtering applied to hop 1 only). The BFS is bounded by `limit`.
+
+### Fix — entity extraction type accuracy (R3)
+
+`extractEntities` no longer lets a type indicator in one candidate's surrounding text bleed onto
+neighbours (e.g. "Corp" in "Sarah Chen founded Acme Corp" no longer types "Sarah Chen" as
+Organization). Each candidate is typed by its own span; context may only reinforce the same type.
+
+---
+
+## v7.22.1 — 2026-05-26
+
+**Affected products:** Anyone using `extractEntities()` / `extractConcepts()`, native
+aggregation (`find({ aggregate })`), or multi-hop graph traversal
+(`find({ connected: { depth } })`). Three advanced-API correctness fixes — all reproduce on
+Node, so they were **not** Bun-specific despite the original report (BR-ADV-FEATURES-BUN).
+
+### Entity/concept extraction no longer returns `[]`
+
+`extractEntities()` / `extractConcepts()` returned empty for entity-rich text. The SmartExtractor
+ensemble scored agreeing signals with a weighted **sum** against an absolute 0.60 gate, so a
+confident low-weight signal (e.g. a name pattern at 0.82) lost selection to a mediocre
+high-weight signal that then failed the gate — dropping the whole result. It now selects and
+gates on a normalized weighted **average**. Also: the `confidence` option now actually controls
+the threshold (was a dead hardcoded 0.60), and the embedding-signal timeout was raised
+100ms → 2000ms so the neural signal isn't silently dropped on slower runtimes.
+
+*Known limitation:* type accuracy on dense multi-entity sentences is still imperfect — a strong
+indicator in one entity's surrounding text can bleed onto neighbours. Tracked separately.
+
+### `find({ aggregate })` exposes `groupKey` / `metrics` / `count`
+
+Aggregation always computed correct values, but rows nested them under `.metadata`, so callers
+expecting the documented `AggregateResult` shape saw empty-looking rows. Those three fields are
+now also present at the top level of each result row.
+
+### Multi-hop `find({ connected: { depth } })` honours depth
+
+Previously returned only the immediate (1-hop) neighbour at any depth because the graph-search
+path ignored `depth` / `via`. It now performs the full depth-aware traversal.
+
+---
+
+## v7.22.0 — 2026-05-15
+
+**Affected products:** All. Fixes a silent-data-loss class affecting `find()` and
+`brain.stats()`, plus Cortex-compatibility regression from 7.21.0. Recommended
+upgrade for everyone on 7.20.x or 7.21.x.
+
+### Two correctness fixes
+
+#### 1. `find({ where })` no longer silently returns `[]` (BR-FIND-WHERE-ZERO)
+
+The 7.20.0 column-store refactor deleted the legacy sparse-index write path
+but left `getStats()` and `getIds()` reading from sparse indices. New
+workspaces had no sparse-index files, so:
+- `brain.stats().entityCount` was `0` regardless of actual entity count
+- `find({ where: { ... } })` returned `[]` regardless of actual matches
+- `rebuildIndexesIfNeeded()` saw `0` entries → fired the `[Brainy] CRITICAL ...
+ Second rebuild result: 0 entries` log → application saw no indexed data
+
+7.22.0 makes the ColumnStore the single source of truth post-7.20.0:
+- `MetadataIndex.getStats()` reads from `ColumnStore.getIndexedFields()` and
+ `idMapper.size`. Entity count is now accurate across writer → close →
+ reader-open cycles.
+- `MetadataIndex.getIdsFromChunks()` throws `BrainyError(FIELD_NOT_INDEXED)`
+ when neither column store nor legacy sparse index has the field.
+- `MetadataIndex.getIdsForFilter()` catches per-clause, logs
+ `[brainy] find() where-clause referenced unindexed field(s): ...`, returns
+ `[]`. Production `find()` is now consistent with `brain.explain()`'s
+ `path: 'none'` diagnostic.
+
+Separately, `BaseStorage.getNounType()` was hardcoded to return `'thing'`
+since a type-cache removal in commit `42ae5be`. Every noun save attributed
+the entity to `'thing'` regardless of its actual type, poisoning
+`_system/type-statistics.json` and `brain.stats().entitiesByType`. 7.22.0
+restores type-correct attribution:
+- New `nounTypeByIdCache: Map` on `BaseStorage`, populated
+ in `saveNounMetadata_internal` and consumed in `saveNoun_internal`.
+- New `flushCounts()` override that persists `nounCountsByType` /
+ `verbCountsByType` alongside the per-entity counter. Readers opening the
+ same directory after a writer flush now see correct per-type counts.
+
+**Self-heal at init.** If `loadTypeStatistics()` detects the poisoned-state
+signature (all nouns attributed to `'thing'` with multiple non-`thing`
+entities on disk), it auto-runs `rebuildTypeCounts()` once and rewrites the
+file. A `[BaseStorage] Detected poisoned type-statistics.json signature ...`
+warning is logged. Operators don't need to take any action — the first 7.22
+open of a directory poisoned by 7.20.0/7.21.0 fixes it.
+
+**`brainy inspect repair`** can also be used to manually trigger
+`rebuildTypeCounts()`. It's now public on `BaseStorage`.
+
+#### 2. Cortex 2.2.x no longer crashes 7.21.0 boot (BR-DEFENSIVE-INTERFACE)
+
+7.21.0 added new storage-adapter methods (`supportsMultiProcessLocking`,
+`acquireWriterLock`, etc.) and called them unconditionally. Cortex 2.2.0's
+mmap storage adapter bundles a pre-7.21 `BaseStorage` and doesn't define
+them, so a consumer's 7.21.0 upgrade attempt threw
+`TypeError: this.storage.supportsMultiProcessLocking is not a function`
+at boot.
+
+7.22.0 introduces `hasStorageMethod(name)` on Brainy and guards every new-
+method call site. Older adapters degrade with a one-line warning at init:
+```
+[brainy] Storage adapter `CortexMmapStorage` predates the 7.21
+multi-process methods. Writer locking and the flush-request RPC are
+disabled for this directory. Upgrade the plugin (e.g.
+`@soulcraft/cortex` ≥2.3.0) for full enforcement.
+```
+
+### Dead-code cleanup
+
+Removed `MetadataIndexManager.dirtyChunks`, `dirtySparseIndices`, and the
+`flushDirtyMetadata()` no-op machinery. These accumulators stopped being
+populated when the sparse-index write path was deleted in 7.20.0; the
+remaining 6 callers wasted async hops on an empty Map iteration.
+
+### Upgrade notes
+
+- **Behaviour change:** `find({ where: { unindexedField: ... } })` previously
+ returned `[]` silently. It now logs a one-line warning and still returns
+ `[]`. The diagnostic message names the field — use
+ `brain.explain({ where: { ... } })` for full diagnosis. No code change
+ needed for callers that already handle empty results.
+- **Behaviour change (good):** `brain.stats()` and `brain.health()` now
+ report accurate per-type counts. If you previously relied on the buggy
+ `entitiesByType.thing` over-count, update consumers.
+- **No API breaks.** Pure additive on the public surface.
+
+### Cortex consumers
+
+Cortex 2.2.x continues to work against 7.22.0 thanks to the defensive
+guards, but for full multi-process protection on Cortex-backed stores and
+to make sure Cortex's native MetadataIndex picks up the find-where-zero
+fix, Cortex 2.3.0 is recommended. Tracked as `CX-MULTIPROC-METHOD` in the
+internal platform handoff.
+
+---
+
+## v7.21.0 — 2026-05-15
+
+**Affected products:** All consumers using filesystem storage (production deploys,
+local development).
+
+### Multi-process safety + first-class read-only inspector
+
+Filesystem storage now enforces **single-writer, many-reader**. Previously, opening
+a second writer process against a live data directory silently produced wrong query
+results — no error, no warning, just empty or stale data. This release replaces
+that footgun with a hard refusal at init time and a dedicated read-only inspection
+mode.
+
+#### New: `Brainy.openReadOnly()`
+```ts
+const reader = await Brainy.openReadOnly({
+ storage: { type: 'filesystem', rootDirectory: '/data/brain' }
+})
+const bookings = await reader.find({ where: { entityType: 'booking' } })
+```
+- Does NOT acquire the writer lock — coexists with a live writer.
+- Every mutation method throws `Cannot mutate a read-only Brainy instance`.
+- `flush()` and `close()` are safe (no-op + clean shutdown).
+
+#### New: writer lock at init
+- `new Brainy({ ... })` (default `mode: 'writer'`) acquires
+ `/locks/_writer.lock` containing `{ pid, hostname, startedAt, lastHeartbeat, version }`.
+- A second writer on the same directory **throws** with the PID, hostname,
+ heartbeat, and a pointer to `openReadOnly()`.
+- Stale-lock detection: same hostname + dead PID OR heartbeat > 60s ago → overwritten with a warning.
+- Heartbeat: writer rewrites `lastHeartbeat` every 10s (unref'd timer).
+- Override: pass `{ force: true }` to bypass when you've verified the existing lock is stale.
+
+#### New: `brain.requestFlush({ timeoutMs })`
+Cross-process RPC for inspectors to force a fresh snapshot:
+- In-process: just calls `flush()`.
+- Out-of-process: writes a request file, polls for ack. Times out gracefully.
+- Watcher runs in every writer instance (filesystem only).
+
+#### New: `brain.stats()`
+Operator-facing summary: `entityCount`, `entitiesByType`, `relationCount`,
+`relationsByType`, `fieldRegistry`, `indexHealth`, `storage.backend`, `writerLock`, `version`.
+Designed for `/api/health` endpoints and incident triage.
+
+#### New: `brain.explain(findParams)`
+Shows which index path serves each `where` clause: `column-store` | `sparse-chunked` | `none`.
+**This is the answer to "why is `find()` returning empty?"** — `path: "none"` means
+the field has no index entries and the query will silently return `[]`. Includes
+notes on likely causes (writer hasn't flushed; typo; field genuinely absent).
+
+#### New: `brain.health()`
+Invariant-check battery: HNSW vs metadata count parity, field registry sanity,
+`_seeded` entity sweep, writer heartbeat freshness. Returns `{ overall: 'pass' | 'warn' | 'fail', checks: [...] }`.
+
+#### New: `brainy inspect` CLI (13 subcommands)
+All read-only by default (internally uses `openReadOnly()`):
+```
+brainy inspect stats
+brainy inspect find --type Event --where '{"status":"paid"}' --limit 20
+brainy inspect get
+brainy inspect relations --direction both
+brainy inspect explain --where '{"entityType":"booking"}'
+brainy inspect health
+brainy inspect sample --type Event --n 20
+brainy inspect fields
+brainy inspect dump --type Event > backup.jsonl
+brainy inspect watch --type Event
+brainy inspect backup /backups/brain.tar
+brainy inspect repair # writer mode — stop live writer first
+brainy inspect diff
+```
+Default behaviour: ask the writer to flush via the RPC first (skip with `--no-fresh`).
+
+### Brainy + Cortex
+
+Cortex segments (`*.cidx`) are immutable mmap files; `MANIFEST.json` uses atomic
+rename. The single writer lock at `/locks/_writer.lock` covers both Brainy
+and Cortex. Read-only inspectors mmap Cortex segments with zero coordination.
+
+### What's NOT enforced yet
+
+- **Cloud storage backends** (S3, GCS, R2, Azure) — no multi-process locking.
+ A best-effort warning logs in writer mode against non-filesystem backends.
+- **The `find({ where })`-returns-0 root-cause bug** — tracked separately. The
+ new `brain.explain()` and `brain.health()` surface it loudly
+ (`path: 'none'`, `index-parity warn`) instead of letting it be silent.
+
+### Upgrade notes
+
+- **Default behaviour change:** opening a Brainy directory in writer mode while
+ another writer is live now THROWS where it previously silently produced wrong
+ results. If you have scripts that intentionally co-opened a writer directory,
+ switch them to `Brainy.openReadOnly()` or pass `{ force: true }`.
+- **Cloud backends unchanged** — no lock acquisition for S3/GCS/R2/Azure.
+- All existing APIs unchanged. Pure addition.
+
+### Docs
+
+- README "Single-Writer Model" section.
+- `docs/concepts/multi-process.md` — full model + Cortex compatibility.
+- `docs/guides/inspection.md` — operator recipes.
+- JSDoc on every new API.
+
+---
+
+## v7.19.10 — 2026-02-24
+
+**Affected products:** All Bun/ESM consumers
+
+### ESM crypto fix in SSTable
+
+Replaced `require('crypto')` with `import { createHash } from 'node:crypto'` in the
+SSTable implementation. Fixes a crash in Bun and strict ESM environments where
+CommonJS `require` is unavailable.
+
+No API changes — upgrade and redeploy.
+
+---
+
+## v7.19.2 — 2026-02-18
+
+**Affected products:** All
+
+### Metadata index cleanup on delete
+
+Fixed: metadata indexes were not cleaned up after `delete()` / `deleteMany()`. Stale
+index entries could cause phantom results in metadata-filtered queries after deletion.
+
+No API changes. If you were seeing ghost results in filtered queries, this fixes it.
+
+---
+
+## v7.18.0 — 2026-02-16
+
+**Affected products:** Analytics, reporting, and session-summary consumers
+
+### Aggregation engine
+
+New `brain.aggregate()` API — incremental SUM, COUNT, AVG, MIN, MAX with GROUP BY
+and time window support. Computes over entity collections without loading all records
+into memory.
+
+```typescript
+const result = await brain.aggregate({
+ collection: 'bookings',
+ metrics: [
+ { field: 'revenue', fn: 'SUM' },
+ { field: 'id', fn: 'COUNT' },
+ ],
+ groupBy: 'staffId',
+ timeWindow: { field: 'createdAt', from: startOfMonth, to: now },
+})
+```
+
+SDK exposure: `sdk.brainy.aggregate()` — available once SDK is updated to pass through.
+
+---
+
+## v7.17.0 — 2026-02-09
+
+**Affected products:** All (schema evolution, data migrations)
+
+### Migration system
+
+New `brain.migrate()` API with error handling, validation, and enterprise hardening.
+Run schema migrations reliably across Brainy data directories.
+
+```typescript
+await brain.migrate({
+ version: 3,
+ up: async (brain) => {
+ // transform entities, rename fields, etc.
+ },
+})
+```
+
+---
+
+## v7.16.0 — 2026-02-09
+
+**Affected products:** All
+
+### Data/metadata separation enforced + numeric range queries
+
+- Entity `data` and `metadata` fields are now strictly separated at the storage layer
+- Numeric range queries now supported in metadata filters: `{ age: { $gte: 18, $lt: 65 } }`
+- Fixes edge cases where mixed data/metadata storage caused inconsistent query results
+
+**Breaking for anyone storing numeric values in metadata and relying on range queries:**
+verify your filter syntax matches the new `$gte/$lte/$gt/$lt` operators.
+
+---
+
+## v7.15.5 — 2026-02-02
+
+**Affected products:** Anyone using `@soulcraft/cortex` plugin
+
+### Plugin opt-in clarified
+
+Cortex and other plugins are opt-in. Pass explicitly:
+```typescript
+new Brainy({ plugins: ['@soulcraft/cortex'] })
+```
+Without `plugins`, no external plugins are loaded regardless of what's installed.
+
+---
+
+## v7.15.2 — 2026-02-01
+
+**Affected products:** All (data safety)
+
+### Graph LSM flush on close
+
+Fixed: graph LSM-trees were not flushed on `brain.close()`, risking data loss across
+restarts. Graph edges written in the final seconds before shutdown are now guaranteed
+to be persisted.
+
+No API changes — upgrade immediately if running Brainy in a long-lived server process.
diff --git a/docs/ADR-001-generational-mvcc.md b/docs/ADR-001-generational-mvcc.md
new file mode 100644
index 00000000..b4c4d8dd
--- /dev/null
+++ b/docs/ADR-001-generational-mvcc.md
@@ -0,0 +1,295 @@
+# ADR-001: Generational MVCC storage and the immutable Db API
+
+**Status:** Accepted (ships in 8.0)
+**Date:** 2026-06-10
+
+## Context
+
+Before 8.0, Brainy carried two overlapping version-control subsystems: a
+copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a
+separate versioning subsystem (`versions.save/list/compare/restore/prune`),
+plus a read-only historical adapter for commit-based time travel. Together
+they were ~5,100 LOC of mechanism for one product need: *read a consistent
+past state while the store keeps moving, and snapshot/restore cheaply.*
+
+Neither subsystem gave a precise isolation guarantee. Reads raced in-place
+JSON overwrites, so a "snapshot" was only as immutable as the bytes it
+happened to share with the live store.
+
+8.0 replaces both with **one mechanism**: generational MVCC over immutable,
+generation-stamped records, exposed through a Datomic-style immutable
+database value (`Db`). The same model is implemented natively by versioned
+index providers (LSM snapshots), so semantics are identical on the pure-JS
+path and the native path.
+
+## Decision
+
+### The model
+
+- A **monotonic u64 generation counter** is the store's logical clock. It
+ advances once per committed `transact()` batch and once per
+ single-operation write (`add`/`update`/`remove`/`relate`/…), so
+ `brain.generation()` is always a meaningful watermark. It is persisted in
+ `_system/generation.json` and never reissued for anything durable.
+- `brain.now()` **pins** the current generation in O(1) and returns a `Db` —
+ an immutable view. Pins are refcounted; `db.release()` (with a
+ `FinalizationRegistry` backstop for leaked values) ends the pin.
+- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative
+ batch atomically as **exactly one generation**, with whole-store
+ compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and
+ reified transaction metadata appended to `_system/tx-log.jsonl`.
+- `brain.asOf(generation | Date | snapshotPath)` opens past state;
+ `db.with(ops)` layers a speculative in-memory overlay (never touching
+ disk, the counter, or index providers); `db.persist(path)` cuts an
+ instant snapshot; `brain.restore(path, { confirm: true })` replaces state
+ from one; `Brainy.load(path)` opens a snapshot read-only with the full
+ query surface.
+
+### Persisted layout
+
+All paths are storage-root-relative:
+
+```
+_system/generation.json { generation, updatedAt } atomic tmp+rename
+_system/manifest.json { version, generation, atomic tmp+rename
+ committedAt, horizon } (the commit point)
+_system/tx-log.jsonl one line per committed append-only
+ transact: { generation,
+ timestamp, meta? }
+_generations//tx.json the generation-N delta: immutable
+ touched noun/verb ids + meta
+_generations//prev/.json before-image of as of immutable
+ commit N (raw stored bytes;
+ null parts = file was absent)
+```
+
+**Why per-generation deltas instead of a global `id → latest generation`
+map in the manifest:** a global map makes every commit O(all ids) — the
+whole map must be rewritten to swap it atomically. The delta layout makes a
+commit O(ids touched) and keeps the manifest a fixed-size watermark, while
+point-in-time resolution stays correct (see "Read resolution" below). The
+trade is that resolution at a pinned generation scans the deltas of later
+commits — bounded by the number of commits since the pin, which is exactly
+the window compaction keeps short.
+
+Before-images are deliberately the *only* per-id records. They serve both
+roles the layer needs — the crash-recovery undo log and the point-in-time
+read source. After-images would duplicate state that is already readable
+(the canonical entity files hold the latest bytes; earlier states resolve
+from later before-images) and would double record I/O per commit.
+
+### Commit protocol (durability)
+
+`transact()` commits under a store-wide mutex:
+
+1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError`
+ before anything is staged.
+2. **Reserve** generation `N` (counter increment).
+3. **Stage the undo log:** write the before-image of every touched id plus
+ `tx.json` under `_generations/N/`, then **fsync** the files and their
+ directories. From this point, any crash is recoverable to the exact
+ pre-transaction bytes.
+4. **Execute** the planned batch through the TransactionManager (which has
+ its own operation-level rollback for in-flight failures).
+5. **Commit point:** persist the counter, then write `_system/manifest.json`
+ via atomic tmp+rename and fsync it. The rename *is* the commit: a
+ generation directory is committed if and only if `N ≤
+ manifest.generation`.
+6. Append the tx-log line (advisory metadata — a crash between 5 and 6
+ keeps the transaction).
+
+**Crash recovery (on open):** any `_generations/` directory with
+`N > manifest.generation` is an uncommitted transaction. Its before-images
+are restored to the canonical paths (idempotently — recovery itself can
+crash and rerun) and the directory is removed. Because recovery runs before
+any index is built, and a recovery that rolled something back forces a full
+index rebuild, derived indexes never observe rolled-back state. Reader-mode
+instances skip recovery (readers never write; the next writer repairs).
+
+A failed (non-crash) transaction takes the same staging directory down the
+abort path: the TransactionManager rolls back applied operations, the
+staging directory is removed, and the generation reservation is returned —
+a failed batch leaves the generation counter unchanged.
+
+### Read resolution at a pinned generation
+
+The state of id X at pinned generation G is:
+
+- the before-image stored by the **first committed generation after G that
+ touched X**, or
+- the live canonical bytes, when nothing after G touched X.
+
+While nothing has committed past G, *every* read on the `Db` delegates to
+the live fast paths untouched — `now()` adds no read overhead until history
+actually moves.
+
+**Two read paths, one result set.** `get()`, metadata-level `find()`, and
+filter-based `related()` resolve directly through the record layer at any
+reachable pinned generation — no extra cost beyond scanning the deltas of
+later commits. Index-accelerated dimensions (semantic/vector search, graph
+traversal, cursors, aggregation) are served by **at-generation index
+materialization**: the first such query on a historical `Db` copies the
+exact at-G record set (live bytes for ids untouched since the pin,
+before-images for the rest; a final reconciliation pass runs under the
+commit mutex so transactions racing the copy cannot skew it) into an
+ephemeral in-memory store and opens a read-only engine over it — the same
+vector/metadata/graph index classes the live brain uses, sharing the host's
+embedder and aggregate definitions. The handle is cached on the `Db` and
+freed by `release()`.
+
+**Cost, stated plainly:** materialization is O(n at G) time and memory,
+once per `Db`. That is the open-core price of historical index queries. A
+native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
+retained LSM segments) serves the same reads with no rebuild at all — the
+materializer is the correctness baseline, the provider is the accelerator.
+
+**The one remaining boundary.** Speculative `with()` overlays throw
+`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
+overlay entities carry no embeddings (`with()` never invokes the embedder),
+so a "full" index query over an overlay would silently exclude the
+overlay's own entities. Commit with `transact()` to get the full surface.
+
+**History granularity (Model-B).** EVERY write is its own immutable generation
+— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
+Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/
+`diff()`/`history()` exactly like transacts; a pin always freezes against later
+writes. `transact()` groups several operations into ONE atomic generation.
+
+Single-op history durability is **async group-commit**: the live write hits
+canonical storage immediately (acknowledged), while its before-image is buffered
+and persisted to disk in one batched fsync on a size/timer trigger (or forced by
+`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in
+point-in-time resolution exactly like on-disk generations, so the synchronous
+`now()` freezes with no forced flush. A hard crash before the flush loses only
+the buffered *history* of the last window — never live data — and a crash
+*mid-flush* is recovered by **drop-without-restore** (the partial generation's
+before-images are discarded, never replayed, because the live write was already
+acknowledged; restoring them would silently revert it).
+
+### Pinning, retention, compaction
+
+- Each live `Db` holds one refcounted pin on its generation (plus a
+ `pin(generation)` on every registered `VersionedIndexProvider`, whose
+ explicit pin lifetime overrides any time-based snapshot retention the
+ provider has).
+- The constructor **`retention`** knob governs auto-compaction (on `flush()`/
+ `close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config;
+ driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) ·
+ `'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit
+ CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
+ manually on the same caps — the oldest unpinned record-sets are reclaimed
+ while ANY supplied cap is exceeded.
+- A record-set `N` is reclaimed only when `N` is at or below **every** live
+ pin — deleting `N` can only break readers pinned *below* `N`, because
+ resolution reads before-images from generations strictly greater than the
+ pin. Live pins are ALWAYS exempt, in every retention mode.
+- The manifest records the **horizon** (highest reclaimed generation).
+ Generations below the horizon are unreachable; `asOf()` on them throws
+ `GenerationCompactedError`. The horizon itself stays reachable, resolved
+ from the record-sets above it. To keep a generation readable forever,
+ `persist()` it first — snapshots are self-contained.
+
+### Snapshots and restore
+
+`db.persist(path)` flushes indexes, then cuts the snapshot under the
+store's commit mutex (no commit, compaction, or counter write can
+interleave). On filesystem storage it is a **hard-link farm**: every data
+file is immutable-by-rename, so linking is safe — later rewrites swap
+inodes and the snapshot keeps the old bytes. The two exceptions are handled
+explicitly: the append-in-place tx-log is byte-copied, and process-local
+lock state is excluded. Cross-device targets (and filesystems that refuse
+links) fall back to per-file byte copies. In-memory stores serialize to the
+same directory layout, so persisting a memory brain produces a real,
+durable, loadable store.
+
+`persist()` requires the view to still be the store's latest generation
+(a snapshot captures current bytes); a view that history has moved past
+throws rather than persisting the wrong state.
+
+`restore(path, { confirm: true })` replaces the store's contents from a
+snapshot via byte copy (never links — the snapshot stays independent),
+reloads all adapter-internal derived state, rebuilds all indexes, and
+floors the generation counter at its pre-restore value so observed
+generation numbers are never reissued. Live pins do not survive a restore;
+a warning is logged when any exist.
+
+### Versioned index providers
+
+Native index providers may implement the optional 4-method
+`VersionedIndexProvider` capability (`generation()`,
+`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the
+boundary). The locked consistency model: providers are **post-commit
+appliers**. The storage-record commit is the source of truth; provider
+index state is derived. On open, a provider behind the committed watermark
+replays the gap from storage (or requests a rebuild) — there are no
+provider rollback hooks, because uncommitted transactions are repaired at
+the storage layer before any index opens. Speculative `with()` overlays
+never reach providers.
+
+## Guarantees (and their proofs)
+
+Each stated guarantee has a test that proves it, not merely exercises it
+(`tests/integration/db-mvcc.test.ts`, plus
+`tests/unit/db/generationStore.test.ts` for the record layer in isolation):
+
+| Guarantee | Proof |
+|---|---|
+| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) |
+| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) |
+| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) |
+| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c |
+| Compaction never breaks a pinned read; release enables reclaim | proof 5 |
+| `with()` overlays touch nothing durable | proof 6 |
+| Generation monotonicity across close/reopen | proof 7 |
+| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) |
+| Balanced provider pin/release lockstep | proof 9 |
+
+One deliberate softness: single-operation generation bumps persist the
+counter coalesced (per write burst), not per write. Durable artifacts —
+records, manifests, snapshots — always persist the counter synchronously at
+their own commit points, so a crash inside the coalescing window can lose
+only counter values that nothing durable ever referenced.
+
+## Failure modes
+
+| Failure | Outcome |
+|---|---|
+| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. |
+| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. |
+| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. |
+| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
+| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
+| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
+| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
+| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
+| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
+
+## Lineage
+
+The design is an assembly of well-understood prior art, chosen for being
+boring where it counts:
+
+- **Datomic** — the database-as-a-value: an immutable `Db` you query, with
+ `with()` for speculation and reified transaction metadata instead of
+ commit messages.
+- **LMDB** — reader pins: readers never block writers; a reader's view
+ stays valid because nothing overwrites the pages (here: records) it
+ references; reclamation waits for the last reader.
+- **LSM trees / Cassandra** — immutable segments make snapshots hard links
+ and make compaction a retention policy instead of a locking problem.
+
+## Consequences
+
+- One mechanism replaces the COW and versioning subsystems (their removal
+ is the companion change to this ADR).
+- In-place branch switching (`checkout`) is gone by design; the replacement
+ is opening a persisted snapshot as a separate instance — a name→path
+ mapping where a product needs named branches.
+- Every commit pays O(ids touched) extra writes (before-images + delta +
+ manifest). Single-operation writes pay only an in-memory counter bump
+ with coalesced persistence.
+- The full query surface works at every reachable pinned generation.
+ Record-path reads (`get`, metadata `find`, filter `related`) are
+ effectively free; index-accelerated historical queries pay a one-time
+ O(n at G) materialization per `Db` on the open-core path (freed on
+ `release()`), and run rebuild-free on a native `VersionedIndexProvider`.
diff --git a/docs/API_DECISION_TREE.md b/docs/API_DECISION_TREE.md
deleted file mode 100644
index 775c0d22..00000000
--- a/docs/API_DECISION_TREE.md
+++ /dev/null
@@ -1,481 +0,0 @@
-# 🧠 Brainy API Decision Tree
-
-*Choose the right API for your use case with confidence*
-
-This guide helps you navigate Brainy's comprehensive API surface by asking the right questions to find the perfect method for your specific needs.
-
-## 🎯 Quick Start: What do you want to do?
-
-### 📝 **Adding Data**
-- **Single entity** → [`brainy.add()`](#adding-single-entities)
-- **Multiple entities** → [`brainy.addMany()`](#adding-multiple-entities)
-- **Streaming/real-time data** → [Streaming Pipeline](#streaming-data)
-
-### 🔍 **Finding Data**
-- **Natural language search** → [`brainy.find("search query")`](#natural-language-search)
-- **Structured/filtered search** → [`brainy.find({ query, where, type })`](#structured-search)
-- **Similar entities** → [`brainy.similar()`](#similarity-search)
-- **Get by ID** → [`brainy.get()`](#retrieval-by-id)
-
-### 🔗 **Relationships**
-- **Create relationships** → [`brainy.relate()`](#creating-relationships)
-- **Query relationships** → [`brainy.getRelations()`](#querying-relationships)
-- **Graph traversal** → [Graph Navigation](#graph-operations)
-
-### 📊 **Advanced Features**
-- **File management** → [VFS (Virtual File System)](#file-operations)
-- **AI-powered analysis** → [Neural API](#neural-analysis)
-- **Clustering/insights** → [Intelligence Systems](#intelligence-systems)
-
----
-
-## 🔀 Decision Tree Flow
-
-```mermaid
-graph TD
- A[What are you trying to do?] --> B[Store Data]
- A --> C[Find Data]
- A --> D[Manage Relationships]
- A --> E[Work with Files]
- A --> F[AI Analysis]
-
- B --> B1[Single Item]
- B --> B2[Multiple Items]
- B --> B3[Real-time Stream]
-
- C --> C1[I know the ID]
- C --> C2[Natural language query]
- C --> C3[Complex filters]
- C --> C4[Find similar items]
-
- D --> D1[Create relationship]
- D --> D2[Query relationships]
- D --> D3[Graph traversal]
-
- E --> E1[File operations]
- E --> E2[Knowledge-enhanced files]
-
- F --> F1[Clustering]
- F --> F2[Similarity analysis]
- F --> F3[Insights generation]
-```
-
----
-
-## 📝 Adding Data
-
-### Adding Single Entities
-
-**Use `brainy.add()` when:**
-- Adding one entity at a time
-- You need the ID immediately for further operations
-- Working with user input or real-time data
-
-```typescript
-// ✅ Perfect for single entities
-const id = await brainy.add({
- data: "New research paper on quantum computing",
- type: NounType.Document,
- metadata: { category: "research", priority: "high" }
-})
-```
-
-**Decision factors:**
-- **Single item?** → `add()`
-- **Need immediate ID?** → `add()`
-- **Interactive application?** → `add()`
-
-### Adding Multiple Entities
-
-**Use `brainy.addMany()` when:**
-- Bulk importing data
-- Processing batches (>10 items)
-- Performance is critical
-
-```typescript
-// ✅ Perfect for bulk operations
-const result = await brainy.addMany({
- items: documents.map(doc => ({
- data: doc.content,
- type: NounType.Document,
- metadata: doc.metadata
- })),
- chunkSize: 100,
- parallel: true
-})
-```
-
-**Decision factors:**
-- **Multiple items (>10)?** → `addMany()`
-- **Batch processing?** → `addMany()`
-- **Can tolerate some failures?** → `addMany()` with `continueOnError: true`
-
-### Streaming Data
-
-**Use Streaming Pipeline when:**
-- Real-time data ingestion
-- Processing large datasets that don't fit in memory
-- Need transformation during ingestion
-
-```typescript
-// ✅ Perfect for streaming
-const pipeline = brainy.streaming.pipeline()
- .transform(data => ({ ...data, processed: true }))
- .batch(50)
- .into(brainy)
-```
-
----
-
-## 🔍 Finding Data
-
-### Natural Language Search
-
-**Use `brainy.find("query string")` when:**
-- User is typing search queries
-- You want semantic understanding
-- Building search interfaces
-
-```typescript
-// ✅ Perfect for user searches
-const results = await brainy.find("documents about machine learning")
-```
-
-**Decision factors:**
-- **User-generated query?** → Natural language `find()`
-- **Semantic understanding needed?** → Natural language `find()`
-- **Search interface?** → Natural language `find()`
-
-### Structured Search
-
-**Use `brainy.find({ query, where, type })` when:**
-- Complex filtering requirements
-- Combining text search with metadata filters
-- Performance-critical searches
-
-```typescript
-// ✅ Perfect for complex queries
-const results = await brainy.find({
- query: "neural networks",
- type: NounType.Document,
- where: {
- status: "published",
- year: { $gte: 2020 }
- },
- limit: 20
-})
-```
-
-**Decision factors:**
-- **Need metadata filtering?** → Structured `find()`
-- **Performance critical?** → Structured `find()`
-- **Complex criteria?** → Structured `find()`
-
-### Similarity Search
-
-**Use `brainy.similar()` when:**
-- Finding "more like this" content
-- Recommendation systems
-- Duplicate detection
-
-```typescript
-// ✅ Perfect for recommendations
-const similar = await brainy.similar({
- to: "document-id-123",
- limit: 10,
- type: NounType.Document
-})
-```
-
-**Decision factors:**
-- **"More like this" feature?** → `similar()`
-- **Recommendations?** → `similar()`
-- **Duplicate detection?** → `similar()`
-
-### Retrieval by ID
-
-**Use `brainy.get()` when:**
-- You know the exact ID
-- Loading specific entities
-- Following relationships
-
-```typescript
-// ✅ Perfect for direct access
-const entity = await brainy.get("known-id-123")
-```
-
-**Decision factors:**
-- **Known ID?** → `get()`
-- **Direct access needed?** → `get()`
-- **Following relationships?** → `get()`
-
----
-
-## 🔗 Relationships
-
-### Creating Relationships
-
-**Use `brainy.relate()` when:**
-- Connecting two entities
-- Building knowledge graphs
-- Modeling real-world relationships
-
-```typescript
-// ✅ Perfect for connections
-await brainy.relate({
- from: "user-123",
- to: "project-456",
- type: VerbType.WorksOn,
- metadata: { role: "lead", since: "2024-01-01" }
-})
-```
-
-**Decision factors:**
-- **Connecting entities?** → `relate()`
-- **Need relationship metadata?** → `relate()`
-- **Building graphs?** → `relate()`
-
-### Querying Relationships
-
-**Use `brainy.getRelations()` when:**
-- Finding all connections for an entity
-- Exploring relationship patterns
-- Building relationship views
-
-```typescript
-// ✅ Perfect for relationship queries
-const relations = await brainy.getRelations({
- from: "user-123",
- type: VerbType.WorksOn
-})
-```
-
----
-
-## 📁 File Operations
-
-### Basic File Operations
-
-**Use VFS when:**
-- Managing files and directories
-- Need hierarchical structure
-- Building file explorers
-
-```typescript
-// ✅ Perfect for file management
-const vfs = brainy.vfs({ storage: 'filesystem' })
-await vfs.writeFile('/docs/readme.md', 'content')
-const files = await vfs.getDirectChildren('/docs')
-```
-
-**Decision factors:**
-- **File management?** → VFS
-- **Directory structure?** → VFS
-- **File explorer interface?** → VFS
-
-### Intelligent File Management
-
-**Use VFS (Semantic VFS) when:**
-- Need semantic file search
-- Want AI-powered concept extraction
-- Building smart file systems
-- Require multi-dimensional file access
-
-```typescript
-// ✅ Perfect for intelligent file systems
-const knowledgeVFS = await vfs.withKnowledge(brainy)
-const insights = await knowledgeVFS.getFileInsights('/project')
-```
-
----
-
-## 🧠 AI Analysis
-
-### Clustering
-
-**Use Neural API clustering when:**
-- Discovering data patterns
-- Organizing large datasets
-- Creating automatic categories
-
-```typescript
-// ✅ Perfect for pattern discovery
-const neural = brainy.neural()
-const clusters = await neural.cluster({
- entities: entityIds,
- k: 5,
- method: 'hierarchical'
-})
-```
-
-### Intelligence Systems
-
-**Use Triple Intelligence when:**
-- Complex multi-criteria searches
-- Advanced relationship queries
-- Performance-critical operations
-
-```typescript
-// ✅ Perfect for complex queries
-const intelligence = brainy.getTripleIntelligence()
-const results = await intelligence.query({
- vector: queryVector,
- metadata: { category: 'research' },
- graph: { connected: 'user-123' }
-})
-```
-
----
-
-## 🚀 Performance Optimization Guide
-
-### When Performance Matters
-
-| Scenario | Best Choice | Why |
-|----------|-------------|-----|
-| **Bulk Import** | `addMany()` | Batched operations, parallel processing |
-| **Metadata-only Search** | `find({ where: {...} })` | Skips vector computation |
-| **Known ID Access** | `get()` | Direct index lookup |
-| **Large Result Sets** | Pagination with `offset`/`limit` | Memory efficient |
-| **Real-time Streams** | Streaming Pipeline | Memory efficient, scalable |
-
-### Memory Usage Optimization
-
-```typescript
-// ❌ Memory intensive
-const allResults = await brainy.find({ limit: 10000 })
-
-// ✅ Memory efficient
-for (let offset = 0; offset < total; offset += 100) {
- const batch = await brainy.find({
- query: "...",
- limit: 100,
- offset
- })
- await processBatch(batch)
-}
-```
-
----
-
-## 🎯 Common Use Case Patterns
-
-### Building a Search Interface
-
-```typescript
-// User types query → Natural language search
-const searchResults = await brainy.find(userQuery)
-
-// User applies filters → Structured search
-const filteredResults = await brainy.find({
- query: userQuery,
- where: selectedFilters,
- type: selectedTypes
-})
-
-// User clicks "more like this" → Similarity search
-const similar = await brainy.similar({ to: selectedId })
-```
-
-### Building a Recommendation System
-
-```typescript
-// 1. Get user's interaction history
-const user = await brainy.get(userId)
-
-// 2. Find similar users
-const similarUsers = await brainy.similar({ to: userId, type: NounType.Person })
-
-// 3. Get their liked content
-const recommendations = []
-for (const similarUser of similarUsers) {
- const relations = await brainy.getRelations({
- from: similarUser.id,
- type: VerbType.Likes
- })
- recommendations.push(...relations)
-}
-```
-
-### Building a Knowledge Graph
-
-```typescript
-// 1. Add entities
-const entities = await Promise.all([
- brainy.add({ data: "Person: Alice", type: NounType.Person }),
- brainy.add({ data: "Company: TechCorp", type: NounType.Organization }),
- brainy.add({ data: "Project: AI Assistant", type: NounType.Thing })
-])
-
-// 2. Create relationships
-await brainy.relate({
- from: entities[0], // Alice
- to: entities[1], // TechCorp
- type: VerbType.WorksFor
-})
-
-await brainy.relate({
- from: entities[0], // Alice
- to: entities[2], // AI Assistant
- type: VerbType.WorksOn
-})
-
-// 3. Query the graph
-const aliceConnections = await brainy.getRelations({ from: entities[0] })
-```
-
----
-
-## 🔧 Migration Guide
-
-### From v2.x to v3.x APIs
-
-| v2.x (Deprecated) | v3.x (Current) | When to Use |
-|-------------------|----------------|-------------|
-| `brain.store()` | `brainy.add()` | Adding entities |
-| `brain.search()` | `brainy.find()` | Searching content |
-| `brain.query()` | `brainy.find({ ... })` | Complex queries |
-| `brain.similar()` | `brainy.similar()` | ✅ Same API |
-| `brain.connect()` | `brainy.relate()` | Creating relationships |
-
-### Legacy Type Migration
-
-```typescript
-// ❌ v2.x way
-import { ISenseAugmentation } from '@soulcraft/brainy/types/augmentations'
-
-// ✅ v3.x way
-import { BrainyAugmentation } from '@soulcraft/brainy'
-```
-
----
-
-## 🎪 Decision Quick Reference
-
-**Need to add data?**
-- 1 item → `add()`
-- Many items → `addMany()`
-- Streaming → Pipeline
-
-**Need to find data?**
-- Know ID → `get()`
-- Natural search → `find("query")`
-- Complex filters → `find({ query, where })`
-- Similar items → `similar()`
-
-**Need relationships?**
-- Create → `relate()`
-- Query → `getRelations()`
-- Complex graph → Triple Intelligence
-
-**Need files?**
-- Basic → VFS (standard operations)
-- Smart → Semantic VFS (6 dimensional access + neural extraction)
-
-**Need AI analysis?**
-- Patterns → Neural clustering
-- Complex queries → Triple Intelligence
-
----
-
-*This guide covers 95% of use cases. For edge cases or custom requirements, check the [Core API Patterns](./CORE_API_PATTERNS.md) and [Neural API Patterns](./NEURAL_API_PATTERNS.md) guides.*
\ No newline at end of file
diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md
deleted file mode 100644
index e9b6711b..00000000
--- a/docs/API_REFERENCE.md
+++ /dev/null
@@ -1,1867 +0,0 @@
-# 🧠 Brainy 3.0 Complete API Reference
-
-> The neural database that thinks - Complete API documentation for all public methods
-
-## Table of Contents
-
-- [Quick Start](#quick-start)
-- [Core API](#core-api)
-- [Batch Operations](#batch-operations)
-- [Search & Discovery](#search--discovery)
-- [Security API](#security-api)
-- [Configuration API](#configuration-api)
-- [Data Management API](#data-management-api)
-- [Query API](#query-api)
-- [Neural API](#neural-api)
-- [NLP API](#nlp-api)
-- [Streaming Pipeline API](#streaming-pipeline-api)
-- [Type Definitions](#type-definitions)
-
----
-
-## Quick Start
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-// Initialize
-const brain = new Brainy({
- storage: { type: 'filesystem', path: './brainy-data' },
- model: { type: 'fast', precision: 'Q8' }
-})
-await brain.init()
-
-// Add entities
-const id = await brain.add({
- data: 'John Smith is a software engineer',
- type: NounType.Person,
- metadata: { role: 'engineer' }
-})
-
-// Search
-const results = await brain.find('engineers')
-
-// Clean up
-await brain.close()
-```
-
----
-
-## Core API
-
-### `new Brainy(config?: BrainyConfig)`
-Creates a new Brainy instance.
-
-**Parameters:**
-- `config` - Optional configuration object
-
-**Returns:** `Brainy` instance
-
-**Example:**
-```typescript
-const brain = new Brainy({
- storage: { type: 'filesystem', options: { path: './data' } },
- model: { type: 'accurate', precision: 'FP32' },
- cache: { maxSize: 5000, ttl: 600000 }
-})
-```
-
----
-
-### `async init(): Promise`
-Initializes the brain, loading models and preparing storage.
-
-**Must be called before any other operations.**
-
-**Example:**
-```typescript
-await brain.init()
-```
-
----
-
-### `async add(params: AddParams): Promise`
-Adds a new entity to the brain.
-
-**Parameters:**
-- `data` (required) - Content to embed and store
-- `type` (required) - NounType classification
-- `metadata` - Custom metadata object
-- `id` - Custom ID (auto-generated if not provided)
-- `vector` - Pre-computed embedding vector
-- `service` - Service name for multi-tenancy
-- `confidence` - Type classification confidence (0-1) ✨ *New in v4.3.0*
-- `weight` - Entity importance/salience (0-1) ✨ *New in v4.3.0*
-- `writeOnly` - Skip validation for high-speed ingestion
-
-**Returns:** Entity ID
-
-**Example:**
-```typescript
-const id = await brain.add({
- data: 'Important meeting notes from Q4 planning',
- type: NounType.Document,
- metadata: {
- date: '2024-01-15',
- author: 'John Smith',
- tags: ['planning', 'Q4']
- },
- confidence: 0.95, // High confidence in Document classification
- weight: 0.85 // High importance
-})
-```
-
----
-
-### `async get(id: string, options?: GetOptions): Promise`
-Retrieves an entity by ID.
-
-✨ **v5.11.1 Performance Optimization**:
-- **Default (metadata-only)**: 76-81% faster, 95% less bandwidth - perfect for VFS, existence checks, metadata access
-- **Opt-in vectors**: Use `{ includeVectors: true }` when computing similarity
-
-**Parameters:**
-- `id` - Entity ID
-- `options` (optional):
- - `includeVectors?: boolean` - Include 384-dim vectors (default: false for 76-81% speedup)
-
-**Returns:** Entity object or null if not found
-
-**Entity Properties:** ✨ *Updated in v4.3.0, v5.11.1*
-- `id` - Unique identifier
-- `type` - NounType classification
-- `data` - Original content
-- `metadata` - Custom metadata
-- `vector` - Embedding vector (empty array `[]` if includeVectors: false)
-- `confidence` - Type classification confidence (0-1)
-- `weight` - Entity importance/salience (0-1)
-- `createdAt` - Creation timestamp
-- `updatedAt` - Last update timestamp
-- `service` - Service name (multi-tenancy)
-
-**Example (Metadata-Only - Default, 76-81% faster):**
-```typescript
-// Perfect for VFS, metadata access, existence checks
-const entity = await brain.get('uuid-1234')
-if (entity) {
- console.log(entity.type) // NounType.Document
- console.log(entity.metadata) // { date: '2024-01-15', ... }
- console.log(entity.vector) // [] (empty - not loaded for performance)
-}
-```
-
-**Example (Full Entity with Vectors):**
-```typescript
-// Use when computing similarity on this specific entity
-const entity = await brain.get('uuid-1234', { includeVectors: true })
-if (entity) {
- console.log(entity.vector.length) // 384 (full embeddings loaded)
- // Now can use for similarity calculations
-}
-```
-
----
-
-### `async update(params: UpdateParams): Promise`
-Updates an existing entity.
-
-**Parameters:**
-- `id` (required) - Entity ID to update
-- `data` - New content (will re-embed)
-- `type` - New type classification
-- `metadata` - New or partial metadata
-- `merge` - Merge metadata (true) or replace (false), default: true
-- `vector` - New embedding vector
-- `confidence` - Update type classification confidence ✨ *New in v4.3.0*
-- `weight` - Update entity importance/salience ✨ *New in v4.3.0*
-
-**Example:**
-```typescript
-await brain.update({
- id: 'uuid-1234',
- metadata: { status: 'reviewed' },
- confidence: 0.98, // Increase confidence after review
- weight: 0.90, // Boost importance
- merge: true // Keeps existing metadata, adds status
-})
-```
-
----
-
-### `async delete(id: string): Promise`
-Deletes an entity and all its relationships.
-
-**Parameters:**
-- `id` - Entity ID to delete
-
-**Example:**
-```typescript
-await brain.delete('uuid-1234')
-```
-
----
-
-### `async relate(params: RelateParams): Promise`
-Creates a relationship between two entities.
-
-**Parameters:**
-- `from` (required) - Source entity ID
-- `to` (required) - Target entity ID
-- `type` (required) - VerbType enum value
-- `weight` - Relationship strength (0-1), default: 1
-- `metadata` - Relationship metadata
-- `bidirectional` - Create reverse relationship
-- `service` - Service name for multi-tenancy
-- `writeOnly` - Skip validation
-
-**Validation:**
-- `from` and `to` must be different (no self-referential relationships)
-- `type` must be a valid VerbType enum value
-- `weight` if provided must be between 0 and 1
-
-**Returns:** Relationship ID
-
-**Example:**
-```typescript
-const relationId = await brain.relate({
- from: 'person-123',
- to: 'org-456',
- type: VerbType.WorksWith,
- weight: 0.95,
- metadata: { since: '2020-01-01' },
- bidirectional: true
-})
-```
-
----
-
-### `async unrelate(id: string): Promise`
-Removes a relationship.
-
-**Parameters:**
-- `id` - Relationship ID
-
-**Example:**
-```typescript
-await brain.unrelate('relation-789')
-```
-
----
-
-### `async getRelations(params?: GetRelationsParams): Promise`
-Gets relationships for entities.
-
-**Parameters:**
-- `from` - Source entity ID
-- `to` - Target entity ID
-- `type` - Filter by VerbType(s)
-- `limit` - Maximum results (default: 100)
-- `offset` - Pagination offset
-- `service` - Filter by service
-
-**Example:**
-```typescript
-// Get all relationships from an entity
-const relations = await brain.getRelations({
- from: 'person-123',
- type: [VerbType.WorksWith, VerbType.ReportsTo],
- limit: 50
-})
-```
-
----
-
-### `async close(): Promise`
-Shuts down the brain, cleaning up resources.
-
-**Example:**
-```typescript
-await brain.close()
-```
-
----
-
-## Batch Operations
-
-### `async addMany(params: AddManyParams): Promise`
-Adds multiple entities in batch.
-
-**Parameters:**
-- `items` (required) - Array of AddParams
-- `parallel` - Process in parallel (default: true)
-- `chunkSize` - Batch size (default: 100)
-- `continueOnError` - Continue if some fail
-- `onProgress` - Progress callback
-
-**Returns:** BatchResult with successful/failed counts
-
-**Example:**
-```typescript
-const result = await brain.addMany({
- items: [
- { data: 'Doc 1', type: NounType.Document },
- { data: 'Doc 2', type: NounType.Document },
- { data: 'Doc 3', type: NounType.Document }
- ],
- parallel: true,
- onProgress: (done, total) => console.log(`${done}/${total}`)
-})
-
-console.log(`Added: ${result.successful.length}`)
-console.log(`Failed: ${result.failed.length}`)
-```
-
----
-
-### `async updateMany(params: UpdateManyParams): Promise`
-Updates multiple entities in batch.
-
-**Parameters:**
-- `updates` (required) - Array of UpdateParams
-- `parallel` - Process in parallel
-- `continueOnError` - Continue on failures
-- `onProgress` - Progress callback
-
-**Example:**
-```typescript
-const result = await brain.updateMany({
- updates: ids.map(id => ({
- id,
- metadata: { processed: true },
- merge: true
- })),
- parallel: true
-})
-```
-
----
-
-### `async deleteMany(params: DeleteManyParams): Promise`
-Deletes multiple entities.
-
-**Parameters:**
-- `ids` - Specific IDs to delete
-- `type` - Delete all of a type
-- `where` - Delete by metadata filter
-- `limit` - Maximum to delete (safety limit)
-- `onProgress` - Progress callback
-
-**Example:**
-```typescript
-// Delete specific IDs
-await brain.deleteMany({
- ids: ['id1', 'id2', 'id3']
-})
-
-// Delete by type
-await brain.deleteMany({
- type: NounType.Document,
- where: { status: 'draft' },
- limit: 100
-})
-```
-
----
-
-### `async relateMany(params: RelateManyParams): Promise`
-Creates multiple relationships in batch.
-
-**Parameters:**
-- `relations` (required) - Array of RelateParams
-- `parallel` - Process in parallel
-- `continueOnError` - Continue on failures
-- `onProgress` - Progress callback
-
-**Example:**
-```typescript
-const result = await brain.relateMany({
- relations: [
- { from: 'a', to: 'b', type: VerbType.RelatedTo },
- { from: 'b', to: 'c', type: VerbType.DependsOn },
- { from: 'c', to: 'a', type: VerbType.References }
- ]
-})
-```
-
----
-
-## Search & Discovery
-
-### `async find(query: string | FindParams): Promise`
-Universal search with Triple Intelligence fusion.
-
-**Parameters:**
-- `query` - Natural language query or structured params
-- `vector` - Direct vector search
-- `type` - Filter by NounType(s)
-- `where` - Metadata filters
-- `connected` - Graph constraints
-- `near` - Proximity search
-- `fusion` - Fusion strategy and weights
-- `limit` - Maximum results
-- `offset` - Pagination offset
-- `explain` - Include score explanation
-- `service` - Filter by service
-- `writeOnly` - Skip validation
-
-**Returns:** Array of Result objects with scores
-
-**Result Properties:** ✨ *Enhanced in v4.3.0*
-- `id` - Entity ID
-- `score` - Relevance score (0-1)
-- `type` - Entity type (flattened for convenience) *Enhanced*
-- `metadata` - Entity metadata (flattened) *Enhanced*
-- `data` - Entity data (flattened) *Enhanced*
-- `confidence` - Type classification confidence (flattened) *New*
-- `weight` - Entity importance (flattened) *New*
-- `entity` - Full Entity object (preserved for backward compatibility)
-- `explanation` - Score explanation (if `explain: true`)
-
-**Example:**
-```typescript
-// Natural language search
-const results = await brain.find('recent product launches')
-
-// NEW in v4.3.0: Direct access to flattened fields
-console.log(results[0].metadata) // Direct access (convenient!)
-console.log(results[0].confidence) // Type confidence
-console.log(results[0].weight) // Entity importance
-
-// Backward compatible: Nested access still works
-console.log(results[0].entity.metadata) // Also works
-
-// Structured search with fusion
-const results = await brain.find({
- query: 'machine learning',
- type: [NounType.Document, NounType.Project],
- where: { year: 2024 },
- connected: {
- to: 'research-dept',
- via: VerbType.CreatedBy
- },
- fusion: {
- strategy: 'adaptive',
- weights: { vector: 0.5, graph: 0.3, field: 0.2 }
- },
- limit: 20,
- explain: true
-})
-
-// Access results with clean, predictable patterns
-for (const result of results) {
- console.log(`Score: ${result.score}`)
- console.log(`Type: ${result.type}`)
- console.log(`Confidence: ${result.confidence ?? 'N/A'}`)
- console.log(`Weight: ${result.weight ?? 'N/A'}`)
- console.log(`Metadata:`, result.metadata)
-}
-```
-
----
-
-### `async similar(params: SimilarParams): Promise`
-Finds similar entities using vector similarity.
-
-**Parameters:**
-- `to` (required) - Entity ID, Entity object, or Vector
-- `limit` - Maximum results (default: 10)
-- `threshold` - Minimum similarity (0-1)
-- `type` - Filter by type(s)
-- `where` - Metadata filters
-
-**Returns:** Array of Result objects (same structure as `find()`) ✨ *Enhanced in v4.3.0*
-
-**Example:**
-```typescript
-const similar = await brain.similar({
- to: 'doc-123',
- limit: 5,
- threshold: 0.8,
- type: NounType.Document
-})
-
-// NEW in v4.3.0: Access flattened fields directly
-for (const result of similar) {
- console.log(`Similarity: ${result.score}`)
- console.log(`Type: ${result.type}`) // Flattened
- console.log(`Confidence: ${result.confidence}`) // Flattened
- console.log(`Metadata:`, result.metadata) // Flattened
-}
-```
-
----
-
-### `async embed(data: any): Promise` ✨ *v7.1.0*
-Generates an embedding vector from data.
-
-**Parameters:**
-- `data` - Text, array, or object to embed
-
-**Returns:** 384-dimensional embedding vector
-
-**Example:**
-```typescript
-const vector = await brain.embed('Machine learning concepts')
-console.log(vector.length) // 384
-```
-
----
-
-### `async embedBatch(texts: string[]): Promise` ✨ *v7.1.0*
-Batch embed multiple texts efficiently.
-
-**Parameters:**
-- `texts` - Array of strings to embed
-
-**Returns:** Array of 384-dimensional vectors
-
-**Example:**
-```typescript
-const embeddings = await brain.embedBatch([
- 'Machine learning is fascinating',
- 'Deep neural networks',
- 'Natural language processing'
-])
-console.log(embeddings.length) // 3
-console.log(embeddings[0].length) // 384
-```
-
----
-
-### `async similarity(textA: string, textB: string): Promise` ✨ *v7.1.0*
-Calculate semantic similarity between two texts.
-
-**Parameters:**
-- `textA` - First text
-- `textB` - Second text
-
-**Returns:** Similarity score between 0 (different) and 1 (identical)
-
-**Example:**
-```typescript
-const score = await brain.similarity(
- 'The cat sat on the mat',
- 'A feline was resting on the rug'
-)
-console.log(score) // ~0.85 (high semantic similarity)
-```
-
----
-
-### `async neighbors(entityId: string, options?): Promise` ✨ *v7.1.0*
-Get graph neighbors of an entity.
-
-**Parameters:**
-- `entityId` - Entity to get neighbors for
-- `options.direction` - 'outgoing', 'incoming', or 'both' (default: 'both')
-- `options.depth` - Traversal depth (default: 1)
-- `options.verbType` - Filter by relationship type
-- `options.limit` - Maximum neighbors to return
-
-**Returns:** Array of neighbor entity IDs
-
-**Example:**
-```typescript
-// Get all connected entities
-const neighbors = await brain.neighbors(entityId)
-
-// Get outgoing connections only
-const outgoing = await brain.neighbors(entityId, {
- direction: 'outgoing',
- limit: 10
-})
-
-// Multi-hop traversal
-const extended = await brain.neighbors(entityId, {
- depth: 2,
- direction: 'both'
-})
-```
-
----
-
-### `async findDuplicates(options?): Promise` ✨ *v7.1.0*
-Find semantic duplicates in the database.
-
-**Parameters:**
-- `options.threshold` - Minimum similarity (default: 0.85)
-- `options.type` - Filter by NounType
-- `options.limit` - Maximum duplicate groups (default: 100)
-
-**Returns:** Array of duplicate groups with similarity scores
-
-**Example:**
-```typescript
-// Find all duplicates
-const duplicates = await brain.findDuplicates()
-
-for (const group of duplicates) {
- console.log('Original:', group.entity.id)
- for (const dup of group.duplicates) {
- console.log(` Duplicate: ${dup.entity.id} (${dup.similarity.toFixed(2)})`)
- }
-}
-
-// Find person duplicates with higher threshold
-const personDupes = await brain.findDuplicates({
- type: NounType.PERSON,
- threshold: 0.9,
- limit: 50
-})
-```
-
----
-
-### `async indexStats(): Promise` ✨ *v7.1.0*
-Get comprehensive index statistics.
-
-**Returns:**
-- `entities` - Total entity count
-- `vectors` - Total vectors in HNSW index
-- `relationships` - Total relationships in graph
-- `metadataFields` - Indexed metadata fields
-- `memoryUsage.vectors` - Vector memory in bytes
-- `memoryUsage.graph` - Graph memory in bytes
-- `memoryUsage.metadata` - Metadata index memory in bytes
-- `memoryUsage.total` - Total memory usage
-
-**Example:**
-```typescript
-const stats = await brain.indexStats()
-console.log(`Entities: ${stats.entities}`)
-console.log(`Vectors: ${stats.vectors}`)
-console.log(`Relationships: ${stats.relationships}`)
-console.log(`Memory: ${(stats.memoryUsage.total / 1024 / 1024).toFixed(1)}MB`)
-console.log(`Fields: ${stats.metadataFields.join(', ')}`)
-```
-
----
-
-### `async cluster(options?): Promise` ✨ *v7.1.0*
-Cluster entities by semantic similarity.
-
-Groups entities into clusters based on their embedding similarity using
-a greedy algorithm with HNSW-based neighbor lookup.
-
-**Parameters:**
-- `options.threshold` - Similarity threshold (default: 0.8)
-- `options.type` - Filter by NounType
-- `options.minClusterSize` - Minimum entities per cluster (default: 2)
-- `options.limit` - Maximum clusters to return (default: 100)
-- `options.includeCentroid` - Calculate cluster centroids (default: false)
-
-**Returns:** Array of clusters with entities
-
-**Example:**
-```typescript
-// Find all clusters
-const clusters = await brain.cluster()
-
-for (const cluster of clusters) {
- console.log(`${cluster.clusterId}: ${cluster.entities.length} entities`)
-}
-
-// Find document clusters with centroids
-const docClusters = await brain.cluster({
- type: NounType.Document,
- threshold: 0.85,
- minClusterSize: 3,
- includeCentroid: true
-})
-
-// Use centroids for cluster comparison
-for (const cluster of docClusters) {
- console.log(`${cluster.clusterId}: ${cluster.entities.length} entities`)
- if (cluster.centroid) {
- console.log(` Centroid dimensions: ${cluster.centroid.length}`)
- }
-}
-```
-
----
-
-### `async insights(): Promise`
-Gets statistics and insights about the data.
-
-**Returns:**
-- `entities` - Total entity count
-- `relationships` - Total relationship count
-- `types` - Count by NounType
-- `services` - List of services
-- `density` - Relationships per entity
-
-**Example:**
-```typescript
-const insights = await brain.insights()
-console.log(`Entities: ${insights.entities}`)
-console.log(`Graph density: ${insights.density.toFixed(2)}`)
-```
-
----
-
-### `async suggest(params?: SuggestParams): Promise`
-AI-powered suggestions based on current data.
-
-**Parameters:**
-- `context` - Context for suggestions
-- `type` - Filter by type(s)
-- `limit` - Maximum suggestions per category
-- `service` - Filter by service
-
-**Returns:**
-- `queries` - Suggested search queries
-- `connections` - Suggested relationships
-- `insights` - Data insights
-- `patterns` - Detected patterns
-
-**Example:**
-```typescript
-const suggestions = await brain.suggest({
- context: 'product development',
- limit: 5
-})
-
-// Use suggestions
-for (const query of suggestions.queries) {
- console.log(`Try searching: ${query}`)
-}
-```
-
----
-
-## Security API
-
-Access via: `const security = await brain.security()`
-
-### `async encrypt(data: string): Promise`
-Encrypts data using AES-256-CBC.
-
-**Example:**
-```typescript
-const encrypted = await security.encrypt('sensitive data')
-```
-
----
-
-### `async decrypt(encryptedData: string): Promise`
-Decrypts previously encrypted data.
-
-**Example:**
-```typescript
-const decrypted = await security.decrypt(encrypted)
-```
-
----
-
-### `async hash(data: string, algorithm?: 'sha256'|'sha512'): Promise`
-Creates cryptographic hash.
-
-**Example:**
-```typescript
-const hash = await security.hash('password', 'sha512')
-```
-
----
-
-### `async compare(data: string, hash: string): Promise`
-Compares data against hash (constant-time).
-
-**Example:**
-```typescript
-const matches = await security.compare('password', hash)
-```
-
----
-
-### `async generateToken(bytes?: number): Promise`
-Generates secure random token.
-
-**Example:**
-```typescript
-const token = await security.generateToken(32)
-```
-
----
-
-### `async deriveKey(password: string, salt?: string): Promise<{key: string, salt: string}>`
-Derives key from password using PBKDF2-like iterations.
-
-**Example:**
-```typescript
-const { key, salt } = await security.deriveKey('userPassword')
-```
-
----
-
-### `async sign(data: string, secret?: string): Promise`
-Signs data with HMAC-SHA256.
-
-**Example:**
-```typescript
-const signature = await security.sign(data, secret)
-```
-
----
-
-### `async verify(data: string, signature: string, secret: string): Promise`
-Verifies HMAC signature.
-
-**Example:**
-```typescript
-const valid = await security.verify(data, signature, secret)
-```
-
----
-
-## Storage Configuration
-
-Brainy supports multiple storage backends for different deployment scenarios.
-
-### Storage Types
-
-#### File System Storage (Recommended for Development)
-Persistent local storage using the filesystem.
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'filesystem',
- rootDirectory: './brainy-data'
- }
-})
-```
-
-#### Amazon S3 Storage
-Scalable cloud storage with S3.
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 's3',
- s3Storage: {
- bucketName: 'my-bucket',
- region: 'us-east-1',
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
- }
- }
-})
-```
-
-#### Cloudflare R2 Storage
-Scalable cloud storage with Cloudflare R2 (S3-compatible).
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'r2',
- r2Storage: {
- bucketName: 'my-bucket',
- accountId: process.env.CF_ACCOUNT_ID,
- accessKeyId: process.env.R2_ACCESS_KEY_ID,
- secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
- }
- }
-})
-```
-
-#### Google Cloud Storage (Native SDK) 🆕
-**Recommended for GCS deployments.** Uses native `@google-cloud/storage` SDK with automatic authentication.
-
-**Key Benefits:**
-- ✅ Application Default Credentials (ADC) - Zero config in Cloud Run/GCE
-- ✅ Better performance with native GCS optimizations
-- ✅ No HMAC key management required
-- ✅ Automatic service account integration
-
-**Option 1: Explicit Type (Recommended)**
-
-With Application Default Credentials (Cloud Run/GCE):
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs-native', // ⚠️ Must be 'gcs-native' for native SDK
- gcsNativeStorage: {
- bucketName: 'my-bucket'
- // No credentials needed - ADC automatic!
- }
- }
-})
-```
-
-With Service Account Key File:
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs-native',
- gcsNativeStorage: {
- bucketName: 'my-bucket',
- keyFilename: '/path/to/service-account.json'
- }
- }
-})
-```
-
-With Service Account Credentials Object:
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs-native',
- gcsNativeStorage: {
- bucketName: 'my-bucket',
- credentials: {
- client_email: 'service@project.iam.gserviceaccount.com',
- private_key: process.env.GCS_PRIVATE_KEY
- }
- }
- }
-})
-```
-
-**Option 2: Auto-Detection**
-
-You can omit the `type` field and let Brainy auto-detect based on the config object:
-```typescript
-const brain = new Brainy({
- storage: {
- gcsNativeStorage: {
- bucketName: 'my-bucket'
- // type defaults to 'auto', will use native SDK
- }
- }
-})
-```
-
-#### Google Cloud Storage (S3-Compatible) - Legacy
-GCS using HMAC keys for S3-compatible access. **Consider migrating to 'gcs-native' for better performance.**
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs', // ⚠️ Must be 'gcs' for S3-compatible mode
- gcsStorage: { // ⚠️ Must use 'gcsStorage' (not 'gcsNativeStorage')
- bucketName: 'my-bucket',
- region: 'us-central1',
- accessKeyId: process.env.GCS_ACCESS_KEY_ID,
- secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY,
- endpoint: 'https://storage.googleapis.com'
- }
- }
-})
-```
-
-**⚠️ Common Mistakes:**
-```typescript
-// ❌ WRONG - type/config mismatch (will fall back to memory storage)
-{
- type: 'gcs',
- gcsNativeStorage: { bucketName: 'my-bucket' }
-}
-
-// ❌ WRONG - type/config mismatch (will fall back to memory storage)
-{
- type: 'gcs-native',
- gcsStorage: { bucketName: 'my-bucket', accessKeyId: '...', secretAccessKey: '...' }
-}
-
-// ✅ CORRECT - type matches config object
-{
- type: 'gcs-native',
- gcsNativeStorage: { bucketName: 'my-bucket' }
-}
-
-// ✅ CORRECT - auto-detection
-{
- gcsNativeStorage: { bucketName: 'my-bucket' }
-}
-```
-
-### Storage Features
-
-All storage adapters support:
-- ✅ **UUID-based sharding** - 256 buckets (00-ff) for scalability
-- ✅ **Pagination** - Efficient cursor-based pagination across shards
-- ✅ **Statistics** - O(1) count operations
-- ✅ **Caching** - Multi-level cache for performance
-- ✅ **Backpressure** - Automatic flow control
-- ✅ **Throttling detection** - Adaptive retry on rate limits
-
-### Migration from HMAC to Native GCS
-
-If you're currently using `type: 'gcs'` with HMAC keys, migrating to `type: 'gcs-native'` is straightforward:
-
-**Before (S3-Compatible with HMAC):**
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs', // ⚠️ Old: S3-compatible mode
- gcsStorage: { // ⚠️ Old: HMAC credentials
- bucketName: 'my-bucket',
- accessKeyId: process.env.GCS_ACCESS_KEY_ID,
- secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY
- }
- }
-})
-```
-
-**After (Native SDK with ADC):**
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs-native', // ✅ New: Native SDK mode
- gcsNativeStorage: { // ✅ New: ADC authentication
- bucketName: 'my-bucket'
- // ADC handles authentication automatically
- }
- }
-})
-```
-
-**⚠️ Important Migration Notes:**
-
-1. **Change BOTH the type AND the config object:**
- - `type: 'gcs'` → `type: 'gcs-native'`
- - `gcsStorage` → `gcsNativeStorage`
-
-2. **Remove HMAC keys** - Not needed with ADC:
- - Remove `accessKeyId`
- - Remove `secretAccessKey`
- - Remove `region` (optional with native SDK)
-
-3. **Set up ADC in your environment:**
- ```bash
- # Cloud Run/GCE: Nothing needed, ADC is automatic
-
- # Local development:
- gcloud auth application-default login
-
- # Or set GOOGLE_APPLICATION_CREDENTIALS:
- export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
- ```
-
-**Data Migration:**
-✅ **No data migration required!** Both adapters use the same path structure:
-- `entities/nouns/vectors/{shard}/{id}.json`
-- `entities/nouns/metadata/{shard}/{id}.json`
-- `entities/verbs/vectors/{shard}/{id}.json`
-
-Simply change your code and restart your application. Existing data will work immediately.
-
----
-
-## Configuration API
-
-Access via: `const config = await brain.config()`
-
-### `async set(params): Promise`
-Sets configuration value.
-
-**Parameters:**
-- `key` (required) - Configuration key
-- `value` (required) - Value to store
-- `encrypt` - Encrypt value
-
-**Example:**
-```typescript
-await config.set({
- key: 'api.key',
- value: 'secret-key-123',
- encrypt: true
-})
-```
-
----
-
-### `async get(params): Promise`
-Gets configuration value.
-
-**Parameters:**
-- `key` (required) - Configuration key
-- `decrypt` - Decrypt if encrypted
-- `defaultValue` - Default if not found
-
-**Example:**
-```typescript
-const apiKey = await config.get({
- key: 'api.key',
- decrypt: true,
- defaultValue: 'default-key'
-})
-```
-
----
-
-### `async delete(key: string): Promise`
-Deletes configuration key.
-
----
-
-### `async list(): Promise`
-Lists all configuration keys.
-
----
-
-### `async has(key: string): Promise`
-Checks if key exists.
-
----
-
-### `async clear(): Promise`
-Clears all configuration.
-
----
-
-### `async export(): Promise>`
-Exports all configuration.
-
----
-
-### `async import(config: Record): Promise`
-Imports configuration.
-
----
-
-## Data Management API
-
-Access via: `const data = await brain.data()`
-
-### `async backup(options?: BackupOptions): Promise`
-Creates backup of all data.
-
-**Parameters:**
-- `includeVectors` - Include embeddings (default: true)
-- `compress` - Compress backup (default: false)
-
-**Example:**
-```typescript
-const backup = await data.backup({
- includeVectors: true,
- compress: true
-})
-// Save backup to file
-fs.writeFileSync('backup.json', JSON.stringify(backup))
-```
-
----
-
-### `async restore(params): Promise`
-Restores from backup.
-
-**Parameters:**
-- `backup` (required) - Backup data
-- `merge` - Merge with existing data
-- `overwrite` - Overwrite conflicts
-- `validate` - Validate backup format
-
-**Example:**
-```typescript
-const backup = JSON.parse(fs.readFileSync('backup.json'))
-await data.restore({
- backup,
- merge: false,
- overwrite: true
-})
-```
-
----
-
-### `async clear(params): Promise`
-Clears specified data types.
-
-**Parameters:**
-- `entities` - Clear entities
-- `relations` - Clear relationships
-- `config` - Clear configuration
-
-**Example:**
-```typescript
-await data.clear({
- entities: true,
- relations: true
-})
-```
-
----
-
-### `async import(params): Promise`
-Imports data from various formats with automatic detection.
-
-**Parameters:**
-- `data` (required) - Data to import (Buffer, string, array, or file path)
-- `format` - 'auto' | 'json' | 'csv' | 'excel' | 'pdf' | 'yaml' | 'text' (default: 'auto')
-- `mapping` - Field mapping configuration
-- `batchSize` - Import batch size (default: 50)
-- `validate` - Validate items before import
-- `relationships` - Extract relationships automatically (default: true)
-
-**CSV-specific options:**
-- `csvDelimiter` - Column delimiter (auto-detected if not specified)
-- `csvHeaders` - First row contains headers (default: true)
-- `encoding` - Character encoding (auto-detected if not specified)
-
-**Excel-specific options:**
-- `excelSheets` - Sheet names array or 'all' for all sheets
-
-**PDF-specific options:**
-- `pdfExtractTables` - Extract tables from PDFs (default: true)
-- `pdfPreserveLayout` - Preserve text layout (default: true)
-
-**Examples:**
-```typescript
-// Import CSV file with auto-detection
-const csvResult = await brain.import('customers.csv')
-// Auto-detects: format, encoding, delimiter, field types
-
-// Import Excel workbook
-const excelResult = await brain.import('sales-data.xlsx', {
- excelSheets: ['Q1', 'Q2'] // Import specific sheets
-})
-
-// Import PDF with table extraction
-const pdfResult = await brain.import('report.pdf', {
- pdfExtractTables: true
-})
-
-// Import data array
-const dataResult = await brain.import([
- { name: 'Alice', role: 'Engineer' },
- { name: 'Bob', role: 'Designer' }
-], {
- batchSize: 100,
- relationships: true // Auto-extract relationships
-})
-
-// Import with custom CSV delimiter
-const tsvResult = await brain.import('data.tsv', {
- format: 'csv',
- csvDelimiter: '\t'
-})
-```
-
-**Returns:**
-```typescript
-{
- success: boolean
- imported: number // Number of items successfully imported
- failed: number // Number of items that failed
- entityIds: string[] // IDs of created entities
- metadata: {
- format: string // Detected format
- encoding?: string // Detected encoding (CSV)
- delimiter?: string // Detected delimiter (CSV)
- sheets?: string[] // Processed sheets (Excel)
- pageCount?: number // Number of pages (PDF)
- }
-}
-```
-
----
-
-### `async export(params?: ExportOptions): Promise`
-Exports data in various formats.
-
-**Parameters:**
-- `format` - 'json' | 'csv' | 'parquet'
-- `filter` - Filter options
-- `includeVectors` - Include embeddings
-
-**Example:**
-```typescript
-const exported = await data.export({
- format: 'csv',
- where: { type: NounType.Document },
- includeVectors: false
-})
-```
-
----
-
-### `getStats(): StatsResult`
-Gets complete statistics about entities and relationships. All stats are **O(1) pre-calculated** - updated when entities/relationships are added/removed.
-
-**Returns:**
-```typescript
-{
- entities: {
- total: number // Total entity count
- byType: Record // Entity count by type
- }
- relationships: {
- totalRelationships: number // Total relationship/edge count
- relationshipsByType: Record // Relationship count by type
- uniqueSourceNodes: number // Number of unique source entities
- uniqueTargetNodes: number // Number of unique target entities
- totalNodes: number // Total unique entities in relationships
- }
- density: number // Relationships per entity ratio
-}
-```
-
-**Example:**
-```typescript
-const stats = brain.getStats()
-
-// Total counts (O(1) operations)
-const totalNouns = stats.entities.total
-const totalVerbs = stats.relationships.totalRelationships
-const totalRelations = stats.relationships.totalRelationships // alias
-
-// Counts by type (O(1) operations)
-const nounTypes = stats.entities.byType
-const verbTypes = stats.relationships.relationshipsByType
-
-// Graph metrics
-console.log(`Entities: ${totalNouns}`)
-console.log(`Relationships: ${totalVerbs}`)
-console.log(`Density: ${stats.density.toFixed(2)}`)
-console.log(`Types:`, Object.keys(nounTypes))
-```
-
-**Performance:**
-- ✅ All counts pre-calculated in memory
-- ✅ O(1) access time
-- ✅ Updated automatically on add/remove
-- ✅ No expensive full scans required
-
-**Note:** For more granular counting operations, see the `brain.counts` API below.
-
----
-
-## Query API
-
-Access via: `brain.query`
-
-### `async entities(params?): Promise>`
-Query entities with pagination.
-
-**Parameters:**
-- `type` - Filter by type
-- `where` - Metadata filters
-- `limit` - Page size
-- `cursor` - Pagination cursor
-- `service` - Filter by service
-
-**Example:**
-```typescript
-const result = await brain.query.entities({
- type: NounType.Document,
- where: { status: 'published' },
- limit: 50
-})
-
-// Get next page
-if (result.nextCursor) {
- const nextPage = await brain.query.entities({
- cursor: result.nextCursor
- })
-}
-```
-
----
-
-### `async relations(params?): Promise>`
-Query relationships with pagination.
-
-**Parameters:**
-- `from` - Source entity
-- `to` - Target entity
-- `type` - Relationship type
-- `limit` - Page size
-- `cursor` - Pagination cursor
-
----
-
-## Neural API
-
-Access via: `brain.neural()`
-
-### Similarity Operations
-
-#### `async similar(a, b, options?): Promise`
-Calculates similarity between items.
-
-**Example:**
-```typescript
-const similarity = await neural.similar('doc1', 'doc2', {
- explain: true
-})
-```
-
----
-
-### Clustering Operations
-
-#### `async clusters(options?): Promise`
-General purpose clustering.
-
-**Parameters:**
-- `algorithm` - 'hierarchical' | 'kmeans' | 'dbscan' | 'spectral'
-- `k` - Number of clusters (for kmeans)
-- `threshold` - Distance threshold
-- `minPoints` - Minimum points per cluster
-
-**Example:**
-```typescript
-const clusters = await neural.clusters({
- algorithm: 'kmeans',
- k: 5
-})
-```
-
----
-
-#### `async clustersByDomain(params): Promise`
-Domain-specific clustering.
-
-**Example:**
-```typescript
-const clusters = await neural.clustersByDomain({
- domain: 'technology',
- field: 'category',
- maxClusters: 10
-})
-```
-
----
-
-### Outlier Detection
-
-#### `async outliers(options?): Promise`
-Detects anomalous entities.
-
-**Parameters:**
-- `method` - 'isolation' | 'lof' | 'statistical' | 'autoencoder'
-- `threshold` - Outlier threshold (default: 2.5 std deviations)
-- `returnScores` - Return anomaly scores
-
-**Example:**
-```typescript
-const outliers = await neural.outliers({
- method: 'isolation',
- threshold: 3.0,
- returnScores: true
-})
-```
-
----
-
-### Visualization
-
-#### `async visualize(options?): Promise`
-Generates visualization data for entities.
-
-**Parameters:**
-- `layout` - 'force' | 'circular' | 'hierarchical' | 'random'
-- `dimensions` - 2D or 3D
-- `includeEdges` - Include relationships
-
-**Example:**
-```typescript
-const vizData = await neural.visualize({
- layout: 'force',
- dimensions: 3,
- includeEdges: true
-})
-```
-
----
-
-## NLP API
-
-Access via: `brain.nlp()`
-
-### `async processNaturalQuery(query: string): Promise`
-Converts natural language to structured query.
-
-**Example:**
-```typescript
-const structured = await nlp.processNaturalQuery(
- "Find all documents about AI created last month"
-)
-// Returns structured query with type, time filters, etc.
-```
-
----
-
-### `async extract(text: string, options?): Promise`
-Extracts entities from text using neural matching to NounTypes.
-
-**Parameters:**
-- `types` - Target NounTypes to extract
-- `confidence` - Minimum confidence (0-1)
-- `includeVectors` - Include embeddings
-- `neuralMatching` - Use neural type matching (default: true)
-
-**Returns:** Array of extracted entities with proper NounType classification
-
-**Example:**
-```typescript
-const entities = await nlp.extract(
- "John Smith from Microsoft visited New York on Jan 15",
- {
- types: [NounType.Person, NounType.Organization, NounType.Location],
- confidence: 0.7,
- neuralMatching: true
- }
-)
-// Returns:
-// [
-// { text: "John Smith", type: NounType.Person, confidence: 0.92 },
-// { text: "Microsoft", type: NounType.Organization, confidence: 0.88 },
-// { text: "New York", type: NounType.Location, confidence: 0.85 }
-// ]
-```
-
----
-
-### `async sentiment(text: string, options?): Promise`
-Analyzes text sentiment.
-
-**Parameters:**
-- `granularity` - 'document' | 'sentence' | 'aspect'
-- `aspects` - Aspects to analyze
-
-**Returns:**
-- `overall` - Document sentiment (-1 to 1)
-- `sentences` - Sentence-level sentiment
-- `aspects` - Aspect-based sentiment
-
-**Example:**
-```typescript
-const sentiment = await nlp.sentiment(
- "The product quality is excellent but the price is too high",
- {
- granularity: 'aspect',
- aspects: ['quality', 'price']
- }
-)
-// Returns:
-// overall: { score: 0.2, label: 'mixed' }
-// aspects: {
-// quality: { score: 0.9, magnitude: 0.8 },
-// price: { score: -0.7, magnitude: 0.7 }
-// }
-```
-
----
-
-## Streaming Pipeline API
-
-Access via: `brain.stream()`
-
-### Source Operations
-
-#### `source(generator): Pipeline`
-Sets data source.
-
-**Example:**
-```typescript
-async function* dataGenerator() {
- for (let i = 0; i < 100; i++) {
- yield { id: i, data: `Item ${i}` }
- }
-}
-
-brain.stream()
- .source(dataGenerator())
- .map(item => item.data)
- .sink(console.log)
- .run()
-```
-
----
-
-### Transform Operations
-
-#### `map(fn): Pipeline`
-Transforms each item.
-
-#### `flatMap(fn): Pipeline`
-Maps and flattens arrays.
-
-#### `filter(predicate): Pipeline`
-Filters items.
-
-#### `tap(fn): Pipeline`
-Side effects without modification.
-
-#### `retry(fn, maxRetries?, backoff?): Pipeline`
-Retries failed operations.
-
----
-
-### Batching & Windowing
-
-#### `batch(size, timeoutMs?): Pipeline`
-Groups items into batches.
-
-#### `window(size, type?): Pipeline`
-Creates sliding or tumbling windows.
-
-#### `buffer(size, strategy?): Pipeline`
-Buffers with backpressure handling.
-
----
-
-### Sink Operations
-
-#### `sink(handler): Pipeline`
-Custom sink handler.
-
-#### `toBrainy(options?): Pipeline`
-Sinks data to Brainy.
-
-**Example:**
-```typescript
-brain.stream()
- .source(dataSource)
- .map(transform)
- .batch(100)
- .toBrainy({
- type: NounType.Document,
- metadata: { source: 'stream' }
- })
- .run()
-```
-
----
-
-### Execution
-
-#### `run(options?): Promise`
-Executes the pipeline.
-
-**Parameters:**
-- `workers` - Number of workers or 'auto'
-- `monitoring` - Enable monitoring
-- `maxThroughput` - Rate limiting
-- `backpressure` - 'drop' | 'buffer' | 'pause'
-- `errorHandler` - Error callback
-
----
-
-## Type Definitions
-
-### Entity Types (NounType)
-31 types including:
-- `Person`, `Organization`, `Location`
-- `Document`, `File`, `Message`, `Content`
-- `Product`, `Service`, `Resource`
-- `Event`, `Task`, `Project`, `Process`
-- `User`, `Role`, `State`
-- `Concept`, `Topic`, `Hypothesis`
-- And more...
-
-### Relationship Types (VerbType)
-40 types including:
-- `RelatedTo`, `Contains`, `PartOf`
-- `Creates`, `Modifies`, `Transforms`
-- `DependsOn`, `Requires`, `Uses`
-- `Owns`, `BelongsTo`, `MemberOf`
-- `Supervises`, `ReportsTo`, `WorksWith`
-- And more...
-
-### Core Interfaces
-
-✨ *Updated in v4.3.0 - Added confidence/weight to Entity, flattened Result fields*
-
-```typescript
-interface Entity {
- id: string
- vector: Vector
- type: NounType
- data?: any
- metadata?: T
- service?: string
- createdAt: number
- updatedAt?: number
- confidence?: number // NEW: Type classification confidence (0-1)
- weight?: number // NEW: Entity importance/salience (0-1)
-}
-
-interface Relation {
- id: string
- from: string
- to: string
- type: VerbType
- weight?: number
- confidence?: number // Relationship confidence
- metadata?: T
- evidence?: RelationEvidence // Why this relationship exists
- service?: string
- createdAt: number
-}
-
-interface Result {
- // Search metadata
- id: string
- score: number
-
- // NEW: Flattened entity fields for convenience
- type?: NounType
- metadata?: T
- data?: any
- confidence?: number
- weight?: number
-
- // Full entity (backward compatible)
- entity: Entity
-
- // Score explanation
- explanation?: ScoreExplanation
-}
-```
-
-**Key Changes in v4.3.0:**
-- ✅ `Entity` now exposes `confidence` and `weight`
-- ✅ `Result` flattens commonly-used entity fields to top level
-- ✅ Direct access: `result.metadata` instead of `result.entity.metadata`
-- ✅ Backward compatible: `result.entity` still available
-
----
-
-## Performance Characteristics
-
-### Speed Benchmarks
-- **Add entity**: ~5-10ms (with embedding)
-- **Vector search**: ~1-5ms for 1M entities
-- **Graph traversal**: ~10-20ms (2-hop)
-- **Batch operations**: 1000+ items/second
-
-### Scalability
-- **Entities**: Tested to 10M+
-- **Relationships**: Tested to 100M+
-- **Concurrent operations**: 1000+ parallel
-- **Memory usage**: ~1GB per million entities
-
-### Storage Requirements
-- **Per entity**: ~2KB (with vector)
-- **Per relationship**: ~200 bytes
-- **Indexes**: ~20% overhead
-
----
-
-## Best Practices
-
-### 1. Always Initialize
-```typescript
-const brain = new Brainy()
-await brain.init() // Required before operations
-```
-
-### 2. Use Proper Types
-```typescript
-// ✅ Good - specific type
-await brain.add({ data: 'John', type: NounType.Person })
-
-// ❌ Bad - generic type
-await brain.add({ data: 'John', type: NounType.Thing })
-```
-
-### 3. Batch When Possible
-```typescript
-// ✅ Good - batch operation
-await brain.addMany({ items: documents })
-
-// ❌ Bad - individual adds in loop
-for (const doc of documents) {
- await brain.add(doc) // Slow!
-}
-```
-
-### 4. Use Write-Only for Speed
-```typescript
-// For high-speed ingestion
-await brain.add({
- data: content,
- type: NounType.Document,
- writeOnly: true // Skip validation
-})
-```
-
-### 5. Clean Up Resources
-```typescript
-try {
- // Your operations
-} finally {
- await brain.close() // Always close
-}
-```
-
----
-
-## Input Validation
-
-Brainy uses a **zero-config validation system** that automatically adapts to your system resources:
-
-### Auto-Configured Limits
-- `limit` parameter maximum: Based on available memory (e.g., 8GB RAM = 80K max limit)
-- Query string length: Auto-scaled based on memory
-- Vector dimensions: Must be exactly 384 for all-MiniLM-L6-v2 model
-
-### Common Validation Rules
-- **Pagination**: `limit` and `offset` must be non-negative
-- **Thresholds**: Values like `weight` and `threshold` must be between 0 and 1
-- **Mutual Exclusion**: Cannot use both `query` and `vector` in same request
-- **Type Safety**: `NounType` and `VerbType` must be valid enum values
-- **Self-Reference**: Cannot create relationships from an entity to itself
-
-### Performance Auto-Tuning
-The validation system monitors query performance and adjusts limits automatically:
-- Fast queries with large results → increases limits
-- Slow queries → reduces limits to maintain performance
-
-## Error Handling
-
-All methods validate parameters and throw descriptive errors:
-
-```typescript
-try {
- await brain.add({ data: '', type: NounType.Document })
-} catch (error) {
- // Error: "must provide either data or vector"
-}
-
-try {
- await brain.find({ limit: -1 })
-} catch (error) {
- // Error: "limit must be non-negative"
-}
-
-try {
- await brain.update({ id: 'xyz', metadata: null, merge: false })
-} catch (error) {
- // Error: "must specify at least one field to update"
- // Note: Use metadata: {} to clear, not null
-}
-```
-
----
-
-## Migration from v2
-
-### Old (v2.15)
-```typescript
-brain.add({ data, type: NounType.Document, metadata })
-brain.find({ query, limit: 10 })
-brain.insights()
-```
-
-### New (v3.0)
-```typescript
-brain.add({ data, type: NounType.Document, metadata })
-brain.find({ query, limit: 10 })
-brain.insights()
-```
-
----
-
-## Support
-
-- **GitHub**: https://github.com/soulcraft/brainy
-- **NPM**: https://www.npmjs.com/package/@soulcraft/brainy
-- **Discord**: https://discord.gg/brainy
-
----
-
-*Brainy v3.0 - The Neural Database That Thinks*
\ No newline at end of file
diff --git a/docs/BATCHING.md b/docs/BATCHING.md
index acc11fe6..e70a9e18 100644
--- a/docs/BATCHING.md
+++ b/docs/BATCHING.md
@@ -1,23 +1,28 @@
-# Batch Operations API v5.12.0
+---
+title: Batch Operations
+slug: guides/batching
+public: true
+category: guides
+template: guide
+order: 5
+description: Eliminate N+1 query patterns with batchGet() and storage-level batch APIs for fast multi-entity reads against filesystem and memory storage.
+next:
+ - api/reference
+ - guides/find-system
+---
-> **Enterprise Production-Ready** | Zero N+1 Query Patterns | 90%+ Performance Improvement
+# Batch Operations API
+> **Production-Ready** | Zero N+1 Query Patterns
## Overview
-Brainy v5.12.0 introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
+Brainy provides batch operations at the storage layer to eliminate N+1 query patterns for VFS operations, relationship queries, and entity retrieval.
### Problem Solved
-**Before v5.12.0:**
-- VFS `getTreeStructure()` on cloud storage: **12.7 seconds** for directory with 12 files
-- N+1 query pattern: 1 directory query + N individual file queries (22 sequential calls × 580ms latency)
+The naive pattern of looping and calling `brain.get(id)` once per item issues sequential reads through the storage layer. Batched APIs collapse that into a single read pass.
-**After v5.12.0:**
-- VFS `getTreeStructure()`: **<1 second** for 12 files (90%+ improvement)
-- 2-3 batched calls instead of 22 sequential calls
-- Native cloud storage batch APIs for maximum throughput
-
-**IMPORTANT:** The v5.12.0 batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See v6.0.0 changes for comprehensive storage path optimizations.
+**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`** at the VFS layer and the explicit `batchGet()` / `getNounMetadataBatch()` calls — not to `readFile()` or individual `get()` operations.
---
@@ -42,8 +47,7 @@ results.size // → 3 (number of found entities)
**Performance:**
- Memory storage: Instant (parallel reads)
-- Cloud storage (GCS/S3/Azure): <500ms for 100 entities
-- Throughput: 50-200+ entities/second depending on adapter
+- Filesystem storage: Parallel reads, scales with available IOPS
**Use Cases:**
- Loading multiple entities for display
@@ -56,7 +60,7 @@ results.size // → 3 (number of found entities)
### 2. `storage.getNounMetadataBatch(ids)`
-Batch metadata retrieval with direct O(1) path construction (v6.0.0+).
+Batch metadata retrieval with direct O(1) path construction.
```typescript
const storage = brain.storage as BaseStorage
@@ -65,21 +69,21 @@ const ids = ['id1', 'id2', 'id3']
const metadataMap: Map = await storage.getNounMetadataBatch(ids)
for (const [id, metadata] of metadataMap) {
- console.log(metadata.noun) // Type: 'document', 'person', etc.
- console.log(metadata.data) // Entity data
+ console.log(metadata.noun) // Type: 'document', 'person', etc.
+ console.log(metadata.data) // Entity data
}
```
**Features:**
- ✅ Direct O(1) path construction from ID (no type lookup needed!)
- ✅ Sharding preservation (all paths include `{shard}/{id}`)
-- ✅ COW-aware (respects branch paths)
-- ✅ 40x faster than v5.x type-first architecture
+- ✅ Write-cache coherent (read-after-write consistency)
+- ✅ O(1) path construction — eliminates the per-entity type search of the old type-first layout
**Performance:**
-- ~1ms per 100 entities (consistent, no cache misses!)
-- Cloud storage: Parallel downloads (100-150 concurrent)
-- No type search delays - every ID maps directly to storage path
+- Constant-time path construction per ID — no type-cache misses
+- Filesystem: parallel reads bounded by IOPS
+- No type search delays — every ID maps directly to storage path
---
@@ -92,22 +96,22 @@ const storage = brain.storage as BaseStorage
// Get all relationships from multiple sources
const results: Map = await storage.getVerbsBySourceBatch([
- 'person1',
- 'person2'
+ 'person1',
+ 'person2'
])
// Filter by verb type
const createsResults = await storage.getVerbsBySourceBatch(
- ['person1', 'person2'],
- 'creates'
+ ['person1', 'person2'],
+ 'creates'
)
// Process results
for (const [sourceId, verbs] of results) {
- console.log(`${sourceId} has ${verbs.length} relationships`)
- verbs.forEach(verb => {
- console.log(` → ${verb.verb} → ${verb.targetId}`)
- })
+ console.log(`${sourceId} has ${verbs.length} relationships`)
+ verbs.forEach(verb => {
+ console.log(` → ${verb.verb} → ${verb.targetId}`)
+ })
}
```
@@ -117,130 +121,8 @@ for (const [sourceId, verbs] of results) {
- Bulk export of relationship data
**Performance:**
-- Memory storage: <10ms for 1000 relationships
-- Cloud storage: Batched reads with parallel metadata fetches
-
----
-
-### 4. `storage.readBatchWithInheritance(paths, targetBranch?)`
-
-COW-aware batch path resolution with branch inheritance.
-
-```typescript
-const storage = brain.storage as BaseStorage
-
-const paths = [
- 'entities/nouns/{shard}/id1/metadata.json',
- 'entities/nouns/{shard}/id2/metadata.json'
-]
-
-// Resolves to: branches/{branch}/entities/nouns/{shard}/{id}/metadata.json
-const results: Map = await storage.readBatchWithInheritance(paths, 'my-branch')
-
-// Automatically inherits from parent branches for missing entities
-```
-
-**Features:**
-- ✅ Branch path resolution (`branches/{branch}/...`)
-- ✅ Write cache integration (read-after-write consistency)
-- ✅ COW inheritance (fallback to parent commits for missing entities)
-- ✅ Adapter-agnostic (works with all storage adapters)
-
----
-
-## Cloud Adapter Native Batch APIs
-
-### GCS Storage
-
-```typescript
-const gcsStorage = new GCSStorage({ bucketName: 'my-bucket' })
-
-// Native batch API with 100 concurrent downloads
-const results = await gcsStorage.readBatch(paths)
-
-// Configuration
-gcsStorage.getBatchConfig() // → {
-// maxBatchSize: 1000,
-// maxConcurrent: 100,
-// operationsPerSecond: 1000
-// }
-```
-
-**Performance:**
-- 100 concurrent downloads
-- ~300-500ms for 100 objects
-- HTTP/2 multiplexing for optimal throughput
-
----
-
-### S3 Compatible Storage
-
-Works with Amazon S3, Cloudflare R2, and other S3-compatible services.
-
-```typescript
-const s3Storage = new S3CompatibleStorage({ bucketName: 'my-bucket' })
-
-// Native batch API with 150 concurrent downloads
-const results = await s3Storage.readBatch(paths)
-
-// Configuration
-s3Storage.getBatchConfig() // → {
-// maxBatchSize: 1000,
-// maxConcurrent: 150,
-// operationsPerSecond: 5000
-// }
-```
-
-**Performance:**
-- 150 concurrent downloads
-- ~200-500ms for 150 objects
-- S3 handles 5000+ ops/second with burst capacity
-
----
-
-### R2 Storage (Cloudflare)
-
-```typescript
-const r2Storage = new R2Storage({ bucketName: 'my-bucket' })
-
-// Fastest cloud storage with zero egress fees
-const results = await r2Storage.readBatch(paths)
-
-// Configuration
-r2Storage.getBatchConfig() // → {
-// maxBatchSize: 1000,
-// maxConcurrent: 150,
-// operationsPerSecond: 6000
-// }
-```
-
-**Performance:**
-- 150 concurrent downloads
-- ~200-400ms for 150 objects (fastest!)
-- Zero egress fees enable aggressive caching
-
----
-
-### Azure Blob Storage
-
-```typescript
-const azureStorage = new AzureBlobStorage({ containerName: 'my-container' })
-
-// Native batch API with 100 concurrent downloads
-const results = await azureStorage.readBatch(paths)
-
-// Configuration
-azureStorage.getBatchConfig() // → {
-// maxBatchSize: 1000,
-// maxConcurrent: 100,
-// operationsPerSecond: 3000
-// }
-```
-
-**Performance:**
-- 100 concurrent downloads
-- ~400-600ms for 100 blobs
-- Good throughput with Azure's global network
+- Memory storage: single in-memory pass over the metadata index
+- Filesystem storage: parallel reads through the metadata index
---
@@ -251,12 +133,10 @@ VFS operations automatically use batch APIs for maximum performance.
### Directory Traversal
```typescript
-// OLD: Sequential N+1 pattern (12.7 seconds for 12 files)
+// Tree traversal uses batched reads under the hood
const tree = await brain.vfs.getTreeStructure('/my-dir')
-
-// NEW v5.12.0: Parallel breadth-first with batching (<1 second)
// ✅ PathResolver.getChildren() uses brain.batchGet() internally
-// ✅ Parallel traversal of directories at same tree level
+// ✅ Parallel traversal of directories at the same tree level
// ✅ 2-3 batched calls instead of 22 sequential calls
```
@@ -264,32 +144,26 @@ const tree = await brain.vfs.getTreeStructure('/my-dir')
```
VFS.getTreeStructure()
- ↓ PARALLEL (breadth-first traversal)
- → PathResolver.getChildren() [all dirs at level processed in parallel]
- ↓ BATCHED
- → brain.batchGet(childIds) [1 call instead of N]
- ↓ BATCHED
- → storage.getNounMetadataBatch(ids) [1 call instead of N]
- ↓ ADAPTER-SPECIFIC
- → GCS: readBatch() with 100 concurrent downloads
- → S3: readBatch() with 150 concurrent downloads
- → Memory: Promise.all() parallel reads
+ ↓ PARALLEL (breadth-first traversal)
+ → PathResolver.getChildren() [all dirs at level processed in parallel]
+ ↓ BATCHED
+ → brain.batchGet(childIds) [1 call instead of N]
+ ↓ BATCHED
+ → storage.getNounMetadataBatch(ids) [1 call instead of N]
+ ↓ ADAPTER
+ → Filesystem: Promise.all() parallel reads
+ → Memory: Promise.all() parallel reads
```
-**Performance Gains:**
-- **Before**: 22 sequential calls × 580ms = 12.7 seconds
-- **After**: 2-3 batched calls = <1 second
-- **Improvement**: **90%+ faster** on cloud storage
-
---
## Advanced Features Compatibility
-### ✅ ID-First Storage Architecture (v6.0.0+)
+### ✅ ID-First Storage Architecture
All batch operations use direct ID-first paths - no type lookup needed!
-**NEW v6.0.0 Path Structure:**
+**ID-First Path Structure:**
```
entities/nouns/{SHARD}/{ID}/metadata.json
entities/verbs/{SHARD}/{ID}/metadata.json
@@ -297,9 +171,9 @@ entities/verbs/{SHARD}/{ID}/metadata.json
**Direct O(1) Path Construction:**
```typescript
-// Every ID maps directly to exactly ONE path - 40x faster!
+// Every ID maps directly to exactly ONE path - O(1), no type search
const id = 'abc-123'
-const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
+const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
const path = `entities/nouns/${shard}/${id}/metadata.json`
// No type cache needed!
@@ -309,9 +183,9 @@ const path = `entities/nouns/${shard}/${id}/metadata.json`
```
**Benefits:**
-- **40x faster** on GCS/S3 (eliminates 42-type sequential search)
+- **O(1)** path lookups (eliminates the 42-type sequential search the old type-first layout required)
- **Simpler code** - removed 500+ lines of type cache complexity
-- **Scalable** - works at billion-scale without type tracking overhead
+- **Scalable** - works at large scale without type tracking overhead
---
@@ -329,95 +203,39 @@ const path = `entities/nouns/${shard}/${id}/metadata.json`
---
-### ✅ COW (Copy-on-Write)
+### ✅ Generational MVCC (8.0)
-Batch operations respect branch isolation and time-travel:
+Batch reads always serve the **live** generation through the fast paths
+shown above. Point-in-time reads go through the Db API instead: a pinned
+`Db` (`brain.now()`, `brain.asOf()`) resolves changed ids from immutable
+generation records and unchanged ids from the same live paths batch reads
+use — see the [consistency model](concepts/consistency-model.md).
```typescript
-// Main branch
-const brain = await Brainy.create({ enableCOW: true })
-await brain.add({ type: 'document', data: 'Main' })
-
-// Create fork
-const fork = await brain.fork('experiment')
-
-// Batch operations are isolated
-await brain.batchGet([id1, id2]) // → Reads from: branches/main/...
-await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/...
-```
-
-**Inheritance:**
-- Entities missing from child branch automatically inherit from parent commits
-- `readBatchWithInheritance()` walks commit history for missing items
-- Preserves fork semantics while maintaining performance
-
----
-
-### ✅ fork() and checkout()
-
-```typescript
-const fork = await brain.fork('my-branch')
-await fork.add({ type: 'document', data: 'Fork entity' })
-
-// Batch operations use correct branch
-const results = await fork.batchGet([id1, id2])
-// → Reads from: branches/my-branch/...
-
-// Checkout changes active branch
-await fork.checkout('main')
-const mainResults = await fork.batchGet([id1, id2])
-// → Reads from: branches/main/...
+const db = brain.now() // pinned view
+const entity = await db.get(id) // correct at the pinned generation
+const results = await brain.batchGet(ids) // live state, batched
+await db.release()
```
---
-### ✅ asOf() Time-Travel
+## Why Batching Is Faster
-```typescript
-// Create historical snapshot
-await brain.commit('v1.0')
-const snapshot = await brain.asOf('v1.0')
+Batching's advantage is structural, not a fixed multiplier (the actual speedup
+depends on storage backend, IOPS, and batch size):
-// Batch operations on historical data
-const results = await snapshot.batchGet([id1, id2])
-// → Reads from historical tree state
-```
+- **N+1 elimination** — N sequential reads collapse into a single parallel pass
+ (`Promise.all` over the batch).
+- **O(1) path construction** — every ID maps directly to one storage path, with
+ no per-type cache lookup.
+- **One metadata round-trip** — relationship batches fetch all sources' metadata
+ in a single pass instead of one query per source.
-Historical queries use `HistoricalStorageAdapter` which wraps batch operations to point at specific commits.
-
----
-
-## Performance Benchmarks
-
-### VFS Operations (12 Files)
-
-| Storage | Before v5.12.0 | After v5.12.0 | Improvement |
-|---------|---------------|---------------|-------------|
-| **GCS** | 12.7s | <1s | **92% faster** |
-| **S3** | 13.2s | <1s | **92% faster** |
-| **R2** | 11.8s | <0.8s | **93% faster** |
-| **Azure** | 14.5s | <1s | **93% faster** |
-| **Memory** | 150ms | 50ms | **67% faster** |
-
-### Entity Batch Retrieval (100 Entities)
-
-| Storage | Individual Gets | Batch Get | Improvement |
-|---------|----------------|-----------|-------------|
-| **GCS** | 5.8s | 0.4s | **93% faster** |
-| **S3** | 5.2s | 0.3s | **94% faster** |
-| **R2** | 4.9s | 0.25s | **95% faster** |
-| **Azure** | 6.5s | 0.5s | **92% faster** |
-| **Memory** | 180ms | 15ms | **92% faster** |
-
-### Throughput (Entities/Second)
-
-| Storage | Individual | Batch | Improvement |
-|---------|-----------|-------|-------------|
-| **GCS** | 17 ent/s | 250 ent/s | **14.7x** |
-| **S3** | 19 ent/s | 333 ent/s | **17.5x** |
-| **R2** | 20 ent/s | 400 ent/s | **20x** |
-| **Azure** | 15 ent/s | 200 ent/s | **13.3x** |
-| **Memory** | 556 ent/s | 6667 ent/s | **12x** |
+The integration test `tests/integration/storage-batch-operations.test.ts`
+exercises batch vs. individual reads and asserts that batch retrieval is not
+slower than the per-entity loop for large batches; it does not pin a specific
+multiplier, since that is hardware- and IOPS-dependent.
---
@@ -430,8 +248,8 @@ Batch operations gracefully handle missing or invalid entities:
```typescript
const validId = 'abc-123-...'
const invalidIds = [
- '11111111-1111-1111-1111-111111111111',
- '22222222-2222-2222-2222-222222222222'
+ '11111111-1111-1111-1111-111111111111',
+ '22222222-2222-2222-2222-222222222222'
]
const results = await brain.batchGet([validId, ...invalidIds])
@@ -471,8 +289,8 @@ results.size // → 1 (deduplicated automatically)
```typescript
const entities = []
for (const id of ids) {
- const entity = await brain.get(id)
- if (entity) entities.push(entity)
+ const entity = await brain.get(id)
+ if (entity) entities.push(entity)
}
```
@@ -482,7 +300,7 @@ const results = await brain.batchGet(ids)
const entities = Array.from(results.values())
```
-**Performance Gain:** 10-20x faster on cloud storage.
+**Performance Gain:** Replaces N sequential reads with a single batched pass — no fixed multiplier, it scales with storage IOPS.
---
@@ -492,8 +310,8 @@ const entities = Array.from(results.values())
```typescript
const allVerbs = []
for (const sourceId of sourceIds) {
- const verbs = await brain.getRelations({ from: sourceId })
- allVerbs.push(...verbs)
+ const verbs = await brain.related({ from: sourceId })
+ allVerbs.push(...verbs)
}
```
@@ -504,11 +322,11 @@ const results = await storage.getVerbsBySourceBatch(sourceIds)
const allVerbs = []
for (const verbs of results.values()) {
- allVerbs.push(...verbs)
+ allVerbs.push(...verbs)
}
```
-**Performance Gain:** 5-10x faster due to batched metadata fetches.
+**Performance Gain:** One batched metadata fetch instead of one query per source entity.
---
@@ -522,19 +340,16 @@ const results = await brain.batchGet(ids)
// ❌ BAD: Individual gets in loop
for (const id of ids) {
- await brain.get(id)
+ await brain.get(id)
}
```
### 2. **Batch Size Recommendations**
| Storage | Optimal Batch Size | Max Batch Size |
-|---------|-------------------|----------------|
+|---------|--------------------|----------------|
| **Memory** | Unlimited | Unlimited |
-| **FileSystem** | 100-500 | 1000 |
-| **GCS** | 100-500 | 1000 |
-| **S3/R2** | 100-1000 | 1000 |
-| **Azure** | 100-500 | 1000 |
+| **Filesystem** | 100-500 | 1000 |
**Guideline:** For batches >1000, split into chunks of 500-1000.
@@ -556,13 +371,13 @@ const results = await brain.batchGet(ids)
// Check results
for (const id of ids) {
- if (results.has(id)) {
- // Entity exists
- const entity = results.get(id)
- } else {
- // Entity missing (not an error)
- console.log(`Entity ${id} not found`)
- }
+ if (results.has(id)) {
+ // Entity exists
+ const entity = results.get(id)
+ } else {
+ // Entity missing (not an error)
+ console.log(`Entity ${id} not found`)
+ }
}
```
@@ -597,73 +412,47 @@ npx vitest run tests/integration/storage-batch-operations.test.ts
```
User Code (brain.batchGet)
- ↓
+ ↓
High-Level API (src/brainy.ts)
- ↓
+ ↓
Storage Layer (src/storage/baseStorage.ts)
- ↓
-COW Layer (readBatchWithInheritance)
- ↓
+ ↓
Adapter Layer (readBatchFromAdapter)
- ↓
-Cloud Adapter (GCS/S3/Azure native batch APIs)
+ ↓
+Storage Adapter (FileSystemStorage / MemoryStorage)
```
-### Automatic Fallback
+### Parallel Reads
-If an adapter doesn't implement `readBatch()`, the system automatically falls back to parallel individual reads:
+Both shipped adapters fall back to `Promise.all` over individual reads:
```typescript
// BaseStorage.readBatchFromAdapter()
-if (typeof selfWithBatch.readBatch === 'function') {
- // Use native batch API
- return await selfWithBatch.readBatch(resolvedPaths)
-} else {
- // Automatic parallel fallback
- return await Promise.all(resolvedPaths.map(path => this.read(path)))
-}
+return await Promise.all(resolvedPaths.map(path => this.read(path)))
```
-**Adapters with Native Batch:**
-- ✅ GCSStorage
-- ✅ S3CompatibleStorage
-- ✅ R2Storage
-- ✅ AzureBlobStorage
-
-**Adapters with Parallel Fallback:**
+**Shipped Adapters:**
- MemoryStorage
- FileSystemStorage
-- OPFSStorage
-- HistoricalStorageAdapter (delegates to underlying)
---
-## Release Notes
+## API Summary
-**Version:** 5.12.0
-**Release Date:** 2025-11-19
-**Status:** Production-Ready
-
-**Breaking Changes:** None (backward compatible)
-
-**New APIs:**
- `brain.batchGet(ids, options?)` - High-level batch entity retrieval
- `storage.getNounMetadataBatch(ids)` - Storage-level metadata batch
- `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries
-- `storage.readBatchWithInheritance(paths, targetBranch?)` - COW-aware batch reads
**Performance Improvements:**
-- VFS operations: 90%+ faster on cloud storage
-- Entity retrieval: 10-20x throughput improvement
+- VFS operations: single batched pass instead of N sequential reads
+- Entity retrieval: N+1 reads collapsed into one batched pass
- Zero N+1 query patterns
**Compatibility:**
-- ✅ ID-first storage (v6.0.0+)
+- ✅ ID-first storage
- ✅ Sharding (256 shards)
-- ✅ COW (branch isolation, inheritance)
-- ✅ fork() and checkout()
-- ✅ asOf() time-travel
-- ✅ All 6 indexes respected (HNSW, TypeAwareHNSW, MetadataIndex, GraphAdjacency, Version, DeletedItems)
+- ✅ Generational MVCC — batch reads serve the live generation; pinned `Db` views serve the past
+- ✅ All indexes respected (vector, metadata, graph adjacency)
---
diff --git a/docs/CORE_API_PATTERNS.md b/docs/CORE_API_PATTERNS.md
deleted file mode 100644
index 717686c8..00000000
--- a/docs/CORE_API_PATTERNS.md
+++ /dev/null
@@ -1,643 +0,0 @@
-# 🧠 Core API Patterns: Modern Brainy v3.x
-
-> Learn the correct patterns for Brainy's core operations. Avoid v2.x confusion and use modern, efficient APIs.
-
-## 🚨 Critical: Use v3.x APIs Only
-
-### ❌ **WRONG - Deprecated v2.x APIs**
-
-```typescript
-// DON'T DO THIS - These methods don't exist in v3.x!
-await brain.addNoun(text, type, metadata) // ❌ Removed
-await brain.getNouns({ pagination }) // ❌ Removed
-await brain.addVerb(source, target, type) // ❌ Removed
-await brain.getVerbs() // ❌ Removed
-await brain.deleteNoun(id) // ❌ Removed
-await brain.deleteVerb(id) // ❌ Removed
-```
-
-### ✅ **CORRECT - Modern v3.x APIs**
-
-```typescript
-// ✅ Use these modern methods instead
-await brain.add({ data, type, metadata }) // Modern unified add
-await brain.find({ limit: 100 }) // Natural language search
-await brain.relate({ from, to, type }) // Clean relationship creation
-await brain.getRelations() // Modern relationship queries
-await brain.delete(id) // Unified deletion
-// Relationships auto-cascade when entities are deleted
-```
-
-## 📋 Entity Management Patterns
-
-### ❌ **WRONG - v2.x Style**
-
-```typescript
-// DON'T DO THIS - Old API patterns
-import { Brainy } from 'old-brainy' // ❌ Wrong import
-
-const brain = new Brainy({ // ❌ Old class name
- complexConfig: true
-})
-
-const id = await brain.addNoun( // ❌ Deprecated method
- "John Smith is a developer",
- "Person",
- { role: "engineer" }
-)
-```
-
-### ✅ **CORRECT - Modern Patterns**
-
-```typescript
-// ✅ Pattern 1: Basic entity creation
-import { Brainy, NounType } from '@soulcraft/brainy'
-
-const brain = new Brainy() // ✅ Zero config
-await brain.init()
-
-const id = await brain.add({
- data: "John Smith is a developer",
- type: NounType.Person,
- metadata: { role: "engineer", team: "backend" }
-})
-
-// ✅ Pattern 2: Bulk entity creation
-const entities = [
- { data: "React framework", type: NounType.Technology },
- { data: "Vue.js framework", type: NounType.Technology },
- { data: "Angular framework", type: NounType.Technology }
-]
-
-const ids = await Promise.all(
- entities.map(entity => brain.add(entity))
-)
-
-// ✅ Pattern 3: Entity with pre-computed vector
-const customVector = await brain.embed("Custom text")
-const vectorId = await brain.add({
- data: "Optimized content",
- type: NounType.Document,
- vector: customVector, // Skip re-embedding
- metadata: { source: "api", optimized: true }
-})
-```
-
-## 🔍 Search & Discovery Patterns
-
-### ❌ **WRONG - Confusing Old Patterns**
-
-```typescript
-// DON'T DO THIS - Mixing old and new APIs
-const results1 = await brain.searchText("query") // ❌ Old method
-const results2 = await brain.getNouns({ filter }) // ❌ Doesn't exist
-const results3 = await brain.findSimilar(text) // ❌ Unclear naming
-```
-
-### ✅ **CORRECT - Clean Search Patterns**
-
-```typescript
-// ✅ Pattern 1: Natural language search
-const results = await brain.find("React developers working on authentication")
-
-// ✅ Pattern 2: Structured search with filters
-const filteredResults = await brain.find({
- like: "machine learning", // Vector similarity
- where: { // Metadata filtering
- type: NounType.Document,
- year: { $gte: 2020 },
- status: "published"
- },
- limit: 50,
- orderBy: 'relevance'
-})
-
-// ✅ Pattern 3: Similarity search
-const similarItems = await brain.similar({
- to: existingEntityId, // Find items similar to this
- threshold: 0.8, // Minimum similarity
- limit: 10,
- exclude: [existingEntityId] // Don't include the source
-})
-
-// ✅ Pattern 4: Advanced search with relationships
-const connectedResults = await brain.find({
- like: "frontend frameworks",
- connected: {
- to: reactId, // Connected to React
- via: "related-to", // Through this relationship
- depth: 2 // Up to 2 hops away
- }
-})
-```
-
-## 🔗 Relationship Patterns
-
-### ❌ **WRONG - Old Relationship APIs**
-
-```typescript
-// DON'T DO THIS - Old relationship patterns
-await brain.addVerb(sourceId, targetId, "uses", { strength: 0.9 }) // ❌ Old API
-const verbs = await brain.getVerbsBySource(sourceId) // ❌ Removed
-await brain.deleteVerb(verbId) // ❌ Old pattern
-```
-
-### ✅ **CORRECT - Modern Relationship Management**
-
-```typescript
-// ✅ Pattern 1: Create relationships
-const relationId = await brain.relate({
- from: developerId,
- to: frameworkId,
- type: VerbType.Uses,
- metadata: {
- since: "2023-01-01",
- proficiency: "expert",
- hours_per_week: 40
- }
-})
-
-// ✅ Pattern 2: Query relationships
-const relationships = await brain.getRelations({
- from: developerId, // Relationships from this entity
- type: VerbType.Uses, // Of this type
- limit: 100
-})
-
-// ✅ Pattern 3: Bidirectional relationships
-await brain.relate({
- from: projectId,
- to: developerId,
- type: VerbType.AssignedTo,
- bidirectional: true, // Creates reverse relationship
- metadata: { role: "lead", start_date: "2024-01-01" }
-})
-
-// ✅ Pattern 4: Relationship-based discovery
-const collaborators = await brain.find({
- connected: {
- to: currentProjectId,
- via: VerbType.WorksOn,
- direction: "incoming" // Who works on this project
- }
-})
-```
-
-## 🗃️ Data Retrieval Patterns
-
-### ❌ **WRONG - Inefficient Patterns**
-
-```typescript
-// DON'T DO THIS - Loading everything
-const everything = await brain.getNouns({ limit: 1000000 }) // ❌ Crashes
-const allData = await brain.exportAll() // ❌ Memory explosion
-```
-
-### ✅ **CORRECT - Efficient Data Access**
-
-```typescript
-// ✅ Pattern 1: Paginated retrieval
-async function getAllEntitiesPaginated() {
- const pageSize = 100
- let offset = 0
- let allEntities = []
-
- while (true) {
- const page = await brain.find({
- limit: pageSize,
- offset: offset
- })
-
- if (page.length === 0) break
-
- allEntities.push(...page)
- offset += pageSize
-
- // Optional: Progress reporting
- console.log(`Loaded ${allEntities.length} entities...`)
- }
-
- return allEntities
-}
-
-// ✅ Pattern 2: Streaming large datasets
-async function* streamEntities() {
- const pageSize = 50
- let offset = 0
-
- while (true) {
- const page = await brain.find({
- limit: pageSize,
- offset: offset
- })
-
- if (page.length === 0) break
-
- for (const entity of page) {
- yield entity
- }
-
- offset += pageSize
- }
-}
-
-// Usage
-for await (const entity of streamEntities()) {
- await processEntity(entity)
-}
-
-// ✅ Pattern 3: Specific entity retrieval
-const entity = await brain.get(entityId)
-if (entity) {
- console.log('Entity data:', entity.data)
- console.log('Metadata:', entity.metadata)
-} else {
- console.log('Entity not found')
-}
-```
-
-## 🔄 Update & Delete Patterns
-
-### ❌ **WRONG - Manual Update Patterns**
-
-```typescript
-// DON'T DO THIS - Recreating entities
-await brain.delete(oldId)
-const newId = await brain.add(updatedData) // ❌ Loses relationships
-```
-
-### ✅ **CORRECT - Update Operations**
-
-```typescript
-// ✅ Pattern 1: Update entity data
-await brain.update(entityId, {
- data: "Updated content here",
- metadata: {
- lastModified: Date.now(),
- version: "2.0"
- }
-})
-
-// ✅ Pattern 2: Partial metadata updates
-await brain.updateMetadata(entityId, {
- status: "published",
- tags: ["important", "featured"]
- // Merges with existing metadata
-})
-
-// ✅ Pattern 3: Safe deletion with cascade options
-await brain.delete(entityId, {
- cascade: true, // Delete related relationships
- backup: true // Create backup before deletion
-})
-
-// ✅ Pattern 4: Bulk operations
-const updateOperations = entities.map(entity => ({
- id: entity.id,
- changes: { status: "processed" }
-}))
-
-await brain.updateMany(updateOperations)
-```
-
-## 🧮 Vector & Embedding Patterns
-
-### ❌ **WRONG - Manual Vector Handling**
-
-```typescript
-// DON'T DO THIS - Manual embedding without understanding
-const vector = await brain.embed(text)
-// Store vector somewhere manually // ❌ Missing integration
-```
-
-### ✅ **CORRECT - Smart Vector Operations**
-
-```typescript
-// ✅ Pattern 1: Automatic embedding (recommended)
-const id = await brain.add({
- data: "Content to be embedded",
- type: NounType.Document
- // Vector computed automatically
-})
-
-// ✅ Pattern 2: Pre-computed vectors for optimization
-const texts = ["Text 1", "Text 2", "Text 3"]
-const vectors = await Promise.all(
- texts.map(text => brain.embed(text))
-)
-
-const entities = await Promise.all(
- texts.map((text, i) => brain.add({
- data: text,
- type: NounType.Document,
- vector: vectors[i] // Skip re-embedding
- }))
-)
-
-// ✅ Pattern 3: Vector similarity search
-const queryVector = await brain.embed("search query")
-const similar = await brain.similar({
- vector: queryVector, // Use vector directly
- threshold: 0.75,
- limit: 20
-})
-
-// ✅ Pattern 4: Compare vectors directly
-const vector1 = await brain.embed("First text")
-const vector2 = await brain.embed("Second text")
-const similarity = brain.computeSimilarity(vector1, vector2)
-console.log(`Similarity: ${similarity}`)
-```
-
-## 🏗️ Configuration Patterns
-
-### ❌ **WRONG - Over-Configuration**
-
-```typescript
-// DON'T DO THIS - Complex configurations that break
-const brain = new Brainy({
- storage: {
- type: 'complex',
- options: {
- nested: {
- configuration: true,
- that: "breaks"
- }
- }
- },
- embedding: {
- customModel: "broken-model",
- dimensions: 999999
- }
-})
-```
-
-### ✅ **CORRECT - Smart Configuration**
-
-```typescript
-// ✅ Pattern 1: Zero configuration (recommended)
-const brain = new Brainy() // Auto-detects everything
-await brain.init()
-
-// ✅ Pattern 2: Simple storage selection
-const fsBrain = new Brainy({
- storage: { type: 'filesystem', path: './data' }
-})
-
-const cloudBrain = new Brainy({
- storage: { type: 's3', bucket: 'my-data' }
-})
-
-// ✅ Pattern 3: Production configuration
-const prodBrain = new Brainy({
- storage: {
- type: 's3',
- bucket: process.env.BRAINY_BUCKET,
- region: process.env.AWS_REGION
- },
- silent: true, // No console output
- distributed: true, // Enable clustering
- cache: { maxSize: 10000 } // Larger cache
-})
-
-// ✅ Pattern 4: Development vs production
-const isDev = process.env.NODE_ENV === 'development'
-
-const brain = new Brainy({
- storage: isDev
- ? { type: 'memory' } // Fast for dev
- : { type: 'filesystem', path: './brainy-data' }, // Persistent for prod
- silent: !isDev, // Verbose in dev, quiet in prod
- cache: { maxSize: isDev ? 100 : 5000 }
-})
-```
-
-## 🔄 Migration from v2.x
-
-### ✅ **Migration Patterns**
-
-```typescript
-// If you have old v2.x code, here's how to migrate:
-
-// OLD v2.x:
-// await brain.addNoun(text, type, metadata)
-// NEW v3.x:
-await brain.add({ data: text, type, metadata })
-
-// OLD v2.x:
-// await brain.getNouns({ pagination: { limit: 100 } })
-// NEW v3.x:
-await brain.find({ limit: 100 })
-
-// OLD v2.x:
-// await brain.addVerb(sourceId, targetId, verbType, metadata)
-// NEW v3.x:
-await brain.relate({ from: sourceId, to: targetId, type: verbType, metadata })
-
-// OLD v2.x:
-// await brain.searchText(query)
-// NEW v3.x:
-await brain.find(query) // More powerful natural language search
-```
-
-## 🚀 Performance Patterns
-
-### ✅ **High-Performance Patterns**
-
-```typescript
-// ✅ Pattern 1: Batch operations
-const entities = [/* large array */]
-const batchSize = 100
-
-for (let i = 0; i < entities.length; i += batchSize) {
- const batch = entities.slice(i, i + batchSize)
- await Promise.all(
- batch.map(entity => brain.add(entity))
- )
-
- // Optional: Rate limiting
- await new Promise(resolve => setTimeout(resolve, 100))
-}
-
-// ✅ Pattern 2: Connection pooling for distributed
-const brain = new Brainy({
- distributed: true,
- connectionPool: {
- min: 5,
- max: 50,
- acquireTimeoutMillis: 30000
- }
-})
-
-// ✅ Pattern 3: Efficient caching
-const brain = new Brainy({
- cache: {
- maxSize: 10000, // Number of items
- ttl: 300000, // 5 minutes
- updateAgeOnGet: true // LRU behavior
- }
-})
-
-// ✅ Pattern 4: Memory-conscious operations
-const results = await brain.find({
- query: "large dataset query",
- limit: 1000, // Reasonable limit
- includeVectors: false // Exclude vectors if not needed
-})
-```
-
-## 🛡️ Error Handling Patterns
-
-### ✅ **Robust Error Handling**
-
-```typescript
-// ✅ Pattern 1: Specific error handling
-try {
- const result = await brain.add({ data, type, metadata })
- return result
-} catch (error) {
- if (error.code === 'DUPLICATE_ENTITY') {
- console.log('Entity already exists, updating instead...')
- return await brain.update(error.existingId, { data, metadata })
- } else if (error.code === 'STORAGE_FULL') {
- throw new Error('Storage capacity exceeded')
- } else if (error.code === 'EMBEDDING_FAILED') {
- console.warn('Embedding failed, retrying with simpler text...')
- return await brain.add({
- data: data.substring(0, 1000), // Truncate
- type,
- metadata
- })
- }
- throw error
-}
-
-// ✅ Pattern 2: Retry with exponential backoff
-async function resilientAdd(data: any, maxRetries = 3) {
- for (let attempt = 1; attempt <= maxRetries; attempt++) {
- try {
- return await brain.add(data)
- } catch (error) {
- if (attempt === maxRetries) throw error
-
- const delay = Math.pow(2, attempt) * 1000
- console.warn(`Attempt ${attempt} failed, retrying in ${delay}ms...`)
- await new Promise(resolve => setTimeout(resolve, delay))
- }
- }
-}
-
-// ✅ Pattern 3: Graceful degradation
-async function robustSearch(query: string) {
- try {
- // Try advanced semantic search first
- return await brain.find({
- like: query,
- threshold: 0.8,
- limit: 50
- })
- } catch (error) {
- console.warn('Semantic search failed, falling back to basic search:', error.message)
-
- try {
- // Fallback to simple text search
- return await brain.find(query)
- } catch (fallbackError) {
- console.error('All search methods failed:', fallbackError.message)
- return [] // Return empty results rather than crash
- }
- }
-}
-```
-
-## 📊 Monitoring Patterns
-
-### ✅ **Production Monitoring**
-
-```typescript
-// ✅ Pattern 1: Performance monitoring
-const startTime = Date.now()
-const result = await brain.add(data)
-const duration = Date.now() - startTime
-
-if (duration > 1000) {
- console.warn(`Slow add operation: ${duration}ms`)
-}
-
-// ✅ Pattern 2: Health checks
-async function healthCheck() {
- try {
- // Test basic operations
- const testId = await brain.add({
- data: "health check",
- type: NounType.System,
- metadata: { test: true }
- })
-
- await brain.get(testId)
- await brain.delete(testId)
-
- return { status: 'healthy', timestamp: Date.now() }
- } catch (error) {
- return {
- status: 'unhealthy',
- error: error.message,
- timestamp: Date.now()
- }
- }
-}
-
-// ✅ Pattern 3: Metrics collection
-class BrainyMetrics {
- private metrics = {
- operations: 0,
- errors: 0,
- totalTime: 0
- }
-
- async timedOperation(operation: () => Promise): Promise {
- const start = Date.now()
- try {
- const result = await operation()
- this.metrics.operations++
- this.metrics.totalTime += Date.now() - start
- return result
- } catch (error) {
- this.metrics.errors++
- throw error
- }
- }
-
- getStats() {
- return {
- ...this.metrics,
- avgTime: this.metrics.totalTime / this.metrics.operations || 0,
- errorRate: this.metrics.errors / this.metrics.operations || 0
- }
- }
-}
-```
-
-## 🎯 Summary: Modern Brainy v3.x Best Practices
-
-| ❌ **Avoid v2.x** | ✅ **Use v3.x** |
-|------------------|----------------|
-| `addNoun()` | `add()` |
-| `getNouns()` | `find()` |
-| `addVerb()` | `relate()` |
-| `getVerbs()` | `getRelations()` |
-| `deleteNoun()` | `delete()` |
-| Complex configs | Zero-config with `new Brainy()` |
-| Manual pagination | Built-in smart pagination |
-| String-based search | Natural language queries |
-
----
-
-**🎉 Following these patterns gives you:**
-- 🚀 **Modern APIs** that are actively maintained
-- ⚡ **Better performance** with intelligent defaults
-- 🛡️ **Robust error handling** with specific error types
-- 📈 **Scalable patterns** for production applications
-- 🧠 **Natural language** search capabilities
-
-**Next:** [Neural API Patterns →](./NEURAL_API_PATTERNS.md) | [VFS Patterns →](./vfs/COMMON_PATTERNS.md)
\ No newline at end of file
diff --git a/docs/CREATING-AUGMENTATIONS.md b/docs/CREATING-AUGMENTATIONS.md
deleted file mode 100644
index 1c7fd04a..00000000
--- a/docs/CREATING-AUGMENTATIONS.md
+++ /dev/null
@@ -1,435 +0,0 @@
-# Creating Augmentations for Brainy
-
-> **Updated for v4.0.0** - Includes metadata structure changes and type system improvements
-
-## The BrainyAugmentation Interface
-
-Every augmentation implements this simple yet powerful interface:
-
-```typescript
-interface BrainyAugmentation {
- // Identification
- name: string // Unique name for your augmentation
-
- // Execution control
- timing: 'before' | 'after' | 'around' | 'replace' // When to execute
- operations: string[] // Which operations to intercept
- priority: number // Execution order (higher = first)
-
- // Lifecycle methods
- initialize(context: AugmentationContext): Promise
- execute(operation: string, params: any, next: () => Promise): Promise
- shutdown?(): Promise // Optional cleanup
-}
-```
-
-## v4.0.0 Breaking Changes for Augmentation Developers
-
-### 1. Metadata Structure Separation
-v4.0.0 introduces strict metadata/vector separation for billion-scale performance:
-
-```typescript
-// ✅ v4.0.0: Metadata has required type field
-interface NounMetadata {
- noun: NounType // Required! Must be a valid noun type
- [key: string]: any // Your custom metadata
-}
-
-interface VerbMetadata {
- verb: VerbType // Required! Must be a valid verb type
- sourceId: string
- targetId: string
- [key: string]: any
-}
-```
-
-### 2. Storage Adapter Return Types
-Storage adapters now return different types at different boundaries:
-
-```typescript
-// Internal methods: Pure structures (no metadata)
-abstract _getNoun(id: string): Promise
-
-// Public API: WithMetadata structures
-abstract getNoun(id: string): Promise
-```
-
-### 3. Verb Property Renamed
-The verb relationship field changed from `type` to `verb`:
-
-```typescript
-// ❌ v3.x
-verb.type === 'relatedTo'
-
-// ✅ v4.0.0
-verb.verb === 'relatedTo'
-```
-
-## Creating a Storage Augmentation
-
-Storage augmentations are special - they provide the storage backend for Brainy.
-
-### Important: v4.0.0 Storage Requirements
-
-Your storage adapter MUST:
-1. **Wrap metadata** with required `noun`/`verb` fields
-2. **Return pure structures** from internal `_methods`
-3. **Return WithMetadata types** from public methods
-
-```typescript
-import { StorageAugmentation } from 'brainy/augmentations'
-import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy'
-
-export class MyCustomStorage extends BaseStorageAdapter {
- // Internal method: Returns pure structure
- async _getNoun(id: string): Promise {
- const data = await this.fetchFromDatabase(id)
- return data ? {
- id: data.id,
- vector: data.vector,
- nounType: data.type
- } : null
- }
-
- // Public method: Returns WithMetadata structure
- async getNoun(id: string): Promise {
- const noun = await this._getNoun(id)
- if (!noun) return null
-
- // Fetch metadata separately (v4.0.0 pattern)
- const metadata = await this.getNounMetadata(id)
-
- return {
- ...noun,
- metadata: metadata || { noun: noun.nounType || 'thing' }
- }
- }
-
- // CRITICAL: Always save with proper metadata structure
- async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise {
- // Validate metadata has required 'noun' field
- if (!metadata?.noun) {
- throw new Error('v4.0.0: NounMetadata requires "noun" field')
- }
-
- await this.database.save({
- id: noun.id,
- vector: noun.vector,
- nounType: noun.nounType,
- metadata: metadata // Stored separately in v4.0.0
- })
- }
-}
-
-export class MyStorageAugmentation extends StorageAugmentation {
- private config: MyStorageConfig
-
- constructor(config: MyStorageConfig) {
- super()
- this.name = 'my-custom-storage'
- this.config = config
- }
-
- // Called during storage resolution phase
- async provideStorage(): Promise {
- const storage = new MyCustomStorage(this.config)
- this.storageAdapter = storage
- return storage
- }
-
- // Called during augmentation initialization
- protected async onInitialize(): Promise {
- await this.storageAdapter!.init()
- this.log(`Custom storage initialized`)
- }
-}
-```
-
-### Using Your Storage Augmentation
-
-```typescript
-// Register before brain.init()
-const brain = new Brainy()
-brain.augmentations.register(new MyStorageAugmentation({
- connectionString: 'redis://localhost:6379'
-}))
-await brain.init() // Will use your storage!
-```
-
-## Creating a Feature Augmentation
-
-Here's a complete example of a caching augmentation:
-
-```typescript
-import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations'
-
-export class CachingAugmentation extends BaseAugmentation {
- private cache = new Map()
-
- constructor() {
- super()
- this.name = 'smart-cache'
- this.timing = 'around' // Wrap operations
- this.operations = ['search'] // Only cache searches
- this.priority = 50 // Mid-priority
- }
-
- async execute(operation: string, params: any, next: () => Promise): Promise {
- if (operation === 'search') {
- // Check cache
- const cacheKey = JSON.stringify(params)
- if (this.cache.has(cacheKey)) {
- this.log('Cache hit!')
- return this.cache.get(cacheKey)
- }
-
- // Execute and cache
- const result = await next()
- this.cache.set(cacheKey, result)
- return result
- }
-
- // Pass through other operations
- return next()
- }
-
- protected async onInitialize(): Promise {
- this.log('Cache initialized')
- }
-
- async shutdown(): Promise {
- this.cache.clear()
- await super.shutdown()
- }
-}
-```
-
-## The Four Timing Modes
-
-### 1. `before` - Pre-processing
-```typescript
-timing = 'before'
-async execute(op, params, next) {
- // Validate/transform input
- const validated = await validate(params)
- return next(validated) // Pass modified params
-}
-```
-
-### 2. `after` - Post-processing
-```typescript
-timing = 'after'
-async execute(op, params, next) {
- const result = await next()
- // Log, analyze, or modify result
- console.log(`Operation ${op} returned:`, result)
- return result
-}
-```
-
-### 3. `around` - Wrapping (middleware)
-```typescript
-timing = 'around'
-async execute(op, params, next) {
- console.log('Starting', op)
- try {
- const result = await next()
- console.log('Success', op)
- return result
- } catch (error) {
- console.log('Failed', op, error)
- throw error
- }
-}
-```
-
-### 4. `replace` - Complete replacement
-```typescript
-timing = 'replace'
-async execute(op, params, next) {
- // Don't call next() - replace entirely!
- return myCustomImplementation(params)
-}
-```
-
-## Operations You Can Intercept
-
-Common operations in Brainy:
-- `'storage'` - Storage resolution (special)
-- `'add'` - Adding data
-- `'search'`, `'similar'` - Searching
-- `'update'`, `'delete'` - Modifications
-- `'saveNoun'`, `'saveVerb'` - Storage operations
-- `'all'` - Intercept everything
-
-## Context Available to Augmentations
-
-```typescript
-interface AugmentationContext {
- brain: Brainy // The brain instance
- storage: StorageAdapter // Storage backend
- config: BrainyConfig // Configuration
- log: (message: string, level?: 'info' | 'warn' | 'error') => void
-}
-```
-
-## Real-World Examples
-
-### 1. Redis Storage Augmentation
-```typescript
-export class RedisStorageAugmentation extends StorageAugmentation {
- async provideStorage(): Promise {
- return new RedisAdapter({
- host: 'localhost',
- port: 6379,
- // Implement full StorageAdapter interface
- })
- }
-}
-```
-
-### 2. Audit Trail Augmentation
-```typescript
-export class AuditAugmentation extends BaseAugmentation {
- timing = 'after'
- operations = ['add', 'update', 'delete']
-
- async execute(op, params, next) {
- const result = await next()
-
- // Log to audit trail
- await this.logAudit({
- operation: op,
- params,
- result,
- timestamp: new Date(),
- user: this.context.config.currentUser
- })
-
- return result
- }
-}
-```
-
-### 3. Rate Limiting Augmentation
-```typescript
-export class RateLimitAugmentation extends BaseAugmentation {
- timing = 'before'
- operations = ['search']
- private limiter = new RateLimiter({ rps: 100 })
-
- async execute(op, params, next) {
- await this.limiter.acquire() // Wait if rate limited
- return next()
- }
-}
-```
-
-## Publishing to Brain Cloud Marketplace
-
-Future capability for premium augmentations:
-
-```typescript
-// package.json
-{
- "name": "@brain-cloud/redis-storage",
- "brainy": {
- "type": "augmentation",
- "category": "storage",
- "premium": true
- }
-}
-
-// Users can install via:
-// brainy augment install redis-storage
-```
-
-## Best Practices
-
-### General Practices
-
-1. **Use BaseAugmentation** - Provides common functionality
-2. **Set appropriate priority** - Storage (100), System (80-99), Features (10-50)
-3. **Be selective with operations** - Don't use 'all' unless necessary
-4. **Handle errors gracefully** - Don't break the chain
-5. **Clean up in shutdown()** - Release resources
-6. **Log appropriately** - Use context.log() for consistent output
-7. **Document your augmentation** - Include examples
-
-### v4.0.0 Specific Best Practices
-
-8. **Always include `noun` field** when creating/modifying NounMetadata:
- ```typescript
- const metadata: NounMetadata = {
- noun: 'thing', // REQUIRED!
- yourField: 'value'
- }
- ```
-
-9. **Use `verb` property** not `type` when working with relationships:
- ```typescript
- // ✅ Correct
- if (verb.verb === 'relatedTo') { ... }
-
- // ❌ Wrong (v3.x pattern)
- if (verb.type === 'relatedTo') { ... }
- ```
-
-10. **Access metadata correctly** from storage:
- ```typescript
- // ✅ Correct - metadata is already structured
- const nounType = noun.metadata.noun
-
- // ⚠️ Fallback pattern for robustness
- const nounType = noun.metadata?.noun || 'thing'
- ```
-
-11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations:
- ```typescript
- // ✅ Good - Separate concerns
- await storage.saveNoun(noun)
- await storage.saveMetadata(noun.id, metadata)
-
- // ❌ Bad - Mixing concerns
- await storage.saveNounWithEverything(combinedData)
- ```
-
-## Testing Your Augmentation
-
-```typescript
-import { Brainy } from 'brainy'
-import { MyAugmentation } from './my-augmentation'
-
-describe('MyAugmentation', () => {
- let brain: Brainy
-
- beforeEach(async () => {
- brain = new Brainy()
- brain.augmentations.register(new MyAugmentation())
- await brain.init()
- })
-
- afterEach(async () => {
- await brain.destroy()
- })
-
- it('should enhance searches', async () => {
- // Test your augmentation's effect
- const results = await brain.search('test')
- expect(results).toHaveProperty('enhanced', true)
- })
-})
-```
-
-## Summary
-
-Augmentations are Brainy's extension system. They can:
-- Replace storage backends
-- Add caching layers
-- Implement audit trails
-- Add rate limiting
-- Sync with external systems
-- Transform data
-- And much more!
-
-The unified BrainyAugmentation interface makes it easy to create powerful extensions while maintaining consistency across the entire system.
\ No newline at end of file
diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md
new file mode 100644
index 00000000..aa3cd351
--- /dev/null
+++ b/docs/DATA_MODEL.md
@@ -0,0 +1,271 @@
+# Data Model
+
+> How Brainy stores entities and relationships, and the critical distinction between `data` and `metadata`.
+
+---
+
+## Entity (Noun)
+
+An entity is the fundamental data unit in Brainy. Every entity has:
+
+| Field | Type | Indexed | Description |
+|-------|------|---------|-------------|
+| `id` | `string` | Primary key | UUID v4 (auto-generated or custom) |
+| `data` | `any` | **HNSW vector index** | Content used for semantic/hybrid search. Strings auto-embed. |
+| `metadata` | `object` | **MetadataIndex** | Structured queryable fields (tags, dates, flags, etc.) |
+| `type` | `NounType` | MetadataIndex (as `noun`) | Entity type classification |
+| `vector` | `number[]` | HNSW | 384-dim embedding (auto-computed from `data` or user-provided) |
+| `confidence` | `number` | MetadataIndex | Type classification confidence (0-1) |
+| `weight` | `number` | MetadataIndex | Entity importance/salience (0-1) |
+| `service` | `string` | MetadataIndex | Multi-tenancy identifier |
+| `createdAt` | `number` | MetadataIndex | Creation timestamp (ms since epoch) |
+| `updatedAt` | `number` | MetadataIndex | Last update timestamp (ms since epoch) |
+| `createdBy` | `object` | MetadataIndex | Source augmentation info |
+
+### Example
+
+```typescript
+const id = await brain.add({
+ data: 'John Smith is a software engineer at Acme Corp', // → embedded into vector
+ type: NounType.Person,
+ metadata: { // → indexed, queryable via where filters
+ role: 'engineer',
+ department: 'backend',
+ yearsExperience: 8
+ },
+ confidence: 0.95,
+ weight: 0.7
+})
+```
+
+---
+
+## Relationship (Verb)
+
+A relationship is a typed, directed edge connecting two entities.
+
+| Field | Type | Indexed | Description |
+|-------|------|---------|-------------|
+| `id` | `string` | Primary key | UUID v4 (auto-generated) |
+| `from` | `string` | **GraphAdjacencyIndex** | Source entity ID |
+| `to` | `string` | **GraphAdjacencyIndex** | Target entity ID |
+| `type` | `VerbType` | GraphAdjacencyIndex (as `verb`) | Relationship type classification |
+| `data` | `any` | — | Opaque content (overrides auto-computed vector if provided) |
+| `metadata` | `object` | — | Structured fields on the edge |
+| `weight` | `number` | — | Connection strength (0-1, default: 1.0) |
+| `confidence` | `number` | — | Relationship certainty (0-1) |
+| `evidence` | `RelationEvidence` | — | Why this relationship was detected |
+| `createdAt` | `number` | — | Creation timestamp (ms since epoch) |
+| `updatedAt` | `number` | — | Last update timestamp (ms since epoch) |
+| `service` | `string` | — | Multi-tenancy identifier |
+
+### Example
+
+```typescript
+const relId = await brain.relate({
+ from: personId,
+ to: projectId,
+ type: VerbType.WorksOn,
+ data: 'Lead engineer on the AI module', // Optional: content for this edge
+ metadata: { // Optional: queryable edge fields
+ role: 'lead',
+ startDate: '2024-01-15'
+ },
+ weight: 0.9
+})
+```
+
+---
+
+## Data vs Metadata
+
+This is the most important concept in Brainy's storage model:
+
+### `data` — Content for Semantic Search
+
+- Embedded into a 384-dimensional vector via the WASM embedding engine
+- Searchable via **semantic similarity** (HNSW vector index) and **hybrid text+semantic** search
+- Queried by passing `query` to `find()`:
+ ```typescript
+ brain.find({ query: 'machine learning algorithms' })
+ ```
+- **NOT** indexed by MetadataIndex — you cannot use `where` filters on `data`
+- Stored opaquely: strings, objects, numbers — anything goes
+
+### `metadata` — Structured Queryable Fields
+
+- Indexed by MetadataIndex with O(1) lookups per field
+- Queryable via `where` filters using [BFO operators](./QUERY_OPERATORS.md):
+ ```typescript
+ brain.find({
+ where: {
+ department: 'engineering',
+ yearsExperience: { greaterThan: 5 },
+ tags: { contains: 'senior' }
+ }
+ })
+ ```
+- **NOT** used for vector/semantic search
+- Must be a flat or lightly nested object
+
+### Quick Reference
+
+| | `data` | `metadata` |
+|---|---|---|
+| **Purpose** | Content for embedding / semantic search | Structured fields for filtering |
+| **Searched by** | `find({ query })` — vector similarity, hybrid text+semantic | `find({ where })` — exact, range, set operators |
+| **Indexed by** | HNSW vector index | MetadataIndex |
+| **Queryable with operators?** | No | Yes (`equals`, `greaterThan`, `oneOf`, etc.) |
+| **Auto-embedded?** | Yes (strings → 384-dim vectors) | No |
+| **Typical content** | Text descriptions, document content | Tags, dates, status flags, categories, numeric fields |
+
+### Common Pattern
+
+```typescript
+// Add an article
+await brain.add({
+ data: 'A deep dive into transformer architectures and attention mechanisms',
+ type: NounType.Document,
+ metadata: {
+ title: 'Transformer Deep Dive',
+ author: 'Dr. Chen',
+ publishedYear: 2024,
+ tags: ['AI', 'transformers', 'NLP'],
+ status: 'published'
+ }
+})
+
+// Search by content (semantic — searches data)
+const results = await brain.find({ query: 'neural network attention' })
+
+// Filter by fields (exact — queries metadata)
+const recent = await brain.find({
+ where: {
+ publishedYear: { greaterThan: 2023 },
+ status: 'published'
+ }
+})
+
+// Combine both (Triple Intelligence)
+const precise = await brain.find({
+ query: 'attention mechanisms', // Semantic search on data
+ where: { author: 'Dr. Chen' }, // Metadata filter
+ connected: { from: authorId, depth: 1 } // Graph traversal
+})
+```
+
+---
+
+## Storage Field Naming
+
+Internally, Brainy uses different field names in storage vs the public API:
+
+| Public API (Entity/Relation) | Storage (metadata object) | Notes |
+|------------------------------|--------------------------|-------|
+| `type` | `noun` | Entity type stored as `noun` |
+| `from` | `sourceId` | Relationship source |
+| `to` | `targetId` | Relationship target |
+| `type` (on Relation) | `verb` | Relationship type stored as `verb` |
+
+When querying with `find()`, you can use:
+- `type` parameter (convenience alias, equivalent to `where.noun`)
+- `where.noun` directly
+
+```typescript
+// These are equivalent:
+brain.find({ type: NounType.Person })
+brain.find({ where: { noun: NounType.Person } })
+```
+
+---
+
+## Standard Metadata Fields
+
+When you add an entity, Brainy stores these standard fields in the metadata object alongside your custom fields:
+
+| Field | Set By | Description |
+|-------|--------|-------------|
+| `noun` | System | Entity type (NounType enum value) |
+| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. |
+| `data` | System | The raw `data` value (stored opaquely) |
+| `createdAt` | System | Creation timestamp |
+| `updatedAt` | System | Last update timestamp |
+| `confidence` | User | Type classification confidence |
+| `weight` | User | Entity importance |
+| `service` | User | Multi-tenancy identifier |
+| `createdBy` | User/System | Source augmentation |
+
+On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**.
+
+### Subtype — sub-classification within a NounType
+
+`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type:
+
+```typescript
+// A Person who is an employee:
+await brain.add({
+ data: 'Avery Brooks — runs the AI lab',
+ type: NounType.Person,
+ subtype: 'employee',
+ metadata: { department: 'ai-lab' }
+})
+
+// A Document that is an invoice:
+await brain.add({
+ data: 'INV-2026-001',
+ type: NounType.Document,
+ subtype: 'invoice',
+ metadata: { amount: 1500 }
+})
+```
+
+`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`.
+
+### Subtype — sub-classification within a VerbType (7.30+)
+
+Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive — a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field on `HNSWVerbWithMetadata` and on the public `Relation`:
+
+```typescript
+await brain.relate({
+ from: ceoId,
+ to: vpId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct', // top-level standard field
+ metadata: { since: '2025-Q1' } // user-custom fields stay in metadata
+})
+```
+
+Fast-path filter on the verb side:
+
+```typescript
+const direct = await brain.related({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct'
+})
+```
+
+The verb-side rollup at `_system/verb-subtype-statistics.json` mirrors the noun-side `_system/subtype-statistics.json` — same shape, same self-heal machinery. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`.
+
+Verbs and nouns now have full capability parity — every API on the noun side has a verb-side mirror, including the new `brain.updateRelation()` (which closed a pre-7.30 gap where relationships had no update path).
+
+### Standard verb fields
+
+The verb-side equivalent of `STANDARD_ENTITY_FIELDS` is `STANDARD_VERB_FIELDS`, exported from `src/coreTypes.ts`. Verb-specific standard fields:
+
+| Field | Description |
+|---|---|
+| `verb` | The VerbType enum value |
+| `sourceId` / `targetId` | The two endpoints of the relationship |
+| `subtype` | Sub-classification within the VerbType (7.30+) |
+| `confidence`, `weight`, `createdAt`, `updatedAt`, `service`, `createdBy`, `data` | Same semantics as the noun-side standard fields |
+
+The companion `resolveVerbField(verb, field)` helper resolves field paths the same way `resolveEntityField` does for nouns: standard fields first, metadata fallback for everything else.
+
+---
+
+## See Also
+
+- [API Reference](./api/README.md) — Complete API documentation
+- [Query Operators](./QUERY_OPERATORS.md) — All BFO operators with examples
+- [Find System](./FIND_SYSTEM.md) — Natural language find() details
diff --git a/docs/DEVELOPER_LEARNING_PATH.md b/docs/DEVELOPER_LEARNING_PATH.md
index 3436d922..4134ae63 100644
--- a/docs/DEVELOPER_LEARNING_PATH.md
+++ b/docs/DEVELOPER_LEARNING_PATH.md
@@ -193,21 +193,21 @@ console.log('✅ Created relationships')
console.log('\n🔍 Querying relationships...')
// Who works for TechCorp?
-const techcorpEmployees = await brain.getRelations({
+const techcorpEmployees = await brain.related({
to: techcorpId,
type: VerbType.WorksWith
})
console.log(`TechCorp has ${techcorpEmployees.length} employees`)
// Who works on the AI project?
-const projectContributors = await brain.getRelations({
+const projectContributors = await brain.related({
to: projectId,
type: [VerbType.WorksOn, VerbType.Manages]
})
console.log(`AI Project has ${projectContributors.length} contributors`)
// Who does John collaborate with?
-const johnsCollaborators = await brain.getRelations({
+const johnsCollaborators = await brain.related({
from: johnId,
type: VerbType.CollaboratesWith
})
@@ -294,7 +294,7 @@ await brain.update({
1. Create an organizational hierarchy (CEO → Managers → Engineers)
2. Build a project dependency graph
3. Model a social network with CollaboratesWith relationships
-4. Query "Who reports to Alice?" using getRelations()
+4. Query "Who reports to Alice?" using related()
5. Batch update all projects to add a "year: 2024" field
### Next Steps
@@ -429,23 +429,7 @@ fusionResults.forEach(r => {
}
})
-// 5. NEURAL API: Automatic clustering
-console.log('\n\n🤖 NEURAL API: Automatic Clustering')
-const neural = brain.neural()
-
-const clusters = await neural.clusters({
- maxClusters: 3,
- minClusterSize: 1
-})
-
-console.log(`Found ${clusters.length} semantic clusters:`)
-clusters.forEach((cluster, i) => {
- console.log(`\n Cluster ${i + 1}: ${cluster.label || cluster.id}`)
- console.log(` Members: ${cluster.members.length}`)
- console.log(` Centroid topics: ${cluster.metadata?.topics?.join(', ') || 'N/A'}`)
-})
-
-// 6. SIMILARITY: Find similar documents
+// 5. SIMILARITY: Find similar documents
console.log('\n\n🔍 SIMILARITY: Find Similar Documents')
const similarTo = await brain.similar({
to: paper1, // Entity ID of first AI paper
@@ -459,19 +443,6 @@ similarTo.forEach(r => {
console.log(` [${r.score.toFixed(3)}] ${r.entity.data?.substring(0, 50)}...`)
})
-// 7. OUTLIER DETECTION
-console.log('\n\n🚨 OUTLIER DETECTION')
-const outliers = await neural.outliers({
- method: 'statistical',
- threshold: 2.0 // 2 standard deviations
-})
-
-console.log(`Found ${outliers.length} outlier documents:`)
-outliers.forEach(o => {
- const entity = await brain.get(o.id)
- console.log(` [Anomaly score: ${o.score.toFixed(3)}] ${entity?.data?.substring(0, 50)}...`)
-})
-
await brain.close()
```
@@ -613,7 +584,7 @@ vfsFiles.forEach(f => {
})
// 3. VFS FILTERING IN KNOWLEDGE QUERIES
-console.log('\n🔍 Understanding VFS filtering (v4.4.0)...\n')
+console.log('\n🔍 Understanding VFS filtering...\n')
// Create some knowledge entities
const conceptId = await brain.add({
@@ -676,7 +647,7 @@ if (readmeEntity.length > 0) {
}
// Query relationships
-const conceptDocs = await brain.getRelations({
+const conceptDocs = await brain.related({
from: conceptId,
type: VerbType.DocumentedBy
})
@@ -758,7 +729,7 @@ await brain.close()
### Key Concepts
-#### 1. **VFS Filtering Architecture (v4.4.0)**
+#### 1. **VFS Filtering Architecture**
```typescript
// 🎯 DEFAULT BEHAVIOR: Clean Separation
@@ -798,7 +769,7 @@ await brain.relate({
})
// Query across boundaries
-const conceptDocs = await brain.getRelations({
+const conceptDocs = await brain.related({
from: conceptId,
type: VerbType.DocumentedBy
})
@@ -852,7 +823,7 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
## Level 5: Production Scale (90 minutes)
### What You'll Learn
-- Cloud storage (GCS, S3, R2)
+- Production filesystem storage and off-site backup
- Performance optimization
- Batch imports (CSV, Excel, PDF)
- Metadata query optimization
@@ -863,18 +834,13 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
-// 1. PRODUCTION STORAGE - Google Cloud Storage (Native SDK)
-console.log('☁️ Initializing production storage...\n')
+// 1. PRODUCTION STORAGE - Filesystem with off-site snapshots
+console.log('Initializing production storage...\n')
const brain = new Brainy({
storage: {
- type: 'gcs-native', // Native GCS SDK (recommended)
- gcsNativeStorage: {
- bucketName: 'my-brainy-production',
- // ADC (Application Default Credentials) - zero config in Cloud Run/GCE!
- // Or provide credentials:
- // keyFilename: '/path/to/service-account.json'
- }
+ type: 'filesystem',
+ path: '/var/lib/brainy'
},
// Performance tuning
@@ -888,7 +854,8 @@ const brain = new Brainy({
})
await brain.init()
-console.log('✅ Brainy initialized with GCS Native storage\n')
+console.log('Brainy initialized with filesystem storage')
+console.log('Snapshot /var/lib/brainy off-site via cron with `gsutil rsync` / `aws s3 sync` / `rclone`.\n')
// 2. BATCH IMPORT - CSV File
console.log('📊 Importing CSV data...\n')
@@ -967,26 +934,6 @@ console.log(` Failed: ${batchResult.failed.length}`)
console.log(` Duration: ${duration}ms`)
console.log(` Throughput: ${(batchResult.successful.length / (duration / 1000)).toFixed(0)} entities/sec`)
-// 5. ADVANCED CLUSTERING (Large Dataset)
-console.log('\n\n🤖 Clustering 1000 entities...\n')
-
-const neural = brain.neural()
-
-// Use fast clustering for large datasets
-const clusters = await neural.clusterFast({
- maxClusters: 5
-})
-
-console.log(`Found ${clusters.length} clusters:`)
-clusters.forEach((cluster, i) => {
- console.log(`\n Cluster ${i + 1}: ${cluster.label || cluster.id}`)
- console.log(` Size: ${cluster.members.length} members`)
- console.log(` Density: ${(cluster.density || 0).toFixed(3)}`)
- if (cluster.metadata?.keywords) {
- console.log(` Keywords: ${cluster.metadata.keywords.slice(0, 5).join(', ')}`)
- }
-})
-
// 6. PRODUCTION STATISTICS
console.log('\n\n📊 Production Statistics:\n')
@@ -1044,7 +991,7 @@ console.log('✅ Brain closed cleanly')
console.log('\n\n🎓 Production Deployment Complete!')
console.log('\n📚 Key Production Learnings:')
-console.log(' 1. Use native cloud storage (GCS, S3, R2) for persistence')
+console.log(' 1. Use filesystem storage and snapshot the data directory off-site from your scheduler')
console.log(' 2. Batch operations = 100x faster than individual ops')
console.log(' 3. Metadata query optimization for complex filters')
console.log(' 4. Monitor query performance with explain: true')
@@ -1057,41 +1004,22 @@ console.log(' 7. Stream large imports with progress callbacks')
#### 1. **Storage Options Comparison**
-| Storage | Use Case | Performance | Cost | Setup |
-|---------|----------|-------------|------|-------|
-| Memory | Dev/testing | Fastest | Free | Zero config |
-| Filesystem | Local prod | Fast | Free | Local path |
-| GCS Native | GCP prod | Fast | $$$ | Service account |
-| S3 | AWS prod | Fast | $$$ | Access keys |
-| R2 | Cloudflare | Fast | $ | Access keys |
+| Storage | Use Case | Performance | Setup |
+|---------|----------|-------------|-------|
+| Memory | Dev/testing | Fastest | Zero config |
+| Filesystem | Production | Fast | Local path + scheduled off-site backup |
-#### 2. **GCS Native vs S3-Compatible**
+#### 2. **Off-Site Backup**
-```typescript
-// ✅ RECOMMENDED: GCS Native SDK
-{
- storage: {
- type: 'gcs-native',
- gcsNativeStorage: {
- bucketName: 'my-bucket'
- // ADC handles auth automatically in Cloud Run/GCE!
- }
- }
-}
-
-// ⚠️ LEGACY: S3-compatible mode
-{
- storage: {
- type: 'gcs',
- gcsStorage: {
- bucketName: 'my-bucket',
- accessKeyId: process.env.GCS_ACCESS_KEY,
- secretAccessKey: process.env.GCS_SECRET_KEY
- }
- }
-}
+```bash
+# Cron / systemd timer / k8s CronJob — pick whatever you already operate
+*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
+*/15 * * * * aws s3 sync /var/lib/brainy s3://my-bucket/brainy-backup
+*/15 * * * * gsutil rsync -r /var/lib/brainy gs://my-bucket/brainy-backup
```
+Brainy itself never reaches out to an object store. Snapshot `path` from your scheduler.
+
#### 3. **Performance Optimization**
```typescript
@@ -1164,8 +1092,8 @@ const brain = new Brainy({ verbose: true })
#### Before Deployment
-- [ ] Choose cloud storage (GCS/S3/R2)
-- [ ] Set up authentication (service account/access keys)
+- [ ] Provision a writable `path` on the host
+- [ ] Set up an off-site snapshot job (`rclone` / `aws s3 sync` / `gsutil rsync`)
- [ ] Configure caching
- [ ] Test batch operations
- [ ] Benchmark query performance
@@ -1189,12 +1117,12 @@ const brain = new Brainy({ verbose: true })
### Practice Exercises
-1. Deploy Brainy with GCS Native storage
+1. Deploy Brainy with filesystem storage + scheduled off-site backup
2. Import a 10,000 row CSV file
3. Measure query performance for different filters
4. Optimize a slow query using getOptimalQueryPlan()
5. Set up monitoring dashboard
-6. Create backup/restore scripts
+6. Test restore from off-site snapshot
---
@@ -1212,7 +1140,7 @@ const brain = new Brainy({ verbose: true })
#### Advanced Topics
-- **Distributed Systems**: Sharding, replication, coordination
+- **Multi-instance Deployments**: Run multiple Brainy processes behind your own routing layer
- **Custom Augmentations**: Extend Brainy with plugins
- **Streaming Pipelines**: Real-time data ingestion
- **Security**: Encryption, access control, audit logs
@@ -1220,10 +1148,9 @@ const brain = new Brainy({ verbose: true })
#### Resources
-- 📚 [API Reference](../API_REFERENCE.md) - Complete API documentation
+- 📚 [API Reference](../api/README.md) - Complete API documentation
- 📁 [VFS Guide](../vfs/VFS_API_GUIDE.md) - Virtual Filesystem deep dive
- 🤖 [Neural API](../guides/neural-api.md) - Advanced neural operations
-- 🌐 [Distributed Guide](../guides/distributed-system.md) - Planet-scale architecture
- 💬 [Discord Community](https://discord.gg/brainy) - Get help, share projects
#### Share Your Success
diff --git a/docs/EXTENDING_STORAGE.md b/docs/EXTENDING_STORAGE.md
deleted file mode 100644
index 8ecc700a..00000000
--- a/docs/EXTENDING_STORAGE.md
+++ /dev/null
@@ -1,396 +0,0 @@
-# 🔌 Extending Brainy Storage with Augmentations
-
-## Overview
-
-Brainy's zero-config system is **fully extensible**. Augmentations can register new storage providers, presets, and auto-detection logic that integrates seamlessly with the existing system.
-
-## How Storage Extensions Work
-
-### 1. Storage Provider Registration
-
-When an augmentation is installed, it can register a new storage provider:
-
-```typescript
-import { StorageProvider, registerStorageAugmentation } from '@soulcraft/brainy/config'
-
-const redisProvider: StorageProvider = {
- type: 'redis',
- name: 'Redis Storage',
- description: 'High-performance in-memory data store',
- priority: 10, // Higher priority = checked first in auto-detection
-
- // Auto-detection logic
- async detect(): Promise {
- // Check if Redis is available
- if (process.env.REDIS_URL) {
- try {
- const redis = await import('ioredis')
- const client = new redis.default(process.env.REDIS_URL)
- await client.ping()
- await client.quit()
- return true
- } catch {
- return false
- }
- }
- return false
- },
-
- // Configuration builder
- async getConfig(): Promise {
- return {
- type: 'redis',
- redisStorage: {
- url: process.env.REDIS_URL,
- prefix: 'brainy:',
- ttl: 3600
- }
- }
- }
-}
-
-// Register the provider
-registerStorageAugmentation(redisProvider)
-```
-
-### 2. Using Extended Storage
-
-Once registered, the new storage type works with zero-config:
-
-```typescript
-// Auto-detection will now check Redis
-const brain = new Brainy() // Will use Redis if available!
-
-// Or explicitly specify
-const brain = new Brainy({ storage: 'redis' })
-
-// Or with custom config
-const brain = new Brainy({
- storage: {
- type: 'redis',
- redisStorage: {
- url: 'redis://localhost:6379',
- prefix: 'myapp:'
- }
- }
-})
-```
-
-## Real-World Examples
-
-### Redis Augmentation
-
-```typescript
-// @soulcraft/brainy-redis package
-export class RedisStorageAugmentation {
- async init() {
- // Register the storage provider
- registerStorageAugmentation({
- type: 'redis',
- name: 'Redis Storage',
- priority: 10,
-
- async detect() {
- return !!(process.env.REDIS_URL || process.env.REDIS_HOST)
- },
-
- async getConfig() {
- return {
- type: 'redis',
- redisStorage: {
- url: process.env.REDIS_URL ||
- `redis://${process.env.REDIS_HOST}:${process.env.REDIS_PORT || 6379}`
- }
- }
- }
- })
-
- // Register Redis-specific presets
- registerPresetAugmentation('redis-cache', {
- storage: 'redis',
- model: ModelPrecision.Q8,
- features: ['core', 'cache'],
- distributed: true,
- description: 'Redis-backed cache layer',
- category: PresetCategory.SERVICE
- })
- }
-}
-```
-
-### MongoDB Augmentation
-
-```typescript
-// @soulcraft/brainy-mongodb package
-export class MongoStorageAugmentation {
- async init() {
- registerStorageAugmentation({
- type: 'mongodb',
- name: 'MongoDB Storage',
- priority: 8,
-
- async detect() {
- return !!(process.env.MONGODB_URI || process.env.MONGO_URL)
- },
-
- async getConfig() {
- return {
- type: 'mongodb',
- mongoStorage: {
- uri: process.env.MONGODB_URI,
- database: 'brainy',
- collection: 'vectors'
- }
- }
- }
- })
- }
-}
-```
-
-### PostgreSQL + pgvector Augmentation
-
-```typescript
-// @soulcraft/brainy-postgres package
-export class PostgresStorageAugmentation {
- async init() {
- registerStorageAugmentation({
- type: 'postgres',
- name: 'PostgreSQL + pgvector',
- priority: 9,
-
- async detect() {
- const url = process.env.DATABASE_URL
- if (url?.includes('postgres')) {
- // Check for pgvector extension
- const client = new Client({ connectionString: url })
- await client.connect()
- const result = await client.query(
- "SELECT * FROM pg_extension WHERE extname = 'vector'"
- )
- await client.end()
- return result.rows.length > 0
- }
- return false
- },
-
- async getConfig() {
- return {
- type: 'postgres',
- postgresStorage: {
- connectionString: process.env.DATABASE_URL,
- table: 'brainy_vectors'
- }
- }
- }
- })
- }
-}
-```
-
-## Auto-Detection Priority
-
-Storage providers are checked in priority order:
-
-1. **Custom providers** (highest priority first)
-2. **Cloud storage** (S3, GCS, R2)
-3. **Database storage** (Redis, MongoDB, PostgreSQL)
-4. **Local storage** (filesystem, OPFS)
-5. **Memory** (fallback)
-
-```typescript
-// Example priority chain
-Redis (priority: 10) → PostgreSQL (9) → MongoDB (8) → S3 (5) → Filesystem (1) → Memory (0)
-```
-
-## Creating Custom Presets
-
-Augmentations can also register new presets:
-
-```typescript
-registerPresetAugmentation('redis-cluster', {
- storage: 'redis',
- model: ModelPrecision.Q8,
- features: ['core', 'cache', 'cluster'],
- distributed: true,
- role: DistributedRole.HYBRID,
- cache: {
- hotCacheMaxSize: 100000, // Large distributed cache
- autoTune: true
- },
- description: 'Redis Cluster configuration',
- category: PresetCategory.SERVICE
-})
-
-// Users can then use:
-const brain = new Brainy('redis-cluster')
-```
-
-## Type Safety with Extensions
-
-To maintain type safety with dynamic storage types:
-
-```typescript
-// Augmentation declares its types
-declare module '@soulcraft/brainy' {
- interface StorageTypes {
- redis: {
- url: string
- prefix?: string
- ttl?: number
- }
- }
-
- interface PresetNames {
- 'redis-cache': 'redis-cache'
- 'redis-cluster': 'redis-cluster'
- }
-}
-```
-
-## Best Practices for Storage Augmentations
-
-1. **Always provide auto-detection** - Check environment variables and connectivity
-2. **Set appropriate priority** - Higher for specialized storage, lower for general
-3. **Handle failures gracefully** - Return false from detect() if not available
-4. **Document requirements** - List required packages and environment variables
-5. **Provide presets** - Include common configuration patterns
-6. **Maintain compatibility** - Ensure model precision matches across instances
-
-## Example: Complete Redis Augmentation
-
-```typescript
-import {
- StorageProvider,
- registerStorageAugmentation,
- registerPresetAugmentation,
- PresetCategory,
- ModelPrecision,
- DistributedRole
-} from '@soulcraft/brainy/config'
-import Redis from 'ioredis'
-
-export class BrainyRedisAugmentation {
- private client: Redis
-
- async init() {
- // Register storage provider
- registerStorageAugmentation({
- type: 'redis',
- name: 'Redis Vector Storage',
- description: 'Redis with RediSearch for vector similarity',
- priority: 10,
-
- requirements: {
- env: ['REDIS_URL'],
- packages: ['ioredis', 'redis']
- },
-
- async detect() {
- if (!process.env.REDIS_URL) return false
-
- try {
- const client = new Redis(process.env.REDIS_URL)
-
- // Check for RediSearch module
- const modules = await client.call('MODULE', 'LIST')
- const hasRediSearch = modules.some(m => m[1] === 'search')
-
- await client.quit()
- return hasRediSearch
- } catch {
- return false
- }
- },
-
- async getConfig() {
- return {
- type: 'redis',
- redisStorage: {
- url: process.env.REDIS_URL,
- prefix: process.env.REDIS_PREFIX || 'brainy:',
- index: process.env.REDIS_INDEX || 'brainy-vectors',
- ttl: process.env.REDIS_TTL ? parseInt(process.env.REDIS_TTL) : undefined
- }
- }
- }
- })
-
- // Register presets
- this.registerPresets()
- }
-
- private registerPresets() {
- // Fast cache preset
- registerPresetAugmentation('redis-fast-cache', {
- storage: 'redis' as any,
- model: ModelPrecision.Q8,
- features: ['core', 'cache', 'search'],
- distributed: false,
- cache: {
- hotCacheMaxSize: 10000,
- autoTune: true
- },
- description: 'Redis-backed fast cache',
- category: PresetCategory.SERVICE
- })
-
- // Distributed cache preset
- registerPresetAugmentation('redis-distributed', {
- storage: 'redis' as any,
- model: ModelPrecision.AUTO,
- features: ['core', 'cache', 'search', 'cluster'],
- distributed: true,
- role: DistributedRole.HYBRID,
- cache: {
- hotCacheMaxSize: 50000,
- autoTune: true
- },
- description: 'Redis distributed cache cluster',
- category: PresetCategory.SERVICE
- })
-
- // Session store preset
- registerPresetAugmentation('redis-sessions', {
- storage: 'redis' as any,
- model: ModelPrecision.Q8,
- features: ['core', 'cache'],
- distributed: false,
- cache: {
- hotCacheMaxSize: 5000,
- autoTune: false
- },
- description: 'Redis session storage',
- category: PresetCategory.SERVICE
- })
- }
-}
-
-// Usage after installing the augmentation:
-import { Brainy } from '@soulcraft/brainy'
-import '@soulcraft/brainy-redis' // Registers the augmentation
-
-// Now Redis is automatically detected!
-const brain = new Brainy() // Uses Redis if REDIS_URL is set
-
-// Or use a Redis preset
-const brain = new Brainy('redis-fast-cache')
-
-// Or explicitly configure
-const brain = new Brainy({
- storage: 'redis',
- model: ModelPrecision.FP32
-})
-```
-
-## Summary
-
-The extensible configuration system allows:
-
-1. **New storage types** via `registerStorageAugmentation()`
-2. **Custom presets** via `registerPresetAugmentation()`
-3. **Auto-detection logic** that integrates with zero-config
-4. **Type-safe extensions** with TypeScript declarations
-5. **Priority-based selection** for intelligent defaults
-
-This ensures Brainy can grow with new storage technologies while maintaining its zero-configuration philosophy!
\ No newline at end of file
diff --git a/docs/FIND_SYSTEM.md b/docs/FIND_SYSTEM.md
index f43df101..1cc38ce9 100644
--- a/docs/FIND_SYSTEM.md
+++ b/docs/FIND_SYSTEM.md
@@ -1,29 +1,49 @@
+---
+title: The Find System
+slug: guides/find-system
+public: true
+category: guides
+template: guide
+order: 3
+description: Complete guide to Brainy's find() method — four intelligence systems, query execution phases, NLP patterns, and performance from 1K to 10M entities.
+next:
+ - concepts/triple-intelligence
+ - api/reference
+---
+
# Brainy's Find System - Complete Guide
## Overview
Brainy's `find()` method is the most advanced query system in any vector database, combining **Triple Intelligence** (vector + metadata + graph) with **Type-Aware NLP** for natural language understanding.
-## Architecture: Three Intelligence Systems
+## Architecture: Four Intelligence Systems
### 1. Vector Intelligence (HNSW Index)
- **Purpose**: Semantic similarity search using embeddings
- **Algorithm**: Hierarchical Navigable Small World (HNSW)
-- **Performance**: O(log n) search, ~1.8ms typical
+- **Performance**: O(log n) search
- **Data Structure**: Multi-layer graph with 16 connections per node
- **Use Cases**: "Find similar documents", "Content like this"
-### 2. Metadata Intelligence (Incremental Indices)
+### 2. Text Intelligence (Word Index) - **Purpose**: Keyword/exact text matching
+- **Algorithm**: Inverted word index with FNV-1a hashing
+- **Performance**: O(log C) where C = chunks (~50 values each)
+- **Data Structure**: `__words__ → hash → Roaring Bitmap of entity IDs`
+- **Use Cases**: "Find exact name", "Keyword search"
+- **Integration**: Automatically combined with Vector via RRF fusion
+
+### 3. Metadata Intelligence (Incremental Indices)
- **Purpose**: Fast filtering on structured data
- **Algorithm**: HashMap for exact matches, Sorted arrays for ranges
-- **Performance**: O(1) exact, O(log n) ranges, <1ms typical
+- **Performance**: O(1) exact, O(log n) ranges
- **Data Structure**: `Map>` + sorted value arrays
- **Use Cases**: "Documents from 2023", "Status equals active"
-### 3. Graph Intelligence (Adjacency Maps)
+### 4. Graph Intelligence (Adjacency Maps)
- **Purpose**: Relationship traversal and connection analysis
- **Algorithm**: Pure O(1) neighbor lookups via Map operations
-- **Performance**: O(1) per hop, ~0.1ms typical
+- **Performance**: O(1) per hop (measured <1 ms per neighbor lookup, validated up to 1M relationships — `tests/performance/graph-scale-performance.test.ts:238`)
- **Data Structure**: `Map>`
- **Use Cases**: "Papers connected to MIT", "Authors who collaborated"
@@ -46,29 +66,29 @@ await brain.find("documents by Smith published after 2020 with high citations")
```typescript
// Direct query objects with full control
await brain.find({
- // Vector search
- query: "machine learning research",
-
- // Metadata filters
- where: {
- publishDate: { greaterThan: 2020 },
- citations: { between: [50, 1000] },
- status: "published"
- },
-
- // Type constraints
- type: NounType.Document,
-
- // Graph traversal
- connected: {
- to: "mit-ai-lab",
- via: VerbType.AffiliatedWith,
- depth: 2
- },
-
- // Control options
- limit: 20,
- explain: true // Get scoring breakdown
+ // Vector search
+ query: "machine learning research",
+
+ // Metadata filters
+ where: {
+ publishDate: { greaterThan: 2020 },
+ citations: { between: [50, 1000] },
+ status: "published"
+ },
+
+ // Type constraints
+ type: NounType.Document,
+
+ // Graph traversal
+ connected: {
+ to: "mit-ai-lab",
+ via: VerbType.AffiliatedWith,
+ depth: 2
+ },
+
+ // Control options
+ limit: 20,
+ explain: true // Get scoring breakdown
})
```
@@ -76,14 +96,92 @@ await brain.find({
```typescript
// Find entities similar to a specific item
await brain.find({
- near: {
- id: "doc-123",
- threshold: 0.8 // Minimum similarity
- },
- type: NounType.Document
+ near: {
+ id: "doc-123",
+ threshold: 0.8 // Minimum similarity
+ },
+ type: NounType.Document
})
```
+### 4. Hybrid Search
+```typescript
+// Zero-config hybrid: automatically combines text + semantic search
+await brain.find({ query: "David Smith" })
+// Uses Reciprocal Rank Fusion (RRF) to combine results
+
+// Force text-only search
+await brain.find({ query: "exact keyword", searchMode: 'text' })
+
+// Force semantic-only search
+await brain.find({ query: "AI concepts", searchMode: 'semantic' })
+
+// Custom hybrid weighting (0 = text only, 1 = semantic only)
+await brain.find({ query: "search term", hybridAlpha: 0.3 })
+```
+
+**How Auto-Alpha Works:**
+- Short queries (1-2 words): alpha = 0.3 (favor text matching)
+- Medium queries (3-4 words): alpha = 0.5 (balanced)
+- Long queries (5+ words): alpha = 0.7 (favor semantic matching)
+
+### 5. Match Visibility
+
+Search results include match details showing what matched:
+
+```typescript
+const results = await brain.find({ query: 'david the warrior' })
+
+// Each result has:
+results[0].textMatches // ["david", "warrior"] - exact words found
+results[0].textScore // 0.25 - text match quality (0-1)
+results[0].semanticScore // 0.87 - semantic similarity (0-1)
+results[0].matchSource // 'both' | 'text' | 'semantic'
+```
+
+Use this for:
+- **Highlighting** exact matches in UI (textMatches)
+- **Explaining** why a result was found (matchSource)
+- **Debugging** search behavior (separate scores)
+
+### 6. Semantic Highlighting
+
+Highlight which concepts/words in text matched your query:
+
+```typescript
+// Find semantically similar words + exact matches
+const highlights = await brain.highlight({
+ query: "david the warrior",
+ text: "David Smith is a brave fighter who battles dragons"
+})
+
+// Returns:
+// [
+// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' },
+// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' },
+// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
+// ]
+```
+
+**Features:**
+- `matchType: 'text'` - Exact word match (score = 1.0)
+- `matchType: 'semantic'` - Concept match (score varies)
+- `position` - [start, end] for precise highlighting
+- `granularity` - 'word' (default), 'phrase', or 'sentence'
+- `threshold` - Minimum semantic score (default: 0.5)
+
+**UI Usage Pattern:**
+```typescript
+// Highlight search results with different styles
+function highlightResult(text: string, highlights: Highlight[]) {
+ return highlights.map(h => ({
+ text: h.text,
+ position: h.position,
+ style: h.matchType === 'text' ? 'strong' : 'emphasis' // Different UI styles
+ }))
+}
+```
+
## Index Usage in Detail
### Metadata Index Operations
@@ -96,7 +194,7 @@ await brain.find({
// Memory: ~40 bytes per unique field-value combination
```
-#### Sorted Index (Range Queries)
+#### Sorted Index (Range Queries)
```typescript
// Query: {publishDate: {greaterThan: 2020}}
// Index: sortedIndices.get("publishDate") → [[2019, Set], [2020, Set], [2021, Set]]
@@ -122,7 +220,7 @@ await brain.find({
// Query: {query: "machine learning"}
// Process:
// 1. Embed query text → 384-dimensional vector
-// 2. Start at top layer (entry point)
+// 2. Start at top layer (entry point)
// 3. Greedy search for nearest neighbor at each layer
// 4. Move down layers for progressively finer search
// 5. Return k nearest neighbors with similarity scores
@@ -135,10 +233,10 @@ await brain.find({
#### O(1) Neighbor Lookups
```typescript
// Query: {connected: {to: "entity-123"}}
-// Index lookup:
-// - Outgoing: sourceIndex.get("entity-123") → Set
-// - Incoming: targetIndex.get("entity-123") → Set
-// - Both directions: union of both Sets
+// Index lookup:
+// - Outgoing: sourceIndex.get("entity-123") → Set
+// - Incoming: targetIndex.get("entity-123") → Set
+// - Both directions: union of both Sets
// Performance: O(1) per hop, no matter the graph size
// Memory: ~24 bytes per relationship (source + target + metadata)
```
@@ -148,19 +246,19 @@ await brain.find({
### Phase 1: Query Parsing
```typescript
if (typeof query === 'string') {
- // Natural Language Processing
- const nlpParams = await this.parseNaturalQuery(query)
-
- // Type-aware parsing:
- // 1. Detect NounType using pre-embedded type vectors
- // 2. Get type-specific fields from real data patterns
- // 3. Semantic field matching with type affinity boosting
- // 4. Validate field-type compatibility
- // 5. Generate optimized query plan
-
- params = nlpParams
+ // Natural Language Processing
+ const nlpParams = await this.parseNaturalQuery(query)
+
+ // Type-aware parsing:
+ // 1. Detect NounType using pre-embedded type vectors
+ // 2. Get type-specific fields from real data patterns
+ // 3. Semantic field matching with type affinity boosting
+ // 4. Validate field-type compatibility
+ // 5. Generate optimized query plan
+
+ params = nlpParams
} else {
- params = query // Direct structured query
+ params = query // Direct structured query
}
```
@@ -171,12 +269,12 @@ const searchPromises = []
// Vector search (if query text or vector provided)
if (params.query || params.vector) {
- searchPromises.push(this.executeVectorSearch(params))
+ searchPromises.push(this.executeVectorSearch(params))
}
-// Proximity search (if near parameter provided)
+// Proximity search (if near parameter provided)
if (params.near) {
- searchPromises.push(this.executeProximitySearch(params))
+ searchPromises.push(this.executeProximitySearch(params))
}
// Wait for all searches to complete
@@ -187,41 +285,41 @@ const searchResults = await Promise.all(searchPromises)
```typescript
// Apply metadata filters using optimized indices
if (params.where || params.type || params.service) {
- const filter = {
- ...params.where,
- ...(params.type && { noun: params.type }),
- ...(params.service && { service: params.service })
- }
-
- // Get optimal query plan based on field cardinalities
- const queryPlan = await this.getOptimalQueryPlan(filter)
-
- // Execute filters in optimal order (low cardinality first)
- const filteredIds = await this.metadataIndex.getIdsForFilter(filter)
-
- if (results.length > 0) {
- // Intersect with vector search results
- results = results.filter(r => filteredIds.includes(r.id))
- } else {
- // Create results from metadata matches (metadata-only query)
- results = await this.createResultsFromIds(filteredIds)
- }
+ const filter = {
+ ...params.where,
+ ...(params.type && { noun: params.type }),
+ ...(params.service && { service: params.service })
+ }
+
+ // Get optimal query plan based on field cardinalities
+ const queryPlan = await this.getOptimalQueryPlan(filter)
+
+ // Execute filters in optimal order (low cardinality first)
+ const filteredIds = await this.metadataIndex.getIdsForFilter(filter)
+
+ if (results.length > 0) {
+ // Intersect with vector search results
+ results = results.filter(r => filteredIds.includes(r.id))
+ } else {
+ // Create results from metadata matches (metadata-only query)
+ results = await this.createResultsFromIds(filteredIds)
+ }
}
```
-### Phase 4: Graph Traversal
+### Phase 4: Graph Traversal
```typescript
// Apply graph constraints using O(1) lookups
if (params.connected) {
- const connectedIds = await this.graphIndex.getConnectedIds(params.connected)
-
- if (results.length > 0) {
- // Filter existing results to only connected entities
- results = results.filter(r => connectedIds.includes(r.id))
- } else {
- // Create results from connected entities
- results = await this.createResultsFromIds(connectedIds)
- }
+ const connectedIds = await this.graphIndex.getConnectedIds(params.connected)
+
+ if (results.length > 0) {
+ // Filter existing results to only connected entities
+ results = results.filter(r => connectedIds.includes(r.id))
+ } else {
+ // Create results from connected entities
+ results = await this.createResultsFromIds(connectedIds)
+ }
}
```
@@ -229,12 +327,12 @@ if (params.connected) {
```typescript
// Combine scores from multiple intelligence sources
if (params.fusion && results.length > 0) {
- results = this.applyFusionScoring(results, params.fusion)
-
- // Example fusion strategies:
- // - Weighted: vectorScore × 0.6 + metadataScore × 0.2 + graphScore × 0.2
- // - Adaptive: adjust weights based on query characteristics
- // - Progressive: prioritize based on result confidence
+ results = this.applyFusionScoring(results, params.fusion)
+
+ // Example fusion strategies:
+ // - Weighted: vectorScore × 0.6 + metadataScore × 0.2 + graphScore × 0.2
+ // - Adaptive: adjust weights based on query characteristics
+ // - Progressive: prioritize based on result confidence
}
// Sort by final score and apply pagination
@@ -265,7 +363,7 @@ return results.slice(offset, offset + limit)
### 3. Field-Type Validation
```typescript
// Prevents invalid queries and suggests alternatives:
-// "people with publishDate > 2020"
+// "people with publishDate > 2020"
// → Warning: "Person entities rarely have publishDate field"
// → Suggestion: "Did you mean createdAt, updatedAt, or birthDate?"
// → Auto-correction: Use most likely alternative based on affinity data
@@ -275,36 +373,40 @@ return results.slice(offset, offset + limit)
### Query Performance by Type
-| Query Type | Index Used | Performance | Example |
-|------------|------------|-------------|---------|
-| **Semantic Search** | HNSW Vector | O(log n), ~1.8ms | `"AI research papers"` |
-| **Exact Metadata** | HashMap | O(1), ~0.8ms | `{status: "published"}` |
-| **Range Metadata** | Sorted Array | O(log n), ~0.6ms | `{year: {greaterThan: 2020}}` |
-| **Graph Traversal** | Adjacency Map | O(1), ~0.1ms | `{connected: {to: "mit"}}` |
-| **Type Detection** | Pre-embedded Types | O(t), ~0.3ms | `"documents"` → `NounType.Document` |
-| **Field Matching** | Field Embeddings | O(f), ~0.1ms | `"by"` → `"author"` |
-| **Combined Query** | All Indices | O(log n), ~1.8ms | NLP + filters + graph |
+| Query Type | Index Used | Complexity | Example |
+|------------|------------|------------|---------|
+| **Semantic Search** | HNSW Vector | O(log n) | `"AI research papers"` |
+| **Exact Metadata** | HashMap | O(1) | `{status: "published"}` |
+| **Range Metadata** | Sorted Array | O(log n) | `{year: {greaterThan: 2020}}` |
+| **Graph Traversal** | Adjacency Map | O(1) | `{connected: {to: "mit"}}` |
+| **Type Detection** | Pre-embedded Types | O(t) | `"documents"` → `NounType.Document` |
+| **Field Matching** | Field Embeddings | O(f) | `"by"` → `"author"` |
+| **Combined Query** | All Indices | O(log n) | NLP + filters + graph |
Where:
- n = number of entities in database
-- t = number of types (169 total: 42 noun + 127 verb)
+- t = number of types (169 total: 42 noun + 127 verb)
- f = number of fields for detected entity type (typically 5-15)
+Absolute latencies depend on hardware, embedding model, and storage backend; the columns above describe how each stage scales. The graph stage is the one path with a committed scale benchmark (see [Scalability](#scalability)).
+
### Scalability
-| Database Size | Vector Search | Metadata Filter | Graph Query | Combined |
-|---------------|---------------|-----------------|-------------|----------|
-| **1K entities** | 0.8ms | 0.3ms | 0.05ms | 1.1ms |
-| **10K entities** | 1.2ms | 0.5ms | 0.08ms | 1.5ms |
-| **100K entities** | 1.8ms | 0.8ms | 0.1ms | 2.1ms |
-| **1M entities** | 2.5ms | 1.2ms | 0.1ms | 2.8ms |
-| **10M entities** | 3.8ms | 1.8ms | 0.1ms | 4.2ms |
+The cost of each query stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion:
+
+| Query stage | Complexity | Scaling behavior |
+|-------------|------------|------------------|
+| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
+| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
+| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
+| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
+| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
**Key Performance Notes:**
- Graph queries stay O(1) regardless of scale
- Metadata ranges scale as O(log n), not O(n)
- Vector search degrades gracefully due to HNSW
-- Type-aware NLP adds minimal overhead (~0.4ms)
+- Type-aware NLP adds minimal overhead (single embedding pass, no full scan)
## Example Query Flows
@@ -315,27 +417,27 @@ Where:
// Phase 1: NLP Parsing
// - "papers" → NounType.Document (0.94 confidence)
// - "Stanford researchers" → entity search + NounType.Person
-// - "recent" → publishDate: {greaterThan: 2023}
+// - "recent" → publishDate: {greaterThan: 2023}
// - "high citations" → citations: {greaterThan: 100}
// - "connected to industry" → graph traversal via VerbType.AffiliatedWith
// Phase 2: Generated Query
{
- type: NounType.Document,
- where: {
- publishDate: { greaterThan: 2023 },
- citations: { greaterThan: 100 }
- },
- connected: {
- to: ["stanford-researchers"],
- via: VerbType.AffiliatedWith,
- depth: 2
- }
+ type: NounType.Document,
+ where: {
+ publishDate: { greaterThan: 2023 },
+ citations: { greaterThan: 100 }
+ },
+ connected: {
+ to: ["stanford-researchers"],
+ via: VerbType.AffiliatedWith,
+ depth: 2
+ }
}
// Phase 3: Execution Plan
// 1. Metadata filter: publishDate > 2023 (O(log n) via sorted index)
-// 2. Metadata filter: citations > 100 (O(log n) via sorted index)
+// 2. Metadata filter: citations > 100 (O(log n) via sorted index)
// 3. Graph traversal: connected to Stanford (O(1) per hop)
// 4. Intersection: entities matching all constraints
// 5. Sort by relevance score
@@ -345,22 +447,22 @@ Where:
```typescript
// Query: Direct structured query for maximum performance
await brain.find({
- type: NounType.Document, // O(1) type filter
- where: {
- status: "published", // O(1) exact match
- year: 2024, // O(1) exact match
- citations: { greaterThan: 50 } // O(log n) range query
- },
- connected: {
- from: "author-123", // O(1) graph lookup
- via: VerbType.Creates
- },
- limit: 10
+ type: NounType.Document, // O(1) type filter
+ where: {
+ status: "published", // O(1) exact match
+ year: 2024, // O(1) exact match
+ citations: { greaterThan: 50 } // O(log n) range query
+ },
+ connected: {
+ from: "author-123", // O(1) graph lookup
+ via: VerbType.Creates
+ },
+ limit: 10
})
-// Total performance: ~1.2ms for 100K entities
+// Executes as O(1) type filter + O(1)/O(log n) metadata + O(1) graph lookup — no full scan
```
-## Filter Syntax Reference (v5.8.0+)
+## Filter Syntax Reference
### Where Clause: Complete Operator Guide
@@ -371,25 +473,25 @@ Brainy provides a comprehensive set of operators for filtering entities by metad
**Exact Match** (shorthand):
```typescript
await brain.find({
- where: {
- status: 'active', // Shorthand for { eq: 'active' }
- year: 2024, // Exact match for numbers
- verified: true // Boolean matching
- }
+ where: {
+ status: 'active', // Shorthand for { eq: 'active' }
+ year: 2024, // Exact match for numbers
+ verified: true // Boolean matching
+ }
})
```
**Comparison Operators**:
```typescript
await brain.find({
- where: {
- age: { gt: 18 }, // Greater than
- score: { gte: 80 }, // Greater than or equal
- price: { lt: 100 }, // Less than
- stock: { lte: 10 }, // Less than or equal
- status: { eq: 'active' }, // Equals (explicit)
- role: { ne: 'guest' } // Not equals
- }
+ where: {
+ age: { gt: 18 }, // Greater than
+ score: { gte: 80 }, // Greater than or equal
+ price: { lt: 100 }, // Less than
+ stock: { lte: 10 }, // Less than or equal
+ status: { eq: 'active' }, // Equals (explicit)
+ role: { ne: 'guest' } // Not equals
+ }
})
```
@@ -400,11 +502,11 @@ await brain.find({
**Between** (inclusive):
```typescript
await brain.find({
- where: {
- publishDate: { between: [2020, 2024] }, // Year range
- price: { between: [10.00, 99.99] }, // Price range
- timestamp: { between: [startMs, endMs] } // Time range
- }
+ where: {
+ publishDate: { between: [2020, 2024] }, // Year range
+ price: { between: [10.00, 99.99] }, // Price range
+ timestamp: { between: [startMs, endMs] } // Time range
+ }
})
```
@@ -415,11 +517,11 @@ await brain.find({
**In/Not In**:
```typescript
await brain.find({
- where: {
- category: { in: ['tech', 'science', 'research'] },
- status: { notIn: ['draft', 'deleted'] },
- priority: { in: [1, 2, 3] }
- }
+ where: {
+ category: { in: ['tech', 'science', 'research'] },
+ status: { notIn: ['draft', 'deleted'] },
+ priority: { in: [1, 2, 3] }
+ }
})
```
@@ -430,11 +532,11 @@ await brain.find({
**Contains/Starts/Ends**:
```typescript
await brain.find({
- where: {
- title: { contains: 'machine learning' }, // Substring search
- email: { startsWith: 'admin@' }, // Prefix match
- filename: { endsWith: '.pdf' } // Suffix match
- }
+ where: {
+ title: { contains: 'machine learning' }, // Substring search
+ email: { startsWith: 'admin@' }, // Prefix match
+ filename: { endsWith: '.pdf' } // Suffix match
+ }
})
```
@@ -454,11 +556,11 @@ query: 'artificial intelligence'
**Exists/Missing**:
```typescript
await brain.find({
- where: {
- email: { exists: true }, // Has email field
- deletedAt: { exists: false }, // No deletedAt field (not deleted)
- profileImage: { exists: true } // Has profile image
- }
+ where: {
+ email: { exists: true }, // Has email field
+ deletedAt: { exists: false }, // No deletedAt field (not deleted)
+ profileImage: { exists: true } // Has profile image
+ }
})
```
@@ -474,11 +576,11 @@ All conditions at the same level are implicitly AND:
```typescript
await brain.find({
- where: {
- status: 'published', // AND
- year: { gte: 2020 }, // AND
- citations: { gte: 50 } // AND
- }
+ where: {
+ status: 'published', // AND
+ year: { gte: 2020 }, // AND
+ citations: { gte: 50 } // AND
+ }
})
// Returns: entities matching ALL three conditions
```
@@ -486,13 +588,13 @@ await brain.find({
**Explicit AND with `allOf`**:
```typescript
await brain.find({
- where: {
- allOf: [
- { status: 'published' },
- { year: { gte: 2020 } },
- { citations: { gte: 50 } }
- ]
- }
+ where: {
+ allOf: [
+ { status: 'published' },
+ { year: { gte: 2020 } },
+ { citations: { gte: 50 } }
+ ]
+ }
})
```
@@ -504,13 +606,13 @@ Match ANY condition:
```typescript
await brain.find({
- where: {
- anyOf: [
- { status: 'urgent' },
- { priority: { gte: 8 } },
- { assignee: 'admin' }
- ]
- }
+ where: {
+ anyOf: [
+ { status: 'urgent' },
+ { priority: { gte: 8 } },
+ { assignee: 'admin' }
+ ]
+ }
})
// Returns: entities matching ANY condition
```
@@ -518,13 +620,13 @@ await brain.find({
**Combined AND + OR**:
```typescript
await brain.find({
- where: {
- status: 'active', // Must be active
- anyOf: [ // AND (urgent OR high priority)
- { tags: { contains: 'urgent' } },
- { priority: { gte: 8 } }
- ]
- }
+ where: {
+ status: 'active', // Must be active
+ anyOf: [ // AND (urgent OR high priority)
+ { tags: { contains: 'urgent' } },
+ { priority: { gte: 8 } }
+ ]
+ }
})
```
@@ -536,17 +638,17 @@ Complex boolean expressions:
```typescript
await brain.find({
- where: {
- allOf: [
- { status: 'published' },
- {
- anyOf: [
- { featured: true },
- { citations: { gte: 100 } }
- ]
- }
- ]
- }
+ where: {
+ allOf: [
+ { status: 'published' },
+ {
+ anyOf: [
+ { featured: true },
+ { citations: { gte: 100 } }
+ ]
+ }
+ ]
+ }
})
// Returns: published AND (featured OR highly cited)
```
@@ -591,8 +693,8 @@ Filter entities by NounType:
```typescript
await brain.find({
- type: NounType.Document,
- where: { year: { gte: 2020 } }
+ type: NounType.Document,
+ where: { year: { gte: 2020 } }
})
```
@@ -600,8 +702,8 @@ await brain.find({
```typescript
await brain.find({
- type: [NounType.Person, NounType.Organization],
- where: { verified: true }
+ type: [NounType.Person, NounType.Organization],
+ where: { verified: true }
})
```
@@ -642,11 +744,11 @@ Traverse relationships using the GraphIndex:
```typescript
await brain.find({
- connected: {
- to: 'entity-id-123', // Connected to this entity
- via: VerbType.WorksFor, // Through this relationship type
- direction: 'out' // Direction: 'in', 'out', or 'both'
- }
+ connected: {
+ to: 'entity-id-123', // Connected to this entity
+ via: VerbType.WorksFor, // Through this relationship type
+ direction: 'out' // Direction: 'in', 'out', or 'both'
+ }
})
```
@@ -656,11 +758,11 @@ await brain.find({
```typescript
await brain.find({
- connected: {
- to: 'research-institution',
- via: VerbType.AffiliatedWith,
- depth: 2 // Up to 2 hops away
- }
+ connected: {
+ to: 'research-institution',
+ via: VerbType.AffiliatedWith,
+ depth: 2 // Up to 2 hops away
+ }
})
```
@@ -670,35 +772,35 @@ await brain.find({
```typescript
await brain.find({
- type: NounType.Person,
- where: {
- verified: true,
- reputation: { gte: 100 }
- },
- connected: {
- to: 'stanford-ai-lab',
- via: VerbType.WorksAt,
- direction: 'out'
- },
- limit: 20
+ type: NounType.Person,
+ where: {
+ verified: true,
+ reputation: { gte: 100 }
+ },
+ connected: {
+ to: 'stanford-ai-lab',
+ via: VerbType.WorksAt,
+ direction: 'out'
+ },
+ limit: 20
})
// Returns: Verified people with high reputation who work at Stanford AI Lab
```
-#### Pagination with Graph Queries (v5.8.0+)
+#### Pagination with Graph Queries
```typescript
// Page through high-degree nodes efficiently
const neighbors = await brain.graphIndex.getNeighbors('hub-entity-id', {
- direction: 'out',
- limit: 50,
- offset: 0
+ direction: 'out',
+ limit: 50,
+ offset: 0
})
// Get verb IDs with pagination
const verbIds = await brain.graphIndex.getVerbIdsBySource('source-id', {
- limit: 100,
- offset: 0
+ limit: 100,
+ offset: 0
})
```
@@ -706,36 +808,36 @@ const verbIds = await brain.graphIndex.getVerbIdsBySource('source-id', {
**Note**: See `src/graph/graphAdjacencyIndex.ts` for low-level graph operations.
-### Sorting Results (v4.5.4+)
+### Sorting Results
Sort query results by any field, including timestamps:
```typescript
// Sort by timestamp (descending - newest first)
await brain.find({
- type: NounType.Document,
- orderBy: 'createdAt',
- order: 'desc',
- limit: 10
+ type: NounType.Document,
+ orderBy: 'createdAt',
+ order: 'desc',
+ limit: 10
})
// Sort by custom field (ascending)
await brain.find({
- where: { status: { eq: 'published' } },
- orderBy: 'priority',
- order: 'asc'
+ where: { status: { eq: 'published' } },
+ orderBy: 'priority',
+ order: 'asc'
})
// Sort with filtering and pagination
await brain.find({
- where: {
- publishDate: { gte: startDate },
- citations: { gte: 50 }
- },
- orderBy: 'citations',
- order: 'desc',
- limit: 20,
- offset: 0
+ where: {
+ publishDate: { gte: startDate },
+ citations: { gte: 50 }
+ },
+ orderBy: 'citations',
+ order: 'desc',
+ limit: 20,
+ offset: 0
})
```
@@ -750,17 +852,17 @@ await brain.find({
```typescript
// Range query + sorting
await brain.find({
- where: {
- createdAt: { gte: Date.now() - 86400000 } // Last 24 hours
- },
- orderBy: 'createdAt',
- order: 'desc' // Newest first
+ where: {
+ createdAt: { gte: Date.now() - 86400000 } // Last 24 hours
+ },
+ orderBy: 'createdAt',
+ order: 'desc' // Newest first
})
// Works with updatedAt, accessed, modified
await brain.find({
- orderBy: 'updatedAt',
- order: 'desc'
+ orderBy: 'updatedAt',
+ order: 'desc'
})
```
@@ -768,22 +870,22 @@ await brain.find({
```typescript
// Sort search results by custom field instead of relevance
await brain.find({
- query: "machine learning",
- where: { publishDate: { gte: 2023 } },
- orderBy: 'citations', // Sort by citations, not relevance
- order: 'desc'
+ query: "machine learning",
+ where: { publishDate: { gte: 2023 } },
+ orderBy: 'citations', // Sort by citations, not relevance
+ order: 'desc'
})
// Paginated sorted results
async function getDocumentsByDate(page: number, pageSize: number = 20) {
- return await brain.find({
- type: NounType.Document,
- where: { status: { eq: 'published' } },
- orderBy: 'publishDate',
- order: 'desc',
- limit: pageSize,
- offset: page * pageSize
- })
+ return await brain.find({
+ type: NounType.Document,
+ where: { status: { eq: 'published' } },
+ orderBy: 'publishDate',
+ order: 'desc',
+ limit: pageSize,
+ offset: page * pageSize
+ })
}
```
@@ -794,30 +896,30 @@ async function getDocumentsByDate(page: number, pageSize: number = 20) {
**Offset-based pagination**:
```typescript
async function getPaginatedResults(page: number, pageSize: number = 20) {
- return await brain.find({
- type: NounType.Document,
- where: { status: 'published' },
- orderBy: 'createdAt',
- order: 'desc',
- limit: pageSize,
- offset: page * pageSize
- })
+ return await brain.find({
+ type: NounType.Document,
+ where: { status: 'published' },
+ orderBy: 'createdAt',
+ order: 'desc',
+ limit: pageSize,
+ offset: page * pageSize
+ })
}
// Usage
-const page1 = await getPaginatedResults(0) // First 20
-const page2 = await getPaginatedResults(1) // Next 20
+const page1 = await getPaginatedResults(0) // First 20
+const page2 = await getPaginatedResults(1) // Next 20
```
-**Graph pagination** (v5.8.0+):
+**Graph pagination**:
```typescript
// Paginate through high-degree node relationships
async function getNeighborPage(entityId: string, page: number, pageSize: number = 50) {
- return await brain.graphIndex.getNeighbors(entityId, {
- direction: 'out',
- limit: pageSize,
- offset: page * pageSize
- })
+ return await brain.graphIndex.getNeighbors(entityId, {
+ direction: 'out',
+ limit: pageSize,
+ offset: page * pageSize
+ })
}
```
@@ -830,23 +932,23 @@ async function getNeighborPage(entityId: string, page: number, pageSize: number
// Last 24 hours
const oneDayAgo = Date.now() - (24 * 60 * 60 * 1000)
await brain.find({
- where: {
- createdAt: { gte: oneDayAgo }
- },
- orderBy: 'createdAt',
- order: 'desc'
+ where: {
+ createdAt: { gte: oneDayAgo }
+ },
+ orderBy: 'createdAt',
+ order: 'desc'
})
// Last 7 days with additional filters
const oneWeekAgo = Date.now() - (7 * 24 * 60 * 60 * 1000)
await brain.find({
- type: NounType.Document,
- where: {
- createdAt: { gte: oneWeekAgo },
- status: 'published'
- },
- orderBy: 'createdAt',
- order: 'desc'
+ type: NounType.Document,
+ where: {
+ createdAt: { gte: oneWeekAgo },
+ status: 'published'
+ },
+ orderBy: 'createdAt',
+ order: 'desc'
})
```
@@ -854,21 +956,21 @@ await brain.find({
```typescript
// Specific year
await brain.find({
- where: {
- publishDate: { between: [
- new Date('2023-01-01').getTime(),
- new Date('2023-12-31').getTime()
- ]}
- }
+ where: {
+ publishDate: { between: [
+ new Date('2023-01-01').getTime(),
+ new Date('2023-12-31').getTime()
+ ]}
+ }
})
// Quarter
const Q1_2024_start = new Date('2024-01-01').getTime()
const Q1_2024_end = new Date('2024-03-31').getTime()
await brain.find({
- where: {
- createdAt: { between: [Q1_2024_start, Q1_2024_end] }
- }
+ where: {
+ createdAt: { between: [Q1_2024_start, Q1_2024_end] }
+ }
})
```
@@ -878,32 +980,32 @@ await brain.find({
```typescript
// Find: AI research papers from verified authors at top institutions
const results = await brain.find({
- // Vector search (semantic)
- query: 'artificial intelligence machine learning',
+ // Vector search (semantic)
+ query: 'artificial intelligence machine learning',
- // Metadata filters
- type: NounType.Document,
- where: {
- publishDate: { gte: 2020 },
- citations: { gte: 50 },
- peerReviewed: true
- },
+ // Metadata filters
+ type: NounType.Document,
+ where: {
+ publishDate: { gte: 2020 },
+ citations: { gte: 50 },
+ peerReviewed: true
+ },
- // Graph traversal
- connected: {
- to: topInstitutionIds, // Array of institution entity IDs
- via: VerbType.AffiliatedWith,
- depth: 2 // Authors affiliated with institutions (2 hops)
- },
+ // Graph traversal
+ connected: {
+ to: topInstitutionIds, // Array of institution entity IDs
+ via: VerbType.AffiliatedWith,
+ depth: 2 // Authors affiliated with institutions (2 hops)
+ },
- // Results
- limit: 50,
- orderBy: 'citations',
- order: 'desc'
+ // Results
+ limit: 50,
+ orderBy: 'citations',
+ order: 'desc'
})
```
-**Performance**: O(log n) vector search + O(log n) metadata filters + O(1) graph traversal = ~2-3ms total.
+**Performance**: O(log n) vector search + O(log n) metadata filters + O(1) graph traversal — bounded by the vector stage.
### Excluding Soft-Deleted Entities
@@ -911,19 +1013,19 @@ const results = await brain.find({
```typescript
// Standard query excludes deleted
await brain.find({
- where: {
- deletedAt: { exists: false } // Not soft-deleted
- }
+ where: {
+ deletedAt: { exists: false } // Not soft-deleted
+ }
})
// Or use compound filter
await brain.find({
- where: {
- allOf: [
- { status: 'active' },
- { deletedAt: { exists: false } }
- ]
- }
+ where: {
+ allOf: [
+ { status: 'active' },
+ { deletedAt: { exists: false } }
+ ]
+ }
})
```
@@ -935,21 +1037,21 @@ await brain.find({
```typescript
// Find documents similar to a specific document
await brain.find({
- near: {
- id: 'doc-123',
- threshold: 0.8 // Minimum 80% similarity
- },
- type: NounType.Document,
- limit: 10
+ near: {
+ id: 'doc-123',
+ threshold: 0.8 // Minimum 80% similarity
+ },
+ type: NounType.Document,
+ limit: 10
})
// With metadata constraints
await brain.find({
- near: { id: 'paper-456', threshold: 0.75 },
- where: {
- publishDate: { gte: 2020 },
- language: 'en'
- }
+ near: { id: 'paper-456', threshold: 0.75 },
+ where: {
+ publishDate: { gte: 2020 },
+ language: 'en'
+ }
})
```
@@ -961,8 +1063,8 @@ await brain.find({
```typescript
// Get total count (metadata-only query is fastest)
const results = await brain.find({
- where: { status: 'published' },
- limit: 1 // We only need the count
+ where: { status: 'published' },
+ limit: 1 // We only need the count
})
// Note: Current API returns results, not counts
// For production, consider caching counts or using metadata indices directly
@@ -973,10 +1075,10 @@ const results = await brain.find({
// Find all entities, then group by type in application
const allEntities = await brain.find({ limit: 10000 })
const byType = allEntities.reduce((acc, entity) => {
- const type = entity.noun || 'unknown'
- if (!acc[type]) acc[type] = []
- acc[type].push(entity)
- return acc
+ const type = entity.noun || 'unknown'
+ if (!acc[type]) acc[type] = []
+ acc[type].push(entity)
+ return acc
}, {})
```
@@ -985,14 +1087,14 @@ const byType = allEntities.reduce((acc, entity) => {
**Any of multiple values**:
```typescript
await brain.find({
- where: {
- anyOf: [
- { priority: 'urgent' },
- { priority: 'high' },
- { assignee: 'admin' },
- { dueDate: { lte: Date.now() } }
- ]
- }
+ where: {
+ anyOf: [
+ { priority: 'urgent' },
+ { priority: 'high' },
+ { assignee: 'admin' },
+ { dueDate: { lte: Date.now() } }
+ ]
+ }
})
// Returns: urgent OR high priority OR assigned to admin OR overdue
```
@@ -1001,23 +1103,23 @@ await brain.find({
```typescript
// Find: (Premium users OR trial users with activity) AND not banned
await brain.find({
- type: NounType.Person,
- where: {
- allOf: [
- {
- anyOf: [
- { subscription: 'premium' },
- {
- allOf: [
- { subscription: 'trial' },
- { lastActive: { gte: Date.now() - 86400000 } } // 24h
- ]
- }
- ]
- },
- { banned: { ne: true } }
- ]
- }
+ type: NounType.Person,
+ where: {
+ allOf: [
+ {
+ anyOf: [
+ { subscription: 'premium' },
+ {
+ allOf: [
+ { subscription: 'trial' },
+ { lastActive: { gte: Date.now() - 86400000 } } // 24h
+ ]
+ }
+ ]
+ },
+ { banned: { ne: true } }
+ ]
+ }
})
```
@@ -1029,8 +1131,8 @@ await brain.find({
```typescript
// List all entities of a type
const all = await brain.find({
- type: NounType.Document,
- limit: 10
+ type: NounType.Document,
+ limit: 10
})
console.log(`Found ${all.length} documents`)
```
@@ -1038,11 +1140,11 @@ console.log(`Found ${all.length} documents`)
**Check 2: Test filters individually**
```typescript
// Remove filters one by one to find the culprit
-await brain.find({ where: { status: 'published' } }) // Works?
-await brain.find({ where: { year: 2024 } }) // Works?
+await brain.find({ where: { status: 'published' } }) // Works?
+await brain.find({ where: { year: 2024 } }) // Works?
await brain.find({ where: {
- status: 'published',
- year: 2024 // Combined - works?
+ status: 'published',
+ year: 2024 // Combined - works?
}})
```
@@ -1050,7 +1152,7 @@ await brain.find({ where: {
```typescript
// Get a sample entity to see actual field names
const sample = await brain.find({ type: NounType.Document, limit: 1 })
-console.log(Object.keys(sample[0].data)) // Actual fields
+console.log(Object.keys(sample[0].data)) // Actual fields
```
**Common issues**:
@@ -1064,9 +1166,9 @@ console.log(Object.keys(sample[0].data)) // Actual fields
```typescript
// Use explain mode (if available)
const results = await brain.find({
- query: 'machine learning',
- where: { title: { contains: 'AI' } }, // ⚠️ O(n) substring search
- explain: true
+ query: 'machine learning',
+ where: { title: { contains: 'AI' } }, // ⚠️ O(n) substring search
+ explain: true
})
```
@@ -1089,15 +1191,15 @@ where: { status: 'published' }
```typescript
// ❌ Suboptimal: Slow filter first
where: {
- description: { contains: 'AI' }, // O(n) - runs first
- year: 2024 // O(1) - runs second
+ description: { contains: 'AI' }, // O(n) - runs first
+ year: 2024 // O(1) - runs second
}
// ✅ Optimal: Fast filter first (automatic optimization)
where: {
- year: 2024, // O(1) - narrow results
- status: 'published' // O(1) - further narrow
- // Only then apply O(n) operations if needed
+ year: 2024, // O(1) - narrow results
+ status: 'published' // O(1) - further narrow
+ // Only then apply O(n) operations if needed
}
```
@@ -1119,10 +1221,10 @@ import { NounType } from '@soulcraft/brainy'
await brain.find({ type: NounType.Document })
// ❌ Error: Operator not recognized
-where: { age: { greaterThan: 18 } } // Old API
+where: { age: { greaterThan: 18 } } // Old API
// ✅ Correct: Use canonical operators
-where: { age: { gt: 18 } } // v5.0.0+
+where: { age: { gt: 18 } }
```
### Graph Traversal Issues
@@ -1130,26 +1232,26 @@ where: { age: { gt: 18 } } // v5.0.0+
**No connected entities found**:
```typescript
// Verify relationship exists
-const relations = await brain.getRelations({
- from: 'entity-a',
- to: 'entity-b'
+const relations = await brain.related({
+ from: 'entity-a',
+ to: 'entity-b'
})
console.log('Relationships:', relations)
// Check direction
await brain.find({
- connected: {
- to: 'entity-id',
- direction: 'in' // Try 'out' or 'both'
- }
+ connected: {
+ to: 'entity-id',
+ direction: 'in' // Try 'out' or 'both'
+ }
})
// Verify verb type
await brain.find({
- connected: {
- to: 'entity-id',
- via: VerbType.WorksFor // Correct VerbType?
- }
+ connected: {
+ to: 'entity-id',
+ via: VerbType.WorksFor // Correct VerbType?
+ }
})
```
@@ -1170,12 +1272,12 @@ await brain.update(entity.id, { data: entity.data })
```typescript
// ❌ Too strict: May return nothing
await brain.find({
- near: { id: 'doc-123', threshold: 0.95 }
+ near: { id: 'doc-123', threshold: 0.95 }
})
// ✅ Reasonable: 0.7-0.85 is typical
await brain.find({
- near: { id: 'doc-123', threshold: 0.75 }
+ near: { id: 'doc-123', threshold: 0.75 }
})
```
@@ -1189,8 +1291,8 @@ console.log('Entity type:', entity.noun)
// Verify type filter is working
await brain.find({
- type: NounType.Document,
- where: { id: 'unexpected-id' } // Should not return if wrong type
+ type: NounType.Document,
+ where: { id: 'unexpected-id' } // Should not return if wrong type
})
```
@@ -1205,7 +1307,7 @@ console.log(`Results: ${results.length}, Unique: ${uniqueIds.size}`)
// Brainy should never return duplicates - report if found
```
-## VFS (Virtual File System) Visibility (v4.7.0+)
+## VFS (Virtual File System) Visibility
### Default Behavior
@@ -1226,8 +1328,8 @@ If you need to exclude VFS entities from specific queries, use the `excludeVFS`
```typescript
// Exclude VFS files from results
await brain.find({
- query: 'machine learning',
- excludeVFS: true // Only return non-file entities
+ query: 'machine learning',
+ excludeVFS: true // Only return non-file entities
})
```
@@ -1236,14 +1338,14 @@ await brain.find({
```typescript
// Explicit filtering (same as excludeVFS: true)
await brain.find({
- query: 'machine learning',
- where: { vfsType: { exists: false } }
+ query: 'machine learning',
+ where: { vfsType: { exists: false } }
})
// Or only search VFS files
await brain.find({
- query: 'setup instructions',
- where: { vfsType: 'file' } // Only files
+ query: 'setup instructions',
+ where: { vfsType: 'file' } // Only files
})
```
@@ -1254,26 +1356,62 @@ await brain.find({
- No performance penalty - same speed as any metadata filter
- Works seamlessly with vector + metadata + graph queries
+## 5. Aggregate Queries
+
+The `find()` method also supports aggregate queries via the `aggregate` parameter. When set, `find()` bypasses all vector/metadata/graph search paths and returns pre-computed aggregate results from the incremental aggregation engine.
+
+```typescript
+// Define the aggregate (once — survives restarts)
+brain.defineAggregate({
+ name: 'monthly_spending',
+ source: { type: NounType.Event, where: { domain: 'financial' } },
+ groupBy: ['category', { field: 'date', window: 'month' }],
+ metrics: {
+ total: { op: 'sum', field: 'amount' },
+ count: { op: 'count' }
+ }
+})
+
+// Query it through find()
+const results = await brain.find({
+ aggregate: 'monthly_spending',
+ where: { category: 'food' }, // Filters aggregate groups (not raw entities)
+ orderBy: 'total',
+ order: 'desc',
+ limit: 12
+})
+```
+
+**Key characteristics:**
+- **O(1) read performance** — results come from running totals, no query-time computation
+- **Updated incrementally** — every `add()`, `update()`, `delete()` updates matching aggregates
+- **Same Result[] format** — aggregate results are returned as `NounType.Measurement` entities
+- **Combinable with `where`/`orderBy`/`limit`/`offset`** — standard find() parameters apply to group filtering
+
+See the **[API Reference → Aggregation Engine](./api/README.md#aggregation-engine)** for the full API.
+
+---
+
### Migration from v4.6.x
-**BREAKING CHANGE** (v4.7.0): The `includeVFS` parameter has been removed:
+**BREAKING CHANGE**: The `includeVFS` parameter has been removed:
```typescript
// ❌ Old (v4.6.x and earlier)
await brain.find({
- query: 'docs',
- includeVFS: true // No longer needed!
+ query: 'docs',
+ includeVFS: true // No longer needed!
})
-// ✅ New (v4.7.0+)
+// ✅ New
await brain.find({
- query: 'docs' // VFS included by default
+ query: 'docs' // VFS included by default
})
// ✅ To exclude VFS (if needed)
await brain.find({
- query: 'concepts',
- excludeVFS: true
+ query: 'concepts',
+ excludeVFS: true
})
```
diff --git a/docs/METADATA_CONTRACT_IMPLEMENTATION.md b/docs/METADATA_CONTRACT_IMPLEMENTATION.md
deleted file mode 100644
index 0020990f..00000000
--- a/docs/METADATA_CONTRACT_IMPLEMENTATION.md
+++ /dev/null
@@ -1,177 +0,0 @@
-# Metadata Contract Implementation Plan
-
-## New Required Interface
-
-```typescript
-export interface BrainyAugmentation {
- // Identity
- name: string
- timing: 'before' | 'after' | 'around' | 'replace'
- operations: string[]
- priority: number
-
- // REQUIRED metadata contract
- metadata: 'none' | 'readonly' | MetadataAccess
-
- // Methods
- initialize(context: AugmentationContext): Promise
- execute(operation: string, params: any, next: () => Promise): Promise
- shouldExecute?(operation: string, params: any): boolean
- shutdown?(): Promise
-}
-
-interface MetadataAccess {
- reads?: string[] | '*' // Fields to read, or '*' for all
- writes?: string[] | '*' // Fields to write, or '*' for all
- namespace?: string // Optional: custom namespace like '_myAug'
-}
-```
-
-## Augmentation Analysis & Classification
-
-### Category 1: No Metadata Access ('none')
-These augmentations don't read or write metadata at all:
-
-1. **CacheAugmentation** - Only caches search results
-2. **RequestDeduplicatorAugmentation** - Only deduplicates requests
-3. **ConnectionPoolAugmentation** - Only manages storage connections
-4. **StorageAugmentation** - Base storage layer, metadata handled by Brainy
-
-### Category 2: Read-Only Access ('readonly')
-These augmentations read metadata but never modify it:
-
-6. **IndexAugmentation** - Reads metadata to build indexes
-7. **MonitoringAugmentation** - Reads metadata for monitoring
-8. **MetricsAugmentation** - Reads metadata for metrics collection
-9. **BatchProcessingAugmentation** - Reads metadata to check for external IDs
-10. **EntityRegistryAugmentation** - Reads metadata to register entities
-11. **AutoRegisterEntitiesAugmentation** - Reads metadata for auto-registration
-12. **ConduitAugmentation** - Reads metadata to pass through operations
-
-### Category 3: Metadata Writers (needs specific access)
-These augmentations modify metadata and need specific field declarations:
-
-13. **SynapseAugmentation** - Writes to '_synapse' field
- ```typescript
- metadata: {
- reads: '*',
- writes: ['_synapse', '_synapseTimestamp'],
- namespace: '_synapse' // Uses its own namespace
- }
- ```
-
-14. **IntelligentVerbScoringAugmentation** - Adds scoring to verbs
- ```typescript
- metadata: {
- reads: ['type', 'verb', 'source', 'target'],
- writes: ['weight', 'confidence', 'intelligentScoring']
- }
- ```
-
-15. **ServerSearchAugmentation** - Might add server metadata
- ```typescript
- metadata: {
- reads: '*',
- writes: ['_server', '_syncedAt']
- }
- ```
-
-16. **NeuralImportAugmentation** - Enriches imported data
- ```typescript
- metadata: {
- reads: '*',
- writes: ['nounType', 'verbType', '_importedAt', '_enriched']
- }
- ```
-
-### Category 4: API/Server (needs analysis)
-17. **ApiServerAugmentation** - Likely read-only for serving data
-18. **StorageAugmentations** (plural) - Collection of storage implementations
-19. **ConduitAugmentations** (plural) - Collection of conduit types
-
-## Implementation Steps
-
-### Phase 1: Update Base Interface
-1. Update `BrainyAugmentation` interface to require `metadata` field
-2. Update `BaseAugmentation` class to have abstract `metadata` property
-3. Add runtime enforcement in augmentation executor
-
-### Phase 2: Update Each Augmentation
-For each augmentation, add the appropriate metadata declaration:
-
-#### Example Updates:
-
-**CacheAugmentation:**
-```typescript
-export class CacheAugmentation extends BaseAugmentation {
- readonly name = 'cache'
- readonly metadata = 'none' as const // ✅ No metadata access
- // ... rest unchanged
-}
-```
-
-```typescript
- readonly metadata = 'readonly' as const // ✅ Only reads for logging
- // ... rest unchanged
-}
-```
-
-**SynapseAugmentation:**
-```typescript
-export abstract class SynapseAugmentation extends BaseAugmentation {
- readonly name = 'synapse'
- readonly metadata = {
- reads: '*',
- writes: ['_synapse', '_synapseTimestamp'],
- namespace: '_synapse'
- } as const
- // ... rest unchanged
-}
-```
-
-### Phase 3: Runtime Enforcement
-Add a metadata access enforcer that:
-1. Wraps metadata objects based on declared access
-2. Throws errors if augmentation violates its contract
-3. Logs warnings in development mode
-
-```typescript
-class MetadataEnforcer {
- enforce(augmentation: BrainyAugmentation, metadata: any): any {
- if (augmentation.metadata === 'none') {
- return null // No access at all
- }
-
- if (augmentation.metadata === 'readonly') {
- return Object.freeze(deepClone(metadata)) // Read-only copy
- }
-
- // For specific access, create proxy that validates
- return new Proxy(metadata, {
- set(target, prop, value) {
- const access = augmentation.metadata as MetadataAccess
- if (!access.writes?.includes(String(prop)) && access.writes !== '*') {
- throw new Error(`Augmentation '${augmentation.name}' cannot write to field '${String(prop)}'`)
- }
- target[prop] = value
- return true
- }
- })
- }
-}
-```
-
-## Benefits
-1. **Type Safety** - TypeScript enforces metadata declaration
-2. **Runtime Safety** - Violations caught immediately
-3. **Documentation** - Contract shows exactly what each augmentation does
-4. **Brain-cloud Ready** - Registry can validate augmentations
-5. **Developer Friendly** - Most use simple 'none' or 'readonly'
-
-## Migration Checklist
-- [ ] Update BrainyAugmentation interface
-- [ ] Update BaseAugmentation class
-- [ ] Add MetadataEnforcer
-- [ ] Update all 19 augmentations with metadata declarations
-- [ ] Add tests for metadata enforcement
-- [ ] Update documentation
\ No newline at end of file
diff --git a/docs/MIGRATION-V3-TO-V4.md b/docs/MIGRATION-V3-TO-V4.md
index 9b9dec3f..29c409ac 100644
--- a/docs/MIGRATION-V3-TO-V4.md
+++ b/docs/MIGRATION-V3-TO-V4.md
@@ -48,7 +48,7 @@ await storage.setLifecyclePolicy({
```typescript
// v3: Delete one at a time (slow, expensive)
for (const id of idsToDelete) {
- await brain.delete(id) // 1000 API calls for 1000 entities
+ await brain.remove(id) // 1000 API calls for 1000 entities
}
// v4.0.0: Batch delete (fast, cheap)
@@ -426,16 +426,6 @@ npm install @soulcraft/brainy@^3.50.0
7. Verify data integrity thoroughly
8. Enable lifecycle policies gradually
-### Scenario 4: Multi-Node Distributed System
-
-**Recommended approach:**
-1. Perform blue-green deployment:
- - Keep v3 nodes running (blue)
- - Deploy v4 nodes (green)
- - Migrate data once
- - Switch traffic to v4 nodes
- - Decommission v3 nodes
-
## Cost Savings After Migration
### Enable All v4.0.0 Features
diff --git a/docs/NEURAL_API_PATTERNS.md b/docs/NEURAL_API_PATTERNS.md
deleted file mode 100644
index 4edd3505..00000000
--- a/docs/NEURAL_API_PATTERNS.md
+++ /dev/null
@@ -1,736 +0,0 @@
-# 🧠 Neural API Patterns: AI-Powered Intelligence
-
-> Learn the correct patterns for Brainy's Neural API. Avoid performance pitfalls and use AI features effectively.
-
-## 🚨 Critical: Access Neural APIs Correctly
-
-### ❌ **WRONG - Outdated Access Patterns**
-
-```typescript
-// DON'T DO THIS - Outdated documentation patterns
-import { Brainy } from '@soulcraft/brainy' // ❌ Wrong import
-const brain = new Brainy() // ❌ Old class name
-
-// These may not work as expected:
-const neural = brain.neural // ❌ May be undefined
-```
-
-### ✅ **CORRECT - Modern Neural Access**
-
-```typescript
-// ✅ Use modern Brainy class
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// ✅ Neural API is available after initialization
-const clusters = await brain.neural.clusters()
-const similarity = await brain.neural.similar('item1', 'item2')
-```
-
-## 🔍 Similarity Analysis Patterns
-
-### ❌ **WRONG - Inefficient Similarity Checks**
-
-```typescript
-// DON'T DO THIS - N² comparisons
-const items = await brain.find({ limit: 1000 })
-const similarities = []
-
-for (const item1 of items) {
- for (const item2 of items) {
- if (item1.id !== item2.id) {
- const sim = await brain.neural.similar(item1.id, item2.id) // ❌ Millions of calls
- similarities.push({ from: item1.id, to: item2.id, score: sim })
- }
- }
-}
-```
-
-### ✅ **CORRECT - Efficient Similarity Patterns**
-
-```typescript
-// ✅ Pattern 1: Find neighbors (much more efficient)
-const item = await brain.get('target-item-id')
-const neighbors = await brain.neural.neighbors(item.id, {
- limit: 10, // Top 10 most similar
- threshold: 0.7, // Minimum similarity
- includeScores: true // Include similarity scores
-})
-
-console.log(`Found ${neighbors.length} similar items`)
-
-// ✅ Pattern 2: Batch similarity for specific pairs
-const itemPairs = [
- ['item1', 'item2'],
- ['item1', 'item3'],
- ['item2', 'item3']
-]
-
-const similarities = await Promise.all(
- itemPairs.map(async ([a, b]) => ({
- from: a,
- to: b,
- score: await brain.neural.similar(a, b)
- }))
-)
-
-// ✅ Pattern 3: Text-to-text similarity (no need for IDs)
-const textSimilarity = await brain.neural.similar(
- "Machine learning is fascinating",
- "AI and deep learning are interesting",
- { detailed: true } // Get explanation of similarity
-)
-
-console.log(`Similarity: ${textSimilarity.score}`)
-console.log(`Explanation: ${textSimilarity.explanation}`)
-
-// ✅ Pattern 4: Vector-level similarity for optimization
-const vector1 = await brain.embed("First concept")
-const vector2 = await brain.embed("Second concept")
-const vectorSimilarity = await brain.neural.similar(vector1, vector2)
-```
-
-## 🎯 Clustering Patterns
-
-### ❌ **WRONG - Uncontrolled Clustering**
-
-```typescript
-// DON'T DO THIS - Clustering everything without limits
-const everything = await brain.find({ limit: 100000 }) // ❌ Too much data
-const clusters = await brain.neural.clusters() // ❌ May crash or timeout
-```
-
-### ✅ **CORRECT - Smart Clustering Patterns**
-
-```typescript
-// ✅ Pattern 1: Controlled clustering with limits
-const recentItems = await brain.find({
- where: {
- createdAt: { $gte: Date.now() - 30 * 24 * 60 * 60 * 1000 } // Last 30 days
- },
- limit: 1000 // Reasonable limit
-})
-
-const clusters = await brain.neural.clusters(
- recentItems.map(item => item.id),
- {
- algorithm: 'kmeans', // Reliable algorithm
- maxClusters: 10, // Reasonable number
- threshold: 0.75, // High similarity required
- iterations: 50 // Convergence limit
- }
-)
-
-// ✅ Pattern 2: Domain-specific clustering
-const techDocs = await brain.find({
- where: { category: 'technology', type: 'document' },
- limit: 500
-})
-
-const techClusters = await brain.neural.clusterByDomain(
- 'category', // Group by this field
- {
- items: techDocs.map(doc => doc.id),
- minClusterSize: 3, // Minimum items per cluster
- maxClusters: 8
- }
-)
-
-// ✅ Pattern 3: Temporal clustering for time-series data
-const timebasedClusters = await brain.neural.clusterByTime(
- 'createdAt', // Time field
- 'week', // Time window (hour, day, week, month)
- {
- items: recentItems.map(item => item.id),
- overlap: 0.2, // 20% overlap between windows
- minPerWindow: 5 // Minimum items per time window
- }
-)
-
-// ✅ Pattern 4: Streaming clustering for large datasets
-async function clusterLargeDataset() {
- const clusterStream = brain.neural.clusterStream({
- batchSize: 100, // Process 100 items at a time
- updateInterval: 1000, // Update clusters every 1000 items
- maxMemory: 512 * 1024 * 1024 // 512MB memory limit
- })
-
- const allClusters = []
- for await (const batch of clusterStream) {
- console.log(`Processed ${batch.processed} items, found ${batch.clusters.length} clusters`)
- allClusters.push(...batch.clusters)
- }
-
- return allClusters
-}
-```
-
-## 🔍 Neighbor Discovery Patterns
-
-### ❌ **WRONG - Manual Similarity Searches**
-
-```typescript
-// DON'T DO THIS - Reinventing neighbor search
-async function findSimilarManually(targetId: string) {
- const allItems = await brain.find({ limit: 10000 }) // ❌ Load everything
- const similarities = []
-
- for (const item of allItems) {
- if (item.id !== targetId) {
- const score = await brain.neural.similar(targetId, item.id) // ❌ Slow
- if (score > 0.7) {
- similarities.push({ id: item.id, score })
- }
- }
- }
-
- return similarities.sort((a, b) => b.score - a.score).slice(0, 10) // ❌ Inefficient
-}
-```
-
-### ✅ **CORRECT - Optimized Neighbor Patterns**
-
-```typescript
-// ✅ Pattern 1: Basic neighbor search
-const neighbors = await brain.neural.neighbors('target-item-id', {
- limit: 20, // Top 20 neighbors
- threshold: 0.6, // Minimum similarity
- includeMetadata: true, // Include item metadata
- includeDistances: true // Include exact similarity scores
-})
-
-// ✅ Pattern 2: Filtered neighbor search
-const filteredNeighbors = await brain.neural.neighbors('article-id', {
- limit: 10,
- filter: {
- type: 'document', // Only find similar documents
- status: 'published', // Only published content
- language: 'en' // Only English content
- },
- excludeIds: ['self-id', 'duplicate-id'] // Exclude specific items
-})
-
-// ✅ Pattern 3: Multi-level neighbor discovery
-async function discoverNeighborNetwork(rootId: string, maxDepth = 2) {
- const network = new Map()
- const visited = new Set()
- const queue = [{ id: rootId, depth: 0 }]
-
- while (queue.length > 0) {
- const { id, depth } = queue.shift()!
-
- if (visited.has(id) || depth >= maxDepth) continue
- visited.add(id)
-
- const neighbors = await brain.neural.neighbors(id, {
- limit: 5,
- threshold: 0.8
- })
-
- network.set(id, neighbors)
-
- // Add neighbors to queue for next depth level
- if (depth < maxDepth - 1) {
- neighbors.forEach(neighbor => {
- queue.push({ id: neighbor.id, depth: depth + 1 })
- })
- }
- }
-
- return network
-}
-
-// ✅ Pattern 4: Recommendation engine
-async function getRecommendations(userId: string) {
- // Get user's liked items
- const userItems = await brain.find({
- connected: { to: userId, via: 'liked-by' }
- })
-
- // Find neighbors for each liked item
- const allNeighbors = await Promise.all(
- userItems.map(item =>
- brain.neural.neighbors(item.id, {
- limit: 10,
- threshold: 0.7,
- excludeConnected: { to: userId, via: 'liked-by' } // Exclude already liked
- })
- )
- )
-
- // Aggregate and rank recommendations
- const recommendations = new Map()
- allNeighbors.flat().forEach(neighbor => {
- const current = recommendations.get(neighbor.id) || { score: 0, count: 0 }
- current.score += neighbor.score
- current.count += 1
- recommendations.set(neighbor.id, current)
- })
-
- // Return top recommendations by average score
- return Array.from(recommendations.entries())
- .map(([id, stats]) => ({
- id,
- avgScore: stats.score / stats.count,
- mentions: stats.count
- }))
- .sort((a, b) => b.avgScore - a.avgScore)
- .slice(0, 10)
-}
-```
-
-## 🏗️ Hierarchy & Structure Patterns
-
-### ❌ **WRONG - Manual Hierarchy Building**
-
-```typescript
-// DON'T DO THIS - Building hierarchies manually
-async function buildHierarchyManually(rootId: string) {
- const root = await brain.get(rootId)
- const allItems = await brain.find({ limit: 1000 }) // ❌ Load everything
-
- // Manual tree building with nested loops
- const hierarchy = { root, children: [] }
- // ... complex manual logic
-}
-```
-
-### ✅ **CORRECT - Semantic Hierarchy Patterns**
-
-```typescript
-// ✅ Pattern 1: Automatic semantic hierarchy
-const hierarchy = await brain.neural.hierarchy('root-concept-id', {
- maxDepth: 4, // Maximum tree depth
- minSimilarity: 0.6, // Minimum similarity for inclusion
- branchingFactor: 5, // Maximum children per node
- algorithm: 'semantic' // Use semantic clustering
-})
-
-// ✅ Pattern 2: Domain-specific hierarchy
-const techHierarchy = await brain.neural.hierarchy('technology-id', {
- filter: { category: 'technology' },
- weights: {
- semantic: 0.7, // 70% based on content similarity
- metadata: 0.3 // 30% based on metadata similarity
- },
- includeMetrics: true // Include hierarchy quality metrics
-})
-
-// ✅ Pattern 3: Multi-root hierarchy for complex domains
-async function buildMultiRootHierarchy(rootIds: string[]) {
- const hierarchies = await Promise.all(
- rootIds.map(rootId =>
- brain.neural.hierarchy(rootId, {
- maxDepth: 3,
- crossReference: true // Allow cross-hierarchy connections
- })
- )
- )
-
- // Merge hierarchies and find connections
- const merged = {
- roots: hierarchies,
- connections: await findCrossHierarchyConnections(hierarchies)
- }
-
- return merged
-}
-
-async function findCrossHierarchyConnections(hierarchies: any[]) {
- const connections = []
-
- for (let i = 0; i < hierarchies.length; i++) {
- for (let j = i + 1; j < hierarchies.length; j++) {
- const leafNodes1 = extractLeafNodes(hierarchies[i])
- const leafNodes2 = extractLeafNodes(hierarchies[j])
-
- // Find connections between leaf nodes of different hierarchies
- for (const leaf1 of leafNodes1) {
- const neighbors = await brain.neural.neighbors(leaf1.id, {
- limit: 5,
- threshold: 0.8,
- filter: { id: { $in: leafNodes2.map(l => l.id) } }
- })
-
- connections.push(...neighbors.map(n => ({
- from: leaf1.id,
- to: n.id,
- hierarchyPair: [i, j],
- similarity: n.score
- })))
- }
- }
- }
-
- return connections
-}
-```
-
-## 🚨 Outlier Detection Patterns
-
-### ❌ **WRONG - Manual Outlier Detection**
-
-```typescript
-// DON'T DO THIS - Manual statistical outlier detection
-async function findOutliersManually() {
- const items = await brain.find({ limit: 1000 })
- const similarities = []
-
- // Calculate average similarity for each item (expensive)
- for (const item of items) {
- let totalSim = 0
- let count = 0
- for (const other of items) {
- if (item.id !== other.id) {
- totalSim += await brain.neural.similar(item.id, other.id) // ❌ N² operations
- count++
- }
- }
- similarities.push({ id: item.id, avgSimilarity: totalSim / count })
- }
-
- // Manual outlier calculation
- const threshold = calculateManualThreshold(similarities) // ❌ Complex statistics
- return similarities.filter(s => s.avgSimilarity < threshold)
-}
-```
-
-### ✅ **CORRECT - AI-Powered Outlier Detection**
-
-```typescript
-// ✅ Pattern 1: Automatic outlier detection
-const outliers = await brain.neural.outliers({
- threshold: 0.3, // Items with < 30% avg similarity to others
- method: 'isolation-forest', // AI-based outlier detection
- contamination: 0.1, // Expect ~10% outliers
- includeReasons: true // Explain why each item is an outlier
-})
-
-console.log(`Found ${outliers.length} outliers`)
-outliers.forEach(outlier => {
- console.log(`Outlier: ${outlier.id}, Score: ${outlier.score}`)
- console.log(`Reason: ${outlier.reason}`)
-})
-
-// ✅ Pattern 2: Domain-specific outlier detection
-const techOutliers = await brain.neural.outliers({
- filter: { category: 'technology' },
- features: ['content', 'metadata.tags', 'metadata.complexity'],
- method: 'local-outlier-factor',
- neighbors: 20 // Consider 20 nearest neighbors
-})
-
-// ✅ Pattern 3: Temporal outlier detection
-const recentOutliers = await brain.neural.outliers({
- timeWindow: '7days', // Look at last 7 days
- baseline: '30days', // Compare to 30-day baseline
- method: 'statistical', // Use statistical methods
- autoThreshold: true // Automatically determine threshold
-})
-
-// ✅ Pattern 4: Streaming outlier detection
-async function detectOutliersInStream() {
- const outlierStream = brain.neural.outlierStream({
- batchSize: 50,
- updateInterval: 100, // Check every 100 new items
- adaptiveThreshold: true // Threshold adapts as data changes
- })
-
- for await (const batch of outlierStream) {
- console.log(`Batch ${batch.batchNumber}: ${batch.outliers.length} outliers detected`)
-
- // Process outliers immediately
- for (const outlier of batch.outliers) {
- await handleOutlier(outlier)
- }
- }
-}
-
-async function handleOutlier(outlier: any) {
- // Flag for manual review
- await brain.update(outlier.id, {
- metadata: {
- flagged: true,
- outlierScore: outlier.score,
- outlierReason: outlier.reason,
- flaggedAt: Date.now()
- }
- })
-}
-```
-
-## 📊 Visualization Patterns
-
-### ❌ **WRONG - Manual Visualization Data Preparation**
-
-```typescript
-// DON'T DO THIS - Manual coordinate calculation
-async function prepareVisualizationManually() {
- const items = await brain.find({ limit: 500 })
- const coordinates = []
-
- // Manual dimensionality reduction (complex math)
- for (const item of items) {
- const vector = await brain.embed(item.data)
- // Complex PCA/t-SNE calculations manually
- const x = complexMathFunction(vector) // ❌ Error-prone
- const y = anotherComplexFunction(vector)
- coordinates.push({ id: item.id, x, y })
- }
-
- return coordinates
-}
-```
-
-### ✅ **CORRECT - AI-Powered Visualization**
-
-```typescript
-// ✅ Pattern 1: Automatic 2D visualization
-const visualization = await brain.neural.visualize({
- dimensions: 2, // 2D plot
- algorithm: 'umap', // UMAP for better clustering preservation
- maxItems: 1000, // Performance limit
- includeMetadata: true, // Include item metadata in output
- colorBy: 'cluster' // Color points by cluster membership
-})
-
-// Result format:
-// {
-// points: [{ id, x, y, cluster, metadata }, ...],
-// clusters: [{ id, centroid: [x, y], members: [...] }, ...],
-// stats: { stress, kruskalStress, trustworthiness }
-// }
-
-// ✅ Pattern 2: 3D visualization for complex data
-const viz3D = await brain.neural.visualize({
- dimensions: 3,
- algorithm: 'tsne',
- perplexity: 30, // t-SNE parameter
- learningRate: 200, // t-SNE learning rate
- iterations: 1000 // Number of optimization steps
-})
-
-// ✅ Pattern 3: Interactive visualization with filtering
-const interactiveViz = await brain.neural.visualize({
- filter: {
- type: 'document',
- createdAt: { $gte: Date.now() - 7 * 24 * 60 * 60 * 1000 }
- },
- groupBy: 'category', // Group points by metadata field
- showLabels: true, // Include text labels
- labelField: 'title', // Field to use for labels
- includeEdges: true, // Show connections between similar items
- edgeThreshold: 0.8 // Only show high-similarity connections
-})
-
-// ✅ Pattern 4: Real-time visualization updates
-class LiveVisualization {
- private visualization: any = null
- private updateInterval: NodeJS.Timeout | null = null
-
- async start() {
- // Initial visualization
- this.visualization = await brain.neural.visualize({
- dimensions: 2,
- algorithm: 'umap',
- maxItems: 500,
- includeMetadata: true
- })
-
- // Update every 30 seconds
- this.updateInterval = setInterval(async () => {
- await this.update()
- }, 30000)
- }
-
- async update() {
- // Get recent items
- const recentItems = await brain.find({
- where: {
- createdAt: { $gte: Date.now() - 30000 } // Last 30 seconds
- },
- limit: 50
- })
-
- if (recentItems.length > 0) {
- // Incrementally update visualization
- const updates = await brain.neural.updateVisualization(
- this.visualization.id,
- {
- newItems: recentItems.map(item => item.id),
- algorithm: 'incremental' // Faster incremental updates
- }
- )
-
- this.visualization = { ...this.visualization, ...updates }
- this.onUpdate(updates)
- }
- }
-
- onUpdate(updates: any) {
- // Emit updates to frontend
- console.log(`Visualization updated: ${updates.newPoints.length} new points`)
- }
-
- stop() {
- if (this.updateInterval) {
- clearInterval(this.updateInterval)
- }
- }
-}
-```
-
-## 🚀 Performance Optimization Patterns
-
-### ✅ **High-Performance Neural Operations**
-
-```typescript
-// ✅ Pattern 1: Batch processing for similarity
-async function batchSimilarityCalculation(itemPairs: Array<[string, string]>) {
- const batchSize = 100
- const results = []
-
- for (let i = 0; i < itemPairs.length; i += batchSize) {
- const batch = itemPairs.slice(i, i + batchSize)
-
- const batchResults = await Promise.all(
- batch.map(async ([a, b]) => ({
- from: a,
- to: b,
- similarity: await brain.neural.similar(a, b)
- }))
- )
-
- results.push(...batchResults)
-
- // Progress reporting
- console.log(`Processed ${Math.min(i + batchSize, itemPairs.length)}/${itemPairs.length} pairs`)
- }
-
- return results
-}
-
-// ✅ Pattern 2: Caching expensive operations
-class NeuralCache {
- private clusterCache = new Map()
- private similarityCache = new Map()
- private readonly TTL = 5 * 60 * 1000 // 5 minutes
-
- async getClusters(options: any) {
- const key = JSON.stringify(options)
- const cached = this.clusterCache.get(key)
-
- if (cached && Date.now() - cached.timestamp < this.TTL) {
- return cached.data
- }
-
- const clusters = await brain.neural.clusters(undefined, options)
- this.clusterCache.set(key, {
- data: clusters,
- timestamp: Date.now()
- })
-
- return clusters
- }
-
- async getSimilarity(id1: string, id2: string) {
- // Create consistent cache key regardless of order
- const key = [id1, id2].sort().join('-')
- const cached = this.similarityCache.get(key)
-
- if (cached && Date.now() - cached.timestamp < this.TTL) {
- return cached.data
- }
-
- const similarity = await brain.neural.similar(id1, id2)
- this.similarityCache.set(key, {
- data: similarity,
- timestamp: Date.now()
- })
-
- return similarity
- }
-}
-
-// ✅ Pattern 3: Memory-efficient streaming
-async function processLargeDatasetEfficiently() {
- const stream = brain.neural.clusterStream({
- batchSize: 50, // Small batches for memory efficiency
- maxMemoryMB: 256, // Memory limit
- diskCache: true, // Use disk for temporary storage
- compression: true // Compress cached data
- })
-
- const results = []
- let totalProcessed = 0
-
- for await (const batch of stream) {
- // Process batch immediately, don't accumulate in memory
- const processedBatch = await processBatch(batch)
-
- // Save to disk or send to another service
- await saveBatchToDisk(processedBatch)
-
- totalProcessed += batch.items.length
- console.log(`Processed ${totalProcessed} items`)
-
- // Clear memory
- batch.items = null
- }
-
- return { totalProcessed }
-}
-
-// ✅ Pattern 4: Parallel processing with worker threads
-async function parallelNeuralProcessing(items: string[]) {
- const numWorkers = require('os').cpus().length
- const batchSize = Math.ceil(items.length / numWorkers)
-
- const workers = []
- for (let i = 0; i < numWorkers; i++) {
- const batch = items.slice(i * batchSize, (i + 1) * batchSize)
- if (batch.length > 0) {
- workers.push(processWorkerBatch(batch))
- }
- }
-
- const results = await Promise.all(workers)
- return results.flat()
-}
-
-async function processWorkerBatch(batch: string[]) {
- // This would run in a worker thread in real implementation
- return Promise.all(
- batch.map(async itemId => {
- const neighbors = await brain.neural.neighbors(itemId, { limit: 5 })
- return { itemId, neighbors }
- })
- )
-}
-```
-
-## 🎯 Summary: Neural API Best Practices
-
-| ❌ **Avoid These Patterns** | ✅ **Use These Instead** |
-|---------------------------|------------------------|
-| Manual similarity loops | `brain.neural.neighbors()` |
-| Uncontrolled clustering | Limit items and set maxClusters |
-| Manual outlier detection | `brain.neural.outliers()` |
-| Manual visualization prep | `brain.neural.visualize()` |
-| Loading entire datasets | Streaming and batch processing |
-| No caching | Cache expensive operations |
-| Blocking operations | Parallel and async patterns |
-
----
-
-**🎉 Following these patterns gives you:**
-- 🚀 **Optimized performance** with intelligent algorithms
-- 🧠 **AI-powered insights** instead of manual statistics
-- 📊 **Rich visualizations** for data exploration
-- 🎯 **Accurate clustering** with semantic understanding
-- 🚨 **Smart outlier detection** for quality control
-- ⚡ **Scalable processing** for large datasets
-
-**Next:** [Augmentation Patterns →](./AUGMENTATION_PATTERNS.md) | [Core API Patterns →](./CORE_API_PATTERNS.md)
\ No newline at end of file
diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md
index 15344b1e..248a2c70 100644
--- a/docs/PERFORMANCE.md
+++ b/docs/PERFORMANCE.md
@@ -2,21 +2,23 @@
## Performance Characteristics
-Brainy achieves industry-leading performance through carefully optimized data structures and algorithms. All performance claims are verified through actual benchmarks on production code.
+Brainy achieves high performance through carefully optimized data structures and algorithms. The tables below describe each component by its **algorithmic complexity** — the durable, defensible guarantee. The example latencies are figures from a single 100-item run on one machine (see [Benchmarks](#benchmarks)); they are illustrative, not a committed benchmark, and vary with hardware, embedding model, and storage backend. The one component with a committed scale assertion is the graph adjacency index (`tests/performance/graph-scale-performance.test.ts:238`).
### Core Performance Summary
-| Component | Operation | Time Complexity | Measured Performance | Data Structure |
+| Component | Operation | Time Complexity | Example latency (100-item run)\* | Data Structure |
|-----------|-----------|-----------------|---------------------|----------------|
| **Metadata Index** | Exact match | **O(1)** | 0.8ms | `Map>` |
| **Metadata Index** | Range query | **O(log n) + O(k)** | 0.6ms | Sorted array + binary search |
| **Graph Index** | Get neighbors | **O(1)** | 0.09ms | `Map>` |
-| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | HNSW hierarchical graph |
+| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | Hierarchical graph |
| **NLP Parser** | Query parsing | **O(m)** | 8.9ms | 220 pre-computed patterns |
| **Type-Field Affinity** | Field matching | **O(f)** | 0.1ms | Type-specific field cache |
| **Type Detection** | Noun/Verb matching | **O(t)** | 0.3ms | Pre-embedded type vectors |
| **Triple Intelligence** | Combined query | **O(1) to O(log n)** | 1.8ms | Parallel execution |
+\* Illustrative single-run figures at 100 items on one machine — not a committed benchmark. Only the graph index carries an asserted scale bound (measured <1 ms per neighbor lookup up to 1M relationships, `tests/performance/graph-scale-performance.test.ts:238`).
+
Where:
- `n` = number of items in index
- `k` = number of results returned
@@ -24,27 +26,34 @@ Where:
- `f` = number of fields for entity type
- `t` = number of types (42 nouns, 127 verbs)
-### v5.11.1: brain.get() Metadata-Only Optimization
+### brain.get() Metadata-Only Optimization
-✨ **Massive Performance Improvement**: `brain.get()` is now **76-81% faster** by default!
+`brain.get()` returns **metadata only by default**, skipping the 384-dimensional
+embedding — the bulk of an entity's payload. Callers that need the vector opt in
+with `{ includeVectors: true }`.
-| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup | Use Case |
-|-----------|------------------|-----------------|---------|----------|
-| **brain.get() (metadata-only)** | 43ms, 6KB | **10ms, 300 bytes** | **76-81%** | VFS, existence checks, metadata |
-| **brain.get({ includeVectors: true })** | 43ms, 6KB | 43ms, 6KB | 0% | Similarity calculations |
-| **VFS readFile()** | 53ms | **~13ms** | **75%** | File operations |
-| **VFS readdir(100 files)** | 5.3s | **~1.3s** | **75%** | Directory listings |
+| Operation | Default (metadata-only) | With `includeVectors: true` | Use Case |
+|-----------|-------------------------|-----------------------------|----------|
+| **brain.get()** | Skips vector load | Loads full vector | VFS, existence checks, metadata |
+| **VFS readFile() / readdir()** | Inherits metadata-only path | n/a | File operations, directory listings |
-**Key Innovation**: Lazy vector loading - only load 384-dimensional embeddings when explicitly needed.
+**Key Innovation**: Lazy vector loading — only load the 384-dimensional embedding when explicitly needed.
+
+The integration test `tests/integration/metadata-only-comprehensive.test.ts:306`
+asserts metadata-only `get()` is faster than the full-entity `get()`
+(`metadataTime < fullTime`). The *magnitude* of the speedup is
+environment-dependent (the percentage assertion in
+`tests/integration/vfs-performance-v5.11.1.test.ts` is intentionally skipped on
+CI for that reason), so no fixed percentage is quoted here.
**Why this matters**:
-- **94% of brain.get() calls** don't need vectors (VFS, admin tools, import utilities, data APIs)
-- **Vector data is 95% of entity size** (6KB vectors vs 300 bytes metadata)
-- **Zero code changes** for most applications - automatic speedup!
+- Most `brain.get()` calls don't need vectors (VFS, admin tools, import utilities, data APIs)
+- The embedding dominates an entity's serialized size, so skipping it is the largest win
+- **Zero code changes** for most applications — automatic by default
**When to use what**:
```typescript
-// DEFAULT: Metadata-only (76-81% faster) - use for:
+// DEFAULT: Metadata-only (skips the vector load) - use for:
const entity = await brain.get(id)
// - VFS operations (readFile, stat, readdir)
// - Existence checks: if (await brain.get(id)) ...
@@ -55,7 +64,7 @@ const entity = await brain.get(id)
const entity = await brain.get(id, { includeVectors: true })
// - Computing similarity on THIS entity
// - Manual vector operations
-// - HNSW graph traversal
+// - Vector index graph traversal
```
## Architecture Deep Dive
@@ -143,14 +152,14 @@ class GraphAdjacencyIndex {
**Key Innovation:** Pure Map/Set operations - no database queries, no loops, just direct memory access.
-### 4. HNSW Vector Search - O(log n)
+### 4. Vector Index - O(log n)
-Hierarchical Navigable Small World graphs provide logarithmic approximate nearest neighbor search:
+The default vector index (`JsHnswVectorIndex`) provides logarithmic approximate nearest neighbor search through a hierarchical graph:
```typescript
-class HNSWIndex {
+class JsHnswVectorIndex {
private nouns: Map = new Map()
-
+
interface HNSWNoun {
id: string
vector: number[]
@@ -238,7 +247,7 @@ const results = await Promise.all(searchPromises)
|-----------|--------------|---------|
| Metadata Index | ~40 bytes/entry | `(key_size + 8) × unique_values + 8 × total_items` |
| Graph Index | ~24 bytes/edge | `16 × edges + 8 × nodes` |
-| HNSW | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
+| Vector Index | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
| Pattern Library | 394KB fixed | Pre-computed, shared across instances |
| Type Embeddings | ~60KB fixed | 70 types × 384 dimensions × 4 bytes, cached |
| Field Embeddings | ~5KB dynamic | Actual fields × 384 dimensions × 4 bytes |
@@ -252,34 +261,40 @@ const results = await Promise.all(searchPromises)
## Benchmarks
-### Real-world Performance Test (100 items)
+### Illustrative Single Run (100 items, one machine)
+
+Example output from a single 100-item run — illustrative only, not a committed
+benchmark; absolute numbers vary by hardware. The values feed the
+[Core Performance Summary](#core-performance-summary) example-latency column.
```
-📊 Metadata exact match: 0.818ms (50 items matched)
-📊 Metadata range query: 0.631ms (40 items in range)
-🔗 Graph neighbor lookup: 0.092ms (2 connections)
-🎯 Vector k-NN search: 1.773ms (10 nearest neighbors)
-🧠 NLP query parsing: 8.906ms (full natural language)
-⚡ Triple Intelligence: 1.830ms (combined query)
+Metadata exact match: 0.818ms (50 items matched)
+Metadata range query: 0.631ms (40 items in range)
+Graph neighbor lookup: 0.092ms (2 connections)
+Vector k-NN search: 1.773ms (10 nearest neighbors)
+NLP query parsing: 8.906ms (full natural language)
+Triple Intelligence: 1.830ms (combined query)
```
### Scaling Characteristics
-| Items | Metadata O(1) | Range O(log n) | Graph O(1) | Vector O(log n) |
-|-------|---------------|----------------|------------|-----------------|
-| 100 | 0.8ms | 0.6ms | 0.09ms | 1.8ms |
-| 1,000 | 0.8ms | 0.9ms | 0.09ms | 2.5ms |
-| 10,000 | 0.8ms | 1.2ms | 0.09ms | 3.2ms |
-| 100,000 | 0.8ms | 1.5ms | 0.09ms | 4.1ms |
-| 1,000,000 | 0.8ms | 1.8ms | 0.09ms | 5.0ms |
+Each stage scales by its algorithmic complexity, not a fixed millisecond figure
+— absolute latency depends on hardware, embedding model, and storage backend.
+Only the graph adjacency index carries a committed scale assertion:
-*Note: O(1) operations maintain constant time regardless of scale*
+| Query stage | Complexity | Scaling behavior |
+|-------------|------------|------------------|
+| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
+| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
+| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
+| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
+| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
## Comparison with Other Systems
| System | Metadata Filter | Graph Traversal | Vector Search | Natural Language |
|--------|-----------------|-----------------|---------------|------------------|
-| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) HNSW | 220 patterns |
+| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) vector index | 220 patterns |
| Neo4j | O(log n) B-tree | O(k) traversal | Not native | Not native |
| Elasticsearch | O(log n) inverted | Not native | O(n) brute force* | Basic tokenization |
| PostgreSQL | O(log n) B-tree | O(k) recursive | O(n) brute force* | Full-text only |
@@ -305,10 +320,10 @@ const results = await Promise.all(searchPromises)
- ✅ **No Network Calls**: Everything runs locally, including embeddings
- ✅ **Thread-Safe**: Immutable data structures where possible
- ✅ **Memory Bounded**: Configurable cache sizes and automatic cleanup
-- ✅ **Horizontally Scalable**: Stateless operations support clustering
+- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
- ✅ **Zero Stubs**: Every line of code is production-ready
-## Lazy Loading Performance (v5.7.7+)
+## Lazy Loading Performance
Brainy supports two initialization modes for optimal performance across different use cases:
@@ -324,7 +339,7 @@ await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
- First query: Instant (indexes already loaded)
- Use case: Traditional applications, long-running servers
-### Mode 2: Lazy Loading (v5.7.7+)
+### Mode 2: Lazy Loading
```javascript
const brain = new Brainy({ disableAutoRebuild: true })
@@ -375,110 +390,47 @@ const brain = new Brainy({ disableAutoRebuild: true })
await brain.init() // Instant (0-10ms)
```
-### Automatic Self-Tuning (Current & Planned)
+### Automatic Self-Tuning
-**✅ Currently Implemented:**
- **Metadata Index**: Auto-builds sorted indices for range queries on first use
- **Graph Index**: Auto-flushes every 30 seconds
-- **Default Tuning**: Research-based defaults (M=16, ef=200)
+- **Default Tuning**: Research-based vector index defaults
- **Lazy Loading**: Indices built only when needed
- **Cache Management**: LRU caches with TTL
-**🚧 Planned Enhancements:**
-- **Dynamic Storage Selection**: Auto-switch between memory/disk based on size
-- **Adaptive Index Parameters**: Adjust M and ef based on query patterns
-- **Smart Cache Sizing**: Scale caches based on available memory
-- **Predictive Optimization**: Learn from usage patterns
-
### Intelligent Defaults
-All defaults are research-based and production-tested:
-- **HNSW M=16**: Optimal balance of recall/speed for most datasets
-- **efConstruction=200**: High quality graph construction
-- **Cache TTL=5min**: Balances freshness with performance
-- **Flush Interval=30s**: Non-blocking background persistence
+- **Vector recall** = `'balanced'` (M=16, ef=200): right for most datasets
+- **Cache TTL** = 5 min: balances freshness and performance
+- **Flush interval** = 30 s: non-blocking background persistence
-### Progressive Enhancement
+### Vector Index Tuning Knobs
-Brainy learns and improves over time:
-1. **Query Pattern Learning**: Frequently used patterns get cached
-2. **Index Optimization**: Auto-rebuilds indices when fragmented
-3. **Memory Management**: Coordinates caches across all components
-4. **Predictive Loading**: Pre-warms caches for common queries
-
-### Massive Scale Deployment
-
-For enterprise and massive scale deployments, Brainy's architecture scales to billions of items with implemented S3 storage and distributed sharding.
-
-**Currently Implemented:**
-- Memory storage (production-ready)
-- Disk storage (production-ready)
-- S3-compatible storage (AWS S3, Cloudflare R2, Google Cloud Storage, MinIO, Backblaze B2)
-- Distributed sharding with ConsistentHashRing
-- Single-node deployment (scales to ~1M items)
-- Multi-node deployment with sharding (scales to billions)
-
-**Available Today:**
+Brainy 8.0 exposes two knobs on `config.vector`:
```javascript
-// S3-compatible storage for unlimited scale - WORKS NOW
-const brain = new Brainy({
- storage: {
- type: 's3',
- bucketName: 'my-brainy-data',
- region: 'us-east-1',
- credentials: {
- accessKeyId: 'YOUR_ACCESS_KEY',
- secretAccessKey: 'YOUR_SECRET_KEY'
- }
- // Works with: AWS S3, MinIO, Cloudflare R2, Backblaze B2, Google Cloud Storage
- }
-})
-
-// Cloudflare R2 storage - WORKS NOW
-const brain = new Brainy({
- storage: {
- type: 'r2',
- bucketName: 'my-brainy-data',
- accountId: 'YOUR_ACCOUNT_ID',
- accessKeyId: 'YOUR_R2_ACCESS_KEY',
- secretAccessKey: 'YOUR_R2_SECRET_KEY'
- }
-})
-
-// Google Cloud Storage - WORKS NOW
-const brain = new Brainy({
- storage: {
- type: 'gcs',
- bucketName: 'my-brainy-data',
- region: 'us-central1',
- credentials: {
- accessKeyId: 'YOUR_ACCESS_KEY',
- secretAccessKey: 'YOUR_SECRET_KEY'
- }
+const brain = new Brainy({
+ vector: {
+ recall: 'fast', // 'fast' | 'balanced' | 'accurate'
+ persistMode: 'deferred' // 'immediate' | 'deferred'
}
})
```
+The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cor`) can replace it with a higher-performing implementation; the public knobs stay the same.
+
### Scale Scenarios
-| Scale | Items | Storage Strategy | Performance | Status |
-|-------|-------|-----------------|-------------|--------|
-| **Small** | <10K | Memory (automatic) | Sub-millisecond | ✅ Implemented |
-| **Medium** | 10K-1M | Disk with memory cache | 1-5ms | ✅ Implemented |
-| **Large** | 1M-100M | S3 with memory cache | 2-10ms | ✅ Implemented |
-| **Massive** | 100M-10B | S3 + distributed sharding | 5-20ms | ✅ Implemented |
-| **Planetary** | 10B+ | Multi-region S3 + Edge cache | 10-50ms | 🚧 Roadmap |
+| Scale | Items | Storage Strategy | Performance |
+|-------|-------|------------------|-------------|
+| **Small** | <10K | Memory | Sub-millisecond |
+| **Medium** | 10K-1M | Filesystem | 1-5ms |
+| **Large** | 1M-10M | Filesystem + tuned cache | 2-10ms |
+| **Massive** | 10M+ | Filesystem + native vector provider + service-layer sharding | 5-20ms |
-### S3-Compatible Storage Benefits
+For >10M entities, run multiple Brainy processes behind your own routing layer — Brainy 8.0 doesn't ship cluster coordination.
-- **Unlimited Scale**: No practical limit on dataset size
-- **Cost Effective**: $0.023/GB/month for standard storage
-- **Durability**: 99.999999999% (11 9's) durability
-- **Global**: Multi-region replication available
-- **Compatible**: Works with any S3-compatible API (MinIO, R2, B2)
-
-### Distributed Architecture (Implemented)
+### Architecture
```
┌─────────────────────────────────────────┐
@@ -490,103 +442,51 @@ const brain = new Brainy({
│ Brainy Core │
│ (Triple Intelligence Engine) │
├─────────────────────────────────────────┤
-│ Memory │ Shard │ Metadata │
-│ Cache │ Manager │ Index │
+│ Memory │ Vector │ Metadata │
+│ Cache │ Index │ Index │
└─────────────┬───────────────────────────┘
│
┌─────────────▼───────────────────────────┐
│ Storage Layer │
├──────────┬──────────┬──────────────────┤
-│ HNSW │ Graph │ Objects │
-│ Vectors │ Edges │ (S3/R2/GCS) │
+│ Vectors │ Graph │ Files │
+│ (sharded)│ Edges │ (filesystem) │
└──────────┴──────────┴──────────────────┘
```
-**Distributed Sharding (Implemented):**
-- ConsistentHashRing with 150 virtual nodes
-- 64 shards by default
-- Replication factor of 3
-- Automatic rebalancing on node addition/removal
-
-### Auto-Sharding for Horizontal Scale (Implemented)
-
-Brainy includes a complete sharding implementation with ConsistentHashRing:
-
-```javascript
-import { ShardManager } from '@soulcraft/brainy/distributed'
-
-// Create shard manager with custom configuration
-const shardManager = new ShardManager({
- shardCount: 64, // Default: 64 shards
- replicationFactor: 3, // Default: 3 replicas
- virtualNodes: 150, // Default: 150 virtual nodes
- autoRebalance: true // Default: true
-})
-
-// Add nodes to the cluster
-shardManager.addNode('node-1')
-shardManager.addNode('node-2')
-shardManager.addNode('node-3')
-
-// Sharding automatically:
-// - Uses consistent hashing for even distribution
-// - Maintains replicas for fault tolerance
-// - Rebalances on node changes
-// - Provides O(1) shard lookups
-```
+For off-site replication, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`).
### Performance at Scale
-Even at massive scale, Brainy maintains excellent performance:
-
-- **Metadata queries**: Still O(1) with distributed hash tables
-- **Graph traversal**: O(1) with edge locality optimization
-- **Vector search**: O(log n) with hierarchical sharding
-- **Write throughput**: 100K+ writes/second with S3 batching
+- **Metadata queries**: O(1) HashMap
+- **Graph traversal**: O(1) adjacency lookup
+- **Vector search**: O(log n)
+- **Write throughput**: 50K+ writes/second per process (filesystem, batched)
- **Read throughput**: 1M+ reads/second with caching
-### Zero-Config with Autoscaling (Implemented)
+### Zero-Config with Autoscaling
-Brainy includes extensive autoscaling capabilities:
-
-**✅ Implemented Autoscaling:**
- **AutoConfiguration System**: Detects environment and adjusts settings
- **Learning from Performance**: `learnFromPerformance()` adapts based on metrics
- **Auto-flush**: Graph index (30s), Metadata index (configurable)
-- **Auto-optimize**: Enabled by default in Graph and HNSW indices
-- **Auto-rebalance**: Shards automatically rebalance on node changes
+- **Auto-optimize**: Enabled by default in graph and vector indices
- **Zero-config presets**: Production, development, minimal modes
- **Adaptive memory**: Scales caches based on available memory
-- **Environment detection**: Browser vs Node.js vs Serverless
-
-**🚧 Roadmap Autoscaling:**
-- Dynamic HNSW parameter adjustment (M, ef)
-- Predictive query pattern caching
-- Multi-region auto-replication
-- Automatic cross-node data migration
## Implementation Status
-### ✅ Fully Implemented and Production-Ready
+### Fully Implemented and Production-Ready
- **O(1) metadata lookups** via HashMaps (exact match)
- **O(log n) range queries** via sorted arrays with lazy building
- **O(1) graph traversal** via adjacency maps
-- **O(log n) vector search** via HNSW
+- **O(log n) vector search** via the default JS index, swappable for a native provider
- **220 NLP patterns** with pre-computed embeddings
-- **S3-compatible storage** (AWS S3, R2, GCS, MinIO, B2)
-- **Distributed sharding** with ConsistentHashRing
+- **Filesystem and memory storage** adapters
- **Auto-configuration system** with environment detection
- **Zero-config operation** with intelligent defaults
- **Auto-flush and auto-optimize** in indices
-- **Sub-2ms response times** for complex queries
-
-### 🚧 Roadmap Features
-- Dynamic HNSW parameter tuning
-- Predictive query pattern caching
-- Multi-region S3 replication
-- Automatic cross-node data migration
-- Edge caching layer
+- **Low-latency Triple Intelligence queries** (O(log n) vector + O(1) metadata/graph)
## Conclusion
-Brainy delivers on its promise of **production-ready Triple Intelligence** with measured, verified performance characteristics. All listed features are fully implemented, tested, and benchmarked. No stubs, no mocks, no theoretical claims - just real, working code with measured performance.
\ No newline at end of file
+Brainy delivers on its promise of **production-ready Triple Intelligence** with documented algorithmic-complexity guarantees and a committed graph-scale benchmark (`tests/performance/graph-scale-performance.test.ts`). All listed features are fully implemented and tested. No stubs, no mocks — just real, working code with characterized performance.
\ No newline at end of file
diff --git a/docs/PLUGINS.md b/docs/PLUGINS.md
new file mode 100644
index 00000000..d9a4d3e7
--- /dev/null
+++ b/docs/PLUGINS.md
@@ -0,0 +1,486 @@
+---
+title: Plugin System
+slug: guides/plugins
+public: true
+category: guides
+template: guide
+order: 4
+description: Replace any Brainy subsystem — distance functions, embeddings, vector index, metadata index, aggregation — with a custom implementation or optional native acceleration.
+next:
+ - guides/storage-adapters
+---
+
+# Plugin Development Guide
+
+Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
+
+## Architecture Overview
+
+Brainy's plugin system uses **named providers** — string keys mapped to implementations. During `init()`, brainy:
+
+1. Imports each package listed in the `plugins` config array
+2. Activates each plugin, passing a `BrainyPluginContext`
+3. The plugin calls `context.registerProvider(key, implementation)` for each subsystem it provides
+4. Brainy checks each provider key and wires the implementation into its internal pipeline
+
+Installing the first-party accelerator is the opt-in: with the default config, brainy probes for `@soulcraft/cor` and loads it when present. Everything except "not installed" fails **loud** — a present-but-broken accelerator makes `init()` throw rather than silently degrading to the JS engines.
+
+```typescript
+const brain = new Brainy() // @soulcraft/cor auto-detected when installed
+const pinned = new Brainy({ plugins: ['@soulcraft/cor'] }) // or pin exactly what loads
+const plain = new Brainy({ plugins: [] }) // or opt out of detection entirely
+```
+
+| `plugins` value | Behavior |
+|---|---|
+| `undefined` (default) | Guarded auto-detection of `@soulcraft/cor`: not installed → no plugins, silently; installed → loads + announces; installed-but-broken → `init()` throws |
+| `false` / `[]` | No plugins, no detection (explicit opt-out) |
+| `['@soulcraft/cor']` | Load only the listed packages; a listed plugin that fails to load throws |
+
+Plugins registered programmatically via `brain.use(plugin)` are always activated regardless of the `plugins` config.
+
+If no plugin provides a given key, brainy uses its built-in JavaScript implementation. This means brainy works perfectly standalone — plugins only enhance performance or add capabilities.
+
+## Creating a Plugin
+
+### 1. Implement the `BrainyPlugin` interface
+
+```typescript
+import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
+
+const myPlugin: BrainyPlugin = {
+ name: 'my-brainy-plugin', // Must be unique (typically your npm package name)
+
+ async activate(context: BrainyPluginContext): Promise {
+ // Register your providers here
+ context.registerProvider('distance', myFastDistanceFunction)
+
+ // Return true if activation succeeded, false to skip
+ return true
+ },
+
+ async deactivate(): Promise {
+ // Optional cleanup when brainy.close() is called
+ }
+}
+
+export default myPlugin
+```
+
+### 2. Package exports
+
+Your package must export the plugin as the default export so brainy's plugin loader can resolve it:
+
+```typescript
+// index.ts
+export { default } from './plugin.js'
+```
+
+### 3. Registration
+
+**Config-based:** List your package name in the brainy config:
+
+```typescript
+const brain = new Brainy({
+ plugins: ['my-brainy-plugin']
+})
+await brain.init()
+```
+
+**Programmatic registration:** For plugins not installed as npm packages, use `brain.use()`:
+
+```typescript
+import { Brainy } from '@soulcraft/brainy'
+import myPlugin from './my-plugin.js'
+
+const brain = new Brainy()
+brain.use(myPlugin)
+await brain.init()
+```
+
+## Provider Keys Reference
+
+Each key has a specific expected signature. Brainy checks for these during `init()` and wires them into the appropriate code paths.
+
+### Core Providers
+
+#### `distance`
+**Type:** `(a: number[], b: number[]) => number`
+
+Replaces the default cosine distance function used in vector search and neural APIs. This is the highest-impact single provider — it's called for every vector comparison.
+
+```typescript
+context.registerProvider('distance', (a: number[], b: number[]): number => {
+ // Your SIMD-accelerated or GPU distance calculation
+ return myFastCosineDistance(a, b)
+})
+```
+
+#### `embeddings`
+**Type:** `(text: string | string[]) => Promise`
+
+Replaces the built-in WASM embedding engine. Called for every `brain.add()`, `brain.update()`, and `brain.find()` operation that involves text.
+
+```typescript
+context.registerProvider('embeddings', async (text: string | string[]) => {
+ if (Array.isArray(text)) {
+ return myEngine.embedBatch(text)
+ }
+ return myEngine.embed(text)
+})
+```
+
+#### `embedBatch`
+**Type:** `(texts: string[]) => Promise`
+
+Dedicated batch embedding provider. When registered, brainy uses this for bulk operations (import, reindex, batch add) instead of calling the `embeddings` provider N times. This enables true single-forward-pass batch processing.
+
+Priority order for batch operations:
+1. `embedBatch` provider (single forward pass — fastest)
+2. `embeddings` provider with `Promise.all()` (N individual calls)
+3. Built-in WASM batch API (fallback)
+
+```typescript
+context.registerProvider('embedBatch', async (texts: string[]) => {
+ // Process all texts in a single forward pass
+ return myEngine.batchEmbed(texts)
+})
+```
+
+### Index Providers
+
+> **Write-path invariant (the change-feed contract).** Every canonical
+> mutation flows through Brainy's generation-store commit points — index
+> providers are invoked *inside* that commit and never originate canonical
+> writes of their own. The `brain.onChange` change feed is emitted from those
+> commit points and relies on this: **a plugin must never introduce a write
+> path that bypasses the generation-store commit.** If a future provider ever
+> needs a direct native ingest path, it must either route through the commit
+> or emit equivalent change events — otherwise every `onChange` consumer
+> (live UIs, cache invalidation, realtime sync) silently develops a blind
+> spot.
+
+#### `vector`
+**Type:** `(config: object, distanceFunction: Function, options: object) => VectorIndexProvider-compatible`
+
+Factory function that creates a vector index instance. The returned object must implement the `VectorIndexProvider` public API:
+
+- `addItem(item: { id: string, vector: number[] }): Promise`
+- `search(queryVector: number[], k: number, filter?, options?): Promise>`
+- `removeItem(id: string): Promise`
+- `size(): number`
+- `clear(): void`
+- `flush(): Promise`
+- `rebuild(options?): Promise`
+- `getDirtyNodeCount(): number`
+- `getPersistMode(): 'immediate' | 'deferred'`
+- `getEntryPointId(): string | null`
+- `getMaxLevel(): number`
+- `getDimension(): number | null`
+- `getConfig(): object`
+- `getDistanceFunction(): Function`
+- `enableCOW(parent): void`
+- `setUseParallelization(boolean): void`
+
+For type-aware indexes (separate graph per noun type), also implement:
+- `getIndexForType(type: string): VectorIndexProvider` (duck-typed detection)
+- `search(queryVector, k, type?, filter?, options?): Promise>`
+
+```typescript
+context.registerProvider('vector', (config, distanceFn, options) => {
+ return new MyNativeVectorIndex(config, distanceFn, options)
+})
+```
+
+#### The readiness contract (all three index providers)
+
+A provider that **persists its derived index** should implement the optional readiness
+members so a warm reopen never pays a redundant rebuild-from-canonical:
+
+- **`init?(): Promise`** — eager cold-load. Brainy awaits it once during
+ `brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
+ and **before the rebuild gate**.
+- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
+ loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
+ exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
+ `totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
+ while fully durable. Never return `true` if the durable state failed to load: the
+ signal is honest in both directions, and a not-ready provider gets its rebuild even
+ when `size() > 0`.
+- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
+ migration); brainy skips its rebuild entirely.
+
+Providers that implement none of these keep the size/count heuristics — correct for
+engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
+
+#### `metadataIndex`
+**Type:** `(storage: StorageAdapter) => MetadataIndexManager-compatible`
+
+Factory function that creates a metadata index. The returned object must implement the `MetadataIndexManager` interface including `init()`, `addEntity()`, `removeEntity()`, `query()`, `flush()`, `clear()`, etc.
+
+```typescript
+context.registerProvider('metadataIndex', (storage) => {
+ return new MyNativeMetadataIndex(storage)
+})
+```
+
+#### `graphIndex`
+**Type:** `(storage: StorageAdapter) => GraphAdjacencyIndex-compatible`
+
+Factory function that creates a graph adjacency index for relationship tracking (verbs/triples). Must implement the `GraphAdjacencyIndex` interface including `addVerb()`, `getVerbsBySource()`, `getVerbsByTarget()`, `flush()`, etc.
+
+```typescript
+context.registerProvider('graphIndex', (storage) => {
+ return new MyNativeGraphIndex(storage)
+})
+```
+
+#### `aggregation`
+**Type:** `(storage: StorageAdapter) => AggregationProvider-compatible`
+
+Factory function that creates an aggregation engine for write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows. The returned object must implement the `AggregationProvider` interface.
+
+```typescript
+context.registerProvider('aggregation', (storage) => {
+ return new MyNativeAggregationEngine(storage)
+})
+```
+
+When provided by an optional native acceleration plugin (such as `@soulcraft/cor`), this enables:
+- Compiled source filters (vs per-entity JS object traversal)
+- Precise MIN/MAX via sorted data structures (vs lazy recompute)
+- Parallel aggregate rebuild across CPU cores
+- SIMD-accelerated timestamp bucketing
+
+### Utility Providers
+
+#### `cache`
+**Type:** `UnifiedCache`
+
+Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`).
+
+```typescript
+import type { UnifiedCache } from '@soulcraft/brainy/internals'
+
+context.registerProvider('cache', myNativeCache)
+```
+
+#### `entityIdMapper`
+**Type:** `(storage: StorageAdapter) => EntityIdMapper-compatible`
+
+Factory for bidirectional UUID ↔ integer mapping used by roaring bitmaps. Must implement `getOrAssign()`, `getUuid()`, `getInt()`, `has()`, `remove()`, `flush()`, `clear()`.
+
+#### `roaring`
+**Type:** `RoaringBitmap32 class`
+
+Replacement for the roaring bitmap implementation. Used internally by the metadata index for set operations. Must be API-compatible with `roaring-wasm`.
+
+#### `msgpack`
+**Type:** `{ encode: (data: any) => Buffer, decode: (buffer: Buffer) => any }`
+
+Native msgpack encode/decode for SSTable serialization.
+
+### Analytics Providers (Native-Only)
+
+These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cor`) is installed.
+
+Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it.
+
+#### `analytics:hyperloglog`
+Approximate distinct counts. Count unique values (e.g., unique merchants) across millions of records using ~16KB of memory with ~1% error. Each update is O(1).
+
+#### `analytics:tdigest`
+Streaming percentiles. Compute P50/P90/P95/P99 from streaming data without storing all values. Uses ~4KB per digest with ~1% accuracy at the tails.
+
+#### `analytics:countmin`
+Frequency estimation. Find the most common values (e.g., top-K merchants) using ~40KB with 0.1% error. O(1) per update.
+
+#### `analytics:anomaly`
+Real-time anomaly detection. Flag statistically unusual values at write-time using exponentially weighted moving averages. 64 bytes per group, sub-microsecond decisions.
+
+#### `aggregation:mmap`
+Persistent aggregate storage via memory-mapped files. Aggregate state survives process crashes without explicit flush. Zero serialization overhead.
+
+---
+
+## Storage Adapter Plugins
+
+Plugins can register custom storage backends that users reference by name.
+
+### Implementing a Storage Adapter
+
+```typescript
+import type { StorageAdapterFactory } from '@soulcraft/brainy/plugin'
+import type { StorageAdapter } from '@soulcraft/brainy'
+
+class MyStorageAdapter implements StorageAdapter {
+ async init(): Promise { /* ... */ }
+ async saveNoun(noun: HNSWNoun): Promise { /* ... */ }
+ async getNoun(id: string): Promise { /* ... */ }
+ async deleteNoun(id: string): Promise { /* ... */ }
+ // ... implement all StorageAdapter methods
+}
+```
+
+### Registering a Storage Adapter
+
+```typescript
+context.registerProvider('storage:my-backend', {
+ name: 'my-backend',
+ create: (config: Record) => {
+ return new MyStorageAdapter(config)
+ }
+} satisfies StorageAdapterFactory)
+```
+
+Users can then use your storage:
+
+```typescript
+const brain = new Brainy({ storage: 'my-backend', myBackendOption: 'value' })
+```
+
+## Import Paths
+
+Brainy provides three entry points for plugin developers:
+
+| Import Path | Contents | Stability |
+|-------------|----------|-----------|
+| `@soulcraft/brainy` | Public API, types, StorageAdapter | Stable (semver) |
+| `@soulcraft/brainy/plugin` | BrainyPlugin, BrainyPluginContext, StorageAdapterFactory | Stable (semver) |
+| `@soulcraft/brainy/internals` | UnifiedCache, EntityIdMapper, logger utilities | Internal (may change between minor versions) |
+
+## Diagnostics
+
+Brainy provides a `diagnostics()` method to verify plugin wiring:
+
+```typescript
+const brain = new Brainy()
+await brain.init()
+
+const diag = brain.diagnostics()
+console.log(diag)
+// {
+// version: '7.14.0',
+// plugins: { active: ['my-plugin'], count: 1 },
+// providers: {
+// metadataIndex: { source: 'default' },
+// graphIndex: { source: 'default' },
+// embeddings: { source: 'plugin' },
+// embedBatch: { source: 'plugin' },
+// distance: { source: 'plugin' },
+// vector: { source: 'default' },
+// ...
+// },
+// indexes: {
+// vector: { size: 0, type: 'JsHnswVectorIndex' },
+// metadata: { type: 'MetadataIndexManager', initialized: true },
+// graph: { type: 'GraphAdjacencyIndex', initialized: true, wiredToStorage: true }
+// }
+// }
+```
+
+The CLI also supports diagnostics:
+
+```bash
+brainy diagnostics
+```
+
+### Init-Time Summary
+
+When a plugin is active, brainy automatically logs a provider summary after `init()`:
+
+```
+[brainy] Plugin activated: @soulcraft/cor
+[brainy] Providers: 8/10 native (@soulcraft/cor) | default: vector, cache
+```
+
+This tells you at a glance how many subsystems are accelerated and which ones are falling back to JavaScript. The log respects `config.silent`.
+
+### Fail-Fast for Production
+
+Use `requireProviders()` after `init()` to guarantee specific providers are plugin-supplied. This prevents silent fallback to JavaScript in deployments where you expect native acceleration:
+
+```typescript
+const brain = new Brainy()
+await brain.init()
+
+// Throws immediately if any of these are using JS fallback
+brain.requireProviders(['distance', 'embeddings', 'metadataIndex', 'graphIndex'])
+```
+
+If a required provider is missing, the error message tells you exactly what's wrong:
+
+```
+[brainy] Required providers using JS fallback: graphIndex.
+Active plugins: @soulcraft/cor.
+These providers must be supplied by a plugin for this deployment.
+Check plugin installation, license, and native module availability.
+```
+
+This is the recommended pattern for production deployments with paid plugins — fail at startup rather than silently degrading performance.
+
+## Complete Example: Distance Acceleration Plugin
+
+A minimal but useful plugin that provides SIMD-accelerated distance calculations:
+
+```typescript
+// simd-distance-plugin/src/plugin.ts
+import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
+
+// Hypothetical native module
+import { simdCosineDistance } from './native.js'
+
+const simdDistancePlugin: BrainyPlugin = {
+ name: 'brainy-simd-distance',
+
+ async activate(context: BrainyPluginContext): Promise {
+ // Check if SIMD is available on this platform
+ if (!checkSimdSupport()) {
+ console.log('[simd-distance] SIMD not available, skipping')
+ return false // Don't activate — brainy uses JS fallback
+ }
+
+ context.registerProvider('distance', simdCosineDistance)
+ return true
+ }
+}
+
+export default simdDistancePlugin
+```
+
+```json
+// simd-distance-plugin/package.json
+{
+ "name": "brainy-simd-distance",
+ "main": "./dist/plugin.js",
+ "types": "./dist/plugin.d.ts",
+ "peerDependencies": {
+ "@soulcraft/brainy": ">=7.0.0"
+ }
+}
+```
+
+Usage:
+
+```typescript
+import { Brainy } from '@soulcraft/brainy'
+
+const brain = new Brainy({ plugins: ['brainy-simd-distance'] })
+await brain.init()
+
+// Verify it's active
+const diag = brain.diagnostics()
+console.log(diag.providers.distance) // { source: 'plugin' }
+```
+
+## Design Principles
+
+1. **Brainy works perfectly without plugins.** Every provider has a JavaScript fallback. Plugins only improve performance or add capabilities.
+
+2. **Provider keys are string-based.** The plugin system is not coupled to any specific plugin. Any package can register any provider.
+
+3. **Clean separation.** Plugins access brainy through the documented `BrainyPluginContext` interface. No direct access to internal classes is needed.
+
+4. **Fail-safe activation.** If a plugin throws during `activate()`, brainy logs a warning and continues with defaults. A broken plugin never prevents brainy from working.
+
+5. **Lifecycle management.** `deactivate()` is called during `brainy.close()` for resource cleanup. Native resources, connections, and file handles should be released here.
diff --git a/docs/PRODUCTION_SERVICE_ARCHITECTURE.md b/docs/PRODUCTION_SERVICE_ARCHITECTURE.md
index a7e088f4..4568cd31 100644
--- a/docs/PRODUCTION_SERVICE_ARCHITECTURE.md
+++ b/docs/PRODUCTION_SERVICE_ARCHITECTURE.md
@@ -215,9 +215,9 @@ console.log('Server running on http://localhost:3000')
```
**Benefits:**
-- ✅ Native Bun performance (~2x faster than Node.js)
+- ✅ Native Bun runtime performance
- ✅ No framework dependencies
-- ✅ Works with `bun --compile` for single-binary deployment
+- ✅ Pure WASM — no native binaries, bundler-friendly
- ✅ Built-in TypeScript support
### Pattern 4: Express/Node.js Middleware (Legacy)
diff --git a/docs/QUERY_OPERATORS.md b/docs/QUERY_OPERATORS.md
new file mode 100644
index 00000000..f4ac04e6
--- /dev/null
+++ b/docs/QUERY_OPERATORS.md
@@ -0,0 +1,298 @@
+# Query Operators (BFO)
+
+> Brainy Field Operators — the complete reference for `where` filters in `find()`.
+
+All operators work with `find({ where: { ... } })` and filter on **metadata fields** (not `data`).
+
+---
+
+## Equality
+
+| Operator | Alias | Description | Example |
+|----------|-------|-------------|---------|
+| `eq` | `equals` | Exact match | `{ status: { eq: 'active' } }` |
+| `ne` | `notEquals` | Not equal | `{ status: { ne: 'deleted' } }` |
+
+**Shorthand:** A bare value is treated as `equals`:
+
+```typescript
+// These are equivalent:
+brain.find({ where: { status: 'active' } })
+brain.find({ where: { status: { equals: 'active' } } })
+```
+
+---
+
+## Comparison
+
+| Operator | Alias | Description | Example |
+|----------|-------|-------------|---------|
+| `gt` | `greaterThan` | Greater than | `{ age: { gt: 18 } }` |
+| `gte` | `greaterThanOrEqual` | Greater or equal | `{ score: { gte: 90 } }` |
+| `lt` | `lessThan` | Less than | `{ price: { lt: 100 } }` |
+| `lte` | `lessThanOrEqual` | Less or equal | `{ rating: { lte: 3 } }` |
+| `between` | — | Inclusive range `[min, max]` | `{ year: { between: [2020, 2025] } }` |
+
+```typescript
+// Range query
+const recent = await brain.find({
+ where: {
+ createdAt: { between: [Date.now() - 86400000, Date.now()] }
+ }
+})
+```
+
+---
+
+## Array / Set
+
+| Operator | Alias | Description | Example |
+|----------|-------|-------------|---------|
+| `oneOf` | `in` | Value is one of the given options | `{ color: { oneOf: ['red', 'blue'] } }` |
+| `noneOf` | — | Value is NOT one of the given options | `{ status: { noneOf: ['deleted', 'archived'] } }` |
+| `contains` | — | Array field contains value | `{ tags: { contains: 'ai' } }` |
+| `excludes` | — | Array field does NOT contain value | `{ tags: { excludes: 'spam' } }` |
+| `hasAll` | — | Array field contains ALL listed values | `{ skills: { hasAll: ['js', 'ts'] } }` |
+
+```typescript
+// Find entities tagged with 'ai'
+const aiEntities = await brain.find({
+ where: { tags: { contains: 'ai' } }
+})
+
+// Find entities of specific types
+const people = await brain.find({
+ where: { noun: { oneOf: ['Person', 'Agent'] } }
+})
+```
+
+---
+
+## Existence
+
+| Operator | Description | Example |
+|----------|-------------|---------|
+| `exists: true` | Field exists (has any value) | `{ email: { exists: true } }` |
+| `exists: false` | Field does NOT exist | `{ email: { exists: false } }` |
+| `missing: true` | Field does NOT exist (alias for `exists: false`) | `{ email: { missing: true } }` |
+| `missing: false` | Field exists (alias for `exists: true`) | `{ email: { missing: false } }` |
+
+```typescript
+// Find entities that have an email field
+const withEmail = await brain.find({
+ where: { email: { exists: true } }
+})
+```
+
+---
+
+## Pattern (In-Memory Only)
+
+These operators work via the in-memory filter path. They are applied **after** the indexed query, so use them with other indexed operators for best performance.
+
+| Operator | Description | Example |
+|----------|-------------|---------|
+| `matches` | Regex or string pattern match | `{ name: { matches: /^Dr\./ } }` |
+| `startsWith` | String prefix | `{ name: { startsWith: 'John' } }` |
+| `endsWith` | String suffix | `{ email: { endsWith: '@gmail.com' } }` |
+
+```typescript
+const doctors = await brain.find({
+ where: {
+ type: NounType.Person, // Indexed — fast
+ name: { startsWith: 'Dr.' } // In-memory — applied after
+ }
+})
+```
+
+---
+
+## Logical
+
+Combine multiple conditions:
+
+| Operator | Description | Example |
+|----------|-------------|---------|
+| `allOf` | ALL sub-filters must match (AND) | `{ allOf: [{ status: 'active' }, { role: 'admin' }] }` |
+| `anyOf` | ANY sub-filter must match (OR) | `{ anyOf: [{ role: 'admin' }, { role: 'owner' }] }` |
+| `not` | Invert a filter | `{ not: { status: 'deleted' } }` |
+
+```typescript
+// Complex OR query
+const adminsOrOwners = await brain.find({
+ where: {
+ anyOf: [
+ { role: 'admin' },
+ { role: 'owner' }
+ ]
+ }
+})
+
+// NOT query
+const notDeleted = await brain.find({
+ where: {
+ not: { status: 'deleted' }
+ }
+})
+
+// Combined AND + OR
+const results = await brain.find({
+ where: {
+ allOf: [
+ { department: 'engineering' },
+ { anyOf: [
+ { level: 'senior' },
+ { yearsExperience: { greaterThan: 5 } }
+ ]}
+ ]
+ }
+})
+```
+
+---
+
+## Indexed vs In-Memory Operators
+
+Brainy's MetadataIndex supports a subset of operators natively for O(1) field lookups. Other operators fall back to in-memory filtering.
+
+| Operator | MetadataIndex (Indexed) | In-Memory Fallback |
+|----------|:-----------------------:|:------------------:|
+| `equals` / `eq` | Yes | Yes |
+| `notEquals` / `ne` | — | Yes |
+| `greaterThan` / `gt` | Yes | Yes |
+| `greaterThanOrEqual` / `gte` | Yes | Yes |
+| `lessThan` / `lt` | Yes | Yes |
+| `lessThanOrEqual` / `lte` | Yes | Yes |
+| `between` | Yes | Yes |
+| `oneOf` / `in` | Yes | Yes |
+| `noneOf` | — | Yes |
+| `contains` | Yes | Yes |
+| `exists` / `missing` | Yes | Yes |
+| `matches` | — | Yes |
+| `startsWith` | — | Yes |
+| `endsWith` | — | Yes |
+| `allOf` | Partial | Yes |
+| `anyOf` | Partial | Yes |
+| `not` | — | Yes |
+
+**Performance tip:** Combine indexed operators (equals, greaterThan, oneOf, between, contains, exists) with pattern operators for optimal speed — the index narrows results first, then patterns filter in memory.
+
+---
+
+## Practical Examples
+
+### Filter by entity type
+
+```typescript
+// Using the type shorthand (recommended)
+brain.find({ type: NounType.Person })
+
+// Using where.noun directly
+brain.find({ where: { noun: NounType.Person } })
+
+// Multiple types
+brain.find({ type: [NounType.Person, NounType.Agent] })
+```
+
+### Filter by subtype
+
+`subtype` is a top-level standard field — takes the column-store fast path, not the metadata fallback. Pair with `type` for the typical "Person who is an employee" query:
+
+```typescript
+// Equality on subtype:
+brain.find({ type: NounType.Person, subtype: 'employee' })
+
+// Set membership:
+brain.find({ type: NounType.Person, subtype: ['employee', 'contractor'] })
+
+// Operator-form predicates use `where`:
+brain.find({
+ type: NounType.Person,
+ where: { subtype: { exists: true } }
+})
+```
+
+See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the full surface.
+
+### Filter relationships by subtype (7.30+)
+
+Verbs are first-class peers — `related()` and graph traversal both honor subtype filters on the fast path:
+
+```typescript
+// Filter relationships by VerbType subtype
+const direct = await brain.related({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct'
+})
+
+// Set membership on verb subtype
+const all = await brain.related({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: ['direct', 'dotted-line']
+})
+
+// Graph traversal — subtype filters traversal edges (depth-1 in 7.30 JS path;
+// multi-hop subtype filtering lands on Cor native)
+const reports = await brain.find({
+ connected: {
+ from: ceoId,
+ via: VerbType.ReportsTo,
+ subtype: 'direct',
+ depth: 1
+ }
+})
+```
+
+### Combine semantic search with filters
+
+```typescript
+const results = await brain.find({
+ query: 'machine learning engineer', // Semantic search (on data)
+ type: NounType.Person, // Type filter (indexed)
+ where: {
+ department: 'engineering', // Exact match (indexed)
+ yearsExperience: { greaterThan: 3 } // Range filter (indexed)
+ },
+ limit: 10
+})
+```
+
+### Temporal queries
+
+```typescript
+const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000
+const recentEntities = await brain.find({
+ where: {
+ createdAt: { greaterThan: lastWeek }
+ },
+ orderBy: 'createdAt',
+ order: 'desc',
+ limit: 50
+})
+```
+
+### Graph + metadata combination
+
+```typescript
+const results = await brain.find({
+ connected: {
+ from: teamLeadId,
+ via: VerbType.WorksWith,
+ depth: 2
+ },
+ where: {
+ role: { oneOf: ['engineer', 'designer'] },
+ active: true
+ }
+})
+```
+
+---
+
+## See Also
+
+- [Data Model](./DATA_MODEL.md) — Entity structure, data vs metadata
+- [API Reference](./api/README.md) — Complete API documentation
+- [Find System](./FIND_SYSTEM.md) — Natural language find() details
diff --git a/docs/README.md b/docs/README.md
index bed4e26f..3290001f 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,339 +1,130 @@
-# Brainy Documentation (v6.5.0)
+# Brainy Documentation
-Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
+> The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API.
-## 🆕 What's New in v4.0.0
-
-**Production-Ready Cost Optimization:**
-- **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!)
-- **Intelligent-Tiering**: S3 Intelligent-Tiering and GCS Autoclass support
-- **Batch Operations**: Efficient bulk delete operations (1000 objects per request)
-- **Compression**: Gzip compression for FileSystem storage (60-80% space savings)
-- **Quota Monitoring**: Real-time OPFS quota tracking for browser apps
-- **Tier Management**: Azure Hot/Cool/Archive tier management
-
-**Cost Impact Example (500TB dataset):**
-- Before: $138,000/year
-- After v4.0.0: $5,940/year
-- **Savings: $132,060/year (96%)**
-
-## 📊 Implementation Status
-
-- ✅ **Production Ready**: Core features working today
-- 🚧 **In Development**: Features coming soon
-- 📅 **Roadmap**: See [ROADMAP.md](../ROADMAP.md)
-
-## Quick Links
-
-### Getting Started
-- [API Reference](./api/README.md) - Complete API documentation (start here!)
-- [Enterprise for Everyone](./guides/enterprise-for-everyone.md) - **No limits, no tiers, everything free**
-- [Natural Language Queries](./guides/natural-language.md) - Query with plain English
-
-### Core Concepts
-- [Zero Configuration](./architecture/zero-config.md) - **Auto-adapts to any environment**
-- [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) - **Revolutionary data model**
-- [Triple Intelligence](./architecture/triple-intelligence.md) - Unified query system
-- [Architecture Overview](./architecture/overview.md) - System design
-
-### API Documentation
-- [API Reference](./api/README.md) - Complete API documentation
-- [TypeScript Types](./api/types.md) - Type definitions
-
-### Advanced Topics
-- [Augmentations System](./architecture/augmentations.md) - **Enterprise plugins & neural import**
-- [Storage Architecture](./architecture/storage.md) - Storage adapter system
-- [Performance Tuning](./guides/performance.md) - Optimization guide
-- [Migration Guide](../MIGRATION.md) - Upgrading from 1.x
-
-## What is Brainy?
-
-Brainy is a next-generation AI database that combines:
-- **Vector Search**: Semantic similarity using HNSW indexing
-- **Graph Relationships**: Complex relationship mapping and traversal
-- **Field Filtering**: Precise metadata filtering with O(1) lookups
-- **Natural Language**: Query in plain English
-
-## Key Features
-
-### 🧠 Triple Intelligence Engine
-All three intelligence types (vector, graph, field) work together in every query for optimal results.
-
-### 📝 Noun-Verb Taxonomy
-Model your data naturally as entities (nouns) and relationships (verbs) - no complex schemas needed.
-
-### 🌍 Natural Language Queries
-Ask questions in plain English and Brainy understands your intent:
-```typescript
-await brain.find("recent articles about AI with high ratings")
-```
-
-### ⚡ Production Ready
-- Universal storage (FileSystem, S3, OPFS, Memory)
-- Zero configuration with intelligent defaults
-- Full TypeScript support
-- Cross-platform compatibility
-
-## Quick Example
+## Quick Start
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-// Initialize
const brain = new Brainy()
await brain.init()
-// Add entities (nouns)
-const articleId = await brain.add({
- data: "Revolutionary AI Breakthrough",
+// Add entities — data is embedded for semantic search, metadata is indexed for filtering
+const id = await brain.add({
+ data: 'Revolutionary AI Breakthrough',
type: NounType.Document,
- metadata: { category: "technology", rating: 4.8 }
+ metadata: { category: 'technology', rating: 4.8 }
})
-const authorId = await brain.add({
- data: "Dr. Sarah Chen",
- type: NounType.Person,
- metadata: { role: "researcher" }
+// Search with Triple Intelligence
+const results = await brain.find({
+ query: 'artificial intelligence', // Semantic search (on data)
+ where: { rating: { greaterThan: 4.0 } }, // Metadata filter
+ connected: { from: authorId, depth: 2 } // Graph traversal
})
-
-// Create relationships (verbs)
-await brain.relate({
- from: authorId,
- to: articleId,
- type: VerbType.CreatedBy,
- metadata: { date: "2024-01-15", contribution: "primary" }
-})
-
-// Query naturally
-const results = await brain.find("highly rated technology articles by researchers")
```
-## 📚 Complete Documentation Index
+---
-### 🚀 Quick Start Guides
+## Core Documentation
| Document | Description |
|----------|-------------|
-| [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples |
-| [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds |
+| **[API Reference](./api/README.md)** | Complete API documentation — **start here** |
+| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
+| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
+| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
+| [Consistency Model](./concepts/consistency-model.md) | The Db API guarantees — snapshot isolation, atomic transactions, time travel |
-### 🆕 v4.0.0 Migration & Optimization
+---
+
+## Architecture
| Document | Description |
|----------|-------------|
-| [v3→v4 Migration Guide](./MIGRATION-V3-TO-V4.md) | **NEW** - Upgrade guide with zero breaking changes |
-| [AWS S3 Cost Optimization](./operations/cost-optimization-aws-s3.md) | **NEW** - 96% cost savings with lifecycle policies |
-| [GCS Cost Optimization](./operations/cost-optimization-gcs.md) | **NEW** - 94% savings with Autoclass |
-| [Azure Cost Optimization](./operations/cost-optimization-azure.md) | **NEW** - 95% savings with tier management |
-| [Cloudflare R2 Cost Guide](./operations/cost-optimization-cloudflare-r2.md) | **NEW** - Zero egress fees + S3-compatible API |
-
-### 🎯 Core Concepts
-
-| Document | Description |
-|----------|-------------|
-| [Architecture Overview](./architecture/overview.md) | High-level system design and components |
-| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships |
-| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system |
+| [Architecture Overview](./architecture/overview.md) | High-level system design |
+| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Metadata unified query |
+| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | 42 nouns + 127 verbs type system |
+| [Stage 3 Canonical Taxonomy](./STAGE3-CANONICAL-TAXONOMY.md) | Complete type reference |
+| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
+| [Index Architecture](./architecture/index-architecture.md) | Vector, Graph, and Metadata indexing |
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
-| [Storage Architecture](./architecture/storage-architecture.md) | **v4.0.0** - Storage adapters and optimization |
-| [Data Storage Architecture](./architecture/data-storage-architecture.md) | **v4.0.0** - Metadata/vector separation, sharding |
-| [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing |
-### 💾 Storage & Deployment
+---
+
+## Virtual Filesystem (VFS)
| Document | Description |
|----------|-------------|
-| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | **v4.0.0** - Deploy on AWS, GCP, Azure, Cloudflare |
-| [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns |
-| [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment |
-| [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations |
-| [Distributed Storage](./architecture/distributed-storage.md) | Multi-node storage coordination |
-| [Extending Storage](./EXTENDING_STORAGE.md) | Create custom storage adapters |
+| [VFS Quick Start](./vfs/QUICK_START.md) | Get started in 30 seconds |
+| [VFS Core](./vfs/VFS_CORE.md) | Core concepts and architecture |
+| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
+| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
+
+See [vfs/](./vfs/) for the complete VFS documentation set.
+
+---
+
+## Guides
+
+| Document | Description |
+|----------|-------------|
+| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
+| [Snapshots & Time Travel](./guides/snapshots-and-time-travel.md) | Backups, restore, what-if analysis, audit trails |
+| [Natural Language](./guides/natural-language.md) | Query in plain English |
+| [Neural API](./guides/neural-api.md) | AI-powered features |
+| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
+| [Framework Integration](./guides/framework-integration.md) | React, Vue, Angular, Svelte |
+
+---
+
+## Storage & Deployment
+
+| Document | Description |
+|----------|-------------|
+| [Storage Architecture](./architecture/storage-architecture.md) | Filesystem and memory adapters, on-disk artifact layout, operator-layer backup |
| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities |
-### 📊 API Documentation
+---
+
+## Plugins
| Document | Description |
|----------|-------------|
-| [API Reference](./API_REFERENCE.md) | Complete API documentation |
-| [Comprehensive API Overview](./api/COMPREHENSIVE_API_OVERVIEW.md) | All APIs with examples |
-| [API Decision Tree](./API_DECISION_TREE.md) | Choose the right API for your use case |
-| [Core API Patterns](./CORE_API_PATTERNS.md) | Common patterns and best practices |
-| [Neural API Patterns](./NEURAL_API_PATTERNS.md) | AI-powered query patterns |
-| [Find System](./FIND_SYSTEM.md) | Natural language find() API |
-| [API Surface Design](./architecture/API_SURFACE_DESIGN.md) | API design principles |
-| [API Returns](./api-returns.md) | Return types and structures |
+| [Plugins](./PLUGINS.md) | Plugin system overview — providers, `plugins` config, `brain.use()` |
-### 🔧 Framework Integration
+---
+
+## Performance & Scaling
| Document | Description |
|----------|-------------|
-| [Framework Integration Guide](./guides/framework-integration.md) | React, Vue, Angular, Svelte, etc. |
-| [Next.js Integration](./guides/nextjs-integration.md) | Server-side rendering with Brainy |
-| [Vue.js Integration](./guides/vue-integration.md) | Vue 3 integration patterns |
+| [Performance](./PERFORMANCE.md) | Optimization techniques |
+| [Scaling](./SCALING.md) | Scale to billions of entities |
+| [Batching](./BATCHING.md) | Batch operations guide |
-### 📁 Virtual Filesystem (VFS)
+---
+
+## Migration & Reference
| Document | Description |
|----------|-------------|
-| [VFS Core Documentation](./vfs/VFS_CORE.md) | Core VFS concepts and architecture |
-| [Semantic VFS Guide](./vfs/SEMANTIC_VFS.md) | Semantic filesystem projections |
-| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
-| [Neural Extraction](./vfs/NEURAL_EXTRACTION.md) | AI-powered file analysis |
-| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
-| [User Functions](./vfs/USER_FUNCTIONS.md) | Custom VFS functions |
-| [VFS Graph Types](./vfs/VFS_GRAPH_TYPES.md) | Graph-based VFS projections |
-| [VFS Examples & Scenarios](./vfs/VFS_EXAMPLES_SCENARIOS.md) | Real-world VFS examples |
-| [VFS Initialization](./vfs/VFS_INITIALIZATION.md) | Setup and configuration |
-| [Projection Strategy API](./vfs/PROJECTION_STRATEGY_API.md) | Custom projection strategies |
-| [Building File Explorers](./vfs/building-file-explorers.md) | Build custom file browsers |
-| [VFS Troubleshooting](./vfs/TROUBLESHOOTING.md) | Common issues and solutions |
-
-### 🧠 Advanced Topics
-
-| Document | Description |
-|----------|-------------|
-| [Natural Language Queries](./guides/natural-language.md) | Query in plain English |
-| [Neural API](./guides/neural-api.md) | AI-powered features |
-| [Import Anything](./guides/import-anything.md) | Import data from any source |
-| [Distributed Systems](./guides/distributed-system.md) | Multi-node deployments |
-| [Model Loading](./guides/model-loading.md) | Load and manage AI models |
-| [Model Loading Quick Reference](./MODEL_LOADING_QUICK_REFERENCE.md) | Model loading cheat sheet |
-| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No paywalls, no tiers |
-
-### 🔌 Augmentations (Plugins)
-
-| Document | Description |
-|----------|-------------|
-| [Creating Augmentations](./CREATING-AUGMENTATIONS.md) | Build custom plugins |
-| [Augmentations Complete Reference](./augmentations/COMPLETE-REFERENCE.md) | Full augmentation API |
-| [Augmentations Developer Guide](./augmentations/DEVELOPER-GUIDE.md) | Plugin development guide |
-| [Augmentation Configuration](./augmentations/CONFIGURATION.md) | Configure augmentations |
-| [Augmentation System](./architecture/augmentations.md) | System architecture |
-| [Augmentation System Audit](./architecture/augmentation-system-audit.md) | Actual vs planned features |
-| [Augmentations Actual](./architecture/augmentations-actual.md) | Currently implemented augmentations |
-| [API Server Augmentation](./augmentations/api-server.md) | REST API server plugin |
-
-### ⚡ Performance & Scaling
-
-| Document | Description |
-|----------|-------------|
-| [Performance Guide](./PERFORMANCE.md) | Optimization techniques |
-| [Performance Analysis](./architecture/PERFORMANCE_ANALYSIS.md) | Benchmarks and analysis |
-| [Scaling Guide](./SCALING.md) | Scale to billions of entities |
-| [Clustering Algorithms Analysis](./architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md) | HNSW algorithm details |
-| [Initialization & Rebuild](./architecture/initialization-and-rebuild.md) | Index management |
-
-### 📋 Reference
-
-| Document | Description |
-|----------|-------------|
-| [Complete Feature List](./features/complete-feature-list.md) | All Brainy features |
-| [v3 Features](./features/v3-features.md) | v3.x feature list |
-| [Metadata Architecture](./architecture/METADATA_ARCHITECTURE.md) | Metadata namespacing |
-| [Metadata Contract Implementation](./METADATA_CONTRACT_IMPLEMENTATION.md) | Metadata API contracts |
-| [Finite Type System](./architecture/finite-type-system.md) | Type-safe noun/verb taxonomy |
-| [Validation](./VALIDATION.md) | Data validation rules |
-| [Universal Display Augmentation](./universal-display-augmentation.md) | Display system |
-
-### 🛠️ Development
-
-| Document | Description |
-|----------|-------------|
-| [Troubleshooting](./troubleshooting.md) | Common issues and solutions |
+| [v3 to v4 Migration](./MIGRATION-V3-TO-V4.md) | Upgrade guide |
| [Release Guide](./RELEASE-GUIDE.md) | How to release new versions |
+| [Production Architecture](./PRODUCTION_SERVICE_ARCHITECTURE.md) | Ops reference |
-### 🔍 Internal Documentation
+---
+
+## Internal
| Document | Description |
|----------|-------------|
| [Audit Report](./internal/AUDIT_REPORT.md) | Feature audit |
-| [Cleanup Summary](./internal/CLEANUP_SUMMARY.md) | Codebase cleanup notes |
| [Honest Status](./internal/HONEST_STATUS.md) | Actual implementation status |
---
-## Documentation Structure
-
-```
-docs/
-├── README.md (this file) # Complete documentation index
-├── MIGRATION-V3-TO-V4.md # v4.0.0 migration guide
-│
-├── guides/ # User guides
-│ ├── natural-language.md
-│ ├── neural-api.md
-│ ├── import-anything.md
-│ ├── framework-integration.md
-│ ├── nextjs-integration.md
-│ ├── vue-integration.md
-│ ├── distributed-system.md
-│ ├── model-loading.md
-│ └── enterprise-for-everyone.md
-│
-├── architecture/ # System architecture
-│ ├── overview.md
-│ ├── noun-verb-taxonomy.md
-│ ├── triple-intelligence.md
-│ ├── zero-config.md
-│ ├── storage-architecture.md # v4.0.0
-│ ├── data-storage-architecture.md # v4.0.0
-│ ├── index-architecture.md
-│ ├── distributed-storage.md
-│ ├── augmentations.md
-│ ├── augmentation-system-audit.md
-│ ├── augmentations-actual.md
-│ ├── finite-type-system.md
-│ └── ...
-│
-├── operations/ # Operations guides
-│ ├── cost-optimization-aws-s3.md # v4.0.0
-│ ├── cost-optimization-gcs.md # v4.0.0
-│ ├── cost-optimization-azure.md # v4.0.0
-│ ├── cost-optimization-cloudflare-r2.md # v4.0.0
-│ └── capacity-planning.md
-│
-├── deployment/ # Deployment guides
-│ ├── CLOUD_DEPLOYMENT_GUIDE.md # v4.0.0
-│ ├── aws-deployment.md
-│ ├── gcp-deployment.md
-│ └── kubernetes-deployment.md
-│
-├── vfs/ # Virtual Filesystem docs
-│ ├── QUICK_START.md
-│ ├── VFS_CORE.md
-│ ├── SEMANTIC_VFS.md
-│ ├── VFS_API_GUIDE.md
-│ ├── NEURAL_EXTRACTION.md
-│ ├── COMMON_PATTERNS.md
-│ └── ...
-│
-├── api/ # API documentation
-│ ├── README.md
-│ └── COMPREHENSIVE_API_OVERVIEW.md
-│
-├── augmentations/ # Augmentation docs
-│ ├── COMPLETE-REFERENCE.md
-│ ├── DEVELOPER-GUIDE.md
-│ ├── CONFIGURATION.md
-│ └── api-server.md
-│
-├── features/ # Feature documentation
-│ ├── complete-feature-list.md
-│ └── v3-features.md
-│
-└── internal/ # Internal docs
- ├── AUDIT_REPORT.md
- ├── CLEANUP_SUMMARY.md
- └── HONEST_STATUS.md
-```
-
-## Community
-
-- **GitHub**: [github.com/brainy-org/brainy](https://github.com/brainy-org/brainy)
-- **Issues**: [Report bugs or request features](https://github.com/brainy-org/brainy/issues)
-- **Discussions**: [Join the conversation](https://github.com/brainy-org/brainy/discussions)
-
## License
-Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
\ No newline at end of file
+Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
diff --git a/docs/RELEASE-GUIDE.md b/docs/RELEASE-GUIDE.md
index bf8abb9f..94c6a6cb 100644
--- a/docs/RELEASE-GUIDE.md
+++ b/docs/RELEASE-GUIDE.md
@@ -48,7 +48,7 @@ git commit -m "chore: remove unused dependency"
# DON'T use BREAKING CHANGE for internal changes
git commit -m "feat: improve model delivery
-BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers v3.0.0
+BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers
```
## Release Workflow Checklist
@@ -119,4 +119,13 @@ gh release create vY.Y.Y --generate-notes
- **Most releases should be MINOR or PATCH**
- **Major versions should be RARE**
- **When in doubt, it's probably MINOR**
-- **NEVER use "BREAKING CHANGE" for internal changes**
\ No newline at end of file
+- **NEVER use "BREAKING CHANGE" for internal changes**
+## Hard Ordering Constraints (check before EVERY release)
+
+- **Embedding model changes are SEQUENCED, not free.** No release may change the
+ embedding model (or its quantization/dimensions) before **vector model-version
+ stamping + hard-error-on-mismatch** ships. Stored vectors carry no model version
+ today; mixing vectors from two models silently corrupts every similarity
+ comparison. If a model bump is ever proposed, the stamping work moves ahead of
+ it in the schedule — coordinate with the native provider so both engines stamp
+ and enforce identically. (Registered with the native-provider team 2026-07-07.)
diff --git a/docs/SCALING.md b/docs/SCALING.md
index d071a838..e9ae1136 100644
--- a/docs/SCALING.md
+++ b/docs/SCALING.md
@@ -1,502 +1,239 @@
-# 🚀 Brainy Scaling Guide - Enterprise for Everyone
+# Brainy Scaling Guide
-> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required.
+> **One Line Summary**: Single-node by design — Brainy scales by getting the most out of one machine plus operator-layer backup.
-## 📖 Table of Contents
+## Table of Contents
- [Quick Start](#quick-start)
-- [How It Works](#how-it-works)
+- [How Brainy Scales](#how-brainy-scales)
- [Storage Configurations](#storage-configurations)
- [Scaling Patterns](#scaling-patterns)
- [Real World Examples](#real-world-examples)
## Quick Start
-### Single Node (Default)
+### In-Memory
```typescript
import Brainy from '@soulcraft/brainy'
-const brain = new Brainy() // That's it!
+const brain = new Brainy({ storage: { type: 'memory' } })
```
-### Multi-Node (Auto-Discovery)
+### On-Disk (Default for Node)
```typescript
-// Node 1
-const brain = new Brainy() // Starts as primary
-
-// Node 2 (different server)
-const brain = new Brainy() // Auto-discovers Node 1, becomes replica!
-```
-
-**That's literally all you need!** Brainy handles everything else automatically.
-
-## How It Works
-
-### 🎯 The Magic: Zero Configuration
-
-Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration:
-
-1. **First node starts** → Becomes primary automatically
-2. **Second node starts** → Discovers first node via UDP broadcast
-3. **Nodes negotiate** → Elect leader, distribute shards
-4. **Data flows** → Automatic replication and routing
-5. **Node fails** → Automatic failover in <1 second
-
-### 🔄 Automatic Node Discovery
-
-```typescript
-// Three ways Brainy finds other nodes (auto-selected):
-
-// 1. LOCAL NETWORK (Default)
-// Uses UDP broadcast on port 7946
-// Perfect for: On-premise, same VPC
-
-// 2. CLOUD NATIVE (Auto-detected)
-// Kubernetes: Uses k8s DNS service discovery
-// AWS: Uses EC2 tags or Route53
-// Azure: Uses Azure DNS
-
-// 3. EXPLICIT (When needed)
const brain = new Brainy({
- peers: ['node1.example.com', 'node2.example.com']
+ storage: { type: 'filesystem', path: './brainy-data' }
})
```
-### 📊 Data Distribution
+## How Brainy Scales
-When you add data, Brainy automatically:
+Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery, no S3 coordination. Scaling means:
-```typescript
-brain.add({ name: "John" }, 'person')
+- **Up**: give the process more RAM, CPU, and IOPS
+- **Out**: stand up multiple independent Brainy instances behind your own service layer
+- **Cold storage**: snapshot the on-disk artifact off-site so you can rehydrate elsewhere
-// Behind the scenes:
-// 1. Hash ID to determine shard (consistent hashing)
-// 2. Find nodes responsible for this shard
-// 3. Write to primary shard owner
-// 4. Replicate to N backup nodes (default: 2)
-// 5. Confirm write when majority acknowledge
-```
+The three knobs that matter most:
+
+1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
+2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
+
+The native vector provider (via the optional `@soulcraft/cor` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
+
+## Measured Performance
+
+Numbers below are **measured** by `tests/benchmarks/find-composition-scale.js` (a single
+Node 22 process, in-memory storage, 384-dim vectors, `balanced` recall). They are the
+open-core (pure-TypeScript) path — what you get from `@soulcraft/brainy` with no native
+provider installed. Run it yourself: `node --max-old-space-size=8192 tests/benchmarks/find-composition-scale.js 100000`.
+
+`find()` query latency, p50 / p95 (200 queries each):
+
+| Query | 5,000 entities | 100,000 entities |
+|---|---|---|
+| Vector similarity (`{ vector }`) | 0.8 / 1.3 ms | 1.4 / 4.7 ms |
+| Graph 1-hop (`{ connected }`) | 0.5 / 0.7 ms | 0.7 / 0.8 ms |
+| Metadata filter (`{ where }`, low-selectivity) | 0.7 / 1.2 ms | 23.5 / 30.1 ms |
+| Vector + metadata | 7.7 / 8.3 ms | 78.8 / 93.8 ms |
+
+What the shape tells you:
+
+- **Vector and graph lookups scale ~logarithmically** — they barely move from 5k to 100k,
+ because HNSW search is ~O(ef·log n) and graph adjacency is O(degree).
+- **Metadata-filtered paths scale with the size of the match set, not the database.** The
+ benchmark's `category` filter matches ~10% of rows (10,000 at 100k); the cost is
+ materializing that candidate set and running the vector search *inside* it (`find()` does
+ metadata-first hard filtering, then ranks within the candidates — see
+ [How find works](./FIND_SYSTEM.md)). A **high-selectivity** filter (few matches) is far
+ cheaper; a 10%-of-everything filter is the worst case. This candidate-restricted search is
+ precisely the path the native provider accelerates (Rust roaring-bitmap candidate
+ intersection).
+- **Composition is correct, not lossy.** Combining vector + metadata + graph returns exactly
+ the entities satisfying all constraints — verified by
+ `tests/integration/find-triple-composition.test.ts`.
+
+Memory: ~62 KB resident per entity at 100k (6.2 GB RSS for 100k × 384-dim including the HNSW
+graph, metadata index, and 100k edges).
+
+**Scale ceiling (open-core).** The pure-JS HNSW *build* cost (~100 inserts/s at 384-dim on
+one core) makes the in-process open-core path most appropriate up to ~10⁵–10⁶ entities.
+*Query* latency stays low well beyond that, but for the 10⁸–10¹⁰ regime install the native
+provider (`@soulcraft/cor`, on-disk DiskANN) — same API, no code change. _Projected from
+the two measured points, vector p50 at 1M is ~2 ms; metadata-heavy composition grows with
+match-set size and is the path to move onto the native provider first._
## Storage Configurations
-### 🗂️ Storage Adapter Patterns
-
-Brainy intelligently adapts to your storage setup:
-
-#### Pattern 1: Separate Storage Per Node (Recommended)
-
+### Filesystem (Recommended for Production)
```typescript
-// Node 1 - Own filesystem
-const brain1 = new Brainy({
- storage: '/data/node1' // or auto: './brainy-data'
-})
-
-// Node 2 - Own filesystem
-const brain2 = new Brainy({
- storage: '/data/node2' // or auto: './brainy-data'
-})
-
-// ✅ BENEFITS:
-// - No conflicts between nodes
-// - Fast local reads
-// - True horizontal scaling
-// - Survives network partitions
-```
-
-#### Pattern 2: Separate S3 Buckets Per Node
-
-```typescript
-// Node 1 - Own S3 bucket
-const brain1 = new Brainy({
- storage: 's3://brainy-node-1' // Auto-uses AWS credentials
-})
-
-// Node 2 - Own S3 bucket
-const brain2 = new Brainy({
- storage: 's3://brainy-node-2'
-})
-
-// ✅ BENEFITS:
-// - Infinite storage capacity
-// - Geographic distribution
-// - No local disk needed
-// - Built-in durability
-```
-
-#### Pattern 3: Shared S3 Bucket (Coordinated)
-
-```typescript
-// All nodes - Shared bucket with coordination
-const brain = new Brainy({
- storage: 's3://shared-brainy-data',
- // Brainy automatically adds node-specific prefixes!
-})
-
-// What happens automatically:
-// - Node 1 writes to: s3://shared-brainy-data/node-1/
-// - Node 2 writes to: s3://shared-brainy-data/node-2/
-// - Metadata in: s3://shared-brainy-data/_cluster/
-// - Coordination via S3 conditional writes
-
-// ✅ BENEFITS:
-// - Single bucket to manage
-// - Easy backup/restore
-// - Cost effective
-// - Automatic namespace isolation
-```
-
-#### Pattern 4: Mixed Storage (Hybrid)
-
-```typescript
-// Hot data on local SSD, cold data in S3
const brain = new Brainy({
storage: {
- hot: '/fast-ssd/brainy', // Recent/frequent data
- cold: 's3://brainy-archive' // Older data
+ type: 'filesystem',
+ path: '/var/lib/brainy'
}
- // Brainy automatically promotes/demotes data!
})
```
+- Stores everything in a sharded JSON tree under `path`
+- Atomic writes via rename
+- Survives process restarts
+- Snapshot it off-site with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` from your scheduler
-### 🌍 Cloud Provider Auto-Detection
+### Memory
+```typescript
+const brain = new Brainy({ storage: { type: 'memory' } })
+```
+- Zero I/O, fastest possible
+- No persistence — process exit discards everything
+- Use for tests and ephemeral caches
+### Auto
```typescript
const brain = new Brainy({
- storage: 'cloud://brainy-data' // Auto-detects provider!
+ storage: { type: 'auto', path: './data' }
})
-
-// Automatically uses:
-// - AWS: S3 + DynamoDB for metadata
-// - Google Cloud: GCS + Firestore
-// - Azure: Blob Storage + Cosmos DB
-// - Cloudflare: R2 + D1
-// - Vercel: Blob + KV
```
-
-### 📝 Storage Coordination Rules
-
-When multiple nodes share storage, Brainy automatically:
-
-1. **Namespace Isolation**: Each node gets unique prefix
-2. **Lock-Free Writes**: Uses atomic operations
-3. **Consistent Metadata**: Coordinated via consensus
-4. **Conflict Resolution**: Version vectors for conflicts
-5. **Garbage Collection**: Automatic cleanup of old data
+- Picks `filesystem` when running on Node with a writable `path`
+- Falls back to `memory` otherwise
## Scaling Patterns
-### 📈 Progressive Scaling Journey
-
-#### Stage 1: Prototype (1 node, memory)
+### Stage 1: Prototype (Memory)
```typescript
-const brain = new Brainy() // Memory storage, single node
-// Perfect for: Development, testing, <1000 items
+const brain = new Brainy({ storage: { type: 'memory' } })
+// Development, tests, <100K items
```
-#### Stage 2: Production (1 node, disk)
+### Stage 2: Production (Filesystem)
```typescript
const brain = new Brainy({
- storage: './data' // Persistent storage
+ storage: { type: 'filesystem', path: '/var/lib/brainy' }
})
-// Perfect for: Small apps, <100K items
+// Most production workloads up to ~10M entities on a single host
```
-#### Stage 3: High Availability (2-3 nodes)
+### Stage 3: Higher Throughput (Tune the Vector Index)
```typescript
-// Just start same code on multiple servers!
const brain = new Brainy({
- storage: './data' // Each node's own storage
+ storage: { type: 'filesystem', path: '/var/lib/brainy' },
+ vector: {
+ recall: 'fast', // Trade recall for latency
+ persistMode: 'deferred' // Batch persistence
+ }
})
-// Automatic: Leader election, replication, failover
-// Perfect for: Critical apps, <1M items
```
-#### Stage 4: Scale Out (N nodes)
-```typescript
-// Same code, more servers!
-const brain = new Brainy({
- storage: 's3://brainy-{{nodeId}}' // Template auto-filled
-})
-// Automatic: Sharding, load balancing, geo-distribution
-// Perfect for: Large apps, unlimited items
-```
-
-### 🎯 Common Scaling Scenarios
-
-#### Scenario: Read-Heavy Application
-```typescript
-// Brainy auto-detects read-heavy pattern and:
-// 1. Increases cache size
-// 2. Creates more read replicas
-// 3. Routes reads to nearest node
-// 4. Caches popular items on all nodes
-
-const brain = new Brainy() // No config needed!
-```
-
-#### Scenario: Multi-Tenant SaaS
-```typescript
-// Brainy auto-detects tenant patterns and:
-// 1. Shards by tenant ID
-// 2. Isolates tenant data
-// 3. Routes by tenant
-// 4. Separate rate limits per tenant
-
-const brain = new Brainy() // Detects from your queries!
-```
-
-#### Scenario: Geographic Distribution
-```typescript
-// Deploy nodes in different regions
-// Brainy automatically:
-// 1. Detects node locations (via latency)
-// 2. Replicates data geographically
-// 3. Routes to nearest node
-// 4. Handles region failures
-
-// US-East
-const brain = new Brainy({ region: 'us-east' }) // Optional hint
-
-// EU-West (auto-discovers US-East)
-const brain = new Brainy({ region: 'eu-west' })
-```
+### Stage 4: Multi-Instance (Operator-Layer)
+Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `path`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes.
## Real World Examples
-### Example 1: Blog Platform
-
+### Example 1: Single-Node App With Backup
```typescript
-// Day 1: Single server
const brain = new Brainy({
- storage: './blog-data'
+ storage: { type: 'filesystem', path: '/var/lib/brainy' }
})
-
-// Month 6: Add redundancy (on second server)
-const brain = new Brainy({
- storage: './blog-data' // Different machine!
-})
-// Automatically syncs with first server
-
-// Year 2: Global scale
-// US Server
-const brain = new Brainy({
- storage: 's3://blog-us/data'
-})
-
-// EU Server
-const brain = new Brainy({
- storage: 's3://blog-eu/data'
-})
-
-// Asia Server
-const brain = new Brainy({
- storage: 's3://blog-asia/data'
-})
-// All automatically coordinate!
+```
+Schedule (cron / systemd timer):
+```bash
+*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
```
-### Example 2: E-Commerce Site
-
+### Example 2: Tests
```typescript
-// Development
-const brain = new Brainy() // Memory storage
-
-// Staging (Kubernetes)
-const brain = new Brainy({
- storage: process.env.STORAGE_PATH // Uses PVC
-})
-// Auto-discovers other pods via K8s DNS
-
-// Production (AWS)
-const brain = new Brainy({
- storage: 's3://shop-data',
- cache: 'elasticache://shop-cache' // Optional
-})
-// Auto-scales with ECS/EKS
+const brain = new Brainy({ storage: { type: 'memory' } })
+// Fast, no cleanup needed between runs
```
-### Example 3: Analytics Platform
-
+### Example 3: Multi-Tenant Service
+Spin up one Brainy instance per tenant, each in its own directory:
```typescript
-// Ingestion nodes (write-optimized)
-const brain = new Brainy({
- role: 'writer', // Hint for optimization
- storage: '/fast-nvme/ingest'
-})
-
-// Query nodes (read-optimized)
-const brain = new Brainy({
- role: 'reader', // More cache, indexes
- storage: 's3://analytics-archive'
-})
-
-// Automatically coordinates between writers and readers!
-```
-
-## 🔧 Storage Adapter Specifics
-
-### Local Filesystem
-```typescript
-{
- storage: './data' // or absolute: '/var/lib/brainy'
- // Each node MUST have separate directory
- // Can be network mounted (NFS, EFS)
+function brainForTenant(tenantId: string) {
+ return new Brainy({
+ storage: {
+ type: 'filesystem',
+ path: `/var/lib/brainy/${tenantId}`
+ }
+ })
}
```
+Your service layer handles routing and isolation; Brainy stays simple.
-### AWS S3
+### Example 4: Higher Recall at Scale
```typescript
-{
- storage: 's3://bucket-name/prefix'
- // Uses AWS SDK credentials (env, IAM role, etc)
- // Supports S3-compatible (MinIO, Ceph)
-}
-```
-
-### Cloudflare R2
-```typescript
-{
- storage: 'r2://bucket-name'
- // Uses Wrangler or API tokens
- // Zero egress fees!
-}
-```
-
-### Google Cloud Storage
-```typescript
-{
- storage: 'gs://bucket-name'
- // Uses Application Default Credentials
-}
-```
-
-### Azure Blob Storage
-```typescript
-{
- storage: 'azure://container-name'
- // Uses DefaultAzureCredential
-}
-```
-
-### Mixed/Tiered
-```typescript
-{
- storage: {
- hot: './local-cache', // Fast SSD
- warm: 's3://regular-data', // Standard storage
- cold: 's3://glacier-archive' // Cheap archive
+const brain = new Brainy({
+ storage: { type: 'filesystem', path: '/var/lib/brainy' },
+ vector: {
+ recall: 'accurate'
}
- // Automatic tiering based on access patterns
-}
-```
-
-## 🎭 Advanced Patterns
-
-### Pattern: Blue-Green Deployment
-```typescript
-// Blue cluster (current)
-const brain = new Brainy({
- cluster: 'blue',
- storage: 's3://prod-blue'
-})
-
-// Green cluster (new version)
-const brain = new Brainy({
- cluster: 'green',
- storage: 's3://prod-green',
- syncFrom: 'blue' // Real-time sync during migration
})
```
-### Pattern: Federation
-```typescript
-// Region 1 Cluster
-const brain1 = new Brainy({
- federation: 'global',
- region: 'us-east',
- storage: 's3://us-east-data'
-})
+## Tuning Knobs Summary
-// Region 2 Cluster
-const brain2 = new Brainy({
- federation: 'global',
- region: 'eu-west',
- storage: 's3://eu-west-data'
-})
-// Clusters coordinate for global queries!
-```
+| Setting | Values | When to change |
+|---------|--------|----------------|
+| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
+| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
+| `storage.cache.maxSize` | integer | Hot-path read cache size |
+| `storage.cache.ttl` | ms | Cache freshness |
-### Pattern: Edge Computing
-```typescript
-// Edge nodes (in CDN POPs)
-const brain = new Brainy({
- mode: 'edge',
- storage: 'memory', // RAM only
- upstream: 'https://main-cluster.example.com'
-})
-// Caches frequently accessed data at edge
-```
-
-## 📊 Monitoring & Observability
-
-Brainy automatically exposes metrics:
+## Monitoring & Observability
```typescript
-const metrics = brain.getMetrics()
+const stats = await brain.stats()
// {
-// nodes: { total: 5, healthy: 5 },
-// shards: { total: 20, local: 4 },
-// replication: { factor: 2, lag: 45 },
-// operations: { reads: 10000, writes: 1000 },
-// storage: { used: '45GB', available: '955GB' }
+// nounCount: 50000,
+// verbCount: 80000,
+// vectorIndex: { ... },
+// storage: { used: '45GB' }
// }
```
-## 🚨 Troubleshooting
+## Troubleshooting
-### Issue: Nodes don't discover each other
-```typescript
-// Solution 1: Check network allows UDP 7946
-// Solution 2: Use explicit peers
-const brain = new Brainy({
- peers: ['10.0.0.1:7946', '10.0.0.2:7946']
-})
-```
+### Issue: Slow queries
+1. Switch to `vector.recall: 'fast'`
+2. Increase the read cache (`storage.cache.maxSize`)
+3. Consider the optional native vector provider via `@soulcraft/cor`
-### Issue: Storage conflicts
-```typescript
-// Ensure each node has unique storage path
-// ❌ WRONG: All nodes use './data'
-// ✅ RIGHT: Node1: './data1', Node2: './data2'
-// ✅ RIGHT: Use {{nodeId}} template
-```
+### Issue: Memory pressure
+1. Reduce `storage.cache.maxSize`
+2. Move to `vector.persistMode: 'deferred'` to batch writes
+3. Consider the optional native vector provider via `@soulcraft/cor` for at-scale index acceleration
-### Issue: Slow performance
-```typescript
-// Brainy auto-tunes, but you can hint:
-const brain = new Brainy({
- profile: 'read-heavy' // or 'write-heavy', 'balanced'
-})
-```
+### Issue: Slow startup after a crash
+1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
+2. Verify backup integrity periodically
-## 🎯 Best Practices
+## Best Practices
-1. **Let Brainy Auto-Configure**: Don't over-configure
-2. **Separate Storage Per Node**: Avoids conflicts
-3. **Use S3 for Large Scale**: Infinite capacity
-4. **Start Simple**: Single node → Scale when needed
-5. **Monitor Metrics**: Watch for bottlenecks
-6. **Trust Auto-Scaling**: It learns your patterns
+1. **One process = one `path`** — never share a directory between processes
+2. **Snapshot from your scheduler** — Brainy doesn't ship cloud SDKs; use `rclone` / `aws s3 sync` / `gsutil`
+3. **Profile before tuning** — `recall: 'balanced'` is right for most workloads
+4. **Install the native vector provider only when measured profiling shows it pays off**
-## 🚀 Summary
+## Summary
-- **Zero Config**: Just `new Brainy()` at any scale
-- **Auto-Discovery**: Nodes find each other
-- **Smart Storage**: Adapts to any backend
-- **Progressive Scaling**: 1 → 100 nodes seamlessly
-- **Self-Tuning**: Learns and optimizes
-- **No DevOps**: It just works!
-
-**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!**
-
----
-
-*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)*
\ No newline at end of file
+- Brainy 8.0 is a **library**, not a cluster
+- Storage adapters: `filesystem`, `memory`, `auto`
+- Vector tuning: `recall`, `persistMode`
+- Backup is an operator-layer concern — snapshot `path`
diff --git a/docs/STAGE3-CANONICAL-TAXONOMY.md b/docs/STAGE3-CANONICAL-TAXONOMY.md
index b1fab35a..bfafb9e5 100644
--- a/docs/STAGE3-CANONICAL-TAXONOMY.md
+++ b/docs/STAGE3-CANONICAL-TAXONOMY.md
@@ -1,4 +1,4 @@
-# Brainy Stage 3: Canonical Taxonomy (v6.0.0)
+# Brainy Stage 3: Canonical Taxonomy
**Status:** FINAL - This is the definitive, timeless taxonomy
**Total Types:** 169 (42 nouns + 127 verbs)
diff --git a/docs/VALIDATION.md b/docs/VALIDATION.md
deleted file mode 100644
index a0c42668..00000000
--- a/docs/VALIDATION.md
+++ /dev/null
@@ -1,340 +0,0 @@
-# Brainy Validation System
-
-## Zero-Config Philosophy
-
-Brainy's validation system automatically adapts to your system resources without any configuration. It enforces universal truths while dynamically adjusting limits based on available memory and observed performance.
-
-## Core Principles
-
-### 1. Universal Truths Only
-We only validate things that are mathematically or logically impossible:
-- Negative pagination values (there's no page -1)
-- Probabilities outside 0-1 range
-- Self-referential relationships
-- Invalid enum values
-
-### 2. Auto-Configuration
-The system automatically configures based on:
-- **Available Memory**: More RAM = higher limits
-- **System Performance**: Adjusts based on query response times
-- **Usage Patterns**: Learns from your actual workload
-
-### 3. Performance Monitoring
-Every query is monitored to tune future limits:
-```typescript
-// Automatic adjustment based on performance
-if (avgQueryTime < 100ms && resultCount > 80% of limit) {
- // Increase limits - system can handle more
- maxLimit *= 1.5
-} else if (avgQueryTime > 1000ms) {
- // Reduce limits - system is struggling
- maxLimit *= 0.8
-}
-```
-
-## Validation Rules by Method
-
-### `add(params: AddParams)`
-
-**Required:**
-- Either `data` or `vector` must be provided
-- `type` must be a valid NounType enum
-
-**Constraints:**
-- `vector` must have exactly 384 dimensions (for all-MiniLM-L6-v2)
-- Custom `id` must be unique
-
-**Example:**
-```typescript
-// ✅ Valid
-await brain.add({
- data: "Hello world",
- type: NounType.Document
-})
-
-// ✅ Valid - pre-computed vector
-await brain.add({
- vector: new Array(384).fill(0),
- type: NounType.Document
-})
-
-// ❌ Invalid - missing both data and vector
-await brain.add({
- type: NounType.Document
-})
-// Error: "must provide either data or vector"
-
-// ❌ Invalid - wrong vector dimensions
-await brain.add({
- vector: new Array(100).fill(0),
- type: NounType.Document
-})
-// Error: "vector must have exactly 384 dimensions"
-```
-
-### `update(params: UpdateParams)`
-
-**Required:**
-- `id` must be provided
-- At least one field must be updated
-
-**Important Metadata Behavior:**
-- `metadata: null` with `merge: false` → **Keeps existing metadata** (does nothing)
-- `metadata: {}` with `merge: false` → **Clears metadata** ✅
-- `metadata: undefined` → No change to metadata
-
-**Example:**
-```typescript
-// ✅ Valid - update metadata
-await brain.update({
- id: "xyz",
- metadata: { status: "published" }
-})
-
-// ✅ Valid - clear metadata properly
-await brain.update({
- id: "xyz",
- metadata: {},
- merge: false
-})
-
-// ❌ Invalid - null doesn't clear metadata
-await brain.update({
- id: "xyz",
- metadata: null,
- merge: false
-})
-// Error: "must specify at least one field to update"
-// (because null metadata doesn't actually update anything)
-
-// ❌ Invalid - no fields to update
-await brain.update({
- id: "xyz"
-})
-// Error: "must specify at least one field to update"
-```
-
-### `relate(params: RelateParams)`
-
-**Required:**
-- `from` entity ID
-- `to` entity ID
-- `type` must be valid VerbType enum
-
-**Constraints:**
-- `from` and `to` must be different (no self-loops)
-- `weight` must be between 0 and 1
-
-**Example:**
-```typescript
-// ✅ Valid
-await brain.relate({
- from: "entity1",
- to: "entity2",
- type: VerbType.RelatedTo
-})
-
-// ❌ Invalid - self-referential
-await brain.relate({
- from: "entity1",
- to: "entity1",
- type: VerbType.RelatedTo
-})
-// Error: "cannot create self-referential relationship"
-
-// ❌ Invalid - weight out of range
-await brain.relate({
- from: "entity1",
- to: "entity2",
- type: VerbType.RelatedTo,
- weight: 1.5
-})
-// Error: "weight must be between 0 and 1"
-```
-
-### `find(params: FindParams)`
-
-**Constraints:**
-- `limit` must be non-negative and below auto-configured maximum
-- `offset` must be non-negative
-- Cannot specify both `query` and `vector` (mutually exclusive)
-- Cannot use both `cursor` and `offset` pagination
-- `threshold` must be between 0 and 1
-
-**Auto-Configured Limits:**
-```typescript
-// Based on available memory
-// 1GB RAM → max limit: 10,000
-// 8GB RAM → max limit: 80,000
-// 16GB RAM → max limit: 100,000 (capped)
-
-// Query length also scales with memory
-// 1GB RAM → max query: 5,000 characters
-// 8GB RAM → max query: 40,000 characters
-```
-
-**Example:**
-```typescript
-// ✅ Valid
-await brain.find({
- query: "machine learning",
- limit: 50
-})
-
-// ❌ Invalid - negative limit
-await brain.find({
- query: "test",
- limit: -1
-})
-// Error: "limit must be non-negative"
-
-// ❌ Invalid - both query and vector
-await brain.find({
- query: "test",
- vector: new Array(384).fill(0)
-})
-// Error: "cannot specify both query and vector - they are mutually exclusive"
-
-// ❌ Invalid - exceeds auto-configured limit
-await brain.find({
- limit: 1000000
-})
-// Error: "limit exceeds auto-configured maximum of 80000 (based on available memory)"
-```
-
-## Auto-Configuration Details
-
-### Memory-Based Scaling
-
-The validation system checks available memory on initialization:
-
-```typescript
-const availableMemory = os.freemem()
-
-// Scale limits based on available memory
-maxLimit = Math.min(
- 100000, // Absolute maximum for safety
- Math.floor(availableMemory / (1024 * 1024 * 100)) * 1000
-)
-
-// Scale query length similarly
-maxQueryLength = Math.min(
- 50000,
- Math.floor(availableMemory / (1024 * 1024 * 10)) * 1000
-)
-```
-
-### Performance-Based Tuning
-
-The system continuously monitors and adjusts:
-
-1. **After each query**, performance is recorded
-2. **Limits adjust** based on response times
-3. **Gradual optimization** towards optimal throughput
-
-### Checking Current Configuration
-
-You can inspect the current validation configuration:
-
-```typescript
-import { getValidationConfig } from '@soulcraft/brainy/validation'
-
-const config = getValidationConfig()
-console.log(config)
-// {
-// maxLimit: 80000,
-// maxQueryLength: 40000,
-// maxVectorDimensions: 384,
-// systemMemory: 17179869184,
-// availableMemory: 8589934592
-// }
-```
-
-## Best Practices
-
-### 1. Clearing Metadata
-```typescript
-// ❌ Wrong - doesn't clear
-await brain.update({ id, metadata: null, merge: false })
-
-// ✅ Correct - actually clears
-await brain.update({ id, metadata: {}, merge: false })
-```
-
-### 2. Type Safety
-```typescript
-// ❌ Wrong - string type
-await brain.add({ data: "test", type: "document" })
-
-// ✅ Correct - enum type
-import { NounType } from '@soulcraft/brainy'
-await brain.add({ data: "test", type: NounType.Document })
-```
-
-### 3. Pagination
-```typescript
-// ✅ Let the system auto-configure limits
-const results = await brain.find({
- query: "test",
- limit: 100 // Will be capped at system maximum
-})
-
-// ✅ For large datasets, use pagination
-let offset = 0
-const pageSize = 1000
-while (true) {
- const results = await brain.find({
- query: "test",
- limit: pageSize,
- offset
- })
- if (results.length === 0) break
- offset += pageSize
-}
-```
-
-## Error Messages
-
-All validation errors are descriptive and actionable:
-
-| Error | Cause | Solution |
-|-------|-------|----------|
-| `"must provide either data or vector"` | Missing content in add() | Provide either data to embed or pre-computed vector |
-| `"limit must be non-negative"` | Negative pagination | Use positive limit value |
-| `"invalid NounType: xyz"` | Invalid enum value | Use valid NounType enum |
-| `"cannot create self-referential relationship"` | from === to | Use different entity IDs |
-| `"must specify at least one field to update"` | Empty update | Provide at least one field to change |
-| `"vector must have exactly 384 dimensions"` | Wrong vector size | Use 384-dimensional vectors |
-
-## Performance Impact
-
-The validation system adds minimal overhead:
-- **Validation time**: <1ms per operation
-- **Memory usage**: ~1KB for configuration tracking
-- **Auto-tuning**: Happens asynchronously, no blocking
-
-## FAQ
-
-**Q: Why can't I set metadata to null?**
-A: Setting metadata to `null` with `merge: false` doesn't actually clear it - it falls back to existing metadata. Use `{}` to clear.
-
-**Q: Why are my limits being reduced?**
-A: If queries are taking >1 second, the system automatically reduces limits to maintain performance.
-
-**Q: Can I override the auto-configured limits?**
-A: No, this is by design. The system knows better than static configuration what your hardware can handle.
-
-**Q: Why exactly 384 dimensions for vectors?**
-A: Brainy uses the all-MiniLM-L6-v2 model which produces 384-dimensional embeddings. This ensures consistency.
-
-## Summary
-
-Brainy's validation system:
-- ✅ **Zero configuration** - adapts to your system
-- ✅ **Universal truths** - only prevents impossible operations
-- ✅ **Performance aware** - adjusts based on actual performance
-- ✅ **Type safe** - enforces enum types
-- ✅ **Minimal overhead** - <1ms validation time
-- ✅ **Clear errors** - actionable error messages
-
-The philosophy is simple: prevent impossible operations, adapt to reality, and get out of the way.
\ No newline at end of file
diff --git a/docs/ZERO_CONFIG.md b/docs/ZERO_CONFIG.md
deleted file mode 100644
index 6f20a74f..00000000
--- a/docs/ZERO_CONFIG.md
+++ /dev/null
@@ -1,406 +0,0 @@
-# 🚀 Brainy Zero-Configuration Guide
-
-## Overview
-
-Starting with v2.10, Brainy introduces a **Zero-Configuration System** that automatically configures everything based on your environment. No more environment variables, no more complex configuration objects - just create and use.
-
-## Quick Start
-
-### True Zero Config
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-// That's it! Everything auto-configures
-const brain = new Brainy()
-await brain.init()
-```
-
-### Using Strongly-Typed Presets
-
-```typescript
-import { Brainy, PresetName } from '@soulcraft/brainy'
-
-// Type-safe preset selection
-const brain = new Brainy(PresetName.PRODUCTION)
-await brain.init()
-```
-
-### Common Scenarios
-
-#### Development
-```typescript
-const brain = new Brainy(PresetName.DEVELOPMENT)
-// ✅ Filesystem storage for persistence
-// ✅ FP32 models for best quality
-// ✅ Verbose logging
-// ✅ All features enabled
-```
-
-#### Production
-```typescript
-const brain = new Brainy(PresetName.PRODUCTION)
-// ✅ Disk storage for persistence
-// ✅ Auto-selected model precision
-// ✅ Silent logging
-// ✅ Optimized features
-```
-
-#### Minimal
-```typescript
-const brain = new Brainy(PresetName.MINIMAL)
-// ✅ Filesystem storage
-// ✅ Q8 models for small size
-// ✅ Core features only
-// ✅ Minimal resource usage
-```
-
-## Distributed Architecture Presets
-
-Brainy includes specialized presets for distributed and microservice architectures:
-
-### Basic Distributed Roles
-```typescript
-import { Brainy, PresetName } from '@soulcraft/brainy'
-
-// Write-only instance (data ingestion)
-const writer = new Brainy(PresetName.WRITER)
-// ✅ Optimized for writes
-// ✅ No search index loading
-// ✅ Minimal memory usage
-
-// Read-only instance (search API)
-const reader = new Brainy(PresetName.READER)
-// ✅ Optimized for search
-// ✅ Lazy index loading
-// ✅ Large cache
-```
-
-### Service-Specific Presets
-```typescript
-// High-throughput data ingestion
-const ingestion = new Brainy(PresetName.INGESTION_SERVICE)
-
-// Low-latency search API
-const searchApi = new Brainy(PresetName.SEARCH_API)
-
-// Analytics processing
-const analytics = new Brainy(PresetName.ANALYTICS_SERVICE)
-
-// Edge location cache
-const edge = new Brainy(PresetName.EDGE_CACHE)
-
-// Batch processing
-const batch = new Brainy(PresetName.BATCH_PROCESSOR)
-
-// Real-time streaming
-const streaming = new Brainy(PresetName.STREAMING_SERVICE)
-
-// ML training
-const training = new Brainy(PresetName.ML_TRAINING)
-
-// Lightweight sidecar
-const sidecar = new Brainy(PresetName.SIDECAR)
-```
-
-## Model Precision Control
-
-You can **explicitly specify** model precision when needed:
-
-```typescript
-import { ModelPrecision } from '@soulcraft/brainy'
-
-// Force FP32 (full precision)
-const brain = new Brainy({ model: ModelPrecision.FP32 })
-
-// Force Q8 (quantized, smaller)
-const brain = new Brainy({ model: ModelPrecision.Q8 })
-
-// Use presets
-const brain = new Brainy({ model: ModelPrecision.FAST }) // Maps to fp32
-const brain = new Brainy({ model: ModelPrecision.SMALL }) // Maps to q8
-
-// Auto-detection (default)
-const brain = new Brainy({ model: ModelPrecision.AUTO })
-```
-
-### Auto-Detection Logic
-
-When not specified, Brainy automatically selects the best model:
-
-- **Browser**: Q8 (smaller download)
-- **Serverless**: Q8 (faster cold starts)
-- **Low Memory (<512MB)**: Q8
-- **Development**: FP32 (best quality)
-- **Production (>2GB RAM)**: FP32
-- **Default**: Q8 (balanced)
-
-## Storage Configuration
-
-### Automatic Storage Detection
-
-Brainy automatically detects the best storage option:
-
-1. **Cloud Storage** (if credentials found)
- - AWS S3 (checks AWS_ACCESS_KEY_ID, AWS_PROFILE)
- - Google Cloud Storage (checks GOOGLE_APPLICATION_CREDENTIALS)
- - Cloudflare R2 (checks R2_ACCESS_KEY_ID)
-
-2. **Browser Storage**
- - OPFS (if supported)
- - Filesystem fallback
-
-3. **Node.js Storage**
- - Filesystem (`./brainy-data` or `~/.brainy/data`)
-
-### Manual Storage Control
-
-```typescript
-import { StorageOption } from '@soulcraft/brainy'
-
-// Force specific storage with enum
-const brain = new Brainy({ storage: StorageOption.DISK })
-const brain = new Brainy({ storage: StorageOption.CLOUD })
-const brain = new Brainy({ storage: StorageOption.AUTO })
-
-// Custom storage configuration
-const brain = new Brainy({
- storage: {
- s3Storage: {
- bucket: 'my-bucket',
- region: 'us-east-1'
- }
- }
-})
-```
-
-## Feature Sets
-
-Control which features are enabled:
-
-```typescript
-import { FeatureSet } from '@soulcraft/brainy'
-
-// Preset feature sets with enum
-const brain = new Brainy({ features: FeatureSet.MINIMAL }) // Core only
-const brain = new Brainy({ features: FeatureSet.DEFAULT }) // Balanced
-const brain = new Brainy({ features: FeatureSet.FULL }) // Everything
-
-// Custom features
-const brain = new Brainy({
- features: ['core', 'search', 'cache', 'triple-intelligence']
-})
-```
-
-## Simplified Configuration Interface
-
-The new configuration is dramatically simpler:
-
-```typescript
-interface BrainyZeroConfig {
- // Mode preset - now with distributed options
- mode?: PresetName // All strongly typed presets
-
- // Model configuration with enum
- model?: ModelPrecision
-
- // Storage configuration with enum
- storage?: StorageOption | StorageConfig
-
- // Feature set with enum
- features?: FeatureSet | string[]
-
- // Logging
- verbose?: boolean
-
- // Escape hatch for advanced users
- advanced?: any
-}
-```
-
-### Available Enums
-
-```typescript
-enum PresetName {
- // Basic
- PRODUCTION = 'production',
- DEVELOPMENT = 'development',
- MINIMAL = 'minimal',
- ZERO = 'zero',
-
- // Distributed
- WRITER = 'writer',
- READER = 'reader',
-
- // Services
- INGESTION_SERVICE = 'ingestion-service',
- SEARCH_API = 'search-api',
- ANALYTICS_SERVICE = 'analytics-service',
- EDGE_CACHE = 'edge-cache',
- BATCH_PROCESSOR = 'batch-processor',
- STREAMING_SERVICE = 'streaming-service',
- ML_TRAINING = 'ml-training',
- SIDECAR = 'sidecar'
-}
-
-enum ModelPrecision {
- FP32 = 'fp32',
- Q8 = 'q8',
- AUTO = 'auto',
- FAST = 'fast', // Maps to fp32
- SMALL = 'small' // Maps to q8
-}
-
-enum StorageOption {
- AUTO = 'auto',
- DISK = 'disk',
- CLOUD = 'cloud'
-}
-
-enum FeatureSet {
- MINIMAL = 'minimal',
- DEFAULT = 'default',
- FULL = 'full'
-}
-```
-
-## Multi-Instance with Shared Storage
-
-When multiple Brainy instances connect to the same storage (like S3), you **must ensure they use compatible configurations**:
-
-```typescript
-import { ModelPrecision } from '@soulcraft/brainy'
-
-// Container A - Writer
-const writer = new Brainy({
- mode: PresetName.WRITER,
- model: ModelPrecision.FP32, // ⚠️ MUST match across instances!
- storage: { s3Storage: { bucket: 'shared-data' }}
-})
-
-// Container B - Reader
-const reader = new Brainy({
- mode: PresetName.READER,
- model: ModelPrecision.FP32, // ✅ Matches Container A
- storage: { s3Storage: { bucket: 'shared-data' }}
-})
-```
-
-### Distributed Architecture Example
-
-```typescript
-// Ingestion Service (Writer)
-const ingestion = new Brainy({
- mode: PresetName.INGESTION_SERVICE,
- model: ModelPrecision.Q8, // All instances must use Q8
- storage: { s3Storage: { bucket: 'production-data' }}
-})
-
-// Search API (Reader)
-const search = new Brainy({
- mode: PresetName.SEARCH_API,
- model: ModelPrecision.Q8, // Matches ingestion service
- storage: { s3Storage: { bucket: 'production-data' }}
-})
-
-// Analytics (Hybrid)
-const analytics = new Brainy({
- mode: PresetName.ANALYTICS_SERVICE,
- model: ModelPrecision.Q8, // Matches other services
- storage: { s3Storage: { bucket: 'production-data' }}
-})
-```
-
-## Migration from Old Configuration
-
-### Before (Complex)
-```typescript
-const brain = new Brainy({
- hnsw: {
- M: 16,
- efConstruction: 200,
- seed: 42
- },
- storage: {
- s3Storage: {
- bucketName: 'my-bucket',
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
- region: 'us-east-1'
- },
- cacheConfig: {
- hotCacheMaxSize: 5000,
- hotCacheEvictionThreshold: 0.8,
- warmCacheTTL: 3600000,
- batchSize: 100
- }
- },
- cache: {
- autoTune: true,
- autoTuneInterval: 60000,
- hotCacheMaxSize: 10000
- },
- embeddingFunction: customFunction,
- readOnly: false,
- logging: { verbose: true }
-})
-```
-
-### After (Simple)
-```typescript
-const brain = new Brainy('production')
-// Everything above is auto-configured!
-```
-
-## Environment Variables (No Longer Needed!)
-
-These environment variables are **no longer required**:
-
-- ❌ `BRAINY_ALLOW_REMOTE_MODELS` - Models auto-download when needed
-- ❌ `BRAINY_MODELS_PATH` - Path auto-selected based on environment
-- ❌ `BRAINY_Q8_CONFIRMED` - Warnings auto-suppressed in production
-- ❌ `BRAINY_LOG_LEVEL` - Auto-set based on NODE_ENV
-- ❌ AWS credentials - Use AWS SDK credential chain
-
-## Performance Impact
-
-The zero-config system has **zero performance overhead**:
-
-- Configuration happens once during initialization
-- Auto-detected values are cached
-- Same optimized code paths as manual configuration
-- Actually **faster** startup due to reduced parsing
-
-## Troubleshooting
-
-### Models Not Downloading
-- Check internet connection
-- Ensure firewall allows HTTPS to Hugging Face / CDN
-- Run `npm run download-models` to pre-download
-
-### Wrong Model Precision
-- Explicitly specify: `{ model: 'fp32' }` or `{ model: 'q8' }`
-- Check shared storage compatibility
-
-### Storage Detection Issues
-- Check cloud credentials are properly configured
-- Verify write permissions for filesystem paths
-- Use explicit storage configuration if needed
-
-## Best Practices
-
-1. **Use zero-config for single instances** - Let Brainy handle everything
-2. **Specify precision for shared storage** - Ensure compatibility
-3. **Use presets for common scenarios** - 'development', 'production', 'minimal'
-4. **Override only what you need** - Start simple, add complexity only if required
-
-## Summary
-
-The new zero-config system reduces configuration from **100+ parameters** to **0-3 decisions**:
-
-| Scenario | Old Config Lines | New Config Lines |
-|----------|-----------------|------------------|
-| Development | 50+ | 1 |
-| Production | 100+ | 1 |
-| Custom | 200+ | 3-5 |
-
-**Result**: 95% less configuration, 100% of the power! 🚀
\ No newline at end of file
diff --git a/docs/api-returns.md b/docs/api-returns.md
deleted file mode 100644
index a9f5e76a..00000000
--- a/docs/api-returns.md
+++ /dev/null
@@ -1,152 +0,0 @@
-# Brainy API Return Values
-
-## Core Operations
-
-### `brain.add()` - Adding Entities
-
-**Returns:** `Promise` - The ID of the created entity
-
-```typescript
-// ✅ Correct usage - add() returns the ID string directly
-const id = await brain.add({
- type: 'document',
- data: 'My document content'
-})
-
-console.log('Created entity with ID:', id)
-
-// Use the ID to create relationships
-await brain.relate({
- from: id,
- to: anotherId,
- type: 'references'
-})
-```
-
-```typescript
-// ❌ Incorrect - trying to access .id property
-const result = await brain.add({
- type: 'document',
- data: 'My content'
-})
-
-console.log(result.id) // ❌ undefined - result IS the ID, not an object!
-```
-
-### Getting the Full Entity
-
-If you need the full entity object after creation, use `brain.get()`:
-
-```typescript
-// Add entity and get its ID
-const id = await brain.add({
- type: 'document',
- data: 'My content',
- metadata: { label: 'Important Doc' }
-})
-
-// Get the full entity
-const entity = await brain.get(id)
-console.log(entity.id) // The ID
-console.log(entity.type) // 'document'
-console.log(entity.data) // 'My content'
-console.log(entity.metadata) // { label: 'Important Doc', ... }
-console.log(entity.vector) // The embedding vector
-console.log(entity.createdAt) // Timestamp
-```
-
-### `brain.find()` - Finding Entities
-
-**Returns:** `Promise` - Array of full entity objects
-
-```typescript
-const entities = await brain.find({
- query: 'machine learning',
- limit: 10
-})
-
-// Each entity has full information
-for (const entity of entities) {
- console.log(entity.id)
- console.log(entity.type)
- console.log(entity.data)
- console.log(entity.metadata)
-}
-```
-
-### `brain.relate()` - Creating Relationships
-
-**Returns:** `Promise` - The ID of the created relationship
-
-```typescript
-const relationId = await brain.relate({
- from: entityId1,
- to: entityId2,
- type: 'references'
-})
-
-console.log('Created relationship with ID:', relationId)
-```
-
-## Data Field Behavior
-
-### String Data - Used for Embeddings
-
-When `data` is a string, it's used to generate embeddings for semantic search:
-
-```typescript
-await brain.add({
- type: 'document',
- data: 'This text will be converted to an embedding vector',
- metadata: {
- title: 'My Document',
- year: 2024
- }
-})
-```
-
-### Object Data - Structured Information
-
-When `data` is an object, it's treated as structured data:
-
-```typescript
-await brain.add({
- type: 'product',
- data: {
- name: 'Widget',
- price: 29.99,
- category: 'Tools'
- },
- vector: precomputedVector // Must provide vector when using object data
-})
-```
-
-**Important:** If you provide object data without a `vector`, you must include a string somewhere for embedding generation, or the operation will fail.
-
-### Metadata vs Data
-
-- **`data`**: Primary content - used for embeddings (if string) or stored as structured data (if object)
-- **`metadata`**: Auxiliary information - always stored as structured data, used for filtering
-
-**Best practice for labels:**
-```typescript
-await brain.add({
- type: 'document',
- data: 'The full text content of the document...', // For semantic search
- metadata: {
- label: 'Quick Reference Label', // For display
- author: 'John Doe',
- category: 'Technical'
- }
-})
-```
-
-## Summary
-
-| Method | Returns | Contains |
-|--------|---------|----------|
-| `brain.add()` | `string` | The ID of the created entity |
-| `brain.get()` | `Entity \| null` | Full entity object with all fields |
-| `brain.find()` | `Entity[]` | Array of full entity objects |
-| `brain.relate()` | `string` | The ID of the created relationship |
-| `brain.getRelations()` | `Relation[]` | Array of relationship objects |
diff --git a/docs/api/COMPREHENSIVE_API_OVERVIEW.md b/docs/api/COMPREHENSIVE_API_OVERVIEW.md
deleted file mode 100644
index 3a729be5..00000000
--- a/docs/api/COMPREHENSIVE_API_OVERVIEW.md
+++ /dev/null
@@ -1,563 +0,0 @@
-# Brainy Complete Public API Reference
-
-> **Accurate API documentation for Brainy v6.5.0+**
-
-## Initialization
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-// Zero-config (just works)
-const brain = new Brainy()
-await brain.init()
-
-// With configuration
-const brain = new Brainy({
- storage: { type: 'memory' }, // or { path: './my-data' } for filesystem
- embeddingModel: 'Q8', // Q4, Q8, F16, F32
- silent: true // Suppress logs
-})
-await brain.init()
-```
-
-## Readiness API (v7.3.0+)
-
-For reliable initialization detection, especially in cloud environments with progressive initialization:
-
-### `brain.ready` - Await Initialization
-
-```typescript
-const brain = new Brainy()
-brain.init() // Fire and forget
-
-// Elsewhere (e.g., API handler)
-await brain.ready // Wait until init() completes
-const results = await brain.find({ query: 'test' })
-```
-
-### `brain.isInitialized` - Check Basic Readiness
-
-```typescript
-if (brain.isInitialized) {
- // Safe to use brain methods
-}
-```
-
-### `brain.isFullyInitialized()` - Check Background Tasks
-
-```typescript
-// Returns true when ALL initialization is complete, including background tasks
-// Useful for cloud storage adapters with progressive initialization
-if (brain.isFullyInitialized()) {
- console.log('All background tasks complete')
-}
-```
-
-### `brain.awaitBackgroundInit()` - Wait for Background Tasks
-
-```typescript
-const brain = new Brainy({ storage: { type: 'gcs', ... } })
-await brain.init() // Fast return in cloud (<200ms)
-
-// Optional: wait for all background tasks (bucket validation, count sync)
-await brain.awaitBackgroundInit()
-console.log('Fully initialized including background tasks')
-```
-
-### Health Check Pattern
-
-```typescript
-app.get('/health', async (req, res) => {
- try {
- await brain.ready
- res.json({
- status: 'ready',
- fullyInitialized: brain.isFullyInitialized()
- })
- } catch (error) {
- res.status(503).json({ status: 'initializing', error: error.message })
- }
-})
-```
-
-## Core CRUD Operations
-
-### `brain.add(params)` - Add Entity
-
-```typescript
-// Add with data (auto-embedded)
-const id = await brain.add({
- data: 'Machine learning is a subset of AI',
- type: NounType.Document,
- metadata: { author: 'Alice', tags: ['ml', 'ai'] }
-})
-
-// Add with pre-computed vector
-const id = await brain.add({
- data: 'Content here',
- vector: [0.1, 0.2, ...], // 384 dimensions
- type: NounType.Concept
-})
-```
-
-**Parameters:**
-- `data` (required): Content to embed (string, object, or any serializable data)
-- `type?`: NounType enum value
-- `metadata?`: Custom key-value pairs
-- `vector?`: Pre-computed embedding vector
-- `id?`: Custom ID (auto-generated if not provided)
-
-### `brain.get(id, options?)` - Get Entity
-
-```typescript
-const entity = await brain.get('entity-id')
-
-// Include vector embeddings (not loaded by default for performance)
-const entity = await brain.get('entity-id', { includeVectors: true })
-```
-
-### `brain.update(params)` - Update Entity
-
-```typescript
-await brain.update({
- id: 'entity-id',
- data: 'Updated content', // Re-embeds if changed
- metadata: { reviewed: true } // Merges with existing
-})
-```
-
-### `brain.delete(id)` - Delete Entity
-
-```typescript
-await brain.delete('entity-id')
-```
-
-### `brain.clear()` - Clear All Data
-
-```typescript
-await brain.clear()
-```
-
-## Relationships
-
-### `brain.relate(params)` - Create Relationship
-
-```typescript
-const relationId = await brain.relate({
- from: 'source-entity-id',
- to: 'target-entity-id',
- type: VerbType.RelatedTo,
- metadata: { strength: 0.9 }
-})
-```
-
-**Parameters:**
-- `from` (required): Source entity ID
-- `to` (required): Target entity ID
-- `type` (required): VerbType enum value
-- `metadata?`: Custom relationship metadata
-
-### `brain.getRelations(params)` - Query Relationships
-
-```typescript
-// Get all relationships from an entity
-const relations = await brain.getRelations({ from: 'entity-id' })
-
-// Get relationships to an entity
-const relations = await brain.getRelations({ to: 'entity-id' })
-
-// Filter by type
-const relations = await brain.getRelations({
- from: 'entity-id',
- type: VerbType.Contains
-})
-```
-
-### `brain.unrelate(id)` - Delete Relationship
-
-```typescript
-await brain.unrelate('relationship-id')
-```
-
-## Search & Query
-
-### `brain.find(query)` - Semantic Search
-
-The primary search method with Triple Intelligence (semantic + graph + metadata).
-
-```typescript
-// Simple text search
-const results = await brain.find('machine learning algorithms')
-
-// With options
-const results = await brain.find({
- query: 'machine learning',
- limit: 10,
- threshold: 0.7,
- type: NounType.Document,
- where: { author: 'Alice' },
- excludeVFS: true // Exclude VFS files from results
-})
-
-// Natural language query (Triple Intelligence)
-const results = await brain.find(
- 'Show me documents about AI written by Alice in 2024'
-)
-```
-
-**Parameters:**
-- `query`: Search text (required)
-- `limit?`: Max results (default: 10)
-- `threshold?`: Minimum similarity (0-1)
-- `type?`: Filter by NounType
-- `where?`: Metadata filters
-- `excludeVFS?`: Exclude VFS entities
-
-### `brain.similar(params)` - Find Similar Entities
-
-```typescript
-// Find similar to entity ID
-const similar = await brain.similar({
- to: 'entity-id',
- limit: 5
-})
-
-// Find similar to vector
-const similar = await brain.similar({
- to: [0.1, 0.2, ...], // Vector
- limit: 10,
- type: NounType.Document
-})
-```
-
-## Batch Operations
-
-### `brain.addMany(params)` - Batch Add
-
-```typescript
-const result = await brain.addMany({
- items: [
- { data: 'First item', type: NounType.Document },
- { data: 'Second item', type: NounType.Concept },
- { data: 'Third item', metadata: { priority: 'high' } }
- ],
- continueOnError: true, // Don't stop on failures
- onProgress: (completed, total) => {
- console.log(`${completed}/${total} complete`)
- }
-})
-
-console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}`)
-```
-
-### `brain.deleteMany(params)` - Batch Delete
-
-```typescript
-// Delete by IDs
-const result = await brain.deleteMany({
- ids: ['id1', 'id2', 'id3'],
- continueOnError: true
-})
-
-// Delete by type
-const result = await brain.deleteMany({
- type: NounType.TempData
-})
-
-// Delete by metadata filter
-const result = await brain.deleteMany({
- where: { status: 'archived' }
-})
-```
-
-### `brain.relateMany(params)` - Batch Relate
-
-```typescript
-const ids = await brain.relateMany({
- items: [
- { from: 'a', to: 'b', type: VerbType.RelatedTo },
- { from: 'b', to: 'c', type: VerbType.Contains },
- { from: 'c', to: 'd', type: VerbType.References }
- ],
- continueOnError: true
-})
-```
-
-### `brain.updateMany(params)` - Batch Update
-
-```typescript
-const result = await brain.updateMany({
- items: [
- { id: 'id1', metadata: { reviewed: true } },
- { id: 'id2', data: 'Updated content' }
- ],
- continueOnError: true
-})
-```
-
-## Neural API
-
-Access advanced AI/ML features via `brain.neural()`:
-
-```typescript
-const neural = brain.neural()
-
-// Similarity calculation
-const similarity = await neural.similar('text1', 'text2')
-const detailed = await neural.similar('text1', 'text2', { detailed: true })
-
-// Clustering
-const clusters = await neural.clusters()
-const clusters = await neural.clusters({
- algorithm: 'hierarchical',
- maxClusters: 10
-})
-
-// K-nearest neighbors
-const neighbors = await neural.neighbors('entity-id', { k: 5 })
-
-// Semantic hierarchy
-const hierarchy = await neural.hierarchy('entity-id')
-
-// Outlier/anomaly detection
-const outliers = await neural.outliers({ threshold: 2.0 })
-
-// Domain-aware clustering
-const domainClusters = await neural.clusterByDomain('category')
-
-// Temporal clustering
-const temporalClusters = await neural.clusterByTime('createdAt', [
- { start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' },
- { start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' }
-])
-
-// Streaming clusters (for large datasets)
-for await (const batch of neural.clusterStream({ batchSize: 100 })) {
- console.log(`Progress: ${batch.progress.percentage}%`)
-}
-```
-
-## Virtual File System (VFS)
-
-Access via `brain.vfs`:
-
-```typescript
-const vfs = brain.vfs
-await vfs.init()
-
-// File operations
-await vfs.writeFile('/docs/readme.md', '# Hello')
-const content = await vfs.readFile('/docs/readme.md')
-await vfs.unlink('/docs/readme.md')
-
-// Directory operations
-await vfs.mkdir('/project/src', { recursive: true })
-const files = await vfs.readdir('/project')
-await vfs.rmdir('/project', { recursive: true })
-
-// Bulk operations (v6.5.0+)
-const result = await vfs.bulkWrite([
- { type: 'mkdir', path: '/data' },
- { type: 'write', path: '/data/config.json', data: '{}' },
- { type: 'write', path: '/data/users.json', data: '[]' }
-])
-// Note: mkdir operations run first (sequentially), then other ops in parallel
-
-// Semantic search
-const results = await vfs.search('authentication code', { path: '/src' })
-
-// File metadata
-const stats = await vfs.stat('/file.txt')
-await vfs.setMetadata('/file.txt', { author: 'Alice' })
-```
-
-## Counts (O(1) Performance)
-
-```typescript
-// Entity counts
-const total = brain.counts.entities()
-const byType = await brain.counts.byType(NounType.Document)
-const nonVFS = await brain.counts.byType({ excludeVFS: true })
-
-// Relationship counts
-const relations = brain.counts.relationships()
-const byVerb = await brain.counts.byVerbType(VerbType.Contains)
-```
-
-## Versioning & Branching
-
-```typescript
-// Create branch
-await brain.fork('feature-branch')
-
-// List branches
-const branches = await brain.listBranches()
-
-// Switch branch
-await brain.checkout('feature-branch')
-
-// Get current branch
-const current = await brain.getCurrentBranch()
-
-// Commit changes
-await brain.commit({ message: 'Added new features' })
-
-// View history
-const history = await brain.getHistory({ limit: 10 })
-
-// Time travel (read-only snapshot)
-const snapshot = await brain.asOf('commit-id')
-```
-
-## Streaming API
-
-```typescript
-// Stream all entities
-for await (const entity of brain.streaming.entities()) {
- console.log(entity.id)
-}
-
-// Stream with filters
-for await (const entity of brain.streaming.entities({
- type: NounType.Document,
- where: { status: 'active' }
-})) {
- // Process each entity
-}
-
-// Stream relationships
-for await (const relation of brain.streaming.relations({
- from: 'entity-id'
-})) {
- console.log(relation)
-}
-```
-
-## Pagination API
-
-```typescript
-// Paginated queries
-const page1 = await brain.pagination.find({
- query: 'machine learning',
- page: 1,
- pageSize: 20
-})
-
-console.log(`Page ${page1.page} of ${page1.totalPages}`)
-console.log(`Total results: ${page1.total}`)
-
-// Get next page
-const page2 = await brain.pagination.find({
- query: 'machine learning',
- page: 2,
- pageSize: 20
-})
-```
-
-## Augmentations
-
-```typescript
-// List active augmentations
-const augmentations = brain.augmentations.list()
-
-// Get specific augmentation
-const cache = brain.augmentations.get('cache')
-
-// Augmentations are auto-loaded based on config
-// Common augmentations: cache, display, metrics, intelligent-import
-```
-
-## Utilities
-
-```typescript
-// Manual embedding
-const vector = await brain.embed('text to embed')
-
-// Flush pending writes
-await brain.flush()
-
-// Get statistics
-const stats = await brain.getStats()
-const statsNoVFS = await brain.getStats({ excludeVFS: true })
-
-// Close (cleanup)
-await brain.close()
-```
-
-## Type Enums
-
-```typescript
-import { NounType, VerbType } from '@soulcraft/brainy'
-
-// NounType - Entity types
-NounType.Document
-NounType.Person
-NounType.Concept
-NounType.Event
-NounType.Location
-NounType.Organization
-NounType.Product
-NounType.Content
-NounType.Collection
-// ... and more
-
-// VerbType - Relationship types
-VerbType.RelatedTo
-VerbType.Contains
-VerbType.References
-VerbType.DependsOn
-VerbType.Precedes
-VerbType.Follows
-VerbType.CreatedBy
-VerbType.ModifiedBy
-// ... and more
-```
-
-## Error Handling
-
-```typescript
-try {
- await brain.get('nonexistent-id')
-} catch (error) {
- if (error.message.includes('not found')) {
- // Handle missing entity
- }
-}
-
-// VFS errors use POSIX codes
-try {
- await vfs.readFile('/nonexistent')
-} catch (error) {
- if (error.code === 'ENOENT') {
- // File not found
- }
-}
-```
-
-## Configuration Reference
-
-```typescript
-const brain = new Brainy({
- // Storage
- storage: {
- type: 'memory' | 'filesystem',
- path: './data', // For filesystem
- forceMemoryStorage: false // Force memory even if path exists
- },
-
- // Embeddings
- embeddingModel: 'Q8', // Q4, Q8, F16, F32
- dimensions: 384, // Auto-detected
-
- // Performance
- silent: false, // Suppress console output
- verbose: false, // Extra logging
-
- // Augmentations (auto-enabled by default)
- augmentations: {
- cache: true,
- display: true,
- metrics: true
- }
-})
-```
diff --git a/docs/api/README.md b/docs/api/README.md
index 3873474a..4ca84364 100644
--- a/docs/api/README.md
+++ b/docs/api/README.md
@@ -1,9 +1,22 @@
-# 🧠 Brainy v7.1.0 API Reference
+---
+title: API Reference
+slug: api/reference
+public: true
+category: api
+template: api
+order: 1
+description: Complete API reference for all Brainy methods — add, find, relate, update, delete, batch operations, the Db API (transactions, snapshots, time travel), VFS, neural API, and more.
+next:
+ - getting-started/quick-start
+ - guides/find-system
+---
-> **Complete API documentation for Brainy v7.1.0**
-> Zero Configuration • Triple Intelligence • Git-Style Branching • Entity Versioning • Candle WASM Embeddings
+# 🧠 Brainy API Reference
-**Updated:** 2026-01-06 for v7.1.0
+> **Complete API documentation for Brainy**
+> Zero Configuration • Triple Intelligence • Database as a Value • Atomic Transactions • Time Travel
+
+**Updated:** 2026-06-11
**All APIs verified against actual code**
---
@@ -13,32 +26,36 @@
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-const brain = new Brainy() // Zero config!
-await brain.init() // VFS auto-initialized in v5.1.0!
+const brain = new Brainy() // Zero config!
+await brain.init() // VFS auto-initialized!
// Add data (text auto-embeds!)
const id = await brain.add({
- data: 'The future of AI is here',
- type: NounType.Concept,
- metadata: { category: 'technology' }
+ data: 'The future of AI is here',
+ type: NounType.Concept,
+ metadata: { category: 'technology' }
})
// Search with Triple Intelligence
const results = await brain.find({
- query: 'artificial intelligence',
- where: { year: { greaterThan: 2020 } },
- connected: { from: id, depth: 2 }
+ query: 'artificial intelligence',
+ where: { year: { greaterThan: 2020 } },
+ connected: { from: id, depth: 2 }
})
-// Fork for safe experimentation (v5.0.0+)
-const experiment = await brain.fork('test-feature')
-await experiment.add({ data: 'test', type: NounType.Document })
-await experiment.commit({ message: 'Add test data' })
+// Pin the current state as an immutable value
+const db = brain.now()
-// Entity versioning (v5.3.0+)
-await brain.versions.save(id, { tag: 'v1.0', description: 'Initial version' })
-await brain.update(id, { category: 'AI' })
-await brain.versions.save(id, { tag: 'v2.0' })
+// Commit an atomic multi-write batch (all-or-nothing)
+await brain.transact([
+ { op: 'update', id, metadata: { category: 'AI' } }
+], { meta: { author: 'docs-example' } })
+
+await db.get(id) // still sees the pre-transaction state — snapshot isolation
+await db.release()
+
+// Time travel: query any past state
+const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000))
```
---
@@ -49,16 +66,19 @@ await brain.versions.save(id, { tag: 'v2.0' })
Semantic vectors with metadata and relationships - the fundamental data unit in Brainy.
### 🔗 Relationships (Verbs)
-Typed connections between entities - building knowledge graphs.
+Typed connections between entities with optional `data` and `metadata` - building knowledge graphs.
+
+### 📊 Data vs Metadata
+- **`data`**: Content embedded into vectors. Searchable via **semantic similarity** (vector index) and **hybrid text+semantic** search. NOT queryable via `where` filters.
+- **`metadata`**: Structured fields indexed by MetadataIndex. Queryable via `where` filters in `find()`.
+
+See **[Data Model](../DATA_MODEL.md)** for the full explanation.
### 🧠 Triple Intelligence
Vector search + Graph traversal + Metadata filtering in one unified query.
-### 🌳 Git-Style Branching (v5.0.0+)
-Fork, experiment, and commit - Snowflake-style copy-on-write isolation.
-
-### 📜 Entity Versioning (v5.3.0+)
-Time-travel and history tracking for individual entities - Git-like version control with content-addressable storage.
+### 🧊 Database Values (Db)
+The whole store pinned as an immutable value — snapshot isolation, atomic `transact()` batches, time travel with `asOf()`, instant hard-link snapshots with `persist()`. See the [consistency model](../concepts/consistency-model.md).
---
@@ -66,17 +86,17 @@ Time-travel and history tracking for individual entities - Git-like version cont
- [Core CRUD Operations](#core-crud-operations)
- [Search & Query](#search--query)
+- [Aggregation Engine](#aggregation-engine)
- [Relationships](#relationships)
- [Batch Operations](#batch-operations)
-- [Branch Management (v5.0+)](#branch-management-v50)
-- [Entity Versioning (v5.3.0+)](#entity-versioning-v530)
+- [Database Values & Time Travel (Db API)](#database-values--time-travel-db-api)
- [Virtual Filesystem (VFS)](#virtual-filesystem-vfs)
- [Neural API](#neural-api)
- [Import & Export](#import--export)
- [Configuration](#configuration)
- [Storage Adapters](#storage-adapters)
- [Utility Methods](#utility-methods)
-- [Embedding & Analysis APIs (v7.1.0)](#embedding--analysis-apis-v710)
+- [Embedding & Analysis APIs](#embedding--analysis-apis)
- [Type System Reference](#type-system-reference)
---
@@ -89,19 +109,30 @@ Add a single entity to the database.
```typescript
const id = await brain.add({
- data: 'JavaScript is a programming language', // Text or pre-computed vector
- type: NounType.Concept, // Required: Entity type
- metadata: { // Optional metadata
- category: 'programming',
- year: 1995
- }
+ data: 'JavaScript is a programming language', // Text or pre-computed vector
+ type: NounType.Concept, // Required: Entity type
+ subtype: 'language', // Optional: sub-classification
+ metadata: { // Optional: queryable fields
+ category: 'programming',
+ year: 1995
+ }
})
```
**Parameters:**
-- `data`: `string | number[]` - Text (auto-embeds) or vector
+- `data`: `string | number[]` - Content to embed (text auto-embeds) or pre-computed vector
- `type`: `NounType` - Entity type (required)
-- `metadata?`: `object` - Additional metadata
+- `subtype?`: `string` - Per-product sub-classification within the NounType (top-level standard field, indexed on the fast path). See [Subtypes & Facets](../guides/subtypes-and-facets.md).
+- `metadata?`: `object` - Structured queryable fields (indexed by MetadataIndex, used in `where` filters)
+- `id?`: `string` - Custom ID (auto-generated UUID if not provided)
+- `vector?`: `number[]` - Pre-computed vector (skips auto-embedding)
+- `confidence?`: `number` - Type classification confidence (0-1)
+- `weight?`: `number` - Entity importance/salience (0-1)
+- `ifAbsent?`: `boolean` - By-ID idempotent insert. When `true` AND a custom `id` is supplied AND an entity with that `id` already exists, returns the existing `id` without writing (no throw, no overwrite). Ignored without `id`. See [guides/optimistic-concurrency](../guides/optimistic-concurrency.md).
+
+> **`data`** is embedded into vectors for semantic search. **`metadata`** is indexed for `where` filters. See [Data Model](../DATA_MODEL.md).
+
+> **Strict-mode tip:** if a vocabulary is registered for your `type` (via `brain.requireSubtype()` or by an SDK that wraps Brainy), you must pass a matching `subtype`. Run `await brain.audit()` to inventory pre-existing gaps before enabling strict mode; see the [migration recipe](../guides/subtypes-and-facets.md#strict-mode-in-practice-for-sdk-style-vocabulary-consumers).
**Returns:** `Promise` - Entity ID
@@ -113,9 +144,9 @@ Retrieve a single entity by ID.
```typescript
const entity = await brain.get(id)
-console.log(entity?.data) // Original data
-console.log(entity?.metadata) // Metadata
-console.log(entity?.vector) // Embedding vector
+console.log(entity?.data) // Original data
+console.log(entity?.metadata) // Metadata
+console.log(entity?.vector) // Embedding vector
```
**Parameters:**
@@ -131,27 +162,35 @@ Update an existing entity.
```typescript
await brain.update({
- id: entityId,
- data: 'Updated content', // Optional: new data
- metadata: { updated: true } // Optional: new metadata (merges)
+ id: entityId,
+ data: 'Updated content', // Optional: new data
+ subtype: 'archived', // Optional: change sub-classification
+ metadata: { updated: true } // Optional: new metadata (merges)
})
```
**Parameters:**
- `id`: `string` - Entity ID
- `data?`: `string | number[]` - New data/vector
-- `metadata?`: `object` - Metadata to merge
+- `type?`: `NounType` - Change entity type
+- `subtype?`: `string` - Change subtype (omit to preserve existing)
+- `metadata?`: `object` - Metadata to merge (or replace with `merge: false`)
+- `confidence?`: `number` - Update classification confidence
+- `weight?`: `number` - Update entity importance
+- `ifRev?`: `number` - Optimistic-concurrency check. When provided, the update throws `RevisionConflictError` if the persisted entity's `_rev` no longer equals `ifRev`. See [guides/optimistic-concurrency](../guides/optimistic-concurrency.md).
**Returns:** `Promise`
+> **Tip — read-then-CAS.** Every entity returned by `get()` / `find()` / `search()` carries `entity._rev` (a monotonic counter Brainy auto-bumps on every successful `update()`). Pass it back as `ifRev` to make multi-writer coordination safe without an external lock service. Full guide: [guides/optimistic-concurrency](../guides/optimistic-concurrency.md).
+
---
-### `delete(id)` → `Promise`
+### `remove(id)` → `Promise`
-Delete a single entity.
+Remove a single entity (and every relationship where it is source or target).
```typescript
-await brain.delete(id)
+await brain.remove(id)
```
**Parameters:**
@@ -173,58 +212,411 @@ const results = await brain.find('machine learning')
// Advanced Triple Intelligence query
const results = await brain.find({
- query: 'artificial intelligence', // Vector similarity
- where: { // Metadata filtering
- year: { greaterThan: 2020 },
- category: { oneOf: ['AI', 'ML'] }
- },
- connected: { // Graph traversal
- to: conceptId,
- depth: 2,
- type: VerbType.RelatedTo
- },
- limit: 10
+ query: 'artificial intelligence', // Vector similarity
+ where: { // Metadata filtering
+ year: { greaterThan: 2020 },
+ category: { oneOf: ['AI', 'ML'] }
+ },
+ connected: { // Graph traversal
+ to: conceptId,
+ depth: 2,
+ type: VerbType.RelatedTo
+ },
+ limit: 10
})
```
**Parameters:**
- `query`: `string | FindParams`
- - **Simple:** Just text for vector search
- - **Advanced:** Object with vector + graph + metadata filters
+ - **Simple:** Just text for vector search
+ - **Advanced:** Object with vector + graph + metadata filters
**FindParams:**
-- `query?`: `string` - Text for vector similarity
-- `where?`: `object` - Metadata filters (see [Query Operators](#query-operators))
+- `query?`: `string` - Text for semantic + hybrid search (searches `data` via the vector index + text index)
+- `type?`: `NounType | NounType[]` - Filter by entity type(s). Alias for `where.noun`.
+- `subtype?`: `string | string[]` - Filter by sub-classification (top-level standard field, fast path). Single string for equality, array for set membership.
+- `where?`: `object` - Metadata filters. See **[Query Operators](../QUERY_OPERATORS.md)** for all operators.
- `connected?`: `object` - Graph traversal options
- - `to?`: `string` - Target entity ID
- - `from?`: `string` - Source entity ID
- - `type?`: `VerbType` - Relationship type
- - `depth?`: `number` - Traversal depth
+ - `to?`: `string` - Target entity ID
+ - `from?`: `string` - Source entity ID
+ - `via?`: `VerbType | VerbType[]` - Relationship type(s) to traverse
+ - `type?`: `VerbType | VerbType[]` - Alias for `via`
+ - `depth?`: `number` - Traversal depth (default: 1)
+ - `direction?`: `'in' | 'out' | 'both'` - Traversal direction (default: 'both')
- `limit?`: `number` - Max results (default: 10)
- `offset?`: `number` - Skip results
+- `orderBy?`: `string` - Field to sort by (e.g., 'createdAt', 'metadata.priority')
+- `order?`: `'asc' | 'desc'` - Sort direction (default: 'asc')
+- `searchMode?`: `'auto' | 'text' | 'semantic' | 'hybrid'` - Search strategy:
+ - `'auto'` (default): Zero-config hybrid combining text + semantic search
+ - `'text'`: Pure keyword/text matching
+ - `'semantic'`/`'vector'`: Pure vector similarity
+ - `'hybrid'`: Explicit hybrid mode
+- `hybridAlpha?`: `number` - Balance between text (0.0) and semantic (1.0) search. Auto-detected by query length if not specified.
+- `excludeVFS?`: `boolean` - Exclude VFS entities from results (default: false)
+
+> **`limit` tip:** Brainy caps `limit` against an auto-configured maximum (based on container/free memory, ~25 KB per result). Above the cap you get a one-time warning per call site; above 2× the cap it throws. To raise the cap, pass `new Brainy({ maxQueryLimit: N })` or `{ reservedQueryMemory: bytes }`. For queries that need ALL matches, paginate with `{ limit, offset }` — that's the only pattern guaranteed to keep working across Brainy versions. See [Query Limits & Pagination](../guides/find-limits.md).
**Returns:** `Promise` - Matching entities with scores
---
+### Hybrid Search
+
+Brainy automatically combines text (keyword) and semantic (vector) search for optimal results. No configuration needed.
+
+```typescript
+// Zero-config hybrid search (just works)
+const results = await brain.find({
+ query: 'David Smith' // Finds both exact text matches AND semantically similar
+})
+
+// Force text-only search (exact keyword matching)
+const textResults = await brain.find({
+ query: 'exact keyword',
+ searchMode: 'text'
+})
+
+// Force semantic-only search (vector similarity)
+const semanticResults = await brain.find({
+ query: 'artificial intelligence concepts',
+ searchMode: 'semantic'
+})
+
+// Custom hybrid weighting (0 = text only, 1 = semantic only)
+const customResults = await brain.find({
+ query: 'David Smith',
+ hybridAlpha: 0.3 // Favor text matching
+})
+```
+
+**How it works:**
+- Short queries (1-2 words) automatically favor text matching
+- Long queries (5+ words) automatically favor semantic search
+- Results are combined using Reciprocal Rank Fusion (RRF)
+
+---
+
+### Match Visibility
+
+Search results include detailed match information:
+
+```typescript
+const results = await brain.find({ query: 'david the warrior' })
+
+// Each result now includes:
+results[0].textMatches // ["david", "warrior"] - exact query words found
+results[0].textScore // 0.25 - text match quality (0-1)
+results[0].semanticScore // 0.87 - semantic similarity (0-1)
+results[0].matchSource // 'both' | 'text' | 'semantic'
+```
+
+**Use cases:**
+- Highlight exact matches in UI (textMatches)
+- Explain why a result ranked high (matchSource)
+- Debug search behavior (separate scores)
+
+---
+
+### `highlight(params)` → `Promise` ✨
+
+Zero-config highlighting for both exact matches AND semantic concepts.
+Handles plain text, rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill), HTML, and Markdown automatically.
+
+```typescript
+// Plain text (works as before)
+const highlights = await brain.highlight({
+ query: "david the warrior",
+ text: "David Smith is a brave fighter who battles dragons"
+})
+// [
+// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' },
+// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' },
+// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
+// ]
+
+// Rich-text JSON (auto-detected)
+const highlights = await brain.highlight({
+ query: "david the warrior",
+ text: JSON.stringify(tiptapDocument) // TipTap, Slate, Lexical, Draft.js, Quill
+})
+// Extracts text from nodes, annotates with contentCategory:
+// [
+// { text: "David", score: 1.0, matchType: 'text', contentCategory: 'title' },
+// { text: "fighter", score: 0.78, matchType: 'semantic', contentCategory: 'content' }
+// ]
+
+// HTML input (auto-detected)
+const highlights = await brain.highlight({
+ query: "warrior",
+ text: "David the Warrior
A brave fighter.
"
+})
+
+// Custom extractor for proprietary formats
+const highlights = await brain.highlight({
+ query: "function",
+ text: sourceCode,
+ contentExtractor: (text) => treeSitterParse(text) // Your custom parser
+})
+```
+
+**Parameters:**
+- `query`: `string` - The search query
+- `text`: `string` - Text to highlight (plain text, JSON, HTML, or Markdown)
+- `granularity?`: `'word' | 'phrase' | 'sentence'` - Highlight unit (default: 'word')
+- `threshold?`: `number` - Min similarity for semantic matches (default: 0.5)
+- `contentType?`: `ContentType` - Optional hint: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`. Skips auto-detection when provided.
+- `contentExtractor?`: `(text: string) => ExtractedSegment[]` - Custom parser. Bypasses built-in detection entirely.
+
+**Returns:** `Promise`
+- `text` - The matched text
+- `score` - Match score (1.0 for text matches, varies for semantic)
+- `position` - [start, end] indices in extracted text
+- `matchType` - `'text'` (exact) or `'semantic'` (concept)
+- `contentCategory?` - `'title' | 'annotation' | 'content' | 'value' | 'code' | 'structural'` — Role of the source text. Built-in extractors produce `'title'`, `'content'`, `'code'`. All 6 categories are available for custom parsers.
+
+**Supported Rich-Text Formats:**
+
+| Format | Detection | Text nodes |
+|--------|-----------|------------|
+| TipTap / ProseMirror | `{ type: 'doc', content: [...] }` | `{ type: 'text', text }` |
+| Slate.js | `[{ type, children }]` | `{ text }` |
+| Lexical | `{ root: { children } }` | `{ type: 'text', text }` |
+| Draft.js | `{ blocks: [{ text }] }` | `{ text }` in block |
+| Quill Delta | `{ ops: [{ insert }] }` | `{ insert }` |
+| HTML | Tags like ``, `
`, `` | Visible text content |
+| Markdown | `#` headings, ` ``` ` code blocks | Stripped markup |
+
+**Timeout Protection:**
+Semantic matching has a 10-second timeout. If embedding takes too long (e.g., WASM stall), `highlight()` returns text-only matches instead of hanging.
+
+**UI Pattern:**
+```typescript
+// Style differently based on match type and content category
+highlights.forEach(h => {
+ const style = h.matchType === 'text' ? 'font-weight: bold' : 'background: yellow'
+ if (h.contentCategory === 'title') { /* render as heading highlight */ }
+ if (h.contentCategory === 'code') { /* render with code styling */ }
+ if (h.contentCategory === 'annotation') { /* render as comment/caption */ }
+ // Apply style from h.position[0] to h.position[1]
+})
+```
+
+---
+
### Query Operators
-Brainy uses clean, readable operators:
+Brainy uses clean, readable operators (BFO — Brainy Field Operators):
| Operator | Description | Example |
|----------|-------------|---------|
-| `equals` | Exact match | `{age: {equals: 25}}` |
-| `greaterThan` | Greater than | `{age: {greaterThan: 18}}` |
-| `lessThan` | Less than | `{price: {lessThan: 100}}` |
-| `greaterEqual` | Greater or equal | `{score: {greaterEqual: 90}}` |
-| `lessEqual` | Less or equal | `{rating: {lessEqual: 3}}` |
-| `oneOf` | In array | `{color: {oneOf: ['red', 'blue']}}` |
-| `notOneOf` | Not in array | `{status: {notOneOf: ['deleted']}}` |
-| `contains` | Contains value | `{tags: {contains: 'ai'}}` |
+| `equals` / `eq` | Exact match | `{age: {equals: 25}}` |
+| `notEquals` / `ne` | Not equal | `{status: {notEquals: 'deleted'}}` |
+| `greaterThan` / `gt` | Greater than | `{age: {greaterThan: 18}}` |
+| `gte` / `greaterThanOrEqual` | Greater or equal | `{score: {gte: 90}}` |
+| `lessThan` / `lt` | Less than | `{price: {lessThan: 100}}` |
+| `lte` / `lessThanOrEqual` | Less or equal | `{rating: {lte: 3}}` |
+| `between` | Inclusive range | `{year: {between: [2020, 2025]}}` |
+| `oneOf` / `in` | In array | `{color: {oneOf: ['red', 'blue']}}` |
+| `noneOf` | Not in array | `{status: {noneOf: ['deleted']}}` |
+| `contains` | Array contains value | `{tags: {contains: 'ai'}}` |
+| `exists` / `missing` | Field existence | `{email: {exists: true}}` |
| `startsWith` | String prefix | `{name: {startsWith: 'John'}}` |
| `endsWith` | String suffix | `{email: {endsWith: '@gmail.com'}}` |
| `matches` | Pattern match | `{text: {matches: /^[A-Z]/}}` |
-| `between` | Range | `{year: {between: [2020, 2024]}}` |
+| `allOf` | AND combinator | `{allOf: [{active: true}, {role: 'admin'}]}` |
+| `anyOf` | OR combinator | `{anyOf: [{role: 'admin'}, {role: 'owner'}]}` |
+
+**[Complete Operator Reference →](../QUERY_OPERATORS.md)** — all operators, aliases, indexed vs in-memory support matrix, and practical examples.
+
+---
+
+## Aggregation Engine
+
+Brainy's aggregation engine maintains **incremental running totals** at write time, delivering O(1) aggregate reads regardless of dataset size. Define aggregates once, and every `add()`, `update()`, and `delete()` automatically updates the running metrics.
+
+### `defineAggregate(definition)` → `void`
+
+Register a named aggregate for incremental computation.
+
+```typescript
+brain.defineAggregate({
+ name: 'monthly_spending',
+ source: {
+ type: NounType.Event,
+ where: { domain: 'financial' } // matches custom metadata fields
+ },
+ groupBy: [
+ 'category',
+ { field: 'date', window: 'month' } // Time-windowed dimension
+ ],
+ metrics: {
+ total: { op: 'sum', field: 'amount' },
+ count: { op: 'count' },
+ average: { op: 'avg', field: 'amount' },
+ highest: { op: 'max', field: 'amount' },
+ lowest: { op: 'min', field: 'amount' },
+ spread: { op: 'stddev', field: 'amount' } // Welford's online algorithm
+ },
+ materialize: true // Optional: write results as NounType.Measurement entities
+})
+```
+
+**Parameters:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | `string` | Unique identifier for this aggregate |
+| `source.type` | `NounType \| NounType[]` | Entity types that feed into this aggregate |
+| `source.where` | `Record` | Filter on custom **metadata** fields (matched against the entity's `metadata` bag) |
+| `source.service` | `string` | Multi-tenancy filter |
+| `groupBy` | `GroupByDimension[]` | Dimensions to group by — plain field names, `{ field, window }` for time bucketing, or `{ field, unnest: true }` for array fields (one contribution per element) |
+| `metrics` | `Record` | Named metrics with `op` (`sum`, `count`, `avg`, `min`, `max`, `stddev`, `variance`, `percentile`, `distinctCount`) and optional `field`. `percentile` additionally requires `p` in `[0, 1]`. |
+| `materialize` | `boolean \| object` | Write results as `NounType.Measurement` entities (auto-visible in OData/Sheets/SSE) |
+
+**Time window granularities:** `'hour'`, `'day'`, `'week'`, `'month'`, `'quarter'`, `'year'`, or `{ seconds: number }` for custom intervals.
+
+### `removeAggregate(name)` → `void`
+
+Remove a named aggregate and clean up its state.
+
+```typescript
+brain.removeAggregate('monthly_spending')
+```
+
+### Querying Aggregates via `find()`
+
+Aggregate results are queried through the standard `find()` method using the `aggregate` parameter:
+
+```typescript
+// Simple: query by name
+const results = await brain.find({ aggregate: 'monthly_spending' })
+
+// With filtering on group keys
+const foodOnly = await brain.find({
+ aggregate: 'monthly_spending',
+ where: { category: 'food' }
+})
+
+// With sorting and pagination
+const topCategories = await brain.find({
+ aggregate: {
+ name: 'monthly_spending',
+ orderBy: 'total',
+ order: 'desc',
+ limit: 10
+ }
+})
+
+// Combine find-level params (where, orderBy, limit, offset merge automatically)
+const recentFood = await brain.find({
+ aggregate: 'monthly_spending',
+ where: { category: 'food' },
+ orderBy: 'total',
+ order: 'desc',
+ limit: 12
+})
+```
+
+**Result format:** Returns `Result[]` with `type: NounType.Measurement`. Each result contains:
+
+```typescript
+{
+ id: string, // Aggregate group ID (or materialized entity ID)
+ score: 1.0, // Always 1.0 for aggregates
+ type: NounType.Measurement,
+ metadata: {
+ __aggregate: 'monthly_spending', // Source aggregate name
+ category: 'food', // Group key values
+ date: '2024-01', // Time window bucket
+ total: 342.50, // Computed metrics
+ count: 28,
+ average: 12.23,
+ highest: 45.00,
+ lowest: 2.50
+ },
+ entity: Entity // Full entity structure
+}
+```
+
+### `queryAggregate(name, params?)` → `Promise`
+
+The first-class analytics path — returns plain group rows (`{ groupKey, metrics, count }`) instead of `find()`-style `Result` wrappers. Supports `where` (group-key filter), `having` (SQL-HAVING metric filter), `orderBy`, `order`, `limit`, `offset`:
+
+```typescript
+const rows = await brain.queryAggregate('monthly_spending', {
+ having: { total: { greaterThan: 100 } }, // filter by computed metrics
+ orderBy: 'total',
+ order: 'desc',
+ limit: 10
+})
+// [{ groupKey: { category: 'food', date: '2024-01' }, metrics: { total: 342.5, count: 28 }, count: 28 }, ...]
+```
+
+### How It Works
+
+Aggregation hooks run **outside transactions** on every write operation:
+
+- **`add()`**: If the new entity matches any aggregate's `source` filter, its values are added to the matching group's running totals.
+- **`update()`**: The old entity's contribution is reversed and the new entity's contribution is applied (handles group key changes, source filter changes).
+- **`delete()`**: The deleted entity's contribution is reversed from its group.
+
+**Performance:** O(A × G × M) per write where A = matching aggregates, G = groupBy dimensions, M = metrics. For typical configurations (2-5 aggregates, 1-3 dimensions, 3-5 metrics), this is effectively O(1) — measured at **10,000 entities in 13ms** in unit tests.
+
+**Infinite loop prevention:** Materialized `NounType.Measurement` entities (with `service: 'brainy:aggregation'` or `metadata.__aggregate`) are automatically excluded from all aggregate source matching.
+
+**Persistence:** Definitions and running state are persisted to storage on `flush()`/`close()` and reloaded on `init()`. Definition changes are detected via FNV-1a hashing — only changed aggregates reset their state.
+
+**Native acceleration:** Register an `'aggregation'` provider via the plugin system to replace the TypeScript engine with a custom native implementation for higher throughput at scale.
+
+### Financial Data Modeling
+
+Brainy supports financial analytics through **subtypes and metadata conventions** on existing NounTypes — no custom types needed:
+
+```typescript
+// Transaction = NounType.Event + 'transaction' subtype + financial metadata
+await brain.add({
+ data: 'Coffee at Blue Bottle',
+ type: NounType.Event,
+ subtype: 'transaction', // top-level standard field (reserved — never in metadata)
+ metadata: {
+ domain: 'financial',
+ amount: 5.50,
+ currency: 'USD',
+ category: 'food',
+ date: Date.now(),
+ merchant: 'Blue Bottle Coffee'
+ }
+})
+
+// Account = NounType.Collection + 'account' subtype + financial metadata
+await brain.add({
+ data: 'Checking Account',
+ type: NounType.Collection,
+ subtype: 'account',
+ metadata: {
+ domain: 'financial',
+ accountType: 'checking',
+ currency: 'USD',
+ institution: 'Chase'
+ }
+})
+
+// Invoice = NounType.Document + 'invoice' subtype + financial metadata
+await brain.add({
+ data: 'Invoice #1234 from Acme Corp',
+ type: NounType.Document,
+ subtype: 'invoice',
+ metadata: {
+ domain: 'financial',
+ amount: 15000,
+ currency: 'USD',
+ status: 'pending',
+ dueDate: Date.UTC(2024, 2, 15),
+ vendor: 'Acme Corp'
+ }
+})
+```
---
@@ -236,50 +628,107 @@ Create a typed relationship between entities.
```typescript
const relId = await brain.relate({
- from: sourceId,
- to: targetId,
- type: VerbType.RelatedTo,
- metadata: { // Optional
- strength: 0.9,
- confidence: 0.85
- }
+ from: sourceId,
+ to: targetId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct', // Optional: sub-classification
+ data: 'Collaborated on the research paper', // Optional: content for this edge
+ metadata: { // Optional: structured edge fields
+ strength: 0.9,
+ role: 'primary author'
+ }
})
```
**Parameters:**
-- `from`: `string` - Source entity ID
-- `to`: `string` - Target entity ID
+- `from`: `string` - Source entity ID (must exist)
+- `to`: `string` - Target entity ID (must exist)
- `type`: `VerbType` - Relationship type
-- `metadata?`: `object` - Optional metadata
+- `subtype?`: `string` - Per-product sub-classification within the VerbType (top-level standard field, fast-path indexed). See [Subtypes & Facets](../guides/subtypes-and-facets.md).
+- `data?`: `any` - Content for the relationship (overrides auto-computed vector)
+- `metadata?`: `object` - Structured edge fields
+- `weight?`: `number` - Connection strength (0-1, default: 1.0)
+- `bidirectional?`: `boolean` - Create reverse edge too (default: false)
+- `confidence?`: `number` - Relationship certainty (0-1)
+
+> **Strict-mode tip:** same as `add()` — if a vocabulary is registered for your `type`, pass a matching `subtype`. Run `await brain.audit()` first to surface pre-existing gaps.
**Returns:** `Promise` - Relationship ID
---
-### `getRelations(params)` → `Promise`
+### `updateRelation(params)` → `Promise`
-Get relationships for an entity.
+Update an existing relationship. Mirror of `update()` for verbs — closed a long-standing gap (verbs had no update path before 7.30).
+
+```typescript
+// Change the subtype on an existing relationship
+await brain.updateRelation({ id: relId, subtype: 'dotted-line' })
+
+// Update weight + confidence
+await brain.updateRelation({ id: relId, weight: 0.7, confidence: 0.9 })
+
+// Change verb type (re-indexes in graph adjacency, id preserved)
+await brain.updateRelation({ id: relId, type: VerbType.WorksWith })
+```
+
+**Parameters:**
+- `id`: `string` - Relationship ID (required)
+- `type?`: `VerbType` - Change verb type (re-indexes in graph adjacency)
+- `subtype?`: `string` - Change sub-classification (omit to preserve existing)
+- `weight?`: `number` - New weight (0-1)
+- `confidence?`: `number` - New confidence (0-1)
+- `data?`: `any` - New content
+- `metadata?`: `object` - Metadata to merge (or replace with `merge: false`)
+- `merge?`: `boolean` - Merge or replace metadata (default: true)
+
+**Returns:** `Promise`
+
+---
+
+### `related(params)` → `Promise`
+
+Get relationships for an entity. Same name and surface as `db.related()` on a
+pinned `Db` view.
```typescript
// Get all relationships FROM an entity
-const outgoing = await brain.getRelations({ from: entityId })
+const outgoing = await brain.related({ from: entityId })
// Get all relationships TO an entity
-const incoming = await brain.getRelations({ to: entityId })
+const incoming = await brain.related({ to: entityId })
// Filter by type
-const related = await brain.getRelations({
- from: entityId,
- type: VerbType.Contains
+const contains = await brain.related({
+ from: entityId,
+ type: VerbType.Contains
+})
+
+// Filter by subtype (fast path, column-store hit)
+const direct = await brain.related({
+ from: entityId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct'
+})
+
+// Set membership on subtype
+const all = await brain.related({
+ from: entityId,
+ type: VerbType.ReportsTo,
+ subtype: ['direct', 'dotted-line']
})
```
**Parameters:**
- `from?`: `string` - Source entity ID
- `to?`: `string` - Target entity ID
-- `type?`: `VerbType` - Filter by relationship type
+- `type?`: `VerbType | VerbType[]` - Filter by relationship type
+- `subtype?`: `string | string[]` - Filter by VerbType subtype (top-level standard field, fast path)
+- `service?`: `string` - Multi-tenancy filter
+- `limit?`: `number` - Pagination limit (default: 100)
+- `offset?`: `number` - Pagination offset
-**Returns:** `Promise` - Matching relationships
+**Returns:** `Promise` - Matching relationships (each with `subtype` at top level when set)
---
@@ -291,27 +740,27 @@ Add multiple entities in one operation.
```typescript
const result = await brain.addMany({
- items: [
- { data: 'Entity 1', type: NounType.Document },
- { data: 'Entity 2', type: NounType.Concept }
- ]
+ items: [
+ { data: 'Entity 1', type: NounType.Document },
+ { data: 'Entity 2', type: NounType.Concept }
+ ]
})
-console.log(result.successful) // Array of IDs
-console.log(result.failed) // Array of errors
+console.log(result.successful) // Array of IDs
+console.log(result.failed) // Array of errors
```
**Returns:** `Promise>` - Success/failure results
---
-### `deleteMany(params)` → `Promise>`
+### `removeMany(params)` → `Promise>`
-Delete multiple entities.
+Remove multiple entities.
```typescript
-const result = await brain.deleteMany({
- ids: [id1, id2, id3]
+const result = await brain.removeMany({
+ ids: [id1, id2, id3]
})
```
@@ -323,10 +772,10 @@ Update multiple entities.
```typescript
const result = await brain.updateMany({
- updates: [
- { id: id1, metadata: { updated: true } },
- { id: id2, data: 'New content' }
- ]
+ items: [
+ { id: id1, metadata: { updated: true } },
+ { id: id2, data: 'New content' }
+ ]
})
```
@@ -338,617 +787,269 @@ Create multiple relationships.
```typescript
const ids = await brain.relateMany({
- relations: [
- { from: id1, to: id2, type: VerbType.RelatedTo },
- { from: id1, to: id3, type: VerbType.Contains }
- ]
+ items: [
+ { from: id1, to: id2, type: VerbType.RelatedTo },
+ { from: id1, to: id3, type: VerbType.Contains }
+ ]
})
```
---
-## Branch Management (v5.0+)
+## Database Values & Time Travel (Db API)
-**NEW in v5.0.0:** Git-style branching with Snowflake-style copy-on-write.
+Brainy 8.0's generational MVCC exposes the whole store as an immutable
+value: the **`Db`**. Pin the current state in O(1), commit atomic
+multi-write batches, query any past generation with the full query surface,
+cut instant snapshots, and ask what-if questions in memory. The exact
+guarantees live in the **[consistency model](../concepts/consistency-model.md)**;
+recipes live in **[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md)**.
-### `fork(branch?, options?)` → `Promise`
+### `generation()` → `number`
-Create an instant fork (<100ms) with full isolation.
+The store's current generation — a monotonic watermark advanced once per
+committed `transact()` batch and once per single-operation write. Never
+reissued, including across restarts and `restore()`.
```typescript
-// Create a fork
-const experiment = await brain.fork('test-feature')
-
-// Make changes safely in isolation
-await experiment.add({ data: 'Test entity', type: NounType.Document })
-await experiment.update({ id: someId, metadata: { modified: true } })
-
-// Parent is unaffected!
-const parentData = await brain.find({}) // Original data unchanged
-```
-
-**Parameters:**
-- `branch?`: `string` - Branch name (auto-generated if omitted)
-- `options?`: `object`
- - `description?`: `string` - Branch description
-
-**Returns:** `Promise` - New Brainy instance on forked branch
-
-**How it works:** Snowflake-style COW shares HNSW index, copies only modified nodes (10-20% memory overhead).
-
----
-
-### `checkout(branch)` → `Promise`
-
-Switch to a different branch.
-
-```typescript
-await brain.checkout('main')
-await brain.checkout('test-feature')
-```
-
-**Parameters:**
-- `branch`: `string` - Branch name
-
----
-
-### `listBranches()` → `Promise`
-
-List all branches.
-
-```typescript
-const branches = await brain.listBranches()
-// ['main', 'test-feature', 'experiment-2']
+const g = brain.generation()
```
---
-### `getCurrentBranch()` → `Promise`
+### `now()` → `Db`
-Get current branch name.
+Pin the current generation and return an immutable view — O(1), no I/O.
+The view keeps reading exactly this state no matter what commits afterwards.
```typescript
-const current = await brain.getCurrentBranch()
-// 'main'
+const db = brain.now()
+await brain.update({ id, metadata: { v: 2 } })
+
+await db.get(id) // still sees v: 1 — pinned
+await brain.get(id) // sees v: 2 — live
+await db.release() // unpin (enables history compaction)
+```
+
+**Returns:** `Db` — release it when done; pins gate `compactHistory()`.
+
+---
+
+### `transact(ops, options?)` → `Promise`
+
+Execute a declarative operation batch **atomically**: either every
+operation applies and the store advances exactly one generation, or none
+apply and the store is byte-identical to its pre-transaction state. The
+commit point is an atomic manifest rename; a crash anywhere before it rolls
+back to the exact pre-transaction bytes on the next open.
+
+```typescript
+const db = await brain.transact([
+ { op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' },
+ { op: 'update', id: customerId, metadata: { lastOrderAt: Date.now() }, ifRev: customer._rev },
+ { op: 'relate', from: customerId, to: orderId, type: VerbType.Creates, subtype: 'purchase' },
+ { op: 'remove', id: staleDraftId },
+ { op: 'unrelate', id: oldRelationId }
+], {
+ meta: { author: 'order-service', requestId: 'req-9f2' }, // reified, durable
+ ifAtGeneration: expectedGeneration // whole-store CAS
+})
+
+db.receipt.ids // resolved id per operation, in input order
+db.receipt.generation // the committed generation
+```
+
+**Operations** (`op` discriminates; parameters mirror the single-operation methods):
+- `{ op: 'add', ... }` — same parameters as `add()`; optional explicit `id`
+- `{ op: 'update', ... }` — same parameters as `update()`, including per-entity `ifRev` CAS
+- `{ op: 'remove', id }` — deletes the entity plus its relationships (same cascade as `delete()`)
+- `{ op: 'relate', ... }` — same parameters as `relate()`, including `bidirectional`; duplicates dedupe to the existing relationship id
+- `{ op: 'unrelate', id }` — deletes a relationship by id
+
+Operations may reference ids created earlier in the same batch.
+
+**Options:**
+- `meta?`: `Record` — transaction metadata, recorded durably in the transaction log (audit fields: author, reason, request id)
+- `ifAtGeneration?`: `number` — whole-store compare-and-swap; commits only if the store is still at this generation
+
+**Returns:** `Promise` — pinned at the freshly committed generation, carrying a `receipt`.
+
+**Throws:**
+- `GenerationConflictError` — `ifAtGeneration` did not match (nothing staged, generation unchanged)
+- `RevisionConflictError` — an `ifRev` did not match (whole batch rejected)
+
+---
+
+### `asOf(target)` → `Promise`
+
+Open an immutable view of **past** state:
+
+```typescript
+const atGen = await brain.asOf(1041) // generation number
+const lastWeek = await brain.asOf(new Date(Date.now() - 7 * 86_400_000)) // wall-clock
+const fromSnapshot = await brain.asOf('/backups/2026-06-01') // snapshot directory
+```
+
+- **`number`** — pins that generation; reads resolve through the immutable record layer.
+- **`Date`** — resolved via the transaction log to the newest generation committed at or before it.
+- **`string`** — a snapshot directory from `db.persist()`, opened as a self-contained read-only store (equivalent to `Brainy.load()`).
+
+Historical views serve the **full query surface**. Metadata-level reads are
+free; the first index-accelerated query (semantic search, traversal,
+cursors, aggregation) builds an in-memory index materialization — O(n at
+that generation), once per `Db`, freed on `release()`.
+
+**Throws:** `GenerationCompactedError` when the generation's records were reclaimed by `compactHistory()`.
+
+**History granularity:** every write is its own immutable generation —
+`transact()` batches AND single-operation writes — so a pin always freezes and
+every write is addressable via `asOf()`. See the
+[consistency model](../concepts/consistency-model.md).
+
+---
+
+### `transactionLog(options?)` → `Promise`
+
+Read the reified transaction log — one entry per committed generation (every
+`transact()` AND single-op write), newest first: `{ generation, timestamp,
+meta? }`. Single-op generations carry no `meta` (it is a `transact()`-only field).
+
+```typescript
+const [latest] = await brain.transactionLog({ limit: 1 })
+latest.meta // { author: 'order-service', requestId: 'req-9f2' }
```
---
-### `commit(options?)` → `Promise`
+### `compactHistory(options?)` → `Promise`
-Create a commit snapshot.
+Reclaim historical record-sets that no retention cap and no live `Db` pin
+protects. Pinned reads stay correct across compaction, always. (Auto-compaction
+on `flush()`/`close()` is governed by the constructor `retention` knob — unset →
+adaptive, `'all'` → unbounded, `{ … }` → explicit caps.)
```typescript
-const commitId = await brain.commit({
- message: 'Add new features',
- author: 'dev@example.com',
- metadata: { ticket: 'PROJ-123' }
+await brain.compactHistory({
+ maxGenerations: 100, // keep at most the 100 most recent generations
+ maxAge: 7 * 24 * 60 * 60 * 1000 // and only those from the last 7 days
})
```
-**Parameters:**
-- `message?`: `string` - Commit message
-- `author?`: `string` - Author email
-- `metadata?`: `object` - Additional commit metadata
-
-**Returns:** `Promise` - Commit ID
+**Returns:** `{ removedGenerations, horizon }` — `asOf()` below the horizon throws `GenerationCompactedError`.
---
+### `restore(path, { confirm: true })` → `Promise`
-### `deleteBranch(branch)` → `Promise`
-
-Delete a branch (cannot delete 'main').
+Replace the store's **entire** state from a snapshot directory. Destructive
+— requires `{ confirm: true }`. All indexes are rebuilt; the generation
+counter is floored so observed generation numbers are never reissued; live
+pins do not survive.
```typescript
-await brain.deleteBranch('old-experiment')
+await brain.restore('/backups/2026-06-01', { confirm: true })
```
---
-### `getHistory(options?)` → `Promise`
+### `Brainy.load(path)` → `Promise` (static)
-Get commit history.
+Open a persisted snapshot as a self-contained **read-only** store with the
+full query surface, including vector search. Releasing the returned `Db`
+closes the underlying instance.
```typescript
-const history = await brain.getHistory({
- branch: 'main',
- limit: 10
-})
+const db = await Brainy.load('/backups/2026-06-01')
+const hits = await db.find({ query: 'quarterly invoices' })
+await db.release()
```
---
-### `asOf(commitId, options?)` → `Promise`
+### The `Db` value
-Create a read-only snapshot at a specific commit for time-travel queries.
+Every `Db` is pinned at one generation and serves the full query surface at
+exactly that state.
+
+**Properties:**
+
+| Property | Type | Meaning |
+|---|---|---|
+| `generation` | `number` | The pinned generation |
+| `timestamp` | `number` | Pin time (`now()`), commit time (`transact()`), or resolved commit time (`asOf()`) |
+| `receipt` | `TransactReceipt?` | Present only on `transact()` results |
+| `speculative` | `boolean` | Whether this view carries a `with()` overlay |
+| `released` | `boolean` | Whether `release()` has been called |
+
+**Methods:**
```typescript
-// Get commit ID from history
-const commits = await brain.getHistory({ limit: 1 })
-const commitId = commits[0].id
-
-// Create snapshot (lazy-loading, no eager data loading)
-const snapshot = await brain.asOf(commitId, {
- cacheSize: 10000 // LRU cache size (default: 10000)
-})
-
-// Query historical state - full Triple Intelligence works!
-const results = await snapshot.find({
- query: 'AI research',
- where: { category: 'technology' }
-})
-
-// Get historical relationships
-const related = await snapshot.getRelated(entityId, { depth: 2 })
-
-// MUST close when done to free memory
-await snapshot.close()
+await db.get(id) // entity as of this generation
+await db.find({ where: { status: 'open' } }) // full find() surface
+await db.find({ query: 'unpaid invoices' }) // semantic search as of this generation
+await db.related(entityId) // relationships as of this generation
+await db.since(olderDb) // ids changed between two views
+const whatIf = await db.with(ops) // speculative in-memory overlay
+await db.persist('/backups/today') // self-contained hard-link snapshot
+await db.release() // unpin + free cached materialization
```
-**Parameters:**
-- `commitId`: `string` - Commit hash to snapshot from
-- `options?`: `object`
- - `cacheSize?`: `number` - LRU cache size for lazy-loading (default: 10000)
-
-**Returns:** `Promise` - Read-only Brainy instance with historical state
-
-**Features:**
-- **Lazy-Loading** - Loads entities on-demand, not eagerly
-- **Bounded Memory** - LRU cache prevents memory bloat
-- **Full Query Support** - All find(), getRelated(), etc. work on historical data
-- **Read-Only** - Prevents accidental modifications to history
-
-**Important:** Always call `snapshot.close()` when done to release resources.
-
----
-
-## Entity Versioning (v5.3.0+)
-
-**NEW in v5.3.0:** Git-style versioning for individual entities with content-addressable storage.
-
-### Overview
-
-Entity Versioning provides time-travel and history tracking for individual entities:
-
-- **Content-Addressable Storage** - Deduplication via SHA-256 hashing
-- **Zero-Config** - Lazy initialization, uses existing indexes
-- **Branch-Isolated** - Versions isolated per branch
-- **Selective Auto-Versioning** - Optional augmentation for automatic version creation
-- **Production-Scale** - Designed for billions of entities
-- **VFS File Support (v6.3.2+)** - Full versioning for VFS files with actual blob content
-
----
-
-### `versions.save(entityId, options?)` → `Promise`
-
-Save a new version of an entity.
-
-```typescript
-// Save version with tag
-const version = await brain.versions.save('user-123', {
- tag: 'v1.0',
- description: 'Initial user profile',
- metadata: { author: 'dev@example.com' }
-})
-
-console.log(version.version) // 1
-console.log(version.contentHash) // SHA-256 hash
-console.log(version.createdAt) // Timestamp
-```
-
-**Parameters:**
-- `entityId`: `string` - Entity ID to version
-- `options?`: `object`
- - `tag?`: `string` - Version tag (e.g., 'v1.0', 'beta')
- - `description?`: `string` - Version description
- - `metadata?`: `object` - Additional version metadata
-
-**Returns:** `Promise` - Created version
-
-**Features:**
-- Automatic deduplication (identical content = same version)
-- Sequential version numbering (1, 2, 3, ...)
-- Content-addressable storage (SHA-256)
-
----
-
-### `versions.list(entityId, options?)` → `Promise`
-
-List all versions of an entity.
-
-```typescript
-const versions = await brain.versions.list('user-123', {
- limit: 10,
- offset: 0
-})
-
-versions.forEach(v => {
- console.log(`Version ${v.version}: ${v.tag} - ${v.description}`)
-})
-```
-
-**Parameters:**
-- `entityId`: `string` - Entity ID
-- `options?`: `object`
- - `limit?`: `number` - Max versions to return
- - `offset?`: `number` - Skip versions
-
-**Returns:** `Promise` - Versions (newest first)
-
----
-
-### `versions.restore(entityId, versionOrTag)` → `Promise`
-
-Restore entity to a previous version.
-
-```typescript
-// Restore by version number
-await brain.versions.restore('user-123', 1)
-
-// Restore by tag
-await brain.versions.restore('user-123', 'beta')
-```
-
-**Parameters:**
-- `entityId`: `string` - Entity ID
-- `versionOrTag`: `number | string` - Version number or tag
-
----
-
-### `versions.compare(entityId, version1, version2)` → `Promise`
-
-Compare two versions.
-
-```typescript
-const diff = await brain.versions.compare('user-123', 1, 2)
-
-console.log(diff.totalChanges) // Total changes
-console.log(diff.modified) // Modified fields
-console.log(diff.added) // Added fields
-console.log(diff.removed) // Removed fields
-
-// Check specific changes
-const nameChange = diff.modified.find(c => c.path === 'metadata.name')
-console.log(`${nameChange.oldValue} → ${nameChange.newValue}`)
-```
-
-**Returns:** `Promise` - Detailed diff with field-level changes
-
----
-
-### `versions.getContent(entityId, versionOrTag)` → `Promise`
-
-Get version content without restoring.
-
-```typescript
-// View old version without changing current state
-const v1Content = await brain.versions.getContent('user-123', 1)
-console.log(v1Content.metadata.name) // Old name
-
-// Current state unchanged
-const current = await brain.get('user-123')
-console.log(current.metadata.name) // Current name
-```
-
----
-
-### `versions.undo(entityId)` → `Promise`
-
-Undo to previous version (shorthand for restore to latest-1).
-
-```typescript
-// Make a bad change
-await brain.update('user-123', { status: 'deleted' })
-
-// Undo immediately
-await brain.versions.undo('user-123')
-```
-
-**Alias:** `versions.revert(entityId)`
-
----
-
-### `versions.prune(entityId, options)` → `Promise`
-
-Clean up old versions.
-
-```typescript
-const result = await brain.versions.prune('user-123', {
- keepRecent: 10, // Keep 10 most recent
- keepTagged: true, // Always keep tagged versions
- olderThan: Date.now() - 30 * 24 * 60 * 60 * 1000 // Older than 30 days
-})
-
-console.log(`Deleted ${result.deleted}, kept ${result.kept}`)
-```
-
-**Parameters:**
-- `keepRecent?`: `number` - Keep N most recent versions
-- `keepTagged?`: `boolean` - Always keep tagged versions (default: true)
-- `olderThan?`: `number` - Only prune versions older than timestamp
-
----
-
-### `versions.getLatest(entityId)` → `Promise`
-
-Get latest version.
-
-```typescript
-const latest = await brain.versions.getLatest('user-123')
-if (latest) {
- console.log(`Latest: v${latest.version} (${latest.tag})`)
-}
-```
-
----
-
-### `versions.getVersionByTag(entityId, tag)` → `Promise`
-
-Get version by tag.
-
-```typescript
-const beta = await brain.versions.getVersionByTag('user-123', 'beta')
-```
-
----
-
-### `versions.count(entityId)` → `Promise`
-
-Count versions for an entity.
-
-```typescript
-const count = await brain.versions.count('user-123')
-console.log(`${count} versions saved`)
-```
-
----
-
-### `versions.hasVersions(entityId)` → `Promise`
-
-Check if entity has versions.
-
-```typescript
-if (await brain.versions.hasVersions('user-123')) {
- console.log('Entity has version history')
-}
-```
-
----
-
-### Auto-Versioning Augmentation
-
-Automatically create versions on entity updates.
-
-```typescript
-import { VersioningAugmentation } from '@soulcraft/brainy'
-
-// Configure auto-versioning
-const versioning = new VersioningAugmentation({
- enabled: true,
- onUpdate: true, // Version on update()
- onDelete: false, // Don't version on delete
- entities: ['user-*'], // Only version users
- excludeEntities: ['temp-*'],
- excludeTypes: ['temporary'],
- keepRecent: 50, // Auto-prune old versions
- keepTagged: true
-})
-
-// Apply augmentation
-brain.augment(versioning)
-
-// Now updates auto-create versions
-await brain.update('user-123', { name: 'New Name' })
-
-// Version automatically created!
-const versions = await brain.versions.list('user-123')
-console.log(`Auto-created version: ${versions[0].version}`)
-```
-
-**Configuration:**
-- `enabled`: `boolean` - Enable/disable augmentation
-- `onUpdate`: `boolean` - Version on entity updates
-- `onDelete`: `boolean` - Version before deletion
-- `entities`: `string[]` - Entity ID patterns (glob-style)
-- `excludeEntities`: `string[]` - Exclusion patterns
-- `types`: `string[]` - Entity types to version
-- `excludeTypes`: `string[]` - Types to exclude
-- `keepRecent`: `number` - Auto-prune to keep N versions
-- `keepTagged`: `boolean` - Always keep tagged versions
-
-**Pattern Matching:**
-- `['*']` - All entities
-- `['user-*']` - All IDs starting with "user-"
-- `['*-prod']` - All IDs ending with "-prod"
-- `['user-*', 'account-*']` - Multiple patterns
-
----
-
-### Branch Isolation
-
-Versions are isolated per branch.
-
-```typescript
-// Save version on main
-await brain.versions.save('doc-1', { tag: 'main-v1' })
-
-// Fork and create version
-const feature = await brain.fork('feature')
-await feature.update('doc-1', { content: 'Feature update' })
-await feature.versions.save('doc-1', { tag: 'feature-v1' })
-
-// Versions are isolated
-const mainVersions = await brain.versions.list('doc-1')
-const featureVersions = await feature.versions.list('doc-1')
-
-console.log(mainVersions.length !== featureVersions.length) // true
-```
-
----
-
-### Architecture
-
-**Content-Addressable Storage:**
-- SHA-256 hashing for deduplication
-- Identical content = single storage blob
-- Efficient for entities with few changes
-
-**Metadata Indexing:**
-- Leverages existing MetadataIndexManager
-- Fast lookups by entity ID
-- Version number indexing
-
-**Storage Structure:**
-```
-_version:{entityId}:{versionNum}:{branch} // Version metadata
-_version_blob:{contentHash} // Content blob (deduplicated)
-```
-
-**Performance:**
-- Version save: O(1) if duplicate, O(log N) for index update
-- Version list: O(K) where K = version count
-- Version restore: O(log N) lookup + O(1) restore
-- Pruning: O(K) where K = versions pruned
-
----
-
-### Examples
-
-#### Basic Versioning Workflow
-
-```typescript
-// Create entity
-await brain.add({
- data: 'User profile',
- id: 'user-123',
- type: 'user',
- metadata: { name: 'Alice', email: 'alice@example.com' }
-})
-
-// Save v1
-await brain.versions.save('user-123', { tag: 'v1.0' })
-
-// Make changes
-await brain.update('user-123', { name: 'Alice Smith' })
-
-// Save v2
-await brain.versions.save('user-123', { tag: 'v2.0' })
-
-// Compare versions
-const diff = await brain.versions.compare('user-123', 1, 2)
-
-// Restore to v1 if needed
-await brain.versions.restore('user-123', 'v1.0')
-```
-
-#### Release Management
-
-```typescript
-// Development workflow
-await brain.update('app-config', { version: '1.0.0-alpha' })
-await brain.versions.save('app-config', { tag: 'alpha' })
-
-await brain.update('app-config', { version: '1.0.0-beta' })
-await brain.versions.save('app-config', { tag: 'beta' })
-
-await brain.update('app-config', { version: '1.0.0' })
-await brain.versions.save('app-config', { tag: 'release' })
-
-// Rollback to beta if issues found
-await brain.versions.restore('app-config', 'beta')
-```
-
-#### Audit Trail
-
-```typescript
-// Track all changes
-const versioning = new VersioningAugmentation({
- enabled: true,
- onUpdate: true,
- entities: ['audit-*'],
- keepRecent: 100 // Keep 100 versions for audit
-})
-
-brain.augment(versioning)
-
-// All updates now tracked
-await brain.update('audit-record-1', { status: 'modified' })
-await brain.update('audit-record-1', { status: 'approved' })
-
-// View complete history
-const versions = await brain.versions.list('audit-record-1')
-versions.forEach(v => {
- console.log(`${v.createdAt}: ${v.description}`)
-})
-```
-
-#### VFS File Versioning (v6.3.2+)
-
-```typescript
-// VFS files can be versioned with actual blob content
-await brain.vfs.writeFile('/docs/readme.md', 'Version 1 content')
-
-// Get the file's entity ID
-const stat = await brain.vfs.stat('/docs/readme.md')
-
-// Save version 1
-await brain.versions.save(stat.entityId, { tag: 'v1', description: 'Initial draft' })
-
-// Modify the file
-await brain.vfs.writeFile('/docs/readme.md', 'Version 2 - updated content')
-
-// Save version 2
-await brain.versions.save(stat.entityId, { tag: 'v2', description: 'Updated docs' })
-
-// Compare versions - content is DIFFERENT (fixed in v6.3.2)
-const v1 = await brain.versions.getContent(stat.entityId, 1)
-const v2 = await brain.versions.getContent(stat.entityId, 2)
-console.log(v1.data !== v2.data) // true
-
-// Restore to v1 - writes content back to blob storage
-await brain.versions.restore(stat.entityId, 'v1')
-
-// File is now back to v1
-const content = await brain.vfs.readFile('/docs/readme.md')
-console.log(content.toString()) // 'Version 1 content'
-```
-
----
-
-**[📖 Complete Versioning Guide →](../features/entity-versioning.md)**
+- **`with(ops)`** — applies `transact()`-style operations **in memory** on
+ top of the view; nothing touches disk, the generation counter, or index
+ providers. Overlay entities carry no embeddings, so index-accelerated
+ queries and `persist()` on overlays throw `SpeculativeOverlayError`;
+ `get()`, metadata-filter `find()`, and filter-based `related()` work
+ fully. Commit the same ops with `transact()` for the full surface.
+- **`persist(path)`** — cuts an instant snapshot (hard links on filesystem
+ storage; byte copies across devices; in-memory stores serialize to the
+ same layout). Requires the view to still be the store's latest generation
+ — otherwise `GenerationConflictError`.
+- **`release()`** — idempotent; after release every read throws. A
+ `FinalizationRegistry` backstop releases leaked pins at GC, but explicit
+ release is what makes `compactHistory()` deterministic.
+
+### Db API errors
+
+All exported from `@soulcraft/brainy`:
+
+| Error | Thrown by | Meaning |
+|---|---|---|
+| `GenerationConflictError` | `transact({ ifAtGeneration })`, `db.persist()` | The store moved past the expected generation — re-read and retry |
+| `RevisionConflictError` | `update({ ifRev })`, `transact()` update ops | Per-entity revision moved — see [optimistic concurrency](../guides/optimistic-concurrency.md) |
+| `GenerationCompactedError` | `asOf()` | The requested generation's records were reclaimed — persist what you must keep |
+| `SpeculativeOverlayError` | index-accelerated reads / `persist()` on `with()` overlays | Honest boundary: overlay entities carry no embeddings |
---
## Virtual Filesystem (VFS)
-**Auto-initialized in v5.1.0!** Access via `brain.vfs` (property, not method).
+Access via `brain.vfs` (property, not method). Auto-initialized during `brain.init()`.
### Filtering VFS Entities
-**NEW in v5.3.0:** All VFS entities (files/folders) have `metadata.isVFSEntity: true` set automatically.
+All VFS entities (files/folders) have `metadata.isVFSEntity: true` set automatically.
Use this to filter VFS entities from semantic search results:
```typescript
// Exclude VFS entities from semantic search
const semanticOnly = await brain.find({
- query: 'artificial intelligence',
- where: {
- isVFSEntity: { notEquals: true } // Only semantic entities
- }
+ query: 'artificial intelligence',
+ where: {
+ isVFSEntity: { notEquals: true } // Only semantic entities
+ }
})
// Or filter to ONLY VFS entities
const vfsOnly = await brain.find({
- where: {
- isVFSEntity: { equals: true } // Only VFS files/folders
- }
+ where: {
+ isVFSEntity: { equals: true } // Only VFS files/folders
+ }
})
// Check if an entity is a VFS entity
if (entity.metadata.isVFSEntity === true) {
- console.log('This is a VFS file or folder')
+ console.log('This is a VFS file or folder')
}
```
@@ -975,7 +1076,7 @@ Write file content.
```typescript
await brain.vfs.writeFile('/docs/README.md', 'New content', {
- encoding: 'utf-8'
+ encoding: 'utf-8'
})
```
@@ -1013,7 +1114,7 @@ const files = await brain.vfs.readdir('/projects')
// With file types
const entries = await brain.vfs.readdir('/projects', { withFileTypes: true })
entries.forEach(entry => {
- console.log(entry.name, entry.isDirectory() ? 'DIR' : 'FILE')
+ console.log(entry.name, entry.isDirectory() ? 'DIR' : 'FILE')
})
```
@@ -1035,9 +1136,9 @@ Get file/directory stats.
```typescript
const stats = await brain.vfs.stat('/docs/README.md')
-console.log(stats.size) // File size
-console.log(stats.mtime) // Modified time
-console.log(stats.isDirectory()) // Is directory?
+console.log(stats.size) // File size
+console.log(stats.mtime) // Modified time
+console.log(stats.isDirectory()) // Is directory?
```
---
@@ -1050,8 +1151,8 @@ Semantic file search.
```typescript
const results = await brain.vfs.search('React components with hooks', {
- path: '/src',
- limit: 10
+ path: '/src',
+ limit: 10
})
```
@@ -1063,8 +1164,8 @@ Find similar files.
```typescript
const similar = await brain.vfs.findSimilar('/src/App.tsx', {
- limit: 5,
- threshold: 0.7
+ limit: 5,
+ threshold: 0.7
})
```
@@ -1078,7 +1179,7 @@ Get directory tree (prevents infinite recursion).
```typescript
const tree = await brain.vfs.getTreeStructure('/projects', {
- maxDepth: 3
+ maxDepth: 3
})
```
@@ -1090,7 +1191,7 @@ Get all descendants with optional filtering.
```typescript
const files = await brain.vfs.getDescendants('/src', {
- filter: (entity) => entity.name.endsWith('.tsx')
+ filter: (entity) => entity.name.endsWith('.tsx')
})
```
@@ -1104,8 +1205,8 @@ Get file metadata.
```typescript
const meta = await brain.vfs.getMetadata('/src/App.tsx')
-console.log(meta.todos) // Extracted TODOs
-console.log(meta.tags) // Tags
+console.log(meta.todos) // Extracted TODOs
+console.log(meta.tags) // Tags
```
---
@@ -1131,39 +1232,16 @@ const todos = await brain.vfs.getTodos('/src/App.tsx')
---
-#### `vfs.getAllTodos(path?)` → `Promise`
+#### `vfs.searchEntities(query)` → `Promise>`
-Get all TODOs from directory tree.
+Search for semantic entities tracked by the VFS, filtered by type, name, or metadata.
```typescript
-const allTodos = await brain.vfs.getAllTodos('/src')
-```
-
----
-
-### Project Analysis
-
-#### `vfs.getProjectStats(path?)` → `Promise`
-
-Get project statistics.
-
-```typescript
-const stats = await brain.vfs.getProjectStats('/projects/my-app')
-console.log(stats.fileCount)
-console.log(stats.totalSize)
-console.log(stats.fileTypes) // Breakdown by extension
-```
-
----
-
-#### `vfs.searchEntities(query)` → `Promise`
-
-Search for VFS entities by metadata.
-
-```typescript
-const tsxFiles = await brain.vfs.searchEntities({
- type: 'file',
- extension: '.tsx'
+const people = await brain.vfs.searchEntities({
+ type: 'person', // entity type filter
+ name: 'Ada', // semantic name search
+ where: { role: 'author' }, // metadata filters
+ limit: 50
})
```
@@ -1173,119 +1251,6 @@ const tsxFiles = await brain.vfs.searchEntities({
---
-## Neural API
-
-Access advanced AI features via `brain.neural()` (method that returns NeuralAPI instance).
-
-### `neural().similar(a, b, options?)` → `Promise`
-
-Calculate semantic similarity.
-
-```typescript
-// Simple similarity score
-const score = await brain.neural().similar(
- 'renewable energy',
- 'sustainable power'
-) // 0.87
-
-// Detailed result
-const result = await brain.neural().similar('text1', 'text2', {
- detailed: true
-})
-console.log(result.score)
-console.log(result.explanation)
-```
-
----
-
-### `neural().clusters(input?, options?)` → `Promise`
-
-Automatic clustering.
-
-```typescript
-const clusters = await brain.neural().clusters({
- algorithm: 'kmeans',
- k: 5,
- minSize: 3
-})
-
-clusters.forEach(cluster => {
- console.log(cluster.label)
- console.log(cluster.items)
- console.log(cluster.centroid)
-})
-```
-
----
-
-### `neural().neighbors(id, options?)` → `Promise`
-
-Find k-nearest neighbors.
-
-```typescript
-const neighbors = await brain.neural().neighbors(entityId, {
- k: 10,
- threshold: 0.7
-})
-```
-
----
-
-### `neural().outliers(threshold?)` → `Promise`
-
-Detect outlier entities.
-
-```typescript
-const outliers = await brain.neural().outliers(0.3)
-// Returns entity IDs that are outliers
-```
-
----
-
-### `neural().visualize(options?)` → `Promise`
-
-Generate visualization data.
-
-```typescript
-const vizData = await brain.neural().visualize({
- maxNodes: 100,
- dimensions: 3,
- algorithm: 'force',
- includeEdges: true
-})
-// Use with D3.js, Cytoscape, GraphML tools
-```
-
----
-
-### Performance Methods
-
-#### `neural().clusterFast(options)` → `Promise`
-
-Fast clustering for large datasets.
-
-```typescript
-const clusters = await brain.neural().clusterFast({
- k: 10,
- maxIterations: 50
-})
-```
-
----
-
-#### `neural().clusterLarge(options)` → `Promise`
-
-Streaming clustering for very large datasets.
-
-```typescript
-const clusters = await brain.neural().clusterLarge({
- k: 20,
- batchSize: 1000
-})
-```
-
----
-
## Import & Export
### `import(source, options?)` → `Promise`
@@ -1295,21 +1260,19 @@ Smart import with auto-detection (CSV, Excel, PDF, JSON, URLs).
```typescript
// CSV import
await brain.import('data.csv', {
- format: 'csv',
- createEntities: true
+ format: 'csv',
+ createEntities: true
})
-// Excel import
+// Excel import (all sheets processed automatically)
await brain.import('sales.xlsx', {
- format: 'excel',
- sheets: ['Q1', 'Q2']
+ format: 'excel',
+ vfsPath: '/imports/sales', // optional: mirror into the VFS
+ groupBy: 'sheet'
})
-// PDF import
-await brain.import('research.pdf', {
- format: 'pdf',
- extractTables: true
-})
+// PDF import (tables extracted automatically)
+await brain.import('research.pdf', { format: 'pdf' })
// URL import
await brain.import('https://api.example.com/data.json')
@@ -1318,31 +1281,61 @@ await brain.import('https://api.example.com/data.json')
**Parameters:**
- `source`: `string | Buffer | object` - File path, URL, buffer, or object
- `options?`: Import configuration
- - `format?`: `'csv' | 'excel' | 'pdf' | 'json'` - Auto-detected if omitted
- - `createEntities?`: `boolean` - Create entities from rows
- - `sheets?`: `string[]` - Excel sheets to import
- - `extractTables?`: `boolean` - Extract tables from PDF
+ - `format?`: `'excel' | 'pdf' | 'csv' | 'json' | 'markdown' | 'yaml' | 'docx' | 'image'` - Auto-detected if omitted
+ - `vfsPath?`: `string` - Mirror imported content into the VFS at this path
+ - `groupBy?`: `'type' | 'sheet' | 'flat' | 'custom'` - VFS grouping strategy
+ - `createEntities?`: `boolean` - Create entities from rows
+ - `createRelationships?`: `boolean` - Create relationships between extracted entities
+ - `preserveSource?`: `boolean` - Save the original file in the VFS
+ - `enableNeuralExtraction?`: `boolean` - Extract entity names via AI
+ - `enableRelationshipInference?`: `boolean` - Infer relationships via AI
+ - `enableConceptExtraction?`: `boolean` - Extract entity types via AI
+ - `confidenceThreshold?`: `number` - Minimum confidence for extracted entities
+ - `onProgress?`: `(progress) => void` - Progress callback (stage, counts, throughput, ETA)
**Returns:** `Promise` - Import statistics
-**Note:** Import always uses the current branch (v5.1.0 verified).
-
**[📖 Complete Import Guide →](../guides/import-anything.md)**
---
-### Export & Snapshots
+### Export & Import (portable) + Snapshots (native)
+
+**Portable graph export/import** — `brain.export()` / `brain.import()` (`PortableGraph` v1, versioned
+JSON, partial-or-whole, cross-version). `export()` lives on the immutable `Db`, so it composes
+with `now()`/`asOf()`/`with()`:
```typescript
-// Export to file
-await brain.export('/path/to/backup.brainy')
+// Export part or all of the brain to a portable, versioned document
+const graph = await brain.export({ ids }, { includeVectors: true })
-// Create instant snapshot using COW fork
-await brain.fork('backup-2025-01-19')
+// Restore it — import() routes a PortableGraph to the graph round-trip (merge by id)
+await otherBrain.import(graph, { onConflict: 'merge' })
-// Time-travel to specific commit
-const snapshot = await brain.asOf(commitId)
+// Time-travel export (serialize a past generation) / what-if export (a speculative state)
+const past = await brain.asOf(gen)
+const asWas = await past.export({ collection: id })
+await past.release()
+await brain.now().with(ops).export({ ids })
+```
+
+Selectors: `{ ids }`, `{ collection }` (alias `memberOf`), `{ connected: { from, depth } }`,
+`{ vfsPath }`, predicate (`{ type, subtype, where, service }`), or whole brain (omit). See the
+**[Export & Import guide](../guides/export-and-import.md)**. Distinct from `brain.import(file)`
+(CSV/PDF/Excel/JSON ingestion — `import()` dispatches on whether you pass a `PortableGraph` or a file).
+
+**Native whole-brain snapshot** (generation-preserving, not portable JSON):
+
+```typescript
+// Instant hard-link snapshot via the Db API
+const pin = brain.now()
+await pin.persist('/backups/2026-06-11')
+await pin.release()
+
+// Time-travel to a past generation or timestamp
+const snapshot = await brain.asOf(new Date('2026-06-01'))
const entities = await snapshot.find({ limit: 100 })
+await snapshot.release()
```
---
@@ -1355,47 +1348,37 @@ const entities = await snapshot.find({ limit: 100 })
const brain = new Brainy({
// Storage configuration
storage: {
- type: 'memory', // memory | opfs | filesystem | s3 | r2 | gcs | azure
- path: './brainy-data', // For filesystem storage
- compression: true, // Enable gzip compression (60-80% savings)
-
- // Cloud storage configs (see Storage Adapters section)
- s3Storage: { ... },
- r2Storage: { ... },
- gcsStorage: { ... },
- azureStorage: { ... }
+ type: 'filesystem', // 'memory' | 'filesystem' | 'auto'
+ path: './brainy-data'
},
- // HNSW vector index config
- hnsw: {
- M: 16, // Connections per layer
- efConstruction: 200, // Construction quality
- efSearch: 100, // Search quality
- typeAware: true // Enable type-aware indexing (v4.0+)
+ // Vector index configuration (2 knobs)
+ vector: {
+ recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
+ persistMode: 'immediate' // 'immediate' | 'deferred'
},
// Model configuration (embedded in WASM - zero config needed)
- // Model: all-MiniLM-L6-v2 (384 dimensions)
- // Device: CPU via WASM (works everywhere)
+ // Model: all-MiniLM-L6-v2 (384 dimensions)
+ // Device: CPU via WASM (works everywhere)
- // Cache configuration
+ // Cache configuration — `true`/`false`, or an options object
cache: {
- enabled: true,
maxSize: 10000,
- ttl: 3600000 // 1 hour in ms
+ ttl: 3600000 // 1 hour in ms
}
})
-await brain.init() // Required! VFS auto-initialized in v5.1.0
+await brain.init() // Required! VFS auto-initialized
```
---
## Storage Adapters
-All 7 storage adapters support **copy-on-write branching** (v5.0+).
+Brainy 8.0 ships two adapters — both support the full Db API (generational history, snapshots, restore).
-### Memory (Default)
+### Memory (Default for Tests)
```typescript
const brain = new Brainy({
@@ -1407,125 +1390,32 @@ const brain = new Brainy({
---
-### OPFS (Browser)
-
-```typescript
-const brain = new Brainy({
- storage: { type: 'opfs' }
-})
-```
-
-**Use case:** Browser applications with persistent storage
-
----
-
-### Filesystem (Node.js)
+### Filesystem (Default for Node)
```typescript
const brain = new Brainy({
storage: {
type: 'filesystem',
- path: './brainy-data',
- compression: true // 60-80% space savings
+ path: './brainy-data'
}
})
```
-**Use case:** Node.js applications, local persistence
+**Use case:** Node.js applications, single-node production deployments
+
+For off-site backup, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`) — Brainy itself doesn't reach out to cloud object stores.
---
-### AWS S3
+### Auto
```typescript
const brain = new Brainy({
- storage: {
- type: 's3',
- s3Storage: {
- bucketName: 'my-brainy-data',
- region: 'us-east-1',
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
- }
- }
-})
-
-// Enable Intelligent-Tiering for 96% cost savings
-await brain.storage.enableIntelligentTiering('entities/', 'auto-tier')
-```
-
-**Use case:** Production deployments, scalable storage
-
-**[📖 AWS S3 Cost Optimization →](../operations/cost-optimization-aws-s3.md)**
-
----
-
-### Cloudflare R2
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'r2',
- r2Storage: {
- accountId: process.env.CF_ACCOUNT_ID,
- bucketName: 'my-brainy-data',
- accessKeyId: process.env.CF_ACCESS_KEY_ID,
- secretAccessKey: process.env.CF_SECRET_ACCESS_KEY
- }
- }
+ storage: { type: 'auto', path: './brainy-data' }
})
```
-**Use case:** Zero egress fees, cost-effective storage
-
-**[📖 R2 Cost Optimization →](../operations/cost-optimization-cloudflare-r2.md)**
-
----
-
-### Google Cloud Storage (GCS)
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs',
- gcsStorage: {
- bucketName: 'my-brainy-data',
- projectId: process.env.GCP_PROJECT_ID,
- keyFilename: './gcp-key.json'
- }
- }
-})
-
-// Enable auto-tiering
-await brain.storage.enableAutoclass({
- terminalStorageClass: 'ARCHIVE'
-})
-```
-
-**Use case:** Google Cloud ecosystem, global distribution
-
-**[📖 GCS Cost Optimization →](../operations/cost-optimization-gcs.md)**
-
----
-
-### Azure Blob Storage
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'azure',
- azureStorage: {
- accountName: process.env.AZURE_STORAGE_ACCOUNT,
- accountKey: process.env.AZURE_STORAGE_KEY,
- containerName: 'brainy-data'
- }
- }
-})
-```
-
-**Use case:** Azure ecosystem, enterprise deployments
-
-**[📖 Azure Cost Optimization →](../operations/cost-optimization-azure.md)**
+Picks `'filesystem'` on Node with a writable `path`, falls back to `'memory'` otherwise.
---
@@ -1561,7 +1451,205 @@ const count = await brain.getVerbCount()
---
-### `embed(data)` → `Promise` ✨ *Enhanced v7.1.0*
+### Subtype & facet APIs
+
+Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
+
+#### `counts.bySubtype(type, subtype?)` → `Record | number`
+
+O(1) subtype counts for a NounType (backed by the persisted rollup).
+
+```typescript
+brain.counts.bySubtype(NounType.Person)
+// → { employee: 12, customer: 847, vendor: 34 }
+
+brain.counts.bySubtype(NounType.Person, 'employee')
+// → 12
+```
+
+#### `counts.topSubtypes(type, n=10)` → `Array<[subtype, count]>`
+
+Top N subtypes ranked by count.
+
+```typescript
+brain.counts.topSubtypes(NounType.Person, 3)
+// → [['customer', 847], ['employee', 12], ['vendor', 34]]
+```
+
+#### `subtypesOf(type)` → `string[]`
+
+Sorted distinct subtypes seen for a NounType.
+
+```typescript
+brain.subtypesOf(NounType.Person)
+// → ['customer', 'employee', 'vendor']
+```
+
+#### `counts.byRelationshipSubtype(verb, subtype?)` → `Record | number`
+
+Verb-side mirror of `counts.bySubtype`. O(1) per-VerbType-per-subtype counts.
+
+```typescript
+brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
+// → { direct: 12, 'dotted-line': 3 }
+
+brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')
+// → 12
+```
+
+#### `counts.topRelationshipSubtypes(verb, n=10)` → `Array<[subtype, count]>`
+
+Top N subtypes for a `VerbType` ranked by count.
+
+```typescript
+brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
+// → [['direct', 12], ['dotted-line', 3]]
+```
+
+#### `relationshipSubtypesOf(verb)` → `string[]`
+
+Sorted distinct subtypes seen for a `VerbType`.
+
+```typescript
+brain.relationshipSubtypesOf(VerbType.ReportsTo)
+// → ['direct', 'dotted-line']
+```
+
+#### `audit(options?)` → `Promise` (7.30.1+)
+
+Diagnostic — find entities and relationships missing a `subtype` value, grouped by type. The companion to `migrateField()` / `fillSubtypes()` — answers "what would break if I enabled strict subtype enforcement?".
+
+```typescript
+const report = await brain.audit()
+// {
+// entitiesWithoutSubtype: { event: 24, document: 3 },
+// relationshipsWithoutSubtype: { relatedTo: 1402 },
+// total: 1429,
+// scanned: 8400,
+// recommendation: 'Found 1429 entries without subtype. ...'
+// }
+```
+
+**Parameters:**
+- `options.includeVFS?`: `boolean` — When `false` (default), VFS infrastructure entities (`metadata.isVFSEntity` / `metadata.isVFS`) are excluded. They bypass enforcement anyway, so counting them is noise.
+- `options.batchSize?`: `number` — Pagination batch size (default 200).
+- `options.onProgress?`: `(progress: { scanned, missingSubtype }) => void` — Progress callback per batch.
+
+Run before adopting an SDK that registers `requireSubtype()` rules, or before upgrading to Brainy 8.0 (which makes strict mode the default). See the [Strict mode in practice](../guides/subtypes-and-facets.md#strict-mode-in-practice-for-sdk-style-vocabulary-consumers) guide for the full migration recipe.
+
+#### `requireSubtype(type, options?)` → `void`
+
+Register subtype enforcement for a specific `NounType` or `VerbType`. Unified API for nouns and verbs. Composes with the brain-wide `requireSubtype` constructor flag.
+
+```typescript
+// Lock down Person sub-classification
+brain.requireSubtype(NounType.Person, {
+ values: ['employee', 'customer', 'vendor'],
+ required: true
+})
+
+// Lock down management edges
+brain.requireSubtype(VerbType.ReportsTo, {
+ values: ['direct', 'dotted-line'],
+ required: true
+})
+```
+
+**Parameters:**
+- `type`: `NounType | VerbType` - The type to register
+- `options.values?`: `string[]` - Vocabulary whitelist (rejects off-vocab values)
+- `options.required?`: `boolean` - Whether subtype is required (default: `true`)
+
+#### Brain-wide strict mode — `new Brainy({ requireSubtype })`
+
+Constructor option that enforces subtype on every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()` for every type:
+
+```typescript
+// Every write must include subtype
+const brain = new Brainy({ requireSubtype: true })
+
+// Exempt specific types (e.g. catch-all Thing)
+const brain2 = new Brainy({
+ requireSubtype: { except: [NounType.Thing, NounType.Custom] }
+})
+```
+
+When strict mode is on:
+- Every public write path checks the pairing guarantee.
+- `addMany()` / `relateMany()` validate all items BEFORE any storage write — atomic-fail, no partial writes.
+- Brainy's own VFS infrastructure writes bypass via the `metadata.isVFSEntity: true` marker.
+- Per-type registrations always apply regardless of the brain-wide flag.
+
+The default since 8.0.0 — pass `requireSubtype: false` to opt out while migrating pre-8.0 data.
+
+#### `trackField(name, options?)` → `void`
+
+Register a metadata field for cardinality + per-NounType breakdown stats. With `values: [...]`, validates against the whitelist on `add()`/`update()`.
+
+```typescript
+brain.trackField('status') // basic
+brain.trackField('status', { perType: true }) // with per-NounType breakdown
+brain.trackField('priority', { values: ['low', 'med', 'high'] }) // strict vocabulary
+```
+
+#### `counts.byField(name, options?)` → `Promise>`
+
+Counts by value for a tracked field. Requires `perType: true` registration if filtering by NounType.
+
+```typescript
+await brain.counts.byField('status')
+// → { todo: 12, doing: 3, done: 47 }
+
+await brain.counts.byField('status', { type: NounType.Task })
+// → { todo: 8, doing: 2, done: 30 }
+```
+
+#### `migrateField(options)` → `Promise`
+
+Stream-and-rewrite a field across the brain. Supports `metadata.X`, `data.X`, and top-level paths. Idempotent.
+
+```typescript
+// One-shot rewrite
+await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
+
+// Deprecation window — keep source field readable
+await brain.migrateField({ from: 'data.kind', to: 'subtype', readBoth: true })
+
+// With progress reporting
+await brain.migrateField({
+ from: 'metadata.kind',
+ to: 'subtype',
+ batchSize: 500,
+ onProgress: ({ scanned, migrated }) => console.log(`${scanned} / ${migrated}`)
+})
+```
+
+Returns `{ scanned: number, migrated: number, skipped: number, errors: Array<{id, error}> }`.
+
+#### `fillSubtypes(rules, options?)` → `Promise` (8.0+)
+
+Back-fill missing `subtype` values across entities AND relationships in one streaming pass — the migration companion to `audit()`. Keys are `NounType`/`VerbType` values; each rule is a literal subtype string or a function deriving one from the entry (return `undefined` to decline). Idempotent: entries that already carry a subtype are never touched, so a crashed run is resumed safely by re-running.
+
+```typescript
+const report = await brain.fillSubtypes({
+ [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified', // derived
+ [NounType.Document]: 'general', // literal default
+ [VerbType.RelatedTo]: 'unspecified' // relationship rule
+})
+// → { scanned, filled, skipped, errors, byType }
+```
+
+**Parameters:**
+- `rules`: `FillSubtypeRules` - Map of NounType/VerbType → literal subtype or `(entry) => string | undefined`
+- `options.includeVFS?`: `boolean` - Also fill VFS infrastructure entries (default `false`)
+- `options.batchSize?`: `number` - Pagination batch size (default `200`)
+- `options.onProgress?`: `(progress: { scanned, filled, skipped }) => void` - Per-batch callback
+
+Returns `{ scanned, filled, skipped, errors, byType }`. After a clean run, `skipped` equals the remaining `audit().total`. See the [migration recipe](../guides/subtypes-and-facets.md).
+
+---
+
+### `embed(data)` → `Promise` ✨
Generate embedding vector from text or data.
@@ -1573,30 +1661,32 @@ console.log(vector.length) // 384
---
-### `embedBatch(texts)` → `Promise` ✨ *New v7.1.0*
+### `embedBatch(texts)` → `Promise` ✨
-Batch embed multiple texts efficiently.
+Batch embed multiple texts using native WASM batch API (single forward pass).
```typescript
const embeddings = await brain.embedBatch([
- 'Machine learning is fascinating',
- 'Deep neural networks',
- 'Natural language processing'
+ 'Machine learning is fascinating',
+ 'Deep neural networks',
+ 'Natural language processing'
])
-console.log(embeddings.length) // 3
+console.log(embeddings.length) // 3
console.log(embeddings[0].length) // 384
```
+> Uses the WASM engine's native `embed_batch()` for a single model forward pass instead of N individual calls. This is the same batch API used internally by `highlight()`.
+
---
-### `similarity(textA, textB)` → `Promise` ✨ *New v7.1.0*
+### `similarity(textA, textB)` → `Promise` ✨
Calculate semantic similarity between two texts.
```typescript
const score = await brain.similarity(
- 'The cat sat on the mat',
- 'A feline was resting on the rug'
+ 'The cat sat on the mat',
+ 'A feline was resting on the rug'
)
console.log(score) // ~0.85 (high semantic similarity)
```
@@ -1605,7 +1695,7 @@ console.log(score) // ~0.85 (high semantic similarity)
---
-### `neighbors(entityId, options?)` → `Promise
`, fenced/indented code blocks, or code nodes
+- `'prose'` — regular paragraph text
+- `'label'` — labels, captions, metadata-like text
+
+**Custom Content Extractors**
+
+New `contentExtractor` parameter lets developers plug in custom parsers:
+
+```typescript
+const highlights = await brain.highlight({
+ query: "function",
+ text: sourceCode,
+ contentExtractor: (text) => treeSitterParse(text) // Custom parser
+})
+```
+
+**Content Type Hints**
+
+New `contentType` parameter to skip auto-detection:
+
+```typescript
+await brain.highlight({ query: "test", text: input, contentType: 'html' })
+```
+
+### New Types
+
+- `ContentType`: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`
+- `ContentCategory`: `'prose' | 'heading' | 'code' | 'label'`
+- `ExtractedSegment`: `{ text: string, contentCategory: ContentCategory }`
+- `HighlightParams.contentType?` — optional content type hint
+- `HighlightParams.contentExtractor?` — optional custom parser callback
+- `Highlight.contentCategory?` — content role annotation
+
+## [7.7.0](https://github.com/soulcraftlabs/brainy/compare/v7.6.1...v7.7.0) (2026-01-26)
+
+### Features
+
+**Match Visibility in Search Results**
+
+Search results now include detailed match information:
+- `textMatches: string[]` - Query words found in entity
+- `textScore: number` - Text match quality (0-1)
+- `semanticScore: number` - Semantic similarity (0-1)
+- `matchSource: 'text' | 'semantic' | 'both'` - Where result came from
+
+```typescript
+const results = await brain.find({ query: 'david the warrior' })
+results[0].textMatches // ["david", "warrior"]
+results[0].semanticScore // 0.87
+results[0].matchSource // "both"
+```
+
+**Semantic Highlighting API**
+
+New `highlight()` method shows which concepts matched:
+
+```typescript
+const highlights = await brain.highlight({
+ query: "david the warrior",
+ text: "David Smith is a brave fighter who battles dragons"
+})
+// Returns both exact matches and semantic concepts:
+// [
+// { text: "David", score: 1.0, matchType: "text" },
+// { text: "fighter", score: 0.78, matchType: "semantic" },
+// { text: "battles", score: 0.72, matchType: "semantic" }
+// ]
+```
+
+**Scalable Word Indexing**
+
+- Increased word limit from 50 to 5000 words per entity
+- Supports articles, chapters, and large documents
+- Roaring Bitmaps provide efficient compression at scale
+
+### Performance
+
+- O(1) fast path in `findMatchingWords()` for text results
+- 500 chunk limit in `highlight()` for memory safety
+- Stopword filtering reduces embedding overhead
+
+### [7.6.1](https://github.com/soulcraftlabs/brainy/compare/v7.6.0...v7.6.1) (2026-01-26)
+
+- docs: add link to hosted API documentation at soulcraft.com/docs
+
+## [7.6.0](https://github.com/soulcraftlabs/brainy/compare/v7.5.0...v7.6.0) (2026-01-26)
+
+- chore: republish (npm ghost versions in 7.5.x range)
### [7.5.0](https://github.com/soulcraftlabs/brainy/compare/v7.4.1...v7.5.0) (2026-01-26)
@@ -260,7 +1096,7 @@ v7.0.0 introduced **breaking changes** to the embedding system:
| copy 15 files | ~120s | ~20-40s | 3-6x faster |
| move 15 files | ~240s | ~40-60s | 4-6x faster |
-Requested by: Soulcraft Workshop team (BRAINY-VFS-RMDIR-PERFORMANCE)
+Requested by: a consumer team (BRAINY-VFS-RMDIR-PERFORMANCE)
### [6.3.2](https://github.com/soulcraftlabs/brainy/compare/v6.3.1...v6.3.2) (2025-12-09)
@@ -303,7 +1139,7 @@ Requested by: Soulcraft Workshop team (BRAINY-VFS-RMDIR-PERFORMANCE)
**Fixed VFS tree operations on cloud storage (GCS, S3, Azure, R2, OPFS)**
**Issue:** Despite v6.1.0's PathResolver optimization, `vfs.getTreeStructure()` remained critically slow on cloud storage:
-- **Workshop Production (GCS):** 5,304ms for tree with maxDepth=2
+- **Production (GCS) deployment:** 5,304ms for tree with maxDepth=2
- **Root Cause:** Tree traversal made 111+ separate storage calls (one per directory)
- **Why v6.1.0 didn't help:** v6.1.0 optimized path→ID resolution, but tree traversal still called `getChildren()` 111+ times
@@ -331,7 +1167,7 @@ Result: 111 storage calls → 1 storage call
- `src/vfs/VirtualFileSystem.ts:730-762` - Updated `getDescendants()` to use batch fetch
**Impact:**
-- ✅ Workshop file explorer now loads instantly on GCS
+- ✅ Consumer file explorer now loads instantly on GCS
- ✅ Clean architecture: one code path, no fallbacks
- ✅ Production-scale: uses in-memory graph + single batch fetch
- ✅ Works for ALL storage adapters (GCS, S3, Azure, R2, OPFS, FileSystem)
@@ -670,7 +1506,7 @@ for (const id of pageIds) {
**Fix:** Set `isInitialized = true` at the START of `BaseStorage.init()` (before any initialization work) to prevent recursive calls. Flag is reset to `false` on error to allow retries.
**Impact:**
-- ✅ Fixes production blocker reported by Workshop team
+- ✅ Fixes production blocker reported by a consumer team
- ✅ All 8 storage adapters fixed (FileSystem, Memory, S3, R2, GCS, Azure, OPFS, Historical)
- ✅ Init completes in ~1 second on fresh installation (was hanging indefinitely)
- ✅ No new test failures introduced (1178 tests passing)
@@ -950,7 +1786,7 @@ See comprehensive guides:
v5.10.0 reintroduced a critical bug where `BlobStorage.read()` was hashing wrapped binary data instead of unwrapped content, causing all blob integrity checks to fail:
- **Symptom**: `Blob integrity check failed: ` errors on every VFS file read
- **Root Cause**: Missing defense-in-depth unwrap verification in `BlobStorage.read()`
-- **Impact**: 100% failure rate for VFS file operations in Workshop application
+- **Impact**: 100% failure rate for VFS file operations in A consumer application
### The Fix (v5.10.1)
1. **Defense-in-Depth Unwrapping**: Added unwrap verification in `BlobStorage.read()` before hash check
@@ -1098,7 +1934,7 @@ Reverted storage internals to v5.6.3 implementation:
- ✅ No 12+ second delays per entity
### Verification
-Workshop team (production users) should upgrade immediately:
+a consumer team (production users) should upgrade immediately:
```bash
npm install @soulcraft/brainy@5.7.1
```
@@ -1135,7 +1971,7 @@ Expected behavior after upgrade:
### 🐛 Bug Fixes
-* **storage**: Fix `clear()` not deleting COW version control data ([#workshop-bug-report](https://github.com/soulcraftlabs/brainy/issues))
+* **storage**: Fix `clear()` not deleting COW version control data (consumer-reported)
- Fixed all storage adapters to properly delete `_cow/` directory on clear()
- Fixed in-memory entity counters not being reset after clear()
- Prevents COW reinitialization after clear() by setting `cowEnabled = false`
@@ -1233,10 +2069,10 @@ Expected behavior after upgrade:
- Impact: All relationship queries via `getRelations()`, `getRelationships()`
- Reference: `src/storage/baseStorage.ts:2030-2040`, `src/storage/baseStorage.ts:2081-2091`
-* **Workshop blob integrity**: Verified v5.4.0 lazy-loading asOf() prevents corruption
+* **Consumer blob integrity**: Verified v5.4.0 lazy-loading asOf() prevents corruption
- HistoricalStorageAdapter eliminates race conditions
- Snapshots created on-demand (no commit-time snapshot)
- - Verified with 570-entity test matching Workshop production scale
+ - Verified with 570-entity test matching consumer production scale
### ⚡ Performance Adjustments
@@ -1529,7 +2365,7 @@ v5.1.0 delivers a significantly improved developer experience:
- **Problem**: In v5.0.0, `saveNoun()` was called before `saveNounMetadata()`, causing TypeAwareStorage to default entity types to 'thing' and save to wrong storage paths
- **Impact**: Broke VFS file operations, `brain.get()`, `brain.relate()`, and all features depending on entity metadata
- **Solution**: Reversed save order - now saves metadata FIRST, then noun vector
- - **Fixes**: [Workshop Bug Report](https://github.com/soulcraftlabs/brain-cloud/issues/VFS-METADATA-MISSING)
+ - **Fixes**: VFS metadata-missing regression (internal tracker)
**Fork API: Lazy COW Initialization**
@@ -1556,7 +2392,7 @@ v5.1.0 delivers a significantly improved developer experience:
### 📊 Impact
-* **Unblocks**: Workshop team and all VFS users
+* **Unblocks**: a consumer team and all VFS users
* **Fixes**: All metadata-dependent features (get, relate, find, VFS)
* **Maintains**: Full backward compatibility with v4.x data
@@ -1904,7 +2740,7 @@ This release transforms Brainy imports from entity extractors into true knowledg
- **Relationship Type Metadata**: All relationships tagged as `vfs`, `semantic`, or `provenance` for filtering
- **Enhanced Column Detection**: 7 relationship types (vs 1 previously) - Location, Owner, Creator, Uses, Member, Friend, Related
- **Type-Based Inference**: Smart relationship classification based on entity types and context analysis
- - **Impact**: Workshop import now creates ~3,900 relationships (vs 581), with 5-20+ connections per entity
+ - **Impact**: A consumer import now creates ~3,900 relationships (vs 581), with 5-20+ connections per entity
* **import**: New configuration option `createProvenanceLinks` (defaults to `true`)
- Enables/disables provenance relationship creation
@@ -1946,7 +2782,7 @@ Graph: Rich network, 5-20+ connections per entity
### [4.7.4](https://github.com/soulcraftlabs/brainy/compare/v4.7.3...v4.7.4) (2025-10-27)
-**CRITICAL SYSTEMIC VFS BUG FIX - Workshop Team Unblocked!**
+**CRITICAL SYSTEMIC VFS BUG FIX - A consumer team Unblocked!**
This hotfix resolves a systemic bug affecting ALL storage adapters that caused VFS queries to return empty results even when data existed.
@@ -1958,7 +2794,7 @@ This hotfix resolves a systemic bug affecting ALL storage adapters that caused V
- **Bug Pattern**: `if (!metadata) continue` in getNouns()/getVerbs() methods
- **Fixed Locations**: 12 bug sites across 7 adapters (TypeAware, Memory, FileSystem, GCS, S3, R2, OPFS, Azure)
- **Solution**: Allow optional metadata with `metadata: (metadata || {}) as NounMetadata`
- - **Result**: Workshop team UNBLOCKED - VFS entities now queryable
+ - **Result**: a consumer team UNBLOCKED - VFS entities now queryable
* **neural**: Fix SmartExtractor weighted score threshold bug (28 test failures → 4)
- **Root Cause**: Single signal with 0.8 confidence × 0.2 weight = 0.16 < 0.60 threshold
@@ -2026,7 +2862,7 @@ Clean separation between VFS (Virtual File System) entities and knowledge graph
### 🐛 Critical Bug Fixes
* **vfs.initializeRoot()**: add includeVFS to prevent duplicate root creation
- - **Critical Fix**: VFS init was creating ~10 duplicate root entities (Workshop team issue)
+ - **Critical Fix**: VFS init was creating ~10 duplicate root entities (a consumer team issue)
- **Root Cause**: `initializeRoot()` called `brain.find()` without `includeVFS: true`, never found existing VFS root
- **Impact**: Every `vfs.init()` created a new root, causing empty `readdir('/')` results
- **Solution**: Added `includeVFS: true` to root entity lookup (line 171)
@@ -2120,7 +2956,7 @@ const files = await vfs.search('documentation')
- **Root Cause**: HNSW `rebuild()` and Graph `rebuild()` methods still called `getNounsWithPagination()`/`getVerbsWithPagination()` repeatedly
- Each pagination call triggered `getAllShardedFiles()` reading all 256 shard directories
- For 1,157 entities: MetadataIndex (2-3s) + HNSW (~20s) + Graph (~10s) = **30-35 seconds total**
- - Workshop team reported: "v4.2.3 is at batch 7 after ~60 seconds" - still far from claimed 100x improvement
+ - a consumer team reported: "v4.2.3 is at batch 7 after ~60 seconds" - still far from claimed 100x improvement
- **Solution**: Apply v4.2.3 adaptive loading pattern to ALL 3 indexes
- **FileSystemStorage/MemoryStorage/OPFSStorage**: Load all entities at once (limit: 10000000)
- **Cloud storage (GCS/S3/R2/Azure)**: Keep pagination (native APIs are efficient)
@@ -2141,7 +2977,7 @@ const files = await vfs.search('documentation')
- Graph Index: Loads all verbs at once for local, paginated for cloud (lines 300-361)
- Pattern matches v4.2.3 MetadataIndex implementation exactly
- Zero config: Completely automatic based on storage adapter type
- - **Resolution**: Fully resolves Workshop team's v4.2.x performance regression
+ - **Resolution**: Fully resolves a consumer team's v4.2.x performance regression
- **Files Changed**:
- `src/hnsw/hnswIndex.ts` (updated rebuild() with adaptive loading)
- `src/graph/graphAdjacencyIndex.ts` (updated rebuild() with adaptive loading)
@@ -2166,7 +3002,7 @@ const files = await vfs.search('documentation')
- Pagination was designed for cloud storage socket exhaustion
- FileSystem doesn't need pagination - can handle loading thousands of entities at once
- Eliminates repeated directory scans: 3 batches × 256 dirs → 1 batch × 256 dirs
- - **Workshop Team**: This resolves the v4.2.2 stalling issue - rebuild will now complete in seconds
+ - **A consumer team**: This resolves the v4.2.2 stalling issue - rebuild will now complete in seconds
- **Files Changed**: `src/utils/metadataIndex.ts` (rebuilt() method with adaptive loading strategy)
### [4.2.2](https://github.com/soulcraftlabs/brainy/compare/v4.2.1...v4.2.2) (2025-10-23)
@@ -2208,7 +3044,7 @@ const files = await vfs.search('documentation')
- Universal: All storage adapters (FileSystem, GCS, S3, R2, Memory, OPFS)
- **Zero Config**: Completely automatic, no configuration needed
- **Self-Healing**: Gracefully handles missing/corrupt registry (rebuilds once)
- - **Impact**: Fixes Workshop team bug report - production-ready at billion scale
+ - **Impact**: Fixes a consumer team bug report - production-ready at billion scale
- **Files Changed**: `src/utils/metadataIndex.ts` (added saveFieldRegistry/loadFieldRegistry methods, updated init/flush)
### [4.2.0](https://github.com/soulcraftlabs/brainy/compare/v4.1.4...v4.2.0) (2025-10-23)
@@ -2252,7 +3088,7 @@ const files = await vfs.search('documentation')
- Fixed property access bugs: `verb.target` → `verb.to`, `verb.verb` → `verb.type`
- Added comprehensive integration tests (14 tests covering all query patterns)
- Updated JSDoc documentation with usage examples
- - **Impact**: Resolves Workshop team bug where 524 imported relationships were inaccessible
+ - **Impact**: Resolves a consumer team bug where 524 imported relationships were inaccessible
- **Breaking**: None - fully backward compatible
### [4.1.2](https://github.com/soulcraftlabs/brainy/compare/v4.1.1...v4.1.2) (2025-10-21)
diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 00000000..568c10db
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,208 @@
+# Brainy - Claude Code Project Guide
+
+This file provides guidance for Claude Code (and human contributors) when working on the Brainy codebase.
+
+## Cross-Project Coordination
+
+Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
+
+**At session START:** Read the handoff. Find rows where Owner = Brainy. Act on those first.
+
+**At session END:** Mark completed actions ✅, delete rows you finished, delete threads with zero remaining actions. File must not grow. **If you shipped anything consumers need to know about, update `RELEASES.md` before closing.**
+
+**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
+
+**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`)
+
+---
+
+## Project Overview
+
+Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraft/brainy` on npm under the MIT license.
+
+## Getting Started
+
+```bash
+npm install # Install dependencies
+npm run build # Build the project
+npm test # Run test suite (Vitest)
+```
+
+## Architecture
+
+Full architecture reference: `.claude/skills/architecture.md`
+
+### Core Systems
+- **Storage** (`src/storage/`): Pluggable storage backends via StorageAdapter interface (`src/coreTypes.ts`)
+- **Vector Search** (`src/hnsw/`): HNSW approximate nearest neighbor search
+- **Graph Engine** (`src/graph/`): Relationship traversal with adjacency index and pathfinding
+- **Metadata Index** (`src/utils/metadataIndex.ts`): O(1) exact match, O(log n) range queries
+- **Triple Intelligence** (`src/triple/`): Unified query combining all three intelligence types
+- **Aggregation Engine** (`src/aggregation/`): Write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows
+- **Virtual Filesystem** (`src/vfs/`): Full VFS with semantic search
+
+### Type System
+- **NounType** (42 types): Entity classification -- Person, Concept, Collection, Document, Task, etc.
+- **VerbType** (127 types): Relationship types -- Contains, RelatedTo, PartOf, Creates, DependsOn, etc.
+- Defined in `src/types/graphTypes.ts`
+
+## Code Standards
+
+### TypeScript
+- Strict mode enabled
+- Target: ES2020, NodeNext module resolution
+- All new code must be TypeScript
+- Follow existing patterns -- read related code before writing
+
+### Quality
+- All code must compile without errors
+- All code must have working tests that exercise real behavior
+- No stub returns (`return {} as any`)
+- No incomplete implementations with TODO comments
+- If something can't be fully implemented, throw an explicit error rather than faking it
+
+### Verification Before Code Changes
+1. Check that interfaces and methods actually exist before using them
+2. Check that type properties are in the type definitions
+3. Run `npm test` -- tests must pass
+4. Run `npm run build` -- build must succeed
+
+### Testing
+- Framework: Vitest
+- Tests in `tests/` (unit, integration, benchmarks, comprehensive)
+- Use in-memory storage for speed where possible
+- Tests must exercise real behavior, not mock it
+- Benchmarks are in `tests/benchmarks/` (not tests/performance/)
+
+## Commit Conventions
+
+Use [Conventional Commits](https://www.conventionalcommits.org/):
+
+```
+feat: add new feature (minor version bump)
+fix: resolve bug (patch version bump)
+docs: update documentation (patch version bump)
+perf: improve performance (patch version bump)
+refactor: restructure code (patch version bump)
+test: add/update tests (patch version bump)
+```
+
+**Important:** Never use `BREAKING CHANGE` in commit messages. Major version bumps are manual decisions only (`npm run release:major`).
+
+## Docs Pipeline — soulcraft.com/docs
+
+Docs in `docs/**/*.md` are published with the npm package (included in `files`) and synced to soulcraft.com/docs on every portal deploy. Frontmatter controls what appears publicly.
+
+### Docs check triggers
+
+Run the docs check whenever the user says ANY of:
+- "commit, publish, release" / "release" / "publish"
+- "update the docs" / "make sure docs are accurate" / "check the docs"
+- "review docs" / "clean up docs"
+
+### Pre-release docs check (MANDATORY before every release)
+
+When the user says "commit, publish, release" or any variation, **before committing**:
+
+1. **Scan all files changed in this session** (and any recently added `docs/*.md` files)
+2. For each changed/new doc, decide: is this useful to external users?
+ - **Yes** → ensure it has complete frontmatter (add or update it)
+ - **No** (internal, migration, dev-only) → ensure it has no frontmatter or `public: false`
+3. For docs that already have frontmatter, verify:
+ - `description` still matches the actual content
+ - `next` links still exist and are still the right follow-up pages
+ - `title` matches the doc's h1
+4. Include frontmatter changes in the commit
+
+### Frontmatter format
+
+```yaml
+---
+title: Human-readable title
+slug: category/page-name # URL: soulcraft.com/docs/category/page-name
+public: true # false or absent = not published
+category: getting-started | concepts | guides | api
+template: guide | concept | api # controls layout on soulcraft.com
+order: 1 # sidebar position within category (lower = first)
+description: One sentence. What this doc covers and why it matters.
+next: # "Next steps" links shown at bottom of page
+ - category/other-slug
+---
+```
+
+### Category guide
+
+| category | use for |
+|----------|---------|
+| `getting-started` | installation, quick start, first steps |
+| `concepts` | how the system works, mental models |
+| `guides` | how to do specific things, recipes |
+| `api` | method reference, signatures, parameters |
+
+### What stays internal (no frontmatter / `public: false`)
+
+- Release guides, developer learning paths
+- Migration guides for old versions (v3→v4, v5.11)
+- Architecture analysis docs (clustering algorithms, etc.)
+- Anything in `docs/internal/`
+- Deployment/ops/cost docs (cloud-run, kubernetes, cost-optimization)
+
+## Release Process
+
+Fully automated via `scripts/release.sh`:
+
+```bash
+npm run release:dry # Preview (no changes)
+npm run release:patch # Bug fixes
+npm run release:minor # New features
+npm run release:major # Breaking changes (rare, manual decision)
+```
+
+The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release.
+
+After a successful release, remind the user:
+> "Published. Deploy portal to pick up the new docs → go to the portal project and deploy."
+
+Do NOT deploy portal from here. Portal is always deployed separately from within the portal project.
+
+## Closed-Source Product Names — HARD RULE
+
+Brainy is the only Soulcraft open-source project. Nothing in this repo — code, JSDoc, tests,
+docs, RELEASES.md, CHANGELOG.md, commit messages — may reference closed-source Soulcraft
+products by name (Workshop, Venue, Memory, Muse, Hall, Forge, Academy, Pulse, Heart,
+Collective, SDK) or by their specific class/method names (`BookingDraftService`,
+`getDemandHeatmap`, `systemKind`, etc.).
+
+When recording a consumer-reported bug, regression scenario, or release note:
+- Refer to "a consumer", "a downstream application", "a production deployment", or "an
+ internal report" — never name the product.
+- All doc examples must use generic domain values (`'employee'`, `'customer'`, `'invoice'`,
+ `'milestone'`, `OrderService`, `/orders/...`), not product-specific schemas.
+- Internal session artifacts (`.strategy/`, `~/.claude/plans/`, handoff files outside the
+ repo) MAY name products — those are not public.
+
+If you catch yourself typing a product name into a tracked file, stop and rephrase.
+
+## Performance Claims
+
+When documenting performance characteristics:
+- **MEASURED**: Cite the test file and line number
+- **PROJECTED**: Clearly label as extrapolated from tested scale
+- Never claim a performance figure without context or evidence
+
+## Debugging
+
+When a bug persists through 2+ fix attempts, switch to systematic debugging:
+1. Add comprehensive logging at every step
+2. Test with production-like data
+3. Trace the complete execution path
+4. Check both library code and consumer code
+5. Verify with actual test execution before declaring fixed
+
+## Key Paths
+
+- Main class: `src/brainy.ts`
+- Public API: `src/index.ts` (38+ exports)
+- Storage interface: `src/coreTypes.ts`
+- Type definitions: `src/types/`
+- Strategy/planning docs: `.strategy/` (gitignored, not public)
diff --git a/README.md b/README.md
index e0f9e855..d1342f52 100644
--- a/README.md
+++ b/README.md
@@ -1,745 +1,217 @@
-# Brainy
-
-
+
-[](https://www.npmjs.com/package/@soulcraft/brainy)
-[](https://www.npmjs.com/package/@soulcraft/brainy)
-[](LICENSE)
-[](https://www.typescriptlang.org/)
+Brainy
-## The Knowledge Operating System
+
+ Three database paradigms. One API. Zero configuration.
+ The in-process knowledge database for TypeScript — vector search, graph traversal,
+ and metadata filtering unified in a single query.
+
-**Every piece of knowledge in your application — living, connected, and intelligent.**
+
+
+
+
+
+
+
+
-Stop fighting with vector databases, graph databases, and document stores. Stop stitching together Pinecone + Neo4j + MongoDB. **Brainy does all three, in one elegant API, from prototype to planet-scale.**
-
-```javascript
-const brain = new Brainy()
-await brain.init()
-
-// That's it. You now have semantic search, graph relationships,
-// and document filtering. Zero configuration. Just works.
-```
-
-**Built by developers who were tired of:**
-- Spending weeks configuring embeddings, indexes, and schemas
-- Choosing between vector similarity OR graph relationships OR metadata filtering
-- Rewriting everything when you need to scale from 1,000 to 1,000,000,000 entities
-
-**Brainy makes the impossible simple: All three paradigms. One API. Any scale.**
+
+ Quick start ·
+ One query ·
+ Features ·
+ Scale with Cor ·
+ Docs
+
---
-## 👉 Choose Your Path
+Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact **three ways at once** and lets one call query them together:
-**New to Brainy? Pick your starting point:**
+| You write | Brainy indexes it as | You query it with |
+|---|---|---|
+| `data: 'Ada wrote the first program'` | a **384-dim vector** (local embedding — no API key) | `find({ query: 'computing pioneers' })` |
+| `metadata: { field: 'CS', year: 1843 }` | **structured fields** (O(1) exact, O(log n) range) | `find({ where: { year: { lessThan: 1900 } } })` |
+| `relate({ from: ada, to: babbage })` | a **typed, directed graph edge** | `find({ connected: { to: babbage, depth: 2 } })` |
-### 🚀 Path 1: I want to build something NOW
-**→ [Complete API Reference](docs/api/README.md)** ⭐ **Most developers start here** ⭐
-- Every method documented with examples
-- Quick start in 60 seconds
-- 1,870 lines of copy-paste ready code
-- **This is your primary resource**
+It runs **inside your process** — no server, no Docker, nothing to operate — and persists to plain files you can snapshot with a hard link.
-### 🧠 Path 2: I want to understand the big picture first
-**→ Keep reading below** for demos, architecture, and use cases
+**New here?** → **[What is Brainy? — plain-language overview, no jargon](docs/eli5.md)**
-### 📊 Path 3: I'm evaluating database options
-**→ Jump to [Why Revolutionary](#why-brainy-is-revolutionary)** or **[Benchmarks](#benchmarks)**
-
----
-
-## See It In Action
-
-**30 seconds to understand why Brainy is different:**
-
-```javascript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// Add knowledge with context
-const reactId = await brain.add({
- data: "React is a JavaScript library for building user interfaces",
- type: NounType.Concept,
- metadata: { category: "frontend", year: 2013 }
-})
-
-const nextId = await brain.add({
- data: "Next.js framework for React with server-side rendering",
- type: NounType.Concept,
- metadata: { category: "framework", year: 2016 }
-})
-
-// Create relationships
-await brain.relate({ from: nextId, to: reactId, type: VerbType.BuiltOn })
-
-// NOW THE MAGIC: Query with natural language
-const results = await brain.find({
- query: "modern frontend frameworks", // 🔍 Vector similarity
- where: { year: { greaterThan: 2015 } }, // 📊 Document filtering
- connected: { to: reactId, depth: 2 } // 🕸️ Graph traversal
-})
-
-// ALL THREE PARADIGMS. ONE QUERY. 10ms response time.
-```
-
-**This is impossible with traditional databases.** Brainy makes it trivial.
-
----
-
-## Quick Start
+## Quick start
```bash
-npm install @soulcraft/brainy
+bun add @soulcraft/brainy # Bun ≥ 1.1 — recommended
+npm install @soulcraft/brainy # Node.js ≥ 22
```
-### Your First Knowledge Graph (60 seconds)
-
```javascript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-const brain = new Brainy()
+const brain = new Brainy() // in-memory; one line swaps to disk
await brain.init()
-// Add knowledge
-const jsId = await brain.add({
- data: "JavaScript is a programming language",
- type: NounType.Concept,
- metadata: { category: "language", year: 1995 }
+// Text auto-embeds locally; metadata auto-indexes
+const react = await brain.add({
+ data: 'React is a JavaScript library for building user interfaces',
+ type: NounType.Concept,
+ subtype: 'library',
+ metadata: { category: 'frontend', year: 2013 }
})
-const nodeId = await brain.add({
- data: "Node.js runtime environment",
- type: NounType.Concept,
- metadata: { category: "runtime", year: 2009 }
+const next = await brain.add({
+ data: 'Next.js is a React framework with server-side rendering',
+ type: NounType.Concept,
+ subtype: 'framework',
+ metadata: { category: 'frontend', year: 2016 }
})
-// Create relationships
-await brain.relate({ from: nodeId, to: jsId, type: VerbType.Executes })
+await brain.relate({ from: next, to: react, type: VerbType.DependsOn, subtype: 'runtime' })
+```
-// Query with Triple Intelligence
+## One query, three engines
+
+```javascript
const results = await brain.find({
- query: "JavaScript", // 🔍 Vector
- where: { category: "language" }, // 📊 Document
- connected: { from: nodeId, depth: 1 } // 🕸️ Graph
+ query: 'modern frontend frameworks', // vector — what it means
+ where: { year: { greaterThan: 2015 } }, // metadata — what it is
+ connected: { to: react, depth: 2 } // graph — what it touches
})
```
-**Done.** No configuration. No complexity. Production-ready from day one.
+Every clause is optional; any combination composes. Under the hood Brainy plans the query across an HNSW vector index, a roaring-bitmap field index, and an adjacency graph index — and re-validates every result against your predicate before returning it, so a corrupt index can never hand you a wrong answer.
-**→ Ready to dive deeper? [Complete API Documentation](docs/api/README.md)** has every method with examples.
+## Feature tour
----
+### The database is a value
-## Entity Extraction (NEW in v5.7.6)
-
-**Extract entities from text with AI-powered classification:**
+Pin it, rewind it, fork it. Snapshot isolation without a server.
```javascript
-import { Brainy, NounType } from '@soulcraft/brainy'
+const db = brain.now() // pin current state — O(1)
-const brain = new Brainy()
-await brain.init()
+await brain.transact([ // atomic all-or-nothing, CAS-guarded
+ { op: 'update', id: order, metadata: { status: 'paid' } },
+ { op: 'relate', from: invoice, to: order, type: VerbType.References, subtype: 'billing' }
+], { ifAtGeneration: db.generation })
-// Extract all entities
-const entities = await brain.extractEntities('John Smith founded Acme Corp in New York')
-// Returns:
-// [
-// { text: 'John Smith', type: NounType.Person, confidence: 0.95 },
-// { text: 'Acme Corp', type: NounType.Organization, confidence: 0.92 },
-// { text: 'New York', type: NounType.Location, confidence: 0.88 }
-// ]
+await db.get(order) // still 'pending' — pinned forever
+await brain.get(order) // 'paid' — live
-// Extract with filters
-const people = await brain.extractEntities(resume, {
- types: [NounType.Person],
- confidence: 0.8
-})
-
-// Advanced: Direct access to extractors
-import { SmartExtractor } from '@soulcraft/brainy'
-
-const extractor = new SmartExtractor(brain, { minConfidence: 0.7 })
-const result = await extractor.extract('CEO', {
- formatContext: { format: 'excel', columnHeader: 'Title' }
-})
+const lastWeek = await brain.asOf(Date.now() - 7 * 86_400_000) // full query surface, past state
+const whatIf = await db.with([{ op: 'remove', id: order }]) // speculative — never touches disk
+await brain.now().persist('/backups/today') // instant hard-link snapshot
```
-**Features:**
-- 🎯 **4-Signal Ensemble** - ExactMatch (40%) + Embedding (35%) + Pattern (20%) + Context (5%)
-- 📊 **Format Intelligence** - Adapts to Excel, CSV, PDF, YAML, DOCX, JSON, Markdown
-- ⚡ **Fast** - ~15-20ms per extraction with LRU caching
-- 🌍 **42 Types** - Person, Organization, Location, Document, and 38 more
+**[Consistency model](docs/concepts/consistency-model.md)** · **[Snapshots & time travel](docs/guides/snapshots-and-time-travel.md)**
-**→ [Neural Extraction Guide](docs/neural-extraction.md)** | **[Import Preview Mode](docs/neural-extraction.md#import-preview-mode)**
+### Local embeddings — no API keys
----
-
-## From Prototype to Planet Scale
-
-**The same API. Zero rewrites. Any scale.**
-
-### 👤 Individual Developer → Weekend Prototype
-```javascript
-const brain = new Brainy() // Zero config, starts in memory
-await brain.init()
-```
-**Perfect for:** Hackathons, side projects, prototyping, learning
-
-### 👥 Small Team → Production MVP
-```javascript
-const brain = new Brainy({
- storage: { type: 'filesystem', path: './data', compression: true }
-})
-```
-**Scale:** Thousands to hundreds of thousands • **Performance:** <5ms queries
-**→ [Production Service Architecture](docs/PRODUCTION_SERVICE_ARCHITECTURE.md)** — Singleton patterns, caching, and scaling for Bun/Node.js services
-
-### 🏢 Growing Company → Multi-Million Scale
-```javascript
-const brain = new Brainy({
- storage: { type: 's3', s3Storage: { bucketName: 'my-kb', region: 'us-east-1' } },
- hnsw: { typeAware: true } // 87% memory reduction
-})
-```
-**Scale:** Millions of entities • **Performance:** <10ms queries, 12GB @ 10M entities
-
-### 🌍 Enterprise → Billion+ Scale
-```javascript
-const brain = new Brainy({
- storage: { type: 'gcs', gcsStorage: { bucketName: 'global-kb' } },
- hnsw: { typeAware: true, M: 32, efConstruction: 400 }
-})
-```
-**Scale:** Billions (tested @ 1B+) • **Performance:** 18ms queries, 50GB memory
-**Cost:** $138k/year → $6k/year with intelligent tiering (96% savings)
-
-**→ [Capacity Planning Guide](docs/operations/capacity-planning.md)** | **[Cost Optimization](docs/operations/)**
-
-### 🎯 The Point
-
-**Start simple. Scale infinitely. Never rewrite.**
-
-Most systems make you choose: Simple (SQLite) OR Scalable (Kubernetes + 7 databases).
-**Brainy gives you both.** Starts simple as SQLite. Scales like Google.
-
----
-
-## Why Brainy Is Revolutionary
-
-### 🧠 **Triple Intelligence™** — The Impossible Made Possible
-
-**The world's first to unify three database paradigms in ONE API:**
-
-| What You Get | Like Having | But Unified |
-|-------------|-------------|-------------|
-| 🔍 **Vector Search** | Pinecone, Weaviate | Find by meaning |
-| 🕸️ **Graph Relationships** | Neo4j, ArangoDB | Navigate connections |
-| 📊 **Document Filtering** | MongoDB, Elasticsearch | Query metadata |
-
-**Every other system makes you choose.** Brainy does all three together.
-
-**Why this matters:** Your data isn't just vectors or just documents or just graphs. It's all three at once. A research paper is semantically similar to other papers (vector), written by an author (graph), and published in 2023 (document). **Brainy is the only system that understands this.**
-
-### 🎯 **42 Noun Types × 127 Verb Types = Universal Protocol**
-
-Model **any domain** with mathematical completeness:
-
-```
-42 Nouns × 127 Verbs × ∞ Metadata = 5,334+ base combinations
-Stage 3 CANONICAL: 96-97% coverage of all human knowledge
-```
-
-**Real-world expressiveness:**
-- Healthcare: `Patient → diagnoses → Condition`
-- Finance: `Account → transfers → Transaction`
-- Manufacturing: `Product → assembles → Component`
-- Education: `Student → completes → Course`
-- **YOUR domain** → Your types + relationships = Your knowledge graph
-
-[→ See the Mathematical Proof](docs/architecture/noun-verb-taxonomy.md)
-
-### ⚡ **Zero Configuration Philosophy**
-
-**We hate configuration files. So we eliminated them.**
+Strings embed on-device with a bundled MiniLM model (WASM). Semantic search works offline, in CI, and on air-gapped machines, at zero cost per call. Hybrid keyword + semantic ranking is the default:
```javascript
-const brain = new Brainy() // Auto-detects everything
-await brain.init() // Optimizes for your environment
+await brain.find({ query: 'David Smith' }) // auto: text + semantic
+await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // semantic only
```
-Brainy automatically:
-- Detects optimal storage (memory/filesystem/cloud)
-- Configures memory based on available RAM
-- Optimizes for containers (Docker/K8s)
-- Tunes indexes for your data patterns
-- Manages embedding models and caching
+### A typed graph, not a bag of edges
-**You write business logic. Brainy handles infrastructure.**
-
-### 🚀 **Git-Style Version Control** — Database & Entity Level (v5.0.0+)
-
-**Clone your entire database in <100ms. Track every entity change. Full Git-style workflow.**
+42 entity types × 127 relationship types form a shared vocabulary for any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), yours. Your own taxonomy layers on with `subtype`, enforced at write time:
```javascript
-// Fork instantly - Snowflake-style copy-on-write
-const experiment = await brain.fork('test-migration')
+await brain.add({ data: 'Avery Brooks', type: NounType.Person, subtype: 'employee' })
-// Make changes safely in isolation
-await experiment.add({ type: 'user', data: { name: 'Test User' } })
-await experiment.updateAll({ /* migration logic */ })
-
-// Commit your work
-await experiment.commit({ message: 'Add test user', author: 'dev@example.com' })
-
-// Switch to experimental branch to make it active
-await brain.checkout('test-migration')
-
-// Time-travel: Query database at any past commit (read-only)
-const commits = await brain.getHistory({ limit: 10 })
-const snapshot = await brain.asOf(commits[5].id)
-const pastResults = await snapshot.find({ query: 'historical data' })
-await snapshot.close()
-
-// Entity versioning: Track changes to individual entities (v5.3.0+)
-const userId = await brain.add({ type: 'user', data: { name: 'Alice' } })
-await brain.versions.save(userId, { tag: 'v1.0', description: 'Initial profile' })
-
-await brain.update(userId, { data: { name: 'Alice Smith', role: 'admin' } })
-await brain.versions.save(userId, { tag: 'v2.0', description: 'Added role' })
-
-// Compare versions or restore previous state
-const diff = await brain.versions.compare(userId, 1, 2) // See what changed
-await brain.versions.restore(userId, 1) // Restore v1.0
+brain.counts.bySubtype(NounType.Person) // O(1) — { employee: 12, customer: 847 }
+brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
```
-**Database-level version control (v5.0.0):**
-- ✅ `fork()` - Instant clone in <100ms
-- ✅ `merge()` - Merge with conflict resolution
-- ✅ `commit()` - Snapshot state
-- ✅ `asOf()` - Time-travel queries (query at any commit)
-- ✅ `getHistory()` - View commit history
-- ✅ `checkout()`, `listBranches()` - Full branch management
-- ✅ CLI support for all features
+**[Type system](docs/architecture/noun-verb-taxonomy.md)** · **[Subtypes & facets](docs/guides/subtypes-and-facets.md)**
-**Entity-level version control (v5.3.0):**
-- ✅ `versions.save()` - Save entity snapshots with tags
-- ✅ `versions.restore()` - Restore previous versions
-- ✅ `versions.compare()` - Diff between versions
-- ✅ `versions.list()` - View version history
-- ✅ Automatic deduplication (content-addressable storage)
-
-**How it works:** Snowflake-style COW shares HNSW index structures, copying only modified nodes (10-20% memory overhead).
-
-**Perfect for:** Safe migrations, A/B testing, feature branches, distributed development, time-travel debugging, audit trails, document versioning, compliance tracking
-
-[→ See Full Documentation](docs/features/instant-fork.md)
-
----
-
-## What Can You Build?
-
-**If your app needs to remember, understand, or connect information — Brainy makes it trivial.**
-
-### 🤖 **AI Agents with Perfect Memory**
-Give your AI unlimited context that persists forever. Not just chat history — true understanding of relationships, evolution, and meaning over time.
-
-**Examples:** Personal assistants, code assistants, conversational AI, research agents
-
-### 📚 **Living Documentation & Knowledge Bases**
-Documentation that understands itself. Auto-links related concepts, detects outdated information, finds connections across your entire knowledge base.
-
-**Examples:** Internal wikis, research platforms, smart documentation, learning systems
-
-### 🔍 **Semantic Search at Any Scale**
-Find by meaning, not keywords. Search codebases, research papers, customer data, or media libraries with natural language.
-
-**Examples:** Code search, research platforms, content discovery, recommendation engines
-
-### 🏢 **Enterprise Knowledge Management**
-Corporate memory that never forgets. Track every customer interaction, product evolution, and business relationship.
-
-**Examples:** CRM systems, product catalogs, customer intelligence, institutional knowledge
-
-### 🎮 **Rich Interactive Experiences**
-NPCs that remember. Characters that persist across stories. Worlds that evolve based on real relationships.
-
-**Examples:** Game worlds, interactive fiction, educational platforms, creative tools
-
-### 🎨 **Content & Media Platforms**
-Every asset knows its relationships. Intelligent tagging, similarity-based discovery, and relationship-aware management.
-
-**Examples:** DAM systems, media libraries, writing assistants, content management
-
-**The pattern:** Knowledge that needs to live, connect, and evolve. That's what Brainy was built for.
-
----
-
-## Core Features
-
-### 🧠 **Natural Language Queries**
+### Graph analytics built in
```javascript
-// Ask naturally - Brainy understands
-await brain.find("recent React components with tests")
-await brain.find("JavaScript libraries similar to Vue")
-
-// Or use structured Triple Intelligence queries
-await brain.find({
- query: "React",
- where: { type: "library", year: { greaterThan: 2020 } },
- connected: { to: "JavaScript", depth: 2 }
-})
+await brain.graph.rank() // which entities matter most (centrality)
+await brain.graph.communities() // natural clusters
+await brain.graph.path(a, b) // how two things connect
+await brain.graph.subgraph([seed], { depth: 2 }) // bounded neighborhood → { nodes, edges }
+await brain.graph.export() // whole graph, one O(N+E) streaming pass
```
-**→ [See all query methods in API Reference](docs/api/README.md#search--query)**
+### Write-time aggregations
-### 🌐 **Virtual Filesystem** — Intelligent File Management
+`SUM` / `COUNT` / `AVG` / `MIN` / `MAX` with `GROUP BY` and time windows, maintained incrementally on every write — reads are O(1) lookups, not scans. **[Aggregation guide](docs/guides/aggregation.md)**
-Build file explorers and IDEs that never crash:
+### Import anything
```javascript
-const vfs = brain.vfs()
-
-// Tree-aware operations prevent infinite recursion
-const tree = await vfs.getTreeStructure('/projects', { maxDepth: 3 })
-
-// Semantic file search
-const reactFiles = await vfs.search('React components with hooks')
-```
-
-**[📖 VFS Quick Start →](docs/vfs/QUICK_START.md)** | **[Common Patterns →](docs/vfs/COMMON_PATTERNS.md)** | **[Neural Extraction →](docs/vfs/NEURAL_EXTRACTION.md)**
-
-### 🚀 **Import Anything** — CSV, Excel, PDF, URLs
-
-```javascript
-await brain.import('customers.csv') // Auto-detects everything
-await brain.import('sales-data.xlsx', { excelSheets: ['Q1', 'Q2'] })
-await brain.import('research-paper.pdf', { pdfExtractTables: true })
+await brain.import('customers.csv')
+await brain.import('sales.xlsx') // every sheet
+await brain.import('research-paper.pdf') // tables extracted
await brain.import('https://api.example.com/data.json')
```
-**[📖 Complete Import Guide →](docs/guides/import-anything.md)**
+Entities auto-classify on the way in; `brain.extractEntities(text)` exposes the same NER ensemble directly. **[Import guide](docs/guides/import-anything.md)**
-### 🧠 **Neural API** — Advanced Semantic Analysis
+### A filesystem that understands content
```javascript
-// Clustering, similarity, outlier detection, visualization
-const clusters = await brain.neural.clusters({ algorithm: 'kmeans' })
-const similarity = await brain.neural.similar('item1', 'item2')
-const outliers = await brain.neural.outliers(0.3)
-const vizData = await brain.neural.visualize({ maxNodes: 100 })
+await brain.vfs.writeFile('/docs/readme.md', 'Project documentation')
+await brain.vfs.search('React components with hooks') // semantic file search
```
----
+**[VFS quick start](docs/vfs/QUICK_START.md)**
-## Framework Integration
+### Operations-grade by default
-**Works with any modern framework.** React, Vue, Angular, Svelte, Solid.js — your choice.
+- **Single-writer, many-reader** — an exclusive lock protects the data directory; `Brainy.openReadOnly()` and the `brainy inspect` CLI examine a live brain from another process, safely.
+- **Self-upgrading data files** — a 7.x brain opens under 8.x and migrates itself behind an observable lock (`getIndexStatus().migration`), with an automatic pre-upgrade backup. No migration scripts.
+- **No silent wrong answers** — cold-open guards self-heal or throw typed errors (`MetadataIndexNotReadyError`, `GraphIndexNotReadyError`); they never return `[]` for data that exists.
-```javascript
-// React
-const [brain] = useState(() => new Brainy())
-useEffect(() => { brain.init() }, [])
+**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
-// Vue
-async mounted() { this.brain = await new Brainy().init() }
+## From laptop to hundreds of millions
-// Angular
-@Injectable() export class BrainyService { brain = new Brainy() }
-```
-
-**Supports:** All bundlers (Webpack, Vite, Rollup) • SSR/SSG • Edge runtimes • Browser/Node.js
-
-**[📖 Framework Integration Guide →](docs/guides/framework-integration.md)** | **[Next.js →](docs/guides/nextjs-integration.md)** | **[Vue →](docs/guides/vue-integration.md)**
-
----
-
-## Storage — From Memory to Planet-Scale
-
-### Development → Just Works
-```javascript
-const brain = new Brainy() // Memory storage, zero config
-```
-
-### Production → Persistence with Compression
-```javascript
-const brain = new Brainy({
- storage: { type: 'filesystem', path: './data', compression: true }
-})
-// 60-80% space savings with gzip
-```
-
-### Cloud → AWS, GCS, Azure, Cloudflare R2
-```javascript
-// AWS S3 / Cloudflare R2
-const brain = new Brainy({
- storage: {
- type: 's3',
- s3Storage: {
- bucketName: 'my-knowledge-base',
- region: 'us-east-1'
- }
- }
-})
-
-// Enable Intelligent-Tiering: 96% cost savings
-await brain.storage.enableIntelligentTiering('entities/', 'auto-tier')
-```
-
-**Cost optimization at scale:**
-
-| Scale | Standard | With Intelligent Tiering | Annual Savings |
-|-------|----------|--------------------------|----------------|
-| 5TB | $1,380 | $59 | $1,321 (96%) |
-| 50TB | $13,800 | $594 | $13,206 (96%) |
-| 500TB | $138,000 | $5,940 | $132,060 (96%) |
-
-**[📖 Cloud Storage Guide →](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[AWS Cost Optimization →](docs/operations/cost-optimization-aws-s3.md)** | **[GCS →](docs/operations/cost-optimization-gcs.md)** | **[Azure →](docs/operations/cost-optimization-azure.md)**
-
----
-
-## Production Features
-
-### 🎯 Type-Aware HNSW Indexing
-
-Efficient type-based organization for large-scale deployments:
-
-- **Type-based queries:** Faster via directory structure (measured at 1K-1M scale)
-- **Type count tracking:** 284 bytes (Uint32Array, measured)
-- **Billion-scale projections:** NOT tested at 1B entities (extrapolated from 1M)
-
-```javascript
-const brain = new Brainy({ hnsw: { typeAware: true } })
-```
-
-**[📖 How Type-Aware Indexing Works →](docs/architecture/data-storage-architecture.md)**
-
-### ⚡ Enterprise-Ready Operations (v4.0.0)
-
-- **Batch operations** with retry logic (1000x faster deletes)
-- **Gzip compression** (60-80% space savings)
-- **OPFS quota monitoring** (browser storage)
-- **Metadata/Vector separation** (billion-entity scalability)
-- **Circuit breakers & backpressure** (enterprise reliability)
-
-```javascript
-// Batch operations
-await brain.storage.batchDelete(keys, { maxRetries: 3 })
-
-// Monitor storage
-const status = await brain.storage.getStorageStatus()
-```
-
-### 📊 Adaptive Memory Management
-
-Auto-scales 2GB → 128GB+ based on environment:
-
-- Container-aware (Docker/K8s cgroups)
-- Environment-optimized (dev/staging/production)
-- Built-in cache monitoring with tuning recommendations
-
-```javascript
-const stats = brain.getCacheStats() // Performance insights
-```
-
-**[📖 Capacity Planning Guide →](docs/operations/capacity-planning.md)**
-
----
-
-## Benchmarks
-
-| Operation | Performance | Memory |
-|-----------|-------------|--------|
-| Initialize | 450ms | 24MB |
-| Add entity | 12ms | +0.1MB |
-| Vector search (1K) | 3ms | - |
-| Metadata filter (10K) | 0.8ms | - |
-| Bulk import (1K) | 2.3s | +8MB |
-| **10M entities** | **5.8ms** | **12GB** |
-| **1B entities** | **18ms** | **50GB** |
-
----
-
-## 🧠 Deep Dive: How Brainy Actually Works
-
-**Want to understand the magic under the hood?**
-
-### 🔍 Triple Intelligence & find() API
-Understand how vector search, graph relationships, and document filtering work together in one unified query:
-
-**[📖 Triple Intelligence Architecture →](docs/architecture/triple-intelligence.md)**
-**[📖 Natural Language Guide →](docs/guides/natural-language.md)**
-**[📖 API Reference: find() →](docs/api/README.md)**
-
-### 🗂️ Type-Aware Indexing & HNSW
-Learn about our indexing architecture with measured performance optimizations:
-
-**[📖 Data Storage Architecture →](docs/architecture/data-storage-architecture.md)**
-**[📖 Architecture Overview →](docs/architecture/overview.md)**
-
-### 📈 Scaling: Individual → Planet
-Understand how the same code scales from prototype to billions of entities:
-
-**[📖 Capacity Planning →](docs/operations/capacity-planning.md)**
-**[📖 Cloud Deployment Guide →](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)**
-
-### 🎯 The Universal Type System
-Explore the mathematical foundation: 42 nouns × 127 verbs = Stage 3 CANONICAL taxonomy:
-
-**[📖 Noun-Verb Taxonomy →](docs/architecture/noun-verb-taxonomy.md)**
-
----
-
-## CLI Tools
+Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**:
```bash
-npm install -g brainy
-
-brainy add "JavaScript is awesome" --metadata '{"type":"opinion"}'
-brainy find "awesome programming languages"
-brainy search "programming"
+npm install @soulcraft/cor
```
-47 commands available, including storage management, imports, and neural operations.
+```javascript
+const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } })
+await brain.init() // @soulcraft/cor detected — same code, native engines underneath
+```
----
+Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
-## 📖 Complete Documentation
+Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both.
-**For most developers:** Start with the **[Complete API Reference](docs/api/README.md)** ⭐
+## Performance
-This comprehensive guide includes:
-- ✅ Every method with parameters, returns, and examples
-- ✅ Quick start in 60 seconds
-- ✅ Core CRUD → Advanced features (branching, versioning, time-travel)
-- ✅ TypeScript types and patterns
-- ✅ 1,870 lines of copy-paste ready code
+- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
+- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
+- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
----
+## Use cases
-### 🎯 Essential Reading (Start Here)
+**AI agent memory** — persistent semantic recall with relationship tracking · **Knowledge bases** — auto-linking and meaning-aware navigation · **Semantic search** over codebases, documents, media · **Enterprise data** — CRM, catalogs, institutional memory · **Games & simulations** — worlds and characters that remember.
-1. **[📖 Complete API Reference](docs/api/README.md)** ⭐ **START HERE** ⭐
- - Your primary resource for building with Brainy
- - Every method documented with working examples
+## Documentation
-2. **[Filter & Query Syntax Guide](docs/FIND_SYSTEM.md)**
- - Complete reference for operators, compound filters, and optimization tips
-
-3. **[Natural Language Queries](docs/guides/natural-language.md)**
- - Master the `find()` method and Triple Intelligence queries
-
-4. **[v4.0.0 Migration Guide](docs/MIGRATION-V3-TO-V4.md)**
- - Upgrading from v3 (100% backward compatible)
-
-### 🧠 Core Concepts & Architecture
-
-- **[Triple Intelligence Architecture](docs/architecture/triple-intelligence.md)** — How vector + graph + document work together
-- **[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** — The universal type system (42 nouns × 127 verbs)
-- **[Transactions](docs/transactions.md)** — Atomic operations with automatic rollback
-- **[Architecture Overview](docs/architecture/overview.md)** — System design and components
-- **[Data Storage Architecture](docs/architecture/data-storage-architecture.md)** — Type-aware indexing and HNSW
-
-### ☁️ Production & Operations
-
-- **[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** — Deploy to AWS, GCS, Azure
-- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling to billions
-- **Cost Optimization:** **[AWS S3](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[Cloudflare R2](docs/operations/cost-optimization-cloudflare-r2.md)**
-
-### 🌐 Framework Integration
-
-- **[Framework Integration Guide](docs/guides/framework-integration.md)** — React, Vue, Angular, Svelte
-- **[Next.js Integration](docs/guides/nextjs-integration.md)**
-- **[Vue.js Integration](docs/guides/vue-integration.md)**
-
-### 🌳 Virtual Filesystem (VFS)
-
-- **[VFS Quick Start](docs/vfs/QUICK_START.md)** — Build file explorers that never crash
-- **[VFS Core Documentation](docs/vfs/VFS_CORE.md)**
-- **[Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md)** — AI-powered file navigation
-- **[Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md)**
-
-### 📦 Data Import & Processing
-
-- **[Import Anything Guide](docs/guides/import-anything.md)** — CSV, Excel, PDF, URLs with auto-detection
-
----
-
-## What's New in v4.0.0
-
-**Enterprise-scale cost optimization and performance improvements:**
-
-- 🎯 **96% cloud storage cost savings** with intelligent tiering (AWS, GCS, Azure)
-- ⚡ **1000x faster batch deletions** (533 entities/sec vs 0.5/sec)
-- 📦 **60-80% compression** with gzip (FileSystem storage)
-- 🔄 **Enhanced metadata/vector separation** for billion-scale deployments
-
-**[📖 Full v4.0.0 Changelog →](CHANGELOG.md)** | **[Migration Guide →](docs/MIGRATION-V3-TO-V4.md)** (100% backward compatible)
-
----
+| Start | Core | Going deeper |
+|---|---|---|
+| [Brainy explained simply](docs/eli5.md) | [API reference](docs/api/README.md) | [Architecture overview](docs/architecture/overview.md) |
+| [Installation](docs/guides/installation.md) | [Data model](docs/DATA_MODEL.md) | [Consistency model](docs/concepts/consistency-model.md) |
+| [Natural-language queries](docs/guides/natural-language.md) | [Query operators](docs/QUERY_OPERATORS.md) | [Multi-process model](docs/concepts/multi-process.md) |
+| | [Find system](docs/FIND_SYSTEM.md) | [Scaling](docs/SCALING.md) |
## Requirements
-**Bun 1.0+** (recommended) or **Node.js 22 LTS**
+**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
-```bash
-# Bun (recommended - best performance, single-binary deployment)
-bun install @soulcraft/brainy
+## Contributing & license
-# Node.js (fully supported)
-npm install @soulcraft/brainy
-```
-
-> **Why Bun?** Brainy's Candle WASM engine works seamlessly with `bun --compile` for standalone binary deployment. No external model files, no runtime downloads.
-
----
-
-## Why Brainy Exists
-
-**The Vision:** Traditional systems force you to choose between vector databases, graph databases, and document stores. You need all three, but combining them is complex and fragile.
-
-**Brainy solved the impossible:** One API. All three paradigms. Any scale.
-
-Like HTTP standardized web communication, **Brainy standardizes knowledge representation.** One protocol that any AI model understands. One system that scales from prototype to planet.
-
-**[📖 Read the Mathematical Proof →](docs/architecture/noun-verb-taxonomy.md)**
-
----
-
-## Enterprise & Support
-
-**🏢 Brain Cloud** — Managed Brainy with team sync, persistent memory, and enterprise connectors.
-Visit **[soulcraft.com](https://soulcraft.com)** for more information.
-
-**💖 Support Development:**
-- ⭐ Star us on **[GitHub](https://github.com/soulcraftlabs/brainy)**
-- 💝 Sponsor via **[GitHub Sponsors](https://github.com/sponsors/soulcraftlabs)**
-- 🐛 Report issues and contribute code
-- 📣 Share with your team and community
-
-**Brainy is 100% free and open source.** No paywalls, no premium tiers, no feature gates.
-
----
-
-## Contributing
-
-We welcome contributions! See **[CONTRIBUTING.md](CONTRIBUTING.md)** for guidelines.
-
----
-
-## License
-
-MIT © Brainy Contributors
-
----
-
-
- Built with ❤️ by the Brainy community
- The Knowledge Operating System
- From prototype to planet-scale • Zero configuration • Triple Intelligence™
-
+Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors.
diff --git a/RELEASES.md b/RELEASES.md
new file mode 100644
index 00000000..7799c6f6
--- /dev/null
+++ b/RELEASES.md
@@ -0,0 +1,2901 @@
+# @soulcraft/brainy — Release Notes for Consumers
+
+This file is the **quick reference for downstream sessions** tracking Brainy changes.
+Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/soulcraftlabs/brainy/releases
+
+**How to use:** Brainy is the underlying data engine for downstream applications. Read this when:
+- Upgrading `@soulcraft/brainy` in your application
+- Debugging data, query, or storage behaviour
+- A new Brainy feature is available that you want to adopt
+
+## Removed APIs — 7.x → 8.x (the complete ledger)
+
+Every public API removed at the 8.0 major, with its sanctioned replacement. If your code
+still calls a left-column name on 8.x it throws (or the config key is rejected) — the
+replacement is always a one-line change. (Standing contract from 8.9.0 forward: removals
+happen only at majors, after ≥1 minor of loud runtime deprecation naming the replacement.)
+
+| Removed (7.x) | Replacement (8.x) |
+|---|---|
+| `brain.search(query, k)` | `find({ query })` — semantic; `find({ query, searchMode })` for hybrid |
+| `brain.getRelations({...})` | `related(id, opts)` for adjacency; `find({ connected: {...} })` for scoped traversal |
+| `brain.neural()` clustering | `find({ vector })` + aggregation `GROUP BY` |
+| `Db.search()` | `db.find({ vector })` |
+| Pre-8.0 storage path aliases (`directory`, `basePath`, …) | one `storage.path` key (old aliases throw) |
+| Reserved keys inside `metadata` bags (silently remapped in 7.x) | top-level params (`subtype`, `visibility`, `confidence`, `weight`, …) — reserved-in-bag throws |
+| 7.x COW branches layout (`branches/main/`) | generational MVCC (`asOf()`, `now()`, `db.persist(path)`) — on-disk migration is automatic at first 8.x open |
+
+The fork/snapshot family (`brain.snapshot()`, `createSnapshot()`, `restoreSnapshot()`)
+is sometimes cited as a 7.x removal — those methods never existed on 7.x; the 8.0 Db API
+(`asOf`/`persist`/`restore({confirm})`) is their first real implementation.
+
+---
+
+## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
+
+The write path stops paying maintenance costs — the last structural piece of the
+flush-storm class (a production deployment measured single writes blocked 25–191s behind
+history reclaim running inline on flush under memory pressure):
+
+- **`flush()` never compacts history.** It persists the current window's deltas and
+ nothing else — its cost no longer depends on history backlog or retention mode, in any
+ configuration. **`close()` is the auto-compaction site** (time-bounded per pass, ~5s;
+ an early stop is a consistent prefix and the next pass resumes).
+- **`compactHistory()` gains `timeBudgetMs`** — bound your own maintenance windows; the
+ same resumable-prefix guarantee applies.
+- **The documented trade**: a long-lived writer that never closes accumulates history
+ until its next explicit `compactHistory()`. Predictable writes, explicit maintenance.
+ If you run bounded retention on an always-on service, schedule a periodic
+ `compactHistory({ ...caps, timeBudgetMs })` in your maintenance window.
+- **New public doc: `docs/performance-envelopes.md`** — measured per-op envelopes
+ (p50/p95 at stated scales, hardware, and backend, with the measuring script cited).
+ Refresh rule going forward: any release touching a measured path re-runs that op's
+ benchmark and updates the envelope in the same release.
+- **New in this file: the Removed APIs 7.x→8.x table** (top of this document) — every
+ removal with its sanctioned replacement, one place, per the engine-currency contract.
+ Standing from here: removals only at majors, after ≥1 minor of loud runtime deprecation.
+
+## v8.8.2 — 2026-07-19 (one field-resolution law: reserved-field aggregates stop drifting)
+
+Four fixes from a consumer conformance audit, all rooted in the same disease — two field-resolution
+regimes where there must be one:
+
+- **Aggregates grouped by a RESERVED field (`subtype`, `visibility`, …) now decrement on
+ delete.** The delete/update hooks fed the aggregation engine a partial entity view (type,
+ service, data, metadata only), so a reserved-field `groupBy` resolved to a nonexistent group
+ on the way DOWN — counts drifted upward forever after any delete, and updates that moved an
+ entity between reserved-field groups double-counted it. The hooks now pass the full-fidelity
+ entity view (every reserved field top-level, the same shape the add path uses). If your
+ deployment derives stats from reserved-field aggregates, re-define those aggregates once
+ after upgrading (a changed definition triggers one rescan) or run them fresh — the drifted
+ persisted counts do not self-heal retroactively.
+- **Aggregation `source.where` on reserved fields now filters** instead of silently matching
+ nothing: the matcher resolves fields through the same resolver `groupBy` uses (top-level
+ standard fields + custom metadata), so `where: { subtype: 'note' }` means what it says.
+- **`removeMany()` refuses empty/invalid selectors loudly.** A bare array passed positionally
+ (`removeMany([id])` instead of `removeMany({ ids: [id] })`), an empty params object, or
+ `ids: []` used to resolve successfully having deleted nothing. All three now throw.
+- **`find()` accepts both where-key spellings.** Metadata is flattened at index time
+ (`metadata.entry.title` indexes as `entry.title`); a `metadata.`-prefixed where key now
+ falls back to its flattened spelling when the prefixed one isn't indexed — the
+ "unindexed field(s), returning []" confusion for storage-shaped spellings is gone. (A
+ literal nested custom key named `metadata` still wins when indexed as spelled.)
+
+## v8.8.1 — 2026-07-18 (flush no longer walks the whole generation history + the import dedup off-switch is now honest)
+
+### The flush-storm fix (production incident, reported by a long-running deployment)
+
+Under the default adaptive retention, **every `flush()` re-walked the entire committed
+generation history** to compute total history bytes for the budget check — O(all
+generations) with disk re-reads past the 4,096-entry delta-cache bound. On a brain with
+70,000+ accumulated generations that turned every write into a full-tail scan (60-100s
+writes), even though the budget (free-RAM-based) never tripped and nothing was ever
+reclaimed. Fixed:
+
+- `historyBytes()` now maintains a **running total**: seeded by one walk on first use,
+ then updated incrementally at every commit and reclaim — the adaptive retention check
+ on every flush is O(1). Invariant regression-pinned (running total ≡ fresh walk through
+ both commit paths and compaction).
+- New **`brain.historyStats()`** (read-only, exported `HistoryStats`): generation count,
+ total on-disk bytes, generation/timestamp range, compaction horizon, retention mode,
+ and the effective adaptive budget — the one-call fleet-audit for sizing retention
+ exposure per brain.
+- Interim guidance for keep-everything deployments already affected: `retention: 'all'`
+ skips the adaptive accounting entirely (and is the correct policy if you never want
+ history reclaimed). The accumulated files are harmless at rest; this release removes
+ the per-write cost of their existence.
+
+### The import dedup off-switch (lifecycle honesty)
+
+The post-import background deduplication pass (a merge-DELETE writer that runs ~5 minutes
+after an import, merging entities judged duplicates by id / name / vector similarity) had
+three lifecycle defects, all fixed:
+
+- **`enableDeduplication: false` now actually disables it.** The background pass was
+ scheduled unconditionally — an import that explicitly opted out could still have
+ entities auto-removed 5 minutes later. The flag now gates BOTH the inline merge and
+ the background pass (regression-pinned).
+- **One deduplicator per brain, owned by the brain.** Each `import()` call constructed its
+ own coordinator + deduplicator, so the "debounced" timer never actually debounced across
+ imports (N imports = N delete timers). The brain now owns a single instance — the
+ debounce genuinely spans imports — and `close()` cancels pending work, so a delete pass
+ can never fire against a closed brain.
+- **The 5-minute timer is unref'd** — a pending pass no longer holds the process open
+ (the exit-hang class; this timer had escaped the earlier sweep).
+
+Retention note for keep-everything deployments: with `enableDeduplication: false` on
+import calls and `retention: 'all'` in config, no engine path removes records
+automatically.
+
+## v8.8.0 — 2026-07-17 (OS-limit detection for pool-scale deployments)
+
+Small minor: brains now detect the two OS limits that bite at pool scale and warn **before**
+the incident instead of during it.
+
+- At open (once per process, Linux-only, measurement-only), Brainy reads `RLIMIT_NOFILE`
+ (soft/hard, from `/proc/self/limits`) and `vm.max_map_count`, and warns loudly when either
+ sits below the pool-scale floors (soft NOFILE < 65 536; max_map_count < 262 144) — with the
+ exact raise commands (`ulimit -n` / `LimitNOFILE=` / `sysctl vm.max_map_count`). On stock
+ defaults the failure otherwise arrives as `EMFILE` or a failed mmap deep inside an index
+ open, long after the real cause stopped being visible. An unreadable limit produces **no**
+ warning — no measurement, no claim (non-Linux platforms stay silent).
+- Exported for ops doors: `checkOsLimits()` returns the full `OsLimitsReport`
+ (values + warnings) programmatically, with the floors exported as constants.
+
+## v8.7.1 — 2026-07-17 (writer-lock acquisition is race-proof + machine-readable through init)
+
+Two hardenings of the multi-process writer lock (the `locks/_writer.lock` lease that makes a
+second writer on the same brain directory fail loudly):
+
+- **Lock acquisition claims atomically.** The acquire path used read-then-write, leaving a
+ window where two processes racing an *absent* lock could both "succeed" — and the loser
+ kept running unlocked, silently. The claim is now an atomic create-exclusive write
+ (`O_EXCL`): exactly one racer wins; the loser re-evaluates and either fails loudly with
+ the winner's details or performs a verified stale-takeover. Bounded retries; contention
+ beyond them fails loudly rather than degrading into a lockless open.
+- **`BRAINY_WRITER_LOCKED` survives `init()`.** The conflict error documents a
+ machine-readable contract (`err.code`, `err.lockInfo` with the holder's pid/host/
+ heartbeat), but init's error wrapping silently stripped both, leaving consumers a message
+ to regex against. The error now passes through unwrapped.
+
+Measured while verifying (for operators sizing audits): `brain.auditGraph()` at a
+production-consumer scale of ~2,600 relationships / 800 entities costs ~0.1 s warm and
+~0.5 s cold, with exact scar counting across reopen.
+
+## v8.7.0 — 2026-07-17 (bulk-transact ergonomics: scaled budgets + timeout telemetry)
+
+The bulk-import ergonomics release, from a consumer's measured production incident (a serial
+import on network-attached storage at ~2 s/op met a flat 30 s transaction budget):
+
+- **The transact apply budget now scales with the batch** — `max(30 s, opCount × 2 s)` — or
+ is exactly what you pass as the new `TransactOptions.timeoutMs`. A flat 30 s cap silently
+ limited honest bulk work to ~15 operations on slow disks while looking generous for small
+ batches. Internal batch paths (e.g. `removeMany` chunks) get the same scaling.
+- **`TransactionTimeoutError` is a diagnosis, not just a failure**: it now reports the
+ operation it stopped at as `i/N` with the operation's name, elapsed vs budgeted time, and
+ states the batch rolled back atomically and is retryable. Its `context` carries the same
+ fields programmatically.
+- **The transact envelope is documented** — batch sizing, budget math, chunking with
+ `ifAbsent` idempotency, and the precompute pattern (`embedBatch` + per-op `vector`) that
+ keeps model inference out of the commit path. Guide: `docs/guides/optimistic-concurrency.md`.
+- Note: `brain.embed()` / `brain.embedBatch()` (the precompute APIs) already ship — public,
+ with native-provider passthrough, verified end-to-end (batch and single paths produce
+ bit-identical vectors; a vector-supplied `add` is fully searchable). Honest measurement:
+ on the default WASM engine, batch throughput ≈ sequential (~160 ms/text) — the precompute
+ win is keeping inference out of the budgeted commit path, not raw embedding speed.
+
+## v8.6.0 — 2026-07-17 (brain.auditGraph — the graph-truth verification instrument)
+
+A minor release adding one new public API, from the fleet's graph-trust program: a read-only
+audit that **proves whether relationship reads return stored truth** on a given brain.
+
+- **`brain.auditGraph(options?)`** walks every canonical relationship record, queries the same
+ read path applications use (`related()` / VFS `readdir`) with all visibility tiers included,
+ and classifies every discrepancy into its failure family: `missingFromReads` (records the
+ read path omits — a stale adjacency index), `danglingEndpoints` (relationships whose endpoint
+ entity no longer exists — the historical partial-delete scar class), and `readOnlyVerbIds`
+ (read-path edges with no stored record — ghosts). Design-hidden internal/system edges are
+ counted separately so intentional hiding is never misclassified as loss. Counts are exact;
+ example lists cap at `maxExamples` with an explicit `truncatedExamples` flag; the result is
+ narrated loudly on incoherence. Mutates nothing — safe on a live brain.
+- The operational pairing: audit → if incoherent, `repairIndex()` → audit again. A `coherent`
+ report after the repair is the verified all-clear. Run it after any engine upgrade, restore,
+ or migration. Guide: `docs/guides/inspection.md`.
+- Also: `getNounIds` pagination now refuses an undecodable resume cursor loudly (the same
+ contract `getNouns`/`getVerbs` gained in 8.5.2 — the third and final walk brought under it).
+- Types exported: `GraphAuditReport`, `GraphAuditDiscrepancy`.
+
+## v8.5.2 — 2026-07-17 (aggregation backfill: exception-safe, generation-verified, and loud)
+
+Hardening patch from a migration incident (a byte-copied store on new hardware; the service
+entered a silent full-CPU loop at boot). Four changes, all in the aggregation engine's
+backfill/adoption path:
+
+- **Backfill walks are exception-safe and non-destructive.** A rescan now builds into a
+ staging map and swaps in atomically on completion; a mid-walk failure drops the staging map,
+ keeps the previous live state serving, and surfaces the storage error to the failing query.
+ Previously the walk wiped live state *before* a scan that could throw, never cleared the
+ pending flag on failure, and re-ran a full walk on every subsequent query — a silent
+ wipe/walk/throw loop at the caller's retry rate.
+- **Failed walks are latched.** After a walk fails, retries within a 30-second cooldown rethrow
+ the recorded error instantly instead of re-walking — a tight caller-side retry loop now costs
+ one loud error per query, never a full store walk per query.
+- **Adoption is generation-verified.** Persisted aggregation state is stamped with the store's
+ committed generation at flush; reopen adoption requires the stamp to equal the current
+ watermark. Stale state (unclean shutdown) or over-counting state (a fact-log truncation on a
+ copied store pulled the watermark back) triggers exactly one loud rescan — never a silent
+ adopt. Pre-8.5.2 state on generation-aware stores rescans once after upgrade, then is stamped.
+- **The path narrates.** Adoption decisions, no-adoptable-state outcomes, walk start/finish
+ (entity count + duration), and walk failures all log by default; a non-advancing storage
+ pagination cursor aborts the walk loudly instead of looping forever.
+
+Plus three guards from a full audit of every loop in the open/init path:
+
+- **Invalid pagination cursors fail loudly.** A supplied-but-undecodable resume token to
+ `getNouns`/`getVerbs` used to silently restart the walk at offset 0 — to a `while(hasMore)`
+ caller that re-serves page 1 forever (an unbounded silent CPU loop). It now throws with a
+ clear message instead.
+- **The graph cold-load verb walk has a stall guard.** `hasMore=true` with a missing or
+ non-advancing cursor aborts loudly instead of re-reading the same page forever.
+- **A derived index AHEAD of the store is named at open.** Brainy already surfaced a provider
+ generation *behind* the committed watermark; the *ahead* direction (the signature of a
+ byte-copy of a live service, or a log truncation during crash recovery) now logs a loud
+ warning explaining what happened and that `brain.repairIndex()` forces a heal — instead of
+ passing unnamed into whatever the derived index does next.
+
+## v8.5.1 — 2026-07-17 (aggregation state survives restarts + the query-cap ratchet removed)
+
+Patch release from a production incident (aggregate/count paths taking 40–90 s on an idle box
+while vector search stayed fast, and every `find({ limit: 5000 })` suddenly failing against an
+"auto-configured query limit of 1000"). Three fixes, one cosmetic:
+
+- **Aggregation state is actually adopted on reopen.** The boot pattern `defineAggregate()` →
+ query raced the engine's async state load: the synchronous define always won, flagged a
+ backfill, and the first query then wiped the just-loaded persisted state and re-walked the
+ entire store — every restart, forever. Reopening with an unchanged definition now adopts the
+ persisted state directly (zero scans); a backfill runs only on a real definition change, a
+ missing/failed state load, or a write that landed before adoption (exactness wins). Apps that
+ rely on persisted definitions without re-defining at boot also no longer race a spurious
+ "Aggregate not defined".
+- **Backfills are single-flight and batched.** Concurrent queries on a cold aggregate used to
+ each wipe the others' partial state and start their own full store walk — under steady query
+ arrival the store never converged (the 40–90 s loop). Now all concurrent queries share one
+ walk, and one walk fills every aggregate pending backfill (M aggregates ≠ M scans).
+- **The query-cap "learning" ratchet is removed.** `maxLimit` is a memory-protection bound, but
+ a hidden tuner shrank it 20 % per recorded query while the lifetime-average query time
+ exceeded 1 s — down to a floor of 1000, below the documented 10 000 auto floor, with no
+ practical recovery, and the resulting error blamed "available free memory" (stale basis
+ label). The cap now comes from its construction-time basis (or your explicit
+ `maxQueryLimit` / `reservedQueryMemory`) alone and never changes at runtime; query timing is
+ recorded for diagnostics only.
+- **Cosmetic:** the engine's own persistence keys (`__aggregation_*`, `brainy:entityIdMapper`)
+ no longer log `[Storage] Unknown key format` at boot — they were always routed correctly;
+ they're now recognized before the warning fires.
+
+Operationally: if a host was bitten, upgrading and restarting is the whole fix — no repair
+ritual needed. Setting `maxQueryLimit` explicitly remains the valve that bypasses auto-detection
+entirely.
+
+## v8.5.0 — 2026-07-15 (provider access to the fact log + the shared stamp verifier)
+
+Small additive follow-up to 8.4.0, from the native accelerator's first consumption pass:
+
+- **Index providers can now reach the fact log through the storage adapter** — new optional
+ capability `storage.scanFacts()` / `storage.factLogHeadGeneration()` / `storage.factSegmentPaths()`,
+ wired by the host brain at init as a closure over its live log. Providers hold only `storage` and
+ must never construct their own fact-log reader (the log's open path is writer-side); `null` means
+ "no fact log here — use the enumeration walk."
+- **The family-stamp verifier is shared** via `@soulcraft/brainy/internals`
+ (`readFamilyStamp` / `writeFamilyStamp` / `verifyFamilyStamp` + types) so first-party native
+ providers run literally the same verification function, never a synchronized copy.
+- **Rollup stamp invariants accept strings** (`number | string`) — content fingerprints such as a
+ per-tree SHA-256 are valid invariant values; a type mismatch reads as incoherence, never a pass.
+- **`storage.committedGeneration()`** — the committed watermark as a capability, so a provider
+ compares its stamp's `sourceGeneration` against the store's truth without parsing the private
+ manifest format.
+- **Two durability/stability contracts pinned in the suite:** fsync-before-ack (holds for
+ `transact()` today; the single-op path is pinned as the documented future target — group commit
+ becomes latency batching, never durability skipping) and scan-stability-under-rotation (a scan
+ snapshot yields exactly its facts — no gaps, duplicates, or bleed-in — while segments rotate
+ beneath it).
+
+No behavior change for applications; all additions.
+
+## v8.4.0 — 2026-07-15 (the generation fact log — a sequential, self-verifying commit stream)
+
+A minor release, fully backward-compatible (all additions; no behavior change for existing APIs).
+This is infrastructure: it changes nothing about how you query today, and lays the substrate that
+makes index heals and incremental catch-up sequential-read problems instead of directory walks.
+
+- **Every committed write now also appends a "fact" — an after-image commit record.** Alongside the
+ existing before-image history, each committed generation (single-op and `transact()` alike) appends
+ what each touched entity/relationship *became* — or a body-less tombstone for a removal — to an
+ append-only, checksummed segment log under `_generations/facts/`. Crash-safe by construction: a
+ torn tail is detected and ignored; on open the log is reconciled to committed truth, so an absent
+ generation always means "never committed." `transact()` facts are durable when `transact()`
+ returns; single-op facts ride the same group-commit flush as their history.
+
+- **New: `brain.scanFacts()`** — stream committed facts in commit order, in batches, with heal-grade
+ telemetry (total scope up front; per-batch generation range, byte size, and segment; a summary
+ cross-check at the end; loud abort on any gap — never a silent skip). **`brain.factSegmentPaths()`**
+ hands zero-copy consumers the immutable sealed segment files directly. New exported types:
+ `CommitFact`, `FactOp`, `FactScanBatch`, `FactScanHandle`. Facts accumulate from the first write
+ after upgrading — pre-existing history is not retroactively converted (enumeration remains the
+ fallback for old data).
+
+- **New: the entity-tree family stamp.** At every flush/close, brainy stamps which committed
+ generation the canonical entity files reflect plus the rollup invariants (entity/relationship
+ counts) that verify the tree whole. At open, coherence is a comparison — a genuine divergence is
+ loud and names the failing invariant; `repairIndex()` recounts from canonical and re-stamps. New
+ exports: `readFamilyStamp`, `verifyFamilyStamp`, `ENTITY_TREE_STAMP_PATH`, `FamilyStamp` types.
+
+- **Storage adapters** gain optional binary raw-byte primitives (`appendRawBytes`, `readRawBytes`,
+ `writeRawBytes`, `rawByteSize`) — feature-detected; the filesystem and memory adapters implement
+ them; an adapter without them simply hosts no fact log. The fact-log namespace is registered as a
+ protected family: no sweeper or GC can delete under it.
+
+No API breaks. 24 new tests; the full commit-path regression suite is green.
+
+## v8.3.3 — 2026-07-15 (rename moves the containment edge — no ghost in the old directory)
+
+One production-reported fix plus a repair path, completing the delete/move hygiene arc (8.3.1 fixed
+deletes, 8.3.2 fixed counters, this fixes moves).
+
+- **A cross-directory `vfs.rename()` now MOVES the containment edge instead of accumulating one per
+ parent.** The old parent's `Contains` edge was never removed on a move, leaving the entity a child
+ of **both** directories: `readdir(oldDir)` kept listing it after the move, re-creating the old path
+ showed the same name twice, and any tree-walking consumer (sync engines, file browsers) saw the
+ file in two places. The old edge is now removed by edge id, resolved from the graph's own adjacency
+ — a removal never requires reading the thing being removed. Bonus fix in the same seam: a move **to
+ the root** now gets its containment edge (it was previously skipped, orphaning the file out of
+ `readdir('/')`).
+
+- **`repairIndex()` now also reconciles VFS containment** (new `vfs.repairContainment()`): every VFS
+ entity's containment edges are checked against its canonical `metadata.path` — stale old-parent
+ ghosts and duplicate edges are removed, a missing expected edge is restored, and user
+ knowledge-graph edges are never touched (only `vfs-contains` edges are candidates). Loud per
+ repair. Stores that performed cross-directory renames under ≤8.3.2 should run `brain.repairIndex()`
+ once after upgrading — the same single ritual now heals orphan directories, counters, **and**
+ containment edges.
+
+- Also ships a permanent lens-consistency regression suite (combined type+subtype vs subtype-only vs
+ canonical ground truth, id-for-id, warm and after a cold reopen), ported from the field
+ investigation that closed the historical lens-drop report.
+
+No API changes beyond the new optional `vfs.repairContainment()` (also invoked by `repairIndex()`).
+
+## v8.3.2 — 2026-07-14 (honest counters — the recount + removal-without-re-reading)
+
+Completes 8.3.1's delete-hygiene story at the counter layer, from a production proof chain reported
+by a downstream deployment: persisted entity totals were permanently **inflated** — deletes whose
+count decrement was silently skipped — and because paginated `totalCount` serves
+`Math.max(persistedTotal, scanned)`, the inflated number always won and **no disk cleanup could ever
+lower it**.
+
+- **A removal's count decrement no longer requires re-reading the record being removed.** The
+ decrement was sourced from re-reading the entity's metadata inside the delete; if that read
+ returned `null` (a replace race, or a ghost left by a pre-8.3.1 partial delete) the decrement was
+ silently skipped while the paired add had counted — minting drift on every write→delete→re-create
+ cycle. The caller's pre-delete read now rides through the whole delete path
+ (`remove()`/`removeMany()` → the delete operation → `deleteNoun`/`deleteVerb` →
+ `deleteNounMetadata`/`deleteVerbMetadata`, both sides symmetric): a null internal read falls back
+ to the known prior record instead of skipping. The `StorageAdapter` signatures gain an optional
+ `priorMetadata` parameter (additive; existing adapters unaffected).
+
+- **`repairIndex()` is the sanctioned counter recount — unconditional, and it actually persists.**
+ `rebuildTypeCounts()` previously rebuilt only the type-statistics arrays and computed the total
+ *just to log it* — the persisted scalar (`counts.json`) survived every "rebuild" untouched, so an
+ already-inflated brain could never be corrected. One canonical walk now rebuilds **every** counter
+ rollup — scalar totals, per-type maps, and type statistics — and persists them, and `repairIndex()`
+ runs it unconditionally (not only when orphan directories are found: counters can be inflated over
+ perfectly clean shelves). Brains with delete history should run `brain.repairIndex()` once after
+ upgrading; the correction survives reopen.
+
+No API breaks (optional-parameter additions only). Regression tests cover the drift cycle, the
+null-read decrement fallback, and the persisted recount across reopen.
+
+## v8.3.1 — 2026-07-14 (full-removal deletes + family-scoped migration gate)
+
+Two production-reported fixes in the write/index spine, plus an operator repair path. No API changes;
+all behavior changes make previously-wrong states honest.
+
+- **Deleting an entity now removes it completely — no more "ghost" leftovers on disk.** A canonical
+ noun delete removed the metadata (content) leg but left the entity's `vectors.json` and its `/`
+ directory behind. Consequences observed in a production deployment: deleted rows were
+ indistinguishable on disk from damage scars, enumerated counts inflated monotonically with every
+ delete (the leftovers were counted forever), and locator-style reads hit unreadable ghost rows.
+ `remove()`/`removeMany()`/`deleteNoun`/`deleteVerb` now remove **both legs and the entity
+ container**, with a full two-leg before-image rollback inside the transaction. The generation log
+ still holds the delete's before-image, so `asOf()` time-travel reconstructs deleted entities exactly
+ as before — this is live-HEAD hygiene, not a history change. This also fixes **duplicate `readdir`
+ entries for re-created VFS paths** at the root: with no ghost state, a delete always unposts its
+ index rows, so a delete→recreate cycle lists the path exactly once (regression-tested across
+ repeated cycles).
+
+- **`repairIndex()` prunes ghost/scar directories left by earlier versions.** Stores that deleted
+ entities under ≤8.3.0 may hold orphaned entity directories (a vector-only leg, or an empty dir).
+ `brain.repairIndex()` now sweeps them: it removes only containers with **no metadata content leg**
+ (never a directory that still holds content), logs every removal, and recomputes type/subtype
+ counts afterward so totals stop counting ghosts.
+
+- **Reads no longer hang behind an unrelated index migration (family-scoped gate).** During a native
+ provider's one-time background migration, *every* read — including plain `get()`, VFS
+ `readdir`/`readFile`, and metadata-only `find({ where })` — blocked on the whole-brain migration
+ lock until timeout, even when the migrating index was irrelevant to the read. The gate is now
+ scoped to the index families a read actually consults: canonical reads (`get`, `batchGet`, VFS
+ content) never wait; a `find` waits only on the families its query shape needs (vector for
+ semantic, metadata for `where`/type, graph for `connected`); graph traversals wait only on the
+ graph family. Writes and unclassified operations keep the conservative whole-brain wait. A read
+ that *does* need the migrating family still blocks (bounded by `migrationWaitTimeoutMs`) and
+ surfaces the retryable `MigrationInProgressError` — never a partial result.
+
+No breaking API change. Each fix ships with regression tests.
+
+## v8.3.0 — 2026-07-13 (faster index heals + the cross-layer integrity contract)
+
+Three additive changes. The first is an immediate, standalone performance win; the other two are the
+brainy side of the write/index-spine integrity contract, inert until a native accelerator
+that implements the matching hooks is present — so this release changes nothing for a JS-only brain
+beyond the speedup.
+
+- **Canonical enumeration is up to ~16× faster — the dominant term in an index heal.** The paginated
+ entity walk (`getNounsWithPagination`) hydrated each entity's vector + metadata one-at-a-time; since
+ every index rebuild enumerates canonical storage, that serial per-item latency dominated multi-minute
+ heals. Hydration is now 16-way bounded-concurrency, with the pagination contract (order, cursor
+ resume, filters, totalCount) byte-identical to before. New **`getNounIdsWithPagination()`** returns
+ ids without hydrating anything (zero per-entity reads when unfiltered) for callers that own their own
+ IO schedule.
+
+- **Cross-layer integrity — `validateIndexConsistency()` is no longer blind to native providers.** It
+ only ran the JS metadata index's own check, so a native provider whose manifest/segments/counts had
+ diverged still read as "healthy". It now feature-detects and aggregates each provider's optional
+ `validateInvariants()` self-report, names every failing invariant with its numbers, and exposes the
+ per-provider reports. `repairIndex()` now reconciles native derived state from canonical too
+ (rebuilding any provider whose failing invariant asks for it). New exported types
+ `ProviderInvariantReport` / `InvariantResult` / `InvariantHeal`.
+
+- **Registered-blob families — declared index files are undeletable through the storage layer.** A
+ provider can declare a derived-index blob *family* (a set of members that are load-bearing together);
+ once declared, `deleteBinaryBlob` / `removeRawPrefix` refuse to remove a member (new exported
+ `ProtectedArtifactError`), so a stray in-process sweeper cannot delete a load-bearing index file. The
+ declaration persists across reopen; `checkDerivedFamiliesPresent()` names any member missing on open.
+ New optional `StorageAdapter` surface (`registerDerivedFamily` / `unregisterDerivedFamily` /
+ `listDerivedFamilies` + `DerivedFamilyDeclaration`) and exported `DerivedArtifactMissingError`.
+
+No breaking API change (all additions are optional/new). Each change ships with regression tests.
+
+## v8.2.8 — 2026-07-13 (honest index readiness — no more silently-empty queries on a cold index)
+
+Closes the last of the three spine anti-patterns: the "dishonest readiness proxy," where `size() > 0`
+was treated as "this index actually serves queries." On a cold open (fresh boot, restart, crash
+recovery) a native index can load its **count** before its **serving structure** — so for a brief
+window it has data but cannot answer, and a query returned a silent empty result indistinguishable
+from "no such data." Every fix here asks the index whether it can *actually serve*, and if not,
+self-heals or fails loudly instead of returning `[]`.
+
+- **Semantic search no longer returns a silent `[]` on a cold vector index.** A pure semantic
+ `find({ query })` has no filter, so nothing previously guarded the vector index. A new one-shot
+ guard verifies the vector index serves a known persisted vector on the first semantic/proximity
+ search — preferring the provider's honest `isReady()` signal, else a known-vector self-match probe.
+ It rebuilds from canonical records if the serving structure did not load, and throws the new
+ **`VectorIndexNotReadyError`** only if a rebuild still cannot serve — never a silent empty result.
+
+- **Relationship reads fall back to the canonical scan instead of an empty result on a cold graph
+ index.** `getVerbsBySource`/`getVerbsByTarget` (used by relationship queries and virtual-filesystem
+ traversal) skipped the fast path only on `isInitialized` — which reads true once the manifest loaded
+ even if the source→target adjacency did not. They now consult the honest readiness signal and, when
+ the adjacency is not serving, take the correct-but-slower canonical shard scan. A one-shot self-heal
+ probe covers providers that expose no readiness signal.
+
+- **`getIndexStatus()` tells the truth for readiness probes.** It reported `populated: size>0` only, so
+ a Kubernetes readiness check could route traffic to a brain still warming up. It now folds in the
+ honest per-index `ready` signal (making `populated` honest), plus the degraded states already
+ surfaced by `checkHealth()`/`validateIndexConsistency()` (`rebuildFailed`/`rebuildError` and a
+ `degradedIds` count) — so a probe never reports 200-ready over a known-degraded index.
+
+This completes the write/index-spine hardening end to end. New export: **`VectorIndexNotReadyError`**;
+`getIndexStatus()` gains additive fields (`rebuildFailed`, `rebuildError?`, `degradedIds`, per-index
+`ready?`). No breaking API change. Each fix ships with a dedicated regression test.
+
+## v8.2.7 — 2026-07-13 (loadBinaryBlob fault-propagation — the lockstep completion of 8.2.6)
+
+Completes the Pass-1 spine hardening. `loadBinaryBlob` (the raw-blob read a native accelerator mmaps
+for its index files) previously returned `null` on ANY read error, so a real IO fault
+(EIO/EACCES/EMFILE) on a present-but-unreadable index blob masqueraded as "the blob is absent" —
+driving a needless full rebuild or an empty read. It now distinguishes genuine absence (ENOENT →
+`null`, the documented contract) from a real fault (→ throw), so a transient disk fault surfaces
+loudly instead of silently degrading the index.
+
+This one change was deliberately held out of 8.2.6 and ships now, in lockstep with the native
+accelerator release that hardened its two column-store read sites to handle the throw (a faulted
+segment read marks the field unavailable and throws a named error, instead of relying on
+null-on-error). A consumer on new brainy + an older accelerator was never at risk: 8.2.6 kept the
+prior swallow-on-fault behavior for this method until the accelerator was ready.
+
+No API change. Regression: `tests/unit/storage/blob-save-durability.test.ts` gains the loadBinaryBlob
+leg (absent → null; present → bytes; real fault → throws).
+
+## v8.2.6 — 2026-07-13 (write/index-spine hardening — loud errors, never quiet losses)
+
+Durability + integrity hardening across the write and index paths. Every fix converts a place that
+could *silently* lose data, serve a partial result, or acknowledge a write that did not land into a
+loud, observable failure. No breaking API changes; one new exported error.
+
+- **A durable blob write that stored nothing is no longer acknowledged.** The atomic blob write
+ (`saveBinaryBlob`, used for vector/graph index segments and the native index files) uses a unique
+ per-writer temp file, so a rename `ENOENT` can only mean *our* temp vanished before the rename —
+ the bytes never landed. It previously returned success on that path; it now retries once with a
+ fresh temp and, if the temp vanishes again, throws instead of acknowledging a write that persisted
+ nothing. A downstream deployment that mmaps these files no longer finds a "successfully written"
+ blob missing on the next open.
+
+- **Single-op history durability is enforced, not assumed.** The asynchronous group-commit flush
+ that persists single-op generation history previously swallowed a persist failure as a warning
+ while writes kept succeeding and their history piled up, undurable, in memory. It now tolerates a
+ transient blip (retry with capped backoff) and, after repeated failures, **refuses further writes**
+ with the new **`PendingFlushDurabilityError`** rather than promise a durability it cannot deliver.
+ Live canonical data is untouched; the latch self-heals the moment a flush succeeds. Callers needing
+ hard per-write durability should keep using `transact()` (or `flush()` after a single-op).
+
+- **A degraded derived index is surfaced on reads, not served silently.** When a non-fatal index
+ rebuild fails at open, or a write commits via adopt-forward recovery with an incomplete derived
+ index, `find()` and `get()` now emit one loud warning per degraded window (reads still return —
+ canonical is the source of truth), the state folds into `checkHealth()` /
+ `validateIndexConsistency()`, and `repairIndex()` reconciles and clears it.
+
+- **`clear()` removes the full derived footprint.** It previously left raw index blobs, the native
+ id-mapper, and column-index manifests on disk, so a cleared brain could re-read stale native state
+ on reopen. It now wipes them together as a set.
+
+- **Counts stay honest across deletes.** A delete decremented the per-type breakdown but not the
+ scalar total, so the total inflated permanently (and won pagination). Delete now decrements both;
+ the invariant `total === Σ per-type` holds across any interleaving of add / visibility-flip /
+ delete and across a reopen.
+
+- **Index-maintenance and aggregation failures are loud.** A partial LSM/segment load no longer
+ publishes the manifest's full count as if healthy; an HNSW flush that can't persist a node throws
+ (`HnswFlushError`) instead of returning a lying count and dropping the node; a corrupt or missing
+ manifest-listed column segment throws (`ColumnSegmentLoadError`) instead of silently dropping its
+ entities from every query; and aggregation materialization / state-load failures now warn instead
+ of vanishing into an empty catch.
+
+New export: **`PendingFlushDurabilityError`** (with `.cause` and `.failedAttempts`). No other public
+API change. Each fix ships with a dedicated regression test.
+
+## v8.2.5 — 2026-07-12 (honest response when a transaction rollback can't complete)
+
+Data-integrity fix. When a transaction failed and its rollback then *also* failed to undo a
+canonical write (retries exhausted), the old behavior logged, continued, and threw
+`TransactionRollbackError` — while the record it couldn't undo stayed durable on disk, and the
+transaction even reported its state as `'rolled_back'`. The caller got an error implying the write
+was undone; a read-back showed the record. A failed rollback had no truthful response.
+
+Rollback now tells the truth, with a two-branch contract:
+
+- **Adopt-forward (safe case).** A single-op write (`add`/`update`/…) whose only damage is a
+ durably-present record — the write the caller asked for — is **adopted**: its generation is
+ committed, the write returns success, and a loud warning records that the derived index may be
+ incomplete for that id until the next rebuild/`repairIndex()`. No error, no double-write; the
+ record is immediately durable and retrievable by `get()`.
+- **Fail loud + quarantine (unsafe case).** A multi-operation batch, or *any* case where a record
+ was lost (a remove/update whose restore-undo failed), throws the new **`StoreInconsistentError`**
+ naming every unreconciled record and its disposition (`orphan` vs `loss`), and puts the brain into
+ **write-quarantine**: reads keep working, but writes are refused until `repairIndex()` reconciles
+ the derived indexes against canonical storage and lifts the quarantine. The generation counter is
+ not advanced, and a transaction whose rollback failed is now `'inconsistent'`, never the
+ `'rolled_back'` lie.
+
+The decision is made by *observation*, not guesswork: both commit paths already hold byte-identical
+before-images, so after a failed rollback the store compares current canonical state to them to
+classify exactly which records are orphaned or lost.
+
+New export: `StoreInconsistentError` (with `.records` and `.cause`) and the `UnreconciledRecord` type.
+No other API change; `repairIndex()` gains the quarantine-lift behavior. Regression
+(`tests/integration/rollback-trapdoor.test.ts`) injects the exact failure (index add throws →
+canonical delete-undo fails) and pins adopt-forward (durable, get-able, not quarantined), fail-loud
+(`StoreInconsistentError` + quarantine + reads work + `repairIndex()` lifts it), and the error's
+record naming.
+
+## v8.2.4 — 2026-07-12 (restore can no longer destroy the store it's recovering)
+
+Recovery-safety fix. `restore()` removed the entire live brain directory and THEN copied the
+snapshot in — so any copy failure left the store destroyed with only a partial copy. The sharpest
+edge: the copy (`fs.cp`) materialized the holes of sparse mmap blob files, so a snapshot that fits
+on disk could balloon and `ENOSPC` mid-copy, and the recovery tool would have just destroyed the
+brain it was asked to recover. Reported from a downstream incident's recovery forensics.
+
+Restore is now **non-destructive and crash-resumable**:
+
+- The snapshot is copied into a staging area (`_restore_staging/`) **before any live data is
+ touched**. The copy is **sparse-aware** — all-zero regions are left as holes, so a store of
+ mostly-hole blobs restores at its true allocated size instead of its apparent size.
+- A copy failure (including `ENOSPC`) removes only the half-written staging area and throws; the
+ live store is left **exactly as it was**.
+- Only after the copy succeeds and a completion marker is `fsync`'d does an **atomic per-entry
+ swap** move the staged data into place — same-filesystem renames that cannot fail for disk space.
+- A crash mid-swap is finished **forward** on the next open: startup resumes a committed-but-
+ incomplete swap, or discards an uncommitted staging area (live data still authoritative).
+
+No API change — `restore(path, { confirm: true })` is unchanged. `persist()` was already safe
+(hard-link snapshot). Regression (`tests/integration/restore-nondestructive.test.ts`): a forced
+copy failure leaves live data fully intact, a normal restore round-trips, an interrupted-but-
+committed restore completes on reopen, an uncommitted staging area is discarded, and the sparse
+copy is byte-identical with allocation far below apparent size.
+
+## v8.2.3 — 2026-07-12 (a committed transaction is durable on return)
+
+Durability fix. A `transact()` reported "committed" while its canonical entity writes were still
+only in the OS page cache (written via tmp+rename, not yet `fsync`'d), even though the generation
+counter and manifest WERE fsync'd. A hard kill (power loss, SIGKILL) in that window could leave the
+durable generation counter **ahead of** the persisted entity bytes — so a consumer resuming from the
+counter would see "phantom progress": a generation that claims writes the disk never kept. Reported
+from a downstream migration's crash-lifecycle forensics.
+
+`commitTransaction` now runs a **durability barrier**: it records every canonical write and delete
+the batch's operations make, then `fsync`s that entire footprint (file contents, the rename
+directory entries, and the parent directories of any deletes) **before** advancing the generation
+counter and manifest. A committed transaction is therefore durable the moment `transact()` returns —
+the counter can never outrun the entity bytes.
+
+**Durability contract, now explicit.** `transact()` is durable-on-return (above). A **single-op**
+write (`add`/`update`/`remove`/`relate`/…) is Model-B group-commit: its live bytes are written but
+become durable at the next `flush()` or `close()`, not the instant the call resolves — the counter
+is buffered alongside the data, so a crash loses both together (never a torn counter-ahead-of-state
+store). Need per-write durability? Use `transact()` (even for one op), or `flush()` after the write.
+This deliberately trades single-op fsync latency (a 3-5x write regression) for throughput.
+
+No API change; no accelerator involvement (the barrier is in the filesystem storage adapter). In-memory
+and durable-per-call (cloud object-PUT) adapters treat the barrier as a no-op. Regression:
+`tests/integration/transact-durability-barrier.test.ts` proves entity writes fsync in an earlier
+batch than the manifest, for single-op and multi-op (add+relate) transactions, and that a
+precommit-rejected batch opens no barrier and advances nothing.
+
+## v8.2.2 — 2026-07-11 (P0: a timed-out transaction now rolls back — no torn state)
+
+Data-integrity fix. A transaction that exceeded its time budget **mid-flight** (e.g. a bulk
+`transact()` on slower hardware crossing the 30s ceiling) threw a timeout error WITHOUT rolling
+back the operations it had already applied. The budget check sat outside the per-operation
+rollback path, so only per-operation *failures* rolled back — a timeout stranded the partial
+writes in canonical storage while the generation was never stamped, leaving torn, generation-less
+state. This was caught by a downstream migration that crossed the ceiling on a large batch.
+
+`Transaction.execute()` now has a **single rollback point**: any error that escapes the operation
+loop — an operation failure OR a mid-flight timeout — rolls back every applied operation in
+reverse order, then surfaces the original error (a rollback failure supersedes it, loudly, as
+before). This restores the invariant the generation-store commit path already depended on: a throw
+from execute means the applied operations were undone byte-identically, and the aborted
+transaction leaves `generation()` unchanged and storage byte-identical to its pre-transaction
+state.
+
+No API change. Regression pins the reported requirement — after a mid-flight timeout,
+storage is byte-identical and the transaction ends in the `rolled_back` terminal state
+(`tests/unit/transaction/timeout-rollback.test.ts`), plus the operation-failure and
+rollback-failure paths through the same single rollback point.
+
+**Note on the 30s ceiling itself** (configurable/scaled timeout, batched embedding precompute,
+timeout telemetry) — that ergonomics work is tracked separately; this release fixes only the
+correctness bug (a timeout must never leave partial state), independent of where the ceiling sits.
+
+## v8.2.1 — 2026-07-11 (transact forward references work on the native accelerator)
+
+Parity fix. `transact()` has always promised atomic forward references — `add` an entity and
+`relate` to it in one batch — but the transaction planner resolved relationship endpoint ids at
+**plan** time, before the batch's adds had applied. Asking the id mapper about an entity that
+doesn't exist yet made the accelerator's (correctly strict) native mapper throw in
+`EntityIdMapper.getOrAssign`, so `transact([{op:'add', id:X}, {op:'relate', to:X}])` failed on
+native deployments while the permissive JS mapper masked the bug — and silently leaked an id
+assignment whenever a batch was later rejected by a commit precondition.
+
+Endpoint resolution is now **lazy** — evaluated when the graph operation executes inside the
+commit, after the batch's adds have applied (the same lazy pattern the operation's generation
+stamp already used). Fixed across all transact-planned graph operations: relate (including the
+bidirectional reverse edge), remove's relationship cascade, and unrelate. Single-operation writes
+are unchanged. Also fixed as a byproduct: a rejected batch no longer pollutes the id mapper.
+
+Regression suite covers the exact reported shape, both-endpoints-in-batch, bidirectional,
+add+relate+remove in one batch, and the mapper-cleanliness proof on rejected batches
+(`tests/integration/transact-forward-ref-graph.test.ts`). No API changes; no accelerator version
+pairing required.
+
+## v8.2.0 — 2026-07-10 (temporal VFS — file content joins the immutability model)
+
+Time travel now covers Virtual Filesystem **content**. Previously the temporal model had a hole
+exactly where files were concerned: every entity write was an immutable generation, but the
+content *bytes* lived under an eager reference-count GC — deleting a file could physically destroy
+bytes that in-window history still referenced, while overwriting never released the old content at
+all (an unbounded silent leak that only *accidentally* preserved history). A production consumer's
+recovery from a bad deploy succeeded only because a stale field happened to hold the good value —
+luck, not a guarantee. Both directions are now fixed by making blob reclamation a **history**
+decision instead of a **liveness** decision:
+
+- **Content blobs are retention-protected.** Each blob tracks live references AND history
+ references (one per persisted generation record that carries its hash). Deleting/overwriting a
+ file drops only the live reference; bytes are physically reclaimed in exactly one place —
+ history compaction — once no live reference and no retained generation references the hash.
+ Pinned views are exempt automatically (compaction already respects pins). Crash ordering is
+ over-count-only (never under-count), so a crash can leak until the built-in scrub recounts, but
+ can never reclaim bytes history still needs. Existing stores get a one-time, marker-gated
+ backfill on open; cross-file content dedup is handled exactly.
+- **`vfs.readFile(path, { asOf })`** — the file's exact bytes as of a generation or `Date`,
+ guaranteed present within the retention window.
+- **`vfs.history(path)`** — the file's versions (`{ generation, timestamp, hash, size,
+ mimeType? }`, ascending). Restore = read the old bytes `asOf` and write them back (a new write;
+ history is never rewritten).
+- **Overwrites now refresh the file entity's `data`/embedding text** — previously semantic search
+ and the `data` field served the FIRST version's text forever.
+
+Lifecycle note: deleting a file no longer frees its bytes immediately — old content lives until
+compaction reclaims its generations, under the same `retention` budget as all Model-B history
+(`retention: 'all'` keeps every version forever). Guide: the new "Time travel for files" section in
+`docs/guides/snapshots-and-time-travel.md`. Native accelerator: unaffected (canonical write path
+only) — no version pairing required.
+
+## v8.1.0 — 2026-07-10 (`brain.onChange` — the in-process change feed)
+
+New public API: subscribe to every committed mutation with
+`brain.onChange(cb) → unsubscribe`. One event per affected record, for **every**
+canonical write regardless of origin — direct calls, batch methods, `transact()`,
+imports, and Virtual Filesystem writes all funnel through the same commit point the
+feed is emitted from. This is the authoritative in-process signal for live UIs,
+cache invalidation, and realtime sync layers (downstream SDKs can forward it over
+their own transports to make local and remote brains uniform).
+
+Event shape (`BrainyChangeEvent`, exported): `kind` (`entity`/`relation`/`store`),
+`op` (`add`/`update`/`remove`/`relate`/`unrelate`/`updateRelation`/`clear`/`restore`),
+the post-commit `entity` or `relation` view (type + full custom metadata), the
+committed `generation`, and `timestamp`. Notable properties:
+
+- **Post-commit only** — an aborted write (a losing `ifRev` CAS, a rejected
+ transaction) never emits. If you got the event, the write is durable.
+- **Deletes fully described** — `remove`/`unrelate` carry the record's last
+ committed state (from the commit's own history record), not just an id; batch
+ deletes included.
+- **Batches emit per item**; a `transact()` batch's events share its single
+ generation. Entity deletes also emit `unrelate` for each cascaded relationship.
+- **Commit-ordered, asynchronous, isolated** — events arrive in commit order,
+ dispatched in a microtask so a slow listener never delays a write and a throwing
+ listener never affects the write or other listeners. Zero overhead with no
+ subscribers.
+- **`kind: 'store'`** events for `clear()`/`restore()` mean "refetch everything".
+- Fire-and-forget by design; each event's `generation` composes with
+ `asOf()`/`transactionLog()` for catch-up after a gap.
+
+Guide: `docs/guides/reacting-to-changes.md`. No behavior changes for existing code.
+
+## v8.0.17 — 2026-07-08 (count recovery scans the real layout · ~1,100 lines of dead 7.x machinery removed)
+
+A cleanup of the vestigial 7.x "hnsw sharding" machinery that turned up two real fixes.
+
+**1 — Count recovery now scans the layout the database actually uses.** When `counts.json` is
+lost or corrupted (container restarts, partial copies), the recovery scan rebuilt the entity/
+relationship counters by counting files in `entities/*/hnsw/` — directories the 8.0 write path
+never populates — so an established store recovered to **zero counts** on real data: wrong
+`getNounCount()`/stats, a "New installation"-style boot log, and a mis-sized rebuild-strategy
+decision at open. The scan now counts the canonical `entities////` tree.
+Regression-tested: delete `counts.json` from a populated store → reopen → exact counts.
+
+**2 — A latent init-time deadlock removed.** The recovery scan's type-distribution sampler called
+a guarded accessor (`getNounMetadata` → `ensureInitialized`) from **inside** `init()`, which
+re-enters `init()` and hangs the open. It was unreachable before only because the old scan never
+found anything to sample; the fix reads the sampled metadata files directly.
+
+**3 — The dead machinery itself is gone (−1,138 lines).** Twenty-three methods with zero callers:
+the 7.x hnsw-layout entity/edge CRUD, the sharding-depth prober + depth-migration engine, and an
+orphaned streaming paginator. Verified by reachability analysis before deletion; no public API
+touched; the full suite is green. New stores no longer carry the always-empty legacy directories'
+scan cost, and the boot log keeps the truthful new-vs-established wording from v8.0.13.
+
+## v8.0.16 — 2026-07-08 (concurrency fast-follow: atomic `ifAbsent`/`upsert` + exact blob reference counts)
+
+Closes the two remaining check-then-act races found in the sweep that followed v8.0.15's CAS fix
+(disclosed in that release's notes). Same bug class — a check performed before the serialization
+point that guards the apply — same cure.
+
+**1 — `add({ ifAbsent })` and `add({ upsert })` are now atomic.** The absence check ran before the
+commit mutex, so N concurrent same-id creates could all pass it and all write — the second
+silently overwriting the first, violating ifAbsent's "returns the existing id **without writing**"
+contract and upsert's "merge, never clobber" contract. The insert leg now carries a must-be-absent
+precondition (the same conditional-commit primitive as `ifRev`), verified under the commit mutex:
+exactly one concurrent create wins; every other caller takes its documented resolution — ifAbsent
+returns the existing id with zero writes, upsert merges into the now-existing entity (with a
+bounded retry if a concurrent delete intervenes). Verified: 8 concurrent `ifAbsent` creates
+advance the store by exactly ONE generation; 8 concurrent upserts on an absent id produce one
+create + seven merges (`_rev` lands at exactly 8). Note: `transact()`'s batch-level
+ifAbsent/upsert keep planning-time semantics (the batch converges to a valid entity; the window is
+documented, not silent).
+
+**2 — Blob reference counts are exact under concurrency.** The content-addressed blob store's
+`write()` (dedup: exists → add a reference, absent → create) and `delete()` (decrement → remove at
+zero) were unserialized read-modify-writes over the blob's metadata. Concurrent writes of
+identical content could lose references — making a later delete remove bytes **another file still
+referenced** (data loss), or leak unreferenced blobs. All reference-count-bearing mutations are
+now serialized per content hash (distinct content never contends): N concurrent identical writes
+yield exactly N references, and a blob is physically removed only when the true last reference
+drops. Verified with concurrent write/delete storms.
+
+Both fixes are in-process complete by construction — storage enforces single-writer-per-directory,
+so the process is the whole concurrency domain. No API changes.
+
+## v8.0.15 — 2026-07-08 (`ifRev` CAS is now atomic — exactly one winner under concurrency)
+
+Correctness fix for optimistic concurrency, reported from a production cutover rehearsal. N
+**concurrent** `update({ ifRev })` calls carrying the same expected revision ALL succeeded — zero
+`RevisionConflictError`s, last-writer-wins, the other N−1 writes silently lost. (Sequential calls
+conflicted correctly.) The revision check ran before the commit mutex, so interleaved callers all
+passed it before any apply landed — breaking the exactly-one-winner semantics the
+[optimistic-concurrency guide](docs/guides/optimistic-concurrency.md) promises, which advisory
+locks and per-entity ledgers/counters build on.
+
+The fix makes the check-and-apply a **conditional commit**: the generation store's commit paths
+accept a precondition that runs under the commit mutex against the just-read authoritative
+before-images — the per-record analogue of `ifAtGeneration`, which always ran there. `update()`
+and `transact()` per-op `ifRev` both re-verify at that point (the earlier check remains as a cheap
+fast-fail). A conflict aborts atomically: nothing staged, nothing applied, the same
+`RevisionConflictError` as before. Two adjacent behaviors also became honest:
+
+- **`_rev` is now monotonic under concurrency.** The winner's stamp derives from the
+ authoritative before-image, so N concurrent plain updates advance `_rev` by N (previously they
+ could all stamp the same stale value).
+- **CAS against a concurrently-deleted entity** now throws `EntityNotFoundError` instead of
+ silently resurrecting it (plain updates keep their last-writer-wins re-create semantics).
+
+Verified with a concurrency regression suite: 8 parallel same-rev updates → exactly 1 winner + 7
+conflicts; the documented read→CAS→retry ledger loop converges exactly (8 workers, 8 decrements,
+0 lost) — `tests/integration/ifrev-concurrent-cas.test.ts`. If you serialized writes in your own
+adapter as a workaround, it can come out after this upgrade.
+
+## v8.0.14 — 2026-07-07 (7→8 migration preserves branch-scoped non-entity state instead of deleting it)
+
+Defense-in-depth for the one-time 7→8 layout migration. The migration rescues the head branch's
+entities (`branches//entities/*` → `entities/*`) and then drained the whole `branches//`
+directory. If a 7.x engine had written durable state under the head branch *outside* `entities/`
+(a branch-scoped index, blob area, or field registry), that drain would have silently deleted it —
+the same failure class as the VFS content blobs a 7.x store kept in `_cow/`. The migration now drains
+the branch only when nothing but the moved entities remains; if any non-entity object survives, it
+**preserves the branch** (no delete) and logs a loud warning naming the leftover keys, so the state is
+recoverable rather than lost. No effect on a normal migration (a clean branch still drains); purely a
+guard against silent data loss.
+
+Cosmetic boot-log fix. Every persisted 8.0 store logged `📁 New installation: using depth 1
+sharding` on **every** open — even established brains holding thousands of entities — which is
+alarming to read during a restart or incident. Cause: 8.0 stores nouns in the canonical
+`entities/nouns///vectors.json` layout, but the legacy sharding probe inspected the
+`entities/nouns/hnsw/` directory, which the 8.0 write path never populates, so it always concluded
+"new". The new-vs-existing decision now consults the layout the database actually reads and writes
+(plus the known noun count), so an established store logs `📁 Using depth 1 sharding (N entities)`
+and only a genuinely empty store reports a new installation. No behavior change beyond the log line —
+it never triggered a rebuild or migration; entities were always read correctly.
+
+## v8.0.12 — 2026-07-07 (7→8 VFS-content recovery · zero-rebuild cold open · strict query operators)
+
+Three consumer-facing fixes.
+
+**1 — A 7→8 upgrade recovers Virtual Filesystem content automatically (data-integrity).**
+7.x stored VFS file content as blobs in the branch system's copy-on-write area (`_cow/`). 8.0
+removed that system and stores content blobs in the content-addressed store (`_cas/`), but the
+one-time layout migration only moved entities — it never adopted the `_cow/` content blobs. On a
+store that used the VFS (for example, a CMS with published pages), that left every VFS-backed read
+throwing `Blob metadata not found` and the pages 500ing. **8.0.12 adds an on-open recovery pass**
+that adopts every orphaned `_cow/` blob into `_cas/` — copying both the bytes and the metadata,
+idempotently and **non-destructively** (the `_cow/` originals are never deleted). It heals both a
+fresh 7→8 upgrade and a store **already** upgraded by an earlier 8.0.x that stranded them, with no
+operator action: just open the store under 8.0.12. An explicit force path is exposed as
+`brain.vfs.adoptOrphanedBlobs()` (returns `{ cowBlobs, adopted, alreadyPresent, incomplete }`). The
+automatic pre-upgrade backup is now **retained** if any blob can't be fully adopted
+(`incomplete > 0`) instead of being removed on entity-migration "success" alone. Affects any 7.x
+store that used the VFS; native-8.0 and non-VFS stores no-op on a cheap existence check. Full guide:
+`docs/guides/upgrading-7-to-8.md`.
+
+**2 — A cold open no longer re-derives durable indexes it can just load.**
+Opening a persisted store rebuilt the vector, graph, and metadata indexes from the canonical
+records even when the durable index state was present and loadable — an O(N) cost paid on every
+boot (measured at ~48 s on an ~11k-entity store). The rebuild gate now consults an honest
+per-provider durability signal (`init()` / `isReady()`) instead of an in-memory size heuristic, so
+a provider that has loaded (or can cheaply demand-load) its persisted index is not rebuilt. The
+built-in JS indexes keep today's behavior (a rebuild *is* their load path); the live query-time
+guards that self-heal a genuinely lost index are unchanged. **The zero-rebuild boot activates when
+the accelerator exposes the durability signal (`@soulcraft/cor@3.0.5`);** on earlier accelerator
+versions 8.0.12 falls back to the previous behavior safely — no regression, just no speedup yet.
+
+**3 — Unknown query operators now throw instead of silently returning nothing.**
+`find({ where })` had two gaps: the in-memory matcher (used for egress re-validation and historical
+reads) was missing several documented operators (`in`, `greaterThanOrEqual`, `lessThanOrEqual`), so
+it silently disagreed with the index path; and an unknown operator key (a typo, or `notIn` written
+where `not: { in }` was meant) was treated as a nested-object field and silently matched nothing.
+8.0.12 aligns the matcher to the full documented operator set and **validates the `where` filter
+up front**, throwing a typed `BrainyError('INVALID_QUERY')` naming the bad operator. Dotted paths
+remain the supported form for nested fields. If you relied on an unknown key silently returning `[]`,
+it now throws — the fix is to use the documented operator or dot-notation.
+
+## v8.0.11 — 2026-07-02 (no script shape can hang on brainy's internals)
+
+Completes v8.0.10's exit fix for every operation class. Two further mechanisms found and fixed:
+the `beforeExit` auto-flush hook looped forever on any script that never reaches `close()` (Node
+re-emits `beforeExit` after each event-loop drain and the async flush schedules new work — it now
+self-deregisters before its single flush, which still lands your buffered data before exit), and
+every background-maintenance interval (graph auto-flush, LSM compaction, metadata write-buffer,
+VFS/path-cache maintenance, statistics debounce) is now unref'd at creation. Verified against the
+published package: an `add + relate` script exits cleanly both with `close()` (~0.5 s) and with no
+teardown at all — with the data confirmed durable on reopen. A per-operation-class sweep test now
+asserts no ref'd timer survives `close()`, so this bug class stays closed.
+
+## v8.0.10 — 2026-07-02 (a bare script exits cleanly after `close()`)
+
+A minimal `init → add → close` script used to hang forever after `close()` returned — brainy held
+four process keep-alives it never released: the global SIGTERM/SIGINT shutdown hooks (never
+removed; now deregistered when the last live instance closes), an internal cache-fairness interval
+(unclearable by construction; now unref'd), and the VFS/PathResolver maintenance intervals
+(`close()` never shut the VFS down; now wired). Verified against the published package: the same
+script now exits ~1 ms after `close()` resolves. No API change; servers and long-running processes
+are unaffected.
+
+## v8.0.9 — 2026-07-02 (guarded plugin auto-detection — the "install it and it's on" contract)
+
+With the default config (`plugins` unset), brainy now **auto-detects the first-party accelerator**:
+installing `@soulcraft/cor` is the opt-in. The detection is guarded — everything except "not
+installed" fails loud:
+
+- Not installed → plain brainy, silently (the free path — zero noise, zero cost).
+- Installed and healthy → it loads and announces itself (`[brainy] Plugin activated`).
+- Installed but broken (unresolvable, invalid shape, failed activation, version mismatch) →
+ **`init()` throws.** An installed accelerator never silently vanishes behind the JS engines.
+- `plugins: []` / `false` = explicit opt-out; `plugins: ['@soulcraft/cor']` pins the exact list
+ (unchanged semantics).
+
+Also new: if a plugin activates but registers **zero** native providers (e.g. a licensing gate
+declining to engage), brainy warns loudly instead of leaving you to discover every query is running
+on the JS engines.
+
+> This supersedes v8.0.8's "plugins are explicit opt-in" wording, which documented the pre-GA
+> loader accurately but contradicted the published product contract ("add the package and it
+> activates"). 8.0.9 makes the contract true — with the loud-failure guarantees intact.
+
+## v8.0.8 — 2026-07-02 (docs-only patch on the GA)
+
+Corrects the README's scale-up section: the native provider is **not** auto-detected — plugins are
+**explicit opt-in** (`new Brainy({ plugins: ['@soulcraft/cor'] })`; brainy never auto-imports a
+package you didn't list, and a listed plugin that fails to load throws rather than silently falling
+back to the JS engines). Also fixes the stale `plugins` config comment in the public types and one
+phrase in the plugin-author guide. No code change.
+
+## v8.0.7 — 2026-07-02 (GA, npm tag `latest`)
+
+> **Why 8.0.7:** npm permanently retires unpublished version numbers, and `8.0.0`–`8.0.6` were
+> consumed by a January development cycle (published and immediately unpublished). `8.0.7` is
+> the first stable release of the 8.x line; there are no earlier stable 8.0.x releases.
+
+**8.0.7 is the first stable major on the u64-id core.** It ships in lockstep with the optional
+native provider's `3.0` (the billion-scale path). Everything below works standalone on the open-core
+JS engine — a native provider is feature-detected and only changes the scale ceiling, never the API.
+
+**Upgrading from 7.x: nothing to script.** The first time 8.0 opens a 7.x brain it upgrades itself —
+an automatic, observable, **coordinated migration lock** rebuilds every derived index from your
+canonical records while Brainy **blocks and queues** reads and writes, so no operation ever touches a
+half-built index and no write is ever lost. Budget seconds-to-minutes per brain at large sizes; small
+brains rebuild inline on open. A pre-upgrade hard-link **backup** is taken automatically (removed on
+success, kept on failure for rollback). Observe it via `getIndexStatus().migration`; a caller that
+hits the window gets a typed, retryable **`MigrationInProgressError`** (catch → HTTP 503 +
+`Retry-After`). Bounded by `migrationWaitTimeoutMs` (default 30 s — it bounds *your wait*, not the
+rebuild). Opt out of the backup with `migrationBackup: false`.
+
+### Headline changes
+
+- **The cold-open silent-`[]` class is gone.** On a cold reopen, filtered finds (`find({ where })`)
+ and graph reads (`find({ connected })` / `neighbors()` / `related()`) never silently return `[]`
+ for data that is actually present. With a native provider the durable indexes cold-serve every
+ filter and edge with zero rebuild; the open-core engine adds belt-and-suspenders guards that
+ self-heal from canonical data or throw a loud, typed error — never a silent empty result. Two new
+ exported errors: **`MetadataIndexNotReadyError`**, **`GraphIndexNotReadyError`**.
+
+- **Faster vector search (open-core).** The JS distance functions are allocation-free loops instead
+ of `reduce`: **~6× cosine, ~1.4× euclidean** (MEASURED — `tests/benchmarks/distance-microbench.mjs`,
+ 384-dim, median of 41). Numerically identical, so recall is unchanged. Exact metadata-filter
+ pushdown and scale-aware search width keep `find()` recall-correct as data grows.
+
+- **Per-write immutable history.** Every write is generation-stamped; `now()` / `asOf()` /
+ `transact()` read a consistent point in time, and a retention knob bounds history growth. Single
+ writes participate in the same immutable timeline as transactions.
+
+- **Billion-scale resident memory.** Nothing is O(N)-resident on the write or time-travel path —
+ per-id caches removed, the committed-generation ledger is an interval set, per-id history chains
+ are bounded (hot-window + LRU). Resident memory is independent of entity count.
+
+- **Deterministic, round-trip-free writes.** Supply your own ids, forward-reference not-yet-written
+ entities inside a `transact()`, `ifAbsent` upsert, and `brain.newId()` (uuidv7) — write graphs
+ without a write→wait→get round-trip.
+
+### Breaking changes (summary — the full migration guide follows below)
+
+- **Runtime floor: Node ≥ 22 / Bun ≥ 1.1** (Bun recommended). Compiler target ES2023; the DOM lib is
+ dropped — 8.0 is a Node/Bun/Deno engine with no browser path.
+- **`neural()` removed** and **`Db.search` removed** — use `find()` on the brain.
+- **Storage config is one `path` key.** Removed aliases now throw with a message pointing at `path`.
+- **`get()` omits the vector by default** — pass `{ includeVectors: true }` when you need it.
+- **Export/import type is `PortableGraph`** (was `BackupData`; wire tag `brainy-backup` →
+ `brainy-portable-graph`). Re-export any snapshots you version outside Brainy.
+- **Reserved `visibility` field** (`public` / `internal` / `system`) on nouns and verbs; `internal`
+ and `system` are excluded from normal reads by default.
+- **4 deprecated query operators removed** (`is`/`isNot`/`greaterEqual`/`lessEqual`) and reserved
+ keys in a `metadata` bag now **throw** by default — details in the rc notes below.
+
+> Post-rc.9 hardening folded into GA (no on-disk or provider-contract change vs `8.0.0-rc.9`):
+> the metadata cold-read guard, the auto pre-upgrade backup (with an `_id_mapper/*` mmap byte-copy
+> correctness fix), and a CI fix so a fresh clone builds green.
+
+### RC history (rc.1 → rc.9, npm tag `rc`)
+
+> **rc.9 changes (2026-07-01):**
+> - **Whole-brain auto-upgrade is now a coordinated, observable LOCK — supersedes rc.8's no-freeze
+> approach.** When a large brain's derived-index format changes (7.x→8.0), the native provider
+> rebuilds the indexes from the canonical records **in place** while Brainy **blocks and queues**
+> reads and writes — so no operation ever touches a half-built index. This is a clean *blocking
+> upgrade to a known-good state* (vs rc.8's online background swap): unknown/halfway states are
+> more dangerous than a bounded wait. Small brains still rebuild inline on open. New surface, all
+> additive: a typed, exported, retryable **`MigrationInProgressError`** (catch it → HTTP 503 +
+> `Retry-After`); `getIndexStatus()` gains **`migrating` + `migration`** progress (never gated —
+> the readiness-probe signal); a **`migrationWaitTimeoutMs`** config (default 30 s — it bounds the
+> *caller's wait*, NOT the rebuild, which is unbounded); `health()` / `checkHealth()` report the
+> upgrade without blocking. No effect without a native provider.
+> - **Faster vector search (open-core).** The JS distance functions are rewritten from `reduce` to
+> allocation-free loops — **~6× cosine, ~1.4× euclidean** (MEASURED, `tests/benchmarks/distance-microbench.mjs`,
+> 384-dim, median of 41). Numerically identical → recall unchanged.
+> - **Runtime floor + Bun.** Engines are now **Node ≥22 / Bun ≥1.1** (fixes a Node-24 `EBADENGINE`);
+> compiler target ES2023 with DOM dropped from the type lib (8.0 is Node/Bun/Deno-only, no browser
+> path). **Bun is recommended as a runtime** (`bun add` / `bun run`); the single-binary
+> `bun build --compile` is not a supported target (native addon can't embed + a Bun 1.3.10 codegen
+> regression). `.d.ts` stays TS-5.x-parseable.
+
+> **rc.8 additions (2026-06-30) — additive, no breaking change:**
+> - **No-freeze (online) whole-brain auto-upgrade.** When a derived-index format changes, a large
+> brain upgrades **without blocking** — the native provider rebuilds the new indexes in the
+> background and serves correct reads from the canonical records meanwhile, then atomically swaps;
+> no minutes-long freeze on the first open/query. (Small brains still auto-rebuild inline on open.)
+> New surface: an optional provider `isMigrating()` deference signal, a public `brain.stampBrainFormat()`
+> the provider calls once its swap verifies, and a `@soulcraft/brainy/brain-format` export so the
+> provider shares the `indexEpoch` constant. No effect without a native provider.
+
+> **rc.7 additions (2026-06-30) — additive, no breaking change:**
+> - **8.0 cold-graph self-heal.** `find({ connected })` / `neighbors()` / `related()` never
+> silently return `[]` on a cold open where the graph adjacency loaded its membership but not
+> its source→target edges — it self-heals (rebuild from storage) or throws a loud
+> `GraphIndexNotReadyError`, never empty-for-persisted-data. (The 8.0 equivalent of the 7.33.4
+> fix 7.x consumers already have; gates on the native provider's honest `isReady()` edge-readiness
+> signal.)
+> - **Billion-scale RAM — nothing O(N)-resident on the write/time-travel path.** Eliminated the
+> per-id storage caches (per-type counts now sourced from the record), made the
+> committed-generation ledger an interval set, and bounded the per-id time-travel history chains
+> (hot-window + LRU, with lock-light reconstruction so historical reads never stall writers).
+> Resident memory is now independent of entity count.
+> - **Whole-brain version handshake.** A `_system/brain-format.json` marker + `brain.formatInfo()`
+> + a shared, lockstep `indexEpoch`: a future derived-index format change auto-rebuilds the
+> indexes from the canonical records on open (non-destructively) — the foundation for
+> whole-brain auto-upgrade with the native provider. No effect on an unchanged brain.
+
+> **rc.6 additions (2026-06-29) — additive, no breaking change:**
+> - **Open-core perf**: HNSW delete is now O(in-degree) (was O(N²) for bulk delete) via a
+> reverse-adjacency index; the negation/absence operators (`ne`/`exists:false`/`missing:true`)
+> are served as a roaring-bitmap difference instead of materializing the whole corpus.
+> - **Native provider contract** (cor lockstep, optional/feature-detected — no effect without a
+> native provider): a cold-open `probeConsistency()` self-heal hook, and a `getIdsForFilter`
+> page bound so the native index can early-stop the unsorted `find({ type, where, limit })` path.
+> - **Test hygiene**: re-homed previously-unrun test suites into CI + a guard so a test file can
+> never silently fall outside every config again. No source/API change from these.
+
+
+**Affected products:** every consumer — this is a major release with removed
+surfaces, hard renames (no aliases, no deprecation period), and one flipped
+default. This entry **is** the migration guide: the find/replace pairs, the
+sed snippet, and the upgrade checklist below are the complete story — there is
+no separate migration doc. **`8.0.7` is GA on `latest`** — install with
+`npm i @soulcraft/brainy@latest`.
+
+> **rc.3–rc.5 additions (2026-06-24):**
+> - **Showcase-quality GA hardening (rc.5).** A full readiness audit closed a cold-init
+> version-coupling bug (a matched native provider could be rejected on first open), turned
+> silent storage-read failures into named `BrainyError`s (no more empty-result-on-error),
+> stopped the JS graph LSM from orphaning compacted SSTables, corrected the public docs to
+> the real API, and documented the two headline methods. Plus a dead/deprecated-code sweep.
+> - **BREAKING — 4 deprecated query operators removed.** `is`→`eq`, `isNot`→`ne`,
+> `greaterEqual`→`gte`, `lessEqual`→`lte`. The canonical operators and their clean long-form
+> aliases (`equals`/`notEquals`/`greaterThan`/`greaterThanOrEqual`/`lessThan`/`lessThanOrEqual`)
+> are unchanged — only the four redundant spellings are gone. Find/replace if you used them.
+> - **`find()` search mode** is now the single `searchMode: SearchMode` option (the redundant,
+> silently-ignored `mode` alias and the unwired `explain` flag were removed from `FindParams`).
+> - The phantom index-integrity guard (find() re-validates every result against its predicate)
+> shipped in rc.3; rc.4 added the `accel.isInitialized` gate + #35 at-gen candidate vectors.
+
+> **rc.1 additions (this entry predates them — full writeup lands at GA):** entity-id
+> normalization — `brain.newId()` (UUID v7) + v7 default ids, and non-UUID string ids are
+> transparently normalized to a stable UUID v5 (original key preserved under `_originalId`);
+> `brain.neural()` clustering and `Db.search()` removed (use `find({ vector })` /
+> `find()` + aggregation `GROUP BY`); storage config collapsed to one `path` key (the
+> pre-8.0 aliases now throw); and `queryAggregate` no longer hangs/staled min-max after a delete.
+> **Reserved fields in a `metadata` bag now throw by default** — passing a Brainy-reserved key
+> (`confidence`, `weight`, `subtype`, `visibility`, `service`, `createdBy`, `noun`/`verb`, `data`,
+> `createdAt`, `updatedAt`, `_rev`) inside `metadata` on `add()`/`update()`/`relate()`/`updateRelation()`
+> (untyped/JS callers — TypeScript already blocks it) is rejected with an Error naming the correct
+> param, instead of the old silent remap-or-drop. Pass reserved values as their dedicated top-level
+> params. Opt back into the legacy behavior with `new Brainy({ reservedFieldPolicy: 'warn' | 'remap' })`.
+
+> **rc.2 additions (2026-06-21):**
+> - **Graph performance + the `brain.graph` namespace.** New `brain.graph.subgraph(seeds, opts)`
+> (bounded multi-hop neighborhood → `{ nodes, edges, truncated }`) and `brain.graph.export(opts)`
+> (stream the whole graph in one O(N+E) pass — the right primitive for visualizing all data
+> instead of paging per node). `related({ node })` returns every edge incident to an entity in
+> **both directions** in one O(degree) call. Under the hood: `related({ from/to })` now stays
+> O(degree) under the default visibility filter (was a full scan), and the verb **and** noun
+> pagination walks are cursor-based, so a full edge/node walk is **O(N), not O(N²)** (a consumer
+> measured a 19k-edge whole-graph read drop from ~27s toward a single scan). These transparently
+> use a native graph engine when present (the `@soulcraft/cor` 3.0 acceleration layer) and fall
+> back to pure-TS adjacency otherwise.
+> - **Additive ergonomics:** `add({ upsert: true })` (create-or-update in one call — merges into an
+> existing id instead of overwriting), `find({ includeVectors: true })`, and `removeMany`'s batch
+> size is now storage-adaptive (was a hardcoded 10).
+> - **Correctness fixes (apply to all consumers, not just graph users):** `related()` results now
+> carry `visibility`; whole-graph/`getNouns` streaming no longer leaks `system`/`internal` entities;
+> and small-page cursor walks over nouns no longer loop. No API change — just correct behavior.
+
+> **rc.3 additions (2026-06-23):**
+> - **Graph analytics on the `brain.graph` namespace.** Three intent-level reads that answer
+> whole-graph questions in one call:
+> - **`brain.graph.rank(opts?)`** → `{ id, score }[]` descending — "which entities matter most"
+> (importance / centrality). `topK` to cap.
+> - **`brain.graph.communities(opts?)`** → `{ groups: string[][], count }` — "which things group
+> together". Weakly-connected components by default; `{ directed: true }` returns
+> strongly-connected components.
+> - **`brain.graph.path(from, to, opts?)`** → `{ nodes, relationships, cost } | null` — the best
+> route between two entities. Fewest hops by default; `{ by: 'weight' }` minimizes summed edge
+> weight (the 0–1 connection strength); `direction`, `type`, and `maxDepth` filters apply.
+> These are **intent contracts, not algorithm contracts** — the question is the promise, the
+> algorithm is the engine's choice. They transparently use the native `@soulcraft/cor` 3.0 graph
+> engine when present, and fall back to pure-TS kernels (PageRank, connected components / Tarjan
+> SCC, BFS / Dijkstra) otherwise. Both paths return identical shapes and respect the default
+> visibility filter (opt in with `includeInternal` / `includeSystem`).
+> - **Filtered vector search keeps its recall (`allowedIds` pushdown).** A `find({ query, where })`
+> that combines semantic search with a metadata filter now restricts the vector walk to the
+> matching candidates *inside* the search (walk-all, collect-allowed) instead of filtering the
+> top-k afterward — so a query whose nearest vectors are all filtered out still returns the best
+> matches that DO pass the filter, rather than coming back empty. No API change. With the native
+> `@soulcraft/cor` 3.0 stack the matched universe is forwarded as an opaque roaring buffer with
+> zero id materialization in TypeScript; the pure-JS path restricts the beam walk with a string set.
+> - **Historical (`asOf`) semantic search can skip the rebuild.** A filtered semantic query at a
+> past generation — `db.asOf(g).find({ query, where })` — no longer always rebuilds an ephemeral
+> in-memory vector index over every at-`g` vector (O(n@G)). When a native versioned vector engine
+> is registered and can serve the pinned generation, Brainy resolves the at-`g` filter universe from
+> its record layer (no rebuild) and routes the vector leg to the engine with the matched ids + the
+> generation; without one it falls back to the materialization (unchanged). The `VectorIndexProvider.search`
+> options gained an optional `generation?: bigint` ("omitted = now") to carry this — additive, the
+> built-in index ignores it. No public API change.
+> - **`find({ where, orderBy })` bounds the sort to the page.** A broad filter + `orderBy`
+> returning one page no longer materializes every matching sorted id (it produced the full sorted
+> match set, then sliced) — the page bound (`offset + limit`) is threaded into the column store's
+> top-K sort, so returning 20 rows from a million matches stays a bounded-K heap. No API change;
+> ordering and pagination are unchanged.
+> - **`brain.graph.subgraph()` accepts a query (query→expand).** The seed selector now takes not
+> just entity id(s) but a `find()` result or a `FindParams` query — `brain.graph.subgraph({ where:
+> { team: 'platform' } }, { depth: 1 })` runs the query and expands the neighborhood of every
+> match in one call. With the native engine a metadata-only query's matched universe is handed to
+> the traversal as an opaque set with no id materialization in TypeScript (the query→expand
+> fusion); the pure-JS path materializes the matched ids and expands from them. Existing
+> id-seeded calls are unchanged.
+
+### Headline: Database as a Value
+
+8.0 replaces Brainy's two overlapping version-control subsystems (copy-on-write
+branching and per-entity versioning, ~5,100 LOC combined) with **one
+mechanism**: generational MVCC over immutable, generation-stamped records,
+exposed through a Datomic-style immutable database value — the **`Db`**.
+
+```ts
+const db = brain.now() // pin the current state — O(1), no I/O
+
+await brain.transact([
+ { op: 'update', id: invoiceId, metadata: { status: 'paid' } }
+], { meta: { author: 'billing-service', reason: 'PO-7741' } })
+
+await db.get(invoiceId) // still 'pending' — pinned, forever
+await brain.get(invoiceId) // 'paid' — live
+await db.release() // unpin when done
+```
+
+What you get:
+
+- **`brain.now()`** — pins the current generation as an immutable `Db` view.
+ True snapshot isolation: the view reads exactly its pinned state no matter
+ what commits afterwards, including deletes. Readers never block writers and
+ writers never block readers.
+- **`brain.transact(ops, { meta, ifAtGeneration })`** — atomic multi-write
+ batches (`add` / `update` / `remove` / `relate` / `unrelate`) committed as
+ exactly one generation. Either every operation applies or none do.
+ `ifAtGeneration` is whole-store compare-and-swap (`GenerationConflictError`
+ on conflict) — the big sibling of the per-entity `ifRev` CAS from 7.31.
+ `meta` is reified Datomic-style into an append-only transaction log,
+ readable via `brain.transactionLog()`.
+- **`brain.asOf(generation | Date | snapshotPath, { exclusive? })`** — time
+ travel with the **full query surface**: `get()`, `find()` in every mode,
+ semantic search, graph traversal, cursors, aggregation, all at the pinned past
+ state. `exclusive: true` pins the generation immediately before the target
+ (strict-before).
+- **`db.with(ops)`** — speculative writes applied in memory on top of a view.
+ Nothing touches disk, the generation counter, or the indexes. What-if
+ analysis, then `transact()` the same ops for real.
+- **`db.persist(path)`** — instant self-contained snapshots. On filesystem
+ storage they are built from hard links (no entity data is copied; later
+ writes to the source can never alter the snapshot because rewrites swap
+ inodes). `brain.restore(path, { confirm: true })` replaces the whole store
+ from one; `Brainy.load(path)` opens one read-only with the full query
+ surface.
+- **Range verbs over history** — answer "what happened BETWEEN two points":
+ - **`db.since(Db | generation | Date)`** — the entity and relationship ids
+ committed transactions touched after an **exclusive** lower bound (now
+ accepts a generation or `Date`, not just a prior `Db`).
+ - **`brain.diff(a, b)`** — the touched ids CLASSIFIED as `{ added, removed,
+ modified }` (split by nouns/verbs) by resolving each at both endpoints; a
+ touched-but-reverted id lands in none of the buckets. Endpoints are a
+ generation, `Date`, or `Db`, in either order.
+ - **`brain.history(id, { from?, to? })`** — every distinct version of ONE
+ entity/relationship over a range, oldest first (`value: null` marks a
+ removal); each version equals `asOf(version.generation).get(id)`.
+ - **`brain.transactionLog({ from?, to?, limit? })`** — the commit log over an
+ **inclusive** generation/`Date` window (contrast `since`'s exclusive lower
+ bound); `limit` applies last.
+- **Every write is versioned (Model-B).** History granularity is **per-write**:
+ EVERY write — `transact()` AND a single-operation `add`/`update`/`remove`/
+ `relate` — is its own immutable generation. A `now()` pin always freezes
+ against later writes, and `asOf`/`since`/`diff`/`history`/`transactionLog`
+ reflect single-ops exactly like transacts. `transact()` groups several
+ operations into ONE atomic generation. Single-op history durability is **async
+ group-commit** (the live write is acknowledged immediately; its before-image
+ is batched to disk in one fsync) — a hard crash can lose only the last
+ un-flushed window's *history*, never live data, and a crash mid-flush is
+ recovered by drop-without-restore. A freshly-initialized brain is
+ `generation() === 0` with an empty `transactionLog()` (init-time
+ infrastructure is the un-versioned baseline); the first user write is gen 1.
+- **`new Brainy({ retention })`** — the retention knob governs auto-compaction on
+ `flush()`/`close()`: **unset → ADAPTIVE** (disk/RAM-pressure byte budget,
+ zero-config; a coordinator can drive it via `brain.setRetentionBudget(bytes)`)
+ · **`'all'`** → unbounded · **`{ maxGenerations?, maxAge?, maxBytes? }`** →
+ explicit caps (reclaim oldest-unpinned while ANY cap is exceeded). Live pins
+ are ALWAYS exempt.
+- **`brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? })`** — manual
+ reclaim on the same caps. Compaction never breaks a pinned read. `diff`/`since`
+ throw `GenerationCompactedError` below the horizon; `history` truncates to it.
+
+The precise guarantees are documented in:
+
+- [docs/concepts/consistency-model.md](docs/concepts/consistency-model.md) —
+ the guarantees, each proven by a dedicated test in
+ `tests/integration/db-mvcc.test.ts`
+- [docs/guides/snapshots-and-time-travel.md](docs/guides/snapshots-and-time-travel.md)
+ — the recipes: backup, restore, time-travel debugging, what-if, audit trails
+- [docs/ADR-001-generational-mvcc.md](docs/ADR-001-generational-mvcc.md) — the
+ design record: persisted layout, commit protocol, crash recovery, proof table
+
+### Portable export & import (PortableGraph v1)
+
+A portable, versioned graph format that serializes part or all of a brain to a
+single JSON document and restores it — the cross-environment, cross-version
+(7.x↔8.0), partial-or-whole companion to the native `persist()` snapshot.
+
+```ts
+// Export is a method on the immutable Db, so it composes with now()/asOf()/with()
+const graph = await brain.export({ collection: id }, { includeVectors: true })
+await otherBrain.import(graph, { onConflict: 'merge' }) // dedup-by-id
+
+;(await brain.asOf(gen)).export(sel) // time-travel export
+brain.now().with(ops).export(sel) // what-if export
+```
+
+- **`brain.export(selector?, options?)` / `db.export(...)`** → a versioned
+ `PortableGraph` document. Selectors reuse `find()`'s grammar: `{ ids }`,
+ `{ collection }` (alias `memberOf`, transitive `Contains`),
+ `{ connected: { from, depth } }`, `{ vfsPath }`, predicate
+ (`{ type, subtype, where, service }`), or the whole brain (omit) — and they
+ compose. Options: `includeVectors`, `includeContent` (VFS file bytes),
+ `includeSystem`, `edges: 'induced' | 'incident' | 'none'`.
+- **`brain.import(graph, options?)`** is polymorphic: a `PortableGraph` document is
+ restored as **one atomic transaction** (`onConflict: 'merge' | 'replace' | 'skip'`,
+ `reembed: 'auto' | 'never'`, `remapIds` for cloning); a file/buffer routes to the
+ existing CSV/PDF/Excel/JSON ingestion. No migration for ingestion callers.
+- **`validatePortableGraph(data)`** — dry-run structural/version/endpoint check before import.
+- **Format** (`format:'brainy-portable-graph'`, `formatVersion: 1`) is identical on
+ 7.x and 8.0; standard fields (`subtype`/`visibility`/`data`/…) are top-level,
+ `metadata` is custom-only. Current-state (no generation history — that lives in
+ `persist()`). Types exported from the package root.
+- **Naming:** the type is `PortableGraph` (with `PortableGraphEntity` /
+ `PortableGraphRelation`) — it is the portable interchange form of a graph, not a
+ backup (that role is `persist()`/`load()`). Renamed from the short-lived
+ `BackupData`/`'brainy-backup'` (introduced in 7.32.0, never adopted) with no
+ compatibility shim.
+- Guide: [docs/guides/export-and-import.md](docs/guides/export-and-import.md).
+
+### Aggregation: distinctCount over any value type
+
+`distinctCount` now counts distinct values of **any** type (strings, booleans,
+numbers) — previously it silently returned `0` for non-numeric fields, missing its
+primary use (distinct categories / users / tags). `percentile` (with `p`; median =
+`p: 0.5`), `stddev`, and `variance` are exact and delete-safe alongside the
+write-time `sum`/`count`/`avg`/`min`/`max`.
+
+### Removed surfaces and their replacements
+
+| Removed in 8.0 | Replacement |
+|---|---|
+| `brain.fork(name)` | Speculation: `db.with(ops)` (in-memory). Long-lived writable copy: `brain.restore()` a persisted snapshot into a fresh data directory. |
+| `brain.checkout(branch)` | Open the snapshot you want — `Brainy.load(path)` read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store. |
+| `brain.listBranches()` / `brain.getCurrentBranch()` / `brain.deleteBranch()` | A "branch" is now a name → snapshot-path mapping owned by your application. |
+| `brain.commit({ message })` | `brain.transact(ops, { meta })` — every batch is an atomic, logged, time-travelable commit; audit fields live in the database, not in commit messages. |
+| `brain.getHistory()` / `brain.streamHistory()` | `brain.transactionLog({ limit })` + `db.since(olderDb)`. |
+| `brain.versions.*` (`save` / `list` / `compare` / `restore` / `prune`) | A pinned `Db` or persisted snapshot captures *every* entity at that moment; `asOf()` reads any entity's past state; `restore()` for whole-store rollback. |
+| `brain.data()` (DataAPI: `clear` / `import` / `export` / `getStats`) | `brain.clear()`, `brain.import()`, `db.persist(path)` (the full-fidelity export format), `brain.restore(path, { confirm: true })`, `brain.stats()`. |
+| Cloud + browser storage adapters (`s3` / `gcs` / `r2` / `opfs` storage types, Azure Blob, plus the `storage.branch` option) | `filesystem` and `memory` only. Cloud backup is now an operator concern: `db.persist()` produces a plain directory — sync it with `gsutil` / `aws s3 cp` / `rclone` / `azcopy`. Passing a removed storage type throws at construction with this exact guidance. |
+| Browser support | Node.js 22 LTS or Bun ≥ 1.0 (`engines` enforces `node: 22.x`); Deno works through its Node compatibility layer. |
+| `brain.migrateToDiskAnn()` / `brain.migrateToHnsw()` | Gone — there is one canonical `'vector'` provider slot. A registered native vector provider selects its own operating mode; there is nothing to call. |
+| Distributed-clustering subsystem — `config.distributed`, the `DistributedRole` enum, the 13 `BRAINY_*` cluster env vars (`BRAINY_DISTRIBUTED`, `BRAINY_ROLE`, `BRAINY_HTTP_PORT`, `BRAINY_WS_PORT`, `BRAINY_DNS`, `BRAINY_SERVICE`, `BRAINY_NAMESPACE`, `BRAINY_CONSENSUS`, `BRAINY_COORDINATOR`, `BRAINY_NODES`, `BRAINY_REPLICAS`, `BRAINY_SHARDS`, `BRAINY_TRANSPORT`), the storage `setDistributedComponents` hook, and the `@soulcraft/brainy/config` preset/augmentation registry | Removed. Brainy 8.0 is a single-process library — there is no coordinator, peer discovery, or consensus to operate. Scale via: the optional native provider (`@soulcraft/cor` — on-disk DiskANN to 10B+ vectors on one machine); per-tenant pools (one instance + storage dir per tenant); and horizontal read scaling (many reader processes against one shared store, single writer). The `mode: 'reader' \| 'writer'` multi-process roles are unchanged. |
+| CLI `fork` / `branch` / `checkout` / `migrate` | CLI `snapshot ` / `restore ` / `history` / `generation`. |
+| `BrainyZeroConfig` type | Removed. 8.0 zero-config is automatic and internal — `new Brainy()` auto-detects storage, derives HNSW quality from `config.vector.recall`, picks `persistMode` from the adapter, and sizes caches to the detected container-memory limit. There is no config-generation type to import. |
+| `brain.isFullyInitialized()` / `brain.awaitBackgroundInit()` | `await brain.ready`. The built-in filesystem/memory adapters finish initialization synchronously inside `init()`, so a single readiness promise is the whole story — these methods were no-ops once cloud adapters were removed. |
+
+**Opening a 7.x store auto-migrates it.** 8.0 stores entities at the root; 7.x
+stored them branch-scoped under `branches//`. On first open, 8.0 collapses
+the **HEAD branch** (`config.storage.branch`, default `main`) to the 8.0 layout in
+place and rebuilds all derived state — no action required (`autoMigrate` defaults
+to `true`; set it to `false` to make 8.0 refuse a legacy layout with an explicit
+error instead). Two caveats:
+
+- **Non-HEAD branches are not imported.** 8.0 has no COW branches; only the head
+ branch's data is migrated. If you care about other branches, **export each one
+ while still on 7.x** (`fork → export`, or copy its store).
+- **Back up first for rollback.** The migration mutates the directory in place and
+ 8.0 does not keep the old layout — copy the data directory before the first 8.0
+ open if you need to roll back.
+
+### Renames — find/replace pairs
+
+Hard renames, no aliases. Every pair below is verified against the 8.0 source.
+
+| Brainy 7.x | Brainy 8.0 |
+|---|---|
+| `brain.delete(id)` | `brain.remove(id)` — matches the transact op vocabulary (`add` / `update` / `remove` / `relate` / `unrelate`) |
+| `brain.deleteMany(params)` | `brain.removeMany(params)` |
+| `DeleteManyParams` (type) | `RemoveManyParams` |
+| `brain.getRelations(paramsOrId)` | `brain.related(paramsOrId)` — the same name a pinned `Db` exposes (`db.related()`) |
+| `GetRelationsParams` (type) | `RelatedParams` |
+| CLI `delete ` | CLI `remove ` (JSON output `{ id, removed: true }`, matching `unrelate`) |
+| `HnswProvider` (from `@soulcraft/brainy/plugin`) | `VectorIndexProvider` |
+| `HNSWIndex` (exported class) | `JsHnswVectorIndex` |
+| Provider keys `'hnsw'` and `'diskann'` | `'vector'` (the only key Brainy consults for the vector index) |
+| `config.hnsw = { quantization, vectorStorage }` | `config.vector = { recall?, persistMode? }` (shape change — see below) |
+| `config.vector.quantization` (and 7.x `config.hnsw.quantization`) | **Removed.** The JS vector path is full-precision (exact float32 distances) only — quantization at scale is the native provider's job (e.g. DiskANN PQ). |
+| `hnswPersistMode: 'immediate' \| 'deferred'` (top level) | `config.vector.persistMode` (auto-selected from the storage adapter when omitted: `'immediate'` on filesystem, `'deferred'` on memory) |
+| `config.storage.type: 's3' \| 'gcs' \| 'r2' \| 'opfs'` | `'filesystem'` (or `'memory'` / `'auto'`) |
+| `config.storage.branch` | Removed (no COW branches) |
+| `brain.stats().indexHealth.hnsw` | `brain.stats().indexHealth.vector` |
+| Storage adapter contract `saveHNSWData()` / `getHNSWData()` | `saveVectorIndexData()` / `getVectorIndexData()` |
+| Cache category `'hnsw'` (`CacheProvider` union) | `'vectors'` |
+| `config.history = { retainGenerations, retainMs, autoCompact }` | `config.retention` — `'all'` \| `'adaptive'` \| `{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }`; **unset → adaptive** (was keep-100/7-days). The fields are now **caps**, not floors. |
+| `compactHistory({ retainGenerations, retainMs })` | `compactHistory({ maxGenerations, maxAge, maxBytes })` — `retainGenerations`→`maxGenerations`, `retainMs`→`maxAge` (now upper-bound CAPS), plus new `maxBytes`. |
+| `CompactHistoryOptions.retainGenerations` / `.retainMs` | `.maxGenerations` / `.maxAge` (+ `.maxBytes`) |
+
+Mechanical renames as one command (run from your repo root, review the diff):
+
+```bash
+find . -name '*.ts' -not -path '*/node_modules/*' -print0 | xargs -0 sed -i \
+ -e 's/\bHnswProvider\b/VectorIndexProvider/g' \
+ -e 's/\bHNSWIndex\b/JsHnswVectorIndex/g' \
+ -e 's/\bsaveHNSWData\b/saveVectorIndexData/g' \
+ -e 's/\bgetHNSWData\b/getVectorIndexData/g' \
+ -e 's/indexHealth\.hnsw\b/indexHealth.vector/g' \
+ -e "s/registerProvider('hnsw'/registerProvider('vector'/g" \
+ -e "s/registerProvider('diskann'/registerProvider('vector'/g" \
+ -e "s/getProvider('hnsw'/getProvider('vector'/g" \
+ -e "s/getProvider('diskann'/getProvider('vector'/g" \
+ -e 's/\.getRelations(/.related(/g' \
+ -e 's/\bGetRelationsParams\b/RelatedParams/g' \
+ -e 's/\.deleteMany(/.removeMany(/g' \
+ -e 's/\bDeleteManyParams\b/RemoveManyParams/g'
+```
+
+`delete()` → `remove()` is deliberately **not** in the snippet: `.delete(` is
+too common (`Map`/`Set`/storage adapters) for a blind sed. Find your
+`brain.delete(...)` call sites and rename them by hand.
+
+The config shape changes need a human (they are not 1:1 textual):
+
+```ts
+// 7.x
+new Brainy({
+ hnswPersistMode: 'deferred',
+ hnsw: { quantization: { enabled: true, bits: 8, rerankMultiplier: 3 } }
+})
+
+// 8.0 — config.vector is { recall?, persistMode? }
+new Brainy({
+ vector: {
+ recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
+ persistMode: 'deferred' // optional; auto-selected from storage when omitted
+ }
+})
+```
+
+`config.vector.recall` replaces the algorithm-internal HNSW knobs: `'balanced'`
+(the default) maps to exactly the 7.x default parameters, so an upgrade without
+explicit knobs changes nothing about search behaviour. `config.vector.quantization`
+and `config.hnsw.vectorStorage` are removed: the JS vector path now computes exact
+float32 distances throughout (no rerank/approximate branch), which is what made
+its quantization a memory *increase* — it stored both the full and the quantized
+vectors in RAM. Quantization at scale belongs to the native provider's own PQ.
+The on-disk vector index file names are **unchanged** (e.g. `hnsw-system.json`) —
+existing filesystem stores need no data migration for this rename; only the API
+surface moved.
+
+### Behavior changes
+
+- **`subtype` is required by default.** 7.30's opt-in strict mode is now the
+ default: every `add()` / `addMany()` / `update()` / `relate()` /
+ `relateMany()` / `updateRelation()` rejects writes whose type carries no
+ non-empty `subtype` (`src/brainy.ts:9759` — `requireSubtype ?? true`).
+ Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy
+ data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type
+ allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)`
+ still compose. `brain.audit()` reports entries missing a subtype and the new
+ `brain.fillSubtypes(rules)` migration helper backfills them — one rule per
+ NounType/VerbType (literal default or per-entry function), applied only to
+ entries still missing a subtype, returning
+ `{ scanned, filled, skipped, errors, byType }`. Idempotent: re-running fills
+ nothing, so a crashed run is resumed by running it again.
+- **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws
+ a teaching error if a caller passes an `id` field
+ (`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently
+ ignored (a generated UUID was used anyway); 8.0 says so out loud, because
+ graph-index providers key verb-int interning on the raw UUID bytes.
+- **`update({ vector })` is honored on its own.** 7.x only applied a supplied
+ pre-computed vector when `data` was also passed (it was silently dropped
+ otherwise). 8.0 applies an explicit `params.vector` with dimension
+ validation (`src/brainy.ts:1702-1713`; proven by
+ `tests/unit/brainy/update.test.ts:256`).
+- **`brain.clear()` re-resolves all indexes exactly as `init()` does** —
+ including plugin-provided vector/metadata/entityIdMapper factories and VFS
+ root re-creation. In 7.x, clearing a plugin-accelerated brain could leave
+ the metadata index rebuilt without its native id-mapper wiring.
+- **`find({ connected: { …, subtype } })` with `depth > 1` now works.** 7.30
+ threw `NOT_SUPPORTED` for multi-hop subtype-filtered traversal; 8.0
+ implements the BFS in the JS graph index and routes to a native provider
+ when one is registered.
+- **Every write advances the generation clock; only `transact()` writes
+ history.** Single-operation writes (`add` / `update` / `remove` / `relate`
+ outside `transact()`) bump `brain.generation()` so watermarks and CAS stay
+ sound, but they do not stage historical records — they remain visible
+ through earlier pins and are not reported by `db.since()`. Writes you want
+ to travel back through go through `transact()`. This is the documented
+ contract, stated rather than papered over.
+- **Reserved fields have one canonical location — enforced at every layer.**
+ The Brainy-owned field names (`RESERVED_ENTITY_FIELDS`:
+ `noun`/`subtype`/`createdAt`/`updatedAt`/`confidence`/`weight`/`service`/
+ `data`/`createdBy`/`_rev`; verb mirror `RESERVED_RELATION_FIELDS` with
+ `verb` for the type key) are now (1) a **compile error** inside any
+ `metadata` param — `add`/`update`/`relate`/`updateRelation` and the
+ matching `transact()` ops; (2) **normalized at write time** for untyped
+ callers — user-settable fields remap to their dedicated top-level param
+ (top-level wins; `update({metadata:{confidence}})` no longer silently
+ no-ops, closing a 7.x trap), system-managed fields drop with a one-shot
+ warning naming the right path; (3) **split at read time** through one
+ canonical helper, so `entity.metadata` / `relation.metadata` contain ONLY
+ custom fields on every read path — `get`, `find`, `related` (several
+ paginated/by-source/by-target paths previously echoed the full stored
+ record, including the `verb` type key, inside `metadata`), batch reads,
+ and historical `asOf()` reads. `related()` results now also surface
+ `confidence`/`updatedAt` top-level, and `updateRelation()` no longer
+ erases a relationship's `service`/`createdBy`. See "Reserved fields" in
+ `docs/concepts/consistency-model.md`.
+- **Named errors for not-found contract failures.** `update()`, `relate()`,
+ `updateRelation()`, `similar({ to: id })`, `transact()` planning, and
+ speculative `db.with()` planning now throw `EntityNotFoundError` /
+ `RelationNotFoundError` (both exported from the package root, both carrying
+ the missing `id` as a field) instead of a generic `Error`. Message texts are
+ unchanged, so existing `/not found/` matching keeps working — `instanceof`
+ is now the supported way to branch.
+- **Unchanged from 7.31:** per-entity `_rev`, `update({ ifRev })`
+ (`RevisionConflictError`), and `add({ ifAbsent })` work exactly as before,
+ and `ifRev` is also accepted on `transact()` update operations (a conflict
+ rejects the whole batch).
+
+### For provider and plugin authors
+
+The provider contracts in `@soulcraft/brainy/plugin` (`src/plugin.ts`) changed
+shape:
+
+- **`GraphIndexProvider` speaks BigInt at the boundary.** Reads take entity
+ ints and return entity/verb ints as `bigint[]`; the coordinator owns all
+ UUID ↔ int conversion. `addVerb(verb, sourceInt, targetInt)` receives the
+ resolved endpoint ints (also mirrored on the new optional
+ `GraphVerb.sourceInt` / `GraphVerb.targetInt` fields) and returns the
+ interned verb int. The batch reverse resolver
+ `verbIntsToIds(verbInts: bigint[]): Promise<(string | null)[]>` is
+ **required** — the provider owns durable verb-int interning; Brainy keeps
+ only a bounded in-memory warm cache. Implementations may stay u32 internally
+ (`Number(bigint)` narrowing is lossless under the `EntityIdSpaceExceeded`
+ guard) but must speak the bigint contract.
+- **`VersionedIndexProvider`** is a new optional 4-method capability —
+ `generation()`, `isGenerationVisible(g)`, `pin(g)`, `release(g)`, BigInt
+ generations — feature-detected on every registered index provider. Providers
+ are *post-commit appliers*: the storage-record commit is the source of
+ truth; on open, a provider behind the committed watermark replays the gap or
+ requests a rebuild. Explicit pins override any time-based snapshot retention
+ the provider has. Speculative `db.with()` overlays never reach providers. A
+ provider implementing this serves historical reads with **no rebuild** —
+ the open-core materializer is the correctness baseline, the provider is the
+ accelerator.
+- **The vector index registers under `'vector'`** and implements
+ `VectorIndexProvider`. The `'hnsw'` and `'diskann'` keys are retired and
+ never looked up. `CacheProvider`'s category union uses `'vectors'` in place
+ of `'hnsw'`.
+
+### Scale and cost — how the mechanisms behave
+
+Mechanism descriptions, not benchmark numbers (none are published for 8.0 yet):
+
+- `brain.now()` pins in O(1) and adds **zero read overhead until history
+ actually moves** — while nothing has committed past the pin, every read
+ delegates to the live fast paths.
+- A `transact()` commit pays O(ids touched) extra writes (before-images +
+ delta + manifest) — never O(store size). Single-operation writes pay an
+ in-memory counter bump with coalesced persistence.
+- `db.persist()` on filesystem storage is a hard-link farm: snapshot creation
+ copies no entity data and the snapshot shares disk space with the source.
+ Cross-device targets fall back to per-file byte copies; persisting an
+ in-memory brain serializes a real, durable, loadable store.
+- Historical **index-accelerated** queries (semantic search, traversal,
+ cursors, aggregation) on the open-core path pay a one-time O(n at the
+ pinned generation) in-memory index materialization per `Db`, cached until
+ `release()`. Record-path reads (`get`, metadata `find`, filter `related`)
+ at any reachable generation are served directly from the immutable record
+ layer. A native `VersionedIndexProvider` serves the same reads rebuild-free.
+
+The full cost model is in
+[docs/concepts/consistency-model.md](docs/concepts/consistency-model.md).
+
+### Upgrade checklist (from 7.x)
+
+1. **While still on 7.x:** back up your data directory (a plain file copy is
+ fine). If you used COW branches, materialize each branch you need — 8.0
+ does not read branch state. If you ran a cloud storage adapter, export your
+ data with 7.x tooling (`(await brain.data()).export()`) — 8.0 opens
+ `filesystem` and `memory` stores only.
+2. **Runtime:** Node.js 22 LTS or Bun ≥ 1.0.
+3. **Config:** change removed storage types to `'filesystem'`; delete
+ `storage.branch`; move `config.hnsw` / `hnswPersistMode` to
+ `config.vector` per the shape example above.
+4. **Renames:** run the sed snippet, then fix the manual shape changes.
+5. **Removed APIs:** replace `fork` / `checkout` / branch methods / `commit` /
+ `getHistory` / `streamHistory` / `versions` / `data()` per the table above.
+6. **Subtypes:** if your data predates subtype discipline, start with
+ `requireSubtype: false`, run `brain.audit()`, backfill with
+ `brain.fillSubtypes(rules)` (one rule per type — a literal default or a
+ function deriving the subtype from each entry), re-run `audit()` until
+ `total === 0`, then remove the opt-out so the 8.0 default enforcement
+ protects you going forward.
+7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` →
+ `snapshot ` / `restore ` / `history` / `generation`.
+8. **Plugin authors:** apply the contract renames and the BigInt graph
+ contract; optionally implement `VersionedIndexProvider`.
+9. **Verify, then take your first snapshot:** run your test suite, then
+ `const db = brain.now(); await db.persist('/backups/post-8.0-upgrade');
+ await db.release()`.
+
+What did **not** change: the core query surface (`find` / `search` / `get` /
+`add` / `update` / `relate` and friends), the aggregation engine, the VFS,
+neural extraction, and the on-disk entity/relationship layout — 8.0 creates
+its MVCC bookkeeping (`_system/generation.json`, `_system/manifest.json`,
+`_system/tx-log.jsonl`, `_generations/`) alongside the existing files on
+first use.
+
+### Native-provider (Cor) compatibility
+
+Brainy 8.0 pairs with `@soulcraft/cor` 3.0 — the renamed successor to the
+`@soulcraft/cortex` 2.x native provider — shipping in lockstep. The old
+`@soulcraft/cortex` 2.x cannot accelerate Brainy 8.0: 8.0 never consults the
+`'hnsw'` / `'diskann'` provider keys, and the graph/column provider contracts
+are BigInt at the boundary. Upgrade both packages together (`@soulcraft/brainy@8`
++ `@soulcraft/cor@3`).
+
+---
+
+## v7.31.2 — 2026-06-09
+
+**Affected products:** none in behaviour; affects anyone reading Brainy's TypeScript
+type definitions or coreTypes for the `hnsw.quantization.bits` option. Drop-in from 7.31.1.
+
+### Fix: stale comment claimed SQ4 vector quantization required a paid native provider
+
+Brainy ships a pure-JS SQ4 distance implementation (`distanceSQ4Js` in
+`src/utils/vectorQuantization.ts`) that's been part of the open-core path the whole
+time. Two type-definition comments incorrectly stated SQ4 required a native (paid)
+provider:
+
+```ts
+// coreTypes.ts:472 + types/brainy.types.ts:1123 — before
+bits?: 8 | 4 // default: 8 (SQ8). SQ4 requires cortex native.
+```
+
+The comment was simply wrong — open-core users can use SQ4 today, and the native SIMD
+acceleration is a drop-in via `setSQ4DistanceImplementation`, not a hard requirement.
+Corrected to:
+
+```ts
+bits?: 8 | 4 // default: 8 (SQ8). SQ4 has a pure-JS implementation;
+ // cortex's distance:sq4 SIMD provider accelerates it.
+```
+
+Both locations updated. Pure documentation; no behaviour change. Caught during an
+upstream open-core-boundary audit — thanks to whoever read the types carefully enough
+to spot the misleading comment.
+
+### Cortex compatibility
+
+Zero changes required. The fix is documentation only; runtime behaviour was already
+correct (`vectorQuantization.ts:412` ships `distanceSQ4Js`).
+
+---
+
+## v7.31.1 — 2026-06-09
+
+**Affected products:** any consumer running `@soulcraft/brainy` against `FileSystemStorage`
+that triggers concurrent flush / compaction (e.g. explicit `brain.flush()` calls overlapping
+with periodic background compaction, or a backup job + a flush job racing). Production-impacting
+when present: `brain.flush()` throws `ENOENT` on rename, downstream jobs that rely on flush
+(GCS / S3 / Azure backups, snapshot exports) fail continuously. Drop-in from 7.31.0.
+
+### Fixes a same-key rename race in `saveBinaryBlob`
+
+`FileSystemStorage.saveBinaryBlob` used a bare `${filePath}.tmp` suffix for its atomic-write
+temp file. Two concurrent same-key calls computed the **same** temp path; both `writeFile`d,
+the first `rename` succeeded, the second `rename` fired against a missing temp and threw
+`ENOENT`. The throw propagated up through `brain.flush()` and broke any downstream job that
+called it.
+
+```
+[job-queue] gcs-backup: failed — ENOENT: no such file or directory,
+ rename '/data/brainy-data/.../_column_index/owner/DELETED.bin.tmp'
+ -> '/data/brainy-data/.../_column_index/owner/DELETED.bin'
+```
+
+**Patch:** unique per-writer temp suffix (matches the pattern at six other atomic-write
+sites in the same file) + ENOENT swallow on rename (defensive: if the temp is gone, the
+work has already landed; `saveBinaryBlob` is idempotent for a given key) + temp cleanup
+on any other rename failure (no orphan `.tmp.*` files).
+
+```ts
+// before (7.31.0)
+const tmpPath = filePath + '.tmp'
+await fs.promises.writeFile(tmpPath, data)
+await fs.promises.rename(tmpPath, filePath)
+
+// after (7.31.1)
+const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
+await fs.promises.writeFile(tmpPath, data)
+try {
+ await fs.promises.rename(tmpPath, filePath)
+} catch (err) {
+ if ((err as NodeJS.ErrnoException).code === 'ENOENT') return
+ await fs.promises.unlink(tmpPath).catch(() => {})
+ throw err
+}
+```
+
+### Scope
+
+The fix is one site — `src/storage/adapters/fileSystemStorage.ts:992-999`. Audit results:
+
+- **`FileSystemStorage`** — six sibling atomic-write sites already used unique suffixes;
+ only `saveBinaryBlob` was the outlier. All six were rechecked. Clean.
+- **`OPFSStorage`** — writes via WritableStream (no tmp+rename). Not affected.
+- **`GCSStorage` / `R2Storage` / `AzureBlobStorage` / `S3CompatibleStorage`** — use
+ object-store `PUT` (atomic at the API). Not affected.
+- **`MemoryStorage`** — in-memory. Not affected.
+- **`HistoricalStorageAdapter`** — read-only. Not affected.
+- **COW / versioning / snapshot / HNSW / aggregation** — all delegate to storage adapters
+ via `saveBinaryBlob` / `writeObjectToPath`. They get the fix automatically by virtue of
+ using the patched primitive.
+
+### Beneficiary surfaces (broader than the reported bug)
+
+Column-store compaction (`_column_index//DELETED.bin`, segment files) is what the
+production report named, but `saveBinaryBlob` is also used by HNSW connection persistence
+(`_hnsw_conn/` — `src/hnsw/hnswIndex.ts:252`). If two flush paths ever raced on
+HNSW persistence for the same node, they would have hit the same ENOENT. No production
+reports of HNSW-side failures, but the fix removes the latent race.
+
+### Tests
+
+- **New `tests/integration/savebinaryblob-concurrent-rename.test.ts`** (4 tests):
+ - 20 concurrent `saveBinaryBlob(sameKey, ...)` calls all resolve without throwing
+ - No orphan `.tmp.*` siblings remain in the blob directory after concurrent writes
+ - Production-shape: two concurrent compactor passes over 10 column-index fields
+ (`owner`, `path`, `permissions`, `vfsType`, `modified`, `createdAt`, `accessed`,
+ `updatedAt`, `mimeType`, `size`)
+ - Single-writer path still produces correct bytes (no regression)
+- The first three tests reproduce the production `ENOENT: rename '...DELETED.bin.tmp' ->
+ '...DELETED.bin'` error verbatim on the pre-7.31.1 code path. Verified by stashing the
+ fix and watching the tests fail with the exact production error.
+- Existing suites unchanged: 1468/1468 unit. All integration suites pass.
+
+### Cortex compatibility
+
+Zero changes required. The bug and the fix are pure JS in the filesystem storage adapter;
+the Cortex mmap-filesystem adapter wraps `FileSystemStorage` and inherits the patch
+automatically.
+
+### Forward-compat
+
+The bare-`.tmp` pattern is gone repo-wide. 8.0's `Db.persist()` will route through the
+same patched primitive; no additional work needed when the immutable Db API ships.
+
+---
+
+## v7.31.0 — 2026-06-09
+
+**Affected products:** consumers that need multi-writer coordination (job
+schedulers, idempotent state machines, distributed locks, optimistic UI updates)
+or idempotent bootstrap of well-known singletons. Additive; drop-in from 7.30.2.
+
+### Per-entity `_rev` + `update({ ifRev })` + `add({ ifAbsent })`
+
+CouchDB / PouchDB / ETag-style optimistic concurrency. Every entity now carries a
+monotonic `_rev: number` that Brainy auto-bumps on every successful `update()`.
+Pass it back as `ifRev` to make read-modify-write race-safe without an external
+lock service. `ifAbsent` adds by-ID idempotent insert for singletons / config rows
+/ deterministic-ID bootstraps.
+
+**What's new:**
+
+- **`entity._rev: number`** — initialized to `1` on `add()`, bumped by `1` on every
+ successful `update()` that reaches storage. Surfaced on `get()` (both fast metadata-
+ only and full-vector paths), `find()`, `search()`, and on the `Result` flatten layer
+ for backward compatibility. Pre-7.31.0 entities without `_rev` are read as `1`.
+- **`update({ id, ..., ifRev: number })`** — optimistic-concurrency check. When
+ provided, throws `RevisionConflictError` if the persisted `_rev` no longer matches.
+ Carries `{ id, expected, actual }` for principled recovery. Omitting `ifRev` keeps
+ the prior unconditional-update behavior.
+- **`add({ id, ifAbsent: true })`** — by-ID idempotent insert. Returns the existing
+ `id` without writing if the entity already exists. No throw, no overwrite. Ignored
+ when `id` is omitted (a fresh UUID can never collide).
+- **`addMany({ items, ifAbsent: true })`** — applies the flag to every item; per-item
+ `ifAbsent` overrides the batch flag.
+
+### The lock recipe (single-process or distributed)
+
+```ts
+import { Brainy, RevisionConflictError } from '@soulcraft/brainy'
+
+const LOCK_ID = '...uuid...'
+
+await brain.add({
+ id: LOCK_ID,
+ type: NounType.Document,
+ data: { owner: null, expiresAt: 0 },
+ ifAbsent: true
+})
+
+async function tryAcquireLock(workerId: string, ttlMs: number) {
+ const lock = await brain.get(LOCK_ID)
+ if (!lock) throw new Error('lock document missing')
+ const state = lock.data as { owner: string | null; expiresAt: number }
+ if (state.owner && state.expiresAt > Date.now()) return false
+
+ try {
+ await brain.update({
+ id: LOCK_ID,
+ data: { owner: workerId, expiresAt: Date.now() + ttlMs },
+ ifRev: lock._rev
+ })
+ return true
+ } catch (err) {
+ if (err instanceof RevisionConflictError) return false
+ throw err
+ }
+}
+```
+
+### Why this is scoped to `_rev` + `ifAbsent`
+
+A public `brain.transaction(fn)` wrapper was on the table but was cut. The internal
+`TransactionManager` exposes raw `Operation` classes that take a `StorageAdapter`
+constructor argument; a high-level facade in 7.31.0 would have meant either
+(a) delegating to `brain.add()` / `update()` / `relate()` — each of which commits
+its own internal transaction before the closure returns, so multi-write rollback
+wouldn't actually work (a footgun), or (b) threading `tx?` through every internal
+write path — ~2 days of refactor with regression surface, and the API shape changes
+again in 8.0 anyway.
+
+The SDK-scheduler use case that drove the thread (read-then-CAS lock pattern) is
+fully solved by `_rev` + `ifRev`. Multi-write atomicity is the 8.0 `brain.transact()`
+use case, where the Datomic-style immutable Db makes it atomic by construction. We
+chose not to ship a worse version in the interim.
+
+### How `_rev` interacts with branches and the snapshot API
+
+| System | What it tracks | When it advances |
+|---|---|---|
+| `_rev` (NEW) | Per-entity write counter | Auto, on every successful `update()` |
+| `brain.versions.save()` | Named snapshots per entity | Explicit — you call `save()` |
+| `brain.fork()` / branches | Whole-brain copy-on-write | Explicit — you call `fork(name)` |
+| VFS file versioning | Per-VFS-file snapshots | Same as `brain.versions.save()` |
+
+`_rev` is independent. Each branch has its own copy of every entity, so each
+branch has its own `_rev` per entity (same as every other field under COW).
+Snapshots taken via `brain.versions.save()` capture the entity at that moment
+including its `_rev` at that time; the snapshot's own `version: number` is the
+snapshot index, distinct from `_rev`.
+
+### Docs
+
+- **New `docs/guides/optimistic-concurrency.md`** (public, indexed at
+ soulcraft.com/docs after portal deploy) — full reference: the lock pattern,
+ read-modify-write retry, idempotent bootstrap, how `_rev` interacts with branches /
+ snapshots / VFS, what's coming in 8.0.
+- `docs/api/README.md` `add()` and `update()` entries get the new params + tips
+ pointing at the guide.
+
+### Tests
+
+- **New `tests/integration/rev-and-ifabsent.test.ts`** (18 tests): rev initialization,
+ rev bump on update, ifRev pass / fail / omit / legacy-entity-fallback, ifAbsent
+ with custom id / no id / batch propagation / per-item override, plus a full
+ SDK-scheduler-style two-concurrent-CAS-updaters scenario.
+- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-enforcement
+ 30/30, strict-mode-self-test 13/13, find-limits 9/9. Unit 1468/1468.
+
+### Cortex compatibility
+
+Zero changes required. `_rev` is a metadata column already supported by
+`NativeColumnStore`; the auto-bump runs in Brainy JS before any storage call.
+Cortex never sees the per-entity counter — it's a single integer field updated
+alongside every other metadata field on the existing write paths.
+
+### 8.0 forward-compat
+
+`_rev`, `ifRev`, `RevisionConflictError`, and `ifAbsent` all survive the 8.0 Db
+redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for
+whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record
+patterns, generation-based for "did the world move under me."
+
+---
+
+## v7.30.2 — 2026-06-08
+
+**Affected products:** any consumer with `find({ limit })` call sites that pass values
+≥ ~9000 — a common safety-cap pattern in production code (`limit: 10_000` against
+type-filtered queries that typically return 10–500 entities). Additive; drop-in
+from 7.30.1.
+
+### Why
+
+Brainy 7.30 introduced a memory-derived cap on `find({ limit })` to prevent OOM
+from runaway queries. The cap was sound in intent but **~4× too conservative in
+calibration**: the formula assumed 100 KB per result while typical entity footprint
+is 7-10 KB (384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
+free-memory box this capped `limit` at 9000, breaking production dashboards where
+cascading 500s from queries with `limit: 10_000` silently degraded the `Promise.all`
+loading the canvas.
+
+7.30.2 recalibrates the formula and changes the enforcement from synchronous-throw
+to two-tier (warn-then-throw) so existing pre-7.30 code doesn't break on upgrade.
+
+### Recalibrated formula — `25 KB` per result (was `100 KB`)
+
+The auto-configured cap now uses a realistic per-result size:
+
+| Memory source | 7.30.0–7.30.1 cap | 7.30.2 cap |
+|---|---|---|
+| 4 GB Cloud Run container | 10 000 | 40 000 |
+| 2 GB container | 5 000 | 20 000 |
+| 900 MB free system memory | 9 000 | ~36 000 |
+| `reservedQueryMemory: 1 GB` | 10 000 | 40 000 |
+
+Hard ceiling stays at 100 000. Existing `BrainyConfig.maxQueryLimit` /
+`reservedQueryMemory` overrides continue to work unchanged.
+
+### Two-tier enforcement — warn, then throw
+
+**Below cap (`limit ≤ maxLimit`):** silent pass. No signal, no friction. Unchanged.
+
+**Soft tier (`maxLimit < limit ≤ 2 × maxLimit`)** *NEW*: one-time warning per call
+site (dedup keyed on stack location + limit value), query proceeds. Pre-7.30.2 code
+that relied on the cap silently allowing safety-cap limits keeps working — the
+warning teaches the recipe so consumers can fix it intentionally.
+
+**Hard tier (`limit > 2 × maxLimit`)** *NEW*: throw with the same message format the
+warning uses. Real OOM territory; the cap stops being a recommendation and becomes
+a guardrail.
+
+The 2× soft margin is chosen to absorb existing safety-cap patterns (`limit: 10_000`
+against a 9 K-cap box) without disabling OOM protection. Real OOM territory on a JS
+in-memory brain is hundreds of thousands of results, not 10× the safety cap.
+
+### Improved error / warning message (mirrors the 7.30.1 enforcement-error format)
+
+Before (7.30 → 7.30.1):
+```
+limit exceeds auto-configured maximum of 9000 (based on available memory)
+```
+
+After (7.30.2):
+```
+find({ limit: 10000 }) exceeds the auto-configured query limit of 9000 (basis:
+available free memory). Choose one:
+ • Increase the cap: new Brainy({ maxQueryLimit: 10000 })
+ • Reserve more memory: new Brainy({ reservedQueryMemory: 256000000 })
+ • Paginate: split the query with { limit, offset } pages
+ at OrderService.loadDashboard (/app/src/orders/dashboard.ts:142:18)
+Docs: https://soulcraft.com/docs/guides/find-limits
+```
+
+Caller location comes from the same `findCallerLocation()` helper introduced in
+7.30.1 (now extracted to `src/utils/callerLocation.ts` so both subtype enforcement
+and limit enforcement share it).
+
+### New docs
+
+New `docs/guides/subtypes-and-facets.md`-style guide at
+`docs/guides/find-limits.md` (public, indexed) — explains the cap, the four memory
+sources the auto-config considers, the three escape valves (`maxQueryLimit`,
+`reservedQueryMemory`, pagination), and when to use which. Explicit "pagination is
+the future-proof pattern" callout — the cap can get tighter in 8.0 (Datomic-style
+`Db.find()` may make per-call limits stricter to keep snapshot semantics cheap),
+but pagination keeps working unchanged.
+
+`docs/api/README.md` `find()` entry gets a one-paragraph `limit` tip + pointer
+to the new guide.
+
+### Tests
+
+- **New `tests/integration/find-limits.test.ts`** (9 tests): below-cap silent
+ pass; soft-tier warns once per call site (dedup verified by exercising same vs.
+ different source lines); soft-tier message format (names all three escape
+ valves + docs link); soft-tier message includes caller location; hard-tier
+ throws; hard-tier message format same as soft-tier; consumer `maxQueryLimit`
+ override raises the cap and shifts both tiers accordingly; **the pre-7.30.2
+ regression scenario** explicitly covered — `limit: 10_000` against a
+ `maxQueryLimit: 9000` cap now warns and passes instead of throwing.
+- Existing unit tests in `tests/unit/utils/memoryLimits.test.ts` updated to
+ reflect the recalibrated formula (5 tests: hardcoded values for
+ container / reserved / free-memory paths bumped 4×).
+- Existing `paramValidation.test.ts` "should auto-limit based on system memory"
+ test extended to cover the new three-tier semantics (below-cap pass /
+ soft-tier silent / hard-tier throw).
+- All prior suites still green: subtype-and-facets 26/26, verb-subtype-and-
+ enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
+
+### Cortex compatibility
+
+**No Cortex changes required for 7.30.2.** Every change is JS-side: formula
+recalibration runs in `ValidationConfig.constructor()`, two-tier enforcement runs
+in `validateFindParams()`, both fire before any storage/index/Cortex call. Cortex
+2.x and the pending Cortex 3.0 work both stay compatible without code changes.
+
+The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()`
+may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point
+whose rollout staging now also covers query limits.
+
+### What consumers should do
+
+- **If you've been carrying a `limit: 9000` workaround for the 7.30.0–7.30.1
+ cap:** drop it on bump to 7.30.2 — existing `limit: 10_000` patterns now
+ warn (teaching the recipe) but no longer throw. The warning is
+ one-time-per-call-site, so production logs don't spam.
+- **All other consumers:** no action required if no enforcement-error was being
+ hit. If you see new `[Brainy]` warnings in logs after upgrade, follow the
+ recipe in the warning or read `docs/guides/find-limits.md`.
+- **SDK wrappers:** wrapper-level pools that auto-construct `Brainy` instances
+ should consider surfacing `maxQueryLimit` / `reservedQueryMemory` as
+ consumer-facing config; Brainy now documents the knob explicitly.
+
+---
+
+## v7.30.1 — 2026-06-08
+
+**Affected products:** anyone running a brain where a platform layer (SDK, framework wrapper)
+has registered `brain.requireSubtype()` rules on common NounTypes, AND anyone preparing for
+the upcoming Brainy 8.0 default-on strict mode. Additive; drop-in from 7.30.0. No behavior
+change for consumers not using strict-mode enforcement.
+
+### Why
+
+Production incident 2026-06-08: a consumer using SDK 3.20.0 (which registers
+`requireSubtype()` rules on `NounType.{Event, Collection, Message, Contract, Media, Document}`)
+saw their booking flow start returning 500s because `brain.add({ type: NounType.Event, ... })`
+calls in their codebase lacked `subtype`. An audit of Brainy's OWN source revealed 14 HIGH-risk
+internal write paths that also omit subtype — VFS move/copy/symlink edges, aggregation
+materializer, neural extraction, importers, integrations (Sheets/OData), MCP client, CLI.
+Any consumer running `requireSubtype()` rules on those NounTypes was one step away from breaking
+Brainy's own infrastructure paths, not just their own code. 7.30.1 closes both gaps before
+8.0 ships and makes strict mode the default.
+
+### New — `brain.audit()` diagnostic
+
+Find entities and relationships missing a `subtype` value, grouped by type. The companion to
+`migrateField()` (and to 8.0's `fillSubtypes()`): answers "what would break if I enabled
+strict subtype enforcement?".
+
+```typescript
+const report = await brain.audit()
+// {
+// entitiesWithoutSubtype: { event: 24, document: 3 },
+// relationshipsWithoutSubtype: { relatedTo: 1402 },
+// total: 1429,
+// scanned: 8400,
+// recommendation: 'Found 1429 entries without subtype. Migrate via `brain.migrateField()`
+// (7.x) — or wait for `brain.fillSubtypes()` (8.0) which closes the same
+// gap with caller-supplied rules.'
+// }
+```
+
+VFS infrastructure entities are excluded by default (they bypass enforcement via
+`metadata.isVFSEntity` markers). Pass `{ includeVFS: true }` to surface them.
+
+### New — Improved enforcement error messages
+
+The error fired when subtype enforcement rejects a write now includes:
+
+1. **The caller's source location** — extracted from the JavaScript stack so you see your own
+ call site, not a Brainy internal frame. Eliminates the "grep your repo for `brain.add`" step.
+2. **Specific guidance** — points at the registered vocabulary when one exists; mentions
+ brain-wide strict mode and the `except` escape valve when not; otherwise the
+ `brain.requireSubtype()` registration recipe.
+3. **A documentation link** — `https://soulcraft.com/docs/guides/subtypes-and-facets#strict-mode`
+ for the canonical migration recipe.
+
+Before:
+```
+add(): NounType.Event requires subtype but got undefined. Register vocabulary via brain.requireSubtype().
+```
+
+After:
+```
+add(): NounType.event requires subtype but got undefined.
+ at OrderService.getOrCreateByToken (/app/src/orders/draft.ts:42:23)
+ Pass one of: standard, recurring, milestone.
+ Migration recipe: https://soulcraft.com/docs/guides/subtypes-and-facets#strict-mode
+```
+
+### Internal subtype labels — Brainy's own infrastructure paths
+
+Every internal Brainy write path now sets a stable, queryable `subtype`. Consumers don't need
+to do anything for these — they're documented here so you can query Brainy-managed data:
+
+| Code path | NounType / VerbType | Subtype label |
+|---|---|---|
+| VFS root directory `/` | `Collection` | `'vfs-root'` |
+| VFS subdirectories | `Collection` | `'vfs-directory'` |
+| VFS files | mime-driven (e.g. `Document`/`Code`/`Image`) | `'vfs-file'` |
+| VFS symlinks | `File` | `'vfs-symlink'` (NEW — distinct from `'vfs-file'`) |
+| VFS Contains edges (create + move/copy/symlink/batch) | `Contains` | `'vfs-contains'` |
+| Aggregation materialized output | `Measurement` | `'materialized-aggregate'` |
+| Import-source provenance entity | `Document` | `'import-source'` |
+| Importer-extracted entities | extractor-driven type | `'imported'` |
+| Importer placeholder targets | `Thing` | `'import-placeholder'` |
+| Neural extraction | extractor-driven type | `'extracted'` |
+| GoogleSheets API entity writes | request-driven | `'imported-from-sheets'` |
+| OData API entity writes | request-driven | `'imported-from-odata'` |
+| MCP message storage | `Message` | `'mcp-message'` |
+| `brainy add` CLI default | user-supplied type | `'cli-add'` |
+| `brainy relate` CLI default | user-supplied verb | `'cli-relate'` |
+
+Query examples:
+
+```typescript
+// Every VFS-managed file in your brain
+await brain.find({ subtype: 'vfs-file' })
+
+// Document breakdown — distinguishes import-source from extracted/imported/user content
+brain.counts.bySubtype(NounType.Document)
+// → { 'import-source': 12, 'imported': 847, 'extracted': 34, 'vfs-file': 102, ... }
+```
+
+### Caller-supplied `defaultSubtype` on importers and extraction
+
+Importer + extraction paths now accept a caller-supplied `defaultSubtype` config so consumers
+can tag a whole batch with their own provenance label instead of the Brainy default:
+
+```typescript
+// SmartImportOrchestrator / ImportCoordinator / NeuralImport all accept this
+await brain.importer.import(file, {
+ defaultSubtype: 'customer-upload-2026q2', // your batch label
+ // ...
+})
+```
+
+Precedence: extractor-set subtype (highest) → caller's `defaultSubtype` → Brainy default
+(`'imported'` for importers, `'extracted'` for extractors).
+
+### CLI `--subtype` flag
+
+`brainy add` and `brainy relate` gain a `--subtype ` (`-s`) flag for use with
+strict-mode brains:
+
+```bash
+brainy add "Avery Brooks — runs the AI lab" --type person --subtype employee
+brainy relate alice manages bob --subtype direct
+```
+
+When the flag isn't supplied, the CLI uses `'cli-add'` / `'cli-relate'` as defaults so
+ad-hoc CLI usage still works against strict-mode brains.
+
+### Cortex compatibility
+
+**No Cortex changes required for 7.30.1.** Every change is JS-side: internal subtype labels are
+arbitrary strings stored transparently by Cortex; `brain.audit()` runs purely on JS via existing
+`storage.getNouns()` / `getVerbs()` pagination; the CLI is JS-only; error messages fire in
+Brainy before any native call.
+
+**For Cortex 3.0 (forward-looking, not blocking):**
+
+- **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)).
+ A native implementation reading from a "null-subtype" bitmap in the column store would be
+ O(buckets).
+- **Strict-mode parity test.** Cortex should mirror Brainy's new
+ `tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any
+ latent bug where native writes bypass JS validation.
+- **Reserved-label awareness (optional).** Brainy's internal labels (`'vfs-*'`,
+ `'materialized-aggregate'`, `'imported'`, `'extracted'`, `'mcp-message'`, `'cli-*'`) become
+ a documented part of the 8.0 contract; useful for telemetry that surfaces "X% of entities are
+ Brainy-managed infrastructure".
+
+### Docs
+
+Updated `docs/guides/subtypes-and-facets.md` with a new "Strict mode in practice" section
+covering the SDK_CORE_VOCABULARY pattern, a 4-step migration recipe, the Brainy-internal label
+reference table, and an 8.0 forward-look. `docs/api/README.md` documents `brain.audit()` and
+adds a strict-mode tip to the `add()` / `relate()` reference entries.
+
+### Tests
+
+- **New `tests/integration/strict-mode-self-test.test.ts`** (13 tests): creates a brain under
+ a realistic consumer vocabulary shape + brain-wide strict mode, then exercises every
+ internal Brainy path (VFS root/mkdir/writeFile/cp/mv/ln, aggregation engine, audit
+ diagnostic, error-message UX). Zero rejections expected.
+- Existing 7.30.0 + 7.29.0 integration suites unchanged: 26/26 + 30/30.
+- Unit suite unchanged: 1468/1468.
+
+---
+
+## v7.30.0 — 2026-06-05
+
+**Affected products:** consumers modeling typed relationships with sub-classification
+(direct vs dotted-line management; spouse / sibling / colleague; collaborator vs competitor;
+etc.), and anyone wanting to enforce the pairing of `type` + `subtype` on every write.
+Additive; drop-in from 7.29.x. No deprecations.
+
+### Symmetric — `subtype` on relationships (parity with 7.29.0 nouns)
+
+`subtype?: string` is now a first-class standard field on every relationship, mirroring the
+noun-side work shipped in 7.29.0. Verbs and nouns are now first-class peers — every API
+available on the noun side has a verb-side mirror.
+
+```typescript
+const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' })
+const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' })
+
+await brain.relate({
+ from: ceoId,
+ to: vpId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct' // sub-classification on the edge
+})
+```
+
+**Read & filter:**
+
+```typescript
+// Fast-path filter — column-store hit, not metadata fallback
+const direct = await brain.getRelations({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct'
+})
+
+// Set membership
+const all = await brain.getRelations({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: ['direct', 'dotted-line']
+})
+
+// Traversal filter (depth-1 in JS; multi-hop lands on Cortex native)
+const reports = await brain.find({
+ connected: { from: ceoId, via: VerbType.ReportsTo, subtype: 'direct', depth: 1 }
+})
+```
+
+### New — `updateRelation()` closes a long-standing gap
+
+Verbs previously had no update method — the only way to change a relationship was
+delete-then-recreate, which lost the relation id. 7.30 ships `brain.updateRelation()`:
+
+```typescript
+await brain.updateRelation({ id: relId, subtype: 'dotted-line' })
+await brain.updateRelation({ id: relId, weight: 0.5, confidence: 0.9 })
+
+// Change verb type — re-indexes in graph adjacency, id preserved
+await brain.updateRelation({ id: relId, type: VerbType.WorksWith })
+```
+
+### New — O(1) verb subtype counts via the persisted rollup
+
+`_system/verb-subtype-statistics.json` mirrors the noun-side rollup shipped in 7.29.0.
+Per-VerbType-per-subtype counts are maintained incrementally and persisted; reads are O(1):
+
+```typescript
+brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
+// → { direct: 12, 'dotted-line': 3 }
+
+brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct') // O(1) point
+// → 12
+
+brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
+// → [['direct', 12], ['dotted-line', 3]]
+
+brain.relationshipSubtypesOf(VerbType.ReportsTo)
+// → ['direct', 'dotted-line']
+```
+
+### New — `brain.requireSubtype(type, options)` per-type enforcement
+
+Unified API for noun OR verb types — register specific types as requiring a subtype,
+optionally with a fixed vocabulary:
+
+```typescript
+brain.requireSubtype(NounType.Person, {
+ values: ['employee', 'customer', 'vendor'],
+ required: true
+})
+
+brain.requireSubtype(VerbType.ReportsTo, {
+ values: ['direct', 'dotted-line'],
+ required: true
+})
+
+// Now this throws — Person requires subtype:
+await brain.add({ type: NounType.Person, data: 'no subtype' })
+
+// And this throws — 'matrix' isn't in the registered vocabulary:
+await brain.relate({ from: a, to: b, type: VerbType.ReportsTo, subtype: 'matrix' })
+```
+
+### New — Brain-wide strict mode
+
+`new Brainy({ requireSubtype: true })` enforces subtype on every public write across the
+whole brain. Composes with per-type rules; per-type rules win when both apply.
+
+```typescript
+// Every write must include subtype
+const brain = new Brainy({ requireSubtype: true })
+
+// Exempt specific types (e.g. catch-all Thing)
+const brain2 = new Brainy({
+ requireSubtype: { except: [NounType.Thing, NounType.Custom] }
+})
+```
+
+When strict mode is on:
+- Every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()`
+ rejects writes missing a subtype on a non-exempt type.
+- `addMany()` and `relateMany()` validate every item BEFORE any storage write —
+ atomic-fail semantics, no partial writes.
+- Brainy's own infrastructure writes (VFS root, directories, files) bypass via the
+ `metadata.isVFSEntity: true` marker so existing consumers' VFS usage continues to work.
+
+The brain-wide flag becomes the default in 8.0.0; the type-level `subtype` field becomes
+required at the type system level. The full 8.0 contract upgrade is coordinated through the
+internal platform handoff (`CTX-SUBTYPE-8.0-CONTRACT`).
+
+### `migrateField()` extended to verbs
+
+The migration helper shipped in 7.29.0 now walks verbs too via the new `entityKind` option:
+
+```typescript
+// Migrate verb-side metadata.kind → top-level subtype
+await brain.migrateField({
+ from: 'metadata.kind',
+ to: 'subtype',
+ entityKind: 'verb'
+})
+
+// Or walk nouns and verbs in one pass
+await brain.migrateField({
+ from: 'metadata.kind',
+ to: 'subtype',
+ entityKind: 'both'
+})
+```
+
+Default is `entityKind: 'noun'` (backward-compatible).
+
+### Symmetry — noun + verb capability matrix
+
+7.30 closes every gap between nouns and verbs. Every capability available on the noun side
+has a verb-side mirror:
+
+| Capability | Nouns | Verbs |
+|---|---|---|
+| `subtype` top-level field | ✓ (7.29) | ✓ (7.30 new) |
+| Standard-field set | `STANDARD_ENTITY_FIELDS` | `STANDARD_VERB_FIELDS` (new) |
+| Field resolver helper | `resolveEntityField` | `resolveVerbField` (new) |
+| Statistics rollup | `_system/subtype-statistics.json` | `_system/verb-subtype-statistics.json` (new) |
+| Counts breakdown | `counts.bySubtype` / `topSubtypes` / `subtypesOf` | `counts.byRelationshipSubtype` / `topRelationshipSubtypes` / `relationshipSubtypesOf` (new) |
+| Fast-path filter | `find({type, subtype})` | `getRelations({type, subtype})` + `find({connected, subtype})` (new) |
+| Update method | `update()` | `updateRelation()` (new — closed pre-7.30 gap) |
+| Migration helper | `migrateField()` | `migrateField({entityKind: 'verb'\|'both'})` (new) |
+| Enforcement | `requireSubtype(NounType, ...)` | `requireSubtype(VerbType, ...)` — one unified API |
+| Brain-wide strict mode | `new Brainy({ requireSubtype })` covers both | same |
+
+### Cortex compatibility
+
+Verb subtype works under Cortex out of the box via auto-field-indexing. Cortex parity items
+for the verb side (native query planner recognition, `verbSubtypeCountsByType` native rollup,
+per-edge subtype on native graph adjacency for multi-hop traversal filtering) ship in the
+next Cortex release. Not a Brainy blocker.
+
+### Docs
+
+Full guide: `docs/guides/subtypes-and-facets.md` (extended with Layer V + Enforcement
+sections). The verb subtype + enforcement APIs are documented in `docs/api/README.md`.
+
+---
+
+## v7.29.0 — 2026-06-04
+
+**Affected products:** anyone modeling entities with per-product sub-classification — every
+consumer that's been reaching for `metadata.kind`, a custom `metadata.subtype` field, a
+`data.kind` shape inside the payload, or similar ad-hoc conventions. Additive; drop-in from
+7.28.x. No deprecations.
+
+### New — `subtype` promoted to a top-level standard field
+
+`subtype?: string` is now a first-class standard field on every entity, alongside `type` /
+`confidence` / `weight`. Use it as the platform-standard primitive for sub-classifying entities
+within a NounType (`Person` → `'employee'` / `'customer'`; `Document` → `'invoice'` / `'contract'`;
+`Event` → `'milestone'` / `'meeting'`):
+
+```typescript
+await brain.add({
+ data: 'Avery Brooks — runs the AI lab',
+ type: NounType.Person,
+ subtype: 'employee', // top-level write param
+ metadata: { department: 'ai-lab' }
+})
+
+// Fast-path filter — column-store hit, not metadata fallback:
+const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
+
+// Set membership:
+const internal = await brain.find({
+ type: NounType.Person,
+ subtype: ['employee', 'contractor']
+})
+```
+
+Flat string, no hierarchy — your vocabulary, your choice. Brainy stores and counts, never validates.
+
+### New — O(1) subtype counts via the persisted rollup
+
+A new `_system/subtype-statistics.json` rollup is maintained incrementally as entities are added,
+updated, and deleted — mirroring the existing `nounCountsByType` machinery with the same self-heal
+behavior on poison detection. Counts are O(1) at billion scale:
+
+```typescript
+brain.counts.bySubtype(NounType.Person)
+// → { employee: 12, customer: 847, vendor: 34 }
+
+brain.counts.bySubtype(NounType.Person, 'employee') // O(1) point count
+// → 12
+
+brain.counts.topSubtypes(NounType.Person, 3)
+// → [['customer', 847], ['employee', 12], ['vendor', 34]]
+
+brain.subtypesOf(NounType.Person)
+// → ['customer', 'employee', 'vendor']
+```
+
+### New — `brain.trackField()` for other metadata facets
+
+For facets that aren't the *primary* sub-classification (`status`, `source`, `role`, `paradigm`),
+`trackField` registers a field for cardinality + per-NounType breakdown stats without promoting
+each one to a top-level slot. Piggybacks on the existing aggregation engine, so backfill-on-define
+applies — registering on a populated brain scans existing entities on the first query:
+
+```typescript
+brain.trackField('status', { perType: true })
+
+await brain.counts.byField('status')
+// → { todo: 12, doing: 3, done: 47 }
+
+await brain.counts.byField('status', { type: NounType.Task })
+// → { todo: 8, doing: 2, done: 30 }
+
+// Opt-in vocabulary validation:
+brain.trackField('priority', { values: ['low', 'medium', 'high'] })
+// brain.add({ ..., metadata: { priority: 'urgent' } }) → throws
+```
+
+### New — generic `brain.migrateField()` for one-shot rewrites
+
+Streams every entity and copies a value from one field path to another. Supports top-level
+standard fields, `metadata.*`, and `data.*` paths; idempotent; with optional `readBoth: true`
+deprecation window that preserves the source field alongside the new one:
+
+```typescript
+// Phase 1: dual-populate (legacy readers still work)
+await brain.migrateField({
+ from: 'metadata.kind',
+ to: 'subtype',
+ readBoth: true
+})
+
+// ...readers migrate to subtype at their own pace...
+
+// Phase 2: clear the source field
+await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
+// → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] }
+```
+
+Supports `batchSize` and `onProgress` for large brains.
+
+### Aggregation composition
+
+`subtype` is a standard-field group-by dimension out of the box — no `where:` wrapper, no
+metadata-fallback overhead:
+
+```typescript
+await brain.find({
+ aggregate: {
+ groupBy: ['type', 'subtype'],
+ metrics: { count: { op: 'COUNT' } }
+ }
+})
+// [
+// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 },
+// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 },
+// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 },
+// ...
+// ]
+```
+
+### Cortex compatibility
+
+Subtype works under Cortex out of the box via auto-field-indexing. Cortex will land its own
+native-side query-planner recognition + `subtypeCountsByType` native rollup in the next Cortex
+release for full parity with `type`'s fast path. Not a Brainy blocker — subtype reads/writes
+function correctly with current Cortex.
+
+### Docs
+
+Full guide: `docs/guides/subtypes-and-facets.md`. Subtype is documented as a core primitive
+throughout `README.md`, `docs/DATA_MODEL.md`, `docs/architecture/finite-type-system.md`,
+`docs/api/README.md`, and `docs/QUERY_OPERATORS.md`.
+
+---
+
+## v7.24.0 — 2026-05-26
+
+**Affected products:** aggregation/reporting users + anyone calling `extractEntities` on
+multi-entity text. Additive; drop-in from 7.23.x.
+
+### New — array-unnest `groupBy` (tag frequency / faceted counts)
+
+A `groupBy` dimension can now be `{ field, unnest: true }`: the field holds an array and the
+entity contributes once per **distinct** element. Enables tag-frequency / label-count style
+aggregates:
+
+```typescript
+brain.defineAggregate({
+ name: 'tag_frequency',
+ source: { type: NounType.Document },
+ groupBy: [{ field: 'tags', unnest: true }],
+ metrics: { count: { op: 'count' } }
+})
+// queryAggregate('tag_frequency', { orderBy: 'count', order: 'desc' })
+```
+
+Duplicate elements on one entity count once; an entity with an empty/missing array joins no group.
+
+### Perf — entity extraction batch-embeds candidates
+
+`extractEntities` / `extractConcepts` now embed all unique candidate spans in a single
+`embedBatch` call instead of one `embed()` per candidate (N sequential model calls before). No
+behavior change — and with vectors reliably available, the embedding signal more consistently
+reinforces correct types (e.g. clearer Organization confidence). Falls back to per-candidate
+embedding if a batch call fails.
+
+---
+
+## v7.23.0 — 2026-05-26
+
+**Affected products:** anyone using aggregation (stats/dashboards), graph traversal, or entity
+extraction. Adds two report APIs and fixes three correctness gaps surfaced by BR-ADV-FEATURES-BUN.
+Drop-in upgrade from 7.22.x.
+
+### New — `brain.queryAggregate(name, params)`
+
+A first-class report API returning the clean `AggregateResult[]` shape
+(`{ groupKey, metrics, count }[]`) directly, instead of the search-`Result` wrapper that
+`find({ aggregate })` returns. Accepts `where` / `having` / `orderBy` / `order` / `limit` / `offset`.
+
+### New — HAVING (filter groups by metric value)
+
+`find({ aggregate, having: { revenue: { greaterThan: 1000 } } })` (and the same on
+`queryAggregate`) filters groups by their computed metrics — the analytics equivalent of SQL
+`HAVING`, complementing `where` (which filters group keys). Evaluated per group: **O(groups),
+independent of entity count.**
+
+### Fix — aggregates now backfill when defined over existing data (R1)
+
+Defining an aggregate on a store that already holds matching entities returned `[]` because
+write-time hooks only saw *future* writes. It now backfills from existing entities on first query
+(one-time scan via `getNouns`, then incremental) — storage-agnostic, so it works under durable
+backends (Cortex) where a brain reopens pre-populated. Also: `groupBy:['noun']` now resolves to
+the entity type instead of a single null group. `find({ aggregate })` rows now expose
+`groupKey`/`metrics`/`count` at the top level (previously only under `.metadata`).
+
+### Fix — multi-hop `find({ connected: { depth, via } })` (traversal)
+
+`depth` and `via` are now honored at **every hop** (previously only the immediate neighbour was
+returned, and verb filtering applied to hop 1 only). The BFS is bounded by `limit`.
+
+### Fix — entity extraction type accuracy (R3)
+
+`extractEntities` no longer lets a type indicator in one candidate's surrounding text bleed onto
+neighbours (e.g. "Corp" in "Sarah Chen founded Acme Corp" no longer types "Sarah Chen" as
+Organization). Each candidate is typed by its own span; context may only reinforce the same type.
+
+---
+
+## v7.22.1 — 2026-05-26
+
+**Affected products:** Anyone using `extractEntities()` / `extractConcepts()`, native
+aggregation (`find({ aggregate })`), or multi-hop graph traversal
+(`find({ connected: { depth } })`). Three advanced-API correctness fixes — all reproduce on
+Node, so they were **not** Bun-specific despite the original report (BR-ADV-FEATURES-BUN).
+
+### Entity/concept extraction no longer returns `[]`
+
+`extractEntities()` / `extractConcepts()` returned empty for entity-rich text. The SmartExtractor
+ensemble scored agreeing signals with a weighted **sum** against an absolute 0.60 gate, so a
+confident low-weight signal (e.g. a name pattern at 0.82) lost selection to a mediocre
+high-weight signal that then failed the gate — dropping the whole result. It now selects and
+gates on a normalized weighted **average**. Also: the `confidence` option now actually controls
+the threshold (was a dead hardcoded 0.60), and the embedding-signal timeout was raised
+100ms → 2000ms so the neural signal isn't silently dropped on slower runtimes.
+
+*Known limitation:* type accuracy on dense multi-entity sentences is still imperfect — a strong
+indicator in one entity's surrounding text can bleed onto neighbours. Tracked separately.
+
+### `find({ aggregate })` exposes `groupKey` / `metrics` / `count`
+
+Aggregation always computed correct values, but rows nested them under `.metadata`, so callers
+expecting the documented `AggregateResult` shape saw empty-looking rows. Those three fields are
+now also present at the top level of each result row.
+
+### Multi-hop `find({ connected: { depth } })` honours depth
+
+Previously returned only the immediate (1-hop) neighbour at any depth because the graph-search
+path ignored `depth` / `via`. It now performs the full depth-aware traversal.
+
+---
+
+## v7.22.0 — 2026-05-15
+
+**Affected products:** All. Fixes a silent-data-loss class affecting `find()` and
+`brain.stats()`, plus Cortex-compatibility regression from 7.21.0. Recommended
+upgrade for everyone on 7.20.x or 7.21.x.
+
+### Two correctness fixes
+
+#### 1. `find({ where })` no longer silently returns `[]` (BR-FIND-WHERE-ZERO)
+
+The 7.20.0 column-store refactor deleted the legacy sparse-index write path
+but left `getStats()` and `getIds()` reading from sparse indices. New
+workspaces had no sparse-index files, so:
+- `brain.stats().entityCount` was `0` regardless of actual entity count
+- `find({ where: { ... } })` returned `[]` regardless of actual matches
+- `rebuildIndexesIfNeeded()` saw `0` entries → fired the `[Brainy] CRITICAL ...
+ Second rebuild result: 0 entries` log → application saw no indexed data
+
+7.22.0 makes the ColumnStore the single source of truth post-7.20.0:
+- `MetadataIndex.getStats()` reads from `ColumnStore.getIndexedFields()` and
+ `idMapper.size`. Entity count is now accurate across writer → close →
+ reader-open cycles.
+- `MetadataIndex.getIdsFromChunks()` throws `BrainyError(FIELD_NOT_INDEXED)`
+ when neither column store nor legacy sparse index has the field.
+- `MetadataIndex.getIdsForFilter()` catches per-clause, logs
+ `[brainy] find() where-clause referenced unindexed field(s): ...`, returns
+ `[]`. Production `find()` is now consistent with `brain.explain()`'s
+ `path: 'none'` diagnostic.
+
+Separately, `BaseStorage.getNounType()` was hardcoded to return `'thing'`
+since a type-cache removal in commit `42ae5be`. Every noun save attributed
+the entity to `'thing'` regardless of its actual type, poisoning
+`_system/type-statistics.json` and `brain.stats().entitiesByType`. 7.22.0
+restores type-correct attribution:
+- New `nounTypeByIdCache: Map` on `BaseStorage`, populated
+ in `saveNounMetadata_internal` and consumed in `saveNoun_internal`.
+- New `flushCounts()` override that persists `nounCountsByType` /
+ `verbCountsByType` alongside the per-entity counter. Readers opening the
+ same directory after a writer flush now see correct per-type counts.
+
+**Self-heal at init.** If `loadTypeStatistics()` detects the poisoned-state
+signature (all nouns attributed to `'thing'` with multiple non-`thing`
+entities on disk), it auto-runs `rebuildTypeCounts()` once and rewrites the
+file. A `[BaseStorage] Detected poisoned type-statistics.json signature ...`
+warning is logged. Operators don't need to take any action — the first 7.22
+open of a directory poisoned by 7.20.0/7.21.0 fixes it.
+
+**`brainy inspect repair`** can also be used to manually trigger
+`rebuildTypeCounts()`. It's now public on `BaseStorage`.
+
+#### 2. Cortex 2.2.x no longer crashes 7.21.0 boot (BR-DEFENSIVE-INTERFACE)
+
+7.21.0 added new storage-adapter methods (`supportsMultiProcessLocking`,
+`acquireWriterLock`, etc.) and called them unconditionally. Cortex 2.2.0's
+mmap storage adapter bundles a pre-7.21 `BaseStorage` and doesn't define
+them, so a consumer's 7.21.0 upgrade attempt threw
+`TypeError: this.storage.supportsMultiProcessLocking is not a function`
+at boot.
+
+7.22.0 introduces `hasStorageMethod(name)` on Brainy and guards every new-
+method call site. Older adapters degrade with a one-line warning at init:
+```
+[brainy] Storage adapter `CortexMmapStorage` predates the 7.21
+multi-process methods. Writer locking and the flush-request RPC are
+disabled for this directory. Upgrade the plugin (e.g.
+`@soulcraft/cortex` ≥2.3.0) for full enforcement.
+```
+
+### Dead-code cleanup
+
+Removed `MetadataIndexManager.dirtyChunks`, `dirtySparseIndices`, and the
+`flushDirtyMetadata()` no-op machinery. These accumulators stopped being
+populated when the sparse-index write path was deleted in 7.20.0; the
+remaining 6 callers wasted async hops on an empty Map iteration.
+
+### Upgrade notes
+
+- **Behaviour change:** `find({ where: { unindexedField: ... } })` previously
+ returned `[]` silently. It now logs a one-line warning and still returns
+ `[]`. The diagnostic message names the field — use
+ `brain.explain({ where: { ... } })` for full diagnosis. No code change
+ needed for callers that already handle empty results.
+- **Behaviour change (good):** `brain.stats()` and `brain.health()` now
+ report accurate per-type counts. If you previously relied on the buggy
+ `entitiesByType.thing` over-count, update consumers.
+- **No API breaks.** Pure additive on the public surface.
+
+### Cortex consumers
+
+Cortex 2.2.x continues to work against 7.22.0 thanks to the defensive
+guards, but for full multi-process protection on Cortex-backed stores and
+to make sure Cortex's native MetadataIndex picks up the find-where-zero
+fix, Cortex 2.3.0 is recommended. Tracked as `CX-MULTIPROC-METHOD` in the
+internal platform handoff.
+
+---
+
+## v7.21.0 — 2026-05-15
+
+**Affected products:** All consumers using filesystem storage (production deploys,
+local development).
+
+### Multi-process safety + first-class read-only inspector
+
+Filesystem storage now enforces **single-writer, many-reader**. Previously, opening
+a second writer process against a live data directory silently produced wrong query
+results — no error, no warning, just empty or stale data. This release replaces
+that footgun with a hard refusal at init time and a dedicated read-only inspection
+mode.
+
+#### New: `Brainy.openReadOnly()`
+```ts
+const reader = await Brainy.openReadOnly({
+ storage: { type: 'filesystem', rootDirectory: '/data/brain' }
+})
+const bookings = await reader.find({ where: { entityType: 'booking' } })
+```
+- Does NOT acquire the writer lock — coexists with a live writer.
+- Every mutation method throws `Cannot mutate a read-only Brainy instance`.
+- `flush()` and `close()` are safe (no-op + clean shutdown).
+
+#### New: writer lock at init
+- `new Brainy({ ... })` (default `mode: 'writer'`) acquires
+ `/locks/_writer.lock` containing `{ pid, hostname, startedAt, lastHeartbeat, version }`.
+- A second writer on the same directory **throws** with the PID, hostname,
+ heartbeat, and a pointer to `openReadOnly()`.
+- Stale-lock detection: same hostname + dead PID OR heartbeat > 60s ago → overwritten with a warning.
+- Heartbeat: writer rewrites `lastHeartbeat` every 10s (unref'd timer).
+- Override: pass `{ force: true }` to bypass when you've verified the existing lock is stale.
+
+#### New: `brain.requestFlush({ timeoutMs })`
+Cross-process RPC for inspectors to force a fresh snapshot:
+- In-process: just calls `flush()`.
+- Out-of-process: writes a request file, polls for ack. Times out gracefully.
+- Watcher runs in every writer instance (filesystem only).
+
+#### New: `brain.stats()`
+Operator-facing summary: `entityCount`, `entitiesByType`, `relationCount`,
+`relationsByType`, `fieldRegistry`, `indexHealth`, `storage.backend`, `writerLock`, `version`.
+Designed for `/api/health` endpoints and incident triage.
+
+#### New: `brain.explain(findParams)`
+Shows which index path serves each `where` clause: `column-store` | `sparse-chunked` | `none`.
+**This is the answer to "why is `find()` returning empty?"** — `path: "none"` means
+the field has no index entries and the query will silently return `[]`. Includes
+notes on likely causes (writer hasn't flushed; typo; field genuinely absent).
+
+#### New: `brain.health()`
+Invariant-check battery: HNSW vs metadata count parity, field registry sanity,
+`_seeded` entity sweep, writer heartbeat freshness. Returns `{ overall: 'pass' | 'warn' | 'fail', checks: [...] }`.
+
+#### New: `brainy inspect` CLI (13 subcommands)
+All read-only by default (internally uses `openReadOnly()`):
+```
+brainy inspect stats
+brainy inspect find --type Event --where '{"status":"paid"}' --limit 20
+brainy inspect get
+brainy inspect relations --direction both
+brainy inspect explain --where '{"entityType":"booking"}'
+brainy inspect health
+brainy inspect sample --type Event --n 20
+brainy inspect fields
+brainy inspect dump --type Event > backup.jsonl
+brainy inspect watch --type Event
+brainy inspect backup /backups/brain.tar
+brainy inspect repair # writer mode — stop live writer first
+brainy inspect diff
+```
+Default behaviour: ask the writer to flush via the RPC first (skip with `--no-fresh`).
+
+### Brainy + Cortex
+
+Cortex segments (`*.cidx`) are immutable mmap files; `MANIFEST.json` uses atomic
+rename. The single writer lock at `/locks/_writer.lock` covers both Brainy
+and Cortex. Read-only inspectors mmap Cortex segments with zero coordination.
+
+### What's NOT enforced yet
+
+- **Cloud storage backends** (S3, GCS, R2, Azure) — no multi-process locking.
+ A best-effort warning logs in writer mode against non-filesystem backends.
+- **The `find({ where })`-returns-0 root-cause bug** — tracked separately. The
+ new `brain.explain()` and `brain.health()` surface it loudly
+ (`path: 'none'`, `index-parity warn`) instead of letting it be silent.
+
+### Upgrade notes
+
+- **Default behaviour change:** opening a Brainy directory in writer mode while
+ another writer is live now THROWS where it previously silently produced wrong
+ results. If you have scripts that intentionally co-opened a writer directory,
+ switch them to `Brainy.openReadOnly()` or pass `{ force: true }`.
+- **Cloud backends unchanged** — no lock acquisition for S3/GCS/R2/Azure.
+- All existing APIs unchanged. Pure addition.
+
+### Docs
+
+- README "Single-Writer Model" section.
+- `docs/concepts/multi-process.md` — full model + Cortex compatibility.
+- `docs/guides/inspection.md` — operator recipes.
+- JSDoc on every new API.
+
+---
+
+## v7.19.10 — 2026-02-24
+
+**Affected products:** All Bun/ESM consumers
+
+### ESM crypto fix in SSTable
+
+Replaced `require('crypto')` with `import { createHash } from 'node:crypto'` in the
+SSTable implementation. Fixes a crash in Bun and strict ESM environments where
+CommonJS `require` is unavailable.
+
+No API changes — upgrade and redeploy.
+
+---
+
+## v7.19.2 — 2026-02-18
+
+**Affected products:** All
+
+### Metadata index cleanup on delete
+
+Fixed: metadata indexes were not cleaned up after `delete()` / `deleteMany()`. Stale
+index entries could cause phantom results in metadata-filtered queries after deletion.
+
+No API changes. If you were seeing ghost results in filtered queries, this fixes it.
+
+---
+
+## v7.18.0 — 2026-02-16
+
+**Affected products:** Analytics, reporting, and session-summary consumers
+
+### Aggregation engine
+
+New `brain.aggregate()` API — incremental SUM, COUNT, AVG, MIN, MAX with GROUP BY
+and time window support. Computes over entity collections without loading all records
+into memory.
+
+```typescript
+const result = await brain.aggregate({
+ collection: 'bookings',
+ metrics: [
+ { field: 'revenue', fn: 'SUM' },
+ { field: 'id', fn: 'COUNT' },
+ ],
+ groupBy: 'staffId',
+ timeWindow: { field: 'createdAt', from: startOfMonth, to: now },
+})
+```
+
+SDK exposure: `sdk.brainy.aggregate()` — available once SDK is updated to pass through.
+
+---
+
+## v7.17.0 — 2026-02-09
+
+**Affected products:** All (schema evolution, data migrations)
+
+### Migration system
+
+New `brain.migrate()` API with error handling, validation, and enterprise hardening.
+Run schema migrations reliably across Brainy data directories.
+
+```typescript
+await brain.migrate({
+ version: 3,
+ up: async (brain) => {
+ // transform entities, rename fields, etc.
+ },
+})
+```
+
+---
+
+## v7.16.0 — 2026-02-09
+
+**Affected products:** All
+
+### Data/metadata separation enforced + numeric range queries
+
+- Entity `data` and `metadata` fields are now strictly separated at the storage layer
+- Numeric range queries now supported in metadata filters: `{ age: { $gte: 18, $lt: 65 } }`
+- Fixes edge cases where mixed data/metadata storage caused inconsistent query results
+
+**Breaking for anyone storing numeric values in metadata and relying on range queries:**
+verify your filter syntax matches the new `$gte/$lte/$gt/$lt` operators.
+
+---
+
+## v7.15.5 — 2026-02-02
+
+**Affected products:** Anyone using `@soulcraft/cortex` plugin
+
+### Plugin opt-in clarified
+
+Cortex and other plugins are opt-in. Pass explicitly:
+```typescript
+new Brainy({ plugins: ['@soulcraft/cortex'] })
+```
+Without `plugins`, no external plugins are loaded regardless of what's installed.
+
+---
+
+## v7.15.2 — 2026-02-01
+
+**Affected products:** All (data safety)
+
+### Graph LSM flush on close
+
+Fixed: graph LSM-trees were not flushed on `brain.close()`, risking data loss across
+restarts. Graph edges written in the final seconds before shutdown are now guaranteed
+to be persisted.
+
+No API changes — upgrade immediately if running Brainy in a long-lived server process.
diff --git a/docs/ADR-001-generational-mvcc.md b/docs/ADR-001-generational-mvcc.md
new file mode 100644
index 00000000..b4c4d8dd
--- /dev/null
+++ b/docs/ADR-001-generational-mvcc.md
@@ -0,0 +1,295 @@
+# ADR-001: Generational MVCC storage and the immutable Db API
+
+**Status:** Accepted (ships in 8.0)
+**Date:** 2026-06-10
+
+## Context
+
+Before 8.0, Brainy carried two overlapping version-control subsystems: a
+copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a
+separate versioning subsystem (`versions.save/list/compare/restore/prune`),
+plus a read-only historical adapter for commit-based time travel. Together
+they were ~5,100 LOC of mechanism for one product need: *read a consistent
+past state while the store keeps moving, and snapshot/restore cheaply.*
+
+Neither subsystem gave a precise isolation guarantee. Reads raced in-place
+JSON overwrites, so a "snapshot" was only as immutable as the bytes it
+happened to share with the live store.
+
+8.0 replaces both with **one mechanism**: generational MVCC over immutable,
+generation-stamped records, exposed through a Datomic-style immutable
+database value (`Db`). The same model is implemented natively by versioned
+index providers (LSM snapshots), so semantics are identical on the pure-JS
+path and the native path.
+
+## Decision
+
+### The model
+
+- A **monotonic u64 generation counter** is the store's logical clock. It
+ advances once per committed `transact()` batch and once per
+ single-operation write (`add`/`update`/`remove`/`relate`/…), so
+ `brain.generation()` is always a meaningful watermark. It is persisted in
+ `_system/generation.json` and never reissued for anything durable.
+- `brain.now()` **pins** the current generation in O(1) and returns a `Db` —
+ an immutable view. Pins are refcounted; `db.release()` (with a
+ `FinalizationRegistry` backstop for leaked values) ends the pin.
+- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative
+ batch atomically as **exactly one generation**, with whole-store
+ compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and
+ reified transaction metadata appended to `_system/tx-log.jsonl`.
+- `brain.asOf(generation | Date | snapshotPath)` opens past state;
+ `db.with(ops)` layers a speculative in-memory overlay (never touching
+ disk, the counter, or index providers); `db.persist(path)` cuts an
+ instant snapshot; `brain.restore(path, { confirm: true })` replaces state
+ from one; `Brainy.load(path)` opens a snapshot read-only with the full
+ query surface.
+
+### Persisted layout
+
+All paths are storage-root-relative:
+
+```
+_system/generation.json { generation, updatedAt } atomic tmp+rename
+_system/manifest.json { version, generation, atomic tmp+rename
+ committedAt, horizon } (the commit point)
+_system/tx-log.jsonl one line per committed append-only
+ transact: { generation,
+ timestamp, meta? }
+_generations//tx.json the generation-N delta: immutable
+ touched noun/verb ids + meta
+_generations//prev/.json before-image of as of immutable
+ commit N (raw stored bytes;
+ null parts = file was absent)
+```
+
+**Why per-generation deltas instead of a global `id → latest generation`
+map in the manifest:** a global map makes every commit O(all ids) — the
+whole map must be rewritten to swap it atomically. The delta layout makes a
+commit O(ids touched) and keeps the manifest a fixed-size watermark, while
+point-in-time resolution stays correct (see "Read resolution" below). The
+trade is that resolution at a pinned generation scans the deltas of later
+commits — bounded by the number of commits since the pin, which is exactly
+the window compaction keeps short.
+
+Before-images are deliberately the *only* per-id records. They serve both
+roles the layer needs — the crash-recovery undo log and the point-in-time
+read source. After-images would duplicate state that is already readable
+(the canonical entity files hold the latest bytes; earlier states resolve
+from later before-images) and would double record I/O per commit.
+
+### Commit protocol (durability)
+
+`transact()` commits under a store-wide mutex:
+
+1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError`
+ before anything is staged.
+2. **Reserve** generation `N` (counter increment).
+3. **Stage the undo log:** write the before-image of every touched id plus
+ `tx.json` under `_generations/N/`, then **fsync** the files and their
+ directories. From this point, any crash is recoverable to the exact
+ pre-transaction bytes.
+4. **Execute** the planned batch through the TransactionManager (which has
+ its own operation-level rollback for in-flight failures).
+5. **Commit point:** persist the counter, then write `_system/manifest.json`
+ via atomic tmp+rename and fsync it. The rename *is* the commit: a
+ generation directory is committed if and only if `N ≤
+ manifest.generation`.
+6. Append the tx-log line (advisory metadata — a crash between 5 and 6
+ keeps the transaction).
+
+**Crash recovery (on open):** any `_generations/` directory with
+`N > manifest.generation` is an uncommitted transaction. Its before-images
+are restored to the canonical paths (idempotently — recovery itself can
+crash and rerun) and the directory is removed. Because recovery runs before
+any index is built, and a recovery that rolled something back forces a full
+index rebuild, derived indexes never observe rolled-back state. Reader-mode
+instances skip recovery (readers never write; the next writer repairs).
+
+A failed (non-crash) transaction takes the same staging directory down the
+abort path: the TransactionManager rolls back applied operations, the
+staging directory is removed, and the generation reservation is returned —
+a failed batch leaves the generation counter unchanged.
+
+### Read resolution at a pinned generation
+
+The state of id X at pinned generation G is:
+
+- the before-image stored by the **first committed generation after G that
+ touched X**, or
+- the live canonical bytes, when nothing after G touched X.
+
+While nothing has committed past G, *every* read on the `Db` delegates to
+the live fast paths untouched — `now()` adds no read overhead until history
+actually moves.
+
+**Two read paths, one result set.** `get()`, metadata-level `find()`, and
+filter-based `related()` resolve directly through the record layer at any
+reachable pinned generation — no extra cost beyond scanning the deltas of
+later commits. Index-accelerated dimensions (semantic/vector search, graph
+traversal, cursors, aggregation) are served by **at-generation index
+materialization**: the first such query on a historical `Db` copies the
+exact at-G record set (live bytes for ids untouched since the pin,
+before-images for the rest; a final reconciliation pass runs under the
+commit mutex so transactions racing the copy cannot skew it) into an
+ephemeral in-memory store and opens a read-only engine over it — the same
+vector/metadata/graph index classes the live brain uses, sharing the host's
+embedder and aggregate definitions. The handle is cached on the `Db` and
+freed by `release()`.
+
+**Cost, stated plainly:** materialization is O(n at G) time and memory,
+once per `Db`. That is the open-core price of historical index queries. A
+native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
+retained LSM segments) serves the same reads with no rebuild at all — the
+materializer is the correctness baseline, the provider is the accelerator.
+
+**The one remaining boundary.** Speculative `with()` overlays throw
+`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
+overlay entities carry no embeddings (`with()` never invokes the embedder),
+so a "full" index query over an overlay would silently exclude the
+overlay's own entities. Commit with `transact()` to get the full surface.
+
+**History granularity (Model-B).** EVERY write is its own immutable generation
+— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
+Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/
+`diff()`/`history()` exactly like transacts; a pin always freezes against later
+writes. `transact()` groups several operations into ONE atomic generation.
+
+Single-op history durability is **async group-commit**: the live write hits
+canonical storage immediately (acknowledged), while its before-image is buffered
+and persisted to disk in one batched fsync on a size/timer trigger (or forced by
+`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in
+point-in-time resolution exactly like on-disk generations, so the synchronous
+`now()` freezes with no forced flush. A hard crash before the flush loses only
+the buffered *history* of the last window — never live data — and a crash
+*mid-flush* is recovered by **drop-without-restore** (the partial generation's
+before-images are discarded, never replayed, because the live write was already
+acknowledged; restoring them would silently revert it).
+
+### Pinning, retention, compaction
+
+- Each live `Db` holds one refcounted pin on its generation (plus a
+ `pin(generation)` on every registered `VersionedIndexProvider`, whose
+ explicit pin lifetime overrides any time-based snapshot retention the
+ provider has).
+- The constructor **`retention`** knob governs auto-compaction (on `flush()`/
+ `close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config;
+ driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) ·
+ `'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit
+ CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
+ manually on the same caps — the oldest unpinned record-sets are reclaimed
+ while ANY supplied cap is exceeded.
+- A record-set `N` is reclaimed only when `N` is at or below **every** live
+ pin — deleting `N` can only break readers pinned *below* `N`, because
+ resolution reads before-images from generations strictly greater than the
+ pin. Live pins are ALWAYS exempt, in every retention mode.
+- The manifest records the **horizon** (highest reclaimed generation).
+ Generations below the horizon are unreachable; `asOf()` on them throws
+ `GenerationCompactedError`. The horizon itself stays reachable, resolved
+ from the record-sets above it. To keep a generation readable forever,
+ `persist()` it first — snapshots are self-contained.
+
+### Snapshots and restore
+
+`db.persist(path)` flushes indexes, then cuts the snapshot under the
+store's commit mutex (no commit, compaction, or counter write can
+interleave). On filesystem storage it is a **hard-link farm**: every data
+file is immutable-by-rename, so linking is safe — later rewrites swap
+inodes and the snapshot keeps the old bytes. The two exceptions are handled
+explicitly: the append-in-place tx-log is byte-copied, and process-local
+lock state is excluded. Cross-device targets (and filesystems that refuse
+links) fall back to per-file byte copies. In-memory stores serialize to the
+same directory layout, so persisting a memory brain produces a real,
+durable, loadable store.
+
+`persist()` requires the view to still be the store's latest generation
+(a snapshot captures current bytes); a view that history has moved past
+throws rather than persisting the wrong state.
+
+`restore(path, { confirm: true })` replaces the store's contents from a
+snapshot via byte copy (never links — the snapshot stays independent),
+reloads all adapter-internal derived state, rebuilds all indexes, and
+floors the generation counter at its pre-restore value so observed
+generation numbers are never reissued. Live pins do not survive a restore;
+a warning is logged when any exist.
+
+### Versioned index providers
+
+Native index providers may implement the optional 4-method
+`VersionedIndexProvider` capability (`generation()`,
+`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the
+boundary). The locked consistency model: providers are **post-commit
+appliers**. The storage-record commit is the source of truth; provider
+index state is derived. On open, a provider behind the committed watermark
+replays the gap from storage (or requests a rebuild) — there are no
+provider rollback hooks, because uncommitted transactions are repaired at
+the storage layer before any index opens. Speculative `with()` overlays
+never reach providers.
+
+## Guarantees (and their proofs)
+
+Each stated guarantee has a test that proves it, not merely exercises it
+(`tests/integration/db-mvcc.test.ts`, plus
+`tests/unit/db/generationStore.test.ts` for the record layer in isolation):
+
+| Guarantee | Proof |
+|---|---|
+| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) |
+| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) |
+| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) |
+| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c |
+| Compaction never breaks a pinned read; release enables reclaim | proof 5 |
+| `with()` overlays touch nothing durable | proof 6 |
+| Generation monotonicity across close/reopen | proof 7 |
+| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) |
+| Balanced provider pin/release lockstep | proof 9 |
+
+One deliberate softness: single-operation generation bumps persist the
+counter coalesced (per write burst), not per write. Durable artifacts —
+records, manifests, snapshots — always persist the counter synchronously at
+their own commit points, so a crash inside the coalescing window can lose
+only counter values that nothing durable ever referenced.
+
+## Failure modes
+
+| Failure | Outcome |
+|---|---|
+| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. |
+| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. |
+| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. |
+| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
+| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
+| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
+| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
+| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
+| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
+
+## Lineage
+
+The design is an assembly of well-understood prior art, chosen for being
+boring where it counts:
+
+- **Datomic** — the database-as-a-value: an immutable `Db` you query, with
+ `with()` for speculation and reified transaction metadata instead of
+ commit messages.
+- **LMDB** — reader pins: readers never block writers; a reader's view
+ stays valid because nothing overwrites the pages (here: records) it
+ references; reclamation waits for the last reader.
+- **LSM trees / Cassandra** — immutable segments make snapshots hard links
+ and make compaction a retention policy instead of a locking problem.
+
+## Consequences
+
+- One mechanism replaces the COW and versioning subsystems (their removal
+ is the companion change to this ADR).
+- In-place branch switching (`checkout`) is gone by design; the replacement
+ is opening a persisted snapshot as a separate instance — a name→path
+ mapping where a product needs named branches.
+- Every commit pays O(ids touched) extra writes (before-images + delta +
+ manifest). Single-operation writes pay only an in-memory counter bump
+ with coalesced persistence.
+- The full query surface works at every reachable pinned generation.
+ Record-path reads (`get`, metadata `find`, filter `related`) are
+ effectively free; index-accelerated historical queries pay a one-time
+ O(n at G) materialization per `Db` on the open-core path (freed on
+ `release()`), and run rebuild-free on a native `VersionedIndexProvider`.
diff --git a/docs/API_DECISION_TREE.md b/docs/API_DECISION_TREE.md
deleted file mode 100644
index 775c0d22..00000000
--- a/docs/API_DECISION_TREE.md
+++ /dev/null
@@ -1,481 +0,0 @@
-# 🧠 Brainy API Decision Tree
-
-*Choose the right API for your use case with confidence*
-
-This guide helps you navigate Brainy's comprehensive API surface by asking the right questions to find the perfect method for your specific needs.
-
-## 🎯 Quick Start: What do you want to do?
-
-### 📝 **Adding Data**
-- **Single entity** → [`brainy.add()`](#adding-single-entities)
-- **Multiple entities** → [`brainy.addMany()`](#adding-multiple-entities)
-- **Streaming/real-time data** → [Streaming Pipeline](#streaming-data)
-
-### 🔍 **Finding Data**
-- **Natural language search** → [`brainy.find("search query")`](#natural-language-search)
-- **Structured/filtered search** → [`brainy.find({ query, where, type })`](#structured-search)
-- **Similar entities** → [`brainy.similar()`](#similarity-search)
-- **Get by ID** → [`brainy.get()`](#retrieval-by-id)
-
-### 🔗 **Relationships**
-- **Create relationships** → [`brainy.relate()`](#creating-relationships)
-- **Query relationships** → [`brainy.getRelations()`](#querying-relationships)
-- **Graph traversal** → [Graph Navigation](#graph-operations)
-
-### 📊 **Advanced Features**
-- **File management** → [VFS (Virtual File System)](#file-operations)
-- **AI-powered analysis** → [Neural API](#neural-analysis)
-- **Clustering/insights** → [Intelligence Systems](#intelligence-systems)
-
----
-
-## 🔀 Decision Tree Flow
-
-```mermaid
-graph TD
- A[What are you trying to do?] --> B[Store Data]
- A --> C[Find Data]
- A --> D[Manage Relationships]
- A --> E[Work with Files]
- A --> F[AI Analysis]
-
- B --> B1[Single Item]
- B --> B2[Multiple Items]
- B --> B3[Real-time Stream]
-
- C --> C1[I know the ID]
- C --> C2[Natural language query]
- C --> C3[Complex filters]
- C --> C4[Find similar items]
-
- D --> D1[Create relationship]
- D --> D2[Query relationships]
- D --> D3[Graph traversal]
-
- E --> E1[File operations]
- E --> E2[Knowledge-enhanced files]
-
- F --> F1[Clustering]
- F --> F2[Similarity analysis]
- F --> F3[Insights generation]
-```
-
----
-
-## 📝 Adding Data
-
-### Adding Single Entities
-
-**Use `brainy.add()` when:**
-- Adding one entity at a time
-- You need the ID immediately for further operations
-- Working with user input or real-time data
-
-```typescript
-// ✅ Perfect for single entities
-const id = await brainy.add({
- data: "New research paper on quantum computing",
- type: NounType.Document,
- metadata: { category: "research", priority: "high" }
-})
-```
-
-**Decision factors:**
-- **Single item?** → `add()`
-- **Need immediate ID?** → `add()`
-- **Interactive application?** → `add()`
-
-### Adding Multiple Entities
-
-**Use `brainy.addMany()` when:**
-- Bulk importing data
-- Processing batches (>10 items)
-- Performance is critical
-
-```typescript
-// ✅ Perfect for bulk operations
-const result = await brainy.addMany({
- items: documents.map(doc => ({
- data: doc.content,
- type: NounType.Document,
- metadata: doc.metadata
- })),
- chunkSize: 100,
- parallel: true
-})
-```
-
-**Decision factors:**
-- **Multiple items (>10)?** → `addMany()`
-- **Batch processing?** → `addMany()`
-- **Can tolerate some failures?** → `addMany()` with `continueOnError: true`
-
-### Streaming Data
-
-**Use Streaming Pipeline when:**
-- Real-time data ingestion
-- Processing large datasets that don't fit in memory
-- Need transformation during ingestion
-
-```typescript
-// ✅ Perfect for streaming
-const pipeline = brainy.streaming.pipeline()
- .transform(data => ({ ...data, processed: true }))
- .batch(50)
- .into(brainy)
-```
-
----
-
-## 🔍 Finding Data
-
-### Natural Language Search
-
-**Use `brainy.find("query string")` when:**
-- User is typing search queries
-- You want semantic understanding
-- Building search interfaces
-
-```typescript
-// ✅ Perfect for user searches
-const results = await brainy.find("documents about machine learning")
-```
-
-**Decision factors:**
-- **User-generated query?** → Natural language `find()`
-- **Semantic understanding needed?** → Natural language `find()`
-- **Search interface?** → Natural language `find()`
-
-### Structured Search
-
-**Use `brainy.find({ query, where, type })` when:**
-- Complex filtering requirements
-- Combining text search with metadata filters
-- Performance-critical searches
-
-```typescript
-// ✅ Perfect for complex queries
-const results = await brainy.find({
- query: "neural networks",
- type: NounType.Document,
- where: {
- status: "published",
- year: { $gte: 2020 }
- },
- limit: 20
-})
-```
-
-**Decision factors:**
-- **Need metadata filtering?** → Structured `find()`
-- **Performance critical?** → Structured `find()`
-- **Complex criteria?** → Structured `find()`
-
-### Similarity Search
-
-**Use `brainy.similar()` when:**
-- Finding "more like this" content
-- Recommendation systems
-- Duplicate detection
-
-```typescript
-// ✅ Perfect for recommendations
-const similar = await brainy.similar({
- to: "document-id-123",
- limit: 10,
- type: NounType.Document
-})
-```
-
-**Decision factors:**
-- **"More like this" feature?** → `similar()`
-- **Recommendations?** → `similar()`
-- **Duplicate detection?** → `similar()`
-
-### Retrieval by ID
-
-**Use `brainy.get()` when:**
-- You know the exact ID
-- Loading specific entities
-- Following relationships
-
-```typescript
-// ✅ Perfect for direct access
-const entity = await brainy.get("known-id-123")
-```
-
-**Decision factors:**
-- **Known ID?** → `get()`
-- **Direct access needed?** → `get()`
-- **Following relationships?** → `get()`
-
----
-
-## 🔗 Relationships
-
-### Creating Relationships
-
-**Use `brainy.relate()` when:**
-- Connecting two entities
-- Building knowledge graphs
-- Modeling real-world relationships
-
-```typescript
-// ✅ Perfect for connections
-await brainy.relate({
- from: "user-123",
- to: "project-456",
- type: VerbType.WorksOn,
- metadata: { role: "lead", since: "2024-01-01" }
-})
-```
-
-**Decision factors:**
-- **Connecting entities?** → `relate()`
-- **Need relationship metadata?** → `relate()`
-- **Building graphs?** → `relate()`
-
-### Querying Relationships
-
-**Use `brainy.getRelations()` when:**
-- Finding all connections for an entity
-- Exploring relationship patterns
-- Building relationship views
-
-```typescript
-// ✅ Perfect for relationship queries
-const relations = await brainy.getRelations({
- from: "user-123",
- type: VerbType.WorksOn
-})
-```
-
----
-
-## 📁 File Operations
-
-### Basic File Operations
-
-**Use VFS when:**
-- Managing files and directories
-- Need hierarchical structure
-- Building file explorers
-
-```typescript
-// ✅ Perfect for file management
-const vfs = brainy.vfs({ storage: 'filesystem' })
-await vfs.writeFile('/docs/readme.md', 'content')
-const files = await vfs.getDirectChildren('/docs')
-```
-
-**Decision factors:**
-- **File management?** → VFS
-- **Directory structure?** → VFS
-- **File explorer interface?** → VFS
-
-### Intelligent File Management
-
-**Use VFS (Semantic VFS) when:**
-- Need semantic file search
-- Want AI-powered concept extraction
-- Building smart file systems
-- Require multi-dimensional file access
-
-```typescript
-// ✅ Perfect for intelligent file systems
-const knowledgeVFS = await vfs.withKnowledge(brainy)
-const insights = await knowledgeVFS.getFileInsights('/project')
-```
-
----
-
-## 🧠 AI Analysis
-
-### Clustering
-
-**Use Neural API clustering when:**
-- Discovering data patterns
-- Organizing large datasets
-- Creating automatic categories
-
-```typescript
-// ✅ Perfect for pattern discovery
-const neural = brainy.neural()
-const clusters = await neural.cluster({
- entities: entityIds,
- k: 5,
- method: 'hierarchical'
-})
-```
-
-### Intelligence Systems
-
-**Use Triple Intelligence when:**
-- Complex multi-criteria searches
-- Advanced relationship queries
-- Performance-critical operations
-
-```typescript
-// ✅ Perfect for complex queries
-const intelligence = brainy.getTripleIntelligence()
-const results = await intelligence.query({
- vector: queryVector,
- metadata: { category: 'research' },
- graph: { connected: 'user-123' }
-})
-```
-
----
-
-## 🚀 Performance Optimization Guide
-
-### When Performance Matters
-
-| Scenario | Best Choice | Why |
-|----------|-------------|-----|
-| **Bulk Import** | `addMany()` | Batched operations, parallel processing |
-| **Metadata-only Search** | `find({ where: {...} })` | Skips vector computation |
-| **Known ID Access** | `get()` | Direct index lookup |
-| **Large Result Sets** | Pagination with `offset`/`limit` | Memory efficient |
-| **Real-time Streams** | Streaming Pipeline | Memory efficient, scalable |
-
-### Memory Usage Optimization
-
-```typescript
-// ❌ Memory intensive
-const allResults = await brainy.find({ limit: 10000 })
-
-// ✅ Memory efficient
-for (let offset = 0; offset < total; offset += 100) {
- const batch = await brainy.find({
- query: "...",
- limit: 100,
- offset
- })
- await processBatch(batch)
-}
-```
-
----
-
-## 🎯 Common Use Case Patterns
-
-### Building a Search Interface
-
-```typescript
-// User types query → Natural language search
-const searchResults = await brainy.find(userQuery)
-
-// User applies filters → Structured search
-const filteredResults = await brainy.find({
- query: userQuery,
- where: selectedFilters,
- type: selectedTypes
-})
-
-// User clicks "more like this" → Similarity search
-const similar = await brainy.similar({ to: selectedId })
-```
-
-### Building a Recommendation System
-
-```typescript
-// 1. Get user's interaction history
-const user = await brainy.get(userId)
-
-// 2. Find similar users
-const similarUsers = await brainy.similar({ to: userId, type: NounType.Person })
-
-// 3. Get their liked content
-const recommendations = []
-for (const similarUser of similarUsers) {
- const relations = await brainy.getRelations({
- from: similarUser.id,
- type: VerbType.Likes
- })
- recommendations.push(...relations)
-}
-```
-
-### Building a Knowledge Graph
-
-```typescript
-// 1. Add entities
-const entities = await Promise.all([
- brainy.add({ data: "Person: Alice", type: NounType.Person }),
- brainy.add({ data: "Company: TechCorp", type: NounType.Organization }),
- brainy.add({ data: "Project: AI Assistant", type: NounType.Thing })
-])
-
-// 2. Create relationships
-await brainy.relate({
- from: entities[0], // Alice
- to: entities[1], // TechCorp
- type: VerbType.WorksFor
-})
-
-await brainy.relate({
- from: entities[0], // Alice
- to: entities[2], // AI Assistant
- type: VerbType.WorksOn
-})
-
-// 3. Query the graph
-const aliceConnections = await brainy.getRelations({ from: entities[0] })
-```
-
----
-
-## 🔧 Migration Guide
-
-### From v2.x to v3.x APIs
-
-| v2.x (Deprecated) | v3.x (Current) | When to Use |
-|-------------------|----------------|-------------|
-| `brain.store()` | `brainy.add()` | Adding entities |
-| `brain.search()` | `brainy.find()` | Searching content |
-| `brain.query()` | `brainy.find({ ... })` | Complex queries |
-| `brain.similar()` | `brainy.similar()` | ✅ Same API |
-| `brain.connect()` | `brainy.relate()` | Creating relationships |
-
-### Legacy Type Migration
-
-```typescript
-// ❌ v2.x way
-import { ISenseAugmentation } from '@soulcraft/brainy/types/augmentations'
-
-// ✅ v3.x way
-import { BrainyAugmentation } from '@soulcraft/brainy'
-```
-
----
-
-## 🎪 Decision Quick Reference
-
-**Need to add data?**
-- 1 item → `add()`
-- Many items → `addMany()`
-- Streaming → Pipeline
-
-**Need to find data?**
-- Know ID → `get()`
-- Natural search → `find("query")`
-- Complex filters → `find({ query, where })`
-- Similar items → `similar()`
-
-**Need relationships?**
-- Create → `relate()`
-- Query → `getRelations()`
-- Complex graph → Triple Intelligence
-
-**Need files?**
-- Basic → VFS (standard operations)
-- Smart → Semantic VFS (6 dimensional access + neural extraction)
-
-**Need AI analysis?**
-- Patterns → Neural clustering
-- Complex queries → Triple Intelligence
-
----
-
-*This guide covers 95% of use cases. For edge cases or custom requirements, check the [Core API Patterns](./CORE_API_PATTERNS.md) and [Neural API Patterns](./NEURAL_API_PATTERNS.md) guides.*
\ No newline at end of file
diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md
deleted file mode 100644
index e9b6711b..00000000
--- a/docs/API_REFERENCE.md
+++ /dev/null
@@ -1,1867 +0,0 @@
-# 🧠 Brainy 3.0 Complete API Reference
-
-> The neural database that thinks - Complete API documentation for all public methods
-
-## Table of Contents
-
-- [Quick Start](#quick-start)
-- [Core API](#core-api)
-- [Batch Operations](#batch-operations)
-- [Search & Discovery](#search--discovery)
-- [Security API](#security-api)
-- [Configuration API](#configuration-api)
-- [Data Management API](#data-management-api)
-- [Query API](#query-api)
-- [Neural API](#neural-api)
-- [NLP API](#nlp-api)
-- [Streaming Pipeline API](#streaming-pipeline-api)
-- [Type Definitions](#type-definitions)
-
----
-
-## Quick Start
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-// Initialize
-const brain = new Brainy({
- storage: { type: 'filesystem', path: './brainy-data' },
- model: { type: 'fast', precision: 'Q8' }
-})
-await brain.init()
-
-// Add entities
-const id = await brain.add({
- data: 'John Smith is a software engineer',
- type: NounType.Person,
- metadata: { role: 'engineer' }
-})
-
-// Search
-const results = await brain.find('engineers')
-
-// Clean up
-await brain.close()
-```
-
----
-
-## Core API
-
-### `new Brainy(config?: BrainyConfig)`
-Creates a new Brainy instance.
-
-**Parameters:**
-- `config` - Optional configuration object
-
-**Returns:** `Brainy` instance
-
-**Example:**
-```typescript
-const brain = new Brainy({
- storage: { type: 'filesystem', options: { path: './data' } },
- model: { type: 'accurate', precision: 'FP32' },
- cache: { maxSize: 5000, ttl: 600000 }
-})
-```
-
----
-
-### `async init(): Promise`
-Initializes the brain, loading models and preparing storage.
-
-**Must be called before any other operations.**
-
-**Example:**
-```typescript
-await brain.init()
-```
-
----
-
-### `async add(params: AddParams): Promise`
-Adds a new entity to the brain.
-
-**Parameters:**
-- `data` (required) - Content to embed and store
-- `type` (required) - NounType classification
-- `metadata` - Custom metadata object
-- `id` - Custom ID (auto-generated if not provided)
-- `vector` - Pre-computed embedding vector
-- `service` - Service name for multi-tenancy
-- `confidence` - Type classification confidence (0-1) ✨ *New in v4.3.0*
-- `weight` - Entity importance/salience (0-1) ✨ *New in v4.3.0*
-- `writeOnly` - Skip validation for high-speed ingestion
-
-**Returns:** Entity ID
-
-**Example:**
-```typescript
-const id = await brain.add({
- data: 'Important meeting notes from Q4 planning',
- type: NounType.Document,
- metadata: {
- date: '2024-01-15',
- author: 'John Smith',
- tags: ['planning', 'Q4']
- },
- confidence: 0.95, // High confidence in Document classification
- weight: 0.85 // High importance
-})
-```
-
----
-
-### `async get(id: string, options?: GetOptions): Promise`
-Retrieves an entity by ID.
-
-✨ **v5.11.1 Performance Optimization**:
-- **Default (metadata-only)**: 76-81% faster, 95% less bandwidth - perfect for VFS, existence checks, metadata access
-- **Opt-in vectors**: Use `{ includeVectors: true }` when computing similarity
-
-**Parameters:**
-- `id` - Entity ID
-- `options` (optional):
- - `includeVectors?: boolean` - Include 384-dim vectors (default: false for 76-81% speedup)
-
-**Returns:** Entity object or null if not found
-
-**Entity Properties:** ✨ *Updated in v4.3.0, v5.11.1*
-- `id` - Unique identifier
-- `type` - NounType classification
-- `data` - Original content
-- `metadata` - Custom metadata
-- `vector` - Embedding vector (empty array `[]` if includeVectors: false)
-- `confidence` - Type classification confidence (0-1)
-- `weight` - Entity importance/salience (0-1)
-- `createdAt` - Creation timestamp
-- `updatedAt` - Last update timestamp
-- `service` - Service name (multi-tenancy)
-
-**Example (Metadata-Only - Default, 76-81% faster):**
-```typescript
-// Perfect for VFS, metadata access, existence checks
-const entity = await brain.get('uuid-1234')
-if (entity) {
- console.log(entity.type) // NounType.Document
- console.log(entity.metadata) // { date: '2024-01-15', ... }
- console.log(entity.vector) // [] (empty - not loaded for performance)
-}
-```
-
-**Example (Full Entity with Vectors):**
-```typescript
-// Use when computing similarity on this specific entity
-const entity = await brain.get('uuid-1234', { includeVectors: true })
-if (entity) {
- console.log(entity.vector.length) // 384 (full embeddings loaded)
- // Now can use for similarity calculations
-}
-```
-
----
-
-### `async update(params: UpdateParams): Promise`
-Updates an existing entity.
-
-**Parameters:**
-- `id` (required) - Entity ID to update
-- `data` - New content (will re-embed)
-- `type` - New type classification
-- `metadata` - New or partial metadata
-- `merge` - Merge metadata (true) or replace (false), default: true
-- `vector` - New embedding vector
-- `confidence` - Update type classification confidence ✨ *New in v4.3.0*
-- `weight` - Update entity importance/salience ✨ *New in v4.3.0*
-
-**Example:**
-```typescript
-await brain.update({
- id: 'uuid-1234',
- metadata: { status: 'reviewed' },
- confidence: 0.98, // Increase confidence after review
- weight: 0.90, // Boost importance
- merge: true // Keeps existing metadata, adds status
-})
-```
-
----
-
-### `async delete(id: string): Promise`
-Deletes an entity and all its relationships.
-
-**Parameters:**
-- `id` - Entity ID to delete
-
-**Example:**
-```typescript
-await brain.delete('uuid-1234')
-```
-
----
-
-### `async relate(params: RelateParams): Promise`
-Creates a relationship between two entities.
-
-**Parameters:**
-- `from` (required) - Source entity ID
-- `to` (required) - Target entity ID
-- `type` (required) - VerbType enum value
-- `weight` - Relationship strength (0-1), default: 1
-- `metadata` - Relationship metadata
-- `bidirectional` - Create reverse relationship
-- `service` - Service name for multi-tenancy
-- `writeOnly` - Skip validation
-
-**Validation:**
-- `from` and `to` must be different (no self-referential relationships)
-- `type` must be a valid VerbType enum value
-- `weight` if provided must be between 0 and 1
-
-**Returns:** Relationship ID
-
-**Example:**
-```typescript
-const relationId = await brain.relate({
- from: 'person-123',
- to: 'org-456',
- type: VerbType.WorksWith,
- weight: 0.95,
- metadata: { since: '2020-01-01' },
- bidirectional: true
-})
-```
-
----
-
-### `async unrelate(id: string): Promise`
-Removes a relationship.
-
-**Parameters:**
-- `id` - Relationship ID
-
-**Example:**
-```typescript
-await brain.unrelate('relation-789')
-```
-
----
-
-### `async getRelations(params?: GetRelationsParams): Promise`
-Gets relationships for entities.
-
-**Parameters:**
-- `from` - Source entity ID
-- `to` - Target entity ID
-- `type` - Filter by VerbType(s)
-- `limit` - Maximum results (default: 100)
-- `offset` - Pagination offset
-- `service` - Filter by service
-
-**Example:**
-```typescript
-// Get all relationships from an entity
-const relations = await brain.getRelations({
- from: 'person-123',
- type: [VerbType.WorksWith, VerbType.ReportsTo],
- limit: 50
-})
-```
-
----
-
-### `async close(): Promise`
-Shuts down the brain, cleaning up resources.
-
-**Example:**
-```typescript
-await brain.close()
-```
-
----
-
-## Batch Operations
-
-### `async addMany(params: AddManyParams): Promise`
-Adds multiple entities in batch.
-
-**Parameters:**
-- `items` (required) - Array of AddParams
-- `parallel` - Process in parallel (default: true)
-- `chunkSize` - Batch size (default: 100)
-- `continueOnError` - Continue if some fail
-- `onProgress` - Progress callback
-
-**Returns:** BatchResult with successful/failed counts
-
-**Example:**
-```typescript
-const result = await brain.addMany({
- items: [
- { data: 'Doc 1', type: NounType.Document },
- { data: 'Doc 2', type: NounType.Document },
- { data: 'Doc 3', type: NounType.Document }
- ],
- parallel: true,
- onProgress: (done, total) => console.log(`${done}/${total}`)
-})
-
-console.log(`Added: ${result.successful.length}`)
-console.log(`Failed: ${result.failed.length}`)
-```
-
----
-
-### `async updateMany(params: UpdateManyParams): Promise`
-Updates multiple entities in batch.
-
-**Parameters:**
-- `updates` (required) - Array of UpdateParams
-- `parallel` - Process in parallel
-- `continueOnError` - Continue on failures
-- `onProgress` - Progress callback
-
-**Example:**
-```typescript
-const result = await brain.updateMany({
- updates: ids.map(id => ({
- id,
- metadata: { processed: true },
- merge: true
- })),
- parallel: true
-})
-```
-
----
-
-### `async deleteMany(params: DeleteManyParams): Promise`
-Deletes multiple entities.
-
-**Parameters:**
-- `ids` - Specific IDs to delete
-- `type` - Delete all of a type
-- `where` - Delete by metadata filter
-- `limit` - Maximum to delete (safety limit)
-- `onProgress` - Progress callback
-
-**Example:**
-```typescript
-// Delete specific IDs
-await brain.deleteMany({
- ids: ['id1', 'id2', 'id3']
-})
-
-// Delete by type
-await brain.deleteMany({
- type: NounType.Document,
- where: { status: 'draft' },
- limit: 100
-})
-```
-
----
-
-### `async relateMany(params: RelateManyParams): Promise`
-Creates multiple relationships in batch.
-
-**Parameters:**
-- `relations` (required) - Array of RelateParams
-- `parallel` - Process in parallel
-- `continueOnError` - Continue on failures
-- `onProgress` - Progress callback
-
-**Example:**
-```typescript
-const result = await brain.relateMany({
- relations: [
- { from: 'a', to: 'b', type: VerbType.RelatedTo },
- { from: 'b', to: 'c', type: VerbType.DependsOn },
- { from: 'c', to: 'a', type: VerbType.References }
- ]
-})
-```
-
----
-
-## Search & Discovery
-
-### `async find(query: string | FindParams): Promise`
-Universal search with Triple Intelligence fusion.
-
-**Parameters:**
-- `query` - Natural language query or structured params
-- `vector` - Direct vector search
-- `type` - Filter by NounType(s)
-- `where` - Metadata filters
-- `connected` - Graph constraints
-- `near` - Proximity search
-- `fusion` - Fusion strategy and weights
-- `limit` - Maximum results
-- `offset` - Pagination offset
-- `explain` - Include score explanation
-- `service` - Filter by service
-- `writeOnly` - Skip validation
-
-**Returns:** Array of Result objects with scores
-
-**Result Properties:** ✨ *Enhanced in v4.3.0*
-- `id` - Entity ID
-- `score` - Relevance score (0-1)
-- `type` - Entity type (flattened for convenience) *Enhanced*
-- `metadata` - Entity metadata (flattened) *Enhanced*
-- `data` - Entity data (flattened) *Enhanced*
-- `confidence` - Type classification confidence (flattened) *New*
-- `weight` - Entity importance (flattened) *New*
-- `entity` - Full Entity object (preserved for backward compatibility)
-- `explanation` - Score explanation (if `explain: true`)
-
-**Example:**
-```typescript
-// Natural language search
-const results = await brain.find('recent product launches')
-
-// NEW in v4.3.0: Direct access to flattened fields
-console.log(results[0].metadata) // Direct access (convenient!)
-console.log(results[0].confidence) // Type confidence
-console.log(results[0].weight) // Entity importance
-
-// Backward compatible: Nested access still works
-console.log(results[0].entity.metadata) // Also works
-
-// Structured search with fusion
-const results = await brain.find({
- query: 'machine learning',
- type: [NounType.Document, NounType.Project],
- where: { year: 2024 },
- connected: {
- to: 'research-dept',
- via: VerbType.CreatedBy
- },
- fusion: {
- strategy: 'adaptive',
- weights: { vector: 0.5, graph: 0.3, field: 0.2 }
- },
- limit: 20,
- explain: true
-})
-
-// Access results with clean, predictable patterns
-for (const result of results) {
- console.log(`Score: ${result.score}`)
- console.log(`Type: ${result.type}`)
- console.log(`Confidence: ${result.confidence ?? 'N/A'}`)
- console.log(`Weight: ${result.weight ?? 'N/A'}`)
- console.log(`Metadata:`, result.metadata)
-}
-```
-
----
-
-### `async similar(params: SimilarParams): Promise`
-Finds similar entities using vector similarity.
-
-**Parameters:**
-- `to` (required) - Entity ID, Entity object, or Vector
-- `limit` - Maximum results (default: 10)
-- `threshold` - Minimum similarity (0-1)
-- `type` - Filter by type(s)
-- `where` - Metadata filters
-
-**Returns:** Array of Result objects (same structure as `find()`) ✨ *Enhanced in v4.3.0*
-
-**Example:**
-```typescript
-const similar = await brain.similar({
- to: 'doc-123',
- limit: 5,
- threshold: 0.8,
- type: NounType.Document
-})
-
-// NEW in v4.3.0: Access flattened fields directly
-for (const result of similar) {
- console.log(`Similarity: ${result.score}`)
- console.log(`Type: ${result.type}`) // Flattened
- console.log(`Confidence: ${result.confidence}`) // Flattened
- console.log(`Metadata:`, result.metadata) // Flattened
-}
-```
-
----
-
-### `async embed(data: any): Promise` ✨ *v7.1.0*
-Generates an embedding vector from data.
-
-**Parameters:**
-- `data` - Text, array, or object to embed
-
-**Returns:** 384-dimensional embedding vector
-
-**Example:**
-```typescript
-const vector = await brain.embed('Machine learning concepts')
-console.log(vector.length) // 384
-```
-
----
-
-### `async embedBatch(texts: string[]): Promise` ✨ *v7.1.0*
-Batch embed multiple texts efficiently.
-
-**Parameters:**
-- `texts` - Array of strings to embed
-
-**Returns:** Array of 384-dimensional vectors
-
-**Example:**
-```typescript
-const embeddings = await brain.embedBatch([
- 'Machine learning is fascinating',
- 'Deep neural networks',
- 'Natural language processing'
-])
-console.log(embeddings.length) // 3
-console.log(embeddings[0].length) // 384
-```
-
----
-
-### `async similarity(textA: string, textB: string): Promise` ✨ *v7.1.0*
-Calculate semantic similarity between two texts.
-
-**Parameters:**
-- `textA` - First text
-- `textB` - Second text
-
-**Returns:** Similarity score between 0 (different) and 1 (identical)
-
-**Example:**
-```typescript
-const score = await brain.similarity(
- 'The cat sat on the mat',
- 'A feline was resting on the rug'
-)
-console.log(score) // ~0.85 (high semantic similarity)
-```
-
----
-
-### `async neighbors(entityId: string, options?): Promise` ✨ *v7.1.0*
-Get graph neighbors of an entity.
-
-**Parameters:**
-- `entityId` - Entity to get neighbors for
-- `options.direction` - 'outgoing', 'incoming', or 'both' (default: 'both')
-- `options.depth` - Traversal depth (default: 1)
-- `options.verbType` - Filter by relationship type
-- `options.limit` - Maximum neighbors to return
-
-**Returns:** Array of neighbor entity IDs
-
-**Example:**
-```typescript
-// Get all connected entities
-const neighbors = await brain.neighbors(entityId)
-
-// Get outgoing connections only
-const outgoing = await brain.neighbors(entityId, {
- direction: 'outgoing',
- limit: 10
-})
-
-// Multi-hop traversal
-const extended = await brain.neighbors(entityId, {
- depth: 2,
- direction: 'both'
-})
-```
-
----
-
-### `async findDuplicates(options?): Promise` ✨ *v7.1.0*
-Find semantic duplicates in the database.
-
-**Parameters:**
-- `options.threshold` - Minimum similarity (default: 0.85)
-- `options.type` - Filter by NounType
-- `options.limit` - Maximum duplicate groups (default: 100)
-
-**Returns:** Array of duplicate groups with similarity scores
-
-**Example:**
-```typescript
-// Find all duplicates
-const duplicates = await brain.findDuplicates()
-
-for (const group of duplicates) {
- console.log('Original:', group.entity.id)
- for (const dup of group.duplicates) {
- console.log(` Duplicate: ${dup.entity.id} (${dup.similarity.toFixed(2)})`)
- }
-}
-
-// Find person duplicates with higher threshold
-const personDupes = await brain.findDuplicates({
- type: NounType.PERSON,
- threshold: 0.9,
- limit: 50
-})
-```
-
----
-
-### `async indexStats(): Promise` ✨ *v7.1.0*
-Get comprehensive index statistics.
-
-**Returns:**
-- `entities` - Total entity count
-- `vectors` - Total vectors in HNSW index
-- `relationships` - Total relationships in graph
-- `metadataFields` - Indexed metadata fields
-- `memoryUsage.vectors` - Vector memory in bytes
-- `memoryUsage.graph` - Graph memory in bytes
-- `memoryUsage.metadata` - Metadata index memory in bytes
-- `memoryUsage.total` - Total memory usage
-
-**Example:**
-```typescript
-const stats = await brain.indexStats()
-console.log(`Entities: ${stats.entities}`)
-console.log(`Vectors: ${stats.vectors}`)
-console.log(`Relationships: ${stats.relationships}`)
-console.log(`Memory: ${(stats.memoryUsage.total / 1024 / 1024).toFixed(1)}MB`)
-console.log(`Fields: ${stats.metadataFields.join(', ')}`)
-```
-
----
-
-### `async cluster(options?): Promise` ✨ *v7.1.0*
-Cluster entities by semantic similarity.
-
-Groups entities into clusters based on their embedding similarity using
-a greedy algorithm with HNSW-based neighbor lookup.
-
-**Parameters:**
-- `options.threshold` - Similarity threshold (default: 0.8)
-- `options.type` - Filter by NounType
-- `options.minClusterSize` - Minimum entities per cluster (default: 2)
-- `options.limit` - Maximum clusters to return (default: 100)
-- `options.includeCentroid` - Calculate cluster centroids (default: false)
-
-**Returns:** Array of clusters with entities
-
-**Example:**
-```typescript
-// Find all clusters
-const clusters = await brain.cluster()
-
-for (const cluster of clusters) {
- console.log(`${cluster.clusterId}: ${cluster.entities.length} entities`)
-}
-
-// Find document clusters with centroids
-const docClusters = await brain.cluster({
- type: NounType.Document,
- threshold: 0.85,
- minClusterSize: 3,
- includeCentroid: true
-})
-
-// Use centroids for cluster comparison
-for (const cluster of docClusters) {
- console.log(`${cluster.clusterId}: ${cluster.entities.length} entities`)
- if (cluster.centroid) {
- console.log(` Centroid dimensions: ${cluster.centroid.length}`)
- }
-}
-```
-
----
-
-### `async insights(): Promise`
-Gets statistics and insights about the data.
-
-**Returns:**
-- `entities` - Total entity count
-- `relationships` - Total relationship count
-- `types` - Count by NounType
-- `services` - List of services
-- `density` - Relationships per entity
-
-**Example:**
-```typescript
-const insights = await brain.insights()
-console.log(`Entities: ${insights.entities}`)
-console.log(`Graph density: ${insights.density.toFixed(2)}`)
-```
-
----
-
-### `async suggest(params?: SuggestParams): Promise`
-AI-powered suggestions based on current data.
-
-**Parameters:**
-- `context` - Context for suggestions
-- `type` - Filter by type(s)
-- `limit` - Maximum suggestions per category
-- `service` - Filter by service
-
-**Returns:**
-- `queries` - Suggested search queries
-- `connections` - Suggested relationships
-- `insights` - Data insights
-- `patterns` - Detected patterns
-
-**Example:**
-```typescript
-const suggestions = await brain.suggest({
- context: 'product development',
- limit: 5
-})
-
-// Use suggestions
-for (const query of suggestions.queries) {
- console.log(`Try searching: ${query}`)
-}
-```
-
----
-
-## Security API
-
-Access via: `const security = await brain.security()`
-
-### `async encrypt(data: string): Promise`
-Encrypts data using AES-256-CBC.
-
-**Example:**
-```typescript
-const encrypted = await security.encrypt('sensitive data')
-```
-
----
-
-### `async decrypt(encryptedData: string): Promise`
-Decrypts previously encrypted data.
-
-**Example:**
-```typescript
-const decrypted = await security.decrypt(encrypted)
-```
-
----
-
-### `async hash(data: string, algorithm?: 'sha256'|'sha512'): Promise`
-Creates cryptographic hash.
-
-**Example:**
-```typescript
-const hash = await security.hash('password', 'sha512')
-```
-
----
-
-### `async compare(data: string, hash: string): Promise`
-Compares data against hash (constant-time).
-
-**Example:**
-```typescript
-const matches = await security.compare('password', hash)
-```
-
----
-
-### `async generateToken(bytes?: number): Promise`
-Generates secure random token.
-
-**Example:**
-```typescript
-const token = await security.generateToken(32)
-```
-
----
-
-### `async deriveKey(password: string, salt?: string): Promise<{key: string, salt: string}>`
-Derives key from password using PBKDF2-like iterations.
-
-**Example:**
-```typescript
-const { key, salt } = await security.deriveKey('userPassword')
-```
-
----
-
-### `async sign(data: string, secret?: string): Promise`
-Signs data with HMAC-SHA256.
-
-**Example:**
-```typescript
-const signature = await security.sign(data, secret)
-```
-
----
-
-### `async verify(data: string, signature: string, secret: string): Promise`
-Verifies HMAC signature.
-
-**Example:**
-```typescript
-const valid = await security.verify(data, signature, secret)
-```
-
----
-
-## Storage Configuration
-
-Brainy supports multiple storage backends for different deployment scenarios.
-
-### Storage Types
-
-#### File System Storage (Recommended for Development)
-Persistent local storage using the filesystem.
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'filesystem',
- rootDirectory: './brainy-data'
- }
-})
-```
-
-#### Amazon S3 Storage
-Scalable cloud storage with S3.
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 's3',
- s3Storage: {
- bucketName: 'my-bucket',
- region: 'us-east-1',
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
- }
- }
-})
-```
-
-#### Cloudflare R2 Storage
-Scalable cloud storage with Cloudflare R2 (S3-compatible).
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'r2',
- r2Storage: {
- bucketName: 'my-bucket',
- accountId: process.env.CF_ACCOUNT_ID,
- accessKeyId: process.env.R2_ACCESS_KEY_ID,
- secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
- }
- }
-})
-```
-
-#### Google Cloud Storage (Native SDK) 🆕
-**Recommended for GCS deployments.** Uses native `@google-cloud/storage` SDK with automatic authentication.
-
-**Key Benefits:**
-- ✅ Application Default Credentials (ADC) - Zero config in Cloud Run/GCE
-- ✅ Better performance with native GCS optimizations
-- ✅ No HMAC key management required
-- ✅ Automatic service account integration
-
-**Option 1: Explicit Type (Recommended)**
-
-With Application Default Credentials (Cloud Run/GCE):
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs-native', // ⚠️ Must be 'gcs-native' for native SDK
- gcsNativeStorage: {
- bucketName: 'my-bucket'
- // No credentials needed - ADC automatic!
- }
- }
-})
-```
-
-With Service Account Key File:
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs-native',
- gcsNativeStorage: {
- bucketName: 'my-bucket',
- keyFilename: '/path/to/service-account.json'
- }
- }
-})
-```
-
-With Service Account Credentials Object:
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs-native',
- gcsNativeStorage: {
- bucketName: 'my-bucket',
- credentials: {
- client_email: 'service@project.iam.gserviceaccount.com',
- private_key: process.env.GCS_PRIVATE_KEY
- }
- }
- }
-})
-```
-
-**Option 2: Auto-Detection**
-
-You can omit the `type` field and let Brainy auto-detect based on the config object:
-```typescript
-const brain = new Brainy({
- storage: {
- gcsNativeStorage: {
- bucketName: 'my-bucket'
- // type defaults to 'auto', will use native SDK
- }
- }
-})
-```
-
-#### Google Cloud Storage (S3-Compatible) - Legacy
-GCS using HMAC keys for S3-compatible access. **Consider migrating to 'gcs-native' for better performance.**
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs', // ⚠️ Must be 'gcs' for S3-compatible mode
- gcsStorage: { // ⚠️ Must use 'gcsStorage' (not 'gcsNativeStorage')
- bucketName: 'my-bucket',
- region: 'us-central1',
- accessKeyId: process.env.GCS_ACCESS_KEY_ID,
- secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY,
- endpoint: 'https://storage.googleapis.com'
- }
- }
-})
-```
-
-**⚠️ Common Mistakes:**
-```typescript
-// ❌ WRONG - type/config mismatch (will fall back to memory storage)
-{
- type: 'gcs',
- gcsNativeStorage: { bucketName: 'my-bucket' }
-}
-
-// ❌ WRONG - type/config mismatch (will fall back to memory storage)
-{
- type: 'gcs-native',
- gcsStorage: { bucketName: 'my-bucket', accessKeyId: '...', secretAccessKey: '...' }
-}
-
-// ✅ CORRECT - type matches config object
-{
- type: 'gcs-native',
- gcsNativeStorage: { bucketName: 'my-bucket' }
-}
-
-// ✅ CORRECT - auto-detection
-{
- gcsNativeStorage: { bucketName: 'my-bucket' }
-}
-```
-
-### Storage Features
-
-All storage adapters support:
-- ✅ **UUID-based sharding** - 256 buckets (00-ff) for scalability
-- ✅ **Pagination** - Efficient cursor-based pagination across shards
-- ✅ **Statistics** - O(1) count operations
-- ✅ **Caching** - Multi-level cache for performance
-- ✅ **Backpressure** - Automatic flow control
-- ✅ **Throttling detection** - Adaptive retry on rate limits
-
-### Migration from HMAC to Native GCS
-
-If you're currently using `type: 'gcs'` with HMAC keys, migrating to `type: 'gcs-native'` is straightforward:
-
-**Before (S3-Compatible with HMAC):**
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs', // ⚠️ Old: S3-compatible mode
- gcsStorage: { // ⚠️ Old: HMAC credentials
- bucketName: 'my-bucket',
- accessKeyId: process.env.GCS_ACCESS_KEY_ID,
- secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY
- }
- }
-})
-```
-
-**After (Native SDK with ADC):**
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs-native', // ✅ New: Native SDK mode
- gcsNativeStorage: { // ✅ New: ADC authentication
- bucketName: 'my-bucket'
- // ADC handles authentication automatically
- }
- }
-})
-```
-
-**⚠️ Important Migration Notes:**
-
-1. **Change BOTH the type AND the config object:**
- - `type: 'gcs'` → `type: 'gcs-native'`
- - `gcsStorage` → `gcsNativeStorage`
-
-2. **Remove HMAC keys** - Not needed with ADC:
- - Remove `accessKeyId`
- - Remove `secretAccessKey`
- - Remove `region` (optional with native SDK)
-
-3. **Set up ADC in your environment:**
- ```bash
- # Cloud Run/GCE: Nothing needed, ADC is automatic
-
- # Local development:
- gcloud auth application-default login
-
- # Or set GOOGLE_APPLICATION_CREDENTIALS:
- export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
- ```
-
-**Data Migration:**
-✅ **No data migration required!** Both adapters use the same path structure:
-- `entities/nouns/vectors/{shard}/{id}.json`
-- `entities/nouns/metadata/{shard}/{id}.json`
-- `entities/verbs/vectors/{shard}/{id}.json`
-
-Simply change your code and restart your application. Existing data will work immediately.
-
----
-
-## Configuration API
-
-Access via: `const config = await brain.config()`
-
-### `async set(params): Promise`
-Sets configuration value.
-
-**Parameters:**
-- `key` (required) - Configuration key
-- `value` (required) - Value to store
-- `encrypt` - Encrypt value
-
-**Example:**
-```typescript
-await config.set({
- key: 'api.key',
- value: 'secret-key-123',
- encrypt: true
-})
-```
-
----
-
-### `async get(params): Promise`
-Gets configuration value.
-
-**Parameters:**
-- `key` (required) - Configuration key
-- `decrypt` - Decrypt if encrypted
-- `defaultValue` - Default if not found
-
-**Example:**
-```typescript
-const apiKey = await config.get({
- key: 'api.key',
- decrypt: true,
- defaultValue: 'default-key'
-})
-```
-
----
-
-### `async delete(key: string): Promise`
-Deletes configuration key.
-
----
-
-### `async list(): Promise`
-Lists all configuration keys.
-
----
-
-### `async has(key: string): Promise`
-Checks if key exists.
-
----
-
-### `async clear(): Promise`
-Clears all configuration.
-
----
-
-### `async export(): Promise>`
-Exports all configuration.
-
----
-
-### `async import(config: Record): Promise`
-Imports configuration.
-
----
-
-## Data Management API
-
-Access via: `const data = await brain.data()`
-
-### `async backup(options?: BackupOptions): Promise`
-Creates backup of all data.
-
-**Parameters:**
-- `includeVectors` - Include embeddings (default: true)
-- `compress` - Compress backup (default: false)
-
-**Example:**
-```typescript
-const backup = await data.backup({
- includeVectors: true,
- compress: true
-})
-// Save backup to file
-fs.writeFileSync('backup.json', JSON.stringify(backup))
-```
-
----
-
-### `async restore(params): Promise`
-Restores from backup.
-
-**Parameters:**
-- `backup` (required) - Backup data
-- `merge` - Merge with existing data
-- `overwrite` - Overwrite conflicts
-- `validate` - Validate backup format
-
-**Example:**
-```typescript
-const backup = JSON.parse(fs.readFileSync('backup.json'))
-await data.restore({
- backup,
- merge: false,
- overwrite: true
-})
-```
-
----
-
-### `async clear(params): Promise`
-Clears specified data types.
-
-**Parameters:**
-- `entities` - Clear entities
-- `relations` - Clear relationships
-- `config` - Clear configuration
-
-**Example:**
-```typescript
-await data.clear({
- entities: true,
- relations: true
-})
-```
-
----
-
-### `async import(params): Promise`
-Imports data from various formats with automatic detection.
-
-**Parameters:**
-- `data` (required) - Data to import (Buffer, string, array, or file path)
-- `format` - 'auto' | 'json' | 'csv' | 'excel' | 'pdf' | 'yaml' | 'text' (default: 'auto')
-- `mapping` - Field mapping configuration
-- `batchSize` - Import batch size (default: 50)
-- `validate` - Validate items before import
-- `relationships` - Extract relationships automatically (default: true)
-
-**CSV-specific options:**
-- `csvDelimiter` - Column delimiter (auto-detected if not specified)
-- `csvHeaders` - First row contains headers (default: true)
-- `encoding` - Character encoding (auto-detected if not specified)
-
-**Excel-specific options:**
-- `excelSheets` - Sheet names array or 'all' for all sheets
-
-**PDF-specific options:**
-- `pdfExtractTables` - Extract tables from PDFs (default: true)
-- `pdfPreserveLayout` - Preserve text layout (default: true)
-
-**Examples:**
-```typescript
-// Import CSV file with auto-detection
-const csvResult = await brain.import('customers.csv')
-// Auto-detects: format, encoding, delimiter, field types
-
-// Import Excel workbook
-const excelResult = await brain.import('sales-data.xlsx', {
- excelSheets: ['Q1', 'Q2'] // Import specific sheets
-})
-
-// Import PDF with table extraction
-const pdfResult = await brain.import('report.pdf', {
- pdfExtractTables: true
-})
-
-// Import data array
-const dataResult = await brain.import([
- { name: 'Alice', role: 'Engineer' },
- { name: 'Bob', role: 'Designer' }
-], {
- batchSize: 100,
- relationships: true // Auto-extract relationships
-})
-
-// Import with custom CSV delimiter
-const tsvResult = await brain.import('data.tsv', {
- format: 'csv',
- csvDelimiter: '\t'
-})
-```
-
-**Returns:**
-```typescript
-{
- success: boolean
- imported: number // Number of items successfully imported
- failed: number // Number of items that failed
- entityIds: string[] // IDs of created entities
- metadata: {
- format: string // Detected format
- encoding?: string // Detected encoding (CSV)
- delimiter?: string // Detected delimiter (CSV)
- sheets?: string[] // Processed sheets (Excel)
- pageCount?: number // Number of pages (PDF)
- }
-}
-```
-
----
-
-### `async export(params?: ExportOptions): Promise`
-Exports data in various formats.
-
-**Parameters:**
-- `format` - 'json' | 'csv' | 'parquet'
-- `filter` - Filter options
-- `includeVectors` - Include embeddings
-
-**Example:**
-```typescript
-const exported = await data.export({
- format: 'csv',
- where: { type: NounType.Document },
- includeVectors: false
-})
-```
-
----
-
-### `getStats(): StatsResult`
-Gets complete statistics about entities and relationships. All stats are **O(1) pre-calculated** - updated when entities/relationships are added/removed.
-
-**Returns:**
-```typescript
-{
- entities: {
- total: number // Total entity count
- byType: Record // Entity count by type
- }
- relationships: {
- totalRelationships: number // Total relationship/edge count
- relationshipsByType: Record // Relationship count by type
- uniqueSourceNodes: number // Number of unique source entities
- uniqueTargetNodes: number // Number of unique target entities
- totalNodes: number // Total unique entities in relationships
- }
- density: number // Relationships per entity ratio
-}
-```
-
-**Example:**
-```typescript
-const stats = brain.getStats()
-
-// Total counts (O(1) operations)
-const totalNouns = stats.entities.total
-const totalVerbs = stats.relationships.totalRelationships
-const totalRelations = stats.relationships.totalRelationships // alias
-
-// Counts by type (O(1) operations)
-const nounTypes = stats.entities.byType
-const verbTypes = stats.relationships.relationshipsByType
-
-// Graph metrics
-console.log(`Entities: ${totalNouns}`)
-console.log(`Relationships: ${totalVerbs}`)
-console.log(`Density: ${stats.density.toFixed(2)}`)
-console.log(`Types:`, Object.keys(nounTypes))
-```
-
-**Performance:**
-- ✅ All counts pre-calculated in memory
-- ✅ O(1) access time
-- ✅ Updated automatically on add/remove
-- ✅ No expensive full scans required
-
-**Note:** For more granular counting operations, see the `brain.counts` API below.
-
----
-
-## Query API
-
-Access via: `brain.query`
-
-### `async entities(params?): Promise>`
-Query entities with pagination.
-
-**Parameters:**
-- `type` - Filter by type
-- `where` - Metadata filters
-- `limit` - Page size
-- `cursor` - Pagination cursor
-- `service` - Filter by service
-
-**Example:**
-```typescript
-const result = await brain.query.entities({
- type: NounType.Document,
- where: { status: 'published' },
- limit: 50
-})
-
-// Get next page
-if (result.nextCursor) {
- const nextPage = await brain.query.entities({
- cursor: result.nextCursor
- })
-}
-```
-
----
-
-### `async relations(params?): Promise>`
-Query relationships with pagination.
-
-**Parameters:**
-- `from` - Source entity
-- `to` - Target entity
-- `type` - Relationship type
-- `limit` - Page size
-- `cursor` - Pagination cursor
-
----
-
-## Neural API
-
-Access via: `brain.neural()`
-
-### Similarity Operations
-
-#### `async similar(a, b, options?): Promise`
-Calculates similarity between items.
-
-**Example:**
-```typescript
-const similarity = await neural.similar('doc1', 'doc2', {
- explain: true
-})
-```
-
----
-
-### Clustering Operations
-
-#### `async clusters(options?): Promise`
-General purpose clustering.
-
-**Parameters:**
-- `algorithm` - 'hierarchical' | 'kmeans' | 'dbscan' | 'spectral'
-- `k` - Number of clusters (for kmeans)
-- `threshold` - Distance threshold
-- `minPoints` - Minimum points per cluster
-
-**Example:**
-```typescript
-const clusters = await neural.clusters({
- algorithm: 'kmeans',
- k: 5
-})
-```
-
----
-
-#### `async clustersByDomain(params): Promise`
-Domain-specific clustering.
-
-**Example:**
-```typescript
-const clusters = await neural.clustersByDomain({
- domain: 'technology',
- field: 'category',
- maxClusters: 10
-})
-```
-
----
-
-### Outlier Detection
-
-#### `async outliers(options?): Promise`
-Detects anomalous entities.
-
-**Parameters:**
-- `method` - 'isolation' | 'lof' | 'statistical' | 'autoencoder'
-- `threshold` - Outlier threshold (default: 2.5 std deviations)
-- `returnScores` - Return anomaly scores
-
-**Example:**
-```typescript
-const outliers = await neural.outliers({
- method: 'isolation',
- threshold: 3.0,
- returnScores: true
-})
-```
-
----
-
-### Visualization
-
-#### `async visualize(options?): Promise`
-Generates visualization data for entities.
-
-**Parameters:**
-- `layout` - 'force' | 'circular' | 'hierarchical' | 'random'
-- `dimensions` - 2D or 3D
-- `includeEdges` - Include relationships
-
-**Example:**
-```typescript
-const vizData = await neural.visualize({
- layout: 'force',
- dimensions: 3,
- includeEdges: true
-})
-```
-
----
-
-## NLP API
-
-Access via: `brain.nlp()`
-
-### `async processNaturalQuery(query: string): Promise`
-Converts natural language to structured query.
-
-**Example:**
-```typescript
-const structured = await nlp.processNaturalQuery(
- "Find all documents about AI created last month"
-)
-// Returns structured query with type, time filters, etc.
-```
-
----
-
-### `async extract(text: string, options?): Promise`
-Extracts entities from text using neural matching to NounTypes.
-
-**Parameters:**
-- `types` - Target NounTypes to extract
-- `confidence` - Minimum confidence (0-1)
-- `includeVectors` - Include embeddings
-- `neuralMatching` - Use neural type matching (default: true)
-
-**Returns:** Array of extracted entities with proper NounType classification
-
-**Example:**
-```typescript
-const entities = await nlp.extract(
- "John Smith from Microsoft visited New York on Jan 15",
- {
- types: [NounType.Person, NounType.Organization, NounType.Location],
- confidence: 0.7,
- neuralMatching: true
- }
-)
-// Returns:
-// [
-// { text: "John Smith", type: NounType.Person, confidence: 0.92 },
-// { text: "Microsoft", type: NounType.Organization, confidence: 0.88 },
-// { text: "New York", type: NounType.Location, confidence: 0.85 }
-// ]
-```
-
----
-
-### `async sentiment(text: string, options?): Promise`
-Analyzes text sentiment.
-
-**Parameters:**
-- `granularity` - 'document' | 'sentence' | 'aspect'
-- `aspects` - Aspects to analyze
-
-**Returns:**
-- `overall` - Document sentiment (-1 to 1)
-- `sentences` - Sentence-level sentiment
-- `aspects` - Aspect-based sentiment
-
-**Example:**
-```typescript
-const sentiment = await nlp.sentiment(
- "The product quality is excellent but the price is too high",
- {
- granularity: 'aspect',
- aspects: ['quality', 'price']
- }
-)
-// Returns:
-// overall: { score: 0.2, label: 'mixed' }
-// aspects: {
-// quality: { score: 0.9, magnitude: 0.8 },
-// price: { score: -0.7, magnitude: 0.7 }
-// }
-```
-
----
-
-## Streaming Pipeline API
-
-Access via: `brain.stream()`
-
-### Source Operations
-
-#### `source(generator): Pipeline`
-Sets data source.
-
-**Example:**
-```typescript
-async function* dataGenerator() {
- for (let i = 0; i < 100; i++) {
- yield { id: i, data: `Item ${i}` }
- }
-}
-
-brain.stream()
- .source(dataGenerator())
- .map(item => item.data)
- .sink(console.log)
- .run()
-```
-
----
-
-### Transform Operations
-
-#### `map(fn): Pipeline`
-Transforms each item.
-
-#### `flatMap(fn): Pipeline`
-Maps and flattens arrays.
-
-#### `filter(predicate): Pipeline`
-Filters items.
-
-#### `tap(fn): Pipeline`
-Side effects without modification.
-
-#### `retry(fn, maxRetries?, backoff?): Pipeline`
-Retries failed operations.
-
----
-
-### Batching & Windowing
-
-#### `batch(size, timeoutMs?): Pipeline`
-Groups items into batches.
-
-#### `window(size, type?): Pipeline`
-Creates sliding or tumbling windows.
-
-#### `buffer(size, strategy?): Pipeline`
-Buffers with backpressure handling.
-
----
-
-### Sink Operations
-
-#### `sink(handler): Pipeline`
-Custom sink handler.
-
-#### `toBrainy(options?): Pipeline`
-Sinks data to Brainy.
-
-**Example:**
-```typescript
-brain.stream()
- .source(dataSource)
- .map(transform)
- .batch(100)
- .toBrainy({
- type: NounType.Document,
- metadata: { source: 'stream' }
- })
- .run()
-```
-
----
-
-### Execution
-
-#### `run(options?): Promise`
-Executes the pipeline.
-
-**Parameters:**
-- `workers` - Number of workers or 'auto'
-- `monitoring` - Enable monitoring
-- `maxThroughput` - Rate limiting
-- `backpressure` - 'drop' | 'buffer' | 'pause'
-- `errorHandler` - Error callback
-
----
-
-## Type Definitions
-
-### Entity Types (NounType)
-31 types including:
-- `Person`, `Organization`, `Location`
-- `Document`, `File`, `Message`, `Content`
-- `Product`, `Service`, `Resource`
-- `Event`, `Task`, `Project`, `Process`
-- `User`, `Role`, `State`
-- `Concept`, `Topic`, `Hypothesis`
-- And more...
-
-### Relationship Types (VerbType)
-40 types including:
-- `RelatedTo`, `Contains`, `PartOf`
-- `Creates`, `Modifies`, `Transforms`
-- `DependsOn`, `Requires`, `Uses`
-- `Owns`, `BelongsTo`, `MemberOf`
-- `Supervises`, `ReportsTo`, `WorksWith`
-- And more...
-
-### Core Interfaces
-
-✨ *Updated in v4.3.0 - Added confidence/weight to Entity, flattened Result fields*
-
-```typescript
-interface Entity {
- id: string
- vector: Vector
- type: NounType
- data?: any
- metadata?: T
- service?: string
- createdAt: number
- updatedAt?: number
- confidence?: number // NEW: Type classification confidence (0-1)
- weight?: number // NEW: Entity importance/salience (0-1)
-}
-
-interface Relation {
- id: string
- from: string
- to: string
- type: VerbType
- weight?: number
- confidence?: number // Relationship confidence
- metadata?: T
- evidence?: RelationEvidence // Why this relationship exists
- service?: string
- createdAt: number
-}
-
-interface Result {
- // Search metadata
- id: string
- score: number
-
- // NEW: Flattened entity fields for convenience
- type?: NounType
- metadata?: T
- data?: any
- confidence?: number
- weight?: number
-
- // Full entity (backward compatible)
- entity: Entity
-
- // Score explanation
- explanation?: ScoreExplanation
-}
-```
-
-**Key Changes in v4.3.0:**
-- ✅ `Entity` now exposes `confidence` and `weight`
-- ✅ `Result` flattens commonly-used entity fields to top level
-- ✅ Direct access: `result.metadata` instead of `result.entity.metadata`
-- ✅ Backward compatible: `result.entity` still available
-
----
-
-## Performance Characteristics
-
-### Speed Benchmarks
-- **Add entity**: ~5-10ms (with embedding)
-- **Vector search**: ~1-5ms for 1M entities
-- **Graph traversal**: ~10-20ms (2-hop)
-- **Batch operations**: 1000+ items/second
-
-### Scalability
-- **Entities**: Tested to 10M+
-- **Relationships**: Tested to 100M+
-- **Concurrent operations**: 1000+ parallel
-- **Memory usage**: ~1GB per million entities
-
-### Storage Requirements
-- **Per entity**: ~2KB (with vector)
-- **Per relationship**: ~200 bytes
-- **Indexes**: ~20% overhead
-
----
-
-## Best Practices
-
-### 1. Always Initialize
-```typescript
-const brain = new Brainy()
-await brain.init() // Required before operations
-```
-
-### 2. Use Proper Types
-```typescript
-// ✅ Good - specific type
-await brain.add({ data: 'John', type: NounType.Person })
-
-// ❌ Bad - generic type
-await brain.add({ data: 'John', type: NounType.Thing })
-```
-
-### 3. Batch When Possible
-```typescript
-// ✅ Good - batch operation
-await brain.addMany({ items: documents })
-
-// ❌ Bad - individual adds in loop
-for (const doc of documents) {
- await brain.add(doc) // Slow!
-}
-```
-
-### 4. Use Write-Only for Speed
-```typescript
-// For high-speed ingestion
-await brain.add({
- data: content,
- type: NounType.Document,
- writeOnly: true // Skip validation
-})
-```
-
-### 5. Clean Up Resources
-```typescript
-try {
- // Your operations
-} finally {
- await brain.close() // Always close
-}
-```
-
----
-
-## Input Validation
-
-Brainy uses a **zero-config validation system** that automatically adapts to your system resources:
-
-### Auto-Configured Limits
-- `limit` parameter maximum: Based on available memory (e.g., 8GB RAM = 80K max limit)
-- Query string length: Auto-scaled based on memory
-- Vector dimensions: Must be exactly 384 for all-MiniLM-L6-v2 model
-
-### Common Validation Rules
-- **Pagination**: `limit` and `offset` must be non-negative
-- **Thresholds**: Values like `weight` and `threshold` must be between 0 and 1
-- **Mutual Exclusion**: Cannot use both `query` and `vector` in same request
-- **Type Safety**: `NounType` and `VerbType` must be valid enum values
-- **Self-Reference**: Cannot create relationships from an entity to itself
-
-### Performance Auto-Tuning
-The validation system monitors query performance and adjusts limits automatically:
-- Fast queries with large results → increases limits
-- Slow queries → reduces limits to maintain performance
-
-## Error Handling
-
-All methods validate parameters and throw descriptive errors:
-
-```typescript
-try {
- await brain.add({ data: '', type: NounType.Document })
-} catch (error) {
- // Error: "must provide either data or vector"
-}
-
-try {
- await brain.find({ limit: -1 })
-} catch (error) {
- // Error: "limit must be non-negative"
-}
-
-try {
- await brain.update({ id: 'xyz', metadata: null, merge: false })
-} catch (error) {
- // Error: "must specify at least one field to update"
- // Note: Use metadata: {} to clear, not null
-}
-```
-
----
-
-## Migration from v2
-
-### Old (v2.15)
-```typescript
-brain.add({ data, type: NounType.Document, metadata })
-brain.find({ query, limit: 10 })
-brain.insights()
-```
-
-### New (v3.0)
-```typescript
-brain.add({ data, type: NounType.Document, metadata })
-brain.find({ query, limit: 10 })
-brain.insights()
-```
-
----
-
-## Support
-
-- **GitHub**: https://github.com/soulcraft/brainy
-- **NPM**: https://www.npmjs.com/package/@soulcraft/brainy
-- **Discord**: https://discord.gg/brainy
-
----
-
-*Brainy v3.0 - The Neural Database That Thinks*
\ No newline at end of file
diff --git a/docs/BATCHING.md b/docs/BATCHING.md
index acc11fe6..e70a9e18 100644
--- a/docs/BATCHING.md
+++ b/docs/BATCHING.md
@@ -1,23 +1,28 @@
-# Batch Operations API v5.12.0
+---
+title: Batch Operations
+slug: guides/batching
+public: true
+category: guides
+template: guide
+order: 5
+description: Eliminate N+1 query patterns with batchGet() and storage-level batch APIs for fast multi-entity reads against filesystem and memory storage.
+next:
+ - api/reference
+ - guides/find-system
+---
-> **Enterprise Production-Ready** | Zero N+1 Query Patterns | 90%+ Performance Improvement
+# Batch Operations API
+> **Production-Ready** | Zero N+1 Query Patterns
## Overview
-Brainy v5.12.0 introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
+Brainy provides batch operations at the storage layer to eliminate N+1 query patterns for VFS operations, relationship queries, and entity retrieval.
### Problem Solved
-**Before v5.12.0:**
-- VFS `getTreeStructure()` on cloud storage: **12.7 seconds** for directory with 12 files
-- N+1 query pattern: 1 directory query + N individual file queries (22 sequential calls × 580ms latency)
+The naive pattern of looping and calling `brain.get(id)` once per item issues sequential reads through the storage layer. Batched APIs collapse that into a single read pass.
-**After v5.12.0:**
-- VFS `getTreeStructure()`: **<1 second** for 12 files (90%+ improvement)
-- 2-3 batched calls instead of 22 sequential calls
-- Native cloud storage batch APIs for maximum throughput
-
-**IMPORTANT:** The v5.12.0 batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See v6.0.0 changes for comprehensive storage path optimizations.
+**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`** at the VFS layer and the explicit `batchGet()` / `getNounMetadataBatch()` calls — not to `readFile()` or individual `get()` operations.
---
@@ -42,8 +47,7 @@ results.size // → 3 (number of found entities)
**Performance:**
- Memory storage: Instant (parallel reads)
-- Cloud storage (GCS/S3/Azure): <500ms for 100 entities
-- Throughput: 50-200+ entities/second depending on adapter
+- Filesystem storage: Parallel reads, scales with available IOPS
**Use Cases:**
- Loading multiple entities for display
@@ -56,7 +60,7 @@ results.size // → 3 (number of found entities)
### 2. `storage.getNounMetadataBatch(ids)`
-Batch metadata retrieval with direct O(1) path construction (v6.0.0+).
+Batch metadata retrieval with direct O(1) path construction.
```typescript
const storage = brain.storage as BaseStorage
@@ -65,21 +69,21 @@ const ids = ['id1', 'id2', 'id3']
const metadataMap: Map = await storage.getNounMetadataBatch(ids)
for (const [id, metadata] of metadataMap) {
- console.log(metadata.noun) // Type: 'document', 'person', etc.
- console.log(metadata.data) // Entity data
+ console.log(metadata.noun) // Type: 'document', 'person', etc.
+ console.log(metadata.data) // Entity data
}
```
**Features:**
- ✅ Direct O(1) path construction from ID (no type lookup needed!)
- ✅ Sharding preservation (all paths include `{shard}/{id}`)
-- ✅ COW-aware (respects branch paths)
-- ✅ 40x faster than v5.x type-first architecture
+- ✅ Write-cache coherent (read-after-write consistency)
+- ✅ O(1) path construction — eliminates the per-entity type search of the old type-first layout
**Performance:**
-- ~1ms per 100 entities (consistent, no cache misses!)
-- Cloud storage: Parallel downloads (100-150 concurrent)
-- No type search delays - every ID maps directly to storage path
+- Constant-time path construction per ID — no type-cache misses
+- Filesystem: parallel reads bounded by IOPS
+- No type search delays — every ID maps directly to storage path
---
@@ -92,22 +96,22 @@ const storage = brain.storage as BaseStorage
// Get all relationships from multiple sources
const results: Map = await storage.getVerbsBySourceBatch([
- 'person1',
- 'person2'
+ 'person1',
+ 'person2'
])
// Filter by verb type
const createsResults = await storage.getVerbsBySourceBatch(
- ['person1', 'person2'],
- 'creates'
+ ['person1', 'person2'],
+ 'creates'
)
// Process results
for (const [sourceId, verbs] of results) {
- console.log(`${sourceId} has ${verbs.length} relationships`)
- verbs.forEach(verb => {
- console.log(` → ${verb.verb} → ${verb.targetId}`)
- })
+ console.log(`${sourceId} has ${verbs.length} relationships`)
+ verbs.forEach(verb => {
+ console.log(` → ${verb.verb} → ${verb.targetId}`)
+ })
}
```
@@ -117,130 +121,8 @@ for (const [sourceId, verbs] of results) {
- Bulk export of relationship data
**Performance:**
-- Memory storage: <10ms for 1000 relationships
-- Cloud storage: Batched reads with parallel metadata fetches
-
----
-
-### 4. `storage.readBatchWithInheritance(paths, targetBranch?)`
-
-COW-aware batch path resolution with branch inheritance.
-
-```typescript
-const storage = brain.storage as BaseStorage
-
-const paths = [
- 'entities/nouns/{shard}/id1/metadata.json',
- 'entities/nouns/{shard}/id2/metadata.json'
-]
-
-// Resolves to: branches/{branch}/entities/nouns/{shard}/{id}/metadata.json
-const results: Map = await storage.readBatchWithInheritance(paths, 'my-branch')
-
-// Automatically inherits from parent branches for missing entities
-```
-
-**Features:**
-- ✅ Branch path resolution (`branches/{branch}/...`)
-- ✅ Write cache integration (read-after-write consistency)
-- ✅ COW inheritance (fallback to parent commits for missing entities)
-- ✅ Adapter-agnostic (works with all storage adapters)
-
----
-
-## Cloud Adapter Native Batch APIs
-
-### GCS Storage
-
-```typescript
-const gcsStorage = new GCSStorage({ bucketName: 'my-bucket' })
-
-// Native batch API with 100 concurrent downloads
-const results = await gcsStorage.readBatch(paths)
-
-// Configuration
-gcsStorage.getBatchConfig() // → {
-// maxBatchSize: 1000,
-// maxConcurrent: 100,
-// operationsPerSecond: 1000
-// }
-```
-
-**Performance:**
-- 100 concurrent downloads
-- ~300-500ms for 100 objects
-- HTTP/2 multiplexing for optimal throughput
-
----
-
-### S3 Compatible Storage
-
-Works with Amazon S3, Cloudflare R2, and other S3-compatible services.
-
-```typescript
-const s3Storage = new S3CompatibleStorage({ bucketName: 'my-bucket' })
-
-// Native batch API with 150 concurrent downloads
-const results = await s3Storage.readBatch(paths)
-
-// Configuration
-s3Storage.getBatchConfig() // → {
-// maxBatchSize: 1000,
-// maxConcurrent: 150,
-// operationsPerSecond: 5000
-// }
-```
-
-**Performance:**
-- 150 concurrent downloads
-- ~200-500ms for 150 objects
-- S3 handles 5000+ ops/second with burst capacity
-
----
-
-### R2 Storage (Cloudflare)
-
-```typescript
-const r2Storage = new R2Storage({ bucketName: 'my-bucket' })
-
-// Fastest cloud storage with zero egress fees
-const results = await r2Storage.readBatch(paths)
-
-// Configuration
-r2Storage.getBatchConfig() // → {
-// maxBatchSize: 1000,
-// maxConcurrent: 150,
-// operationsPerSecond: 6000
-// }
-```
-
-**Performance:**
-- 150 concurrent downloads
-- ~200-400ms for 150 objects (fastest!)
-- Zero egress fees enable aggressive caching
-
----
-
-### Azure Blob Storage
-
-```typescript
-const azureStorage = new AzureBlobStorage({ containerName: 'my-container' })
-
-// Native batch API with 100 concurrent downloads
-const results = await azureStorage.readBatch(paths)
-
-// Configuration
-azureStorage.getBatchConfig() // → {
-// maxBatchSize: 1000,
-// maxConcurrent: 100,
-// operationsPerSecond: 3000
-// }
-```
-
-**Performance:**
-- 100 concurrent downloads
-- ~400-600ms for 100 blobs
-- Good throughput with Azure's global network
+- Memory storage: single in-memory pass over the metadata index
+- Filesystem storage: parallel reads through the metadata index
---
@@ -251,12 +133,10 @@ VFS operations automatically use batch APIs for maximum performance.
### Directory Traversal
```typescript
-// OLD: Sequential N+1 pattern (12.7 seconds for 12 files)
+// Tree traversal uses batched reads under the hood
const tree = await brain.vfs.getTreeStructure('/my-dir')
-
-// NEW v5.12.0: Parallel breadth-first with batching (<1 second)
// ✅ PathResolver.getChildren() uses brain.batchGet() internally
-// ✅ Parallel traversal of directories at same tree level
+// ✅ Parallel traversal of directories at the same tree level
// ✅ 2-3 batched calls instead of 22 sequential calls
```
@@ -264,32 +144,26 @@ const tree = await brain.vfs.getTreeStructure('/my-dir')
```
VFS.getTreeStructure()
- ↓ PARALLEL (breadth-first traversal)
- → PathResolver.getChildren() [all dirs at level processed in parallel]
- ↓ BATCHED
- → brain.batchGet(childIds) [1 call instead of N]
- ↓ BATCHED
- → storage.getNounMetadataBatch(ids) [1 call instead of N]
- ↓ ADAPTER-SPECIFIC
- → GCS: readBatch() with 100 concurrent downloads
- → S3: readBatch() with 150 concurrent downloads
- → Memory: Promise.all() parallel reads
+ ↓ PARALLEL (breadth-first traversal)
+ → PathResolver.getChildren() [all dirs at level processed in parallel]
+ ↓ BATCHED
+ → brain.batchGet(childIds) [1 call instead of N]
+ ↓ BATCHED
+ → storage.getNounMetadataBatch(ids) [1 call instead of N]
+ ↓ ADAPTER
+ → Filesystem: Promise.all() parallel reads
+ → Memory: Promise.all() parallel reads
```
-**Performance Gains:**
-- **Before**: 22 sequential calls × 580ms = 12.7 seconds
-- **After**: 2-3 batched calls = <1 second
-- **Improvement**: **90%+ faster** on cloud storage
-
---
## Advanced Features Compatibility
-### ✅ ID-First Storage Architecture (v6.0.0+)
+### ✅ ID-First Storage Architecture
All batch operations use direct ID-first paths - no type lookup needed!
-**NEW v6.0.0 Path Structure:**
+**ID-First Path Structure:**
```
entities/nouns/{SHARD}/{ID}/metadata.json
entities/verbs/{SHARD}/{ID}/metadata.json
@@ -297,9 +171,9 @@ entities/verbs/{SHARD}/{ID}/metadata.json
**Direct O(1) Path Construction:**
```typescript
-// Every ID maps directly to exactly ONE path - 40x faster!
+// Every ID maps directly to exactly ONE path - O(1), no type search
const id = 'abc-123'
-const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
+const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
const path = `entities/nouns/${shard}/${id}/metadata.json`
// No type cache needed!
@@ -309,9 +183,9 @@ const path = `entities/nouns/${shard}/${id}/metadata.json`
```
**Benefits:**
-- **40x faster** on GCS/S3 (eliminates 42-type sequential search)
+- **O(1)** path lookups (eliminates the 42-type sequential search the old type-first layout required)
- **Simpler code** - removed 500+ lines of type cache complexity
-- **Scalable** - works at billion-scale without type tracking overhead
+- **Scalable** - works at large scale without type tracking overhead
---
@@ -329,95 +203,39 @@ const path = `entities/nouns/${shard}/${id}/metadata.json`
---
-### ✅ COW (Copy-on-Write)
+### ✅ Generational MVCC (8.0)
-Batch operations respect branch isolation and time-travel:
+Batch reads always serve the **live** generation through the fast paths
+shown above. Point-in-time reads go through the Db API instead: a pinned
+`Db` (`brain.now()`, `brain.asOf()`) resolves changed ids from immutable
+generation records and unchanged ids from the same live paths batch reads
+use — see the [consistency model](concepts/consistency-model.md).
```typescript
-// Main branch
-const brain = await Brainy.create({ enableCOW: true })
-await brain.add({ type: 'document', data: 'Main' })
-
-// Create fork
-const fork = await brain.fork('experiment')
-
-// Batch operations are isolated
-await brain.batchGet([id1, id2]) // → Reads from: branches/main/...
-await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/...
-```
-
-**Inheritance:**
-- Entities missing from child branch automatically inherit from parent commits
-- `readBatchWithInheritance()` walks commit history for missing items
-- Preserves fork semantics while maintaining performance
-
----
-
-### ✅ fork() and checkout()
-
-```typescript
-const fork = await brain.fork('my-branch')
-await fork.add({ type: 'document', data: 'Fork entity' })
-
-// Batch operations use correct branch
-const results = await fork.batchGet([id1, id2])
-// → Reads from: branches/my-branch/...
-
-// Checkout changes active branch
-await fork.checkout('main')
-const mainResults = await fork.batchGet([id1, id2])
-// → Reads from: branches/main/...
+const db = brain.now() // pinned view
+const entity = await db.get(id) // correct at the pinned generation
+const results = await brain.batchGet(ids) // live state, batched
+await db.release()
```
---
-### ✅ asOf() Time-Travel
+## Why Batching Is Faster
-```typescript
-// Create historical snapshot
-await brain.commit('v1.0')
-const snapshot = await brain.asOf('v1.0')
+Batching's advantage is structural, not a fixed multiplier (the actual speedup
+depends on storage backend, IOPS, and batch size):
-// Batch operations on historical data
-const results = await snapshot.batchGet([id1, id2])
-// → Reads from historical tree state
-```
+- **N+1 elimination** — N sequential reads collapse into a single parallel pass
+ (`Promise.all` over the batch).
+- **O(1) path construction** — every ID maps directly to one storage path, with
+ no per-type cache lookup.
+- **One metadata round-trip** — relationship batches fetch all sources' metadata
+ in a single pass instead of one query per source.
-Historical queries use `HistoricalStorageAdapter` which wraps batch operations to point at specific commits.
-
----
-
-## Performance Benchmarks
-
-### VFS Operations (12 Files)
-
-| Storage | Before v5.12.0 | After v5.12.0 | Improvement |
-|---------|---------------|---------------|-------------|
-| **GCS** | 12.7s | <1s | **92% faster** |
-| **S3** | 13.2s | <1s | **92% faster** |
-| **R2** | 11.8s | <0.8s | **93% faster** |
-| **Azure** | 14.5s | <1s | **93% faster** |
-| **Memory** | 150ms | 50ms | **67% faster** |
-
-### Entity Batch Retrieval (100 Entities)
-
-| Storage | Individual Gets | Batch Get | Improvement |
-|---------|----------------|-----------|-------------|
-| **GCS** | 5.8s | 0.4s | **93% faster** |
-| **S3** | 5.2s | 0.3s | **94% faster** |
-| **R2** | 4.9s | 0.25s | **95% faster** |
-| **Azure** | 6.5s | 0.5s | **92% faster** |
-| **Memory** | 180ms | 15ms | **92% faster** |
-
-### Throughput (Entities/Second)
-
-| Storage | Individual | Batch | Improvement |
-|---------|-----------|-------|-------------|
-| **GCS** | 17 ent/s | 250 ent/s | **14.7x** |
-| **S3** | 19 ent/s | 333 ent/s | **17.5x** |
-| **R2** | 20 ent/s | 400 ent/s | **20x** |
-| **Azure** | 15 ent/s | 200 ent/s | **13.3x** |
-| **Memory** | 556 ent/s | 6667 ent/s | **12x** |
+The integration test `tests/integration/storage-batch-operations.test.ts`
+exercises batch vs. individual reads and asserts that batch retrieval is not
+slower than the per-entity loop for large batches; it does not pin a specific
+multiplier, since that is hardware- and IOPS-dependent.
---
@@ -430,8 +248,8 @@ Batch operations gracefully handle missing or invalid entities:
```typescript
const validId = 'abc-123-...'
const invalidIds = [
- '11111111-1111-1111-1111-111111111111',
- '22222222-2222-2222-2222-222222222222'
+ '11111111-1111-1111-1111-111111111111',
+ '22222222-2222-2222-2222-222222222222'
]
const results = await brain.batchGet([validId, ...invalidIds])
@@ -471,8 +289,8 @@ results.size // → 1 (deduplicated automatically)
```typescript
const entities = []
for (const id of ids) {
- const entity = await brain.get(id)
- if (entity) entities.push(entity)
+ const entity = await brain.get(id)
+ if (entity) entities.push(entity)
}
```
@@ -482,7 +300,7 @@ const results = await brain.batchGet(ids)
const entities = Array.from(results.values())
```
-**Performance Gain:** 10-20x faster on cloud storage.
+**Performance Gain:** Replaces N sequential reads with a single batched pass — no fixed multiplier, it scales with storage IOPS.
---
@@ -492,8 +310,8 @@ const entities = Array.from(results.values())
```typescript
const allVerbs = []
for (const sourceId of sourceIds) {
- const verbs = await brain.getRelations({ from: sourceId })
- allVerbs.push(...verbs)
+ const verbs = await brain.related({ from: sourceId })
+ allVerbs.push(...verbs)
}
```
@@ -504,11 +322,11 @@ const results = await storage.getVerbsBySourceBatch(sourceIds)
const allVerbs = []
for (const verbs of results.values()) {
- allVerbs.push(...verbs)
+ allVerbs.push(...verbs)
}
```
-**Performance Gain:** 5-10x faster due to batched metadata fetches.
+**Performance Gain:** One batched metadata fetch instead of one query per source entity.
---
@@ -522,19 +340,16 @@ const results = await brain.batchGet(ids)
// ❌ BAD: Individual gets in loop
for (const id of ids) {
- await brain.get(id)
+ await brain.get(id)
}
```
### 2. **Batch Size Recommendations**
| Storage | Optimal Batch Size | Max Batch Size |
-|---------|-------------------|----------------|
+|---------|--------------------|----------------|
| **Memory** | Unlimited | Unlimited |
-| **FileSystem** | 100-500 | 1000 |
-| **GCS** | 100-500 | 1000 |
-| **S3/R2** | 100-1000 | 1000 |
-| **Azure** | 100-500 | 1000 |
+| **Filesystem** | 100-500 | 1000 |
**Guideline:** For batches >1000, split into chunks of 500-1000.
@@ -556,13 +371,13 @@ const results = await brain.batchGet(ids)
// Check results
for (const id of ids) {
- if (results.has(id)) {
- // Entity exists
- const entity = results.get(id)
- } else {
- // Entity missing (not an error)
- console.log(`Entity ${id} not found`)
- }
+ if (results.has(id)) {
+ // Entity exists
+ const entity = results.get(id)
+ } else {
+ // Entity missing (not an error)
+ console.log(`Entity ${id} not found`)
+ }
}
```
@@ -597,73 +412,47 @@ npx vitest run tests/integration/storage-batch-operations.test.ts
```
User Code (brain.batchGet)
- ↓
+ ↓
High-Level API (src/brainy.ts)
- ↓
+ ↓
Storage Layer (src/storage/baseStorage.ts)
- ↓
-COW Layer (readBatchWithInheritance)
- ↓
+ ↓
Adapter Layer (readBatchFromAdapter)
- ↓
-Cloud Adapter (GCS/S3/Azure native batch APIs)
+ ↓
+Storage Adapter (FileSystemStorage / MemoryStorage)
```
-### Automatic Fallback
+### Parallel Reads
-If an adapter doesn't implement `readBatch()`, the system automatically falls back to parallel individual reads:
+Both shipped adapters fall back to `Promise.all` over individual reads:
```typescript
// BaseStorage.readBatchFromAdapter()
-if (typeof selfWithBatch.readBatch === 'function') {
- // Use native batch API
- return await selfWithBatch.readBatch(resolvedPaths)
-} else {
- // Automatic parallel fallback
- return await Promise.all(resolvedPaths.map(path => this.read(path)))
-}
+return await Promise.all(resolvedPaths.map(path => this.read(path)))
```
-**Adapters with Native Batch:**
-- ✅ GCSStorage
-- ✅ S3CompatibleStorage
-- ✅ R2Storage
-- ✅ AzureBlobStorage
-
-**Adapters with Parallel Fallback:**
+**Shipped Adapters:**
- MemoryStorage
- FileSystemStorage
-- OPFSStorage
-- HistoricalStorageAdapter (delegates to underlying)
---
-## Release Notes
+## API Summary
-**Version:** 5.12.0
-**Release Date:** 2025-11-19
-**Status:** Production-Ready
-
-**Breaking Changes:** None (backward compatible)
-
-**New APIs:**
- `brain.batchGet(ids, options?)` - High-level batch entity retrieval
- `storage.getNounMetadataBatch(ids)` - Storage-level metadata batch
- `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries
-- `storage.readBatchWithInheritance(paths, targetBranch?)` - COW-aware batch reads
**Performance Improvements:**
-- VFS operations: 90%+ faster on cloud storage
-- Entity retrieval: 10-20x throughput improvement
+- VFS operations: single batched pass instead of N sequential reads
+- Entity retrieval: N+1 reads collapsed into one batched pass
- Zero N+1 query patterns
**Compatibility:**
-- ✅ ID-first storage (v6.0.0+)
+- ✅ ID-first storage
- ✅ Sharding (256 shards)
-- ✅ COW (branch isolation, inheritance)
-- ✅ fork() and checkout()
-- ✅ asOf() time-travel
-- ✅ All 6 indexes respected (HNSW, TypeAwareHNSW, MetadataIndex, GraphAdjacency, Version, DeletedItems)
+- ✅ Generational MVCC — batch reads serve the live generation; pinned `Db` views serve the past
+- ✅ All indexes respected (vector, metadata, graph adjacency)
---
diff --git a/docs/CORE_API_PATTERNS.md b/docs/CORE_API_PATTERNS.md
deleted file mode 100644
index 717686c8..00000000
--- a/docs/CORE_API_PATTERNS.md
+++ /dev/null
@@ -1,643 +0,0 @@
-# 🧠 Core API Patterns: Modern Brainy v3.x
-
-> Learn the correct patterns for Brainy's core operations. Avoid v2.x confusion and use modern, efficient APIs.
-
-## 🚨 Critical: Use v3.x APIs Only
-
-### ❌ **WRONG - Deprecated v2.x APIs**
-
-```typescript
-// DON'T DO THIS - These methods don't exist in v3.x!
-await brain.addNoun(text, type, metadata) // ❌ Removed
-await brain.getNouns({ pagination }) // ❌ Removed
-await brain.addVerb(source, target, type) // ❌ Removed
-await brain.getVerbs() // ❌ Removed
-await brain.deleteNoun(id) // ❌ Removed
-await brain.deleteVerb(id) // ❌ Removed
-```
-
-### ✅ **CORRECT - Modern v3.x APIs**
-
-```typescript
-// ✅ Use these modern methods instead
-await brain.add({ data, type, metadata }) // Modern unified add
-await brain.find({ limit: 100 }) // Natural language search
-await brain.relate({ from, to, type }) // Clean relationship creation
-await brain.getRelations() // Modern relationship queries
-await brain.delete(id) // Unified deletion
-// Relationships auto-cascade when entities are deleted
-```
-
-## 📋 Entity Management Patterns
-
-### ❌ **WRONG - v2.x Style**
-
-```typescript
-// DON'T DO THIS - Old API patterns
-import { Brainy } from 'old-brainy' // ❌ Wrong import
-
-const brain = new Brainy({ // ❌ Old class name
- complexConfig: true
-})
-
-const id = await brain.addNoun( // ❌ Deprecated method
- "John Smith is a developer",
- "Person",
- { role: "engineer" }
-)
-```
-
-### ✅ **CORRECT - Modern Patterns**
-
-```typescript
-// ✅ Pattern 1: Basic entity creation
-import { Brainy, NounType } from '@soulcraft/brainy'
-
-const brain = new Brainy() // ✅ Zero config
-await brain.init()
-
-const id = await brain.add({
- data: "John Smith is a developer",
- type: NounType.Person,
- metadata: { role: "engineer", team: "backend" }
-})
-
-// ✅ Pattern 2: Bulk entity creation
-const entities = [
- { data: "React framework", type: NounType.Technology },
- { data: "Vue.js framework", type: NounType.Technology },
- { data: "Angular framework", type: NounType.Technology }
-]
-
-const ids = await Promise.all(
- entities.map(entity => brain.add(entity))
-)
-
-// ✅ Pattern 3: Entity with pre-computed vector
-const customVector = await brain.embed("Custom text")
-const vectorId = await brain.add({
- data: "Optimized content",
- type: NounType.Document,
- vector: customVector, // Skip re-embedding
- metadata: { source: "api", optimized: true }
-})
-```
-
-## 🔍 Search & Discovery Patterns
-
-### ❌ **WRONG - Confusing Old Patterns**
-
-```typescript
-// DON'T DO THIS - Mixing old and new APIs
-const results1 = await brain.searchText("query") // ❌ Old method
-const results2 = await brain.getNouns({ filter }) // ❌ Doesn't exist
-const results3 = await brain.findSimilar(text) // ❌ Unclear naming
-```
-
-### ✅ **CORRECT - Clean Search Patterns**
-
-```typescript
-// ✅ Pattern 1: Natural language search
-const results = await brain.find("React developers working on authentication")
-
-// ✅ Pattern 2: Structured search with filters
-const filteredResults = await brain.find({
- like: "machine learning", // Vector similarity
- where: { // Metadata filtering
- type: NounType.Document,
- year: { $gte: 2020 },
- status: "published"
- },
- limit: 50,
- orderBy: 'relevance'
-})
-
-// ✅ Pattern 3: Similarity search
-const similarItems = await brain.similar({
- to: existingEntityId, // Find items similar to this
- threshold: 0.8, // Minimum similarity
- limit: 10,
- exclude: [existingEntityId] // Don't include the source
-})
-
-// ✅ Pattern 4: Advanced search with relationships
-const connectedResults = await brain.find({
- like: "frontend frameworks",
- connected: {
- to: reactId, // Connected to React
- via: "related-to", // Through this relationship
- depth: 2 // Up to 2 hops away
- }
-})
-```
-
-## 🔗 Relationship Patterns
-
-### ❌ **WRONG - Old Relationship APIs**
-
-```typescript
-// DON'T DO THIS - Old relationship patterns
-await brain.addVerb(sourceId, targetId, "uses", { strength: 0.9 }) // ❌ Old API
-const verbs = await brain.getVerbsBySource(sourceId) // ❌ Removed
-await brain.deleteVerb(verbId) // ❌ Old pattern
-```
-
-### ✅ **CORRECT - Modern Relationship Management**
-
-```typescript
-// ✅ Pattern 1: Create relationships
-const relationId = await brain.relate({
- from: developerId,
- to: frameworkId,
- type: VerbType.Uses,
- metadata: {
- since: "2023-01-01",
- proficiency: "expert",
- hours_per_week: 40
- }
-})
-
-// ✅ Pattern 2: Query relationships
-const relationships = await brain.getRelations({
- from: developerId, // Relationships from this entity
- type: VerbType.Uses, // Of this type
- limit: 100
-})
-
-// ✅ Pattern 3: Bidirectional relationships
-await brain.relate({
- from: projectId,
- to: developerId,
- type: VerbType.AssignedTo,
- bidirectional: true, // Creates reverse relationship
- metadata: { role: "lead", start_date: "2024-01-01" }
-})
-
-// ✅ Pattern 4: Relationship-based discovery
-const collaborators = await brain.find({
- connected: {
- to: currentProjectId,
- via: VerbType.WorksOn,
- direction: "incoming" // Who works on this project
- }
-})
-```
-
-## 🗃️ Data Retrieval Patterns
-
-### ❌ **WRONG - Inefficient Patterns**
-
-```typescript
-// DON'T DO THIS - Loading everything
-const everything = await brain.getNouns({ limit: 1000000 }) // ❌ Crashes
-const allData = await brain.exportAll() // ❌ Memory explosion
-```
-
-### ✅ **CORRECT - Efficient Data Access**
-
-```typescript
-// ✅ Pattern 1: Paginated retrieval
-async function getAllEntitiesPaginated() {
- const pageSize = 100
- let offset = 0
- let allEntities = []
-
- while (true) {
- const page = await brain.find({
- limit: pageSize,
- offset: offset
- })
-
- if (page.length === 0) break
-
- allEntities.push(...page)
- offset += pageSize
-
- // Optional: Progress reporting
- console.log(`Loaded ${allEntities.length} entities...`)
- }
-
- return allEntities
-}
-
-// ✅ Pattern 2: Streaming large datasets
-async function* streamEntities() {
- const pageSize = 50
- let offset = 0
-
- while (true) {
- const page = await brain.find({
- limit: pageSize,
- offset: offset
- })
-
- if (page.length === 0) break
-
- for (const entity of page) {
- yield entity
- }
-
- offset += pageSize
- }
-}
-
-// Usage
-for await (const entity of streamEntities()) {
- await processEntity(entity)
-}
-
-// ✅ Pattern 3: Specific entity retrieval
-const entity = await brain.get(entityId)
-if (entity) {
- console.log('Entity data:', entity.data)
- console.log('Metadata:', entity.metadata)
-} else {
- console.log('Entity not found')
-}
-```
-
-## 🔄 Update & Delete Patterns
-
-### ❌ **WRONG - Manual Update Patterns**
-
-```typescript
-// DON'T DO THIS - Recreating entities
-await brain.delete(oldId)
-const newId = await brain.add(updatedData) // ❌ Loses relationships
-```
-
-### ✅ **CORRECT - Update Operations**
-
-```typescript
-// ✅ Pattern 1: Update entity data
-await brain.update(entityId, {
- data: "Updated content here",
- metadata: {
- lastModified: Date.now(),
- version: "2.0"
- }
-})
-
-// ✅ Pattern 2: Partial metadata updates
-await brain.updateMetadata(entityId, {
- status: "published",
- tags: ["important", "featured"]
- // Merges with existing metadata
-})
-
-// ✅ Pattern 3: Safe deletion with cascade options
-await brain.delete(entityId, {
- cascade: true, // Delete related relationships
- backup: true // Create backup before deletion
-})
-
-// ✅ Pattern 4: Bulk operations
-const updateOperations = entities.map(entity => ({
- id: entity.id,
- changes: { status: "processed" }
-}))
-
-await brain.updateMany(updateOperations)
-```
-
-## 🧮 Vector & Embedding Patterns
-
-### ❌ **WRONG - Manual Vector Handling**
-
-```typescript
-// DON'T DO THIS - Manual embedding without understanding
-const vector = await brain.embed(text)
-// Store vector somewhere manually // ❌ Missing integration
-```
-
-### ✅ **CORRECT - Smart Vector Operations**
-
-```typescript
-// ✅ Pattern 1: Automatic embedding (recommended)
-const id = await brain.add({
- data: "Content to be embedded",
- type: NounType.Document
- // Vector computed automatically
-})
-
-// ✅ Pattern 2: Pre-computed vectors for optimization
-const texts = ["Text 1", "Text 2", "Text 3"]
-const vectors = await Promise.all(
- texts.map(text => brain.embed(text))
-)
-
-const entities = await Promise.all(
- texts.map((text, i) => brain.add({
- data: text,
- type: NounType.Document,
- vector: vectors[i] // Skip re-embedding
- }))
-)
-
-// ✅ Pattern 3: Vector similarity search
-const queryVector = await brain.embed("search query")
-const similar = await brain.similar({
- vector: queryVector, // Use vector directly
- threshold: 0.75,
- limit: 20
-})
-
-// ✅ Pattern 4: Compare vectors directly
-const vector1 = await brain.embed("First text")
-const vector2 = await brain.embed("Second text")
-const similarity = brain.computeSimilarity(vector1, vector2)
-console.log(`Similarity: ${similarity}`)
-```
-
-## 🏗️ Configuration Patterns
-
-### ❌ **WRONG - Over-Configuration**
-
-```typescript
-// DON'T DO THIS - Complex configurations that break
-const brain = new Brainy({
- storage: {
- type: 'complex',
- options: {
- nested: {
- configuration: true,
- that: "breaks"
- }
- }
- },
- embedding: {
- customModel: "broken-model",
- dimensions: 999999
- }
-})
-```
-
-### ✅ **CORRECT - Smart Configuration**
-
-```typescript
-// ✅ Pattern 1: Zero configuration (recommended)
-const brain = new Brainy() // Auto-detects everything
-await brain.init()
-
-// ✅ Pattern 2: Simple storage selection
-const fsBrain = new Brainy({
- storage: { type: 'filesystem', path: './data' }
-})
-
-const cloudBrain = new Brainy({
- storage: { type: 's3', bucket: 'my-data' }
-})
-
-// ✅ Pattern 3: Production configuration
-const prodBrain = new Brainy({
- storage: {
- type: 's3',
- bucket: process.env.BRAINY_BUCKET,
- region: process.env.AWS_REGION
- },
- silent: true, // No console output
- distributed: true, // Enable clustering
- cache: { maxSize: 10000 } // Larger cache
-})
-
-// ✅ Pattern 4: Development vs production
-const isDev = process.env.NODE_ENV === 'development'
-
-const brain = new Brainy({
- storage: isDev
- ? { type: 'memory' } // Fast for dev
- : { type: 'filesystem', path: './brainy-data' }, // Persistent for prod
- silent: !isDev, // Verbose in dev, quiet in prod
- cache: { maxSize: isDev ? 100 : 5000 }
-})
-```
-
-## 🔄 Migration from v2.x
-
-### ✅ **Migration Patterns**
-
-```typescript
-// If you have old v2.x code, here's how to migrate:
-
-// OLD v2.x:
-// await brain.addNoun(text, type, metadata)
-// NEW v3.x:
-await brain.add({ data: text, type, metadata })
-
-// OLD v2.x:
-// await brain.getNouns({ pagination: { limit: 100 } })
-// NEW v3.x:
-await brain.find({ limit: 100 })
-
-// OLD v2.x:
-// await brain.addVerb(sourceId, targetId, verbType, metadata)
-// NEW v3.x:
-await brain.relate({ from: sourceId, to: targetId, type: verbType, metadata })
-
-// OLD v2.x:
-// await brain.searchText(query)
-// NEW v3.x:
-await brain.find(query) // More powerful natural language search
-```
-
-## 🚀 Performance Patterns
-
-### ✅ **High-Performance Patterns**
-
-```typescript
-// ✅ Pattern 1: Batch operations
-const entities = [/* large array */]
-const batchSize = 100
-
-for (let i = 0; i < entities.length; i += batchSize) {
- const batch = entities.slice(i, i + batchSize)
- await Promise.all(
- batch.map(entity => brain.add(entity))
- )
-
- // Optional: Rate limiting
- await new Promise(resolve => setTimeout(resolve, 100))
-}
-
-// ✅ Pattern 2: Connection pooling for distributed
-const brain = new Brainy({
- distributed: true,
- connectionPool: {
- min: 5,
- max: 50,
- acquireTimeoutMillis: 30000
- }
-})
-
-// ✅ Pattern 3: Efficient caching
-const brain = new Brainy({
- cache: {
- maxSize: 10000, // Number of items
- ttl: 300000, // 5 minutes
- updateAgeOnGet: true // LRU behavior
- }
-})
-
-// ✅ Pattern 4: Memory-conscious operations
-const results = await brain.find({
- query: "large dataset query",
- limit: 1000, // Reasonable limit
- includeVectors: false // Exclude vectors if not needed
-})
-```
-
-## 🛡️ Error Handling Patterns
-
-### ✅ **Robust Error Handling**
-
-```typescript
-// ✅ Pattern 1: Specific error handling
-try {
- const result = await brain.add({ data, type, metadata })
- return result
-} catch (error) {
- if (error.code === 'DUPLICATE_ENTITY') {
- console.log('Entity already exists, updating instead...')
- return await brain.update(error.existingId, { data, metadata })
- } else if (error.code === 'STORAGE_FULL') {
- throw new Error('Storage capacity exceeded')
- } else if (error.code === 'EMBEDDING_FAILED') {
- console.warn('Embedding failed, retrying with simpler text...')
- return await brain.add({
- data: data.substring(0, 1000), // Truncate
- type,
- metadata
- })
- }
- throw error
-}
-
-// ✅ Pattern 2: Retry with exponential backoff
-async function resilientAdd(data: any, maxRetries = 3) {
- for (let attempt = 1; attempt <= maxRetries; attempt++) {
- try {
- return await brain.add(data)
- } catch (error) {
- if (attempt === maxRetries) throw error
-
- const delay = Math.pow(2, attempt) * 1000
- console.warn(`Attempt ${attempt} failed, retrying in ${delay}ms...`)
- await new Promise(resolve => setTimeout(resolve, delay))
- }
- }
-}
-
-// ✅ Pattern 3: Graceful degradation
-async function robustSearch(query: string) {
- try {
- // Try advanced semantic search first
- return await brain.find({
- like: query,
- threshold: 0.8,
- limit: 50
- })
- } catch (error) {
- console.warn('Semantic search failed, falling back to basic search:', error.message)
-
- try {
- // Fallback to simple text search
- return await brain.find(query)
- } catch (fallbackError) {
- console.error('All search methods failed:', fallbackError.message)
- return [] // Return empty results rather than crash
- }
- }
-}
-```
-
-## 📊 Monitoring Patterns
-
-### ✅ **Production Monitoring**
-
-```typescript
-// ✅ Pattern 1: Performance monitoring
-const startTime = Date.now()
-const result = await brain.add(data)
-const duration = Date.now() - startTime
-
-if (duration > 1000) {
- console.warn(`Slow add operation: ${duration}ms`)
-}
-
-// ✅ Pattern 2: Health checks
-async function healthCheck() {
- try {
- // Test basic operations
- const testId = await brain.add({
- data: "health check",
- type: NounType.System,
- metadata: { test: true }
- })
-
- await brain.get(testId)
- await brain.delete(testId)
-
- return { status: 'healthy', timestamp: Date.now() }
- } catch (error) {
- return {
- status: 'unhealthy',
- error: error.message,
- timestamp: Date.now()
- }
- }
-}
-
-// ✅ Pattern 3: Metrics collection
-class BrainyMetrics {
- private metrics = {
- operations: 0,
- errors: 0,
- totalTime: 0
- }
-
- async timedOperation(operation: () => Promise): Promise {
- const start = Date.now()
- try {
- const result = await operation()
- this.metrics.operations++
- this.metrics.totalTime += Date.now() - start
- return result
- } catch (error) {
- this.metrics.errors++
- throw error
- }
- }
-
- getStats() {
- return {
- ...this.metrics,
- avgTime: this.metrics.totalTime / this.metrics.operations || 0,
- errorRate: this.metrics.errors / this.metrics.operations || 0
- }
- }
-}
-```
-
-## 🎯 Summary: Modern Brainy v3.x Best Practices
-
-| ❌ **Avoid v2.x** | ✅ **Use v3.x** |
-|------------------|----------------|
-| `addNoun()` | `add()` |
-| `getNouns()` | `find()` |
-| `addVerb()` | `relate()` |
-| `getVerbs()` | `getRelations()` |
-| `deleteNoun()` | `delete()` |
-| Complex configs | Zero-config with `new Brainy()` |
-| Manual pagination | Built-in smart pagination |
-| String-based search | Natural language queries |
-
----
-
-**🎉 Following these patterns gives you:**
-- 🚀 **Modern APIs** that are actively maintained
-- ⚡ **Better performance** with intelligent defaults
-- 🛡️ **Robust error handling** with specific error types
-- 📈 **Scalable patterns** for production applications
-- 🧠 **Natural language** search capabilities
-
-**Next:** [Neural API Patterns →](./NEURAL_API_PATTERNS.md) | [VFS Patterns →](./vfs/COMMON_PATTERNS.md)
\ No newline at end of file
diff --git a/docs/CREATING-AUGMENTATIONS.md b/docs/CREATING-AUGMENTATIONS.md
deleted file mode 100644
index 1c7fd04a..00000000
--- a/docs/CREATING-AUGMENTATIONS.md
+++ /dev/null
@@ -1,435 +0,0 @@
-# Creating Augmentations for Brainy
-
-> **Updated for v4.0.0** - Includes metadata structure changes and type system improvements
-
-## The BrainyAugmentation Interface
-
-Every augmentation implements this simple yet powerful interface:
-
-```typescript
-interface BrainyAugmentation {
- // Identification
- name: string // Unique name for your augmentation
-
- // Execution control
- timing: 'before' | 'after' | 'around' | 'replace' // When to execute
- operations: string[] // Which operations to intercept
- priority: number // Execution order (higher = first)
-
- // Lifecycle methods
- initialize(context: AugmentationContext): Promise
- execute(operation: string, params: any, next: () => Promise): Promise
- shutdown?(): Promise // Optional cleanup
-}
-```
-
-## v4.0.0 Breaking Changes for Augmentation Developers
-
-### 1. Metadata Structure Separation
-v4.0.0 introduces strict metadata/vector separation for billion-scale performance:
-
-```typescript
-// ✅ v4.0.0: Metadata has required type field
-interface NounMetadata {
- noun: NounType // Required! Must be a valid noun type
- [key: string]: any // Your custom metadata
-}
-
-interface VerbMetadata {
- verb: VerbType // Required! Must be a valid verb type
- sourceId: string
- targetId: string
- [key: string]: any
-}
-```
-
-### 2. Storage Adapter Return Types
-Storage adapters now return different types at different boundaries:
-
-```typescript
-// Internal methods: Pure structures (no metadata)
-abstract _getNoun(id: string): Promise
-
-// Public API: WithMetadata structures
-abstract getNoun(id: string): Promise
-```
-
-### 3. Verb Property Renamed
-The verb relationship field changed from `type` to `verb`:
-
-```typescript
-// ❌ v3.x
-verb.type === 'relatedTo'
-
-// ✅ v4.0.0
-verb.verb === 'relatedTo'
-```
-
-## Creating a Storage Augmentation
-
-Storage augmentations are special - they provide the storage backend for Brainy.
-
-### Important: v4.0.0 Storage Requirements
-
-Your storage adapter MUST:
-1. **Wrap metadata** with required `noun`/`verb` fields
-2. **Return pure structures** from internal `_methods`
-3. **Return WithMetadata types** from public methods
-
-```typescript
-import { StorageAugmentation } from 'brainy/augmentations'
-import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy'
-
-export class MyCustomStorage extends BaseStorageAdapter {
- // Internal method: Returns pure structure
- async _getNoun(id: string): Promise {
- const data = await this.fetchFromDatabase(id)
- return data ? {
- id: data.id,
- vector: data.vector,
- nounType: data.type
- } : null
- }
-
- // Public method: Returns WithMetadata structure
- async getNoun(id: string): Promise {
- const noun = await this._getNoun(id)
- if (!noun) return null
-
- // Fetch metadata separately (v4.0.0 pattern)
- const metadata = await this.getNounMetadata(id)
-
- return {
- ...noun,
- metadata: metadata || { noun: noun.nounType || 'thing' }
- }
- }
-
- // CRITICAL: Always save with proper metadata structure
- async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise {
- // Validate metadata has required 'noun' field
- if (!metadata?.noun) {
- throw new Error('v4.0.0: NounMetadata requires "noun" field')
- }
-
- await this.database.save({
- id: noun.id,
- vector: noun.vector,
- nounType: noun.nounType,
- metadata: metadata // Stored separately in v4.0.0
- })
- }
-}
-
-export class MyStorageAugmentation extends StorageAugmentation {
- private config: MyStorageConfig
-
- constructor(config: MyStorageConfig) {
- super()
- this.name = 'my-custom-storage'
- this.config = config
- }
-
- // Called during storage resolution phase
- async provideStorage(): Promise {
- const storage = new MyCustomStorage(this.config)
- this.storageAdapter = storage
- return storage
- }
-
- // Called during augmentation initialization
- protected async onInitialize(): Promise {
- await this.storageAdapter!.init()
- this.log(`Custom storage initialized`)
- }
-}
-```
-
-### Using Your Storage Augmentation
-
-```typescript
-// Register before brain.init()
-const brain = new Brainy()
-brain.augmentations.register(new MyStorageAugmentation({
- connectionString: 'redis://localhost:6379'
-}))
-await brain.init() // Will use your storage!
-```
-
-## Creating a Feature Augmentation
-
-Here's a complete example of a caching augmentation:
-
-```typescript
-import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations'
-
-export class CachingAugmentation extends BaseAugmentation {
- private cache = new Map()
-
- constructor() {
- super()
- this.name = 'smart-cache'
- this.timing = 'around' // Wrap operations
- this.operations = ['search'] // Only cache searches
- this.priority = 50 // Mid-priority
- }
-
- async execute(operation: string, params: any, next: () => Promise): Promise {
- if (operation === 'search') {
- // Check cache
- const cacheKey = JSON.stringify(params)
- if (this.cache.has(cacheKey)) {
- this.log('Cache hit!')
- return this.cache.get(cacheKey)
- }
-
- // Execute and cache
- const result = await next()
- this.cache.set(cacheKey, result)
- return result
- }
-
- // Pass through other operations
- return next()
- }
-
- protected async onInitialize(): Promise {
- this.log('Cache initialized')
- }
-
- async shutdown(): Promise {
- this.cache.clear()
- await super.shutdown()
- }
-}
-```
-
-## The Four Timing Modes
-
-### 1. `before` - Pre-processing
-```typescript
-timing = 'before'
-async execute(op, params, next) {
- // Validate/transform input
- const validated = await validate(params)
- return next(validated) // Pass modified params
-}
-```
-
-### 2. `after` - Post-processing
-```typescript
-timing = 'after'
-async execute(op, params, next) {
- const result = await next()
- // Log, analyze, or modify result
- console.log(`Operation ${op} returned:`, result)
- return result
-}
-```
-
-### 3. `around` - Wrapping (middleware)
-```typescript
-timing = 'around'
-async execute(op, params, next) {
- console.log('Starting', op)
- try {
- const result = await next()
- console.log('Success', op)
- return result
- } catch (error) {
- console.log('Failed', op, error)
- throw error
- }
-}
-```
-
-### 4. `replace` - Complete replacement
-```typescript
-timing = 'replace'
-async execute(op, params, next) {
- // Don't call next() - replace entirely!
- return myCustomImplementation(params)
-}
-```
-
-## Operations You Can Intercept
-
-Common operations in Brainy:
-- `'storage'` - Storage resolution (special)
-- `'add'` - Adding data
-- `'search'`, `'similar'` - Searching
-- `'update'`, `'delete'` - Modifications
-- `'saveNoun'`, `'saveVerb'` - Storage operations
-- `'all'` - Intercept everything
-
-## Context Available to Augmentations
-
-```typescript
-interface AugmentationContext {
- brain: Brainy // The brain instance
- storage: StorageAdapter // Storage backend
- config: BrainyConfig // Configuration
- log: (message: string, level?: 'info' | 'warn' | 'error') => void
-}
-```
-
-## Real-World Examples
-
-### 1. Redis Storage Augmentation
-```typescript
-export class RedisStorageAugmentation extends StorageAugmentation {
- async provideStorage(): Promise {
- return new RedisAdapter({
- host: 'localhost',
- port: 6379,
- // Implement full StorageAdapter interface
- })
- }
-}
-```
-
-### 2. Audit Trail Augmentation
-```typescript
-export class AuditAugmentation extends BaseAugmentation {
- timing = 'after'
- operations = ['add', 'update', 'delete']
-
- async execute(op, params, next) {
- const result = await next()
-
- // Log to audit trail
- await this.logAudit({
- operation: op,
- params,
- result,
- timestamp: new Date(),
- user: this.context.config.currentUser
- })
-
- return result
- }
-}
-```
-
-### 3. Rate Limiting Augmentation
-```typescript
-export class RateLimitAugmentation extends BaseAugmentation {
- timing = 'before'
- operations = ['search']
- private limiter = new RateLimiter({ rps: 100 })
-
- async execute(op, params, next) {
- await this.limiter.acquire() // Wait if rate limited
- return next()
- }
-}
-```
-
-## Publishing to Brain Cloud Marketplace
-
-Future capability for premium augmentations:
-
-```typescript
-// package.json
-{
- "name": "@brain-cloud/redis-storage",
- "brainy": {
- "type": "augmentation",
- "category": "storage",
- "premium": true
- }
-}
-
-// Users can install via:
-// brainy augment install redis-storage
-```
-
-## Best Practices
-
-### General Practices
-
-1. **Use BaseAugmentation** - Provides common functionality
-2. **Set appropriate priority** - Storage (100), System (80-99), Features (10-50)
-3. **Be selective with operations** - Don't use 'all' unless necessary
-4. **Handle errors gracefully** - Don't break the chain
-5. **Clean up in shutdown()** - Release resources
-6. **Log appropriately** - Use context.log() for consistent output
-7. **Document your augmentation** - Include examples
-
-### v4.0.0 Specific Best Practices
-
-8. **Always include `noun` field** when creating/modifying NounMetadata:
- ```typescript
- const metadata: NounMetadata = {
- noun: 'thing', // REQUIRED!
- yourField: 'value'
- }
- ```
-
-9. **Use `verb` property** not `type` when working with relationships:
- ```typescript
- // ✅ Correct
- if (verb.verb === 'relatedTo') { ... }
-
- // ❌ Wrong (v3.x pattern)
- if (verb.type === 'relatedTo') { ... }
- ```
-
-10. **Access metadata correctly** from storage:
- ```typescript
- // ✅ Correct - metadata is already structured
- const nounType = noun.metadata.noun
-
- // ⚠️ Fallback pattern for robustness
- const nounType = noun.metadata?.noun || 'thing'
- ```
-
-11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations:
- ```typescript
- // ✅ Good - Separate concerns
- await storage.saveNoun(noun)
- await storage.saveMetadata(noun.id, metadata)
-
- // ❌ Bad - Mixing concerns
- await storage.saveNounWithEverything(combinedData)
- ```
-
-## Testing Your Augmentation
-
-```typescript
-import { Brainy } from 'brainy'
-import { MyAugmentation } from './my-augmentation'
-
-describe('MyAugmentation', () => {
- let brain: Brainy
-
- beforeEach(async () => {
- brain = new Brainy()
- brain.augmentations.register(new MyAugmentation())
- await brain.init()
- })
-
- afterEach(async () => {
- await brain.destroy()
- })
-
- it('should enhance searches', async () => {
- // Test your augmentation's effect
- const results = await brain.search('test')
- expect(results).toHaveProperty('enhanced', true)
- })
-})
-```
-
-## Summary
-
-Augmentations are Brainy's extension system. They can:
-- Replace storage backends
-- Add caching layers
-- Implement audit trails
-- Add rate limiting
-- Sync with external systems
-- Transform data
-- And much more!
-
-The unified BrainyAugmentation interface makes it easy to create powerful extensions while maintaining consistency across the entire system.
\ No newline at end of file
diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md
new file mode 100644
index 00000000..aa3cd351
--- /dev/null
+++ b/docs/DATA_MODEL.md
@@ -0,0 +1,271 @@
+# Data Model
+
+> How Brainy stores entities and relationships, and the critical distinction between `data` and `metadata`.
+
+---
+
+## Entity (Noun)
+
+An entity is the fundamental data unit in Brainy. Every entity has:
+
+| Field | Type | Indexed | Description |
+|-------|------|---------|-------------|
+| `id` | `string` | Primary key | UUID v4 (auto-generated or custom) |
+| `data` | `any` | **HNSW vector index** | Content used for semantic/hybrid search. Strings auto-embed. |
+| `metadata` | `object` | **MetadataIndex** | Structured queryable fields (tags, dates, flags, etc.) |
+| `type` | `NounType` | MetadataIndex (as `noun`) | Entity type classification |
+| `vector` | `number[]` | HNSW | 384-dim embedding (auto-computed from `data` or user-provided) |
+| `confidence` | `number` | MetadataIndex | Type classification confidence (0-1) |
+| `weight` | `number` | MetadataIndex | Entity importance/salience (0-1) |
+| `service` | `string` | MetadataIndex | Multi-tenancy identifier |
+| `createdAt` | `number` | MetadataIndex | Creation timestamp (ms since epoch) |
+| `updatedAt` | `number` | MetadataIndex | Last update timestamp (ms since epoch) |
+| `createdBy` | `object` | MetadataIndex | Source augmentation info |
+
+### Example
+
+```typescript
+const id = await brain.add({
+ data: 'John Smith is a software engineer at Acme Corp', // → embedded into vector
+ type: NounType.Person,
+ metadata: { // → indexed, queryable via where filters
+ role: 'engineer',
+ department: 'backend',
+ yearsExperience: 8
+ },
+ confidence: 0.95,
+ weight: 0.7
+})
+```
+
+---
+
+## Relationship (Verb)
+
+A relationship is a typed, directed edge connecting two entities.
+
+| Field | Type | Indexed | Description |
+|-------|------|---------|-------------|
+| `id` | `string` | Primary key | UUID v4 (auto-generated) |
+| `from` | `string` | **GraphAdjacencyIndex** | Source entity ID |
+| `to` | `string` | **GraphAdjacencyIndex** | Target entity ID |
+| `type` | `VerbType` | GraphAdjacencyIndex (as `verb`) | Relationship type classification |
+| `data` | `any` | — | Opaque content (overrides auto-computed vector if provided) |
+| `metadata` | `object` | — | Structured fields on the edge |
+| `weight` | `number` | — | Connection strength (0-1, default: 1.0) |
+| `confidence` | `number` | — | Relationship certainty (0-1) |
+| `evidence` | `RelationEvidence` | — | Why this relationship was detected |
+| `createdAt` | `number` | — | Creation timestamp (ms since epoch) |
+| `updatedAt` | `number` | — | Last update timestamp (ms since epoch) |
+| `service` | `string` | — | Multi-tenancy identifier |
+
+### Example
+
+```typescript
+const relId = await brain.relate({
+ from: personId,
+ to: projectId,
+ type: VerbType.WorksOn,
+ data: 'Lead engineer on the AI module', // Optional: content for this edge
+ metadata: { // Optional: queryable edge fields
+ role: 'lead',
+ startDate: '2024-01-15'
+ },
+ weight: 0.9
+})
+```
+
+---
+
+## Data vs Metadata
+
+This is the most important concept in Brainy's storage model:
+
+### `data` — Content for Semantic Search
+
+- Embedded into a 384-dimensional vector via the WASM embedding engine
+- Searchable via **semantic similarity** (HNSW vector index) and **hybrid text+semantic** search
+- Queried by passing `query` to `find()`:
+ ```typescript
+ brain.find({ query: 'machine learning algorithms' })
+ ```
+- **NOT** indexed by MetadataIndex — you cannot use `where` filters on `data`
+- Stored opaquely: strings, objects, numbers — anything goes
+
+### `metadata` — Structured Queryable Fields
+
+- Indexed by MetadataIndex with O(1) lookups per field
+- Queryable via `where` filters using [BFO operators](./QUERY_OPERATORS.md):
+ ```typescript
+ brain.find({
+ where: {
+ department: 'engineering',
+ yearsExperience: { greaterThan: 5 },
+ tags: { contains: 'senior' }
+ }
+ })
+ ```
+- **NOT** used for vector/semantic search
+- Must be a flat or lightly nested object
+
+### Quick Reference
+
+| | `data` | `metadata` |
+|---|---|---|
+| **Purpose** | Content for embedding / semantic search | Structured fields for filtering |
+| **Searched by** | `find({ query })` — vector similarity, hybrid text+semantic | `find({ where })` — exact, range, set operators |
+| **Indexed by** | HNSW vector index | MetadataIndex |
+| **Queryable with operators?** | No | Yes (`equals`, `greaterThan`, `oneOf`, etc.) |
+| **Auto-embedded?** | Yes (strings → 384-dim vectors) | No |
+| **Typical content** | Text descriptions, document content | Tags, dates, status flags, categories, numeric fields |
+
+### Common Pattern
+
+```typescript
+// Add an article
+await brain.add({
+ data: 'A deep dive into transformer architectures and attention mechanisms',
+ type: NounType.Document,
+ metadata: {
+ title: 'Transformer Deep Dive',
+ author: 'Dr. Chen',
+ publishedYear: 2024,
+ tags: ['AI', 'transformers', 'NLP'],
+ status: 'published'
+ }
+})
+
+// Search by content (semantic — searches data)
+const results = await brain.find({ query: 'neural network attention' })
+
+// Filter by fields (exact — queries metadata)
+const recent = await brain.find({
+ where: {
+ publishedYear: { greaterThan: 2023 },
+ status: 'published'
+ }
+})
+
+// Combine both (Triple Intelligence)
+const precise = await brain.find({
+ query: 'attention mechanisms', // Semantic search on data
+ where: { author: 'Dr. Chen' }, // Metadata filter
+ connected: { from: authorId, depth: 1 } // Graph traversal
+})
+```
+
+---
+
+## Storage Field Naming
+
+Internally, Brainy uses different field names in storage vs the public API:
+
+| Public API (Entity/Relation) | Storage (metadata object) | Notes |
+|------------------------------|--------------------------|-------|
+| `type` | `noun` | Entity type stored as `noun` |
+| `from` | `sourceId` | Relationship source |
+| `to` | `targetId` | Relationship target |
+| `type` (on Relation) | `verb` | Relationship type stored as `verb` |
+
+When querying with `find()`, you can use:
+- `type` parameter (convenience alias, equivalent to `where.noun`)
+- `where.noun` directly
+
+```typescript
+// These are equivalent:
+brain.find({ type: NounType.Person })
+brain.find({ where: { noun: NounType.Person } })
+```
+
+---
+
+## Standard Metadata Fields
+
+When you add an entity, Brainy stores these standard fields in the metadata object alongside your custom fields:
+
+| Field | Set By | Description |
+|-------|--------|-------------|
+| `noun` | System | Entity type (NounType enum value) |
+| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. |
+| `data` | System | The raw `data` value (stored opaquely) |
+| `createdAt` | System | Creation timestamp |
+| `updatedAt` | System | Last update timestamp |
+| `confidence` | User | Type classification confidence |
+| `weight` | User | Entity importance |
+| `service` | User | Multi-tenancy identifier |
+| `createdBy` | User/System | Source augmentation |
+
+On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**.
+
+### Subtype — sub-classification within a NounType
+
+`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type:
+
+```typescript
+// A Person who is an employee:
+await brain.add({
+ data: 'Avery Brooks — runs the AI lab',
+ type: NounType.Person,
+ subtype: 'employee',
+ metadata: { department: 'ai-lab' }
+})
+
+// A Document that is an invoice:
+await brain.add({
+ data: 'INV-2026-001',
+ type: NounType.Document,
+ subtype: 'invoice',
+ metadata: { amount: 1500 }
+})
+```
+
+`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`.
+
+### Subtype — sub-classification within a VerbType (7.30+)
+
+Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive — a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field on `HNSWVerbWithMetadata` and on the public `Relation`:
+
+```typescript
+await brain.relate({
+ from: ceoId,
+ to: vpId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct', // top-level standard field
+ metadata: { since: '2025-Q1' } // user-custom fields stay in metadata
+})
+```
+
+Fast-path filter on the verb side:
+
+```typescript
+const direct = await brain.related({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct'
+})
+```
+
+The verb-side rollup at `_system/verb-subtype-statistics.json` mirrors the noun-side `_system/subtype-statistics.json` — same shape, same self-heal machinery. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`.
+
+Verbs and nouns now have full capability parity — every API on the noun side has a verb-side mirror, including the new `brain.updateRelation()` (which closed a pre-7.30 gap where relationships had no update path).
+
+### Standard verb fields
+
+The verb-side equivalent of `STANDARD_ENTITY_FIELDS` is `STANDARD_VERB_FIELDS`, exported from `src/coreTypes.ts`. Verb-specific standard fields:
+
+| Field | Description |
+|---|---|
+| `verb` | The VerbType enum value |
+| `sourceId` / `targetId` | The two endpoints of the relationship |
+| `subtype` | Sub-classification within the VerbType (7.30+) |
+| `confidence`, `weight`, `createdAt`, `updatedAt`, `service`, `createdBy`, `data` | Same semantics as the noun-side standard fields |
+
+The companion `resolveVerbField(verb, field)` helper resolves field paths the same way `resolveEntityField` does for nouns: standard fields first, metadata fallback for everything else.
+
+---
+
+## See Also
+
+- [API Reference](./api/README.md) — Complete API documentation
+- [Query Operators](./QUERY_OPERATORS.md) — All BFO operators with examples
+- [Find System](./FIND_SYSTEM.md) — Natural language find() details
diff --git a/docs/DEVELOPER_LEARNING_PATH.md b/docs/DEVELOPER_LEARNING_PATH.md
index 3436d922..4134ae63 100644
--- a/docs/DEVELOPER_LEARNING_PATH.md
+++ b/docs/DEVELOPER_LEARNING_PATH.md
@@ -193,21 +193,21 @@ console.log('✅ Created relationships')
console.log('\n🔍 Querying relationships...')
// Who works for TechCorp?
-const techcorpEmployees = await brain.getRelations({
+const techcorpEmployees = await brain.related({
to: techcorpId,
type: VerbType.WorksWith
})
console.log(`TechCorp has ${techcorpEmployees.length} employees`)
// Who works on the AI project?
-const projectContributors = await brain.getRelations({
+const projectContributors = await brain.related({
to: projectId,
type: [VerbType.WorksOn, VerbType.Manages]
})
console.log(`AI Project has ${projectContributors.length} contributors`)
// Who does John collaborate with?
-const johnsCollaborators = await brain.getRelations({
+const johnsCollaborators = await brain.related({
from: johnId,
type: VerbType.CollaboratesWith
})
@@ -294,7 +294,7 @@ await brain.update({
1. Create an organizational hierarchy (CEO → Managers → Engineers)
2. Build a project dependency graph
3. Model a social network with CollaboratesWith relationships
-4. Query "Who reports to Alice?" using getRelations()
+4. Query "Who reports to Alice?" using related()
5. Batch update all projects to add a "year: 2024" field
### Next Steps
@@ -429,23 +429,7 @@ fusionResults.forEach(r => {
}
})
-// 5. NEURAL API: Automatic clustering
-console.log('\n\n🤖 NEURAL API: Automatic Clustering')
-const neural = brain.neural()
-
-const clusters = await neural.clusters({
- maxClusters: 3,
- minClusterSize: 1
-})
-
-console.log(`Found ${clusters.length} semantic clusters:`)
-clusters.forEach((cluster, i) => {
- console.log(`\n Cluster ${i + 1}: ${cluster.label || cluster.id}`)
- console.log(` Members: ${cluster.members.length}`)
- console.log(` Centroid topics: ${cluster.metadata?.topics?.join(', ') || 'N/A'}`)
-})
-
-// 6. SIMILARITY: Find similar documents
+// 5. SIMILARITY: Find similar documents
console.log('\n\n🔍 SIMILARITY: Find Similar Documents')
const similarTo = await brain.similar({
to: paper1, // Entity ID of first AI paper
@@ -459,19 +443,6 @@ similarTo.forEach(r => {
console.log(` [${r.score.toFixed(3)}] ${r.entity.data?.substring(0, 50)}...`)
})
-// 7. OUTLIER DETECTION
-console.log('\n\n🚨 OUTLIER DETECTION')
-const outliers = await neural.outliers({
- method: 'statistical',
- threshold: 2.0 // 2 standard deviations
-})
-
-console.log(`Found ${outliers.length} outlier documents:`)
-outliers.forEach(o => {
- const entity = await brain.get(o.id)
- console.log(` [Anomaly score: ${o.score.toFixed(3)}] ${entity?.data?.substring(0, 50)}...`)
-})
-
await brain.close()
```
@@ -613,7 +584,7 @@ vfsFiles.forEach(f => {
})
// 3. VFS FILTERING IN KNOWLEDGE QUERIES
-console.log('\n🔍 Understanding VFS filtering (v4.4.0)...\n')
+console.log('\n🔍 Understanding VFS filtering...\n')
// Create some knowledge entities
const conceptId = await brain.add({
@@ -676,7 +647,7 @@ if (readmeEntity.length > 0) {
}
// Query relationships
-const conceptDocs = await brain.getRelations({
+const conceptDocs = await brain.related({
from: conceptId,
type: VerbType.DocumentedBy
})
@@ -758,7 +729,7 @@ await brain.close()
### Key Concepts
-#### 1. **VFS Filtering Architecture (v4.4.0)**
+#### 1. **VFS Filtering Architecture**
```typescript
// 🎯 DEFAULT BEHAVIOR: Clean Separation
@@ -798,7 +769,7 @@ await brain.relate({
})
// Query across boundaries
-const conceptDocs = await brain.getRelations({
+const conceptDocs = await brain.related({
from: conceptId,
type: VerbType.DocumentedBy
})
@@ -852,7 +823,7 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
## Level 5: Production Scale (90 minutes)
### What You'll Learn
-- Cloud storage (GCS, S3, R2)
+- Production filesystem storage and off-site backup
- Performance optimization
- Batch imports (CSV, Excel, PDF)
- Metadata query optimization
@@ -863,18 +834,13 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
-// 1. PRODUCTION STORAGE - Google Cloud Storage (Native SDK)
-console.log('☁️ Initializing production storage...\n')
+// 1. PRODUCTION STORAGE - Filesystem with off-site snapshots
+console.log('Initializing production storage...\n')
const brain = new Brainy({
storage: {
- type: 'gcs-native', // Native GCS SDK (recommended)
- gcsNativeStorage: {
- bucketName: 'my-brainy-production',
- // ADC (Application Default Credentials) - zero config in Cloud Run/GCE!
- // Or provide credentials:
- // keyFilename: '/path/to/service-account.json'
- }
+ type: 'filesystem',
+ path: '/var/lib/brainy'
},
// Performance tuning
@@ -888,7 +854,8 @@ const brain = new Brainy({
})
await brain.init()
-console.log('✅ Brainy initialized with GCS Native storage\n')
+console.log('Brainy initialized with filesystem storage')
+console.log('Snapshot /var/lib/brainy off-site via cron with `gsutil rsync` / `aws s3 sync` / `rclone`.\n')
// 2. BATCH IMPORT - CSV File
console.log('📊 Importing CSV data...\n')
@@ -967,26 +934,6 @@ console.log(` Failed: ${batchResult.failed.length}`)
console.log(` Duration: ${duration}ms`)
console.log(` Throughput: ${(batchResult.successful.length / (duration / 1000)).toFixed(0)} entities/sec`)
-// 5. ADVANCED CLUSTERING (Large Dataset)
-console.log('\n\n🤖 Clustering 1000 entities...\n')
-
-const neural = brain.neural()
-
-// Use fast clustering for large datasets
-const clusters = await neural.clusterFast({
- maxClusters: 5
-})
-
-console.log(`Found ${clusters.length} clusters:`)
-clusters.forEach((cluster, i) => {
- console.log(`\n Cluster ${i + 1}: ${cluster.label || cluster.id}`)
- console.log(` Size: ${cluster.members.length} members`)
- console.log(` Density: ${(cluster.density || 0).toFixed(3)}`)
- if (cluster.metadata?.keywords) {
- console.log(` Keywords: ${cluster.metadata.keywords.slice(0, 5).join(', ')}`)
- }
-})
-
// 6. PRODUCTION STATISTICS
console.log('\n\n📊 Production Statistics:\n')
@@ -1044,7 +991,7 @@ console.log('✅ Brain closed cleanly')
console.log('\n\n🎓 Production Deployment Complete!')
console.log('\n📚 Key Production Learnings:')
-console.log(' 1. Use native cloud storage (GCS, S3, R2) for persistence')
+console.log(' 1. Use filesystem storage and snapshot the data directory off-site from your scheduler')
console.log(' 2. Batch operations = 100x faster than individual ops')
console.log(' 3. Metadata query optimization for complex filters')
console.log(' 4. Monitor query performance with explain: true')
@@ -1057,41 +1004,22 @@ console.log(' 7. Stream large imports with progress callbacks')
#### 1. **Storage Options Comparison**
-| Storage | Use Case | Performance | Cost | Setup |
-|---------|----------|-------------|------|-------|
-| Memory | Dev/testing | Fastest | Free | Zero config |
-| Filesystem | Local prod | Fast | Free | Local path |
-| GCS Native | GCP prod | Fast | $$$ | Service account |
-| S3 | AWS prod | Fast | $$$ | Access keys |
-| R2 | Cloudflare | Fast | $ | Access keys |
+| Storage | Use Case | Performance | Setup |
+|---------|----------|-------------|-------|
+| Memory | Dev/testing | Fastest | Zero config |
+| Filesystem | Production | Fast | Local path + scheduled off-site backup |
-#### 2. **GCS Native vs S3-Compatible**
+#### 2. **Off-Site Backup**
-```typescript
-// ✅ RECOMMENDED: GCS Native SDK
-{
- storage: {
- type: 'gcs-native',
- gcsNativeStorage: {
- bucketName: 'my-bucket'
- // ADC handles auth automatically in Cloud Run/GCE!
- }
- }
-}
-
-// ⚠️ LEGACY: S3-compatible mode
-{
- storage: {
- type: 'gcs',
- gcsStorage: {
- bucketName: 'my-bucket',
- accessKeyId: process.env.GCS_ACCESS_KEY,
- secretAccessKey: process.env.GCS_SECRET_KEY
- }
- }
-}
+```bash
+# Cron / systemd timer / k8s CronJob — pick whatever you already operate
+*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
+*/15 * * * * aws s3 sync /var/lib/brainy s3://my-bucket/brainy-backup
+*/15 * * * * gsutil rsync -r /var/lib/brainy gs://my-bucket/brainy-backup
```
+Brainy itself never reaches out to an object store. Snapshot `path` from your scheduler.
+
#### 3. **Performance Optimization**
```typescript
@@ -1164,8 +1092,8 @@ const brain = new Brainy({ verbose: true })
#### Before Deployment
-- [ ] Choose cloud storage (GCS/S3/R2)
-- [ ] Set up authentication (service account/access keys)
+- [ ] Provision a writable `path` on the host
+- [ ] Set up an off-site snapshot job (`rclone` / `aws s3 sync` / `gsutil rsync`)
- [ ] Configure caching
- [ ] Test batch operations
- [ ] Benchmark query performance
@@ -1189,12 +1117,12 @@ const brain = new Brainy({ verbose: true })
### Practice Exercises
-1. Deploy Brainy with GCS Native storage
+1. Deploy Brainy with filesystem storage + scheduled off-site backup
2. Import a 10,000 row CSV file
3. Measure query performance for different filters
4. Optimize a slow query using getOptimalQueryPlan()
5. Set up monitoring dashboard
-6. Create backup/restore scripts
+6. Test restore from off-site snapshot
---
@@ -1212,7 +1140,7 @@ const brain = new Brainy({ verbose: true })
#### Advanced Topics
-- **Distributed Systems**: Sharding, replication, coordination
+- **Multi-instance Deployments**: Run multiple Brainy processes behind your own routing layer
- **Custom Augmentations**: Extend Brainy with plugins
- **Streaming Pipelines**: Real-time data ingestion
- **Security**: Encryption, access control, audit logs
@@ -1220,10 +1148,9 @@ const brain = new Brainy({ verbose: true })
#### Resources
-- 📚 [API Reference](../API_REFERENCE.md) - Complete API documentation
+- 📚 [API Reference](../api/README.md) - Complete API documentation
- 📁 [VFS Guide](../vfs/VFS_API_GUIDE.md) - Virtual Filesystem deep dive
- 🤖 [Neural API](../guides/neural-api.md) - Advanced neural operations
-- 🌐 [Distributed Guide](../guides/distributed-system.md) - Planet-scale architecture
- 💬 [Discord Community](https://discord.gg/brainy) - Get help, share projects
#### Share Your Success
diff --git a/docs/EXTENDING_STORAGE.md b/docs/EXTENDING_STORAGE.md
deleted file mode 100644
index 8ecc700a..00000000
--- a/docs/EXTENDING_STORAGE.md
+++ /dev/null
@@ -1,396 +0,0 @@
-# 🔌 Extending Brainy Storage with Augmentations
-
-## Overview
-
-Brainy's zero-config system is **fully extensible**. Augmentations can register new storage providers, presets, and auto-detection logic that integrates seamlessly with the existing system.
-
-## How Storage Extensions Work
-
-### 1. Storage Provider Registration
-
-When an augmentation is installed, it can register a new storage provider:
-
-```typescript
-import { StorageProvider, registerStorageAugmentation } from '@soulcraft/brainy/config'
-
-const redisProvider: StorageProvider = {
- type: 'redis',
- name: 'Redis Storage',
- description: 'High-performance in-memory data store',
- priority: 10, // Higher priority = checked first in auto-detection
-
- // Auto-detection logic
- async detect(): Promise {
- // Check if Redis is available
- if (process.env.REDIS_URL) {
- try {
- const redis = await import('ioredis')
- const client = new redis.default(process.env.REDIS_URL)
- await client.ping()
- await client.quit()
- return true
- } catch {
- return false
- }
- }
- return false
- },
-
- // Configuration builder
- async getConfig(): Promise {
- return {
- type: 'redis',
- redisStorage: {
- url: process.env.REDIS_URL,
- prefix: 'brainy:',
- ttl: 3600
- }
- }
- }
-}
-
-// Register the provider
-registerStorageAugmentation(redisProvider)
-```
-
-### 2. Using Extended Storage
-
-Once registered, the new storage type works with zero-config:
-
-```typescript
-// Auto-detection will now check Redis
-const brain = new Brainy() // Will use Redis if available!
-
-// Or explicitly specify
-const brain = new Brainy({ storage: 'redis' })
-
-// Or with custom config
-const brain = new Brainy({
- storage: {
- type: 'redis',
- redisStorage: {
- url: 'redis://localhost:6379',
- prefix: 'myapp:'
- }
- }
-})
-```
-
-## Real-World Examples
-
-### Redis Augmentation
-
-```typescript
-// @soulcraft/brainy-redis package
-export class RedisStorageAugmentation {
- async init() {
- // Register the storage provider
- registerStorageAugmentation({
- type: 'redis',
- name: 'Redis Storage',
- priority: 10,
-
- async detect() {
- return !!(process.env.REDIS_URL || process.env.REDIS_HOST)
- },
-
- async getConfig() {
- return {
- type: 'redis',
- redisStorage: {
- url: process.env.REDIS_URL ||
- `redis://${process.env.REDIS_HOST}:${process.env.REDIS_PORT || 6379}`
- }
- }
- }
- })
-
- // Register Redis-specific presets
- registerPresetAugmentation('redis-cache', {
- storage: 'redis',
- model: ModelPrecision.Q8,
- features: ['core', 'cache'],
- distributed: true,
- description: 'Redis-backed cache layer',
- category: PresetCategory.SERVICE
- })
- }
-}
-```
-
-### MongoDB Augmentation
-
-```typescript
-// @soulcraft/brainy-mongodb package
-export class MongoStorageAugmentation {
- async init() {
- registerStorageAugmentation({
- type: 'mongodb',
- name: 'MongoDB Storage',
- priority: 8,
-
- async detect() {
- return !!(process.env.MONGODB_URI || process.env.MONGO_URL)
- },
-
- async getConfig() {
- return {
- type: 'mongodb',
- mongoStorage: {
- uri: process.env.MONGODB_URI,
- database: 'brainy',
- collection: 'vectors'
- }
- }
- }
- })
- }
-}
-```
-
-### PostgreSQL + pgvector Augmentation
-
-```typescript
-// @soulcraft/brainy-postgres package
-export class PostgresStorageAugmentation {
- async init() {
- registerStorageAugmentation({
- type: 'postgres',
- name: 'PostgreSQL + pgvector',
- priority: 9,
-
- async detect() {
- const url = process.env.DATABASE_URL
- if (url?.includes('postgres')) {
- // Check for pgvector extension
- const client = new Client({ connectionString: url })
- await client.connect()
- const result = await client.query(
- "SELECT * FROM pg_extension WHERE extname = 'vector'"
- )
- await client.end()
- return result.rows.length > 0
- }
- return false
- },
-
- async getConfig() {
- return {
- type: 'postgres',
- postgresStorage: {
- connectionString: process.env.DATABASE_URL,
- table: 'brainy_vectors'
- }
- }
- }
- })
- }
-}
-```
-
-## Auto-Detection Priority
-
-Storage providers are checked in priority order:
-
-1. **Custom providers** (highest priority first)
-2. **Cloud storage** (S3, GCS, R2)
-3. **Database storage** (Redis, MongoDB, PostgreSQL)
-4. **Local storage** (filesystem, OPFS)
-5. **Memory** (fallback)
-
-```typescript
-// Example priority chain
-Redis (priority: 10) → PostgreSQL (9) → MongoDB (8) → S3 (5) → Filesystem (1) → Memory (0)
-```
-
-## Creating Custom Presets
-
-Augmentations can also register new presets:
-
-```typescript
-registerPresetAugmentation('redis-cluster', {
- storage: 'redis',
- model: ModelPrecision.Q8,
- features: ['core', 'cache', 'cluster'],
- distributed: true,
- role: DistributedRole.HYBRID,
- cache: {
- hotCacheMaxSize: 100000, // Large distributed cache
- autoTune: true
- },
- description: 'Redis Cluster configuration',
- category: PresetCategory.SERVICE
-})
-
-// Users can then use:
-const brain = new Brainy('redis-cluster')
-```
-
-## Type Safety with Extensions
-
-To maintain type safety with dynamic storage types:
-
-```typescript
-// Augmentation declares its types
-declare module '@soulcraft/brainy' {
- interface StorageTypes {
- redis: {
- url: string
- prefix?: string
- ttl?: number
- }
- }
-
- interface PresetNames {
- 'redis-cache': 'redis-cache'
- 'redis-cluster': 'redis-cluster'
- }
-}
-```
-
-## Best Practices for Storage Augmentations
-
-1. **Always provide auto-detection** - Check environment variables and connectivity
-2. **Set appropriate priority** - Higher for specialized storage, lower for general
-3. **Handle failures gracefully** - Return false from detect() if not available
-4. **Document requirements** - List required packages and environment variables
-5. **Provide presets** - Include common configuration patterns
-6. **Maintain compatibility** - Ensure model precision matches across instances
-
-## Example: Complete Redis Augmentation
-
-```typescript
-import {
- StorageProvider,
- registerStorageAugmentation,
- registerPresetAugmentation,
- PresetCategory,
- ModelPrecision,
- DistributedRole
-} from '@soulcraft/brainy/config'
-import Redis from 'ioredis'
-
-export class BrainyRedisAugmentation {
- private client: Redis
-
- async init() {
- // Register storage provider
- registerStorageAugmentation({
- type: 'redis',
- name: 'Redis Vector Storage',
- description: 'Redis with RediSearch for vector similarity',
- priority: 10,
-
- requirements: {
- env: ['REDIS_URL'],
- packages: ['ioredis', 'redis']
- },
-
- async detect() {
- if (!process.env.REDIS_URL) return false
-
- try {
- const client = new Redis(process.env.REDIS_URL)
-
- // Check for RediSearch module
- const modules = await client.call('MODULE', 'LIST')
- const hasRediSearch = modules.some(m => m[1] === 'search')
-
- await client.quit()
- return hasRediSearch
- } catch {
- return false
- }
- },
-
- async getConfig() {
- return {
- type: 'redis',
- redisStorage: {
- url: process.env.REDIS_URL,
- prefix: process.env.REDIS_PREFIX || 'brainy:',
- index: process.env.REDIS_INDEX || 'brainy-vectors',
- ttl: process.env.REDIS_TTL ? parseInt(process.env.REDIS_TTL) : undefined
- }
- }
- }
- })
-
- // Register presets
- this.registerPresets()
- }
-
- private registerPresets() {
- // Fast cache preset
- registerPresetAugmentation('redis-fast-cache', {
- storage: 'redis' as any,
- model: ModelPrecision.Q8,
- features: ['core', 'cache', 'search'],
- distributed: false,
- cache: {
- hotCacheMaxSize: 10000,
- autoTune: true
- },
- description: 'Redis-backed fast cache',
- category: PresetCategory.SERVICE
- })
-
- // Distributed cache preset
- registerPresetAugmentation('redis-distributed', {
- storage: 'redis' as any,
- model: ModelPrecision.AUTO,
- features: ['core', 'cache', 'search', 'cluster'],
- distributed: true,
- role: DistributedRole.HYBRID,
- cache: {
- hotCacheMaxSize: 50000,
- autoTune: true
- },
- description: 'Redis distributed cache cluster',
- category: PresetCategory.SERVICE
- })
-
- // Session store preset
- registerPresetAugmentation('redis-sessions', {
- storage: 'redis' as any,
- model: ModelPrecision.Q8,
- features: ['core', 'cache'],
- distributed: false,
- cache: {
- hotCacheMaxSize: 5000,
- autoTune: false
- },
- description: 'Redis session storage',
- category: PresetCategory.SERVICE
- })
- }
-}
-
-// Usage after installing the augmentation:
-import { Brainy } from '@soulcraft/brainy'
-import '@soulcraft/brainy-redis' // Registers the augmentation
-
-// Now Redis is automatically detected!
-const brain = new Brainy() // Uses Redis if REDIS_URL is set
-
-// Or use a Redis preset
-const brain = new Brainy('redis-fast-cache')
-
-// Or explicitly configure
-const brain = new Brainy({
- storage: 'redis',
- model: ModelPrecision.FP32
-})
-```
-
-## Summary
-
-The extensible configuration system allows:
-
-1. **New storage types** via `registerStorageAugmentation()`
-2. **Custom presets** via `registerPresetAugmentation()`
-3. **Auto-detection logic** that integrates with zero-config
-4. **Type-safe extensions** with TypeScript declarations
-5. **Priority-based selection** for intelligent defaults
-
-This ensures Brainy can grow with new storage technologies while maintaining its zero-configuration philosophy!
\ No newline at end of file
diff --git a/docs/FIND_SYSTEM.md b/docs/FIND_SYSTEM.md
index f43df101..1cc38ce9 100644
--- a/docs/FIND_SYSTEM.md
+++ b/docs/FIND_SYSTEM.md
@@ -1,29 +1,49 @@
+---
+title: The Find System
+slug: guides/find-system
+public: true
+category: guides
+template: guide
+order: 3
+description: Complete guide to Brainy's find() method — four intelligence systems, query execution phases, NLP patterns, and performance from 1K to 10M entities.
+next:
+ - concepts/triple-intelligence
+ - api/reference
+---
+
# Brainy's Find System - Complete Guide
## Overview
Brainy's `find()` method is the most advanced query system in any vector database, combining **Triple Intelligence** (vector + metadata + graph) with **Type-Aware NLP** for natural language understanding.
-## Architecture: Three Intelligence Systems
+## Architecture: Four Intelligence Systems
### 1. Vector Intelligence (HNSW Index)
- **Purpose**: Semantic similarity search using embeddings
- **Algorithm**: Hierarchical Navigable Small World (HNSW)
-- **Performance**: O(log n) search, ~1.8ms typical
+- **Performance**: O(log n) search
- **Data Structure**: Multi-layer graph with 16 connections per node
- **Use Cases**: "Find similar documents", "Content like this"
-### 2. Metadata Intelligence (Incremental Indices)
+### 2. Text Intelligence (Word Index) - **Purpose**: Keyword/exact text matching
+- **Algorithm**: Inverted word index with FNV-1a hashing
+- **Performance**: O(log C) where C = chunks (~50 values each)
+- **Data Structure**: `__words__ → hash → Roaring Bitmap of entity IDs`
+- **Use Cases**: "Find exact name", "Keyword search"
+- **Integration**: Automatically combined with Vector via RRF fusion
+
+### 3. Metadata Intelligence (Incremental Indices)
- **Purpose**: Fast filtering on structured data
- **Algorithm**: HashMap for exact matches, Sorted arrays for ranges
-- **Performance**: O(1) exact, O(log n) ranges, <1ms typical
+- **Performance**: O(1) exact, O(log n) ranges
- **Data Structure**: `Map>` + sorted value arrays
- **Use Cases**: "Documents from 2023", "Status equals active"
-### 3. Graph Intelligence (Adjacency Maps)
+### 4. Graph Intelligence (Adjacency Maps)
- **Purpose**: Relationship traversal and connection analysis
- **Algorithm**: Pure O(1) neighbor lookups via Map operations
-- **Performance**: O(1) per hop, ~0.1ms typical
+- **Performance**: O(1) per hop (measured <1 ms per neighbor lookup, validated up to 1M relationships — `tests/performance/graph-scale-performance.test.ts:238`)
- **Data Structure**: `Map>`
- **Use Cases**: "Papers connected to MIT", "Authors who collaborated"
@@ -46,29 +66,29 @@ await brain.find("documents by Smith published after 2020 with high citations")
```typescript
// Direct query objects with full control
await brain.find({
- // Vector search
- query: "machine learning research",
-
- // Metadata filters
- where: {
- publishDate: { greaterThan: 2020 },
- citations: { between: [50, 1000] },
- status: "published"
- },
-
- // Type constraints
- type: NounType.Document,
-
- // Graph traversal
- connected: {
- to: "mit-ai-lab",
- via: VerbType.AffiliatedWith,
- depth: 2
- },
-
- // Control options
- limit: 20,
- explain: true // Get scoring breakdown
+ // Vector search
+ query: "machine learning research",
+
+ // Metadata filters
+ where: {
+ publishDate: { greaterThan: 2020 },
+ citations: { between: [50, 1000] },
+ status: "published"
+ },
+
+ // Type constraints
+ type: NounType.Document,
+
+ // Graph traversal
+ connected: {
+ to: "mit-ai-lab",
+ via: VerbType.AffiliatedWith,
+ depth: 2
+ },
+
+ // Control options
+ limit: 20,
+ explain: true // Get scoring breakdown
})
```
@@ -76,14 +96,92 @@ await brain.find({
```typescript
// Find entities similar to a specific item
await brain.find({
- near: {
- id: "doc-123",
- threshold: 0.8 // Minimum similarity
- },
- type: NounType.Document
+ near: {
+ id: "doc-123",
+ threshold: 0.8 // Minimum similarity
+ },
+ type: NounType.Document
})
```
+### 4. Hybrid Search
+```typescript
+// Zero-config hybrid: automatically combines text + semantic search
+await brain.find({ query: "David Smith" })
+// Uses Reciprocal Rank Fusion (RRF) to combine results
+
+// Force text-only search
+await brain.find({ query: "exact keyword", searchMode: 'text' })
+
+// Force semantic-only search
+await brain.find({ query: "AI concepts", searchMode: 'semantic' })
+
+// Custom hybrid weighting (0 = text only, 1 = semantic only)
+await brain.find({ query: "search term", hybridAlpha: 0.3 })
+```
+
+**How Auto-Alpha Works:**
+- Short queries (1-2 words): alpha = 0.3 (favor text matching)
+- Medium queries (3-4 words): alpha = 0.5 (balanced)
+- Long queries (5+ words): alpha = 0.7 (favor semantic matching)
+
+### 5. Match Visibility
+
+Search results include match details showing what matched:
+
+```typescript
+const results = await brain.find({ query: 'david the warrior' })
+
+// Each result has:
+results[0].textMatches // ["david", "warrior"] - exact words found
+results[0].textScore // 0.25 - text match quality (0-1)
+results[0].semanticScore // 0.87 - semantic similarity (0-1)
+results[0].matchSource // 'both' | 'text' | 'semantic'
+```
+
+Use this for:
+- **Highlighting** exact matches in UI (textMatches)
+- **Explaining** why a result was found (matchSource)
+- **Debugging** search behavior (separate scores)
+
+### 6. Semantic Highlighting
+
+Highlight which concepts/words in text matched your query:
+
+```typescript
+// Find semantically similar words + exact matches
+const highlights = await brain.highlight({
+ query: "david the warrior",
+ text: "David Smith is a brave fighter who battles dragons"
+})
+
+// Returns:
+// [
+// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' },
+// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' },
+// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
+// ]
+```
+
+**Features:**
+- `matchType: 'text'` - Exact word match (score = 1.0)
+- `matchType: 'semantic'` - Concept match (score varies)
+- `position` - [start, end] for precise highlighting
+- `granularity` - 'word' (default), 'phrase', or 'sentence'
+- `threshold` - Minimum semantic score (default: 0.5)
+
+**UI Usage Pattern:**
+```typescript
+// Highlight search results with different styles
+function highlightResult(text: string, highlights: Highlight[]) {
+ return highlights.map(h => ({
+ text: h.text,
+ position: h.position,
+ style: h.matchType === 'text' ? 'strong' : 'emphasis' // Different UI styles
+ }))
+}
+```
+
## Index Usage in Detail
### Metadata Index Operations
@@ -96,7 +194,7 @@ await brain.find({
// Memory: ~40 bytes per unique field-value combination
```
-#### Sorted Index (Range Queries)
+#### Sorted Index (Range Queries)
```typescript
// Query: {publishDate: {greaterThan: 2020}}
// Index: sortedIndices.get("publishDate") → [[2019, Set], [2020, Set], [2021, Set]]
@@ -122,7 +220,7 @@ await brain.find({
// Query: {query: "machine learning"}
// Process:
// 1. Embed query text → 384-dimensional vector
-// 2. Start at top layer (entry point)
+// 2. Start at top layer (entry point)
// 3. Greedy search for nearest neighbor at each layer
// 4. Move down layers for progressively finer search
// 5. Return k nearest neighbors with similarity scores
@@ -135,10 +233,10 @@ await brain.find({
#### O(1) Neighbor Lookups
```typescript
// Query: {connected: {to: "entity-123"}}
-// Index lookup:
-// - Outgoing: sourceIndex.get("entity-123") → Set
-// - Incoming: targetIndex.get("entity-123") → Set
-// - Both directions: union of both Sets
+// Index lookup:
+// - Outgoing: sourceIndex.get("entity-123") → Set
+// - Incoming: targetIndex.get("entity-123") → Set
+// - Both directions: union of both Sets
// Performance: O(1) per hop, no matter the graph size
// Memory: ~24 bytes per relationship (source + target + metadata)
```
@@ -148,19 +246,19 @@ await brain.find({
### Phase 1: Query Parsing
```typescript
if (typeof query === 'string') {
- // Natural Language Processing
- const nlpParams = await this.parseNaturalQuery(query)
-
- // Type-aware parsing:
- // 1. Detect NounType using pre-embedded type vectors
- // 2. Get type-specific fields from real data patterns
- // 3. Semantic field matching with type affinity boosting
- // 4. Validate field-type compatibility
- // 5. Generate optimized query plan
-
- params = nlpParams
+ // Natural Language Processing
+ const nlpParams = await this.parseNaturalQuery(query)
+
+ // Type-aware parsing:
+ // 1. Detect NounType using pre-embedded type vectors
+ // 2. Get type-specific fields from real data patterns
+ // 3. Semantic field matching with type affinity boosting
+ // 4. Validate field-type compatibility
+ // 5. Generate optimized query plan
+
+ params = nlpParams
} else {
- params = query // Direct structured query
+ params = query // Direct structured query
}
```
@@ -171,12 +269,12 @@ const searchPromises = []
// Vector search (if query text or vector provided)
if (params.query || params.vector) {
- searchPromises.push(this.executeVectorSearch(params))
+ searchPromises.push(this.executeVectorSearch(params))
}
-// Proximity search (if near parameter provided)
+// Proximity search (if near parameter provided)
if (params.near) {
- searchPromises.push(this.executeProximitySearch(params))
+ searchPromises.push(this.executeProximitySearch(params))
}
// Wait for all searches to complete
@@ -187,41 +285,41 @@ const searchResults = await Promise.all(searchPromises)
```typescript
// Apply metadata filters using optimized indices
if (params.where || params.type || params.service) {
- const filter = {
- ...params.where,
- ...(params.type && { noun: params.type }),
- ...(params.service && { service: params.service })
- }
-
- // Get optimal query plan based on field cardinalities
- const queryPlan = await this.getOptimalQueryPlan(filter)
-
- // Execute filters in optimal order (low cardinality first)
- const filteredIds = await this.metadataIndex.getIdsForFilter(filter)
-
- if (results.length > 0) {
- // Intersect with vector search results
- results = results.filter(r => filteredIds.includes(r.id))
- } else {
- // Create results from metadata matches (metadata-only query)
- results = await this.createResultsFromIds(filteredIds)
- }
+ const filter = {
+ ...params.where,
+ ...(params.type && { noun: params.type }),
+ ...(params.service && { service: params.service })
+ }
+
+ // Get optimal query plan based on field cardinalities
+ const queryPlan = await this.getOptimalQueryPlan(filter)
+
+ // Execute filters in optimal order (low cardinality first)
+ const filteredIds = await this.metadataIndex.getIdsForFilter(filter)
+
+ if (results.length > 0) {
+ // Intersect with vector search results
+ results = results.filter(r => filteredIds.includes(r.id))
+ } else {
+ // Create results from metadata matches (metadata-only query)
+ results = await this.createResultsFromIds(filteredIds)
+ }
}
```
-### Phase 4: Graph Traversal
+### Phase 4: Graph Traversal
```typescript
// Apply graph constraints using O(1) lookups
if (params.connected) {
- const connectedIds = await this.graphIndex.getConnectedIds(params.connected)
-
- if (results.length > 0) {
- // Filter existing results to only connected entities
- results = results.filter(r => connectedIds.includes(r.id))
- } else {
- // Create results from connected entities
- results = await this.createResultsFromIds(connectedIds)
- }
+ const connectedIds = await this.graphIndex.getConnectedIds(params.connected)
+
+ if (results.length > 0) {
+ // Filter existing results to only connected entities
+ results = results.filter(r => connectedIds.includes(r.id))
+ } else {
+ // Create results from connected entities
+ results = await this.createResultsFromIds(connectedIds)
+ }
}
```
@@ -229,12 +327,12 @@ if (params.connected) {
```typescript
// Combine scores from multiple intelligence sources
if (params.fusion && results.length > 0) {
- results = this.applyFusionScoring(results, params.fusion)
-
- // Example fusion strategies:
- // - Weighted: vectorScore × 0.6 + metadataScore × 0.2 + graphScore × 0.2
- // - Adaptive: adjust weights based on query characteristics
- // - Progressive: prioritize based on result confidence
+ results = this.applyFusionScoring(results, params.fusion)
+
+ // Example fusion strategies:
+ // - Weighted: vectorScore × 0.6 + metadataScore × 0.2 + graphScore × 0.2
+ // - Adaptive: adjust weights based on query characteristics
+ // - Progressive: prioritize based on result confidence
}
// Sort by final score and apply pagination
@@ -265,7 +363,7 @@ return results.slice(offset, offset + limit)
### 3. Field-Type Validation
```typescript
// Prevents invalid queries and suggests alternatives:
-// "people with publishDate > 2020"
+// "people with publishDate > 2020"
// → Warning: "Person entities rarely have publishDate field"
// → Suggestion: "Did you mean createdAt, updatedAt, or birthDate?"
// → Auto-correction: Use most likely alternative based on affinity data
@@ -275,36 +373,40 @@ return results.slice(offset, offset + limit)
### Query Performance by Type
-| Query Type | Index Used | Performance | Example |
-|------------|------------|-------------|---------|
-| **Semantic Search** | HNSW Vector | O(log n), ~1.8ms | `"AI research papers"` |
-| **Exact Metadata** | HashMap | O(1), ~0.8ms | `{status: "published"}` |
-| **Range Metadata** | Sorted Array | O(log n), ~0.6ms | `{year: {greaterThan: 2020}}` |
-| **Graph Traversal** | Adjacency Map | O(1), ~0.1ms | `{connected: {to: "mit"}}` |
-| **Type Detection** | Pre-embedded Types | O(t), ~0.3ms | `"documents"` → `NounType.Document` |
-| **Field Matching** | Field Embeddings | O(f), ~0.1ms | `"by"` → `"author"` |
-| **Combined Query** | All Indices | O(log n), ~1.8ms | NLP + filters + graph |
+| Query Type | Index Used | Complexity | Example |
+|------------|------------|------------|---------|
+| **Semantic Search** | HNSW Vector | O(log n) | `"AI research papers"` |
+| **Exact Metadata** | HashMap | O(1) | `{status: "published"}` |
+| **Range Metadata** | Sorted Array | O(log n) | `{year: {greaterThan: 2020}}` |
+| **Graph Traversal** | Adjacency Map | O(1) | `{connected: {to: "mit"}}` |
+| **Type Detection** | Pre-embedded Types | O(t) | `"documents"` → `NounType.Document` |
+| **Field Matching** | Field Embeddings | O(f) | `"by"` → `"author"` |
+| **Combined Query** | All Indices | O(log n) | NLP + filters + graph |
Where:
- n = number of entities in database
-- t = number of types (169 total: 42 noun + 127 verb)
+- t = number of types (169 total: 42 noun + 127 verb)
- f = number of fields for detected entity type (typically 5-15)
+Absolute latencies depend on hardware, embedding model, and storage backend; the columns above describe how each stage scales. The graph stage is the one path with a committed scale benchmark (see [Scalability](#scalability)).
+
### Scalability
-| Database Size | Vector Search | Metadata Filter | Graph Query | Combined |
-|---------------|---------------|-----------------|-------------|----------|
-| **1K entities** | 0.8ms | 0.3ms | 0.05ms | 1.1ms |
-| **10K entities** | 1.2ms | 0.5ms | 0.08ms | 1.5ms |
-| **100K entities** | 1.8ms | 0.8ms | 0.1ms | 2.1ms |
-| **1M entities** | 2.5ms | 1.2ms | 0.1ms | 2.8ms |
-| **10M entities** | 3.8ms | 1.8ms | 0.1ms | 4.2ms |
+The cost of each query stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion:
+
+| Query stage | Complexity | Scaling behavior |
+|-------------|------------|------------------|
+| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
+| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
+| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
+| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
+| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
**Key Performance Notes:**
- Graph queries stay O(1) regardless of scale
- Metadata ranges scale as O(log n), not O(n)
- Vector search degrades gracefully due to HNSW
-- Type-aware NLP adds minimal overhead (~0.4ms)
+- Type-aware NLP adds minimal overhead (single embedding pass, no full scan)
## Example Query Flows
@@ -315,27 +417,27 @@ Where:
// Phase 1: NLP Parsing
// - "papers" → NounType.Document (0.94 confidence)
// - "Stanford researchers" → entity search + NounType.Person
-// - "recent" → publishDate: {greaterThan: 2023}
+// - "recent" → publishDate: {greaterThan: 2023}
// - "high citations" → citations: {greaterThan: 100}
// - "connected to industry" → graph traversal via VerbType.AffiliatedWith
// Phase 2: Generated Query
{
- type: NounType.Document,
- where: {
- publishDate: { greaterThan: 2023 },
- citations: { greaterThan: 100 }
- },
- connected: {
- to: ["stanford-researchers"],
- via: VerbType.AffiliatedWith,
- depth: 2
- }
+ type: NounType.Document,
+ where: {
+ publishDate: { greaterThan: 2023 },
+ citations: { greaterThan: 100 }
+ },
+ connected: {
+ to: ["stanford-researchers"],
+ via: VerbType.AffiliatedWith,
+ depth: 2
+ }
}
// Phase 3: Execution Plan
// 1. Metadata filter: publishDate > 2023 (O(log n) via sorted index)
-// 2. Metadata filter: citations > 100 (O(log n) via sorted index)
+// 2. Metadata filter: citations > 100 (O(log n) via sorted index)
// 3. Graph traversal: connected to Stanford (O(1) per hop)
// 4. Intersection: entities matching all constraints
// 5. Sort by relevance score
@@ -345,22 +447,22 @@ Where:
```typescript
// Query: Direct structured query for maximum performance
await brain.find({
- type: NounType.Document, // O(1) type filter
- where: {
- status: "published", // O(1) exact match
- year: 2024, // O(1) exact match
- citations: { greaterThan: 50 } // O(log n) range query
- },
- connected: {
- from: "author-123", // O(1) graph lookup
- via: VerbType.Creates
- },
- limit: 10
+ type: NounType.Document, // O(1) type filter
+ where: {
+ status: "published", // O(1) exact match
+ year: 2024, // O(1) exact match
+ citations: { greaterThan: 50 } // O(log n) range query
+ },
+ connected: {
+ from: "author-123", // O(1) graph lookup
+ via: VerbType.Creates
+ },
+ limit: 10
})
-// Total performance: ~1.2ms for 100K entities
+// Executes as O(1) type filter + O(1)/O(log n) metadata + O(1) graph lookup — no full scan
```
-## Filter Syntax Reference (v5.8.0+)
+## Filter Syntax Reference
### Where Clause: Complete Operator Guide
@@ -371,25 +473,25 @@ Brainy provides a comprehensive set of operators for filtering entities by metad
**Exact Match** (shorthand):
```typescript
await brain.find({
- where: {
- status: 'active', // Shorthand for { eq: 'active' }
- year: 2024, // Exact match for numbers
- verified: true // Boolean matching
- }
+ where: {
+ status: 'active', // Shorthand for { eq: 'active' }
+ year: 2024, // Exact match for numbers
+ verified: true // Boolean matching
+ }
})
```
**Comparison Operators**:
```typescript
await brain.find({
- where: {
- age: { gt: 18 }, // Greater than
- score: { gte: 80 }, // Greater than or equal
- price: { lt: 100 }, // Less than
- stock: { lte: 10 }, // Less than or equal
- status: { eq: 'active' }, // Equals (explicit)
- role: { ne: 'guest' } // Not equals
- }
+ where: {
+ age: { gt: 18 }, // Greater than
+ score: { gte: 80 }, // Greater than or equal
+ price: { lt: 100 }, // Less than
+ stock: { lte: 10 }, // Less than or equal
+ status: { eq: 'active' }, // Equals (explicit)
+ role: { ne: 'guest' } // Not equals
+ }
})
```
@@ -400,11 +502,11 @@ await brain.find({
**Between** (inclusive):
```typescript
await brain.find({
- where: {
- publishDate: { between: [2020, 2024] }, // Year range
- price: { between: [10.00, 99.99] }, // Price range
- timestamp: { between: [startMs, endMs] } // Time range
- }
+ where: {
+ publishDate: { between: [2020, 2024] }, // Year range
+ price: { between: [10.00, 99.99] }, // Price range
+ timestamp: { between: [startMs, endMs] } // Time range
+ }
})
```
@@ -415,11 +517,11 @@ await brain.find({
**In/Not In**:
```typescript
await brain.find({
- where: {
- category: { in: ['tech', 'science', 'research'] },
- status: { notIn: ['draft', 'deleted'] },
- priority: { in: [1, 2, 3] }
- }
+ where: {
+ category: { in: ['tech', 'science', 'research'] },
+ status: { notIn: ['draft', 'deleted'] },
+ priority: { in: [1, 2, 3] }
+ }
})
```
@@ -430,11 +532,11 @@ await brain.find({
**Contains/Starts/Ends**:
```typescript
await brain.find({
- where: {
- title: { contains: 'machine learning' }, // Substring search
- email: { startsWith: 'admin@' }, // Prefix match
- filename: { endsWith: '.pdf' } // Suffix match
- }
+ where: {
+ title: { contains: 'machine learning' }, // Substring search
+ email: { startsWith: 'admin@' }, // Prefix match
+ filename: { endsWith: '.pdf' } // Suffix match
+ }
})
```
@@ -454,11 +556,11 @@ query: 'artificial intelligence'
**Exists/Missing**:
```typescript
await brain.find({
- where: {
- email: { exists: true }, // Has email field
- deletedAt: { exists: false }, // No deletedAt field (not deleted)
- profileImage: { exists: true } // Has profile image
- }
+ where: {
+ email: { exists: true }, // Has email field
+ deletedAt: { exists: false }, // No deletedAt field (not deleted)
+ profileImage: { exists: true } // Has profile image
+ }
})
```
@@ -474,11 +576,11 @@ All conditions at the same level are implicitly AND:
```typescript
await brain.find({
- where: {
- status: 'published', // AND
- year: { gte: 2020 }, // AND
- citations: { gte: 50 } // AND
- }
+ where: {
+ status: 'published', // AND
+ year: { gte: 2020 }, // AND
+ citations: { gte: 50 } // AND
+ }
})
// Returns: entities matching ALL three conditions
```
@@ -486,13 +588,13 @@ await brain.find({
**Explicit AND with `allOf`**:
```typescript
await brain.find({
- where: {
- allOf: [
- { status: 'published' },
- { year: { gte: 2020 } },
- { citations: { gte: 50 } }
- ]
- }
+ where: {
+ allOf: [
+ { status: 'published' },
+ { year: { gte: 2020 } },
+ { citations: { gte: 50 } }
+ ]
+ }
})
```
@@ -504,13 +606,13 @@ Match ANY condition:
```typescript
await brain.find({
- where: {
- anyOf: [
- { status: 'urgent' },
- { priority: { gte: 8 } },
- { assignee: 'admin' }
- ]
- }
+ where: {
+ anyOf: [
+ { status: 'urgent' },
+ { priority: { gte: 8 } },
+ { assignee: 'admin' }
+ ]
+ }
})
// Returns: entities matching ANY condition
```
@@ -518,13 +620,13 @@ await brain.find({
**Combined AND + OR**:
```typescript
await brain.find({
- where: {
- status: 'active', // Must be active
- anyOf: [ // AND (urgent OR high priority)
- { tags: { contains: 'urgent' } },
- { priority: { gte: 8 } }
- ]
- }
+ where: {
+ status: 'active', // Must be active
+ anyOf: [ // AND (urgent OR high priority)
+ { tags: { contains: 'urgent' } },
+ { priority: { gte: 8 } }
+ ]
+ }
})
```
@@ -536,17 +638,17 @@ Complex boolean expressions:
```typescript
await brain.find({
- where: {
- allOf: [
- { status: 'published' },
- {
- anyOf: [
- { featured: true },
- { citations: { gte: 100 } }
- ]
- }
- ]
- }
+ where: {
+ allOf: [
+ { status: 'published' },
+ {
+ anyOf: [
+ { featured: true },
+ { citations: { gte: 100 } }
+ ]
+ }
+ ]
+ }
})
// Returns: published AND (featured OR highly cited)
```
@@ -591,8 +693,8 @@ Filter entities by NounType:
```typescript
await brain.find({
- type: NounType.Document,
- where: { year: { gte: 2020 } }
+ type: NounType.Document,
+ where: { year: { gte: 2020 } }
})
```
@@ -600,8 +702,8 @@ await brain.find({
```typescript
await brain.find({
- type: [NounType.Person, NounType.Organization],
- where: { verified: true }
+ type: [NounType.Person, NounType.Organization],
+ where: { verified: true }
})
```
@@ -642,11 +744,11 @@ Traverse relationships using the GraphIndex:
```typescript
await brain.find({
- connected: {
- to: 'entity-id-123', // Connected to this entity
- via: VerbType.WorksFor, // Through this relationship type
- direction: 'out' // Direction: 'in', 'out', or 'both'
- }
+ connected: {
+ to: 'entity-id-123', // Connected to this entity
+ via: VerbType.WorksFor, // Through this relationship type
+ direction: 'out' // Direction: 'in', 'out', or 'both'
+ }
})
```
@@ -656,11 +758,11 @@ await brain.find({
```typescript
await brain.find({
- connected: {
- to: 'research-institution',
- via: VerbType.AffiliatedWith,
- depth: 2 // Up to 2 hops away
- }
+ connected: {
+ to: 'research-institution',
+ via: VerbType.AffiliatedWith,
+ depth: 2 // Up to 2 hops away
+ }
})
```
@@ -670,35 +772,35 @@ await brain.find({
```typescript
await brain.find({
- type: NounType.Person,
- where: {
- verified: true,
- reputation: { gte: 100 }
- },
- connected: {
- to: 'stanford-ai-lab',
- via: VerbType.WorksAt,
- direction: 'out'
- },
- limit: 20
+ type: NounType.Person,
+ where: {
+ verified: true,
+ reputation: { gte: 100 }
+ },
+ connected: {
+ to: 'stanford-ai-lab',
+ via: VerbType.WorksAt,
+ direction: 'out'
+ },
+ limit: 20
})
// Returns: Verified people with high reputation who work at Stanford AI Lab
```
-#### Pagination with Graph Queries (v5.8.0+)
+#### Pagination with Graph Queries
```typescript
// Page through high-degree nodes efficiently
const neighbors = await brain.graphIndex.getNeighbors('hub-entity-id', {
- direction: 'out',
- limit: 50,
- offset: 0
+ direction: 'out',
+ limit: 50,
+ offset: 0
})
// Get verb IDs with pagination
const verbIds = await brain.graphIndex.getVerbIdsBySource('source-id', {
- limit: 100,
- offset: 0
+ limit: 100,
+ offset: 0
})
```
@@ -706,36 +808,36 @@ const verbIds = await brain.graphIndex.getVerbIdsBySource('source-id', {
**Note**: See `src/graph/graphAdjacencyIndex.ts` for low-level graph operations.
-### Sorting Results (v4.5.4+)
+### Sorting Results
Sort query results by any field, including timestamps:
```typescript
// Sort by timestamp (descending - newest first)
await brain.find({
- type: NounType.Document,
- orderBy: 'createdAt',
- order: 'desc',
- limit: 10
+ type: NounType.Document,
+ orderBy: 'createdAt',
+ order: 'desc',
+ limit: 10
})
// Sort by custom field (ascending)
await brain.find({
- where: { status: { eq: 'published' } },
- orderBy: 'priority',
- order: 'asc'
+ where: { status: { eq: 'published' } },
+ orderBy: 'priority',
+ order: 'asc'
})
// Sort with filtering and pagination
await brain.find({
- where: {
- publishDate: { gte: startDate },
- citations: { gte: 50 }
- },
- orderBy: 'citations',
- order: 'desc',
- limit: 20,
- offset: 0
+ where: {
+ publishDate: { gte: startDate },
+ citations: { gte: 50 }
+ },
+ orderBy: 'citations',
+ order: 'desc',
+ limit: 20,
+ offset: 0
})
```
@@ -750,17 +852,17 @@ await brain.find({
```typescript
// Range query + sorting
await brain.find({
- where: {
- createdAt: { gte: Date.now() - 86400000 } // Last 24 hours
- },
- orderBy: 'createdAt',
- order: 'desc' // Newest first
+ where: {
+ createdAt: { gte: Date.now() - 86400000 } // Last 24 hours
+ },
+ orderBy: 'createdAt',
+ order: 'desc' // Newest first
})
// Works with updatedAt, accessed, modified
await brain.find({
- orderBy: 'updatedAt',
- order: 'desc'
+ orderBy: 'updatedAt',
+ order: 'desc'
})
```
@@ -768,22 +870,22 @@ await brain.find({
```typescript
// Sort search results by custom field instead of relevance
await brain.find({
- query: "machine learning",
- where: { publishDate: { gte: 2023 } },
- orderBy: 'citations', // Sort by citations, not relevance
- order: 'desc'
+ query: "machine learning",
+ where: { publishDate: { gte: 2023 } },
+ orderBy: 'citations', // Sort by citations, not relevance
+ order: 'desc'
})
// Paginated sorted results
async function getDocumentsByDate(page: number, pageSize: number = 20) {
- return await brain.find({
- type: NounType.Document,
- where: { status: { eq: 'published' } },
- orderBy: 'publishDate',
- order: 'desc',
- limit: pageSize,
- offset: page * pageSize
- })
+ return await brain.find({
+ type: NounType.Document,
+ where: { status: { eq: 'published' } },
+ orderBy: 'publishDate',
+ order: 'desc',
+ limit: pageSize,
+ offset: page * pageSize
+ })
}
```
@@ -794,30 +896,30 @@ async function getDocumentsByDate(page: number, pageSize: number = 20) {
**Offset-based pagination**:
```typescript
async function getPaginatedResults(page: number, pageSize: number = 20) {
- return await brain.find({
- type: NounType.Document,
- where: { status: 'published' },
- orderBy: 'createdAt',
- order: 'desc',
- limit: pageSize,
- offset: page * pageSize
- })
+ return await brain.find({
+ type: NounType.Document,
+ where: { status: 'published' },
+ orderBy: 'createdAt',
+ order: 'desc',
+ limit: pageSize,
+ offset: page * pageSize
+ })
}
// Usage
-const page1 = await getPaginatedResults(0) // First 20
-const page2 = await getPaginatedResults(1) // Next 20
+const page1 = await getPaginatedResults(0) // First 20
+const page2 = await getPaginatedResults(1) // Next 20
```
-**Graph pagination** (v5.8.0+):
+**Graph pagination**:
```typescript
// Paginate through high-degree node relationships
async function getNeighborPage(entityId: string, page: number, pageSize: number = 50) {
- return await brain.graphIndex.getNeighbors(entityId, {
- direction: 'out',
- limit: pageSize,
- offset: page * pageSize
- })
+ return await brain.graphIndex.getNeighbors(entityId, {
+ direction: 'out',
+ limit: pageSize,
+ offset: page * pageSize
+ })
}
```
@@ -830,23 +932,23 @@ async function getNeighborPage(entityId: string, page: number, pageSize: number
// Last 24 hours
const oneDayAgo = Date.now() - (24 * 60 * 60 * 1000)
await brain.find({
- where: {
- createdAt: { gte: oneDayAgo }
- },
- orderBy: 'createdAt',
- order: 'desc'
+ where: {
+ createdAt: { gte: oneDayAgo }
+ },
+ orderBy: 'createdAt',
+ order: 'desc'
})
// Last 7 days with additional filters
const oneWeekAgo = Date.now() - (7 * 24 * 60 * 60 * 1000)
await brain.find({
- type: NounType.Document,
- where: {
- createdAt: { gte: oneWeekAgo },
- status: 'published'
- },
- orderBy: 'createdAt',
- order: 'desc'
+ type: NounType.Document,
+ where: {
+ createdAt: { gte: oneWeekAgo },
+ status: 'published'
+ },
+ orderBy: 'createdAt',
+ order: 'desc'
})
```
@@ -854,21 +956,21 @@ await brain.find({
```typescript
// Specific year
await brain.find({
- where: {
- publishDate: { between: [
- new Date('2023-01-01').getTime(),
- new Date('2023-12-31').getTime()
- ]}
- }
+ where: {
+ publishDate: { between: [
+ new Date('2023-01-01').getTime(),
+ new Date('2023-12-31').getTime()
+ ]}
+ }
})
// Quarter
const Q1_2024_start = new Date('2024-01-01').getTime()
const Q1_2024_end = new Date('2024-03-31').getTime()
await brain.find({
- where: {
- createdAt: { between: [Q1_2024_start, Q1_2024_end] }
- }
+ where: {
+ createdAt: { between: [Q1_2024_start, Q1_2024_end] }
+ }
})
```
@@ -878,32 +980,32 @@ await brain.find({
```typescript
// Find: AI research papers from verified authors at top institutions
const results = await brain.find({
- // Vector search (semantic)
- query: 'artificial intelligence machine learning',
+ // Vector search (semantic)
+ query: 'artificial intelligence machine learning',
- // Metadata filters
- type: NounType.Document,
- where: {
- publishDate: { gte: 2020 },
- citations: { gte: 50 },
- peerReviewed: true
- },
+ // Metadata filters
+ type: NounType.Document,
+ where: {
+ publishDate: { gte: 2020 },
+ citations: { gte: 50 },
+ peerReviewed: true
+ },
- // Graph traversal
- connected: {
- to: topInstitutionIds, // Array of institution entity IDs
- via: VerbType.AffiliatedWith,
- depth: 2 // Authors affiliated with institutions (2 hops)
- },
+ // Graph traversal
+ connected: {
+ to: topInstitutionIds, // Array of institution entity IDs
+ via: VerbType.AffiliatedWith,
+ depth: 2 // Authors affiliated with institutions (2 hops)
+ },
- // Results
- limit: 50,
- orderBy: 'citations',
- order: 'desc'
+ // Results
+ limit: 50,
+ orderBy: 'citations',
+ order: 'desc'
})
```
-**Performance**: O(log n) vector search + O(log n) metadata filters + O(1) graph traversal = ~2-3ms total.
+**Performance**: O(log n) vector search + O(log n) metadata filters + O(1) graph traversal — bounded by the vector stage.
### Excluding Soft-Deleted Entities
@@ -911,19 +1013,19 @@ const results = await brain.find({
```typescript
// Standard query excludes deleted
await brain.find({
- where: {
- deletedAt: { exists: false } // Not soft-deleted
- }
+ where: {
+ deletedAt: { exists: false } // Not soft-deleted
+ }
})
// Or use compound filter
await brain.find({
- where: {
- allOf: [
- { status: 'active' },
- { deletedAt: { exists: false } }
- ]
- }
+ where: {
+ allOf: [
+ { status: 'active' },
+ { deletedAt: { exists: false } }
+ ]
+ }
})
```
@@ -935,21 +1037,21 @@ await brain.find({
```typescript
// Find documents similar to a specific document
await brain.find({
- near: {
- id: 'doc-123',
- threshold: 0.8 // Minimum 80% similarity
- },
- type: NounType.Document,
- limit: 10
+ near: {
+ id: 'doc-123',
+ threshold: 0.8 // Minimum 80% similarity
+ },
+ type: NounType.Document,
+ limit: 10
})
// With metadata constraints
await brain.find({
- near: { id: 'paper-456', threshold: 0.75 },
- where: {
- publishDate: { gte: 2020 },
- language: 'en'
- }
+ near: { id: 'paper-456', threshold: 0.75 },
+ where: {
+ publishDate: { gte: 2020 },
+ language: 'en'
+ }
})
```
@@ -961,8 +1063,8 @@ await brain.find({
```typescript
// Get total count (metadata-only query is fastest)
const results = await brain.find({
- where: { status: 'published' },
- limit: 1 // We only need the count
+ where: { status: 'published' },
+ limit: 1 // We only need the count
})
// Note: Current API returns results, not counts
// For production, consider caching counts or using metadata indices directly
@@ -973,10 +1075,10 @@ const results = await brain.find({
// Find all entities, then group by type in application
const allEntities = await brain.find({ limit: 10000 })
const byType = allEntities.reduce((acc, entity) => {
- const type = entity.noun || 'unknown'
- if (!acc[type]) acc[type] = []
- acc[type].push(entity)
- return acc
+ const type = entity.noun || 'unknown'
+ if (!acc[type]) acc[type] = []
+ acc[type].push(entity)
+ return acc
}, {})
```
@@ -985,14 +1087,14 @@ const byType = allEntities.reduce((acc, entity) => {
**Any of multiple values**:
```typescript
await brain.find({
- where: {
- anyOf: [
- { priority: 'urgent' },
- { priority: 'high' },
- { assignee: 'admin' },
- { dueDate: { lte: Date.now() } }
- ]
- }
+ where: {
+ anyOf: [
+ { priority: 'urgent' },
+ { priority: 'high' },
+ { assignee: 'admin' },
+ { dueDate: { lte: Date.now() } }
+ ]
+ }
})
// Returns: urgent OR high priority OR assigned to admin OR overdue
```
@@ -1001,23 +1103,23 @@ await brain.find({
```typescript
// Find: (Premium users OR trial users with activity) AND not banned
await brain.find({
- type: NounType.Person,
- where: {
- allOf: [
- {
- anyOf: [
- { subscription: 'premium' },
- {
- allOf: [
- { subscription: 'trial' },
- { lastActive: { gte: Date.now() - 86400000 } } // 24h
- ]
- }
- ]
- },
- { banned: { ne: true } }
- ]
- }
+ type: NounType.Person,
+ where: {
+ allOf: [
+ {
+ anyOf: [
+ { subscription: 'premium' },
+ {
+ allOf: [
+ { subscription: 'trial' },
+ { lastActive: { gte: Date.now() - 86400000 } } // 24h
+ ]
+ }
+ ]
+ },
+ { banned: { ne: true } }
+ ]
+ }
})
```
@@ -1029,8 +1131,8 @@ await brain.find({
```typescript
// List all entities of a type
const all = await brain.find({
- type: NounType.Document,
- limit: 10
+ type: NounType.Document,
+ limit: 10
})
console.log(`Found ${all.length} documents`)
```
@@ -1038,11 +1140,11 @@ console.log(`Found ${all.length} documents`)
**Check 2: Test filters individually**
```typescript
// Remove filters one by one to find the culprit
-await brain.find({ where: { status: 'published' } }) // Works?
-await brain.find({ where: { year: 2024 } }) // Works?
+await brain.find({ where: { status: 'published' } }) // Works?
+await brain.find({ where: { year: 2024 } }) // Works?
await brain.find({ where: {
- status: 'published',
- year: 2024 // Combined - works?
+ status: 'published',
+ year: 2024 // Combined - works?
}})
```
@@ -1050,7 +1152,7 @@ await brain.find({ where: {
```typescript
// Get a sample entity to see actual field names
const sample = await brain.find({ type: NounType.Document, limit: 1 })
-console.log(Object.keys(sample[0].data)) // Actual fields
+console.log(Object.keys(sample[0].data)) // Actual fields
```
**Common issues**:
@@ -1064,9 +1166,9 @@ console.log(Object.keys(sample[0].data)) // Actual fields
```typescript
// Use explain mode (if available)
const results = await brain.find({
- query: 'machine learning',
- where: { title: { contains: 'AI' } }, // ⚠️ O(n) substring search
- explain: true
+ query: 'machine learning',
+ where: { title: { contains: 'AI' } }, // ⚠️ O(n) substring search
+ explain: true
})
```
@@ -1089,15 +1191,15 @@ where: { status: 'published' }
```typescript
// ❌ Suboptimal: Slow filter first
where: {
- description: { contains: 'AI' }, // O(n) - runs first
- year: 2024 // O(1) - runs second
+ description: { contains: 'AI' }, // O(n) - runs first
+ year: 2024 // O(1) - runs second
}
// ✅ Optimal: Fast filter first (automatic optimization)
where: {
- year: 2024, // O(1) - narrow results
- status: 'published' // O(1) - further narrow
- // Only then apply O(n) operations if needed
+ year: 2024, // O(1) - narrow results
+ status: 'published' // O(1) - further narrow
+ // Only then apply O(n) operations if needed
}
```
@@ -1119,10 +1221,10 @@ import { NounType } from '@soulcraft/brainy'
await brain.find({ type: NounType.Document })
// ❌ Error: Operator not recognized
-where: { age: { greaterThan: 18 } } // Old API
+where: { age: { greaterThan: 18 } } // Old API
// ✅ Correct: Use canonical operators
-where: { age: { gt: 18 } } // v5.0.0+
+where: { age: { gt: 18 } }
```
### Graph Traversal Issues
@@ -1130,26 +1232,26 @@ where: { age: { gt: 18 } } // v5.0.0+
**No connected entities found**:
```typescript
// Verify relationship exists
-const relations = await brain.getRelations({
- from: 'entity-a',
- to: 'entity-b'
+const relations = await brain.related({
+ from: 'entity-a',
+ to: 'entity-b'
})
console.log('Relationships:', relations)
// Check direction
await brain.find({
- connected: {
- to: 'entity-id',
- direction: 'in' // Try 'out' or 'both'
- }
+ connected: {
+ to: 'entity-id',
+ direction: 'in' // Try 'out' or 'both'
+ }
})
// Verify verb type
await brain.find({
- connected: {
- to: 'entity-id',
- via: VerbType.WorksFor // Correct VerbType?
- }
+ connected: {
+ to: 'entity-id',
+ via: VerbType.WorksFor // Correct VerbType?
+ }
})
```
@@ -1170,12 +1272,12 @@ await brain.update(entity.id, { data: entity.data })
```typescript
// ❌ Too strict: May return nothing
await brain.find({
- near: { id: 'doc-123', threshold: 0.95 }
+ near: { id: 'doc-123', threshold: 0.95 }
})
// ✅ Reasonable: 0.7-0.85 is typical
await brain.find({
- near: { id: 'doc-123', threshold: 0.75 }
+ near: { id: 'doc-123', threshold: 0.75 }
})
```
@@ -1189,8 +1291,8 @@ console.log('Entity type:', entity.noun)
// Verify type filter is working
await brain.find({
- type: NounType.Document,
- where: { id: 'unexpected-id' } // Should not return if wrong type
+ type: NounType.Document,
+ where: { id: 'unexpected-id' } // Should not return if wrong type
})
```
@@ -1205,7 +1307,7 @@ console.log(`Results: ${results.length}, Unique: ${uniqueIds.size}`)
// Brainy should never return duplicates - report if found
```
-## VFS (Virtual File System) Visibility (v4.7.0+)
+## VFS (Virtual File System) Visibility
### Default Behavior
@@ -1226,8 +1328,8 @@ If you need to exclude VFS entities from specific queries, use the `excludeVFS`
```typescript
// Exclude VFS files from results
await brain.find({
- query: 'machine learning',
- excludeVFS: true // Only return non-file entities
+ query: 'machine learning',
+ excludeVFS: true // Only return non-file entities
})
```
@@ -1236,14 +1338,14 @@ await brain.find({
```typescript
// Explicit filtering (same as excludeVFS: true)
await brain.find({
- query: 'machine learning',
- where: { vfsType: { exists: false } }
+ query: 'machine learning',
+ where: { vfsType: { exists: false } }
})
// Or only search VFS files
await brain.find({
- query: 'setup instructions',
- where: { vfsType: 'file' } // Only files
+ query: 'setup instructions',
+ where: { vfsType: 'file' } // Only files
})
```
@@ -1254,26 +1356,62 @@ await brain.find({
- No performance penalty - same speed as any metadata filter
- Works seamlessly with vector + metadata + graph queries
+## 5. Aggregate Queries
+
+The `find()` method also supports aggregate queries via the `aggregate` parameter. When set, `find()` bypasses all vector/metadata/graph search paths and returns pre-computed aggregate results from the incremental aggregation engine.
+
+```typescript
+// Define the aggregate (once — survives restarts)
+brain.defineAggregate({
+ name: 'monthly_spending',
+ source: { type: NounType.Event, where: { domain: 'financial' } },
+ groupBy: ['category', { field: 'date', window: 'month' }],
+ metrics: {
+ total: { op: 'sum', field: 'amount' },
+ count: { op: 'count' }
+ }
+})
+
+// Query it through find()
+const results = await brain.find({
+ aggregate: 'monthly_spending',
+ where: { category: 'food' }, // Filters aggregate groups (not raw entities)
+ orderBy: 'total',
+ order: 'desc',
+ limit: 12
+})
+```
+
+**Key characteristics:**
+- **O(1) read performance** — results come from running totals, no query-time computation
+- **Updated incrementally** — every `add()`, `update()`, `delete()` updates matching aggregates
+- **Same Result[] format** — aggregate results are returned as `NounType.Measurement` entities
+- **Combinable with `where`/`orderBy`/`limit`/`offset`** — standard find() parameters apply to group filtering
+
+See the **[API Reference → Aggregation Engine](./api/README.md#aggregation-engine)** for the full API.
+
+---
+
### Migration from v4.6.x
-**BREAKING CHANGE** (v4.7.0): The `includeVFS` parameter has been removed:
+**BREAKING CHANGE**: The `includeVFS` parameter has been removed:
```typescript
// ❌ Old (v4.6.x and earlier)
await brain.find({
- query: 'docs',
- includeVFS: true // No longer needed!
+ query: 'docs',
+ includeVFS: true // No longer needed!
})
-// ✅ New (v4.7.0+)
+// ✅ New
await brain.find({
- query: 'docs' // VFS included by default
+ query: 'docs' // VFS included by default
})
// ✅ To exclude VFS (if needed)
await brain.find({
- query: 'concepts',
- excludeVFS: true
+ query: 'concepts',
+ excludeVFS: true
})
```
diff --git a/docs/METADATA_CONTRACT_IMPLEMENTATION.md b/docs/METADATA_CONTRACT_IMPLEMENTATION.md
deleted file mode 100644
index 0020990f..00000000
--- a/docs/METADATA_CONTRACT_IMPLEMENTATION.md
+++ /dev/null
@@ -1,177 +0,0 @@
-# Metadata Contract Implementation Plan
-
-## New Required Interface
-
-```typescript
-export interface BrainyAugmentation {
- // Identity
- name: string
- timing: 'before' | 'after' | 'around' | 'replace'
- operations: string[]
- priority: number
-
- // REQUIRED metadata contract
- metadata: 'none' | 'readonly' | MetadataAccess
-
- // Methods
- initialize(context: AugmentationContext): Promise
- execute(operation: string, params: any, next: () => Promise): Promise
- shouldExecute?(operation: string, params: any): boolean
- shutdown?(): Promise
-}
-
-interface MetadataAccess {
- reads?: string[] | '*' // Fields to read, or '*' for all
- writes?: string[] | '*' // Fields to write, or '*' for all
- namespace?: string // Optional: custom namespace like '_myAug'
-}
-```
-
-## Augmentation Analysis & Classification
-
-### Category 1: No Metadata Access ('none')
-These augmentations don't read or write metadata at all:
-
-1. **CacheAugmentation** - Only caches search results
-2. **RequestDeduplicatorAugmentation** - Only deduplicates requests
-3. **ConnectionPoolAugmentation** - Only manages storage connections
-4. **StorageAugmentation** - Base storage layer, metadata handled by Brainy
-
-### Category 2: Read-Only Access ('readonly')
-These augmentations read metadata but never modify it:
-
-6. **IndexAugmentation** - Reads metadata to build indexes
-7. **MonitoringAugmentation** - Reads metadata for monitoring
-8. **MetricsAugmentation** - Reads metadata for metrics collection
-9. **BatchProcessingAugmentation** - Reads metadata to check for external IDs
-10. **EntityRegistryAugmentation** - Reads metadata to register entities
-11. **AutoRegisterEntitiesAugmentation** - Reads metadata for auto-registration
-12. **ConduitAugmentation** - Reads metadata to pass through operations
-
-### Category 3: Metadata Writers (needs specific access)
-These augmentations modify metadata and need specific field declarations:
-
-13. **SynapseAugmentation** - Writes to '_synapse' field
- ```typescript
- metadata: {
- reads: '*',
- writes: ['_synapse', '_synapseTimestamp'],
- namespace: '_synapse' // Uses its own namespace
- }
- ```
-
-14. **IntelligentVerbScoringAugmentation** - Adds scoring to verbs
- ```typescript
- metadata: {
- reads: ['type', 'verb', 'source', 'target'],
- writes: ['weight', 'confidence', 'intelligentScoring']
- }
- ```
-
-15. **ServerSearchAugmentation** - Might add server metadata
- ```typescript
- metadata: {
- reads: '*',
- writes: ['_server', '_syncedAt']
- }
- ```
-
-16. **NeuralImportAugmentation** - Enriches imported data
- ```typescript
- metadata: {
- reads: '*',
- writes: ['nounType', 'verbType', '_importedAt', '_enriched']
- }
- ```
-
-### Category 4: API/Server (needs analysis)
-17. **ApiServerAugmentation** - Likely read-only for serving data
-18. **StorageAugmentations** (plural) - Collection of storage implementations
-19. **ConduitAugmentations** (plural) - Collection of conduit types
-
-## Implementation Steps
-
-### Phase 1: Update Base Interface
-1. Update `BrainyAugmentation` interface to require `metadata` field
-2. Update `BaseAugmentation` class to have abstract `metadata` property
-3. Add runtime enforcement in augmentation executor
-
-### Phase 2: Update Each Augmentation
-For each augmentation, add the appropriate metadata declaration:
-
-#### Example Updates:
-
-**CacheAugmentation:**
-```typescript
-export class CacheAugmentation extends BaseAugmentation {
- readonly name = 'cache'
- readonly metadata = 'none' as const // ✅ No metadata access
- // ... rest unchanged
-}
-```
-
-```typescript
- readonly metadata = 'readonly' as const // ✅ Only reads for logging
- // ... rest unchanged
-}
-```
-
-**SynapseAugmentation:**
-```typescript
-export abstract class SynapseAugmentation extends BaseAugmentation {
- readonly name = 'synapse'
- readonly metadata = {
- reads: '*',
- writes: ['_synapse', '_synapseTimestamp'],
- namespace: '_synapse'
- } as const
- // ... rest unchanged
-}
-```
-
-### Phase 3: Runtime Enforcement
-Add a metadata access enforcer that:
-1. Wraps metadata objects based on declared access
-2. Throws errors if augmentation violates its contract
-3. Logs warnings in development mode
-
-```typescript
-class MetadataEnforcer {
- enforce(augmentation: BrainyAugmentation, metadata: any): any {
- if (augmentation.metadata === 'none') {
- return null // No access at all
- }
-
- if (augmentation.metadata === 'readonly') {
- return Object.freeze(deepClone(metadata)) // Read-only copy
- }
-
- // For specific access, create proxy that validates
- return new Proxy(metadata, {
- set(target, prop, value) {
- const access = augmentation.metadata as MetadataAccess
- if (!access.writes?.includes(String(prop)) && access.writes !== '*') {
- throw new Error(`Augmentation '${augmentation.name}' cannot write to field '${String(prop)}'`)
- }
- target[prop] = value
- return true
- }
- })
- }
-}
-```
-
-## Benefits
-1. **Type Safety** - TypeScript enforces metadata declaration
-2. **Runtime Safety** - Violations caught immediately
-3. **Documentation** - Contract shows exactly what each augmentation does
-4. **Brain-cloud Ready** - Registry can validate augmentations
-5. **Developer Friendly** - Most use simple 'none' or 'readonly'
-
-## Migration Checklist
-- [ ] Update BrainyAugmentation interface
-- [ ] Update BaseAugmentation class
-- [ ] Add MetadataEnforcer
-- [ ] Update all 19 augmentations with metadata declarations
-- [ ] Add tests for metadata enforcement
-- [ ] Update documentation
\ No newline at end of file
diff --git a/docs/MIGRATION-V3-TO-V4.md b/docs/MIGRATION-V3-TO-V4.md
index 9b9dec3f..29c409ac 100644
--- a/docs/MIGRATION-V3-TO-V4.md
+++ b/docs/MIGRATION-V3-TO-V4.md
@@ -48,7 +48,7 @@ await storage.setLifecyclePolicy({
```typescript
// v3: Delete one at a time (slow, expensive)
for (const id of idsToDelete) {
- await brain.delete(id) // 1000 API calls for 1000 entities
+ await brain.remove(id) // 1000 API calls for 1000 entities
}
// v4.0.0: Batch delete (fast, cheap)
@@ -426,16 +426,6 @@ npm install @soulcraft/brainy@^3.50.0
7. Verify data integrity thoroughly
8. Enable lifecycle policies gradually
-### Scenario 4: Multi-Node Distributed System
-
-**Recommended approach:**
-1. Perform blue-green deployment:
- - Keep v3 nodes running (blue)
- - Deploy v4 nodes (green)
- - Migrate data once
- - Switch traffic to v4 nodes
- - Decommission v3 nodes
-
## Cost Savings After Migration
### Enable All v4.0.0 Features
diff --git a/docs/NEURAL_API_PATTERNS.md b/docs/NEURAL_API_PATTERNS.md
deleted file mode 100644
index 4edd3505..00000000
--- a/docs/NEURAL_API_PATTERNS.md
+++ /dev/null
@@ -1,736 +0,0 @@
-# 🧠 Neural API Patterns: AI-Powered Intelligence
-
-> Learn the correct patterns for Brainy's Neural API. Avoid performance pitfalls and use AI features effectively.
-
-## 🚨 Critical: Access Neural APIs Correctly
-
-### ❌ **WRONG - Outdated Access Patterns**
-
-```typescript
-// DON'T DO THIS - Outdated documentation patterns
-import { Brainy } from '@soulcraft/brainy' // ❌ Wrong import
-const brain = new Brainy() // ❌ Old class name
-
-// These may not work as expected:
-const neural = brain.neural // ❌ May be undefined
-```
-
-### ✅ **CORRECT - Modern Neural Access**
-
-```typescript
-// ✅ Use modern Brainy class
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// ✅ Neural API is available after initialization
-const clusters = await brain.neural.clusters()
-const similarity = await brain.neural.similar('item1', 'item2')
-```
-
-## 🔍 Similarity Analysis Patterns
-
-### ❌ **WRONG - Inefficient Similarity Checks**
-
-```typescript
-// DON'T DO THIS - N² comparisons
-const items = await brain.find({ limit: 1000 })
-const similarities = []
-
-for (const item1 of items) {
- for (const item2 of items) {
- if (item1.id !== item2.id) {
- const sim = await brain.neural.similar(item1.id, item2.id) // ❌ Millions of calls
- similarities.push({ from: item1.id, to: item2.id, score: sim })
- }
- }
-}
-```
-
-### ✅ **CORRECT - Efficient Similarity Patterns**
-
-```typescript
-// ✅ Pattern 1: Find neighbors (much more efficient)
-const item = await brain.get('target-item-id')
-const neighbors = await brain.neural.neighbors(item.id, {
- limit: 10, // Top 10 most similar
- threshold: 0.7, // Minimum similarity
- includeScores: true // Include similarity scores
-})
-
-console.log(`Found ${neighbors.length} similar items`)
-
-// ✅ Pattern 2: Batch similarity for specific pairs
-const itemPairs = [
- ['item1', 'item2'],
- ['item1', 'item3'],
- ['item2', 'item3']
-]
-
-const similarities = await Promise.all(
- itemPairs.map(async ([a, b]) => ({
- from: a,
- to: b,
- score: await brain.neural.similar(a, b)
- }))
-)
-
-// ✅ Pattern 3: Text-to-text similarity (no need for IDs)
-const textSimilarity = await brain.neural.similar(
- "Machine learning is fascinating",
- "AI and deep learning are interesting",
- { detailed: true } // Get explanation of similarity
-)
-
-console.log(`Similarity: ${textSimilarity.score}`)
-console.log(`Explanation: ${textSimilarity.explanation}`)
-
-// ✅ Pattern 4: Vector-level similarity for optimization
-const vector1 = await brain.embed("First concept")
-const vector2 = await brain.embed("Second concept")
-const vectorSimilarity = await brain.neural.similar(vector1, vector2)
-```
-
-## 🎯 Clustering Patterns
-
-### ❌ **WRONG - Uncontrolled Clustering**
-
-```typescript
-// DON'T DO THIS - Clustering everything without limits
-const everything = await brain.find({ limit: 100000 }) // ❌ Too much data
-const clusters = await brain.neural.clusters() // ❌ May crash or timeout
-```
-
-### ✅ **CORRECT - Smart Clustering Patterns**
-
-```typescript
-// ✅ Pattern 1: Controlled clustering with limits
-const recentItems = await brain.find({
- where: {
- createdAt: { $gte: Date.now() - 30 * 24 * 60 * 60 * 1000 } // Last 30 days
- },
- limit: 1000 // Reasonable limit
-})
-
-const clusters = await brain.neural.clusters(
- recentItems.map(item => item.id),
- {
- algorithm: 'kmeans', // Reliable algorithm
- maxClusters: 10, // Reasonable number
- threshold: 0.75, // High similarity required
- iterations: 50 // Convergence limit
- }
-)
-
-// ✅ Pattern 2: Domain-specific clustering
-const techDocs = await brain.find({
- where: { category: 'technology', type: 'document' },
- limit: 500
-})
-
-const techClusters = await brain.neural.clusterByDomain(
- 'category', // Group by this field
- {
- items: techDocs.map(doc => doc.id),
- minClusterSize: 3, // Minimum items per cluster
- maxClusters: 8
- }
-)
-
-// ✅ Pattern 3: Temporal clustering for time-series data
-const timebasedClusters = await brain.neural.clusterByTime(
- 'createdAt', // Time field
- 'week', // Time window (hour, day, week, month)
- {
- items: recentItems.map(item => item.id),
- overlap: 0.2, // 20% overlap between windows
- minPerWindow: 5 // Minimum items per time window
- }
-)
-
-// ✅ Pattern 4: Streaming clustering for large datasets
-async function clusterLargeDataset() {
- const clusterStream = brain.neural.clusterStream({
- batchSize: 100, // Process 100 items at a time
- updateInterval: 1000, // Update clusters every 1000 items
- maxMemory: 512 * 1024 * 1024 // 512MB memory limit
- })
-
- const allClusters = []
- for await (const batch of clusterStream) {
- console.log(`Processed ${batch.processed} items, found ${batch.clusters.length} clusters`)
- allClusters.push(...batch.clusters)
- }
-
- return allClusters
-}
-```
-
-## 🔍 Neighbor Discovery Patterns
-
-### ❌ **WRONG - Manual Similarity Searches**
-
-```typescript
-// DON'T DO THIS - Reinventing neighbor search
-async function findSimilarManually(targetId: string) {
- const allItems = await brain.find({ limit: 10000 }) // ❌ Load everything
- const similarities = []
-
- for (const item of allItems) {
- if (item.id !== targetId) {
- const score = await brain.neural.similar(targetId, item.id) // ❌ Slow
- if (score > 0.7) {
- similarities.push({ id: item.id, score })
- }
- }
- }
-
- return similarities.sort((a, b) => b.score - a.score).slice(0, 10) // ❌ Inefficient
-}
-```
-
-### ✅ **CORRECT - Optimized Neighbor Patterns**
-
-```typescript
-// ✅ Pattern 1: Basic neighbor search
-const neighbors = await brain.neural.neighbors('target-item-id', {
- limit: 20, // Top 20 neighbors
- threshold: 0.6, // Minimum similarity
- includeMetadata: true, // Include item metadata
- includeDistances: true // Include exact similarity scores
-})
-
-// ✅ Pattern 2: Filtered neighbor search
-const filteredNeighbors = await brain.neural.neighbors('article-id', {
- limit: 10,
- filter: {
- type: 'document', // Only find similar documents
- status: 'published', // Only published content
- language: 'en' // Only English content
- },
- excludeIds: ['self-id', 'duplicate-id'] // Exclude specific items
-})
-
-// ✅ Pattern 3: Multi-level neighbor discovery
-async function discoverNeighborNetwork(rootId: string, maxDepth = 2) {
- const network = new Map()
- const visited = new Set()
- const queue = [{ id: rootId, depth: 0 }]
-
- while (queue.length > 0) {
- const { id, depth } = queue.shift()!
-
- if (visited.has(id) || depth >= maxDepth) continue
- visited.add(id)
-
- const neighbors = await brain.neural.neighbors(id, {
- limit: 5,
- threshold: 0.8
- })
-
- network.set(id, neighbors)
-
- // Add neighbors to queue for next depth level
- if (depth < maxDepth - 1) {
- neighbors.forEach(neighbor => {
- queue.push({ id: neighbor.id, depth: depth + 1 })
- })
- }
- }
-
- return network
-}
-
-// ✅ Pattern 4: Recommendation engine
-async function getRecommendations(userId: string) {
- // Get user's liked items
- const userItems = await brain.find({
- connected: { to: userId, via: 'liked-by' }
- })
-
- // Find neighbors for each liked item
- const allNeighbors = await Promise.all(
- userItems.map(item =>
- brain.neural.neighbors(item.id, {
- limit: 10,
- threshold: 0.7,
- excludeConnected: { to: userId, via: 'liked-by' } // Exclude already liked
- })
- )
- )
-
- // Aggregate and rank recommendations
- const recommendations = new Map()
- allNeighbors.flat().forEach(neighbor => {
- const current = recommendations.get(neighbor.id) || { score: 0, count: 0 }
- current.score += neighbor.score
- current.count += 1
- recommendations.set(neighbor.id, current)
- })
-
- // Return top recommendations by average score
- return Array.from(recommendations.entries())
- .map(([id, stats]) => ({
- id,
- avgScore: stats.score / stats.count,
- mentions: stats.count
- }))
- .sort((a, b) => b.avgScore - a.avgScore)
- .slice(0, 10)
-}
-```
-
-## 🏗️ Hierarchy & Structure Patterns
-
-### ❌ **WRONG - Manual Hierarchy Building**
-
-```typescript
-// DON'T DO THIS - Building hierarchies manually
-async function buildHierarchyManually(rootId: string) {
- const root = await brain.get(rootId)
- const allItems = await brain.find({ limit: 1000 }) // ❌ Load everything
-
- // Manual tree building with nested loops
- const hierarchy = { root, children: [] }
- // ... complex manual logic
-}
-```
-
-### ✅ **CORRECT - Semantic Hierarchy Patterns**
-
-```typescript
-// ✅ Pattern 1: Automatic semantic hierarchy
-const hierarchy = await brain.neural.hierarchy('root-concept-id', {
- maxDepth: 4, // Maximum tree depth
- minSimilarity: 0.6, // Minimum similarity for inclusion
- branchingFactor: 5, // Maximum children per node
- algorithm: 'semantic' // Use semantic clustering
-})
-
-// ✅ Pattern 2: Domain-specific hierarchy
-const techHierarchy = await brain.neural.hierarchy('technology-id', {
- filter: { category: 'technology' },
- weights: {
- semantic: 0.7, // 70% based on content similarity
- metadata: 0.3 // 30% based on metadata similarity
- },
- includeMetrics: true // Include hierarchy quality metrics
-})
-
-// ✅ Pattern 3: Multi-root hierarchy for complex domains
-async function buildMultiRootHierarchy(rootIds: string[]) {
- const hierarchies = await Promise.all(
- rootIds.map(rootId =>
- brain.neural.hierarchy(rootId, {
- maxDepth: 3,
- crossReference: true // Allow cross-hierarchy connections
- })
- )
- )
-
- // Merge hierarchies and find connections
- const merged = {
- roots: hierarchies,
- connections: await findCrossHierarchyConnections(hierarchies)
- }
-
- return merged
-}
-
-async function findCrossHierarchyConnections(hierarchies: any[]) {
- const connections = []
-
- for (let i = 0; i < hierarchies.length; i++) {
- for (let j = i + 1; j < hierarchies.length; j++) {
- const leafNodes1 = extractLeafNodes(hierarchies[i])
- const leafNodes2 = extractLeafNodes(hierarchies[j])
-
- // Find connections between leaf nodes of different hierarchies
- for (const leaf1 of leafNodes1) {
- const neighbors = await brain.neural.neighbors(leaf1.id, {
- limit: 5,
- threshold: 0.8,
- filter: { id: { $in: leafNodes2.map(l => l.id) } }
- })
-
- connections.push(...neighbors.map(n => ({
- from: leaf1.id,
- to: n.id,
- hierarchyPair: [i, j],
- similarity: n.score
- })))
- }
- }
- }
-
- return connections
-}
-```
-
-## 🚨 Outlier Detection Patterns
-
-### ❌ **WRONG - Manual Outlier Detection**
-
-```typescript
-// DON'T DO THIS - Manual statistical outlier detection
-async function findOutliersManually() {
- const items = await brain.find({ limit: 1000 })
- const similarities = []
-
- // Calculate average similarity for each item (expensive)
- for (const item of items) {
- let totalSim = 0
- let count = 0
- for (const other of items) {
- if (item.id !== other.id) {
- totalSim += await brain.neural.similar(item.id, other.id) // ❌ N² operations
- count++
- }
- }
- similarities.push({ id: item.id, avgSimilarity: totalSim / count })
- }
-
- // Manual outlier calculation
- const threshold = calculateManualThreshold(similarities) // ❌ Complex statistics
- return similarities.filter(s => s.avgSimilarity < threshold)
-}
-```
-
-### ✅ **CORRECT - AI-Powered Outlier Detection**
-
-```typescript
-// ✅ Pattern 1: Automatic outlier detection
-const outliers = await brain.neural.outliers({
- threshold: 0.3, // Items with < 30% avg similarity to others
- method: 'isolation-forest', // AI-based outlier detection
- contamination: 0.1, // Expect ~10% outliers
- includeReasons: true // Explain why each item is an outlier
-})
-
-console.log(`Found ${outliers.length} outliers`)
-outliers.forEach(outlier => {
- console.log(`Outlier: ${outlier.id}, Score: ${outlier.score}`)
- console.log(`Reason: ${outlier.reason}`)
-})
-
-// ✅ Pattern 2: Domain-specific outlier detection
-const techOutliers = await brain.neural.outliers({
- filter: { category: 'technology' },
- features: ['content', 'metadata.tags', 'metadata.complexity'],
- method: 'local-outlier-factor',
- neighbors: 20 // Consider 20 nearest neighbors
-})
-
-// ✅ Pattern 3: Temporal outlier detection
-const recentOutliers = await brain.neural.outliers({
- timeWindow: '7days', // Look at last 7 days
- baseline: '30days', // Compare to 30-day baseline
- method: 'statistical', // Use statistical methods
- autoThreshold: true // Automatically determine threshold
-})
-
-// ✅ Pattern 4: Streaming outlier detection
-async function detectOutliersInStream() {
- const outlierStream = brain.neural.outlierStream({
- batchSize: 50,
- updateInterval: 100, // Check every 100 new items
- adaptiveThreshold: true // Threshold adapts as data changes
- })
-
- for await (const batch of outlierStream) {
- console.log(`Batch ${batch.batchNumber}: ${batch.outliers.length} outliers detected`)
-
- // Process outliers immediately
- for (const outlier of batch.outliers) {
- await handleOutlier(outlier)
- }
- }
-}
-
-async function handleOutlier(outlier: any) {
- // Flag for manual review
- await brain.update(outlier.id, {
- metadata: {
- flagged: true,
- outlierScore: outlier.score,
- outlierReason: outlier.reason,
- flaggedAt: Date.now()
- }
- })
-}
-```
-
-## 📊 Visualization Patterns
-
-### ❌ **WRONG - Manual Visualization Data Preparation**
-
-```typescript
-// DON'T DO THIS - Manual coordinate calculation
-async function prepareVisualizationManually() {
- const items = await brain.find({ limit: 500 })
- const coordinates = []
-
- // Manual dimensionality reduction (complex math)
- for (const item of items) {
- const vector = await brain.embed(item.data)
- // Complex PCA/t-SNE calculations manually
- const x = complexMathFunction(vector) // ❌ Error-prone
- const y = anotherComplexFunction(vector)
- coordinates.push({ id: item.id, x, y })
- }
-
- return coordinates
-}
-```
-
-### ✅ **CORRECT - AI-Powered Visualization**
-
-```typescript
-// ✅ Pattern 1: Automatic 2D visualization
-const visualization = await brain.neural.visualize({
- dimensions: 2, // 2D plot
- algorithm: 'umap', // UMAP for better clustering preservation
- maxItems: 1000, // Performance limit
- includeMetadata: true, // Include item metadata in output
- colorBy: 'cluster' // Color points by cluster membership
-})
-
-// Result format:
-// {
-// points: [{ id, x, y, cluster, metadata }, ...],
-// clusters: [{ id, centroid: [x, y], members: [...] }, ...],
-// stats: { stress, kruskalStress, trustworthiness }
-// }
-
-// ✅ Pattern 2: 3D visualization for complex data
-const viz3D = await brain.neural.visualize({
- dimensions: 3,
- algorithm: 'tsne',
- perplexity: 30, // t-SNE parameter
- learningRate: 200, // t-SNE learning rate
- iterations: 1000 // Number of optimization steps
-})
-
-// ✅ Pattern 3: Interactive visualization with filtering
-const interactiveViz = await brain.neural.visualize({
- filter: {
- type: 'document',
- createdAt: { $gte: Date.now() - 7 * 24 * 60 * 60 * 1000 }
- },
- groupBy: 'category', // Group points by metadata field
- showLabels: true, // Include text labels
- labelField: 'title', // Field to use for labels
- includeEdges: true, // Show connections between similar items
- edgeThreshold: 0.8 // Only show high-similarity connections
-})
-
-// ✅ Pattern 4: Real-time visualization updates
-class LiveVisualization {
- private visualization: any = null
- private updateInterval: NodeJS.Timeout | null = null
-
- async start() {
- // Initial visualization
- this.visualization = await brain.neural.visualize({
- dimensions: 2,
- algorithm: 'umap',
- maxItems: 500,
- includeMetadata: true
- })
-
- // Update every 30 seconds
- this.updateInterval = setInterval(async () => {
- await this.update()
- }, 30000)
- }
-
- async update() {
- // Get recent items
- const recentItems = await brain.find({
- where: {
- createdAt: { $gte: Date.now() - 30000 } // Last 30 seconds
- },
- limit: 50
- })
-
- if (recentItems.length > 0) {
- // Incrementally update visualization
- const updates = await brain.neural.updateVisualization(
- this.visualization.id,
- {
- newItems: recentItems.map(item => item.id),
- algorithm: 'incremental' // Faster incremental updates
- }
- )
-
- this.visualization = { ...this.visualization, ...updates }
- this.onUpdate(updates)
- }
- }
-
- onUpdate(updates: any) {
- // Emit updates to frontend
- console.log(`Visualization updated: ${updates.newPoints.length} new points`)
- }
-
- stop() {
- if (this.updateInterval) {
- clearInterval(this.updateInterval)
- }
- }
-}
-```
-
-## 🚀 Performance Optimization Patterns
-
-### ✅ **High-Performance Neural Operations**
-
-```typescript
-// ✅ Pattern 1: Batch processing for similarity
-async function batchSimilarityCalculation(itemPairs: Array<[string, string]>) {
- const batchSize = 100
- const results = []
-
- for (let i = 0; i < itemPairs.length; i += batchSize) {
- const batch = itemPairs.slice(i, i + batchSize)
-
- const batchResults = await Promise.all(
- batch.map(async ([a, b]) => ({
- from: a,
- to: b,
- similarity: await brain.neural.similar(a, b)
- }))
- )
-
- results.push(...batchResults)
-
- // Progress reporting
- console.log(`Processed ${Math.min(i + batchSize, itemPairs.length)}/${itemPairs.length} pairs`)
- }
-
- return results
-}
-
-// ✅ Pattern 2: Caching expensive operations
-class NeuralCache {
- private clusterCache = new Map()
- private similarityCache = new Map()
- private readonly TTL = 5 * 60 * 1000 // 5 minutes
-
- async getClusters(options: any) {
- const key = JSON.stringify(options)
- const cached = this.clusterCache.get(key)
-
- if (cached && Date.now() - cached.timestamp < this.TTL) {
- return cached.data
- }
-
- const clusters = await brain.neural.clusters(undefined, options)
- this.clusterCache.set(key, {
- data: clusters,
- timestamp: Date.now()
- })
-
- return clusters
- }
-
- async getSimilarity(id1: string, id2: string) {
- // Create consistent cache key regardless of order
- const key = [id1, id2].sort().join('-')
- const cached = this.similarityCache.get(key)
-
- if (cached && Date.now() - cached.timestamp < this.TTL) {
- return cached.data
- }
-
- const similarity = await brain.neural.similar(id1, id2)
- this.similarityCache.set(key, {
- data: similarity,
- timestamp: Date.now()
- })
-
- return similarity
- }
-}
-
-// ✅ Pattern 3: Memory-efficient streaming
-async function processLargeDatasetEfficiently() {
- const stream = brain.neural.clusterStream({
- batchSize: 50, // Small batches for memory efficiency
- maxMemoryMB: 256, // Memory limit
- diskCache: true, // Use disk for temporary storage
- compression: true // Compress cached data
- })
-
- const results = []
- let totalProcessed = 0
-
- for await (const batch of stream) {
- // Process batch immediately, don't accumulate in memory
- const processedBatch = await processBatch(batch)
-
- // Save to disk or send to another service
- await saveBatchToDisk(processedBatch)
-
- totalProcessed += batch.items.length
- console.log(`Processed ${totalProcessed} items`)
-
- // Clear memory
- batch.items = null
- }
-
- return { totalProcessed }
-}
-
-// ✅ Pattern 4: Parallel processing with worker threads
-async function parallelNeuralProcessing(items: string[]) {
- const numWorkers = require('os').cpus().length
- const batchSize = Math.ceil(items.length / numWorkers)
-
- const workers = []
- for (let i = 0; i < numWorkers; i++) {
- const batch = items.slice(i * batchSize, (i + 1) * batchSize)
- if (batch.length > 0) {
- workers.push(processWorkerBatch(batch))
- }
- }
-
- const results = await Promise.all(workers)
- return results.flat()
-}
-
-async function processWorkerBatch(batch: string[]) {
- // This would run in a worker thread in real implementation
- return Promise.all(
- batch.map(async itemId => {
- const neighbors = await brain.neural.neighbors(itemId, { limit: 5 })
- return { itemId, neighbors }
- })
- )
-}
-```
-
-## 🎯 Summary: Neural API Best Practices
-
-| ❌ **Avoid These Patterns** | ✅ **Use These Instead** |
-|---------------------------|------------------------|
-| Manual similarity loops | `brain.neural.neighbors()` |
-| Uncontrolled clustering | Limit items and set maxClusters |
-| Manual outlier detection | `brain.neural.outliers()` |
-| Manual visualization prep | `brain.neural.visualize()` |
-| Loading entire datasets | Streaming and batch processing |
-| No caching | Cache expensive operations |
-| Blocking operations | Parallel and async patterns |
-
----
-
-**🎉 Following these patterns gives you:**
-- 🚀 **Optimized performance** with intelligent algorithms
-- 🧠 **AI-powered insights** instead of manual statistics
-- 📊 **Rich visualizations** for data exploration
-- 🎯 **Accurate clustering** with semantic understanding
-- 🚨 **Smart outlier detection** for quality control
-- ⚡ **Scalable processing** for large datasets
-
-**Next:** [Augmentation Patterns →](./AUGMENTATION_PATTERNS.md) | [Core API Patterns →](./CORE_API_PATTERNS.md)
\ No newline at end of file
diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md
index 15344b1e..248a2c70 100644
--- a/docs/PERFORMANCE.md
+++ b/docs/PERFORMANCE.md
@@ -2,21 +2,23 @@
## Performance Characteristics
-Brainy achieves industry-leading performance through carefully optimized data structures and algorithms. All performance claims are verified through actual benchmarks on production code.
+Brainy achieves high performance through carefully optimized data structures and algorithms. The tables below describe each component by its **algorithmic complexity** — the durable, defensible guarantee. The example latencies are figures from a single 100-item run on one machine (see [Benchmarks](#benchmarks)); they are illustrative, not a committed benchmark, and vary with hardware, embedding model, and storage backend. The one component with a committed scale assertion is the graph adjacency index (`tests/performance/graph-scale-performance.test.ts:238`).
### Core Performance Summary
-| Component | Operation | Time Complexity | Measured Performance | Data Structure |
+| Component | Operation | Time Complexity | Example latency (100-item run)\* | Data Structure |
|-----------|-----------|-----------------|---------------------|----------------|
| **Metadata Index** | Exact match | **O(1)** | 0.8ms | `Map>` |
| **Metadata Index** | Range query | **O(log n) + O(k)** | 0.6ms | Sorted array + binary search |
| **Graph Index** | Get neighbors | **O(1)** | 0.09ms | `Map>` |
-| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | HNSW hierarchical graph |
+| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | Hierarchical graph |
| **NLP Parser** | Query parsing | **O(m)** | 8.9ms | 220 pre-computed patterns |
| **Type-Field Affinity** | Field matching | **O(f)** | 0.1ms | Type-specific field cache |
| **Type Detection** | Noun/Verb matching | **O(t)** | 0.3ms | Pre-embedded type vectors |
| **Triple Intelligence** | Combined query | **O(1) to O(log n)** | 1.8ms | Parallel execution |
+\* Illustrative single-run figures at 100 items on one machine — not a committed benchmark. Only the graph index carries an asserted scale bound (measured <1 ms per neighbor lookup up to 1M relationships, `tests/performance/graph-scale-performance.test.ts:238`).
+
Where:
- `n` = number of items in index
- `k` = number of results returned
@@ -24,27 +26,34 @@ Where:
- `f` = number of fields for entity type
- `t` = number of types (42 nouns, 127 verbs)
-### v5.11.1: brain.get() Metadata-Only Optimization
+### brain.get() Metadata-Only Optimization
-✨ **Massive Performance Improvement**: `brain.get()` is now **76-81% faster** by default!
+`brain.get()` returns **metadata only by default**, skipping the 384-dimensional
+embedding — the bulk of an entity's payload. Callers that need the vector opt in
+with `{ includeVectors: true }`.
-| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup | Use Case |
-|-----------|------------------|-----------------|---------|----------|
-| **brain.get() (metadata-only)** | 43ms, 6KB | **10ms, 300 bytes** | **76-81%** | VFS, existence checks, metadata |
-| **brain.get({ includeVectors: true })** | 43ms, 6KB | 43ms, 6KB | 0% | Similarity calculations |
-| **VFS readFile()** | 53ms | **~13ms** | **75%** | File operations |
-| **VFS readdir(100 files)** | 5.3s | **~1.3s** | **75%** | Directory listings |
+| Operation | Default (metadata-only) | With `includeVectors: true` | Use Case |
+|-----------|-------------------------|-----------------------------|----------|
+| **brain.get()** | Skips vector load | Loads full vector | VFS, existence checks, metadata |
+| **VFS readFile() / readdir()** | Inherits metadata-only path | n/a | File operations, directory listings |
-**Key Innovation**: Lazy vector loading - only load 384-dimensional embeddings when explicitly needed.
+**Key Innovation**: Lazy vector loading — only load the 384-dimensional embedding when explicitly needed.
+
+The integration test `tests/integration/metadata-only-comprehensive.test.ts:306`
+asserts metadata-only `get()` is faster than the full-entity `get()`
+(`metadataTime < fullTime`). The *magnitude* of the speedup is
+environment-dependent (the percentage assertion in
+`tests/integration/vfs-performance-v5.11.1.test.ts` is intentionally skipped on
+CI for that reason), so no fixed percentage is quoted here.
**Why this matters**:
-- **94% of brain.get() calls** don't need vectors (VFS, admin tools, import utilities, data APIs)
-- **Vector data is 95% of entity size** (6KB vectors vs 300 bytes metadata)
-- **Zero code changes** for most applications - automatic speedup!
+- Most `brain.get()` calls don't need vectors (VFS, admin tools, import utilities, data APIs)
+- The embedding dominates an entity's serialized size, so skipping it is the largest win
+- **Zero code changes** for most applications — automatic by default
**When to use what**:
```typescript
-// DEFAULT: Metadata-only (76-81% faster) - use for:
+// DEFAULT: Metadata-only (skips the vector load) - use for:
const entity = await brain.get(id)
// - VFS operations (readFile, stat, readdir)
// - Existence checks: if (await brain.get(id)) ...
@@ -55,7 +64,7 @@ const entity = await brain.get(id)
const entity = await brain.get(id, { includeVectors: true })
// - Computing similarity on THIS entity
// - Manual vector operations
-// - HNSW graph traversal
+// - Vector index graph traversal
```
## Architecture Deep Dive
@@ -143,14 +152,14 @@ class GraphAdjacencyIndex {
**Key Innovation:** Pure Map/Set operations - no database queries, no loops, just direct memory access.
-### 4. HNSW Vector Search - O(log n)
+### 4. Vector Index - O(log n)
-Hierarchical Navigable Small World graphs provide logarithmic approximate nearest neighbor search:
+The default vector index (`JsHnswVectorIndex`) provides logarithmic approximate nearest neighbor search through a hierarchical graph:
```typescript
-class HNSWIndex {
+class JsHnswVectorIndex {
private nouns: Map = new Map()
-
+
interface HNSWNoun {
id: string
vector: number[]
@@ -238,7 +247,7 @@ const results = await Promise.all(searchPromises)
|-----------|--------------|---------|
| Metadata Index | ~40 bytes/entry | `(key_size + 8) × unique_values + 8 × total_items` |
| Graph Index | ~24 bytes/edge | `16 × edges + 8 × nodes` |
-| HNSW | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
+| Vector Index | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
| Pattern Library | 394KB fixed | Pre-computed, shared across instances |
| Type Embeddings | ~60KB fixed | 70 types × 384 dimensions × 4 bytes, cached |
| Field Embeddings | ~5KB dynamic | Actual fields × 384 dimensions × 4 bytes |
@@ -252,34 +261,40 @@ const results = await Promise.all(searchPromises)
## Benchmarks
-### Real-world Performance Test (100 items)
+### Illustrative Single Run (100 items, one machine)
+
+Example output from a single 100-item run — illustrative only, not a committed
+benchmark; absolute numbers vary by hardware. The values feed the
+[Core Performance Summary](#core-performance-summary) example-latency column.
```
-📊 Metadata exact match: 0.818ms (50 items matched)
-📊 Metadata range query: 0.631ms (40 items in range)
-🔗 Graph neighbor lookup: 0.092ms (2 connections)
-🎯 Vector k-NN search: 1.773ms (10 nearest neighbors)
-🧠 NLP query parsing: 8.906ms (full natural language)
-⚡ Triple Intelligence: 1.830ms (combined query)
+Metadata exact match: 0.818ms (50 items matched)
+Metadata range query: 0.631ms (40 items in range)
+Graph neighbor lookup: 0.092ms (2 connections)
+Vector k-NN search: 1.773ms (10 nearest neighbors)
+NLP query parsing: 8.906ms (full natural language)
+Triple Intelligence: 1.830ms (combined query)
```
### Scaling Characteristics
-| Items | Metadata O(1) | Range O(log n) | Graph O(1) | Vector O(log n) |
-|-------|---------------|----------------|------------|-----------------|
-| 100 | 0.8ms | 0.6ms | 0.09ms | 1.8ms |
-| 1,000 | 0.8ms | 0.9ms | 0.09ms | 2.5ms |
-| 10,000 | 0.8ms | 1.2ms | 0.09ms | 3.2ms |
-| 100,000 | 0.8ms | 1.5ms | 0.09ms | 4.1ms |
-| 1,000,000 | 0.8ms | 1.8ms | 0.09ms | 5.0ms |
+Each stage scales by its algorithmic complexity, not a fixed millisecond figure
+— absolute latency depends on hardware, embedding model, and storage backend.
+Only the graph adjacency index carries a committed scale assertion:
-*Note: O(1) operations maintain constant time regardless of scale*
+| Query stage | Complexity | Scaling behavior |
+|-------------|------------|------------------|
+| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
+| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
+| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
+| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
+| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
## Comparison with Other Systems
| System | Metadata Filter | Graph Traversal | Vector Search | Natural Language |
|--------|-----------------|-----------------|---------------|------------------|
-| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) HNSW | 220 patterns |
+| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) vector index | 220 patterns |
| Neo4j | O(log n) B-tree | O(k) traversal | Not native | Not native |
| Elasticsearch | O(log n) inverted | Not native | O(n) brute force* | Basic tokenization |
| PostgreSQL | O(log n) B-tree | O(k) recursive | O(n) brute force* | Full-text only |
@@ -305,10 +320,10 @@ const results = await Promise.all(searchPromises)
- ✅ **No Network Calls**: Everything runs locally, including embeddings
- ✅ **Thread-Safe**: Immutable data structures where possible
- ✅ **Memory Bounded**: Configurable cache sizes and automatic cleanup
-- ✅ **Horizontally Scalable**: Stateless operations support clustering
+- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
- ✅ **Zero Stubs**: Every line of code is production-ready
-## Lazy Loading Performance (v5.7.7+)
+## Lazy Loading Performance
Brainy supports two initialization modes for optimal performance across different use cases:
@@ -324,7 +339,7 @@ await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
- First query: Instant (indexes already loaded)
- Use case: Traditional applications, long-running servers
-### Mode 2: Lazy Loading (v5.7.7+)
+### Mode 2: Lazy Loading
```javascript
const brain = new Brainy({ disableAutoRebuild: true })
@@ -375,110 +390,47 @@ const brain = new Brainy({ disableAutoRebuild: true })
await brain.init() // Instant (0-10ms)
```
-### Automatic Self-Tuning (Current & Planned)
+### Automatic Self-Tuning
-**✅ Currently Implemented:**
- **Metadata Index**: Auto-builds sorted indices for range queries on first use
- **Graph Index**: Auto-flushes every 30 seconds
-- **Default Tuning**: Research-based defaults (M=16, ef=200)
+- **Default Tuning**: Research-based vector index defaults
- **Lazy Loading**: Indices built only when needed
- **Cache Management**: LRU caches with TTL
-**🚧 Planned Enhancements:**
-- **Dynamic Storage Selection**: Auto-switch between memory/disk based on size
-- **Adaptive Index Parameters**: Adjust M and ef based on query patterns
-- **Smart Cache Sizing**: Scale caches based on available memory
-- **Predictive Optimization**: Learn from usage patterns
-
### Intelligent Defaults
-All defaults are research-based and production-tested:
-- **HNSW M=16**: Optimal balance of recall/speed for most datasets
-- **efConstruction=200**: High quality graph construction
-- **Cache TTL=5min**: Balances freshness with performance
-- **Flush Interval=30s**: Non-blocking background persistence
+- **Vector recall** = `'balanced'` (M=16, ef=200): right for most datasets
+- **Cache TTL** = 5 min: balances freshness and performance
+- **Flush interval** = 30 s: non-blocking background persistence
-### Progressive Enhancement
+### Vector Index Tuning Knobs
-Brainy learns and improves over time:
-1. **Query Pattern Learning**: Frequently used patterns get cached
-2. **Index Optimization**: Auto-rebuilds indices when fragmented
-3. **Memory Management**: Coordinates caches across all components
-4. **Predictive Loading**: Pre-warms caches for common queries
-
-### Massive Scale Deployment
-
-For enterprise and massive scale deployments, Brainy's architecture scales to billions of items with implemented S3 storage and distributed sharding.
-
-**Currently Implemented:**
-- Memory storage (production-ready)
-- Disk storage (production-ready)
-- S3-compatible storage (AWS S3, Cloudflare R2, Google Cloud Storage, MinIO, Backblaze B2)
-- Distributed sharding with ConsistentHashRing
-- Single-node deployment (scales to ~1M items)
-- Multi-node deployment with sharding (scales to billions)
-
-**Available Today:**
+Brainy 8.0 exposes two knobs on `config.vector`:
```javascript
-// S3-compatible storage for unlimited scale - WORKS NOW
-const brain = new Brainy({
- storage: {
- type: 's3',
- bucketName: 'my-brainy-data',
- region: 'us-east-1',
- credentials: {
- accessKeyId: 'YOUR_ACCESS_KEY',
- secretAccessKey: 'YOUR_SECRET_KEY'
- }
- // Works with: AWS S3, MinIO, Cloudflare R2, Backblaze B2, Google Cloud Storage
- }
-})
-
-// Cloudflare R2 storage - WORKS NOW
-const brain = new Brainy({
- storage: {
- type: 'r2',
- bucketName: 'my-brainy-data',
- accountId: 'YOUR_ACCOUNT_ID',
- accessKeyId: 'YOUR_R2_ACCESS_KEY',
- secretAccessKey: 'YOUR_R2_SECRET_KEY'
- }
-})
-
-// Google Cloud Storage - WORKS NOW
-const brain = new Brainy({
- storage: {
- type: 'gcs',
- bucketName: 'my-brainy-data',
- region: 'us-central1',
- credentials: {
- accessKeyId: 'YOUR_ACCESS_KEY',
- secretAccessKey: 'YOUR_SECRET_KEY'
- }
+const brain = new Brainy({
+ vector: {
+ recall: 'fast', // 'fast' | 'balanced' | 'accurate'
+ persistMode: 'deferred' // 'immediate' | 'deferred'
}
})
```
+The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cor`) can replace it with a higher-performing implementation; the public knobs stay the same.
+
### Scale Scenarios
-| Scale | Items | Storage Strategy | Performance | Status |
-|-------|-------|-----------------|-------------|--------|
-| **Small** | <10K | Memory (automatic) | Sub-millisecond | ✅ Implemented |
-| **Medium** | 10K-1M | Disk with memory cache | 1-5ms | ✅ Implemented |
-| **Large** | 1M-100M | S3 with memory cache | 2-10ms | ✅ Implemented |
-| **Massive** | 100M-10B | S3 + distributed sharding | 5-20ms | ✅ Implemented |
-| **Planetary** | 10B+ | Multi-region S3 + Edge cache | 10-50ms | 🚧 Roadmap |
+| Scale | Items | Storage Strategy | Performance |
+|-------|-------|------------------|-------------|
+| **Small** | <10K | Memory | Sub-millisecond |
+| **Medium** | 10K-1M | Filesystem | 1-5ms |
+| **Large** | 1M-10M | Filesystem + tuned cache | 2-10ms |
+| **Massive** | 10M+ | Filesystem + native vector provider + service-layer sharding | 5-20ms |
-### S3-Compatible Storage Benefits
+For >10M entities, run multiple Brainy processes behind your own routing layer — Brainy 8.0 doesn't ship cluster coordination.
-- **Unlimited Scale**: No practical limit on dataset size
-- **Cost Effective**: $0.023/GB/month for standard storage
-- **Durability**: 99.999999999% (11 9's) durability
-- **Global**: Multi-region replication available
-- **Compatible**: Works with any S3-compatible API (MinIO, R2, B2)
-
-### Distributed Architecture (Implemented)
+### Architecture
```
┌─────────────────────────────────────────┐
@@ -490,103 +442,51 @@ const brain = new Brainy({
│ Brainy Core │
│ (Triple Intelligence Engine) │
├─────────────────────────────────────────┤
-│ Memory │ Shard │ Metadata │
-│ Cache │ Manager │ Index │
+│ Memory │ Vector │ Metadata │
+│ Cache │ Index │ Index │
└─────────────┬───────────────────────────┘
│
┌─────────────▼───────────────────────────┐
│ Storage Layer │
├──────────┬──────────┬──────────────────┤
-│ HNSW │ Graph │ Objects │
-│ Vectors │ Edges │ (S3/R2/GCS) │
+│ Vectors │ Graph │ Files │
+│ (sharded)│ Edges │ (filesystem) │
└──────────┴──────────┴──────────────────┘
```
-**Distributed Sharding (Implemented):**
-- ConsistentHashRing with 150 virtual nodes
-- 64 shards by default
-- Replication factor of 3
-- Automatic rebalancing on node addition/removal
-
-### Auto-Sharding for Horizontal Scale (Implemented)
-
-Brainy includes a complete sharding implementation with ConsistentHashRing:
-
-```javascript
-import { ShardManager } from '@soulcraft/brainy/distributed'
-
-// Create shard manager with custom configuration
-const shardManager = new ShardManager({
- shardCount: 64, // Default: 64 shards
- replicationFactor: 3, // Default: 3 replicas
- virtualNodes: 150, // Default: 150 virtual nodes
- autoRebalance: true // Default: true
-})
-
-// Add nodes to the cluster
-shardManager.addNode('node-1')
-shardManager.addNode('node-2')
-shardManager.addNode('node-3')
-
-// Sharding automatically:
-// - Uses consistent hashing for even distribution
-// - Maintains replicas for fault tolerance
-// - Rebalances on node changes
-// - Provides O(1) shard lookups
-```
+For off-site replication, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`).
### Performance at Scale
-Even at massive scale, Brainy maintains excellent performance:
-
-- **Metadata queries**: Still O(1) with distributed hash tables
-- **Graph traversal**: O(1) with edge locality optimization
-- **Vector search**: O(log n) with hierarchical sharding
-- **Write throughput**: 100K+ writes/second with S3 batching
+- **Metadata queries**: O(1) HashMap
+- **Graph traversal**: O(1) adjacency lookup
+- **Vector search**: O(log n)
+- **Write throughput**: 50K+ writes/second per process (filesystem, batched)
- **Read throughput**: 1M+ reads/second with caching
-### Zero-Config with Autoscaling (Implemented)
+### Zero-Config with Autoscaling
-Brainy includes extensive autoscaling capabilities:
-
-**✅ Implemented Autoscaling:**
- **AutoConfiguration System**: Detects environment and adjusts settings
- **Learning from Performance**: `learnFromPerformance()` adapts based on metrics
- **Auto-flush**: Graph index (30s), Metadata index (configurable)
-- **Auto-optimize**: Enabled by default in Graph and HNSW indices
-- **Auto-rebalance**: Shards automatically rebalance on node changes
+- **Auto-optimize**: Enabled by default in graph and vector indices
- **Zero-config presets**: Production, development, minimal modes
- **Adaptive memory**: Scales caches based on available memory
-- **Environment detection**: Browser vs Node.js vs Serverless
-
-**🚧 Roadmap Autoscaling:**
-- Dynamic HNSW parameter adjustment (M, ef)
-- Predictive query pattern caching
-- Multi-region auto-replication
-- Automatic cross-node data migration
## Implementation Status
-### ✅ Fully Implemented and Production-Ready
+### Fully Implemented and Production-Ready
- **O(1) metadata lookups** via HashMaps (exact match)
- **O(log n) range queries** via sorted arrays with lazy building
- **O(1) graph traversal** via adjacency maps
-- **O(log n) vector search** via HNSW
+- **O(log n) vector search** via the default JS index, swappable for a native provider
- **220 NLP patterns** with pre-computed embeddings
-- **S3-compatible storage** (AWS S3, R2, GCS, MinIO, B2)
-- **Distributed sharding** with ConsistentHashRing
+- **Filesystem and memory storage** adapters
- **Auto-configuration system** with environment detection
- **Zero-config operation** with intelligent defaults
- **Auto-flush and auto-optimize** in indices
-- **Sub-2ms response times** for complex queries
-
-### 🚧 Roadmap Features
-- Dynamic HNSW parameter tuning
-- Predictive query pattern caching
-- Multi-region S3 replication
-- Automatic cross-node data migration
-- Edge caching layer
+- **Low-latency Triple Intelligence queries** (O(log n) vector + O(1) metadata/graph)
## Conclusion
-Brainy delivers on its promise of **production-ready Triple Intelligence** with measured, verified performance characteristics. All listed features are fully implemented, tested, and benchmarked. No stubs, no mocks, no theoretical claims - just real, working code with measured performance.
\ No newline at end of file
+Brainy delivers on its promise of **production-ready Triple Intelligence** with documented algorithmic-complexity guarantees and a committed graph-scale benchmark (`tests/performance/graph-scale-performance.test.ts`). All listed features are fully implemented and tested. No stubs, no mocks — just real, working code with characterized performance.
\ No newline at end of file
diff --git a/docs/PLUGINS.md b/docs/PLUGINS.md
new file mode 100644
index 00000000..d9a4d3e7
--- /dev/null
+++ b/docs/PLUGINS.md
@@ -0,0 +1,486 @@
+---
+title: Plugin System
+slug: guides/plugins
+public: true
+category: guides
+template: guide
+order: 4
+description: Replace any Brainy subsystem — distance functions, embeddings, vector index, metadata index, aggregation — with a custom implementation or optional native acceleration.
+next:
+ - guides/storage-adapters
+---
+
+# Plugin Development Guide
+
+Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
+
+## Architecture Overview
+
+Brainy's plugin system uses **named providers** — string keys mapped to implementations. During `init()`, brainy:
+
+1. Imports each package listed in the `plugins` config array
+2. Activates each plugin, passing a `BrainyPluginContext`
+3. The plugin calls `context.registerProvider(key, implementation)` for each subsystem it provides
+4. Brainy checks each provider key and wires the implementation into its internal pipeline
+
+Installing the first-party accelerator is the opt-in: with the default config, brainy probes for `@soulcraft/cor` and loads it when present. Everything except "not installed" fails **loud** — a present-but-broken accelerator makes `init()` throw rather than silently degrading to the JS engines.
+
+```typescript
+const brain = new Brainy() // @soulcraft/cor auto-detected when installed
+const pinned = new Brainy({ plugins: ['@soulcraft/cor'] }) // or pin exactly what loads
+const plain = new Brainy({ plugins: [] }) // or opt out of detection entirely
+```
+
+| `plugins` value | Behavior |
+|---|---|
+| `undefined` (default) | Guarded auto-detection of `@soulcraft/cor`: not installed → no plugins, silently; installed → loads + announces; installed-but-broken → `init()` throws |
+| `false` / `[]` | No plugins, no detection (explicit opt-out) |
+| `['@soulcraft/cor']` | Load only the listed packages; a listed plugin that fails to load throws |
+
+Plugins registered programmatically via `brain.use(plugin)` are always activated regardless of the `plugins` config.
+
+If no plugin provides a given key, brainy uses its built-in JavaScript implementation. This means brainy works perfectly standalone — plugins only enhance performance or add capabilities.
+
+## Creating a Plugin
+
+### 1. Implement the `BrainyPlugin` interface
+
+```typescript
+import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
+
+const myPlugin: BrainyPlugin = {
+ name: 'my-brainy-plugin', // Must be unique (typically your npm package name)
+
+ async activate(context: BrainyPluginContext): Promise {
+ // Register your providers here
+ context.registerProvider('distance', myFastDistanceFunction)
+
+ // Return true if activation succeeded, false to skip
+ return true
+ },
+
+ async deactivate(): Promise {
+ // Optional cleanup when brainy.close() is called
+ }
+}
+
+export default myPlugin
+```
+
+### 2. Package exports
+
+Your package must export the plugin as the default export so brainy's plugin loader can resolve it:
+
+```typescript
+// index.ts
+export { default } from './plugin.js'
+```
+
+### 3. Registration
+
+**Config-based:** List your package name in the brainy config:
+
+```typescript
+const brain = new Brainy({
+ plugins: ['my-brainy-plugin']
+})
+await brain.init()
+```
+
+**Programmatic registration:** For plugins not installed as npm packages, use `brain.use()`:
+
+```typescript
+import { Brainy } from '@soulcraft/brainy'
+import myPlugin from './my-plugin.js'
+
+const brain = new Brainy()
+brain.use(myPlugin)
+await brain.init()
+```
+
+## Provider Keys Reference
+
+Each key has a specific expected signature. Brainy checks for these during `init()` and wires them into the appropriate code paths.
+
+### Core Providers
+
+#### `distance`
+**Type:** `(a: number[], b: number[]) => number`
+
+Replaces the default cosine distance function used in vector search and neural APIs. This is the highest-impact single provider — it's called for every vector comparison.
+
+```typescript
+context.registerProvider('distance', (a: number[], b: number[]): number => {
+ // Your SIMD-accelerated or GPU distance calculation
+ return myFastCosineDistance(a, b)
+})
+```
+
+#### `embeddings`
+**Type:** `(text: string | string[]) => Promise`
+
+Replaces the built-in WASM embedding engine. Called for every `brain.add()`, `brain.update()`, and `brain.find()` operation that involves text.
+
+```typescript
+context.registerProvider('embeddings', async (text: string | string[]) => {
+ if (Array.isArray(text)) {
+ return myEngine.embedBatch(text)
+ }
+ return myEngine.embed(text)
+})
+```
+
+#### `embedBatch`
+**Type:** `(texts: string[]) => Promise`
+
+Dedicated batch embedding provider. When registered, brainy uses this for bulk operations (import, reindex, batch add) instead of calling the `embeddings` provider N times. This enables true single-forward-pass batch processing.
+
+Priority order for batch operations:
+1. `embedBatch` provider (single forward pass — fastest)
+2. `embeddings` provider with `Promise.all()` (N individual calls)
+3. Built-in WASM batch API (fallback)
+
+```typescript
+context.registerProvider('embedBatch', async (texts: string[]) => {
+ // Process all texts in a single forward pass
+ return myEngine.batchEmbed(texts)
+})
+```
+
+### Index Providers
+
+> **Write-path invariant (the change-feed contract).** Every canonical
+> mutation flows through Brainy's generation-store commit points — index
+> providers are invoked *inside* that commit and never originate canonical
+> writes of their own. The `brain.onChange` change feed is emitted from those
+> commit points and relies on this: **a plugin must never introduce a write
+> path that bypasses the generation-store commit.** If a future provider ever
+> needs a direct native ingest path, it must either route through the commit
+> or emit equivalent change events — otherwise every `onChange` consumer
+> (live UIs, cache invalidation, realtime sync) silently develops a blind
+> spot.
+
+#### `vector`
+**Type:** `(config: object, distanceFunction: Function, options: object) => VectorIndexProvider-compatible`
+
+Factory function that creates a vector index instance. The returned object must implement the `VectorIndexProvider` public API:
+
+- `addItem(item: { id: string, vector: number[] }): Promise`
+- `search(queryVector: number[], k: number, filter?, options?): Promise>`
+- `removeItem(id: string): Promise`
+- `size(): number`
+- `clear(): void`
+- `flush(): Promise`
+- `rebuild(options?): Promise`
+- `getDirtyNodeCount(): number`
+- `getPersistMode(): 'immediate' | 'deferred'`
+- `getEntryPointId(): string | null`
+- `getMaxLevel(): number`
+- `getDimension(): number | null`
+- `getConfig(): object`
+- `getDistanceFunction(): Function`
+- `enableCOW(parent): void`
+- `setUseParallelization(boolean): void`
+
+For type-aware indexes (separate graph per noun type), also implement:
+- `getIndexForType(type: string): VectorIndexProvider` (duck-typed detection)
+- `search(queryVector, k, type?, filter?, options?): Promise>`
+
+```typescript
+context.registerProvider('vector', (config, distanceFn, options) => {
+ return new MyNativeVectorIndex(config, distanceFn, options)
+})
+```
+
+#### The readiness contract (all three index providers)
+
+A provider that **persists its derived index** should implement the optional readiness
+members so a warm reopen never pays a redundant rebuild-from-canonical:
+
+- **`init?(): Promise`** — eager cold-load. Brainy awaits it once during
+ `brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
+ and **before the rebuild gate**.
+- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
+ loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
+ exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
+ `totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
+ while fully durable. Never return `true` if the durable state failed to load: the
+ signal is honest in both directions, and a not-ready provider gets its rebuild even
+ when `size() > 0`.
+- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
+ migration); brainy skips its rebuild entirely.
+
+Providers that implement none of these keep the size/count heuristics — correct for
+engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
+
+#### `metadataIndex`
+**Type:** `(storage: StorageAdapter) => MetadataIndexManager-compatible`
+
+Factory function that creates a metadata index. The returned object must implement the `MetadataIndexManager` interface including `init()`, `addEntity()`, `removeEntity()`, `query()`, `flush()`, `clear()`, etc.
+
+```typescript
+context.registerProvider('metadataIndex', (storage) => {
+ return new MyNativeMetadataIndex(storage)
+})
+```
+
+#### `graphIndex`
+**Type:** `(storage: StorageAdapter) => GraphAdjacencyIndex-compatible`
+
+Factory function that creates a graph adjacency index for relationship tracking (verbs/triples). Must implement the `GraphAdjacencyIndex` interface including `addVerb()`, `getVerbsBySource()`, `getVerbsByTarget()`, `flush()`, etc.
+
+```typescript
+context.registerProvider('graphIndex', (storage) => {
+ return new MyNativeGraphIndex(storage)
+})
+```
+
+#### `aggregation`
+**Type:** `(storage: StorageAdapter) => AggregationProvider-compatible`
+
+Factory function that creates an aggregation engine for write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows. The returned object must implement the `AggregationProvider` interface.
+
+```typescript
+context.registerProvider('aggregation', (storage) => {
+ return new MyNativeAggregationEngine(storage)
+})
+```
+
+When provided by an optional native acceleration plugin (such as `@soulcraft/cor`), this enables:
+- Compiled source filters (vs per-entity JS object traversal)
+- Precise MIN/MAX via sorted data structures (vs lazy recompute)
+- Parallel aggregate rebuild across CPU cores
+- SIMD-accelerated timestamp bucketing
+
+### Utility Providers
+
+#### `cache`
+**Type:** `UnifiedCache`
+
+Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`).
+
+```typescript
+import type { UnifiedCache } from '@soulcraft/brainy/internals'
+
+context.registerProvider('cache', myNativeCache)
+```
+
+#### `entityIdMapper`
+**Type:** `(storage: StorageAdapter) => EntityIdMapper-compatible`
+
+Factory for bidirectional UUID ↔ integer mapping used by roaring bitmaps. Must implement `getOrAssign()`, `getUuid()`, `getInt()`, `has()`, `remove()`, `flush()`, `clear()`.
+
+#### `roaring`
+**Type:** `RoaringBitmap32 class`
+
+Replacement for the roaring bitmap implementation. Used internally by the metadata index for set operations. Must be API-compatible with `roaring-wasm`.
+
+#### `msgpack`
+**Type:** `{ encode: (data: any) => Buffer, decode: (buffer: Buffer) => any }`
+
+Native msgpack encode/decode for SSTable serialization.
+
+### Analytics Providers (Native-Only)
+
+These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cor`) is installed.
+
+Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it.
+
+#### `analytics:hyperloglog`
+Approximate distinct counts. Count unique values (e.g., unique merchants) across millions of records using ~16KB of memory with ~1% error. Each update is O(1).
+
+#### `analytics:tdigest`
+Streaming percentiles. Compute P50/P90/P95/P99 from streaming data without storing all values. Uses ~4KB per digest with ~1% accuracy at the tails.
+
+#### `analytics:countmin`
+Frequency estimation. Find the most common values (e.g., top-K merchants) using ~40KB with 0.1% error. O(1) per update.
+
+#### `analytics:anomaly`
+Real-time anomaly detection. Flag statistically unusual values at write-time using exponentially weighted moving averages. 64 bytes per group, sub-microsecond decisions.
+
+#### `aggregation:mmap`
+Persistent aggregate storage via memory-mapped files. Aggregate state survives process crashes without explicit flush. Zero serialization overhead.
+
+---
+
+## Storage Adapter Plugins
+
+Plugins can register custom storage backends that users reference by name.
+
+### Implementing a Storage Adapter
+
+```typescript
+import type { StorageAdapterFactory } from '@soulcraft/brainy/plugin'
+import type { StorageAdapter } from '@soulcraft/brainy'
+
+class MyStorageAdapter implements StorageAdapter {
+ async init(): Promise { /* ... */ }
+ async saveNoun(noun: HNSWNoun): Promise { /* ... */ }
+ async getNoun(id: string): Promise { /* ... */ }
+ async deleteNoun(id: string): Promise { /* ... */ }
+ // ... implement all StorageAdapter methods
+}
+```
+
+### Registering a Storage Adapter
+
+```typescript
+context.registerProvider('storage:my-backend', {
+ name: 'my-backend',
+ create: (config: Record) => {
+ return new MyStorageAdapter(config)
+ }
+} satisfies StorageAdapterFactory)
+```
+
+Users can then use your storage:
+
+```typescript
+const brain = new Brainy({ storage: 'my-backend', myBackendOption: 'value' })
+```
+
+## Import Paths
+
+Brainy provides three entry points for plugin developers:
+
+| Import Path | Contents | Stability |
+|-------------|----------|-----------|
+| `@soulcraft/brainy` | Public API, types, StorageAdapter | Stable (semver) |
+| `@soulcraft/brainy/plugin` | BrainyPlugin, BrainyPluginContext, StorageAdapterFactory | Stable (semver) |
+| `@soulcraft/brainy/internals` | UnifiedCache, EntityIdMapper, logger utilities | Internal (may change between minor versions) |
+
+## Diagnostics
+
+Brainy provides a `diagnostics()` method to verify plugin wiring:
+
+```typescript
+const brain = new Brainy()
+await brain.init()
+
+const diag = brain.diagnostics()
+console.log(diag)
+// {
+// version: '7.14.0',
+// plugins: { active: ['my-plugin'], count: 1 },
+// providers: {
+// metadataIndex: { source: 'default' },
+// graphIndex: { source: 'default' },
+// embeddings: { source: 'plugin' },
+// embedBatch: { source: 'plugin' },
+// distance: { source: 'plugin' },
+// vector: { source: 'default' },
+// ...
+// },
+// indexes: {
+// vector: { size: 0, type: 'JsHnswVectorIndex' },
+// metadata: { type: 'MetadataIndexManager', initialized: true },
+// graph: { type: 'GraphAdjacencyIndex', initialized: true, wiredToStorage: true }
+// }
+// }
+```
+
+The CLI also supports diagnostics:
+
+```bash
+brainy diagnostics
+```
+
+### Init-Time Summary
+
+When a plugin is active, brainy automatically logs a provider summary after `init()`:
+
+```
+[brainy] Plugin activated: @soulcraft/cor
+[brainy] Providers: 8/10 native (@soulcraft/cor) | default: vector, cache
+```
+
+This tells you at a glance how many subsystems are accelerated and which ones are falling back to JavaScript. The log respects `config.silent`.
+
+### Fail-Fast for Production
+
+Use `requireProviders()` after `init()` to guarantee specific providers are plugin-supplied. This prevents silent fallback to JavaScript in deployments where you expect native acceleration:
+
+```typescript
+const brain = new Brainy()
+await brain.init()
+
+// Throws immediately if any of these are using JS fallback
+brain.requireProviders(['distance', 'embeddings', 'metadataIndex', 'graphIndex'])
+```
+
+If a required provider is missing, the error message tells you exactly what's wrong:
+
+```
+[brainy] Required providers using JS fallback: graphIndex.
+Active plugins: @soulcraft/cor.
+These providers must be supplied by a plugin for this deployment.
+Check plugin installation, license, and native module availability.
+```
+
+This is the recommended pattern for production deployments with paid plugins — fail at startup rather than silently degrading performance.
+
+## Complete Example: Distance Acceleration Plugin
+
+A minimal but useful plugin that provides SIMD-accelerated distance calculations:
+
+```typescript
+// simd-distance-plugin/src/plugin.ts
+import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
+
+// Hypothetical native module
+import { simdCosineDistance } from './native.js'
+
+const simdDistancePlugin: BrainyPlugin = {
+ name: 'brainy-simd-distance',
+
+ async activate(context: BrainyPluginContext): Promise {
+ // Check if SIMD is available on this platform
+ if (!checkSimdSupport()) {
+ console.log('[simd-distance] SIMD not available, skipping')
+ return false // Don't activate — brainy uses JS fallback
+ }
+
+ context.registerProvider('distance', simdCosineDistance)
+ return true
+ }
+}
+
+export default simdDistancePlugin
+```
+
+```json
+// simd-distance-plugin/package.json
+{
+ "name": "brainy-simd-distance",
+ "main": "./dist/plugin.js",
+ "types": "./dist/plugin.d.ts",
+ "peerDependencies": {
+ "@soulcraft/brainy": ">=7.0.0"
+ }
+}
+```
+
+Usage:
+
+```typescript
+import { Brainy } from '@soulcraft/brainy'
+
+const brain = new Brainy({ plugins: ['brainy-simd-distance'] })
+await brain.init()
+
+// Verify it's active
+const diag = brain.diagnostics()
+console.log(diag.providers.distance) // { source: 'plugin' }
+```
+
+## Design Principles
+
+1. **Brainy works perfectly without plugins.** Every provider has a JavaScript fallback. Plugins only improve performance or add capabilities.
+
+2. **Provider keys are string-based.** The plugin system is not coupled to any specific plugin. Any package can register any provider.
+
+3. **Clean separation.** Plugins access brainy through the documented `BrainyPluginContext` interface. No direct access to internal classes is needed.
+
+4. **Fail-safe activation.** If a plugin throws during `activate()`, brainy logs a warning and continues with defaults. A broken plugin never prevents brainy from working.
+
+5. **Lifecycle management.** `deactivate()` is called during `brainy.close()` for resource cleanup. Native resources, connections, and file handles should be released here.
diff --git a/docs/PRODUCTION_SERVICE_ARCHITECTURE.md b/docs/PRODUCTION_SERVICE_ARCHITECTURE.md
index a7e088f4..4568cd31 100644
--- a/docs/PRODUCTION_SERVICE_ARCHITECTURE.md
+++ b/docs/PRODUCTION_SERVICE_ARCHITECTURE.md
@@ -215,9 +215,9 @@ console.log('Server running on http://localhost:3000')
```
**Benefits:**
-- ✅ Native Bun performance (~2x faster than Node.js)
+- ✅ Native Bun runtime performance
- ✅ No framework dependencies
-- ✅ Works with `bun --compile` for single-binary deployment
+- ✅ Pure WASM — no native binaries, bundler-friendly
- ✅ Built-in TypeScript support
### Pattern 4: Express/Node.js Middleware (Legacy)
diff --git a/docs/QUERY_OPERATORS.md b/docs/QUERY_OPERATORS.md
new file mode 100644
index 00000000..f4ac04e6
--- /dev/null
+++ b/docs/QUERY_OPERATORS.md
@@ -0,0 +1,298 @@
+# Query Operators (BFO)
+
+> Brainy Field Operators — the complete reference for `where` filters in `find()`.
+
+All operators work with `find({ where: { ... } })` and filter on **metadata fields** (not `data`).
+
+---
+
+## Equality
+
+| Operator | Alias | Description | Example |
+|----------|-------|-------------|---------|
+| `eq` | `equals` | Exact match | `{ status: { eq: 'active' } }` |
+| `ne` | `notEquals` | Not equal | `{ status: { ne: 'deleted' } }` |
+
+**Shorthand:** A bare value is treated as `equals`:
+
+```typescript
+// These are equivalent:
+brain.find({ where: { status: 'active' } })
+brain.find({ where: { status: { equals: 'active' } } })
+```
+
+---
+
+## Comparison
+
+| Operator | Alias | Description | Example |
+|----------|-------|-------------|---------|
+| `gt` | `greaterThan` | Greater than | `{ age: { gt: 18 } }` |
+| `gte` | `greaterThanOrEqual` | Greater or equal | `{ score: { gte: 90 } }` |
+| `lt` | `lessThan` | Less than | `{ price: { lt: 100 } }` |
+| `lte` | `lessThanOrEqual` | Less or equal | `{ rating: { lte: 3 } }` |
+| `between` | — | Inclusive range `[min, max]` | `{ year: { between: [2020, 2025] } }` |
+
+```typescript
+// Range query
+const recent = await brain.find({
+ where: {
+ createdAt: { between: [Date.now() - 86400000, Date.now()] }
+ }
+})
+```
+
+---
+
+## Array / Set
+
+| Operator | Alias | Description | Example |
+|----------|-------|-------------|---------|
+| `oneOf` | `in` | Value is one of the given options | `{ color: { oneOf: ['red', 'blue'] } }` |
+| `noneOf` | — | Value is NOT one of the given options | `{ status: { noneOf: ['deleted', 'archived'] } }` |
+| `contains` | — | Array field contains value | `{ tags: { contains: 'ai' } }` |
+| `excludes` | — | Array field does NOT contain value | `{ tags: { excludes: 'spam' } }` |
+| `hasAll` | — | Array field contains ALL listed values | `{ skills: { hasAll: ['js', 'ts'] } }` |
+
+```typescript
+// Find entities tagged with 'ai'
+const aiEntities = await brain.find({
+ where: { tags: { contains: 'ai' } }
+})
+
+// Find entities of specific types
+const people = await brain.find({
+ where: { noun: { oneOf: ['Person', 'Agent'] } }
+})
+```
+
+---
+
+## Existence
+
+| Operator | Description | Example |
+|----------|-------------|---------|
+| `exists: true` | Field exists (has any value) | `{ email: { exists: true } }` |
+| `exists: false` | Field does NOT exist | `{ email: { exists: false } }` |
+| `missing: true` | Field does NOT exist (alias for `exists: false`) | `{ email: { missing: true } }` |
+| `missing: false` | Field exists (alias for `exists: true`) | `{ email: { missing: false } }` |
+
+```typescript
+// Find entities that have an email field
+const withEmail = await brain.find({
+ where: { email: { exists: true } }
+})
+```
+
+---
+
+## Pattern (In-Memory Only)
+
+These operators work via the in-memory filter path. They are applied **after** the indexed query, so use them with other indexed operators for best performance.
+
+| Operator | Description | Example |
+|----------|-------------|---------|
+| `matches` | Regex or string pattern match | `{ name: { matches: /^Dr\./ } }` |
+| `startsWith` | String prefix | `{ name: { startsWith: 'John' } }` |
+| `endsWith` | String suffix | `{ email: { endsWith: '@gmail.com' } }` |
+
+```typescript
+const doctors = await brain.find({
+ where: {
+ type: NounType.Person, // Indexed — fast
+ name: { startsWith: 'Dr.' } // In-memory — applied after
+ }
+})
+```
+
+---
+
+## Logical
+
+Combine multiple conditions:
+
+| Operator | Description | Example |
+|----------|-------------|---------|
+| `allOf` | ALL sub-filters must match (AND) | `{ allOf: [{ status: 'active' }, { role: 'admin' }] }` |
+| `anyOf` | ANY sub-filter must match (OR) | `{ anyOf: [{ role: 'admin' }, { role: 'owner' }] }` |
+| `not` | Invert a filter | `{ not: { status: 'deleted' } }` |
+
+```typescript
+// Complex OR query
+const adminsOrOwners = await brain.find({
+ where: {
+ anyOf: [
+ { role: 'admin' },
+ { role: 'owner' }
+ ]
+ }
+})
+
+// NOT query
+const notDeleted = await brain.find({
+ where: {
+ not: { status: 'deleted' }
+ }
+})
+
+// Combined AND + OR
+const results = await brain.find({
+ where: {
+ allOf: [
+ { department: 'engineering' },
+ { anyOf: [
+ { level: 'senior' },
+ { yearsExperience: { greaterThan: 5 } }
+ ]}
+ ]
+ }
+})
+```
+
+---
+
+## Indexed vs In-Memory Operators
+
+Brainy's MetadataIndex supports a subset of operators natively for O(1) field lookups. Other operators fall back to in-memory filtering.
+
+| Operator | MetadataIndex (Indexed) | In-Memory Fallback |
+|----------|:-----------------------:|:------------------:|
+| `equals` / `eq` | Yes | Yes |
+| `notEquals` / `ne` | — | Yes |
+| `greaterThan` / `gt` | Yes | Yes |
+| `greaterThanOrEqual` / `gte` | Yes | Yes |
+| `lessThan` / `lt` | Yes | Yes |
+| `lessThanOrEqual` / `lte` | Yes | Yes |
+| `between` | Yes | Yes |
+| `oneOf` / `in` | Yes | Yes |
+| `noneOf` | — | Yes |
+| `contains` | Yes | Yes |
+| `exists` / `missing` | Yes | Yes |
+| `matches` | — | Yes |
+| `startsWith` | — | Yes |
+| `endsWith` | — | Yes |
+| `allOf` | Partial | Yes |
+| `anyOf` | Partial | Yes |
+| `not` | — | Yes |
+
+**Performance tip:** Combine indexed operators (equals, greaterThan, oneOf, between, contains, exists) with pattern operators for optimal speed — the index narrows results first, then patterns filter in memory.
+
+---
+
+## Practical Examples
+
+### Filter by entity type
+
+```typescript
+// Using the type shorthand (recommended)
+brain.find({ type: NounType.Person })
+
+// Using where.noun directly
+brain.find({ where: { noun: NounType.Person } })
+
+// Multiple types
+brain.find({ type: [NounType.Person, NounType.Agent] })
+```
+
+### Filter by subtype
+
+`subtype` is a top-level standard field — takes the column-store fast path, not the metadata fallback. Pair with `type` for the typical "Person who is an employee" query:
+
+```typescript
+// Equality on subtype:
+brain.find({ type: NounType.Person, subtype: 'employee' })
+
+// Set membership:
+brain.find({ type: NounType.Person, subtype: ['employee', 'contractor'] })
+
+// Operator-form predicates use `where`:
+brain.find({
+ type: NounType.Person,
+ where: { subtype: { exists: true } }
+})
+```
+
+See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the full surface.
+
+### Filter relationships by subtype (7.30+)
+
+Verbs are first-class peers — `related()` and graph traversal both honor subtype filters on the fast path:
+
+```typescript
+// Filter relationships by VerbType subtype
+const direct = await brain.related({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct'
+})
+
+// Set membership on verb subtype
+const all = await brain.related({
+ from: ceoId,
+ type: VerbType.ReportsTo,
+ subtype: ['direct', 'dotted-line']
+})
+
+// Graph traversal — subtype filters traversal edges (depth-1 in 7.30 JS path;
+// multi-hop subtype filtering lands on Cor native)
+const reports = await brain.find({
+ connected: {
+ from: ceoId,
+ via: VerbType.ReportsTo,
+ subtype: 'direct',
+ depth: 1
+ }
+})
+```
+
+### Combine semantic search with filters
+
+```typescript
+const results = await brain.find({
+ query: 'machine learning engineer', // Semantic search (on data)
+ type: NounType.Person, // Type filter (indexed)
+ where: {
+ department: 'engineering', // Exact match (indexed)
+ yearsExperience: { greaterThan: 3 } // Range filter (indexed)
+ },
+ limit: 10
+})
+```
+
+### Temporal queries
+
+```typescript
+const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000
+const recentEntities = await brain.find({
+ where: {
+ createdAt: { greaterThan: lastWeek }
+ },
+ orderBy: 'createdAt',
+ order: 'desc',
+ limit: 50
+})
+```
+
+### Graph + metadata combination
+
+```typescript
+const results = await brain.find({
+ connected: {
+ from: teamLeadId,
+ via: VerbType.WorksWith,
+ depth: 2
+ },
+ where: {
+ role: { oneOf: ['engineer', 'designer'] },
+ active: true
+ }
+})
+```
+
+---
+
+## See Also
+
+- [Data Model](./DATA_MODEL.md) — Entity structure, data vs metadata
+- [API Reference](./api/README.md) — Complete API documentation
+- [Find System](./FIND_SYSTEM.md) — Natural language find() details
diff --git a/docs/README.md b/docs/README.md
index bed4e26f..3290001f 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,339 +1,130 @@
-# Brainy Documentation (v6.5.0)
+# Brainy Documentation
-Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
+> The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API.
-## 🆕 What's New in v4.0.0
-
-**Production-Ready Cost Optimization:**
-- **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!)
-- **Intelligent-Tiering**: S3 Intelligent-Tiering and GCS Autoclass support
-- **Batch Operations**: Efficient bulk delete operations (1000 objects per request)
-- **Compression**: Gzip compression for FileSystem storage (60-80% space savings)
-- **Quota Monitoring**: Real-time OPFS quota tracking for browser apps
-- **Tier Management**: Azure Hot/Cool/Archive tier management
-
-**Cost Impact Example (500TB dataset):**
-- Before: $138,000/year
-- After v4.0.0: $5,940/year
-- **Savings: $132,060/year (96%)**
-
-## 📊 Implementation Status
-
-- ✅ **Production Ready**: Core features working today
-- 🚧 **In Development**: Features coming soon
-- 📅 **Roadmap**: See [ROADMAP.md](../ROADMAP.md)
-
-## Quick Links
-
-### Getting Started
-- [API Reference](./api/README.md) - Complete API documentation (start here!)
-- [Enterprise for Everyone](./guides/enterprise-for-everyone.md) - **No limits, no tiers, everything free**
-- [Natural Language Queries](./guides/natural-language.md) - Query with plain English
-
-### Core Concepts
-- [Zero Configuration](./architecture/zero-config.md) - **Auto-adapts to any environment**
-- [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) - **Revolutionary data model**
-- [Triple Intelligence](./architecture/triple-intelligence.md) - Unified query system
-- [Architecture Overview](./architecture/overview.md) - System design
-
-### API Documentation
-- [API Reference](./api/README.md) - Complete API documentation
-- [TypeScript Types](./api/types.md) - Type definitions
-
-### Advanced Topics
-- [Augmentations System](./architecture/augmentations.md) - **Enterprise plugins & neural import**
-- [Storage Architecture](./architecture/storage.md) - Storage adapter system
-- [Performance Tuning](./guides/performance.md) - Optimization guide
-- [Migration Guide](../MIGRATION.md) - Upgrading from 1.x
-
-## What is Brainy?
-
-Brainy is a next-generation AI database that combines:
-- **Vector Search**: Semantic similarity using HNSW indexing
-- **Graph Relationships**: Complex relationship mapping and traversal
-- **Field Filtering**: Precise metadata filtering with O(1) lookups
-- **Natural Language**: Query in plain English
-
-## Key Features
-
-### 🧠 Triple Intelligence Engine
-All three intelligence types (vector, graph, field) work together in every query for optimal results.
-
-### 📝 Noun-Verb Taxonomy
-Model your data naturally as entities (nouns) and relationships (verbs) - no complex schemas needed.
-
-### 🌍 Natural Language Queries
-Ask questions in plain English and Brainy understands your intent:
-```typescript
-await brain.find("recent articles about AI with high ratings")
-```
-
-### ⚡ Production Ready
-- Universal storage (FileSystem, S3, OPFS, Memory)
-- Zero configuration with intelligent defaults
-- Full TypeScript support
-- Cross-platform compatibility
-
-## Quick Example
+## Quick Start
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-// Initialize
const brain = new Brainy()
await brain.init()
-// Add entities (nouns)
-const articleId = await brain.add({
- data: "Revolutionary AI Breakthrough",
+// Add entities — data is embedded for semantic search, metadata is indexed for filtering
+const id = await brain.add({
+ data: 'Revolutionary AI Breakthrough',
type: NounType.Document,
- metadata: { category: "technology", rating: 4.8 }
+ metadata: { category: 'technology', rating: 4.8 }
})
-const authorId = await brain.add({
- data: "Dr. Sarah Chen",
- type: NounType.Person,
- metadata: { role: "researcher" }
+// Search with Triple Intelligence
+const results = await brain.find({
+ query: 'artificial intelligence', // Semantic search (on data)
+ where: { rating: { greaterThan: 4.0 } }, // Metadata filter
+ connected: { from: authorId, depth: 2 } // Graph traversal
})
-
-// Create relationships (verbs)
-await brain.relate({
- from: authorId,
- to: articleId,
- type: VerbType.CreatedBy,
- metadata: { date: "2024-01-15", contribution: "primary" }
-})
-
-// Query naturally
-const results = await brain.find("highly rated technology articles by researchers")
```
-## 📚 Complete Documentation Index
+---
-### 🚀 Quick Start Guides
+## Core Documentation
| Document | Description |
|----------|-------------|
-| [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples |
-| [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds |
+| **[API Reference](./api/README.md)** | Complete API documentation — **start here** |
+| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
+| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
+| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
+| [Consistency Model](./concepts/consistency-model.md) | The Db API guarantees — snapshot isolation, atomic transactions, time travel |
-### 🆕 v4.0.0 Migration & Optimization
+---
+
+## Architecture
| Document | Description |
|----------|-------------|
-| [v3→v4 Migration Guide](./MIGRATION-V3-TO-V4.md) | **NEW** - Upgrade guide with zero breaking changes |
-| [AWS S3 Cost Optimization](./operations/cost-optimization-aws-s3.md) | **NEW** - 96% cost savings with lifecycle policies |
-| [GCS Cost Optimization](./operations/cost-optimization-gcs.md) | **NEW** - 94% savings with Autoclass |
-| [Azure Cost Optimization](./operations/cost-optimization-azure.md) | **NEW** - 95% savings with tier management |
-| [Cloudflare R2 Cost Guide](./operations/cost-optimization-cloudflare-r2.md) | **NEW** - Zero egress fees + S3-compatible API |
-
-### 🎯 Core Concepts
-
-| Document | Description |
-|----------|-------------|
-| [Architecture Overview](./architecture/overview.md) | High-level system design and components |
-| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships |
-| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system |
+| [Architecture Overview](./architecture/overview.md) | High-level system design |
+| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Metadata unified query |
+| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | 42 nouns + 127 verbs type system |
+| [Stage 3 Canonical Taxonomy](./STAGE3-CANONICAL-TAXONOMY.md) | Complete type reference |
+| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
+| [Index Architecture](./architecture/index-architecture.md) | Vector, Graph, and Metadata indexing |
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
-| [Storage Architecture](./architecture/storage-architecture.md) | **v4.0.0** - Storage adapters and optimization |
-| [Data Storage Architecture](./architecture/data-storage-architecture.md) | **v4.0.0** - Metadata/vector separation, sharding |
-| [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing |
-### 💾 Storage & Deployment
+---
+
+## Virtual Filesystem (VFS)
| Document | Description |
|----------|-------------|
-| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | **v4.0.0** - Deploy on AWS, GCP, Azure, Cloudflare |
-| [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns |
-| [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment |
-| [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations |
-| [Distributed Storage](./architecture/distributed-storage.md) | Multi-node storage coordination |
-| [Extending Storage](./EXTENDING_STORAGE.md) | Create custom storage adapters |
+| [VFS Quick Start](./vfs/QUICK_START.md) | Get started in 30 seconds |
+| [VFS Core](./vfs/VFS_CORE.md) | Core concepts and architecture |
+| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
+| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
+
+See [vfs/](./vfs/) for the complete VFS documentation set.
+
+---
+
+## Guides
+
+| Document | Description |
+|----------|-------------|
+| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
+| [Snapshots & Time Travel](./guides/snapshots-and-time-travel.md) | Backups, restore, what-if analysis, audit trails |
+| [Natural Language](./guides/natural-language.md) | Query in plain English |
+| [Neural API](./guides/neural-api.md) | AI-powered features |
+| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
+| [Framework Integration](./guides/framework-integration.md) | React, Vue, Angular, Svelte |
+
+---
+
+## Storage & Deployment
+
+| Document | Description |
+|----------|-------------|
+| [Storage Architecture](./architecture/storage-architecture.md) | Filesystem and memory adapters, on-disk artifact layout, operator-layer backup |
| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities |
-### 📊 API Documentation
+---
+
+## Plugins
| Document | Description |
|----------|-------------|
-| [API Reference](./API_REFERENCE.md) | Complete API documentation |
-| [Comprehensive API Overview](./api/COMPREHENSIVE_API_OVERVIEW.md) | All APIs with examples |
-| [API Decision Tree](./API_DECISION_TREE.md) | Choose the right API for your use case |
-| [Core API Patterns](./CORE_API_PATTERNS.md) | Common patterns and best practices |
-| [Neural API Patterns](./NEURAL_API_PATTERNS.md) | AI-powered query patterns |
-| [Find System](./FIND_SYSTEM.md) | Natural language find() API |
-| [API Surface Design](./architecture/API_SURFACE_DESIGN.md) | API design principles |
-| [API Returns](./api-returns.md) | Return types and structures |
+| [Plugins](./PLUGINS.md) | Plugin system overview — providers, `plugins` config, `brain.use()` |
-### 🔧 Framework Integration
+---
+
+## Performance & Scaling
| Document | Description |
|----------|-------------|
-| [Framework Integration Guide](./guides/framework-integration.md) | React, Vue, Angular, Svelte, etc. |
-| [Next.js Integration](./guides/nextjs-integration.md) | Server-side rendering with Brainy |
-| [Vue.js Integration](./guides/vue-integration.md) | Vue 3 integration patterns |
+| [Performance](./PERFORMANCE.md) | Optimization techniques |
+| [Scaling](./SCALING.md) | Scale to billions of entities |
+| [Batching](./BATCHING.md) | Batch operations guide |
-### 📁 Virtual Filesystem (VFS)
+---
+
+## Migration & Reference
| Document | Description |
|----------|-------------|
-| [VFS Core Documentation](./vfs/VFS_CORE.md) | Core VFS concepts and architecture |
-| [Semantic VFS Guide](./vfs/SEMANTIC_VFS.md) | Semantic filesystem projections |
-| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
-| [Neural Extraction](./vfs/NEURAL_EXTRACTION.md) | AI-powered file analysis |
-| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
-| [User Functions](./vfs/USER_FUNCTIONS.md) | Custom VFS functions |
-| [VFS Graph Types](./vfs/VFS_GRAPH_TYPES.md) | Graph-based VFS projections |
-| [VFS Examples & Scenarios](./vfs/VFS_EXAMPLES_SCENARIOS.md) | Real-world VFS examples |
-| [VFS Initialization](./vfs/VFS_INITIALIZATION.md) | Setup and configuration |
-| [Projection Strategy API](./vfs/PROJECTION_STRATEGY_API.md) | Custom projection strategies |
-| [Building File Explorers](./vfs/building-file-explorers.md) | Build custom file browsers |
-| [VFS Troubleshooting](./vfs/TROUBLESHOOTING.md) | Common issues and solutions |
-
-### 🧠 Advanced Topics
-
-| Document | Description |
-|----------|-------------|
-| [Natural Language Queries](./guides/natural-language.md) | Query in plain English |
-| [Neural API](./guides/neural-api.md) | AI-powered features |
-| [Import Anything](./guides/import-anything.md) | Import data from any source |
-| [Distributed Systems](./guides/distributed-system.md) | Multi-node deployments |
-| [Model Loading](./guides/model-loading.md) | Load and manage AI models |
-| [Model Loading Quick Reference](./MODEL_LOADING_QUICK_REFERENCE.md) | Model loading cheat sheet |
-| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No paywalls, no tiers |
-
-### 🔌 Augmentations (Plugins)
-
-| Document | Description |
-|----------|-------------|
-| [Creating Augmentations](./CREATING-AUGMENTATIONS.md) | Build custom plugins |
-| [Augmentations Complete Reference](./augmentations/COMPLETE-REFERENCE.md) | Full augmentation API |
-| [Augmentations Developer Guide](./augmentations/DEVELOPER-GUIDE.md) | Plugin development guide |
-| [Augmentation Configuration](./augmentations/CONFIGURATION.md) | Configure augmentations |
-| [Augmentation System](./architecture/augmentations.md) | System architecture |
-| [Augmentation System Audit](./architecture/augmentation-system-audit.md) | Actual vs planned features |
-| [Augmentations Actual](./architecture/augmentations-actual.md) | Currently implemented augmentations |
-| [API Server Augmentation](./augmentations/api-server.md) | REST API server plugin |
-
-### ⚡ Performance & Scaling
-
-| Document | Description |
-|----------|-------------|
-| [Performance Guide](./PERFORMANCE.md) | Optimization techniques |
-| [Performance Analysis](./architecture/PERFORMANCE_ANALYSIS.md) | Benchmarks and analysis |
-| [Scaling Guide](./SCALING.md) | Scale to billions of entities |
-| [Clustering Algorithms Analysis](./architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md) | HNSW algorithm details |
-| [Initialization & Rebuild](./architecture/initialization-and-rebuild.md) | Index management |
-
-### 📋 Reference
-
-| Document | Description |
-|----------|-------------|
-| [Complete Feature List](./features/complete-feature-list.md) | All Brainy features |
-| [v3 Features](./features/v3-features.md) | v3.x feature list |
-| [Metadata Architecture](./architecture/METADATA_ARCHITECTURE.md) | Metadata namespacing |
-| [Metadata Contract Implementation](./METADATA_CONTRACT_IMPLEMENTATION.md) | Metadata API contracts |
-| [Finite Type System](./architecture/finite-type-system.md) | Type-safe noun/verb taxonomy |
-| [Validation](./VALIDATION.md) | Data validation rules |
-| [Universal Display Augmentation](./universal-display-augmentation.md) | Display system |
-
-### 🛠️ Development
-
-| Document | Description |
-|----------|-------------|
-| [Troubleshooting](./troubleshooting.md) | Common issues and solutions |
+| [v3 to v4 Migration](./MIGRATION-V3-TO-V4.md) | Upgrade guide |
| [Release Guide](./RELEASE-GUIDE.md) | How to release new versions |
+| [Production Architecture](./PRODUCTION_SERVICE_ARCHITECTURE.md) | Ops reference |
-### 🔍 Internal Documentation
+---
+
+## Internal
| Document | Description |
|----------|-------------|
| [Audit Report](./internal/AUDIT_REPORT.md) | Feature audit |
-| [Cleanup Summary](./internal/CLEANUP_SUMMARY.md) | Codebase cleanup notes |
| [Honest Status](./internal/HONEST_STATUS.md) | Actual implementation status |
---
-## Documentation Structure
-
-```
-docs/
-├── README.md (this file) # Complete documentation index
-├── MIGRATION-V3-TO-V4.md # v4.0.0 migration guide
-│
-├── guides/ # User guides
-│ ├── natural-language.md
-│ ├── neural-api.md
-│ ├── import-anything.md
-│ ├── framework-integration.md
-│ ├── nextjs-integration.md
-│ ├── vue-integration.md
-│ ├── distributed-system.md
-│ ├── model-loading.md
-│ └── enterprise-for-everyone.md
-│
-├── architecture/ # System architecture
-│ ├── overview.md
-│ ├── noun-verb-taxonomy.md
-│ ├── triple-intelligence.md
-│ ├── zero-config.md
-│ ├── storage-architecture.md # v4.0.0
-│ ├── data-storage-architecture.md # v4.0.0
-│ ├── index-architecture.md
-│ ├── distributed-storage.md
-│ ├── augmentations.md
-│ ├── augmentation-system-audit.md
-│ ├── augmentations-actual.md
-│ ├── finite-type-system.md
-│ └── ...
-│
-├── operations/ # Operations guides
-│ ├── cost-optimization-aws-s3.md # v4.0.0
-│ ├── cost-optimization-gcs.md # v4.0.0
-│ ├── cost-optimization-azure.md # v4.0.0
-│ ├── cost-optimization-cloudflare-r2.md # v4.0.0
-│ └── capacity-planning.md
-│
-├── deployment/ # Deployment guides
-│ ├── CLOUD_DEPLOYMENT_GUIDE.md # v4.0.0
-│ ├── aws-deployment.md
-│ ├── gcp-deployment.md
-│ └── kubernetes-deployment.md
-│
-├── vfs/ # Virtual Filesystem docs
-│ ├── QUICK_START.md
-│ ├── VFS_CORE.md
-│ ├── SEMANTIC_VFS.md
-│ ├── VFS_API_GUIDE.md
-│ ├── NEURAL_EXTRACTION.md
-│ ├── COMMON_PATTERNS.md
-│ └── ...
-│
-├── api/ # API documentation
-│ ├── README.md
-│ └── COMPREHENSIVE_API_OVERVIEW.md
-│
-├── augmentations/ # Augmentation docs
-│ ├── COMPLETE-REFERENCE.md
-│ ├── DEVELOPER-GUIDE.md
-│ ├── CONFIGURATION.md
-│ └── api-server.md
-│
-├── features/ # Feature documentation
-│ ├── complete-feature-list.md
-│ └── v3-features.md
-│
-└── internal/ # Internal docs
- ├── AUDIT_REPORT.md
- ├── CLEANUP_SUMMARY.md
- └── HONEST_STATUS.md
-```
-
-## Community
-
-- **GitHub**: [github.com/brainy-org/brainy](https://github.com/brainy-org/brainy)
-- **Issues**: [Report bugs or request features](https://github.com/brainy-org/brainy/issues)
-- **Discussions**: [Join the conversation](https://github.com/brainy-org/brainy/discussions)
-
## License
-Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
\ No newline at end of file
+Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
diff --git a/docs/RELEASE-GUIDE.md b/docs/RELEASE-GUIDE.md
index bf8abb9f..94c6a6cb 100644
--- a/docs/RELEASE-GUIDE.md
+++ b/docs/RELEASE-GUIDE.md
@@ -48,7 +48,7 @@ git commit -m "chore: remove unused dependency"
# DON'T use BREAKING CHANGE for internal changes
git commit -m "feat: improve model delivery
-BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers v3.0.0
+BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers
```
## Release Workflow Checklist
@@ -119,4 +119,13 @@ gh release create vY.Y.Y --generate-notes
- **Most releases should be MINOR or PATCH**
- **Major versions should be RARE**
- **When in doubt, it's probably MINOR**
-- **NEVER use "BREAKING CHANGE" for internal changes**
\ No newline at end of file
+- **NEVER use "BREAKING CHANGE" for internal changes**
+## Hard Ordering Constraints (check before EVERY release)
+
+- **Embedding model changes are SEQUENCED, not free.** No release may change the
+ embedding model (or its quantization/dimensions) before **vector model-version
+ stamping + hard-error-on-mismatch** ships. Stored vectors carry no model version
+ today; mixing vectors from two models silently corrupts every similarity
+ comparison. If a model bump is ever proposed, the stamping work moves ahead of
+ it in the schedule — coordinate with the native provider so both engines stamp
+ and enforce identically. (Registered with the native-provider team 2026-07-07.)
diff --git a/docs/SCALING.md b/docs/SCALING.md
index d071a838..e9ae1136 100644
--- a/docs/SCALING.md
+++ b/docs/SCALING.md
@@ -1,502 +1,239 @@
-# 🚀 Brainy Scaling Guide - Enterprise for Everyone
+# Brainy Scaling Guide
-> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required.
+> **One Line Summary**: Single-node by design — Brainy scales by getting the most out of one machine plus operator-layer backup.
-## 📖 Table of Contents
+## Table of Contents
- [Quick Start](#quick-start)
-- [How It Works](#how-it-works)
+- [How Brainy Scales](#how-brainy-scales)
- [Storage Configurations](#storage-configurations)
- [Scaling Patterns](#scaling-patterns)
- [Real World Examples](#real-world-examples)
## Quick Start
-### Single Node (Default)
+### In-Memory
```typescript
import Brainy from '@soulcraft/brainy'
-const brain = new Brainy() // That's it!
+const brain = new Brainy({ storage: { type: 'memory' } })
```
-### Multi-Node (Auto-Discovery)
+### On-Disk (Default for Node)
```typescript
-// Node 1
-const brain = new Brainy() // Starts as primary
-
-// Node 2 (different server)
-const brain = new Brainy() // Auto-discovers Node 1, becomes replica!
-```
-
-**That's literally all you need!** Brainy handles everything else automatically.
-
-## How It Works
-
-### 🎯 The Magic: Zero Configuration
-
-Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration:
-
-1. **First node starts** → Becomes primary automatically
-2. **Second node starts** → Discovers first node via UDP broadcast
-3. **Nodes negotiate** → Elect leader, distribute shards
-4. **Data flows** → Automatic replication and routing
-5. **Node fails** → Automatic failover in <1 second
-
-### 🔄 Automatic Node Discovery
-
-```typescript
-// Three ways Brainy finds other nodes (auto-selected):
-
-// 1. LOCAL NETWORK (Default)
-// Uses UDP broadcast on port 7946
-// Perfect for: On-premise, same VPC
-
-// 2. CLOUD NATIVE (Auto-detected)
-// Kubernetes: Uses k8s DNS service discovery
-// AWS: Uses EC2 tags or Route53
-// Azure: Uses Azure DNS
-
-// 3. EXPLICIT (When needed)
const brain = new Brainy({
- peers: ['node1.example.com', 'node2.example.com']
+ storage: { type: 'filesystem', path: './brainy-data' }
})
```
-### 📊 Data Distribution
+## How Brainy Scales
-When you add data, Brainy automatically:
+Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery, no S3 coordination. Scaling means:
-```typescript
-brain.add({ name: "John" }, 'person')
+- **Up**: give the process more RAM, CPU, and IOPS
+- **Out**: stand up multiple independent Brainy instances behind your own service layer
+- **Cold storage**: snapshot the on-disk artifact off-site so you can rehydrate elsewhere
-// Behind the scenes:
-// 1. Hash ID to determine shard (consistent hashing)
-// 2. Find nodes responsible for this shard
-// 3. Write to primary shard owner
-// 4. Replicate to N backup nodes (default: 2)
-// 5. Confirm write when majority acknowledge
-```
+The three knobs that matter most:
+
+1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
+2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
+
+The native vector provider (via the optional `@soulcraft/cor` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
+
+## Measured Performance
+
+Numbers below are **measured** by `tests/benchmarks/find-composition-scale.js` (a single
+Node 22 process, in-memory storage, 384-dim vectors, `balanced` recall). They are the
+open-core (pure-TypeScript) path — what you get from `@soulcraft/brainy` with no native
+provider installed. Run it yourself: `node --max-old-space-size=8192 tests/benchmarks/find-composition-scale.js 100000`.
+
+`find()` query latency, p50 / p95 (200 queries each):
+
+| Query | 5,000 entities | 100,000 entities |
+|---|---|---|
+| Vector similarity (`{ vector }`) | 0.8 / 1.3 ms | 1.4 / 4.7 ms |
+| Graph 1-hop (`{ connected }`) | 0.5 / 0.7 ms | 0.7 / 0.8 ms |
+| Metadata filter (`{ where }`, low-selectivity) | 0.7 / 1.2 ms | 23.5 / 30.1 ms |
+| Vector + metadata | 7.7 / 8.3 ms | 78.8 / 93.8 ms |
+
+What the shape tells you:
+
+- **Vector and graph lookups scale ~logarithmically** — they barely move from 5k to 100k,
+ because HNSW search is ~O(ef·log n) and graph adjacency is O(degree).
+- **Metadata-filtered paths scale with the size of the match set, not the database.** The
+ benchmark's `category` filter matches ~10% of rows (10,000 at 100k); the cost is
+ materializing that candidate set and running the vector search *inside* it (`find()` does
+ metadata-first hard filtering, then ranks within the candidates — see
+ [How find works](./FIND_SYSTEM.md)). A **high-selectivity** filter (few matches) is far
+ cheaper; a 10%-of-everything filter is the worst case. This candidate-restricted search is
+ precisely the path the native provider accelerates (Rust roaring-bitmap candidate
+ intersection).
+- **Composition is correct, not lossy.** Combining vector + metadata + graph returns exactly
+ the entities satisfying all constraints — verified by
+ `tests/integration/find-triple-composition.test.ts`.
+
+Memory: ~62 KB resident per entity at 100k (6.2 GB RSS for 100k × 384-dim including the HNSW
+graph, metadata index, and 100k edges).
+
+**Scale ceiling (open-core).** The pure-JS HNSW *build* cost (~100 inserts/s at 384-dim on
+one core) makes the in-process open-core path most appropriate up to ~10⁵–10⁶ entities.
+*Query* latency stays low well beyond that, but for the 10⁸–10¹⁰ regime install the native
+provider (`@soulcraft/cor`, on-disk DiskANN) — same API, no code change. _Projected from
+the two measured points, vector p50 at 1M is ~2 ms; metadata-heavy composition grows with
+match-set size and is the path to move onto the native provider first._
## Storage Configurations
-### 🗂️ Storage Adapter Patterns
-
-Brainy intelligently adapts to your storage setup:
-
-#### Pattern 1: Separate Storage Per Node (Recommended)
-
+### Filesystem (Recommended for Production)
```typescript
-// Node 1 - Own filesystem
-const brain1 = new Brainy({
- storage: '/data/node1' // or auto: './brainy-data'
-})
-
-// Node 2 - Own filesystem
-const brain2 = new Brainy({
- storage: '/data/node2' // or auto: './brainy-data'
-})
-
-// ✅ BENEFITS:
-// - No conflicts between nodes
-// - Fast local reads
-// - True horizontal scaling
-// - Survives network partitions
-```
-
-#### Pattern 2: Separate S3 Buckets Per Node
-
-```typescript
-// Node 1 - Own S3 bucket
-const brain1 = new Brainy({
- storage: 's3://brainy-node-1' // Auto-uses AWS credentials
-})
-
-// Node 2 - Own S3 bucket
-const brain2 = new Brainy({
- storage: 's3://brainy-node-2'
-})
-
-// ✅ BENEFITS:
-// - Infinite storage capacity
-// - Geographic distribution
-// - No local disk needed
-// - Built-in durability
-```
-
-#### Pattern 3: Shared S3 Bucket (Coordinated)
-
-```typescript
-// All nodes - Shared bucket with coordination
-const brain = new Brainy({
- storage: 's3://shared-brainy-data',
- // Brainy automatically adds node-specific prefixes!
-})
-
-// What happens automatically:
-// - Node 1 writes to: s3://shared-brainy-data/node-1/
-// - Node 2 writes to: s3://shared-brainy-data/node-2/
-// - Metadata in: s3://shared-brainy-data/_cluster/
-// - Coordination via S3 conditional writes
-
-// ✅ BENEFITS:
-// - Single bucket to manage
-// - Easy backup/restore
-// - Cost effective
-// - Automatic namespace isolation
-```
-
-#### Pattern 4: Mixed Storage (Hybrid)
-
-```typescript
-// Hot data on local SSD, cold data in S3
const brain = new Brainy({
storage: {
- hot: '/fast-ssd/brainy', // Recent/frequent data
- cold: 's3://brainy-archive' // Older data
+ type: 'filesystem',
+ path: '/var/lib/brainy'
}
- // Brainy automatically promotes/demotes data!
})
```
+- Stores everything in a sharded JSON tree under `path`
+- Atomic writes via rename
+- Survives process restarts
+- Snapshot it off-site with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` from your scheduler
-### 🌍 Cloud Provider Auto-Detection
+### Memory
+```typescript
+const brain = new Brainy({ storage: { type: 'memory' } })
+```
+- Zero I/O, fastest possible
+- No persistence — process exit discards everything
+- Use for tests and ephemeral caches
+### Auto
```typescript
const brain = new Brainy({
- storage: 'cloud://brainy-data' // Auto-detects provider!
+ storage: { type: 'auto', path: './data' }
})
-
-// Automatically uses:
-// - AWS: S3 + DynamoDB for metadata
-// - Google Cloud: GCS + Firestore
-// - Azure: Blob Storage + Cosmos DB
-// - Cloudflare: R2 + D1
-// - Vercel: Blob + KV
```
-
-### 📝 Storage Coordination Rules
-
-When multiple nodes share storage, Brainy automatically:
-
-1. **Namespace Isolation**: Each node gets unique prefix
-2. **Lock-Free Writes**: Uses atomic operations
-3. **Consistent Metadata**: Coordinated via consensus
-4. **Conflict Resolution**: Version vectors for conflicts
-5. **Garbage Collection**: Automatic cleanup of old data
+- Picks `filesystem` when running on Node with a writable `path`
+- Falls back to `memory` otherwise
## Scaling Patterns
-### 📈 Progressive Scaling Journey
-
-#### Stage 1: Prototype (1 node, memory)
+### Stage 1: Prototype (Memory)
```typescript
-const brain = new Brainy() // Memory storage, single node
-// Perfect for: Development, testing, <1000 items
+const brain = new Brainy({ storage: { type: 'memory' } })
+// Development, tests, <100K items
```
-#### Stage 2: Production (1 node, disk)
+### Stage 2: Production (Filesystem)
```typescript
const brain = new Brainy({
- storage: './data' // Persistent storage
+ storage: { type: 'filesystem', path: '/var/lib/brainy' }
})
-// Perfect for: Small apps, <100K items
+// Most production workloads up to ~10M entities on a single host
```
-#### Stage 3: High Availability (2-3 nodes)
+### Stage 3: Higher Throughput (Tune the Vector Index)
```typescript
-// Just start same code on multiple servers!
const brain = new Brainy({
- storage: './data' // Each node's own storage
+ storage: { type: 'filesystem', path: '/var/lib/brainy' },
+ vector: {
+ recall: 'fast', // Trade recall for latency
+ persistMode: 'deferred' // Batch persistence
+ }
})
-// Automatic: Leader election, replication, failover
-// Perfect for: Critical apps, <1M items
```
-#### Stage 4: Scale Out (N nodes)
-```typescript
-// Same code, more servers!
-const brain = new Brainy({
- storage: 's3://brainy-{{nodeId}}' // Template auto-filled
-})
-// Automatic: Sharding, load balancing, geo-distribution
-// Perfect for: Large apps, unlimited items
-```
-
-### 🎯 Common Scaling Scenarios
-
-#### Scenario: Read-Heavy Application
-```typescript
-// Brainy auto-detects read-heavy pattern and:
-// 1. Increases cache size
-// 2. Creates more read replicas
-// 3. Routes reads to nearest node
-// 4. Caches popular items on all nodes
-
-const brain = new Brainy() // No config needed!
-```
-
-#### Scenario: Multi-Tenant SaaS
-```typescript
-// Brainy auto-detects tenant patterns and:
-// 1. Shards by tenant ID
-// 2. Isolates tenant data
-// 3. Routes by tenant
-// 4. Separate rate limits per tenant
-
-const brain = new Brainy() // Detects from your queries!
-```
-
-#### Scenario: Geographic Distribution
-```typescript
-// Deploy nodes in different regions
-// Brainy automatically:
-// 1. Detects node locations (via latency)
-// 2. Replicates data geographically
-// 3. Routes to nearest node
-// 4. Handles region failures
-
-// US-East
-const brain = new Brainy({ region: 'us-east' }) // Optional hint
-
-// EU-West (auto-discovers US-East)
-const brain = new Brainy({ region: 'eu-west' })
-```
+### Stage 4: Multi-Instance (Operator-Layer)
+Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `path`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes.
## Real World Examples
-### Example 1: Blog Platform
-
+### Example 1: Single-Node App With Backup
```typescript
-// Day 1: Single server
const brain = new Brainy({
- storage: './blog-data'
+ storage: { type: 'filesystem', path: '/var/lib/brainy' }
})
-
-// Month 6: Add redundancy (on second server)
-const brain = new Brainy({
- storage: './blog-data' // Different machine!
-})
-// Automatically syncs with first server
-
-// Year 2: Global scale
-// US Server
-const brain = new Brainy({
- storage: 's3://blog-us/data'
-})
-
-// EU Server
-const brain = new Brainy({
- storage: 's3://blog-eu/data'
-})
-
-// Asia Server
-const brain = new Brainy({
- storage: 's3://blog-asia/data'
-})
-// All automatically coordinate!
+```
+Schedule (cron / systemd timer):
+```bash
+*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
```
-### Example 2: E-Commerce Site
-
+### Example 2: Tests
```typescript
-// Development
-const brain = new Brainy() // Memory storage
-
-// Staging (Kubernetes)
-const brain = new Brainy({
- storage: process.env.STORAGE_PATH // Uses PVC
-})
-// Auto-discovers other pods via K8s DNS
-
-// Production (AWS)
-const brain = new Brainy({
- storage: 's3://shop-data',
- cache: 'elasticache://shop-cache' // Optional
-})
-// Auto-scales with ECS/EKS
+const brain = new Brainy({ storage: { type: 'memory' } })
+// Fast, no cleanup needed between runs
```
-### Example 3: Analytics Platform
-
+### Example 3: Multi-Tenant Service
+Spin up one Brainy instance per tenant, each in its own directory:
```typescript
-// Ingestion nodes (write-optimized)
-const brain = new Brainy({
- role: 'writer', // Hint for optimization
- storage: '/fast-nvme/ingest'
-})
-
-// Query nodes (read-optimized)
-const brain = new Brainy({
- role: 'reader', // More cache, indexes
- storage: 's3://analytics-archive'
-})
-
-// Automatically coordinates between writers and readers!
-```
-
-## 🔧 Storage Adapter Specifics
-
-### Local Filesystem
-```typescript
-{
- storage: './data' // or absolute: '/var/lib/brainy'
- // Each node MUST have separate directory
- // Can be network mounted (NFS, EFS)
+function brainForTenant(tenantId: string) {
+ return new Brainy({
+ storage: {
+ type: 'filesystem',
+ path: `/var/lib/brainy/${tenantId}`
+ }
+ })
}
```
+Your service layer handles routing and isolation; Brainy stays simple.
-### AWS S3
+### Example 4: Higher Recall at Scale
```typescript
-{
- storage: 's3://bucket-name/prefix'
- // Uses AWS SDK credentials (env, IAM role, etc)
- // Supports S3-compatible (MinIO, Ceph)
-}
-```
-
-### Cloudflare R2
-```typescript
-{
- storage: 'r2://bucket-name'
- // Uses Wrangler or API tokens
- // Zero egress fees!
-}
-```
-
-### Google Cloud Storage
-```typescript
-{
- storage: 'gs://bucket-name'
- // Uses Application Default Credentials
-}
-```
-
-### Azure Blob Storage
-```typescript
-{
- storage: 'azure://container-name'
- // Uses DefaultAzureCredential
-}
-```
-
-### Mixed/Tiered
-```typescript
-{
- storage: {
- hot: './local-cache', // Fast SSD
- warm: 's3://regular-data', // Standard storage
- cold: 's3://glacier-archive' // Cheap archive
+const brain = new Brainy({
+ storage: { type: 'filesystem', path: '/var/lib/brainy' },
+ vector: {
+ recall: 'accurate'
}
- // Automatic tiering based on access patterns
-}
-```
-
-## 🎭 Advanced Patterns
-
-### Pattern: Blue-Green Deployment
-```typescript
-// Blue cluster (current)
-const brain = new Brainy({
- cluster: 'blue',
- storage: 's3://prod-blue'
-})
-
-// Green cluster (new version)
-const brain = new Brainy({
- cluster: 'green',
- storage: 's3://prod-green',
- syncFrom: 'blue' // Real-time sync during migration
})
```
-### Pattern: Federation
-```typescript
-// Region 1 Cluster
-const brain1 = new Brainy({
- federation: 'global',
- region: 'us-east',
- storage: 's3://us-east-data'
-})
+## Tuning Knobs Summary
-// Region 2 Cluster
-const brain2 = new Brainy({
- federation: 'global',
- region: 'eu-west',
- storage: 's3://eu-west-data'
-})
-// Clusters coordinate for global queries!
-```
+| Setting | Values | When to change |
+|---------|--------|----------------|
+| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
+| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
+| `storage.cache.maxSize` | integer | Hot-path read cache size |
+| `storage.cache.ttl` | ms | Cache freshness |
-### Pattern: Edge Computing
-```typescript
-// Edge nodes (in CDN POPs)
-const brain = new Brainy({
- mode: 'edge',
- storage: 'memory', // RAM only
- upstream: 'https://main-cluster.example.com'
-})
-// Caches frequently accessed data at edge
-```
-
-## 📊 Monitoring & Observability
-
-Brainy automatically exposes metrics:
+## Monitoring & Observability
```typescript
-const metrics = brain.getMetrics()
+const stats = await brain.stats()
// {
-// nodes: { total: 5, healthy: 5 },
-// shards: { total: 20, local: 4 },
-// replication: { factor: 2, lag: 45 },
-// operations: { reads: 10000, writes: 1000 },
-// storage: { used: '45GB', available: '955GB' }
+// nounCount: 50000,
+// verbCount: 80000,
+// vectorIndex: { ... },
+// storage: { used: '45GB' }
// }
```
-## 🚨 Troubleshooting
+## Troubleshooting
-### Issue: Nodes don't discover each other
-```typescript
-// Solution 1: Check network allows UDP 7946
-// Solution 2: Use explicit peers
-const brain = new Brainy({
- peers: ['10.0.0.1:7946', '10.0.0.2:7946']
-})
-```
+### Issue: Slow queries
+1. Switch to `vector.recall: 'fast'`
+2. Increase the read cache (`storage.cache.maxSize`)
+3. Consider the optional native vector provider via `@soulcraft/cor`
-### Issue: Storage conflicts
-```typescript
-// Ensure each node has unique storage path
-// ❌ WRONG: All nodes use './data'
-// ✅ RIGHT: Node1: './data1', Node2: './data2'
-// ✅ RIGHT: Use {{nodeId}} template
-```
+### Issue: Memory pressure
+1. Reduce `storage.cache.maxSize`
+2. Move to `vector.persistMode: 'deferred'` to batch writes
+3. Consider the optional native vector provider via `@soulcraft/cor` for at-scale index acceleration
-### Issue: Slow performance
-```typescript
-// Brainy auto-tunes, but you can hint:
-const brain = new Brainy({
- profile: 'read-heavy' // or 'write-heavy', 'balanced'
-})
-```
+### Issue: Slow startup after a crash
+1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
+2. Verify backup integrity periodically
-## 🎯 Best Practices
+## Best Practices
-1. **Let Brainy Auto-Configure**: Don't over-configure
-2. **Separate Storage Per Node**: Avoids conflicts
-3. **Use S3 for Large Scale**: Infinite capacity
-4. **Start Simple**: Single node → Scale when needed
-5. **Monitor Metrics**: Watch for bottlenecks
-6. **Trust Auto-Scaling**: It learns your patterns
+1. **One process = one `path`** — never share a directory between processes
+2. **Snapshot from your scheduler** — Brainy doesn't ship cloud SDKs; use `rclone` / `aws s3 sync` / `gsutil`
+3. **Profile before tuning** — `recall: 'balanced'` is right for most workloads
+4. **Install the native vector provider only when measured profiling shows it pays off**
-## 🚀 Summary
+## Summary
-- **Zero Config**: Just `new Brainy()` at any scale
-- **Auto-Discovery**: Nodes find each other
-- **Smart Storage**: Adapts to any backend
-- **Progressive Scaling**: 1 → 100 nodes seamlessly
-- **Self-Tuning**: Learns and optimizes
-- **No DevOps**: It just works!
-
-**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!**
-
----
-
-*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)*
\ No newline at end of file
+- Brainy 8.0 is a **library**, not a cluster
+- Storage adapters: `filesystem`, `memory`, `auto`
+- Vector tuning: `recall`, `persistMode`
+- Backup is an operator-layer concern — snapshot `path`
diff --git a/docs/STAGE3-CANONICAL-TAXONOMY.md b/docs/STAGE3-CANONICAL-TAXONOMY.md
index b1fab35a..bfafb9e5 100644
--- a/docs/STAGE3-CANONICAL-TAXONOMY.md
+++ b/docs/STAGE3-CANONICAL-TAXONOMY.md
@@ -1,4 +1,4 @@
-# Brainy Stage 3: Canonical Taxonomy (v6.0.0)
+# Brainy Stage 3: Canonical Taxonomy
**Status:** FINAL - This is the definitive, timeless taxonomy
**Total Types:** 169 (42 nouns + 127 verbs)
diff --git a/docs/VALIDATION.md b/docs/VALIDATION.md
deleted file mode 100644
index a0c42668..00000000
--- a/docs/VALIDATION.md
+++ /dev/null
@@ -1,340 +0,0 @@
-# Brainy Validation System
-
-## Zero-Config Philosophy
-
-Brainy's validation system automatically adapts to your system resources without any configuration. It enforces universal truths while dynamically adjusting limits based on available memory and observed performance.
-
-## Core Principles
-
-### 1. Universal Truths Only
-We only validate things that are mathematically or logically impossible:
-- Negative pagination values (there's no page -1)
-- Probabilities outside 0-1 range
-- Self-referential relationships
-- Invalid enum values
-
-### 2. Auto-Configuration
-The system automatically configures based on:
-- **Available Memory**: More RAM = higher limits
-- **System Performance**: Adjusts based on query response times
-- **Usage Patterns**: Learns from your actual workload
-
-### 3. Performance Monitoring
-Every query is monitored to tune future limits:
-```typescript
-// Automatic adjustment based on performance
-if (avgQueryTime < 100ms && resultCount > 80% of limit) {
- // Increase limits - system can handle more
- maxLimit *= 1.5
-} else if (avgQueryTime > 1000ms) {
- // Reduce limits - system is struggling
- maxLimit *= 0.8
-}
-```
-
-## Validation Rules by Method
-
-### `add(params: AddParams)`
-
-**Required:**
-- Either `data` or `vector` must be provided
-- `type` must be a valid NounType enum
-
-**Constraints:**
-- `vector` must have exactly 384 dimensions (for all-MiniLM-L6-v2)
-- Custom `id` must be unique
-
-**Example:**
-```typescript
-// ✅ Valid
-await brain.add({
- data: "Hello world",
- type: NounType.Document
-})
-
-// ✅ Valid - pre-computed vector
-await brain.add({
- vector: new Array(384).fill(0),
- type: NounType.Document
-})
-
-// ❌ Invalid - missing both data and vector
-await brain.add({
- type: NounType.Document
-})
-// Error: "must provide either data or vector"
-
-// ❌ Invalid - wrong vector dimensions
-await brain.add({
- vector: new Array(100).fill(0),
- type: NounType.Document
-})
-// Error: "vector must have exactly 384 dimensions"
-```
-
-### `update(params: UpdateParams)`
-
-**Required:**
-- `id` must be provided
-- At least one field must be updated
-
-**Important Metadata Behavior:**
-- `metadata: null` with `merge: false` → **Keeps existing metadata** (does nothing)
-- `metadata: {}` with `merge: false` → **Clears metadata** ✅
-- `metadata: undefined` → No change to metadata
-
-**Example:**
-```typescript
-// ✅ Valid - update metadata
-await brain.update({
- id: "xyz",
- metadata: { status: "published" }
-})
-
-// ✅ Valid - clear metadata properly
-await brain.update({
- id: "xyz",
- metadata: {},
- merge: false
-})
-
-// ❌ Invalid - null doesn't clear metadata
-await brain.update({
- id: "xyz",
- metadata: null,
- merge: false
-})
-// Error: "must specify at least one field to update"
-// (because null metadata doesn't actually update anything)
-
-// ❌ Invalid - no fields to update
-await brain.update({
- id: "xyz"
-})
-// Error: "must specify at least one field to update"
-```
-
-### `relate(params: RelateParams)`
-
-**Required:**
-- `from` entity ID
-- `to` entity ID
-- `type` must be valid VerbType enum
-
-**Constraints:**
-- `from` and `to` must be different (no self-loops)
-- `weight` must be between 0 and 1
-
-**Example:**
-```typescript
-// ✅ Valid
-await brain.relate({
- from: "entity1",
- to: "entity2",
- type: VerbType.RelatedTo
-})
-
-// ❌ Invalid - self-referential
-await brain.relate({
- from: "entity1",
- to: "entity1",
- type: VerbType.RelatedTo
-})
-// Error: "cannot create self-referential relationship"
-
-// ❌ Invalid - weight out of range
-await brain.relate({
- from: "entity1",
- to: "entity2",
- type: VerbType.RelatedTo,
- weight: 1.5
-})
-// Error: "weight must be between 0 and 1"
-```
-
-### `find(params: FindParams)`
-
-**Constraints:**
-- `limit` must be non-negative and below auto-configured maximum
-- `offset` must be non-negative
-- Cannot specify both `query` and `vector` (mutually exclusive)
-- Cannot use both `cursor` and `offset` pagination
-- `threshold` must be between 0 and 1
-
-**Auto-Configured Limits:**
-```typescript
-// Based on available memory
-// 1GB RAM → max limit: 10,000
-// 8GB RAM → max limit: 80,000
-// 16GB RAM → max limit: 100,000 (capped)
-
-// Query length also scales with memory
-// 1GB RAM → max query: 5,000 characters
-// 8GB RAM → max query: 40,000 characters
-```
-
-**Example:**
-```typescript
-// ✅ Valid
-await brain.find({
- query: "machine learning",
- limit: 50
-})
-
-// ❌ Invalid - negative limit
-await brain.find({
- query: "test",
- limit: -1
-})
-// Error: "limit must be non-negative"
-
-// ❌ Invalid - both query and vector
-await brain.find({
- query: "test",
- vector: new Array(384).fill(0)
-})
-// Error: "cannot specify both query and vector - they are mutually exclusive"
-
-// ❌ Invalid - exceeds auto-configured limit
-await brain.find({
- limit: 1000000
-})
-// Error: "limit exceeds auto-configured maximum of 80000 (based on available memory)"
-```
-
-## Auto-Configuration Details
-
-### Memory-Based Scaling
-
-The validation system checks available memory on initialization:
-
-```typescript
-const availableMemory = os.freemem()
-
-// Scale limits based on available memory
-maxLimit = Math.min(
- 100000, // Absolute maximum for safety
- Math.floor(availableMemory / (1024 * 1024 * 100)) * 1000
-)
-
-// Scale query length similarly
-maxQueryLength = Math.min(
- 50000,
- Math.floor(availableMemory / (1024 * 1024 * 10)) * 1000
-)
-```
-
-### Performance-Based Tuning
-
-The system continuously monitors and adjusts:
-
-1. **After each query**, performance is recorded
-2. **Limits adjust** based on response times
-3. **Gradual optimization** towards optimal throughput
-
-### Checking Current Configuration
-
-You can inspect the current validation configuration:
-
-```typescript
-import { getValidationConfig } from '@soulcraft/brainy/validation'
-
-const config = getValidationConfig()
-console.log(config)
-// {
-// maxLimit: 80000,
-// maxQueryLength: 40000,
-// maxVectorDimensions: 384,
-// systemMemory: 17179869184,
-// availableMemory: 8589934592
-// }
-```
-
-## Best Practices
-
-### 1. Clearing Metadata
-```typescript
-// ❌ Wrong - doesn't clear
-await brain.update({ id, metadata: null, merge: false })
-
-// ✅ Correct - actually clears
-await brain.update({ id, metadata: {}, merge: false })
-```
-
-### 2. Type Safety
-```typescript
-// ❌ Wrong - string type
-await brain.add({ data: "test", type: "document" })
-
-// ✅ Correct - enum type
-import { NounType } from '@soulcraft/brainy'
-await brain.add({ data: "test", type: NounType.Document })
-```
-
-### 3. Pagination
-```typescript
-// ✅ Let the system auto-configure limits
-const results = await brain.find({
- query: "test",
- limit: 100 // Will be capped at system maximum
-})
-
-// ✅ For large datasets, use pagination
-let offset = 0
-const pageSize = 1000
-while (true) {
- const results = await brain.find({
- query: "test",
- limit: pageSize,
- offset
- })
- if (results.length === 0) break
- offset += pageSize
-}
-```
-
-## Error Messages
-
-All validation errors are descriptive and actionable:
-
-| Error | Cause | Solution |
-|-------|-------|----------|
-| `"must provide either data or vector"` | Missing content in add() | Provide either data to embed or pre-computed vector |
-| `"limit must be non-negative"` | Negative pagination | Use positive limit value |
-| `"invalid NounType: xyz"` | Invalid enum value | Use valid NounType enum |
-| `"cannot create self-referential relationship"` | from === to | Use different entity IDs |
-| `"must specify at least one field to update"` | Empty update | Provide at least one field to change |
-| `"vector must have exactly 384 dimensions"` | Wrong vector size | Use 384-dimensional vectors |
-
-## Performance Impact
-
-The validation system adds minimal overhead:
-- **Validation time**: <1ms per operation
-- **Memory usage**: ~1KB for configuration tracking
-- **Auto-tuning**: Happens asynchronously, no blocking
-
-## FAQ
-
-**Q: Why can't I set metadata to null?**
-A: Setting metadata to `null` with `merge: false` doesn't actually clear it - it falls back to existing metadata. Use `{}` to clear.
-
-**Q: Why are my limits being reduced?**
-A: If queries are taking >1 second, the system automatically reduces limits to maintain performance.
-
-**Q: Can I override the auto-configured limits?**
-A: No, this is by design. The system knows better than static configuration what your hardware can handle.
-
-**Q: Why exactly 384 dimensions for vectors?**
-A: Brainy uses the all-MiniLM-L6-v2 model which produces 384-dimensional embeddings. This ensures consistency.
-
-## Summary
-
-Brainy's validation system:
-- ✅ **Zero configuration** - adapts to your system
-- ✅ **Universal truths** - only prevents impossible operations
-- ✅ **Performance aware** - adjusts based on actual performance
-- ✅ **Type safe** - enforces enum types
-- ✅ **Minimal overhead** - <1ms validation time
-- ✅ **Clear errors** - actionable error messages
-
-The philosophy is simple: prevent impossible operations, adapt to reality, and get out of the way.
\ No newline at end of file
diff --git a/docs/ZERO_CONFIG.md b/docs/ZERO_CONFIG.md
deleted file mode 100644
index 6f20a74f..00000000
--- a/docs/ZERO_CONFIG.md
+++ /dev/null
@@ -1,406 +0,0 @@
-# 🚀 Brainy Zero-Configuration Guide
-
-## Overview
-
-Starting with v2.10, Brainy introduces a **Zero-Configuration System** that automatically configures everything based on your environment. No more environment variables, no more complex configuration objects - just create and use.
-
-## Quick Start
-
-### True Zero Config
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-// That's it! Everything auto-configures
-const brain = new Brainy()
-await brain.init()
-```
-
-### Using Strongly-Typed Presets
-
-```typescript
-import { Brainy, PresetName } from '@soulcraft/brainy'
-
-// Type-safe preset selection
-const brain = new Brainy(PresetName.PRODUCTION)
-await brain.init()
-```
-
-### Common Scenarios
-
-#### Development
-```typescript
-const brain = new Brainy(PresetName.DEVELOPMENT)
-// ✅ Filesystem storage for persistence
-// ✅ FP32 models for best quality
-// ✅ Verbose logging
-// ✅ All features enabled
-```
-
-#### Production
-```typescript
-const brain = new Brainy(PresetName.PRODUCTION)
-// ✅ Disk storage for persistence
-// ✅ Auto-selected model precision
-// ✅ Silent logging
-// ✅ Optimized features
-```
-
-#### Minimal
-```typescript
-const brain = new Brainy(PresetName.MINIMAL)
-// ✅ Filesystem storage
-// ✅ Q8 models for small size
-// ✅ Core features only
-// ✅ Minimal resource usage
-```
-
-## Distributed Architecture Presets
-
-Brainy includes specialized presets for distributed and microservice architectures:
-
-### Basic Distributed Roles
-```typescript
-import { Brainy, PresetName } from '@soulcraft/brainy'
-
-// Write-only instance (data ingestion)
-const writer = new Brainy(PresetName.WRITER)
-// ✅ Optimized for writes
-// ✅ No search index loading
-// ✅ Minimal memory usage
-
-// Read-only instance (search API)
-const reader = new Brainy(PresetName.READER)
-// ✅ Optimized for search
-// ✅ Lazy index loading
-// ✅ Large cache
-```
-
-### Service-Specific Presets
-```typescript
-// High-throughput data ingestion
-const ingestion = new Brainy(PresetName.INGESTION_SERVICE)
-
-// Low-latency search API
-const searchApi = new Brainy(PresetName.SEARCH_API)
-
-// Analytics processing
-const analytics = new Brainy(PresetName.ANALYTICS_SERVICE)
-
-// Edge location cache
-const edge = new Brainy(PresetName.EDGE_CACHE)
-
-// Batch processing
-const batch = new Brainy(PresetName.BATCH_PROCESSOR)
-
-// Real-time streaming
-const streaming = new Brainy(PresetName.STREAMING_SERVICE)
-
-// ML training
-const training = new Brainy(PresetName.ML_TRAINING)
-
-// Lightweight sidecar
-const sidecar = new Brainy(PresetName.SIDECAR)
-```
-
-## Model Precision Control
-
-You can **explicitly specify** model precision when needed:
-
-```typescript
-import { ModelPrecision } from '@soulcraft/brainy'
-
-// Force FP32 (full precision)
-const brain = new Brainy({ model: ModelPrecision.FP32 })
-
-// Force Q8 (quantized, smaller)
-const brain = new Brainy({ model: ModelPrecision.Q8 })
-
-// Use presets
-const brain = new Brainy({ model: ModelPrecision.FAST }) // Maps to fp32
-const brain = new Brainy({ model: ModelPrecision.SMALL }) // Maps to q8
-
-// Auto-detection (default)
-const brain = new Brainy({ model: ModelPrecision.AUTO })
-```
-
-### Auto-Detection Logic
-
-When not specified, Brainy automatically selects the best model:
-
-- **Browser**: Q8 (smaller download)
-- **Serverless**: Q8 (faster cold starts)
-- **Low Memory (<512MB)**: Q8
-- **Development**: FP32 (best quality)
-- **Production (>2GB RAM)**: FP32
-- **Default**: Q8 (balanced)
-
-## Storage Configuration
-
-### Automatic Storage Detection
-
-Brainy automatically detects the best storage option:
-
-1. **Cloud Storage** (if credentials found)
- - AWS S3 (checks AWS_ACCESS_KEY_ID, AWS_PROFILE)
- - Google Cloud Storage (checks GOOGLE_APPLICATION_CREDENTIALS)
- - Cloudflare R2 (checks R2_ACCESS_KEY_ID)
-
-2. **Browser Storage**
- - OPFS (if supported)
- - Filesystem fallback
-
-3. **Node.js Storage**
- - Filesystem (`./brainy-data` or `~/.brainy/data`)
-
-### Manual Storage Control
-
-```typescript
-import { StorageOption } from '@soulcraft/brainy'
-
-// Force specific storage with enum
-const brain = new Brainy({ storage: StorageOption.DISK })
-const brain = new Brainy({ storage: StorageOption.CLOUD })
-const brain = new Brainy({ storage: StorageOption.AUTO })
-
-// Custom storage configuration
-const brain = new Brainy({
- storage: {
- s3Storage: {
- bucket: 'my-bucket',
- region: 'us-east-1'
- }
- }
-})
-```
-
-## Feature Sets
-
-Control which features are enabled:
-
-```typescript
-import { FeatureSet } from '@soulcraft/brainy'
-
-// Preset feature sets with enum
-const brain = new Brainy({ features: FeatureSet.MINIMAL }) // Core only
-const brain = new Brainy({ features: FeatureSet.DEFAULT }) // Balanced
-const brain = new Brainy({ features: FeatureSet.FULL }) // Everything
-
-// Custom features
-const brain = new Brainy({
- features: ['core', 'search', 'cache', 'triple-intelligence']
-})
-```
-
-## Simplified Configuration Interface
-
-The new configuration is dramatically simpler:
-
-```typescript
-interface BrainyZeroConfig {
- // Mode preset - now with distributed options
- mode?: PresetName // All strongly typed presets
-
- // Model configuration with enum
- model?: ModelPrecision
-
- // Storage configuration with enum
- storage?: StorageOption | StorageConfig
-
- // Feature set with enum
- features?: FeatureSet | string[]
-
- // Logging
- verbose?: boolean
-
- // Escape hatch for advanced users
- advanced?: any
-}
-```
-
-### Available Enums
-
-```typescript
-enum PresetName {
- // Basic
- PRODUCTION = 'production',
- DEVELOPMENT = 'development',
- MINIMAL = 'minimal',
- ZERO = 'zero',
-
- // Distributed
- WRITER = 'writer',
- READER = 'reader',
-
- // Services
- INGESTION_SERVICE = 'ingestion-service',
- SEARCH_API = 'search-api',
- ANALYTICS_SERVICE = 'analytics-service',
- EDGE_CACHE = 'edge-cache',
- BATCH_PROCESSOR = 'batch-processor',
- STREAMING_SERVICE = 'streaming-service',
- ML_TRAINING = 'ml-training',
- SIDECAR = 'sidecar'
-}
-
-enum ModelPrecision {
- FP32 = 'fp32',
- Q8 = 'q8',
- AUTO = 'auto',
- FAST = 'fast', // Maps to fp32
- SMALL = 'small' // Maps to q8
-}
-
-enum StorageOption {
- AUTO = 'auto',
- DISK = 'disk',
- CLOUD = 'cloud'
-}
-
-enum FeatureSet {
- MINIMAL = 'minimal',
- DEFAULT = 'default',
- FULL = 'full'
-}
-```
-
-## Multi-Instance with Shared Storage
-
-When multiple Brainy instances connect to the same storage (like S3), you **must ensure they use compatible configurations**:
-
-```typescript
-import { ModelPrecision } from '@soulcraft/brainy'
-
-// Container A - Writer
-const writer = new Brainy({
- mode: PresetName.WRITER,
- model: ModelPrecision.FP32, // ⚠️ MUST match across instances!
- storage: { s3Storage: { bucket: 'shared-data' }}
-})
-
-// Container B - Reader
-const reader = new Brainy({
- mode: PresetName.READER,
- model: ModelPrecision.FP32, // ✅ Matches Container A
- storage: { s3Storage: { bucket: 'shared-data' }}
-})
-```
-
-### Distributed Architecture Example
-
-```typescript
-// Ingestion Service (Writer)
-const ingestion = new Brainy({
- mode: PresetName.INGESTION_SERVICE,
- model: ModelPrecision.Q8, // All instances must use Q8
- storage: { s3Storage: { bucket: 'production-data' }}
-})
-
-// Search API (Reader)
-const search = new Brainy({
- mode: PresetName.SEARCH_API,
- model: ModelPrecision.Q8, // Matches ingestion service
- storage: { s3Storage: { bucket: 'production-data' }}
-})
-
-// Analytics (Hybrid)
-const analytics = new Brainy({
- mode: PresetName.ANALYTICS_SERVICE,
- model: ModelPrecision.Q8, // Matches other services
- storage: { s3Storage: { bucket: 'production-data' }}
-})
-```
-
-## Migration from Old Configuration
-
-### Before (Complex)
-```typescript
-const brain = new Brainy({
- hnsw: {
- M: 16,
- efConstruction: 200,
- seed: 42
- },
- storage: {
- s3Storage: {
- bucketName: 'my-bucket',
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
- region: 'us-east-1'
- },
- cacheConfig: {
- hotCacheMaxSize: 5000,
- hotCacheEvictionThreshold: 0.8,
- warmCacheTTL: 3600000,
- batchSize: 100
- }
- },
- cache: {
- autoTune: true,
- autoTuneInterval: 60000,
- hotCacheMaxSize: 10000
- },
- embeddingFunction: customFunction,
- readOnly: false,
- logging: { verbose: true }
-})
-```
-
-### After (Simple)
-```typescript
-const brain = new Brainy('production')
-// Everything above is auto-configured!
-```
-
-## Environment Variables (No Longer Needed!)
-
-These environment variables are **no longer required**:
-
-- ❌ `BRAINY_ALLOW_REMOTE_MODELS` - Models auto-download when needed
-- ❌ `BRAINY_MODELS_PATH` - Path auto-selected based on environment
-- ❌ `BRAINY_Q8_CONFIRMED` - Warnings auto-suppressed in production
-- ❌ `BRAINY_LOG_LEVEL` - Auto-set based on NODE_ENV
-- ❌ AWS credentials - Use AWS SDK credential chain
-
-## Performance Impact
-
-The zero-config system has **zero performance overhead**:
-
-- Configuration happens once during initialization
-- Auto-detected values are cached
-- Same optimized code paths as manual configuration
-- Actually **faster** startup due to reduced parsing
-
-## Troubleshooting
-
-### Models Not Downloading
-- Check internet connection
-- Ensure firewall allows HTTPS to Hugging Face / CDN
-- Run `npm run download-models` to pre-download
-
-### Wrong Model Precision
-- Explicitly specify: `{ model: 'fp32' }` or `{ model: 'q8' }`
-- Check shared storage compatibility
-
-### Storage Detection Issues
-- Check cloud credentials are properly configured
-- Verify write permissions for filesystem paths
-- Use explicit storage configuration if needed
-
-## Best Practices
-
-1. **Use zero-config for single instances** - Let Brainy handle everything
-2. **Specify precision for shared storage** - Ensure compatibility
-3. **Use presets for common scenarios** - 'development', 'production', 'minimal'
-4. **Override only what you need** - Start simple, add complexity only if required
-
-## Summary
-
-The new zero-config system reduces configuration from **100+ parameters** to **0-3 decisions**:
-
-| Scenario | Old Config Lines | New Config Lines |
-|----------|-----------------|------------------|
-| Development | 50+ | 1 |
-| Production | 100+ | 1 |
-| Custom | 200+ | 3-5 |
-
-**Result**: 95% less configuration, 100% of the power! 🚀
\ No newline at end of file
diff --git a/docs/api-returns.md b/docs/api-returns.md
deleted file mode 100644
index a9f5e76a..00000000
--- a/docs/api-returns.md
+++ /dev/null
@@ -1,152 +0,0 @@
-# Brainy API Return Values
-
-## Core Operations
-
-### `brain.add()` - Adding Entities
-
-**Returns:** `Promise` - The ID of the created entity
-
-```typescript
-// ✅ Correct usage - add() returns the ID string directly
-const id = await brain.add({
- type: 'document',
- data: 'My document content'
-})
-
-console.log('Created entity with ID:', id)
-
-// Use the ID to create relationships
-await brain.relate({
- from: id,
- to: anotherId,
- type: 'references'
-})
-```
-
-```typescript
-// ❌ Incorrect - trying to access .id property
-const result = await brain.add({
- type: 'document',
- data: 'My content'
-})
-
-console.log(result.id) // ❌ undefined - result IS the ID, not an object!
-```
-
-### Getting the Full Entity
-
-If you need the full entity object after creation, use `brain.get()`:
-
-```typescript
-// Add entity and get its ID
-const id = await brain.add({
- type: 'document',
- data: 'My content',
- metadata: { label: 'Important Doc' }
-})
-
-// Get the full entity
-const entity = await brain.get(id)
-console.log(entity.id) // The ID
-console.log(entity.type) // 'document'
-console.log(entity.data) // 'My content'
-console.log(entity.metadata) // { label: 'Important Doc', ... }
-console.log(entity.vector) // The embedding vector
-console.log(entity.createdAt) // Timestamp
-```
-
-### `brain.find()` - Finding Entities
-
-**Returns:** `Promise` - Array of full entity objects
-
-```typescript
-const entities = await brain.find({
- query: 'machine learning',
- limit: 10
-})
-
-// Each entity has full information
-for (const entity of entities) {
- console.log(entity.id)
- console.log(entity.type)
- console.log(entity.data)
- console.log(entity.metadata)
-}
-```
-
-### `brain.relate()` - Creating Relationships
-
-**Returns:** `Promise` - The ID of the created relationship
-
-```typescript
-const relationId = await brain.relate({
- from: entityId1,
- to: entityId2,
- type: 'references'
-})
-
-console.log('Created relationship with ID:', relationId)
-```
-
-## Data Field Behavior
-
-### String Data - Used for Embeddings
-
-When `data` is a string, it's used to generate embeddings for semantic search:
-
-```typescript
-await brain.add({
- type: 'document',
- data: 'This text will be converted to an embedding vector',
- metadata: {
- title: 'My Document',
- year: 2024
- }
-})
-```
-
-### Object Data - Structured Information
-
-When `data` is an object, it's treated as structured data:
-
-```typescript
-await brain.add({
- type: 'product',
- data: {
- name: 'Widget',
- price: 29.99,
- category: 'Tools'
- },
- vector: precomputedVector // Must provide vector when using object data
-})
-```
-
-**Important:** If you provide object data without a `vector`, you must include a string somewhere for embedding generation, or the operation will fail.
-
-### Metadata vs Data
-
-- **`data`**: Primary content - used for embeddings (if string) or stored as structured data (if object)
-- **`metadata`**: Auxiliary information - always stored as structured data, used for filtering
-
-**Best practice for labels:**
-```typescript
-await brain.add({
- type: 'document',
- data: 'The full text content of the document...', // For semantic search
- metadata: {
- label: 'Quick Reference Label', // For display
- author: 'John Doe',
- category: 'Technical'
- }
-})
-```
-
-## Summary
-
-| Method | Returns | Contains |
-|--------|---------|----------|
-| `brain.add()` | `string` | The ID of the created entity |
-| `brain.get()` | `Entity \| null` | Full entity object with all fields |
-| `brain.find()` | `Entity[]` | Array of full entity objects |
-| `brain.relate()` | `string` | The ID of the created relationship |
-| `brain.getRelations()` | `Relation[]` | Array of relationship objects |
diff --git a/docs/api/COMPREHENSIVE_API_OVERVIEW.md b/docs/api/COMPREHENSIVE_API_OVERVIEW.md
deleted file mode 100644
index 3a729be5..00000000
--- a/docs/api/COMPREHENSIVE_API_OVERVIEW.md
+++ /dev/null
@@ -1,563 +0,0 @@
-# Brainy Complete Public API Reference
-
-> **Accurate API documentation for Brainy v6.5.0+**
-
-## Initialization
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-// Zero-config (just works)
-const brain = new Brainy()
-await brain.init()
-
-// With configuration
-const brain = new Brainy({
- storage: { type: 'memory' }, // or { path: './my-data' } for filesystem
- embeddingModel: 'Q8', // Q4, Q8, F16, F32
- silent: true // Suppress logs
-})
-await brain.init()
-```
-
-## Readiness API (v7.3.0+)
-
-For reliable initialization detection, especially in cloud environments with progressive initialization:
-
-### `brain.ready` - Await Initialization
-
-```typescript
-const brain = new Brainy()
-brain.init() // Fire and forget
-
-// Elsewhere (e.g., API handler)
-await brain.ready // Wait until init() completes
-const results = await brain.find({ query: 'test' })
-```
-
-### `brain.isInitialized` - Check Basic Readiness
-
-```typescript
-if (brain.isInitialized) {
- // Safe to use brain methods
-}
-```
-
-### `brain.isFullyInitialized()` - Check Background Tasks
-
-```typescript
-// Returns true when ALL initialization is complete, including background tasks
-// Useful for cloud storage adapters with progressive initialization
-if (brain.isFullyInitialized()) {
- console.log('All background tasks complete')
-}
-```
-
-### `brain.awaitBackgroundInit()` - Wait for Background Tasks
-
-```typescript
-const brain = new Brainy({ storage: { type: 'gcs', ... } })
-await brain.init() // Fast return in cloud (<200ms)
-
-// Optional: wait for all background tasks (bucket validation, count sync)
-await brain.awaitBackgroundInit()
-console.log('Fully initialized including background tasks')
-```
-
-### Health Check Pattern
-
-```typescript
-app.get('/health', async (req, res) => {
- try {
- await brain.ready
- res.json({
- status: 'ready',
- fullyInitialized: brain.isFullyInitialized()
- })
- } catch (error) {
- res.status(503).json({ status: 'initializing', error: error.message })
- }
-})
-```
-
-## Core CRUD Operations
-
-### `brain.add(params)` - Add Entity
-
-```typescript
-// Add with data (auto-embedded)
-const id = await brain.add({
- data: 'Machine learning is a subset of AI',
- type: NounType.Document,
- metadata: { author: 'Alice', tags: ['ml', 'ai'] }
-})
-
-// Add with pre-computed vector
-const id = await brain.add({
- data: 'Content here',
- vector: [0.1, 0.2, ...], // 384 dimensions
- type: NounType.Concept
-})
-```
-
-**Parameters:**
-- `data` (required): Content to embed (string, object, or any serializable data)
-- `type?`: NounType enum value
-- `metadata?`: Custom key-value pairs
-- `vector?`: Pre-computed embedding vector
-- `id?`: Custom ID (auto-generated if not provided)
-
-### `brain.get(id, options?)` - Get Entity
-
-```typescript
-const entity = await brain.get('entity-id')
-
-// Include vector embeddings (not loaded by default for performance)
-const entity = await brain.get('entity-id', { includeVectors: true })
-```
-
-### `brain.update(params)` - Update Entity
-
-```typescript
-await brain.update({
- id: 'entity-id',
- data: 'Updated content', // Re-embeds if changed
- metadata: { reviewed: true } // Merges with existing
-})
-```
-
-### `brain.delete(id)` - Delete Entity
-
-```typescript
-await brain.delete('entity-id')
-```
-
-### `brain.clear()` - Clear All Data
-
-```typescript
-await brain.clear()
-```
-
-## Relationships
-
-### `brain.relate(params)` - Create Relationship
-
-```typescript
-const relationId = await brain.relate({
- from: 'source-entity-id',
- to: 'target-entity-id',
- type: VerbType.RelatedTo,
- metadata: { strength: 0.9 }
-})
-```
-
-**Parameters:**
-- `from` (required): Source entity ID
-- `to` (required): Target entity ID
-- `type` (required): VerbType enum value
-- `metadata?`: Custom relationship metadata
-
-### `brain.getRelations(params)` - Query Relationships
-
-```typescript
-// Get all relationships from an entity
-const relations = await brain.getRelations({ from: 'entity-id' })
-
-// Get relationships to an entity
-const relations = await brain.getRelations({ to: 'entity-id' })
-
-// Filter by type
-const relations = await brain.getRelations({
- from: 'entity-id',
- type: VerbType.Contains
-})
-```
-
-### `brain.unrelate(id)` - Delete Relationship
-
-```typescript
-await brain.unrelate('relationship-id')
-```
-
-## Search & Query
-
-### `brain.find(query)` - Semantic Search
-
-The primary search method with Triple Intelligence (semantic + graph + metadata).
-
-```typescript
-// Simple text search
-const results = await brain.find('machine learning algorithms')
-
-// With options
-const results = await brain.find({
- query: 'machine learning',
- limit: 10,
- threshold: 0.7,
- type: NounType.Document,
- where: { author: 'Alice' },
- excludeVFS: true // Exclude VFS files from results
-})
-
-// Natural language query (Triple Intelligence)
-const results = await brain.find(
- 'Show me documents about AI written by Alice in 2024'
-)
-```
-
-**Parameters:**
-- `query`: Search text (required)
-- `limit?`: Max results (default: 10)
-- `threshold?`: Minimum similarity (0-1)
-- `type?`: Filter by NounType
-- `where?`: Metadata filters
-- `excludeVFS?`: Exclude VFS entities
-
-### `brain.similar(params)` - Find Similar Entities
-
-```typescript
-// Find similar to entity ID
-const similar = await brain.similar({
- to: 'entity-id',
- limit: 5
-})
-
-// Find similar to vector
-const similar = await brain.similar({
- to: [0.1, 0.2, ...], // Vector
- limit: 10,
- type: NounType.Document
-})
-```
-
-## Batch Operations
-
-### `brain.addMany(params)` - Batch Add
-
-```typescript
-const result = await brain.addMany({
- items: [
- { data: 'First item', type: NounType.Document },
- { data: 'Second item', type: NounType.Concept },
- { data: 'Third item', metadata: { priority: 'high' } }
- ],
- continueOnError: true, // Don't stop on failures
- onProgress: (completed, total) => {
- console.log(`${completed}/${total} complete`)
- }
-})
-
-console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}`)
-```
-
-### `brain.deleteMany(params)` - Batch Delete
-
-```typescript
-// Delete by IDs
-const result = await brain.deleteMany({
- ids: ['id1', 'id2', 'id3'],
- continueOnError: true
-})
-
-// Delete by type
-const result = await brain.deleteMany({
- type: NounType.TempData
-})
-
-// Delete by metadata filter
-const result = await brain.deleteMany({
- where: { status: 'archived' }
-})
-```
-
-### `brain.relateMany(params)` - Batch Relate
-
-```typescript
-const ids = await brain.relateMany({
- items: [
- { from: 'a', to: 'b', type: VerbType.RelatedTo },
- { from: 'b', to: 'c', type: VerbType.Contains },
- { from: 'c', to: 'd', type: VerbType.References }
- ],
- continueOnError: true
-})
-```
-
-### `brain.updateMany(params)` - Batch Update
-
-```typescript
-const result = await brain.updateMany({
- items: [
- { id: 'id1', metadata: { reviewed: true } },
- { id: 'id2', data: 'Updated content' }
- ],
- continueOnError: true
-})
-```
-
-## Neural API
-
-Access advanced AI/ML features via `brain.neural()`:
-
-```typescript
-const neural = brain.neural()
-
-// Similarity calculation
-const similarity = await neural.similar('text1', 'text2')
-const detailed = await neural.similar('text1', 'text2', { detailed: true })
-
-// Clustering
-const clusters = await neural.clusters()
-const clusters = await neural.clusters({
- algorithm: 'hierarchical',
- maxClusters: 10
-})
-
-// K-nearest neighbors
-const neighbors = await neural.neighbors('entity-id', { k: 5 })
-
-// Semantic hierarchy
-const hierarchy = await neural.hierarchy('entity-id')
-
-// Outlier/anomaly detection
-const outliers = await neural.outliers({ threshold: 2.0 })
-
-// Domain-aware clustering
-const domainClusters = await neural.clusterByDomain('category')
-
-// Temporal clustering
-const temporalClusters = await neural.clusterByTime('createdAt', [
- { start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' },
- { start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' }
-])
-
-// Streaming clusters (for large datasets)
-for await (const batch of neural.clusterStream({ batchSize: 100 })) {
- console.log(`Progress: ${batch.progress.percentage}%`)
-}
-```
-
-## Virtual File System (VFS)
-
-Access via `brain.vfs`:
-
-```typescript
-const vfs = brain.vfs
-await vfs.init()
-
-// File operations
-await vfs.writeFile('/docs/readme.md', '# Hello')
-const content = await vfs.readFile('/docs/readme.md')
-await vfs.unlink('/docs/readme.md')
-
-// Directory operations
-await vfs.mkdir('/project/src', { recursive: true })
-const files = await vfs.readdir('/project')
-await vfs.rmdir('/project', { recursive: true })
-
-// Bulk operations (v6.5.0+)
-const result = await vfs.bulkWrite([
- { type: 'mkdir', path: '/data' },
- { type: 'write', path: '/data/config.json', data: '{}' },
- { type: 'write', path: '/data/users.json', data: '[]' }
-])
-// Note: mkdir operations run first (sequentially), then other ops in parallel
-
-// Semantic search
-const results = await vfs.search('authentication code', { path: '/src' })
-
-// File metadata
-const stats = await vfs.stat('/file.txt')
-await vfs.setMetadata('/file.txt', { author: 'Alice' })
-```
-
-## Counts (O(1) Performance)
-
-```typescript
-// Entity counts
-const total = brain.counts.entities()
-const byType = await brain.counts.byType(NounType.Document)
-const nonVFS = await brain.counts.byType({ excludeVFS: true })
-
-// Relationship counts
-const relations = brain.counts.relationships()
-const byVerb = await brain.counts.byVerbType(VerbType.Contains)
-```
-
-## Versioning & Branching
-
-```typescript
-// Create branch
-await brain.fork('feature-branch')
-
-// List branches
-const branches = await brain.listBranches()
-
-// Switch branch
-await brain.checkout('feature-branch')
-
-// Get current branch
-const current = await brain.getCurrentBranch()
-
-// Commit changes
-await brain.commit({ message: 'Added new features' })
-
-// View history
-const history = await brain.getHistory({ limit: 10 })
-
-// Time travel (read-only snapshot)
-const snapshot = await brain.asOf('commit-id')
-```
-
-## Streaming API
-
-```typescript
-// Stream all entities
-for await (const entity of brain.streaming.entities()) {
- console.log(entity.id)
-}
-
-// Stream with filters
-for await (const entity of brain.streaming.entities({
- type: NounType.Document,
- where: { status: 'active' }
-})) {
- // Process each entity
-}
-
-// Stream relationships
-for await (const relation of brain.streaming.relations({
- from: 'entity-id'
-})) {
- console.log(relation)
-}
-```
-
-## Pagination API
-
-```typescript
-// Paginated queries
-const page1 = await brain.pagination.find({
- query: 'machine learning',
- page: 1,
- pageSize: 20
-})
-
-console.log(`Page ${page1.page} of ${page1.totalPages}`)
-console.log(`Total results: ${page1.total}`)
-
-// Get next page
-const page2 = await brain.pagination.find({
- query: 'machine learning',
- page: 2,
- pageSize: 20
-})
-```
-
-## Augmentations
-
-```typescript
-// List active augmentations
-const augmentations = brain.augmentations.list()
-
-// Get specific augmentation
-const cache = brain.augmentations.get('cache')
-
-// Augmentations are auto-loaded based on config
-// Common augmentations: cache, display, metrics, intelligent-import
-```
-
-## Utilities
-
-```typescript
-// Manual embedding
-const vector = await brain.embed('text to embed')
-
-// Flush pending writes
-await brain.flush()
-
-// Get statistics
-const stats = await brain.getStats()
-const statsNoVFS = await brain.getStats({ excludeVFS: true })
-
-// Close (cleanup)
-await brain.close()
-```
-
-## Type Enums
-
-```typescript
-import { NounType, VerbType } from '@soulcraft/brainy'
-
-// NounType - Entity types
-NounType.Document
-NounType.Person
-NounType.Concept
-NounType.Event
-NounType.Location
-NounType.Organization
-NounType.Product
-NounType.Content
-NounType.Collection
-// ... and more
-
-// VerbType - Relationship types
-VerbType.RelatedTo
-VerbType.Contains
-VerbType.References
-VerbType.DependsOn
-VerbType.Precedes
-VerbType.Follows
-VerbType.CreatedBy
-VerbType.ModifiedBy
-// ... and more
-```
-
-## Error Handling
-
-```typescript
-try {
- await brain.get('nonexistent-id')
-} catch (error) {
- if (error.message.includes('not found')) {
- // Handle missing entity
- }
-}
-
-// VFS errors use POSIX codes
-try {
- await vfs.readFile('/nonexistent')
-} catch (error) {
- if (error.code === 'ENOENT') {
- // File not found
- }
-}
-```
-
-## Configuration Reference
-
-```typescript
-const brain = new Brainy({
- // Storage
- storage: {
- type: 'memory' | 'filesystem',
- path: './data', // For filesystem
- forceMemoryStorage: false // Force memory even if path exists
- },
-
- // Embeddings
- embeddingModel: 'Q8', // Q4, Q8, F16, F32
- dimensions: 384, // Auto-detected
-
- // Performance
- silent: false, // Suppress console output
- verbose: false, // Extra logging
-
- // Augmentations (auto-enabled by default)
- augmentations: {
- cache: true,
- display: true,
- metrics: true
- }
-})
-```
diff --git a/docs/api/README.md b/docs/api/README.md
index 3873474a..4ca84364 100644
--- a/docs/api/README.md
+++ b/docs/api/README.md
@@ -1,9 +1,22 @@
-# 🧠 Brainy v7.1.0 API Reference
+---
+title: API Reference
+slug: api/reference
+public: true
+category: api
+template: api
+order: 1
+description: Complete API reference for all Brainy methods — add, find, relate, update, delete, batch operations, the Db API (transactions, snapshots, time travel), VFS, neural API, and more.
+next:
+ - getting-started/quick-start
+ - guides/find-system
+---
-> **Complete API documentation for Brainy v7.1.0**
-> Zero Configuration • Triple Intelligence • Git-Style Branching • Entity Versioning • Candle WASM Embeddings
+# 🧠 Brainy API Reference
-**Updated:** 2026-01-06 for v7.1.0
+> **Complete API documentation for Brainy**
+> Zero Configuration • Triple Intelligence • Database as a Value • Atomic Transactions • Time Travel
+
+**Updated:** 2026-06-11
**All APIs verified against actual code**
---
@@ -13,32 +26,36 @@
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-const brain = new Brainy() // Zero config!
-await brain.init() // VFS auto-initialized in v5.1.0!
+const brain = new Brainy() // Zero config!
+await brain.init() // VFS auto-initialized!
// Add data (text auto-embeds!)
const id = await brain.add({
- data: 'The future of AI is here',
- type: NounType.Concept,
- metadata: { category: 'technology' }
+ data: 'The future of AI is here',
+ type: NounType.Concept,
+ metadata: { category: 'technology' }
})
// Search with Triple Intelligence
const results = await brain.find({
- query: 'artificial intelligence',
- where: { year: { greaterThan: 2020 } },
- connected: { from: id, depth: 2 }
+ query: 'artificial intelligence',
+ where: { year: { greaterThan: 2020 } },
+ connected: { from: id, depth: 2 }
})
-// Fork for safe experimentation (v5.0.0+)
-const experiment = await brain.fork('test-feature')
-await experiment.add({ data: 'test', type: NounType.Document })
-await experiment.commit({ message: 'Add test data' })
+// Pin the current state as an immutable value
+const db = brain.now()
-// Entity versioning (v5.3.0+)
-await brain.versions.save(id, { tag: 'v1.0', description: 'Initial version' })
-await brain.update(id, { category: 'AI' })
-await brain.versions.save(id, { tag: 'v2.0' })
+// Commit an atomic multi-write batch (all-or-nothing)
+await brain.transact([
+ { op: 'update', id, metadata: { category: 'AI' } }
+], { meta: { author: 'docs-example' } })
+
+await db.get(id) // still sees the pre-transaction state — snapshot isolation
+await db.release()
+
+// Time travel: query any past state
+const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000))
```
---
@@ -49,16 +66,19 @@ await brain.versions.save(id, { tag: 'v2.0' })
Semantic vectors with metadata and relationships - the fundamental data unit in Brainy.
### 🔗 Relationships (Verbs)
-Typed connections between entities - building knowledge graphs.
+Typed connections between entities with optional `data` and `metadata` - building knowledge graphs.
+
+### 📊 Data vs Metadata
+- **`data`**: Content embedded into vectors. Searchable via **semantic similarity** (vector index) and **hybrid text+semantic** search. NOT queryable via `where` filters.
+- **`metadata`**: Structured fields indexed by MetadataIndex. Queryable via `where` filters in `find()`.
+
+See **[Data Model](../DATA_MODEL.md)** for the full explanation.
### 🧠 Triple Intelligence
Vector search + Graph traversal + Metadata filtering in one unified query.
-### 🌳 Git-Style Branching (v5.0.0+)
-Fork, experiment, and commit - Snowflake-style copy-on-write isolation.
-
-### 📜 Entity Versioning (v5.3.0+)
-Time-travel and history tracking for individual entities - Git-like version control with content-addressable storage.
+### 🧊 Database Values (Db)
+The whole store pinned as an immutable value — snapshot isolation, atomic `transact()` batches, time travel with `asOf()`, instant hard-link snapshots with `persist()`. See the [consistency model](../concepts/consistency-model.md).
---
@@ -66,17 +86,17 @@ Time-travel and history tracking for individual entities - Git-like version cont
- [Core CRUD Operations](#core-crud-operations)
- [Search & Query](#search--query)
+- [Aggregation Engine](#aggregation-engine)
- [Relationships](#relationships)
- [Batch Operations](#batch-operations)
-- [Branch Management (v5.0+)](#branch-management-v50)
-- [Entity Versioning (v5.3.0+)](#entity-versioning-v530)
+- [Database Values & Time Travel (Db API)](#database-values--time-travel-db-api)
- [Virtual Filesystem (VFS)](#virtual-filesystem-vfs)
- [Neural API](#neural-api)
- [Import & Export](#import--export)
- [Configuration](#configuration)
- [Storage Adapters](#storage-adapters)
- [Utility Methods](#utility-methods)
-- [Embedding & Analysis APIs (v7.1.0)](#embedding--analysis-apis-v710)
+- [Embedding & Analysis APIs](#embedding--analysis-apis)
- [Type System Reference](#type-system-reference)
---
@@ -89,19 +109,30 @@ Add a single entity to the database.
```typescript
const id = await brain.add({
- data: 'JavaScript is a programming language', // Text or pre-computed vector
- type: NounType.Concept, // Required: Entity type
- metadata: { // Optional metadata
- category: 'programming',
- year: 1995
- }
+ data: 'JavaScript is a programming language', // Text or pre-computed vector
+ type: NounType.Concept, // Required: Entity type
+ subtype: 'language', // Optional: sub-classification
+ metadata: { // Optional: queryable fields
+ category: 'programming',
+ year: 1995
+ }
})
```
**Parameters:**
-- `data`: `string | number[]` - Text (auto-embeds) or vector
+- `data`: `string | number[]` - Content to embed (text auto-embeds) or pre-computed vector
- `type`: `NounType` - Entity type (required)
-- `metadata?`: `object` - Additional metadata
+- `subtype?`: `string` - Per-product sub-classification within the NounType (top-level standard field, indexed on the fast path). See [Subtypes & Facets](../guides/subtypes-and-facets.md).
+- `metadata?`: `object` - Structured queryable fields (indexed by MetadataIndex, used in `where` filters)
+- `id?`: `string` - Custom ID (auto-generated UUID if not provided)
+- `vector?`: `number[]` - Pre-computed vector (skips auto-embedding)
+- `confidence?`: `number` - Type classification confidence (0-1)
+- `weight?`: `number` - Entity importance/salience (0-1)
+- `ifAbsent?`: `boolean` - By-ID idempotent insert. When `true` AND a custom `id` is supplied AND an entity with that `id` already exists, returns the existing `id` without writing (no throw, no overwrite). Ignored without `id`. See [guides/optimistic-concurrency](../guides/optimistic-concurrency.md).
+
+> **`data`** is embedded into vectors for semantic search. **`metadata`** is indexed for `where` filters. See [Data Model](../DATA_MODEL.md).
+
+> **Strict-mode tip:** if a vocabulary is registered for your `type` (via `brain.requireSubtype()` or by an SDK that wraps Brainy), you must pass a matching `subtype`. Run `await brain.audit()` to inventory pre-existing gaps before enabling strict mode; see the [migration recipe](../guides/subtypes-and-facets.md#strict-mode-in-practice-for-sdk-style-vocabulary-consumers).
**Returns:** `Promise` - Entity ID
@@ -113,9 +144,9 @@ Retrieve a single entity by ID.
```typescript
const entity = await brain.get(id)
-console.log(entity?.data) // Original data
-console.log(entity?.metadata) // Metadata
-console.log(entity?.vector) // Embedding vector
+console.log(entity?.data) // Original data
+console.log(entity?.metadata) // Metadata
+console.log(entity?.vector) // Embedding vector
```
**Parameters:**
@@ -131,27 +162,35 @@ Update an existing entity.
```typescript
await brain.update({
- id: entityId,
- data: 'Updated content', // Optional: new data
- metadata: { updated: true } // Optional: new metadata (merges)
+ id: entityId,
+ data: 'Updated content', // Optional: new data
+ subtype: 'archived', // Optional: change sub-classification
+ metadata: { updated: true } // Optional: new metadata (merges)
})
```
**Parameters:**
- `id`: `string` - Entity ID
- `data?`: `string | number[]` - New data/vector
-- `metadata?`: `object` - Metadata to merge
+- `type?`: `NounType` - Change entity type
+- `subtype?`: `string` - Change subtype (omit to preserve existing)
+- `metadata?`: `object` - Metadata to merge (or replace with `merge: false`)
+- `confidence?`: `number` - Update classification confidence
+- `weight?`: `number` - Update entity importance
+- `ifRev?`: `number` - Optimistic-concurrency check. When provided, the update throws `RevisionConflictError` if the persisted entity's `_rev` no longer equals `ifRev`. See [guides/optimistic-concurrency](../guides/optimistic-concurrency.md).
**Returns:** `Promise`
+> **Tip — read-then-CAS.** Every entity returned by `get()` / `find()` / `search()` carries `entity._rev` (a monotonic counter Brainy auto-bumps on every successful `update()`). Pass it back as `ifRev` to make multi-writer coordination safe without an external lock service. Full guide: [guides/optimistic-concurrency](../guides/optimistic-concurrency.md).
+
---
-### `delete(id)` → `Promise`
+### `remove(id)` → `Promise`
-Delete a single entity.
+Remove a single entity (and every relationship where it is source or target).
```typescript
-await brain.delete(id)
+await brain.remove(id)
```
**Parameters:**
@@ -173,58 +212,411 @@ const results = await brain.find('machine learning')
// Advanced Triple Intelligence query
const results = await brain.find({
- query: 'artificial intelligence', // Vector similarity
- where: { // Metadata filtering
- year: { greaterThan: 2020 },
- category: { oneOf: ['AI', 'ML'] }
- },
- connected: { // Graph traversal
- to: conceptId,
- depth: 2,
- type: VerbType.RelatedTo
- },
- limit: 10
+ query: 'artificial intelligence', // Vector similarity
+ where: { // Metadata filtering
+ year: { greaterThan: 2020 },
+ category: { oneOf: ['AI', 'ML'] }
+ },
+ connected: { // Graph traversal
+ to: conceptId,
+ depth: 2,
+ type: VerbType.RelatedTo
+ },
+ limit: 10
})
```
**Parameters:**
- `query`: `string | FindParams`
- - **Simple:** Just text for vector search
- - **Advanced:** Object with vector + graph + metadata filters
+ - **Simple:** Just text for vector search
+ - **Advanced:** Object with vector + graph + metadata filters
**FindParams:**
-- `query?`: `string` - Text for vector similarity
-- `where?`: `object` - Metadata filters (see [Query Operators](#query-operators))
+- `query?`: `string` - Text for semantic + hybrid search (searches `data` via the vector index + text index)
+- `type?`: `NounType | NounType[]` - Filter by entity type(s). Alias for `where.noun`.
+- `subtype?`: `string | string[]` - Filter by sub-classification (top-level standard field, fast path). Single string for equality, array for set membership.
+- `where?`: `object` - Metadata filters. See **[Query Operators](../QUERY_OPERATORS.md)** for all operators.
- `connected?`: `object` - Graph traversal options
- - `to?`: `string` - Target entity ID
- - `from?`: `string` - Source entity ID
- - `type?`: `VerbType` - Relationship type
- - `depth?`: `number` - Traversal depth
+ - `to?`: `string` - Target entity ID
+ - `from?`: `string` - Source entity ID
+ - `via?`: `VerbType | VerbType[]` - Relationship type(s) to traverse
+ - `type?`: `VerbType | VerbType[]` - Alias for `via`
+ - `depth?`: `number` - Traversal depth (default: 1)
+ - `direction?`: `'in' | 'out' | 'both'` - Traversal direction (default: 'both')
- `limit?`: `number` - Max results (default: 10)
- `offset?`: `number` - Skip results
+- `orderBy?`: `string` - Field to sort by (e.g., 'createdAt', 'metadata.priority')
+- `order?`: `'asc' | 'desc'` - Sort direction (default: 'asc')
+- `searchMode?`: `'auto' | 'text' | 'semantic' | 'hybrid'` - Search strategy:
+ - `'auto'` (default): Zero-config hybrid combining text + semantic search
+ - `'text'`: Pure keyword/text matching
+ - `'semantic'`/`'vector'`: Pure vector similarity
+ - `'hybrid'`: Explicit hybrid mode
+- `hybridAlpha?`: `number` - Balance between text (0.0) and semantic (1.0) search. Auto-detected by query length if not specified.
+- `excludeVFS?`: `boolean` - Exclude VFS entities from results (default: false)
+
+> **`limit` tip:** Brainy caps `limit` against an auto-configured maximum (based on container/free memory, ~25 KB per result). Above the cap you get a one-time warning per call site; above 2× the cap it throws. To raise the cap, pass `new Brainy({ maxQueryLimit: N })` or `{ reservedQueryMemory: bytes }`. For queries that need ALL matches, paginate with `{ limit, offset }` — that's the only pattern guaranteed to keep working across Brainy versions. See [Query Limits & Pagination](../guides/find-limits.md).
**Returns:** `Promise` - Matching entities with scores
---
+### Hybrid Search
+
+Brainy automatically combines text (keyword) and semantic (vector) search for optimal results. No configuration needed.
+
+```typescript
+// Zero-config hybrid search (just works)
+const results = await brain.find({
+ query: 'David Smith' // Finds both exact text matches AND semantically similar
+})
+
+// Force text-only search (exact keyword matching)
+const textResults = await brain.find({
+ query: 'exact keyword',
+ searchMode: 'text'
+})
+
+// Force semantic-only search (vector similarity)
+const semanticResults = await brain.find({
+ query: 'artificial intelligence concepts',
+ searchMode: 'semantic'
+})
+
+// Custom hybrid weighting (0 = text only, 1 = semantic only)
+const customResults = await brain.find({
+ query: 'David Smith',
+ hybridAlpha: 0.3 // Favor text matching
+})
+```
+
+**How it works:**
+- Short queries (1-2 words) automatically favor text matching
+- Long queries (5+ words) automatically favor semantic search
+- Results are combined using Reciprocal Rank Fusion (RRF)
+
+---
+
+### Match Visibility
+
+Search results include detailed match information:
+
+```typescript
+const results = await brain.find({ query: 'david the warrior' })
+
+// Each result now includes:
+results[0].textMatches // ["david", "warrior"] - exact query words found
+results[0].textScore // 0.25 - text match quality (0-1)
+results[0].semanticScore // 0.87 - semantic similarity (0-1)
+results[0].matchSource // 'both' | 'text' | 'semantic'
+```
+
+**Use cases:**
+- Highlight exact matches in UI (textMatches)
+- Explain why a result ranked high (matchSource)
+- Debug search behavior (separate scores)
+
+---
+
+### `highlight(params)` → `Promise` ✨
+
+Zero-config highlighting for both exact matches AND semantic concepts.
+Handles plain text, rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill), HTML, and Markdown automatically.
+
+```typescript
+// Plain text (works as before)
+const highlights = await brain.highlight({
+ query: "david the warrior",
+ text: "David Smith is a brave fighter who battles dragons"
+})
+// [
+// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' },
+// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' },
+// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
+// ]
+
+// Rich-text JSON (auto-detected)
+const highlights = await brain.highlight({
+ query: "david the warrior",
+ text: JSON.stringify(tiptapDocument) // TipTap, Slate, Lexical, Draft.js, Quill
+})
+// Extracts text from nodes, annotates with contentCategory:
+// [
+// { text: "David", score: 1.0, matchType: 'text', contentCategory: 'title' },
+// { text: "fighter", score: 0.78, matchType: 'semantic', contentCategory: 'content' }
+// ]
+
+// HTML input (auto-detected)
+const highlights = await brain.highlight({
+ query: "warrior",
+ text: "David the Warrior
A brave fighter.
"
+})
+
+// Custom extractor for proprietary formats
+const highlights = await brain.highlight({
+ query: "function",
+ text: sourceCode,
+ contentExtractor: (text) => treeSitterParse(text) // Your custom parser
+})
+```
+
+**Parameters:**
+- `query`: `string` - The search query
+- `text`: `string` - Text to highlight (plain text, JSON, HTML, or Markdown)
+- `granularity?`: `'word' | 'phrase' | 'sentence'` - Highlight unit (default: 'word')
+- `threshold?`: `number` - Min similarity for semantic matches (default: 0.5)
+- `contentType?`: `ContentType` - Optional hint: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`. Skips auto-detection when provided.
+- `contentExtractor?`: `(text: string) => ExtractedSegment[]` - Custom parser. Bypasses built-in detection entirely.
+
+**Returns:** `Promise`
+- `text` - The matched text
+- `score` - Match score (1.0 for text matches, varies for semantic)
+- `position` - [start, end] indices in extracted text
+- `matchType` - `'text'` (exact) or `'semantic'` (concept)
+- `contentCategory?` - `'title' | 'annotation' | 'content' | 'value' | 'code' | 'structural'` — Role of the source text. Built-in extractors produce `'title'`, `'content'`, `'code'`. All 6 categories are available for custom parsers.
+
+**Supported Rich-Text Formats:**
+
+| Format | Detection | Text nodes |
+|--------|-----------|------------|
+| TipTap / ProseMirror | `{ type: 'doc', content: [...] }` | `{ type: 'text', text }` |
+| Slate.js | `[{ type, children }]` | `{ text }` |
+| Lexical | `{ root: { children } }` | `{ type: 'text', text }` |
+| Draft.js | `{ blocks: [{ text }] }` | `{ text }` in block |
+| Quill Delta | `{ ops: [{ insert }] }` | `{ insert }` |
+| HTML | Tags like ``, `
`, `` | Visible text content |
+| Markdown | `#` headings, ` ``` ` code blocks | Stripped markup |
+
+**Timeout Protection:**
+Semantic matching has a 10-second timeout. If embedding takes too long (e.g., WASM stall), `highlight()` returns text-only matches instead of hanging.
+
+**UI Pattern:**
+```typescript
+// Style differently based on match type and content category
+highlights.forEach(h => {
+ const style = h.matchType === 'text' ? 'font-weight: bold' : 'background: yellow'
+ if (h.contentCategory === 'title') { /* render as heading highlight */ }
+ if (h.contentCategory === 'code') { /* render with code styling */ }
+ if (h.contentCategory === 'annotation') { /* render as comment/caption */ }
+ // Apply style from h.position[0] to h.position[1]
+})
+```
+
+---
+
### Query Operators
-Brainy uses clean, readable operators:
+Brainy uses clean, readable operators (BFO — Brainy Field Operators):
| Operator | Description | Example |
|----------|-------------|---------|
-| `equals` | Exact match | `{age: {equals: 25}}` |
-| `greaterThan` | Greater than | `{age: {greaterThan: 18}}` |
-| `lessThan` | Less than | `{price: {lessThan: 100}}` |
-| `greaterEqual` | Greater or equal | `{score: {greaterEqual: 90}}` |
-| `lessEqual` | Less or equal | `{rating: {lessEqual: 3}}` |
-| `oneOf` | In array | `{color: {oneOf: ['red', 'blue']}}` |
-| `notOneOf` | Not in array | `{status: {notOneOf: ['deleted']}}` |
-| `contains` | Contains value | `{tags: {contains: 'ai'}}` |
+| `equals` / `eq` | Exact match | `{age: {equals: 25}}` |
+| `notEquals` / `ne` | Not equal | `{status: {notEquals: 'deleted'}}` |
+| `greaterThan` / `gt` | Greater than | `{age: {greaterThan: 18}}` |
+| `gte` / `greaterThanOrEqual` | Greater or equal | `{score: {gte: 90}}` |
+| `lessThan` / `lt` | Less than | `{price: {lessThan: 100}}` |
+| `lte` / `lessThanOrEqual` | Less or equal | `{rating: {lte: 3}}` |
+| `between` | Inclusive range | `{year: {between: [2020, 2025]}}` |
+| `oneOf` / `in` | In array | `{color: {oneOf: ['red', 'blue']}}` |
+| `noneOf` | Not in array | `{status: {noneOf: ['deleted']}}` |
+| `contains` | Array contains value | `{tags: {contains: 'ai'}}` |
+| `exists` / `missing` | Field existence | `{email: {exists: true}}` |
| `startsWith` | String prefix | `{name: {startsWith: 'John'}}` |
| `endsWith` | String suffix | `{email: {endsWith: '@gmail.com'}}` |
| `matches` | Pattern match | `{text: {matches: /^[A-Z]/}}` |
-| `between` | Range | `{year: {between: [2020, 2024]}}` |
+| `allOf` | AND combinator | `{allOf: [{active: true}, {role: 'admin'}]}` |
+| `anyOf` | OR combinator | `{anyOf: [{role: 'admin'}, {role: 'owner'}]}` |
+
+**[Complete Operator Reference →](../QUERY_OPERATORS.md)** — all operators, aliases, indexed vs in-memory support matrix, and practical examples.
+
+---
+
+## Aggregation Engine
+
+Brainy's aggregation engine maintains **incremental running totals** at write time, delivering O(1) aggregate reads regardless of dataset size. Define aggregates once, and every `add()`, `update()`, and `delete()` automatically updates the running metrics.
+
+### `defineAggregate(definition)` → `void`
+
+Register a named aggregate for incremental computation.
+
+```typescript
+brain.defineAggregate({
+ name: 'monthly_spending',
+ source: {
+ type: NounType.Event,
+ where: { domain: 'financial' } // matches custom metadata fields
+ },
+ groupBy: [
+ 'category',
+ { field: 'date', window: 'month' } // Time-windowed dimension
+ ],
+ metrics: {
+ total: { op: 'sum', field: 'amount' },
+ count: { op: 'count' },
+ average: { op: 'avg', field: 'amount' },
+ highest: { op: 'max', field: 'amount' },
+ lowest: { op: 'min', field: 'amount' },
+ spread: { op: 'stddev', field: 'amount' } // Welford's online algorithm
+ },
+ materialize: true // Optional: write results as NounType.Measurement entities
+})
+```
+
+**Parameters:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | `string` | Unique identifier for this aggregate |
+| `source.type` | `NounType \| NounType[]` | Entity types that feed into this aggregate |
+| `source.where` | `Record` | Filter on custom **metadata** fields (matched against the entity's `metadata` bag) |
+| `source.service` | `string` | Multi-tenancy filter |
+| `groupBy` | `GroupByDimension[]` | Dimensions to group by — plain field names, `{ field, window }` for time bucketing, or `{ field, unnest: true }` for array fields (one contribution per element) |
+| `metrics` | `Record` | Named metrics with `op` (`sum`, `count`, `avg`, `min`, `max`, `stddev`, `variance`, `percentile`, `distinctCount`) and optional `field`. `percentile` additionally requires `p` in `[0, 1]`. |
+| `materialize` | `boolean \| object` | Write results as `NounType.Measurement` entities (auto-visible in OData/Sheets/SSE) |
+
+**Time window granularities:** `'hour'`, `'day'`, `'week'`, `'month'`, `'quarter'`, `'year'`, or `{ seconds: number }` for custom intervals.
+
+### `removeAggregate(name)` → `void`
+
+Remove a named aggregate and clean up its state.
+
+```typescript
+brain.removeAggregate('monthly_spending')
+```
+
+### Querying Aggregates via `find()`
+
+Aggregate results are queried through the standard `find()` method using the `aggregate` parameter:
+
+```typescript
+// Simple: query by name
+const results = await brain.find({ aggregate: 'monthly_spending' })
+
+// With filtering on group keys
+const foodOnly = await brain.find({
+ aggregate: 'monthly_spending',
+ where: { category: 'food' }
+})
+
+// With sorting and pagination
+const topCategories = await brain.find({
+ aggregate: {
+ name: 'monthly_spending',
+ orderBy: 'total',
+ order: 'desc',
+ limit: 10
+ }
+})
+
+// Combine find-level params (where, orderBy, limit, offset merge automatically)
+const recentFood = await brain.find({
+ aggregate: 'monthly_spending',
+ where: { category: 'food' },
+ orderBy: 'total',
+ order: 'desc',
+ limit: 12
+})
+```
+
+**Result format:** Returns `Result[]` with `type: NounType.Measurement`. Each result contains:
+
+```typescript
+{
+ id: string, // Aggregate group ID (or materialized entity ID)
+ score: 1.0, // Always 1.0 for aggregates
+ type: NounType.Measurement,
+ metadata: {
+ __aggregate: 'monthly_spending', // Source aggregate name
+ category: 'food', // Group key values
+ date: '2024-01', // Time window bucket
+ total: 342.50, // Computed metrics
+ count: 28,
+ average: 12.23,
+ highest: 45.00,
+ lowest: 2.50
+ },
+ entity: Entity // Full entity structure
+}
+```
+
+### `queryAggregate(name, params?)` → `Promise`
+
+The first-class analytics path — returns plain group rows (`{ groupKey, metrics, count }`) instead of `find()`-style `Result` wrappers. Supports `where` (group-key filter), `having` (SQL-HAVING metric filter), `orderBy`, `order`, `limit`, `offset`:
+
+```typescript
+const rows = await brain.queryAggregate('monthly_spending', {
+ having: { total: { greaterThan: 100 } }, // filter by computed metrics
+ orderBy: 'total',
+ order: 'desc',
+ limit: 10
+})
+// [{ groupKey: { category: 'food', date: '2024-01' }, metrics: { total: 342.5, count: 28 }, count: 28 }, ...]
+```
+
+### How It Works
+
+Aggregation hooks run **outside transactions** on every write operation:
+
+- **`add()`**: If the new entity matches any aggregate's `source` filter, its values are added to the matching group's running totals.
+- **`update()`**: The old entity's contribution is reversed and the new entity's contribution is applied (handles group key changes, source filter changes).
+- **`delete()`**: The deleted entity's contribution is reversed from its group.
+
+**Performance:** O(A × G × M) per write where A = matching aggregates, G = groupBy dimensions, M = metrics. For typical configurations (2-5 aggregates, 1-3 dimensions, 3-5 metrics), this is effectively O(1) — measured at **10,000 entities in 13ms** in unit tests.
+
+**Infinite loop prevention:** Materialized `NounType.Measurement` entities (with `service: 'brainy:aggregation'` or `metadata.__aggregate`) are automatically excluded from all aggregate source matching.
+
+**Persistence:** Definitions and running state are persisted to storage on `flush()`/`close()` and reloaded on `init()`. Definition changes are detected via FNV-1a hashing — only changed aggregates reset their state.
+
+**Native acceleration:** Register an `'aggregation'` provider via the plugin system to replace the TypeScript engine with a custom native implementation for higher throughput at scale.
+
+### Financial Data Modeling
+
+Brainy supports financial analytics through **subtypes and metadata conventions** on existing NounTypes — no custom types needed:
+
+```typescript
+// Transaction = NounType.Event + 'transaction' subtype + financial metadata
+await brain.add({
+ data: 'Coffee at Blue Bottle',
+ type: NounType.Event,
+ subtype: 'transaction', // top-level standard field (reserved — never in metadata)
+ metadata: {
+ domain: 'financial',
+ amount: 5.50,
+ currency: 'USD',
+ category: 'food',
+ date: Date.now(),
+ merchant: 'Blue Bottle Coffee'
+ }
+})
+
+// Account = NounType.Collection + 'account' subtype + financial metadata
+await brain.add({
+ data: 'Checking Account',
+ type: NounType.Collection,
+ subtype: 'account',
+ metadata: {
+ domain: 'financial',
+ accountType: 'checking',
+ currency: 'USD',
+ institution: 'Chase'
+ }
+})
+
+// Invoice = NounType.Document + 'invoice' subtype + financial metadata
+await brain.add({
+ data: 'Invoice #1234 from Acme Corp',
+ type: NounType.Document,
+ subtype: 'invoice',
+ metadata: {
+ domain: 'financial',
+ amount: 15000,
+ currency: 'USD',
+ status: 'pending',
+ dueDate: Date.UTC(2024, 2, 15),
+ vendor: 'Acme Corp'
+ }
+})
+```
---
@@ -236,50 +628,107 @@ Create a typed relationship between entities.
```typescript
const relId = await brain.relate({
- from: sourceId,
- to: targetId,
- type: VerbType.RelatedTo,
- metadata: { // Optional
- strength: 0.9,
- confidence: 0.85
- }
+ from: sourceId,
+ to: targetId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct', // Optional: sub-classification
+ data: 'Collaborated on the research paper', // Optional: content for this edge
+ metadata: { // Optional: structured edge fields
+ strength: 0.9,
+ role: 'primary author'
+ }
})
```
**Parameters:**
-- `from`: `string` - Source entity ID
-- `to`: `string` - Target entity ID
+- `from`: `string` - Source entity ID (must exist)
+- `to`: `string` - Target entity ID (must exist)
- `type`: `VerbType` - Relationship type
-- `metadata?`: `object` - Optional metadata
+- `subtype?`: `string` - Per-product sub-classification within the VerbType (top-level standard field, fast-path indexed). See [Subtypes & Facets](../guides/subtypes-and-facets.md).
+- `data?`: `any` - Content for the relationship (overrides auto-computed vector)
+- `metadata?`: `object` - Structured edge fields
+- `weight?`: `number` - Connection strength (0-1, default: 1.0)
+- `bidirectional?`: `boolean` - Create reverse edge too (default: false)
+- `confidence?`: `number` - Relationship certainty (0-1)
+
+> **Strict-mode tip:** same as `add()` — if a vocabulary is registered for your `type`, pass a matching `subtype`. Run `await brain.audit()` first to surface pre-existing gaps.
**Returns:** `Promise` - Relationship ID
---
-### `getRelations(params)` → `Promise`
+### `updateRelation(params)` → `Promise`
-Get relationships for an entity.
+Update an existing relationship. Mirror of `update()` for verbs — closed a long-standing gap (verbs had no update path before 7.30).
+
+```typescript
+// Change the subtype on an existing relationship
+await brain.updateRelation({ id: relId, subtype: 'dotted-line' })
+
+// Update weight + confidence
+await brain.updateRelation({ id: relId, weight: 0.7, confidence: 0.9 })
+
+// Change verb type (re-indexes in graph adjacency, id preserved)
+await brain.updateRelation({ id: relId, type: VerbType.WorksWith })
+```
+
+**Parameters:**
+- `id`: `string` - Relationship ID (required)
+- `type?`: `VerbType` - Change verb type (re-indexes in graph adjacency)
+- `subtype?`: `string` - Change sub-classification (omit to preserve existing)
+- `weight?`: `number` - New weight (0-1)
+- `confidence?`: `number` - New confidence (0-1)
+- `data?`: `any` - New content
+- `metadata?`: `object` - Metadata to merge (or replace with `merge: false`)
+- `merge?`: `boolean` - Merge or replace metadata (default: true)
+
+**Returns:** `Promise`
+
+---
+
+### `related(params)` → `Promise`
+
+Get relationships for an entity. Same name and surface as `db.related()` on a
+pinned `Db` view.
```typescript
// Get all relationships FROM an entity
-const outgoing = await brain.getRelations({ from: entityId })
+const outgoing = await brain.related({ from: entityId })
// Get all relationships TO an entity
-const incoming = await brain.getRelations({ to: entityId })
+const incoming = await brain.related({ to: entityId })
// Filter by type
-const related = await brain.getRelations({
- from: entityId,
- type: VerbType.Contains
+const contains = await brain.related({
+ from: entityId,
+ type: VerbType.Contains
+})
+
+// Filter by subtype (fast path, column-store hit)
+const direct = await brain.related({
+ from: entityId,
+ type: VerbType.ReportsTo,
+ subtype: 'direct'
+})
+
+// Set membership on subtype
+const all = await brain.related({
+ from: entityId,
+ type: VerbType.ReportsTo,
+ subtype: ['direct', 'dotted-line']
})
```
**Parameters:**
- `from?`: `string` - Source entity ID
- `to?`: `string` - Target entity ID
-- `type?`: `VerbType` - Filter by relationship type
+- `type?`: `VerbType | VerbType[]` - Filter by relationship type
+- `subtype?`: `string | string[]` - Filter by VerbType subtype (top-level standard field, fast path)
+- `service?`: `string` - Multi-tenancy filter
+- `limit?`: `number` - Pagination limit (default: 100)
+- `offset?`: `number` - Pagination offset
-**Returns:** `Promise` - Matching relationships
+**Returns:** `Promise` - Matching relationships (each with `subtype` at top level when set)
---
@@ -291,27 +740,27 @@ Add multiple entities in one operation.
```typescript
const result = await brain.addMany({
- items: [
- { data: 'Entity 1', type: NounType.Document },
- { data: 'Entity 2', type: NounType.Concept }
- ]
+ items: [
+ { data: 'Entity 1', type: NounType.Document },
+ { data: 'Entity 2', type: NounType.Concept }
+ ]
})
-console.log(result.successful) // Array of IDs
-console.log(result.failed) // Array of errors
+console.log(result.successful) // Array of IDs
+console.log(result.failed) // Array of errors
```
**Returns:** `Promise>` - Success/failure results
---
-### `deleteMany(params)` → `Promise>`
+### `removeMany(params)` → `Promise>`
-Delete multiple entities.
+Remove multiple entities.
```typescript
-const result = await brain.deleteMany({
- ids: [id1, id2, id3]
+const result = await brain.removeMany({
+ ids: [id1, id2, id3]
})
```
@@ -323,10 +772,10 @@ Update multiple entities.
```typescript
const result = await brain.updateMany({
- updates: [
- { id: id1, metadata: { updated: true } },
- { id: id2, data: 'New content' }
- ]
+ items: [
+ { id: id1, metadata: { updated: true } },
+ { id: id2, data: 'New content' }
+ ]
})
```
@@ -338,617 +787,269 @@ Create multiple relationships.
```typescript
const ids = await brain.relateMany({
- relations: [
- { from: id1, to: id2, type: VerbType.RelatedTo },
- { from: id1, to: id3, type: VerbType.Contains }
- ]
+ items: [
+ { from: id1, to: id2, type: VerbType.RelatedTo },
+ { from: id1, to: id3, type: VerbType.Contains }
+ ]
})
```
---
-## Branch Management (v5.0+)
+## Database Values & Time Travel (Db API)
-**NEW in v5.0.0:** Git-style branching with Snowflake-style copy-on-write.
+Brainy 8.0's generational MVCC exposes the whole store as an immutable
+value: the **`Db`**. Pin the current state in O(1), commit atomic
+multi-write batches, query any past generation with the full query surface,
+cut instant snapshots, and ask what-if questions in memory. The exact
+guarantees live in the **[consistency model](../concepts/consistency-model.md)**;
+recipes live in **[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md)**.
-### `fork(branch?, options?)` → `Promise`
+### `generation()` → `number`
-Create an instant fork (<100ms) with full isolation.
+The store's current generation — a monotonic watermark advanced once per
+committed `transact()` batch and once per single-operation write. Never
+reissued, including across restarts and `restore()`.
```typescript
-// Create a fork
-const experiment = await brain.fork('test-feature')
-
-// Make changes safely in isolation
-await experiment.add({ data: 'Test entity', type: NounType.Document })
-await experiment.update({ id: someId, metadata: { modified: true } })
-
-// Parent is unaffected!
-const parentData = await brain.find({}) // Original data unchanged
-```
-
-**Parameters:**
-- `branch?`: `string` - Branch name (auto-generated if omitted)
-- `options?`: `object`
- - `description?`: `string` - Branch description
-
-**Returns:** `Promise` - New Brainy instance on forked branch
-
-**How it works:** Snowflake-style COW shares HNSW index, copies only modified nodes (10-20% memory overhead).
-
----
-
-### `checkout(branch)` → `Promise`
-
-Switch to a different branch.
-
-```typescript
-await brain.checkout('main')
-await brain.checkout('test-feature')
-```
-
-**Parameters:**
-- `branch`: `string` - Branch name
-
----
-
-### `listBranches()` → `Promise`
-
-List all branches.
-
-```typescript
-const branches = await brain.listBranches()
-// ['main', 'test-feature', 'experiment-2']
+const g = brain.generation()
```
---
-### `getCurrentBranch()` → `Promise`
+### `now()` → `Db`
-Get current branch name.
+Pin the current generation and return an immutable view — O(1), no I/O.
+The view keeps reading exactly this state no matter what commits afterwards.
```typescript
-const current = await brain.getCurrentBranch()
-// 'main'
+const db = brain.now()
+await brain.update({ id, metadata: { v: 2 } })
+
+await db.get(id) // still sees v: 1 — pinned
+await brain.get(id) // sees v: 2 — live
+await db.release() // unpin (enables history compaction)
+```
+
+**Returns:** `Db` — release it when done; pins gate `compactHistory()`.
+
+---
+
+### `transact(ops, options?)` → `Promise`
+
+Execute a declarative operation batch **atomically**: either every
+operation applies and the store advances exactly one generation, or none
+apply and the store is byte-identical to its pre-transaction state. The
+commit point is an atomic manifest rename; a crash anywhere before it rolls
+back to the exact pre-transaction bytes on the next open.
+
+```typescript
+const db = await brain.transact([
+ { op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' },
+ { op: 'update', id: customerId, metadata: { lastOrderAt: Date.now() }, ifRev: customer._rev },
+ { op: 'relate', from: customerId, to: orderId, type: VerbType.Creates, subtype: 'purchase' },
+ { op: 'remove', id: staleDraftId },
+ { op: 'unrelate', id: oldRelationId }
+], {
+ meta: { author: 'order-service', requestId: 'req-9f2' }, // reified, durable
+ ifAtGeneration: expectedGeneration // whole-store CAS
+})
+
+db.receipt.ids // resolved id per operation, in input order
+db.receipt.generation // the committed generation
+```
+
+**Operations** (`op` discriminates; parameters mirror the single-operation methods):
+- `{ op: 'add', ... }` — same parameters as `add()`; optional explicit `id`
+- `{ op: 'update', ... }` — same parameters as `update()`, including per-entity `ifRev` CAS
+- `{ op: 'remove', id }` — deletes the entity plus its relationships (same cascade as `delete()`)
+- `{ op: 'relate', ... }` — same parameters as `relate()`, including `bidirectional`; duplicates dedupe to the existing relationship id
+- `{ op: 'unrelate', id }` — deletes a relationship by id
+
+Operations may reference ids created earlier in the same batch.
+
+**Options:**
+- `meta?`: `Record` — transaction metadata, recorded durably in the transaction log (audit fields: author, reason, request id)
+- `ifAtGeneration?`: `number` — whole-store compare-and-swap; commits only if the store is still at this generation
+
+**Returns:** `Promise` — pinned at the freshly committed generation, carrying a `receipt`.
+
+**Throws:**
+- `GenerationConflictError` — `ifAtGeneration` did not match (nothing staged, generation unchanged)
+- `RevisionConflictError` — an `ifRev` did not match (whole batch rejected)
+
+---
+
+### `asOf(target)` → `Promise`
+
+Open an immutable view of **past** state:
+
+```typescript
+const atGen = await brain.asOf(1041) // generation number
+const lastWeek = await brain.asOf(new Date(Date.now() - 7 * 86_400_000)) // wall-clock
+const fromSnapshot = await brain.asOf('/backups/2026-06-01') // snapshot directory
+```
+
+- **`number`** — pins that generation; reads resolve through the immutable record layer.
+- **`Date`** — resolved via the transaction log to the newest generation committed at or before it.
+- **`string`** — a snapshot directory from `db.persist()`, opened as a self-contained read-only store (equivalent to `Brainy.load()`).
+
+Historical views serve the **full query surface**. Metadata-level reads are
+free; the first index-accelerated query (semantic search, traversal,
+cursors, aggregation) builds an in-memory index materialization — O(n at
+that generation), once per `Db`, freed on `release()`.
+
+**Throws:** `GenerationCompactedError` when the generation's records were reclaimed by `compactHistory()`.
+
+**History granularity:** every write is its own immutable generation —
+`transact()` batches AND single-operation writes — so a pin always freezes and
+every write is addressable via `asOf()`. See the
+[consistency model](../concepts/consistency-model.md).
+
+---
+
+### `transactionLog(options?)` → `Promise`
+
+Read the reified transaction log — one entry per committed generation (every
+`transact()` AND single-op write), newest first: `{ generation, timestamp,
+meta? }`. Single-op generations carry no `meta` (it is a `transact()`-only field).
+
+```typescript
+const [latest] = await brain.transactionLog({ limit: 1 })
+latest.meta // { author: 'order-service', requestId: 'req-9f2' }
```
---
-### `commit(options?)` → `Promise`
+### `compactHistory(options?)` → `Promise`
-Create a commit snapshot.
+Reclaim historical record-sets that no retention cap and no live `Db` pin
+protects. Pinned reads stay correct across compaction, always. (Auto-compaction
+on `flush()`/`close()` is governed by the constructor `retention` knob — unset →
+adaptive, `'all'` → unbounded, `{ … }` → explicit caps.)
```typescript
-const commitId = await brain.commit({
- message: 'Add new features',
- author: 'dev@example.com',
- metadata: { ticket: 'PROJ-123' }
+await brain.compactHistory({
+ maxGenerations: 100, // keep at most the 100 most recent generations
+ maxAge: 7 * 24 * 60 * 60 * 1000 // and only those from the last 7 days
})
```
-**Parameters:**
-- `message?`: `string` - Commit message
-- `author?`: `string` - Author email
-- `metadata?`: `object` - Additional commit metadata
-
-**Returns:** `Promise` - Commit ID
+**Returns:** `{ removedGenerations, horizon }` — `asOf()` below the horizon throws `GenerationCompactedError`.
---
+### `restore(path, { confirm: true })` → `Promise`
-### `deleteBranch(branch)` → `Promise`
-
-Delete a branch (cannot delete 'main').
+Replace the store's **entire** state from a snapshot directory. Destructive
+— requires `{ confirm: true }`. All indexes are rebuilt; the generation
+counter is floored so observed generation numbers are never reissued; live
+pins do not survive.
```typescript
-await brain.deleteBranch('old-experiment')
+await brain.restore('/backups/2026-06-01', { confirm: true })
```
---
-### `getHistory(options?)` → `Promise`
+### `Brainy.load(path)` → `Promise` (static)
-Get commit history.
+Open a persisted snapshot as a self-contained **read-only** store with the
+full query surface, including vector search. Releasing the returned `Db`
+closes the underlying instance.
```typescript
-const history = await brain.getHistory({
- branch: 'main',
- limit: 10
-})
+const db = await Brainy.load('/backups/2026-06-01')
+const hits = await db.find({ query: 'quarterly invoices' })
+await db.release()
```
---
-### `asOf(commitId, options?)` → `Promise`
+### The `Db` value
-Create a read-only snapshot at a specific commit for time-travel queries.
+Every `Db` is pinned at one generation and serves the full query surface at
+exactly that state.
+
+**Properties:**
+
+| Property | Type | Meaning |
+|---|---|---|
+| `generation` | `number` | The pinned generation |
+| `timestamp` | `number` | Pin time (`now()`), commit time (`transact()`), or resolved commit time (`asOf()`) |
+| `receipt` | `TransactReceipt?` | Present only on `transact()` results |
+| `speculative` | `boolean` | Whether this view carries a `with()` overlay |
+| `released` | `boolean` | Whether `release()` has been called |
+
+**Methods:**
```typescript
-// Get commit ID from history
-const commits = await brain.getHistory({ limit: 1 })
-const commitId = commits[0].id
-
-// Create snapshot (lazy-loading, no eager data loading)
-const snapshot = await brain.asOf(commitId, {
- cacheSize: 10000 // LRU cache size (default: 10000)
-})
-
-// Query historical state - full Triple Intelligence works!
-const results = await snapshot.find({
- query: 'AI research',
- where: { category: 'technology' }
-})
-
-// Get historical relationships
-const related = await snapshot.getRelated(entityId, { depth: 2 })
-
-// MUST close when done to free memory
-await snapshot.close()
+await db.get(id) // entity as of this generation
+await db.find({ where: { status: 'open' } }) // full find() surface
+await db.find({ query: 'unpaid invoices' }) // semantic search as of this generation
+await db.related(entityId) // relationships as of this generation
+await db.since(olderDb) // ids changed between two views
+const whatIf = await db.with(ops) // speculative in-memory overlay
+await db.persist('/backups/today') // self-contained hard-link snapshot
+await db.release() // unpin + free cached materialization
```
-**Parameters:**
-- `commitId`: `string` - Commit hash to snapshot from
-- `options?`: `object`
- - `cacheSize?`: `number` - LRU cache size for lazy-loading (default: 10000)
-
-**Returns:** `Promise` - Read-only Brainy instance with historical state
-
-**Features:**
-- **Lazy-Loading** - Loads entities on-demand, not eagerly
-- **Bounded Memory** - LRU cache prevents memory bloat
-- **Full Query Support** - All find(), getRelated(), etc. work on historical data
-- **Read-Only** - Prevents accidental modifications to history
-
-**Important:** Always call `snapshot.close()` when done to release resources.
-
----
-
-## Entity Versioning (v5.3.0+)
-
-**NEW in v5.3.0:** Git-style versioning for individual entities with content-addressable storage.
-
-### Overview
-
-Entity Versioning provides time-travel and history tracking for individual entities:
-
-- **Content-Addressable Storage** - Deduplication via SHA-256 hashing
-- **Zero-Config** - Lazy initialization, uses existing indexes
-- **Branch-Isolated** - Versions isolated per branch
-- **Selective Auto-Versioning** - Optional augmentation for automatic version creation
-- **Production-Scale** - Designed for billions of entities
-- **VFS File Support (v6.3.2+)** - Full versioning for VFS files with actual blob content
-
----
-
-### `versions.save(entityId, options?)` → `Promise`
-
-Save a new version of an entity.
-
-```typescript
-// Save version with tag
-const version = await brain.versions.save('user-123', {
- tag: 'v1.0',
- description: 'Initial user profile',
- metadata: { author: 'dev@example.com' }
-})
-
-console.log(version.version) // 1
-console.log(version.contentHash) // SHA-256 hash
-console.log(version.createdAt) // Timestamp
-```
-
-**Parameters:**
-- `entityId`: `string` - Entity ID to version
-- `options?`: `object`
- - `tag?`: `string` - Version tag (e.g., 'v1.0', 'beta')
- - `description?`: `string` - Version description
- - `metadata?`: `object` - Additional version metadata
-
-**Returns:** `Promise` - Created version
-
-**Features:**
-- Automatic deduplication (identical content = same version)
-- Sequential version numbering (1, 2, 3, ...)
-- Content-addressable storage (SHA-256)
-
----
-
-### `versions.list(entityId, options?)` → `Promise`
-
-List all versions of an entity.
-
-```typescript
-const versions = await brain.versions.list('user-123', {
- limit: 10,
- offset: 0
-})
-
-versions.forEach(v => {
- console.log(`Version ${v.version}: ${v.tag} - ${v.description}`)
-})
-```
-
-**Parameters:**
-- `entityId`: `string` - Entity ID
-- `options?`: `object`
- - `limit?`: `number` - Max versions to return
- - `offset?`: `number` - Skip versions
-
-**Returns:** `Promise` - Versions (newest first)
-
----
-
-### `versions.restore(entityId, versionOrTag)` → `Promise`
-
-Restore entity to a previous version.
-
-```typescript
-// Restore by version number
-await brain.versions.restore('user-123', 1)
-
-// Restore by tag
-await brain.versions.restore('user-123', 'beta')
-```
-
-**Parameters:**
-- `entityId`: `string` - Entity ID
-- `versionOrTag`: `number | string` - Version number or tag
-
----
-
-### `versions.compare(entityId, version1, version2)` → `Promise`
-
-Compare two versions.
-
-```typescript
-const diff = await brain.versions.compare('user-123', 1, 2)
-
-console.log(diff.totalChanges) // Total changes
-console.log(diff.modified) // Modified fields
-console.log(diff.added) // Added fields
-console.log(diff.removed) // Removed fields
-
-// Check specific changes
-const nameChange = diff.modified.find(c => c.path === 'metadata.name')
-console.log(`${nameChange.oldValue} → ${nameChange.newValue}`)
-```
-
-**Returns:** `Promise` - Detailed diff with field-level changes
-
----
-
-### `versions.getContent(entityId, versionOrTag)` → `Promise`
-
-Get version content without restoring.
-
-```typescript
-// View old version without changing current state
-const v1Content = await brain.versions.getContent('user-123', 1)
-console.log(v1Content.metadata.name) // Old name
-
-// Current state unchanged
-const current = await brain.get('user-123')
-console.log(current.metadata.name) // Current name
-```
-
----
-
-### `versions.undo(entityId)` → `Promise`
-
-Undo to previous version (shorthand for restore to latest-1).
-
-```typescript
-// Make a bad change
-await brain.update('user-123', { status: 'deleted' })
-
-// Undo immediately
-await brain.versions.undo('user-123')
-```
-
-**Alias:** `versions.revert(entityId)`
-
----
-
-### `versions.prune(entityId, options)` → `Promise`
-
-Clean up old versions.
-
-```typescript
-const result = await brain.versions.prune('user-123', {
- keepRecent: 10, // Keep 10 most recent
- keepTagged: true, // Always keep tagged versions
- olderThan: Date.now() - 30 * 24 * 60 * 60 * 1000 // Older than 30 days
-})
-
-console.log(`Deleted ${result.deleted}, kept ${result.kept}`)
-```
-
-**Parameters:**
-- `keepRecent?`: `number` - Keep N most recent versions
-- `keepTagged?`: `boolean` - Always keep tagged versions (default: true)
-- `olderThan?`: `number` - Only prune versions older than timestamp
-
----
-
-### `versions.getLatest(entityId)` → `Promise`
-
-Get latest version.
-
-```typescript
-const latest = await brain.versions.getLatest('user-123')
-if (latest) {
- console.log(`Latest: v${latest.version} (${latest.tag})`)
-}
-```
-
----
-
-### `versions.getVersionByTag(entityId, tag)` → `Promise`
-
-Get version by tag.
-
-```typescript
-const beta = await brain.versions.getVersionByTag('user-123', 'beta')
-```
-
----
-
-### `versions.count(entityId)` → `Promise`
-
-Count versions for an entity.
-
-```typescript
-const count = await brain.versions.count('user-123')
-console.log(`${count} versions saved`)
-```
-
----
-
-### `versions.hasVersions(entityId)` → `Promise`
-
-Check if entity has versions.
-
-```typescript
-if (await brain.versions.hasVersions('user-123')) {
- console.log('Entity has version history')
-}
-```
-
----
-
-### Auto-Versioning Augmentation
-
-Automatically create versions on entity updates.
-
-```typescript
-import { VersioningAugmentation } from '@soulcraft/brainy'
-
-// Configure auto-versioning
-const versioning = new VersioningAugmentation({
- enabled: true,
- onUpdate: true, // Version on update()
- onDelete: false, // Don't version on delete
- entities: ['user-*'], // Only version users
- excludeEntities: ['temp-*'],
- excludeTypes: ['temporary'],
- keepRecent: 50, // Auto-prune old versions
- keepTagged: true
-})
-
-// Apply augmentation
-brain.augment(versioning)
-
-// Now updates auto-create versions
-await brain.update('user-123', { name: 'New Name' })
-
-// Version automatically created!
-const versions = await brain.versions.list('user-123')
-console.log(`Auto-created version: ${versions[0].version}`)
-```
-
-**Configuration:**
-- `enabled`: `boolean` - Enable/disable augmentation
-- `onUpdate`: `boolean` - Version on entity updates
-- `onDelete`: `boolean` - Version before deletion
-- `entities`: `string[]` - Entity ID patterns (glob-style)
-- `excludeEntities`: `string[]` - Exclusion patterns
-- `types`: `string[]` - Entity types to version
-- `excludeTypes`: `string[]` - Types to exclude
-- `keepRecent`: `number` - Auto-prune to keep N versions
-- `keepTagged`: `boolean` - Always keep tagged versions
-
-**Pattern Matching:**
-- `['*']` - All entities
-- `['user-*']` - All IDs starting with "user-"
-- `['*-prod']` - All IDs ending with "-prod"
-- `['user-*', 'account-*']` - Multiple patterns
-
----
-
-### Branch Isolation
-
-Versions are isolated per branch.
-
-```typescript
-// Save version on main
-await brain.versions.save('doc-1', { tag: 'main-v1' })
-
-// Fork and create version
-const feature = await brain.fork('feature')
-await feature.update('doc-1', { content: 'Feature update' })
-await feature.versions.save('doc-1', { tag: 'feature-v1' })
-
-// Versions are isolated
-const mainVersions = await brain.versions.list('doc-1')
-const featureVersions = await feature.versions.list('doc-1')
-
-console.log(mainVersions.length !== featureVersions.length) // true
-```
-
----
-
-### Architecture
-
-**Content-Addressable Storage:**
-- SHA-256 hashing for deduplication
-- Identical content = single storage blob
-- Efficient for entities with few changes
-
-**Metadata Indexing:**
-- Leverages existing MetadataIndexManager
-- Fast lookups by entity ID
-- Version number indexing
-
-**Storage Structure:**
-```
-_version:{entityId}:{versionNum}:{branch} // Version metadata
-_version_blob:{contentHash} // Content blob (deduplicated)
-```
-
-**Performance:**
-- Version save: O(1) if duplicate, O(log N) for index update
-- Version list: O(K) where K = version count
-- Version restore: O(log N) lookup + O(1) restore
-- Pruning: O(K) where K = versions pruned
-
----
-
-### Examples
-
-#### Basic Versioning Workflow
-
-```typescript
-// Create entity
-await brain.add({
- data: 'User profile',
- id: 'user-123',
- type: 'user',
- metadata: { name: 'Alice', email: 'alice@example.com' }
-})
-
-// Save v1
-await brain.versions.save('user-123', { tag: 'v1.0' })
-
-// Make changes
-await brain.update('user-123', { name: 'Alice Smith' })
-
-// Save v2
-await brain.versions.save('user-123', { tag: 'v2.0' })
-
-// Compare versions
-const diff = await brain.versions.compare('user-123', 1, 2)
-
-// Restore to v1 if needed
-await brain.versions.restore('user-123', 'v1.0')
-```
-
-#### Release Management
-
-```typescript
-// Development workflow
-await brain.update('app-config', { version: '1.0.0-alpha' })
-await brain.versions.save('app-config', { tag: 'alpha' })
-
-await brain.update('app-config', { version: '1.0.0-beta' })
-await brain.versions.save('app-config', { tag: 'beta' })
-
-await brain.update('app-config', { version: '1.0.0' })
-await brain.versions.save('app-config', { tag: 'release' })
-
-// Rollback to beta if issues found
-await brain.versions.restore('app-config', 'beta')
-```
-
-#### Audit Trail
-
-```typescript
-// Track all changes
-const versioning = new VersioningAugmentation({
- enabled: true,
- onUpdate: true,
- entities: ['audit-*'],
- keepRecent: 100 // Keep 100 versions for audit
-})
-
-brain.augment(versioning)
-
-// All updates now tracked
-await brain.update('audit-record-1', { status: 'modified' })
-await brain.update('audit-record-1', { status: 'approved' })
-
-// View complete history
-const versions = await brain.versions.list('audit-record-1')
-versions.forEach(v => {
- console.log(`${v.createdAt}: ${v.description}`)
-})
-```
-
-#### VFS File Versioning (v6.3.2+)
-
-```typescript
-// VFS files can be versioned with actual blob content
-await brain.vfs.writeFile('/docs/readme.md', 'Version 1 content')
-
-// Get the file's entity ID
-const stat = await brain.vfs.stat('/docs/readme.md')
-
-// Save version 1
-await brain.versions.save(stat.entityId, { tag: 'v1', description: 'Initial draft' })
-
-// Modify the file
-await brain.vfs.writeFile('/docs/readme.md', 'Version 2 - updated content')
-
-// Save version 2
-await brain.versions.save(stat.entityId, { tag: 'v2', description: 'Updated docs' })
-
-// Compare versions - content is DIFFERENT (fixed in v6.3.2)
-const v1 = await brain.versions.getContent(stat.entityId, 1)
-const v2 = await brain.versions.getContent(stat.entityId, 2)
-console.log(v1.data !== v2.data) // true
-
-// Restore to v1 - writes content back to blob storage
-await brain.versions.restore(stat.entityId, 'v1')
-
-// File is now back to v1
-const content = await brain.vfs.readFile('/docs/readme.md')
-console.log(content.toString()) // 'Version 1 content'
-```
-
----
-
-**[📖 Complete Versioning Guide →](../features/entity-versioning.md)**
+- **`with(ops)`** — applies `transact()`-style operations **in memory** on
+ top of the view; nothing touches disk, the generation counter, or index
+ providers. Overlay entities carry no embeddings, so index-accelerated
+ queries and `persist()` on overlays throw `SpeculativeOverlayError`;
+ `get()`, metadata-filter `find()`, and filter-based `related()` work
+ fully. Commit the same ops with `transact()` for the full surface.
+- **`persist(path)`** — cuts an instant snapshot (hard links on filesystem
+ storage; byte copies across devices; in-memory stores serialize to the
+ same layout). Requires the view to still be the store's latest generation
+ — otherwise `GenerationConflictError`.
+- **`release()`** — idempotent; after release every read throws. A
+ `FinalizationRegistry` backstop releases leaked pins at GC, but explicit
+ release is what makes `compactHistory()` deterministic.
+
+### Db API errors
+
+All exported from `@soulcraft/brainy`:
+
+| Error | Thrown by | Meaning |
+|---|---|---|
+| `GenerationConflictError` | `transact({ ifAtGeneration })`, `db.persist()` | The store moved past the expected generation — re-read and retry |
+| `RevisionConflictError` | `update({ ifRev })`, `transact()` update ops | Per-entity revision moved — see [optimistic concurrency](../guides/optimistic-concurrency.md) |
+| `GenerationCompactedError` | `asOf()` | The requested generation's records were reclaimed — persist what you must keep |
+| `SpeculativeOverlayError` | index-accelerated reads / `persist()` on `with()` overlays | Honest boundary: overlay entities carry no embeddings |
---
## Virtual Filesystem (VFS)
-**Auto-initialized in v5.1.0!** Access via `brain.vfs` (property, not method).
+Access via `brain.vfs` (property, not method). Auto-initialized during `brain.init()`.
### Filtering VFS Entities
-**NEW in v5.3.0:** All VFS entities (files/folders) have `metadata.isVFSEntity: true` set automatically.
+All VFS entities (files/folders) have `metadata.isVFSEntity: true` set automatically.
Use this to filter VFS entities from semantic search results:
```typescript
// Exclude VFS entities from semantic search
const semanticOnly = await brain.find({
- query: 'artificial intelligence',
- where: {
- isVFSEntity: { notEquals: true } // Only semantic entities
- }
+ query: 'artificial intelligence',
+ where: {
+ isVFSEntity: { notEquals: true } // Only semantic entities
+ }
})
// Or filter to ONLY VFS entities
const vfsOnly = await brain.find({
- where: {
- isVFSEntity: { equals: true } // Only VFS files/folders
- }
+ where: {
+ isVFSEntity: { equals: true } // Only VFS files/folders
+ }
})
// Check if an entity is a VFS entity
if (entity.metadata.isVFSEntity === true) {
- console.log('This is a VFS file or folder')
+ console.log('This is a VFS file or folder')
}
```
@@ -975,7 +1076,7 @@ Write file content.
```typescript
await brain.vfs.writeFile('/docs/README.md', 'New content', {
- encoding: 'utf-8'
+ encoding: 'utf-8'
})
```
@@ -1013,7 +1114,7 @@ const files = await brain.vfs.readdir('/projects')
// With file types
const entries = await brain.vfs.readdir('/projects', { withFileTypes: true })
entries.forEach(entry => {
- console.log(entry.name, entry.isDirectory() ? 'DIR' : 'FILE')
+ console.log(entry.name, entry.isDirectory() ? 'DIR' : 'FILE')
})
```
@@ -1035,9 +1136,9 @@ Get file/directory stats.
```typescript
const stats = await brain.vfs.stat('/docs/README.md')
-console.log(stats.size) // File size
-console.log(stats.mtime) // Modified time
-console.log(stats.isDirectory()) // Is directory?
+console.log(stats.size) // File size
+console.log(stats.mtime) // Modified time
+console.log(stats.isDirectory()) // Is directory?
```
---
@@ -1050,8 +1151,8 @@ Semantic file search.
```typescript
const results = await brain.vfs.search('React components with hooks', {
- path: '/src',
- limit: 10
+ path: '/src',
+ limit: 10
})
```
@@ -1063,8 +1164,8 @@ Find similar files.
```typescript
const similar = await brain.vfs.findSimilar('/src/App.tsx', {
- limit: 5,
- threshold: 0.7
+ limit: 5,
+ threshold: 0.7
})
```
@@ -1078,7 +1179,7 @@ Get directory tree (prevents infinite recursion).
```typescript
const tree = await brain.vfs.getTreeStructure('/projects', {
- maxDepth: 3
+ maxDepth: 3
})
```
@@ -1090,7 +1191,7 @@ Get all descendants with optional filtering.
```typescript
const files = await brain.vfs.getDescendants('/src', {
- filter: (entity) => entity.name.endsWith('.tsx')
+ filter: (entity) => entity.name.endsWith('.tsx')
})
```
@@ -1104,8 +1205,8 @@ Get file metadata.
```typescript
const meta = await brain.vfs.getMetadata('/src/App.tsx')
-console.log(meta.todos) // Extracted TODOs
-console.log(meta.tags) // Tags
+console.log(meta.todos) // Extracted TODOs
+console.log(meta.tags) // Tags
```
---
@@ -1131,39 +1232,16 @@ const todos = await brain.vfs.getTodos('/src/App.tsx')
---
-#### `vfs.getAllTodos(path?)` → `Promise`
+#### `vfs.searchEntities(query)` → `Promise>`
-Get all TODOs from directory tree.
+Search for semantic entities tracked by the VFS, filtered by type, name, or metadata.
```typescript
-const allTodos = await brain.vfs.getAllTodos('/src')
-```
-
----
-
-### Project Analysis
-
-#### `vfs.getProjectStats(path?)` → `Promise`
-
-Get project statistics.
-
-```typescript
-const stats = await brain.vfs.getProjectStats('/projects/my-app')
-console.log(stats.fileCount)
-console.log(stats.totalSize)
-console.log(stats.fileTypes) // Breakdown by extension
-```
-
----
-
-#### `vfs.searchEntities(query)` → `Promise`
-
-Search for VFS entities by metadata.
-
-```typescript
-const tsxFiles = await brain.vfs.searchEntities({
- type: 'file',
- extension: '.tsx'
+const people = await brain.vfs.searchEntities({
+ type: 'person', // entity type filter
+ name: 'Ada', // semantic name search
+ where: { role: 'author' }, // metadata filters
+ limit: 50
})
```
@@ -1173,119 +1251,6 @@ const tsxFiles = await brain.vfs.searchEntities({
---
-## Neural API
-
-Access advanced AI features via `brain.neural()` (method that returns NeuralAPI instance).
-
-### `neural().similar(a, b, options?)` → `Promise`
-
-Calculate semantic similarity.
-
-```typescript
-// Simple similarity score
-const score = await brain.neural().similar(
- 'renewable energy',
- 'sustainable power'
-) // 0.87
-
-// Detailed result
-const result = await brain.neural().similar('text1', 'text2', {
- detailed: true
-})
-console.log(result.score)
-console.log(result.explanation)
-```
-
----
-
-### `neural().clusters(input?, options?)` → `Promise`
-
-Automatic clustering.
-
-```typescript
-const clusters = await brain.neural().clusters({
- algorithm: 'kmeans',
- k: 5,
- minSize: 3
-})
-
-clusters.forEach(cluster => {
- console.log(cluster.label)
- console.log(cluster.items)
- console.log(cluster.centroid)
-})
-```
-
----
-
-### `neural().neighbors(id, options?)` → `Promise`
-
-Find k-nearest neighbors.
-
-```typescript
-const neighbors = await brain.neural().neighbors(entityId, {
- k: 10,
- threshold: 0.7
-})
-```
-
----
-
-### `neural().outliers(threshold?)` → `Promise`
-
-Detect outlier entities.
-
-```typescript
-const outliers = await brain.neural().outliers(0.3)
-// Returns entity IDs that are outliers
-```
-
----
-
-### `neural().visualize(options?)` → `Promise`
-
-Generate visualization data.
-
-```typescript
-const vizData = await brain.neural().visualize({
- maxNodes: 100,
- dimensions: 3,
- algorithm: 'force',
- includeEdges: true
-})
-// Use with D3.js, Cytoscape, GraphML tools
-```
-
----
-
-### Performance Methods
-
-#### `neural().clusterFast(options)` → `Promise`
-
-Fast clustering for large datasets.
-
-```typescript
-const clusters = await brain.neural().clusterFast({
- k: 10,
- maxIterations: 50
-})
-```
-
----
-
-#### `neural().clusterLarge(options)` → `Promise`
-
-Streaming clustering for very large datasets.
-
-```typescript
-const clusters = await brain.neural().clusterLarge({
- k: 20,
- batchSize: 1000
-})
-```
-
----
-
## Import & Export
### `import(source, options?)` → `Promise`
@@ -1295,21 +1260,19 @@ Smart import with auto-detection (CSV, Excel, PDF, JSON, URLs).
```typescript
// CSV import
await brain.import('data.csv', {
- format: 'csv',
- createEntities: true
+ format: 'csv',
+ createEntities: true
})
-// Excel import
+// Excel import (all sheets processed automatically)
await brain.import('sales.xlsx', {
- format: 'excel',
- sheets: ['Q1', 'Q2']
+ format: 'excel',
+ vfsPath: '/imports/sales', // optional: mirror into the VFS
+ groupBy: 'sheet'
})
-// PDF import
-await brain.import('research.pdf', {
- format: 'pdf',
- extractTables: true
-})
+// PDF import (tables extracted automatically)
+await brain.import('research.pdf', { format: 'pdf' })
// URL import
await brain.import('https://api.example.com/data.json')
@@ -1318,31 +1281,61 @@ await brain.import('https://api.example.com/data.json')
**Parameters:**
- `source`: `string | Buffer | object` - File path, URL, buffer, or object
- `options?`: Import configuration
- - `format?`: `'csv' | 'excel' | 'pdf' | 'json'` - Auto-detected if omitted
- - `createEntities?`: `boolean` - Create entities from rows
- - `sheets?`: `string[]` - Excel sheets to import
- - `extractTables?`: `boolean` - Extract tables from PDF
+ - `format?`: `'excel' | 'pdf' | 'csv' | 'json' | 'markdown' | 'yaml' | 'docx' | 'image'` - Auto-detected if omitted
+ - `vfsPath?`: `string` - Mirror imported content into the VFS at this path
+ - `groupBy?`: `'type' | 'sheet' | 'flat' | 'custom'` - VFS grouping strategy
+ - `createEntities?`: `boolean` - Create entities from rows
+ - `createRelationships?`: `boolean` - Create relationships between extracted entities
+ - `preserveSource?`: `boolean` - Save the original file in the VFS
+ - `enableNeuralExtraction?`: `boolean` - Extract entity names via AI
+ - `enableRelationshipInference?`: `boolean` - Infer relationships via AI
+ - `enableConceptExtraction?`: `boolean` - Extract entity types via AI
+ - `confidenceThreshold?`: `number` - Minimum confidence for extracted entities
+ - `onProgress?`: `(progress) => void` - Progress callback (stage, counts, throughput, ETA)
**Returns:** `Promise` - Import statistics
-**Note:** Import always uses the current branch (v5.1.0 verified).
-
**[📖 Complete Import Guide →](../guides/import-anything.md)**
---
-### Export & Snapshots
+### Export & Import (portable) + Snapshots (native)
+
+**Portable graph export/import** — `brain.export()` / `brain.import()` (`PortableGraph` v1, versioned
+JSON, partial-or-whole, cross-version). `export()` lives on the immutable `Db`, so it composes
+with `now()`/`asOf()`/`with()`:
```typescript
-// Export to file
-await brain.export('/path/to/backup.brainy')
+// Export part or all of the brain to a portable, versioned document
+const graph = await brain.export({ ids }, { includeVectors: true })
-// Create instant snapshot using COW fork
-await brain.fork('backup-2025-01-19')
+// Restore it — import() routes a PortableGraph to the graph round-trip (merge by id)
+await otherBrain.import(graph, { onConflict: 'merge' })
-// Time-travel to specific commit
-const snapshot = await brain.asOf(commitId)
+// Time-travel export (serialize a past generation) / what-if export (a speculative state)
+const past = await brain.asOf(gen)
+const asWas = await past.export({ collection: id })
+await past.release()
+await brain.now().with(ops).export({ ids })
+```
+
+Selectors: `{ ids }`, `{ collection }` (alias `memberOf`), `{ connected: { from, depth } }`,
+`{ vfsPath }`, predicate (`{ type, subtype, where, service }`), or whole brain (omit). See the
+**[Export & Import guide](../guides/export-and-import.md)**. Distinct from `brain.import(file)`
+(CSV/PDF/Excel/JSON ingestion — `import()` dispatches on whether you pass a `PortableGraph` or a file).
+
+**Native whole-brain snapshot** (generation-preserving, not portable JSON):
+
+```typescript
+// Instant hard-link snapshot via the Db API
+const pin = brain.now()
+await pin.persist('/backups/2026-06-11')
+await pin.release()
+
+// Time-travel to a past generation or timestamp
+const snapshot = await brain.asOf(new Date('2026-06-01'))
const entities = await snapshot.find({ limit: 100 })
+await snapshot.release()
```
---
@@ -1355,47 +1348,37 @@ const entities = await snapshot.find({ limit: 100 })
const brain = new Brainy({
// Storage configuration
storage: {
- type: 'memory', // memory | opfs | filesystem | s3 | r2 | gcs | azure
- path: './brainy-data', // For filesystem storage
- compression: true, // Enable gzip compression (60-80% savings)
-
- // Cloud storage configs (see Storage Adapters section)
- s3Storage: { ... },
- r2Storage: { ... },
- gcsStorage: { ... },
- azureStorage: { ... }
+ type: 'filesystem', // 'memory' | 'filesystem' | 'auto'
+ path: './brainy-data'
},
- // HNSW vector index config
- hnsw: {
- M: 16, // Connections per layer
- efConstruction: 200, // Construction quality
- efSearch: 100, // Search quality
- typeAware: true // Enable type-aware indexing (v4.0+)
+ // Vector index configuration (2 knobs)
+ vector: {
+ recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
+ persistMode: 'immediate' // 'immediate' | 'deferred'
},
// Model configuration (embedded in WASM - zero config needed)
- // Model: all-MiniLM-L6-v2 (384 dimensions)
- // Device: CPU via WASM (works everywhere)
+ // Model: all-MiniLM-L6-v2 (384 dimensions)
+ // Device: CPU via WASM (works everywhere)
- // Cache configuration
+ // Cache configuration — `true`/`false`, or an options object
cache: {
- enabled: true,
maxSize: 10000,
- ttl: 3600000 // 1 hour in ms
+ ttl: 3600000 // 1 hour in ms
}
})
-await brain.init() // Required! VFS auto-initialized in v5.1.0
+await brain.init() // Required! VFS auto-initialized
```
---
## Storage Adapters
-All 7 storage adapters support **copy-on-write branching** (v5.0+).
+Brainy 8.0 ships two adapters — both support the full Db API (generational history, snapshots, restore).
-### Memory (Default)
+### Memory (Default for Tests)
```typescript
const brain = new Brainy({
@@ -1407,125 +1390,32 @@ const brain = new Brainy({
---
-### OPFS (Browser)
-
-```typescript
-const brain = new Brainy({
- storage: { type: 'opfs' }
-})
-```
-
-**Use case:** Browser applications with persistent storage
-
----
-
-### Filesystem (Node.js)
+### Filesystem (Default for Node)
```typescript
const brain = new Brainy({
storage: {
type: 'filesystem',
- path: './brainy-data',
- compression: true // 60-80% space savings
+ path: './brainy-data'
}
})
```
-**Use case:** Node.js applications, local persistence
+**Use case:** Node.js applications, single-node production deployments
+
+For off-site backup, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`) — Brainy itself doesn't reach out to cloud object stores.
---
-### AWS S3
+### Auto
```typescript
const brain = new Brainy({
- storage: {
- type: 's3',
- s3Storage: {
- bucketName: 'my-brainy-data',
- region: 'us-east-1',
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
- }
- }
-})
-
-// Enable Intelligent-Tiering for 96% cost savings
-await brain.storage.enableIntelligentTiering('entities/', 'auto-tier')
-```
-
-**Use case:** Production deployments, scalable storage
-
-**[📖 AWS S3 Cost Optimization →](../operations/cost-optimization-aws-s3.md)**
-
----
-
-### Cloudflare R2
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'r2',
- r2Storage: {
- accountId: process.env.CF_ACCOUNT_ID,
- bucketName: 'my-brainy-data',
- accessKeyId: process.env.CF_ACCESS_KEY_ID,
- secretAccessKey: process.env.CF_SECRET_ACCESS_KEY
- }
- }
+ storage: { type: 'auto', path: './brainy-data' }
})
```
-**Use case:** Zero egress fees, cost-effective storage
-
-**[📖 R2 Cost Optimization →](../operations/cost-optimization-cloudflare-r2.md)**
-
----
-
-### Google Cloud Storage (GCS)
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'gcs',
- gcsStorage: {
- bucketName: 'my-brainy-data',
- projectId: process.env.GCP_PROJECT_ID,
- keyFilename: './gcp-key.json'
- }
- }
-})
-
-// Enable auto-tiering
-await brain.storage.enableAutoclass({
- terminalStorageClass: 'ARCHIVE'
-})
-```
-
-**Use case:** Google Cloud ecosystem, global distribution
-
-**[📖 GCS Cost Optimization →](../operations/cost-optimization-gcs.md)**
-
----
-
-### Azure Blob Storage
-
-```typescript
-const brain = new Brainy({
- storage: {
- type: 'azure',
- azureStorage: {
- accountName: process.env.AZURE_STORAGE_ACCOUNT,
- accountKey: process.env.AZURE_STORAGE_KEY,
- containerName: 'brainy-data'
- }
- }
-})
-```
-
-**Use case:** Azure ecosystem, enterprise deployments
-
-**[📖 Azure Cost Optimization →](../operations/cost-optimization-azure.md)**
+Picks `'filesystem'` on Node with a writable `path`, falls back to `'memory'` otherwise.
---
@@ -1561,7 +1451,205 @@ const count = await brain.getVerbCount()
---
-### `embed(data)` → `Promise` ✨ *Enhanced v7.1.0*
+### Subtype & facet APIs
+
+Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
+
+#### `counts.bySubtype(type, subtype?)` → `Record | number`
+
+O(1) subtype counts for a NounType (backed by the persisted rollup).
+
+```typescript
+brain.counts.bySubtype(NounType.Person)
+// → { employee: 12, customer: 847, vendor: 34 }
+
+brain.counts.bySubtype(NounType.Person, 'employee')
+// → 12
+```
+
+#### `counts.topSubtypes(type, n=10)` → `Array<[subtype, count]>`
+
+Top N subtypes ranked by count.
+
+```typescript
+brain.counts.topSubtypes(NounType.Person, 3)
+// → [['customer', 847], ['employee', 12], ['vendor', 34]]
+```
+
+#### `subtypesOf(type)` → `string[]`
+
+Sorted distinct subtypes seen for a NounType.
+
+```typescript
+brain.subtypesOf(NounType.Person)
+// → ['customer', 'employee', 'vendor']
+```
+
+#### `counts.byRelationshipSubtype(verb, subtype?)` → `Record | number`
+
+Verb-side mirror of `counts.bySubtype`. O(1) per-VerbType-per-subtype counts.
+
+```typescript
+brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
+// → { direct: 12, 'dotted-line': 3 }
+
+brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')
+// → 12
+```
+
+#### `counts.topRelationshipSubtypes(verb, n=10)` → `Array<[subtype, count]>`
+
+Top N subtypes for a `VerbType` ranked by count.
+
+```typescript
+brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
+// → [['direct', 12], ['dotted-line', 3]]
+```
+
+#### `relationshipSubtypesOf(verb)` → `string[]`
+
+Sorted distinct subtypes seen for a `VerbType`.
+
+```typescript
+brain.relationshipSubtypesOf(VerbType.ReportsTo)
+// → ['direct', 'dotted-line']
+```
+
+#### `audit(options?)` → `Promise` (7.30.1+)
+
+Diagnostic — find entities and relationships missing a `subtype` value, grouped by type. The companion to `migrateField()` / `fillSubtypes()` — answers "what would break if I enabled strict subtype enforcement?".
+
+```typescript
+const report = await brain.audit()
+// {
+// entitiesWithoutSubtype: { event: 24, document: 3 },
+// relationshipsWithoutSubtype: { relatedTo: 1402 },
+// total: 1429,
+// scanned: 8400,
+// recommendation: 'Found 1429 entries without subtype. ...'
+// }
+```
+
+**Parameters:**
+- `options.includeVFS?`: `boolean` — When `false` (default), VFS infrastructure entities (`metadata.isVFSEntity` / `metadata.isVFS`) are excluded. They bypass enforcement anyway, so counting them is noise.
+- `options.batchSize?`: `number` — Pagination batch size (default 200).
+- `options.onProgress?`: `(progress: { scanned, missingSubtype }) => void` — Progress callback per batch.
+
+Run before adopting an SDK that registers `requireSubtype()` rules, or before upgrading to Brainy 8.0 (which makes strict mode the default). See the [Strict mode in practice](../guides/subtypes-and-facets.md#strict-mode-in-practice-for-sdk-style-vocabulary-consumers) guide for the full migration recipe.
+
+#### `requireSubtype(type, options?)` → `void`
+
+Register subtype enforcement for a specific `NounType` or `VerbType`. Unified API for nouns and verbs. Composes with the brain-wide `requireSubtype` constructor flag.
+
+```typescript
+// Lock down Person sub-classification
+brain.requireSubtype(NounType.Person, {
+ values: ['employee', 'customer', 'vendor'],
+ required: true
+})
+
+// Lock down management edges
+brain.requireSubtype(VerbType.ReportsTo, {
+ values: ['direct', 'dotted-line'],
+ required: true
+})
+```
+
+**Parameters:**
+- `type`: `NounType | VerbType` - The type to register
+- `options.values?`: `string[]` - Vocabulary whitelist (rejects off-vocab values)
+- `options.required?`: `boolean` - Whether subtype is required (default: `true`)
+
+#### Brain-wide strict mode — `new Brainy({ requireSubtype })`
+
+Constructor option that enforces subtype on every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()` for every type:
+
+```typescript
+// Every write must include subtype
+const brain = new Brainy({ requireSubtype: true })
+
+// Exempt specific types (e.g. catch-all Thing)
+const brain2 = new Brainy({
+ requireSubtype: { except: [NounType.Thing, NounType.Custom] }
+})
+```
+
+When strict mode is on:
+- Every public write path checks the pairing guarantee.
+- `addMany()` / `relateMany()` validate all items BEFORE any storage write — atomic-fail, no partial writes.
+- Brainy's own VFS infrastructure writes bypass via the `metadata.isVFSEntity: true` marker.
+- Per-type registrations always apply regardless of the brain-wide flag.
+
+The default since 8.0.0 — pass `requireSubtype: false` to opt out while migrating pre-8.0 data.
+
+#### `trackField(name, options?)` → `void`
+
+Register a metadata field for cardinality + per-NounType breakdown stats. With `values: [...]`, validates against the whitelist on `add()`/`update()`.
+
+```typescript
+brain.trackField('status') // basic
+brain.trackField('status', { perType: true }) // with per-NounType breakdown
+brain.trackField('priority', { values: ['low', 'med', 'high'] }) // strict vocabulary
+```
+
+#### `counts.byField(name, options?)` → `Promise>`
+
+Counts by value for a tracked field. Requires `perType: true` registration if filtering by NounType.
+
+```typescript
+await brain.counts.byField('status')
+// → { todo: 12, doing: 3, done: 47 }
+
+await brain.counts.byField('status', { type: NounType.Task })
+// → { todo: 8, doing: 2, done: 30 }
+```
+
+#### `migrateField(options)` → `Promise`
+
+Stream-and-rewrite a field across the brain. Supports `metadata.X`, `data.X`, and top-level paths. Idempotent.
+
+```typescript
+// One-shot rewrite
+await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
+
+// Deprecation window — keep source field readable
+await brain.migrateField({ from: 'data.kind', to: 'subtype', readBoth: true })
+
+// With progress reporting
+await brain.migrateField({
+ from: 'metadata.kind',
+ to: 'subtype',
+ batchSize: 500,
+ onProgress: ({ scanned, migrated }) => console.log(`${scanned} / ${migrated}`)
+})
+```
+
+Returns `{ scanned: number, migrated: number, skipped: number, errors: Array<{id, error}> }`.
+
+#### `fillSubtypes(rules, options?)` → `Promise` (8.0+)
+
+Back-fill missing `subtype` values across entities AND relationships in one streaming pass — the migration companion to `audit()`. Keys are `NounType`/`VerbType` values; each rule is a literal subtype string or a function deriving one from the entry (return `undefined` to decline). Idempotent: entries that already carry a subtype are never touched, so a crashed run is resumed safely by re-running.
+
+```typescript
+const report = await brain.fillSubtypes({
+ [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified', // derived
+ [NounType.Document]: 'general', // literal default
+ [VerbType.RelatedTo]: 'unspecified' // relationship rule
+})
+// → { scanned, filled, skipped, errors, byType }
+```
+
+**Parameters:**
+- `rules`: `FillSubtypeRules` - Map of NounType/VerbType → literal subtype or `(entry) => string | undefined`
+- `options.includeVFS?`: `boolean` - Also fill VFS infrastructure entries (default `false`)
+- `options.batchSize?`: `number` - Pagination batch size (default `200`)
+- `options.onProgress?`: `(progress: { scanned, filled, skipped }) => void` - Per-batch callback
+
+Returns `{ scanned, filled, skipped, errors, byType }`. After a clean run, `skipped` equals the remaining `audit().total`. See the [migration recipe](../guides/subtypes-and-facets.md).
+
+---
+
+### `embed(data)` → `Promise` ✨
Generate embedding vector from text or data.
@@ -1573,30 +1661,32 @@ console.log(vector.length) // 384
---
-### `embedBatch(texts)` → `Promise` ✨ *New v7.1.0*
+### `embedBatch(texts)` → `Promise` ✨
-Batch embed multiple texts efficiently.
+Batch embed multiple texts using native WASM batch API (single forward pass).
```typescript
const embeddings = await brain.embedBatch([
- 'Machine learning is fascinating',
- 'Deep neural networks',
- 'Natural language processing'
+ 'Machine learning is fascinating',
+ 'Deep neural networks',
+ 'Natural language processing'
])
-console.log(embeddings.length) // 3
+console.log(embeddings.length) // 3
console.log(embeddings[0].length) // 384
```
+> Uses the WASM engine's native `embed_batch()` for a single model forward pass instead of N individual calls. This is the same batch API used internally by `highlight()`.
+
---
-### `similarity(textA, textB)` → `Promise` ✨ *New v7.1.0*
+### `similarity(textA, textB)` → `Promise` ✨
Calculate semantic similarity between two texts.
```typescript
const score = await brain.similarity(
- 'The cat sat on the mat',
- 'A feline was resting on the rug'
+ 'The cat sat on the mat',
+ 'A feline was resting on the rug'
)
console.log(score) // ~0.85 (high semantic similarity)
```
@@ -1605,7 +1695,7 @@ console.log(score) // ~0.85 (high semantic similarity)
---
-### `neighbors(entityId, options?)` → `Promise