.claude/skills/architecture.md

@@ -1,170 +0,0 @@
-# Brainy Architecture Reference
-
-## What Is Brainy
-
-@soulcraft/brainy (v7.17.0) is a Universal Knowledge Protocol -- a Triple Intelligence database combining vector search, graph traversal, and metadata filtering in a single library. Published to npm as a public MIT-licensed package.
-
-## Core Architecture
-
-### Storage Layer (`src/storage/`)
-- **StorageAdapter interface** (`src/coreTypes.ts:576`): The contract ALL storage backends implement. ALWAYS check this interface before adding storage methods.
-- **BaseStorage** (`src/storage/baseStorage.ts`): Base implementation with built-in type-aware partitioning (TypeAwareStorageAdapter was removed -- functionality merged into BaseStorage).
-- **Adapters** (`src/storage/adapters/`):
-  - `fileSystemStorage.ts` -- local filesystem
-  - `memoryStorage.ts` -- in-memory
-  - `baseStorageAdapter.ts` -- shared adapter base (counts, batch ops)
-  - Cloud + OPFS adapters were removed in 8.0 (cloud backup is operator tooling)
-- **Generational MVCC / Db API** (`src/db/`): immutable `Db` values over generation-stamped records
-  - `db.ts` (the `Db` value), `generationStore.ts` (record layer + commit protocol), `types.ts`, `errors.ts`, `whereMatcher.ts`
-  - Design record: `docs/ADR-001-generational-mvcc.md`; replaced the pre-8.0 COW branching + versioning subsystems
-
-### Vector Search (`src/hnsw/`)
-- `hnswIndex.ts` -- HNSW-based approximate nearest neighbor search
-- `typeAwareHNSWIndex.ts` -- type-partitioned vector search
-- NOT in `src/intelligence/` (that directory does not exist)
-
-### Graph Engine (`src/graph/`)
-- `graphAdjacencyIndex.ts` -- adjacency-based graph representation
-- `pathfinding.ts` -- relationship traversal and pathfinding
-- `lsm/` -- LSM tree implementation for graph storage
-
-### Metadata Index (`src/utils/metadataIndex.ts`)
-- O(1) exact match via hash indexes
-- O(log n) range queries via sorted indexes
-- Roaring bitmap set operations for efficient filtering
-- Adaptive chunking strategy (`metadataIndexChunking.ts`)
-- Caching layer (`metadataIndexCache.ts`)
-
-### Triple Intelligence (`src/triple/`)
-- `TripleIntelligenceSystem.ts` -- combines vector + graph + metadata into unified queries
-- Lazy-loaded indexes (loaded on first use, not at startup)
-
-### Neural/AI Components (`src/neural/`)
-- Smart Importers (`src/importers/`): CSV, Excel, PDF, DOCX, YAML, JSON, Markdown, Orchestrator
-- `SmartExtractor.ts` -- entity extraction from unstructured data
-- `SmartRelationshipExtractor.ts` -- relationship detection
-- `NeuralEntityExtractor.ts` -- ML-based entity recognition
-- Natural language processing utilities
-
-### Distributed Systems (`src/distributed/`)
-- Distributed Coordinator for multi-node operation
-- Shard Manager for data partitioning
-- Cache Synchronization across nodes
-- Read/Write separation
-- Network and HTTP transport layers
-- Storage discovery and shard migration
-
-### Transaction Management (`src/transaction/`)
-- TransactionManager for ACID operations
-- Operations: SaveNoun, AddToHNSW, UpdateMetadata, etc.
-- Distributed transaction support
-
-### Integration Hub (`src/integrations/`)
-- Google Sheets integration
-- OData (Open Data Protocol)
-- Server-Sent Events (SSE)
-- Webhooks
-- Event bus system
-
-### Virtual Filesystem (`src/vfs/`)
-- `VirtualFileSystem.ts` -- full VFS implementation (87 KB)
-- `PathResolver.ts`, `FSCompat.ts`, `MimeTypeDetector.ts`, `TreeUtils.ts`
-- Subdirectories: `semantic/` (semantic search), `streams/` (streaming), `importers/`
-
-### MCP Support (`src/mcp/`)
-- BrainyMCPAdapter, MCPAugmentationToolset, BrainyMCPService
-- Model Control Protocol request/response handling
-
-### Aggregation Engine (`src/aggregation/`)
-- **AggregationIndex** (`AggregationIndex.ts`): Write-time incremental aggregation — SUM, COUNT, AVG, MIN, MAX with GROUP BY and time windows
-- **Time Windows** (`timeWindows.ts`): ISO 8601 bucketing — hour, day, week, month, quarter, year, custom intervals
-- **Materializer** (`materializer.ts`): Debounced writes of aggregate results as `NounType.Measurement` entities
-- Integrates into `brain.find({ aggregate })` for unified query API
-- Write hooks in `add()`, `update()`, `delete()` for O(1) incremental updates
-- `'aggregation'` provider key enables native plugin acceleration
-
-### Additional Systems
-- **CLI** (`src/cli/`): Complete command-line tool with interactive mode and catalog system
-- **Migration** (`src/migration/`): MigrationRunner for database schema migrations
-- **Embeddings** (`src/embeddings/`): Embedding manager with Candle-WASM Rust source
-- **Streaming** (`src/streaming/`): Pipeline support with adaptive backpressure
-- **Versioning** (`src/versioning/`): VersioningAPI for data versioning
-- **Plugin System**: Registry-based plugin architecture
-- **Patterns** (`src/patterns/`): 7 pattern library JSON files
-
-## Type System
-- **NounType** (42 types, `src/types/graphTypes.ts:850-893`): Person, Organization, Concept, Collection, Document, Task, Project, etc.
-- **VerbType** (127 types, `src/types/graphTypes.ts:900-1087`): Contains, RelatedTo, PartOf, Creates, DependsOn, MemberOf, etc.
-- All types in `src/types/`
-
-## Module Exports (`src/index.ts`)
-38+ named exports including: Brainy class, configuration types, neural APIs (NeuralImport, NeuralEntityExtractor, SmartExtractor, SmartRelationshipExtractor), distance functions, plugin system, migration system, embedding functions, storage adapters, COW infrastructure, pipeline utilities, graph types, MCP components, integration hub, OData utilities, and more.
-
-## File Structure
-```
-src/
-├── index.ts              # 38+ public exports
-├── brainy.ts             # Main Brainy class (6,500+ lines)
-├── setup.ts              # Initialization polyfills
-├── coreTypes.ts          # StorageAdapter interface + core types
-├── storage/
-│   ├── baseStorage.ts    # Base storage (includes type-aware)
-│   ├── adapters/         # All storage backends + cloud adapters
-│   └── cow/              # Copy-on-Write versioning
-├── hnsw/                 # HNSW vector search
-├── graph/                # Graph engine + pathfinding + LSM
-├── triple/               # Triple Intelligence system
-├── neural/               # Smart extractors + NLP
-├── importers/            # File format importers (8 types)
-├── distributed/          # Distributed database (16 files)
-├── transaction/          # ACID transactions (6 files)
-├── integrations/         # Sheets, OData, SSE, Webhooks
-├── vfs/                  # Virtual filesystem + semantic search
-├── mcp/                  # Model Control Protocol
-├── cli/                  # Command-line interface
-├── migration/            # Schema migrations
-├── embeddings/           # Embedding manager + Candle-WASM
-├── streaming/            # Pipeline + backpressure
-├── versioning/           # Versioning API
-├── types/                # TypeScript type definitions
-├── utils/                # Metadata index, logging, etc.
-├── config/               # Configuration system
-├── patterns/             # Pattern library
-├── api/                  # API layer
-├── interfaces/           # Interface definitions
-├── shared/               # Shared utilities
-├── data/                 # Data utilities
-├── errors/               # Error handling
-├── critical/             # Critical error handling
-├── universal/            # Universal utilities
-├── import/               # Import functionality
-└── scripts/              # Build scripts
-```
-
-## Initialization
-`brainy.ts` `init()` method performs initialization cascade:
-1. Load plugins
-2. Initialize storage
-3. Enable COW (Copy-on-Write)
-4. Set up embeddings
-5. Initialize caches
-6. Set up graph indexes
-7. Initialize VFS
-8. Set up transaction manager
-9. Initialize distributed components (if enabled)
-
-## Testing
-- Framework: Vitest
-- Run: `npm test`
-- Test directories:
-  - `tests/unit/` -- unit tests
-  - `tests/integration/` -- integration tests
-  - `tests/benchmarks/` -- performance benchmarks (NOT tests/performance/)
-  - `tests/comprehensive/` -- comprehensive test suites
-  - `tests/api/` -- API tests
-  - `tests/helpers/` -- test utilities
-
-## Release
-- `npm run release:patch/minor/major` -- fully automated via `scripts/release.sh`
-- `npm run release:dry` -- preview without changes
-- Uses conventional commits for changelog generation

.github/workflows/ci.yml

@@ -1,40 +0,0 @@
-name: CI
-
-on:
-  push:
-  pull_request:
-
-jobs:
-  node:
-    name: Node ${{ matrix.node-version }}
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        node-version: ['22', '24']
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-        with:
-          node-version: ${{ matrix.node-version }}
-          cache: npm
-      - run: npm ci
-      - run: npm run test:unit
-
-  bun:
-    name: Bun (latest)
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-        with:
-          node-version: '22'
-          cache: npm
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: latest
-      - run: npm ci
-      # test:bun imports the built dist/, so build first.
-      - run: npm run build
-      # Bun as a runtime is the supported Bun story (`bun add` / `bun run`).
-      - run: npm run test:bun

.gitignore

@@ -31,9 +31,6 @@ coverage/
 # Test results
 tests/results/
 
-# Filesystem test artifacts (created by integration tests)
-test-*/
-
 # IDE files
 .vscode/
 .idea/
@@ -52,19 +49,21 @@ temp/
 
 # Planning and instruction files
 plan.md
+CLAUDE.md
 
 # Package files
 *.tgz
 
 # Private/confidential files
 PLAN.md
+CLAUDE.md
 INTERNAL_NOTES.md
 TODO_PRIVATE.md
 *.tar.gz
 
 # Strategy and planning documents (private)
 .strategy/
-# Removed: PRODUCTION_*.md (now these should be public documentation)
+PRODUCTION_*.md
 DISTRIBUTED_*.md
 *_ASSESSMENT.md
 *_ANALYSIS.md
@@ -74,11 +73,9 @@ DISTRIBUTED_*.md
 models/
 models-cache/
 
-# But include bundled WASM model assets
-!assets/models/
-
 # Development planning files (not for commit)
 PLAN.md
+CLAUDE.md
 
 # Backup folders
 backup-*
@@ -90,27 +87,9 @@ docs/internal/
 # Cache files
 *.cache
 
-# Rust/Cargo build artifacts
-src/embeddings/candle-wasm/target/
-src/embeddings/candle-wasm/Cargo.lock
-
-# Ignore the wasm-pack output dir's CONTENTS (note the `/*`, not `/`, so the
-# re-includes below can take effect — git cannot re-include a file whose parent
-# DIRECTORY is excluded). Keep the pre-built WASM committed: it ships in the npm
-# package anyway, it lets consumers + CI build without a Rust/wasm-pack toolchain,
-# and versioning it makes the shipped artifact reproducible (not "whatever the
-# maintainer last built").
-src/embeddings/wasm/pkg/*
-!src/embeddings/wasm/pkg/*.wasm
-!src/embeddings/wasm/pkg/*.js
-!src/embeddings/wasm/pkg/*.d.ts
-
 # Log files (redundant but explicit)
 *.log
 
 # Temporary files (redundant but explicit)
 *.tmp
 /.junie/guidelines.md
-
-# Claude Code harness state
-.claude/scheduled_tasks.lock

.versionrc.json

@@ -0,0 +1,24 @@
+{
+  "types": [
+    {"type": "feat", "section": "✨ Features"},
+    {"type": "fix", "section": "🐛 Bug Fixes"},
+    {"type": "docs", "section": "📚 Documentation"},
+    {"type": "refactor", "section": "♻️ Code Refactoring"},
+    {"type": "perf", "section": "⚡ Performance Improvements"},
+    {"type": "test", "section": "✅ Tests"},
+    {"type": "build", "section": "🔧 Build System"},
+    {"type": "ci", "section": "🔄 CI/CD"},
+    {"type": "style", "hidden": true},
+    {"type": "chore", "hidden": true}
+  ],
+  "compareUrlFormat": "https://github.com/soulcraftlabs/brainy/compare/{{previousTag}}...{{currentTag}}",
+  "commitUrlFormat": "https://github.com/soulcraftlabs/brainy/commit/{{hash}}",
+  "issueUrlFormat": "https://github.com/soulcraftlabs/brainy/issues/{{id}}",
+  "userUrlFormat": "https://github.com/{{user}}",
+  "releaseCommitMessageFormat": "chore(release): {{currentTag}}",
+  "issuePrefixes": ["#"],
+  "header": "# Changelog\n\nAll notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.\n",
+  "scripts": {
+    "postbump": "echo '✅ Version bumped to' $(node -p \"require('./package.json').version\")"
+  }
+}

CLAUDE.md

@@ -1,208 +0,0 @@
-# Brainy - Claude Code Project Guide
-
-This file provides guidance for Claude Code (and human contributors) when working on the Brainy codebase.
-
-## Cross-Project Coordination
-
-Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
-
-**At session START:** Read the handoff. Find rows where Owner = Brainy. Act on those first.
-
-**At session END:** Mark completed actions ✅, delete rows you finished, delete threads with zero remaining actions. File must not grow. **If you shipped anything consumers need to know about, update `RELEASES.md` before closing.**
-
-**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
-
-**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`)
-
----
-
-## Project Overview
-
-Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraft/brainy` on npm under the MIT license.
-
-## Getting Started
-
-```bash
-npm install           # Install dependencies
-npm run build         # Build the project
-npm test              # Run test suite (Vitest)
-```
-
-## Architecture
-
-Full architecture reference: `.claude/skills/architecture.md`
-
-### Core Systems
-- **Storage** (`src/storage/`): Pluggable storage backends via StorageAdapter interface (`src/coreTypes.ts`)
-- **Vector Search** (`src/hnsw/`): HNSW approximate nearest neighbor search
-- **Graph Engine** (`src/graph/`): Relationship traversal with adjacency index and pathfinding
-- **Metadata Index** (`src/utils/metadataIndex.ts`): O(1) exact match, O(log n) range queries
-- **Triple Intelligence** (`src/triple/`): Unified query combining all three intelligence types
-- **Aggregation Engine** (`src/aggregation/`): Write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows
-- **Virtual Filesystem** (`src/vfs/`): Full VFS with semantic search
-
-### Type System
-- **NounType** (42 types): Entity classification -- Person, Concept, Collection, Document, Task, etc.
-- **VerbType** (127 types): Relationship types -- Contains, RelatedTo, PartOf, Creates, DependsOn, etc.
-- Defined in `src/types/graphTypes.ts`
-
-## Code Standards
-
-### TypeScript
-- Strict mode enabled
-- Target: ES2020, NodeNext module resolution
-- All new code must be TypeScript
-- Follow existing patterns -- read related code before writing
-
-### Quality
-- All code must compile without errors
-- All code must have working tests that exercise real behavior
-- No stub returns (`return {} as any`)
-- No incomplete implementations with TODO comments
-- If something can't be fully implemented, throw an explicit error rather than faking it
-
-### Verification Before Code Changes
-1. Check that interfaces and methods actually exist before using them
-2. Check that type properties are in the type definitions
-3. Run `npm test` -- tests must pass
-4. Run `npm run build` -- build must succeed
-
-### Testing
-- Framework: Vitest
-- Tests in `tests/` (unit, integration, benchmarks, comprehensive)
-- Use in-memory storage for speed where possible
-- Tests must exercise real behavior, not mock it
-- Benchmarks are in `tests/benchmarks/` (not tests/performance/)
-
-## Commit Conventions
-
-Use [Conventional Commits](https://www.conventionalcommits.org/):
-
-```
-feat: add new feature        (minor version bump)
-fix: resolve bug             (patch version bump)
-docs: update documentation   (patch version bump)
-perf: improve performance    (patch version bump)
-refactor: restructure code   (patch version bump)
-test: add/update tests       (patch version bump)
-```
-
-**Important:** Never use `BREAKING CHANGE` in commit messages. Major version bumps are manual decisions only (`npm run release:major`).
-
-## Docs Pipeline — soulcraft.com/docs
-
-Docs in `docs/**/*.md` are published with the npm package (included in `files`) and synced to soulcraft.com/docs on every portal deploy. Frontmatter controls what appears publicly.
-
-### Docs check triggers
-
-Run the docs check whenever the user says ANY of:
-- "commit, publish, release" / "release" / "publish"
-- "update the docs" / "make sure docs are accurate" / "check the docs"
-- "review docs" / "clean up docs"
-
-### Pre-release docs check (MANDATORY before every release)
-
-When the user says "commit, publish, release" or any variation, **before committing**:
-
-1. **Scan all files changed in this session** (and any recently added `docs/*.md` files)
-2. For each changed/new doc, decide: is this useful to external users?
-   - **Yes** → ensure it has complete frontmatter (add or update it)
-   - **No** (internal, migration, dev-only) → ensure it has no frontmatter or `public: false`
-3. For docs that already have frontmatter, verify:
-   - `description` still matches the actual content
-   - `next` links still exist and are still the right follow-up pages
-   - `title` matches the doc's h1
-4. Include frontmatter changes in the commit
-
-### Frontmatter format
-
-```yaml
----
-title: Human-readable title
-slug: category/page-name          # URL: soulcraft.com/docs/category/page-name
-public: true                      # false or absent = not published
-category: getting-started | concepts | guides | api
-template: guide | concept | api   # controls layout on soulcraft.com
-order: 1                          # sidebar position within category (lower = first)
-description: One sentence. What this doc covers and why it matters.
-next:                             # "Next steps" links shown at bottom of page
-  - category/other-slug
----
-```
-
-### Category guide
-
-| category | use for |
-|----------|---------|
-| `getting-started` | installation, quick start, first steps |
-| `concepts` | how the system works, mental models |
-| `guides` | how to do specific things, recipes |
-| `api` | method reference, signatures, parameters |
-
-### What stays internal (no frontmatter / `public: false`)
-
-- Release guides, developer learning paths
-- Migration guides for old versions (v3→v4, v5.11)
-- Architecture analysis docs (clustering algorithms, etc.)
-- Anything in `docs/internal/`
-- Deployment/ops/cost docs (cloud-run, kubernetes, cost-optimization)
-
-## Release Process
-
-Fully automated via `scripts/release.sh`:
-
-```bash
-npm run release:dry          # Preview (no changes)
-npm run release:patch        # Bug fixes
-npm run release:minor        # New features
-npm run release:major        # Breaking changes (rare, manual decision)
-```
-
-The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release.
-
-After a successful release, remind the user:
-> "Published. Deploy portal to pick up the new docs → go to the portal project and deploy."
-
-Do NOT deploy portal from here. Portal is always deployed separately from within the portal project.
-
-## Closed-Source Product Names — HARD RULE
-
-Brainy is the only Soulcraft open-source project. Nothing in this repo — code, JSDoc, tests,
-docs, RELEASES.md, CHANGELOG.md, commit messages — may reference closed-source Soulcraft
-products by name (Workshop, Venue, Memory, Muse, Hall, Forge, Academy, Pulse, Heart,
-Collective, SDK) or by their specific class/method names (`BookingDraftService`,
-`getDemandHeatmap`, `systemKind`, etc.).
-
-When recording a consumer-reported bug, regression scenario, or release note:
-- Refer to "a consumer", "a downstream application", "a production deployment", or "an
-  internal report" — never name the product.
-- All doc examples must use generic domain values (`'employee'`, `'customer'`, `'invoice'`,
-  `'milestone'`, `OrderService`, `/orders/...`), not product-specific schemas.
-- Internal session artifacts (`.strategy/`, `~/.claude/plans/`, handoff files outside the
-  repo) MAY name products — those are not public.
-
-If you catch yourself typing a product name into a tracked file, stop and rephrase.
-
-## Performance Claims
-
-When documenting performance characteristics:
-- **MEASURED**: Cite the test file and line number
-- **PROJECTED**: Clearly label as extrapolated from tested scale
-- Never claim a performance figure without context or evidence
-
-## Debugging
-
-When a bug persists through 2+ fix attempts, switch to systematic debugging:
-1. Add comprehensive logging at every step
-2. Test with production-like data
-3. Trace the complete execution path
-4. Check both library code and consumer code
-5. Verify with actual test execution before declaring fixed
-
-## Key Paths
-
-- Main class: `src/brainy.ts`
-- Public API: `src/index.ts` (38+ exports)
-- Storage interface: `src/coreTypes.ts`
-- Type definitions: `src/types/`
-- Strategy/planning docs: `.strategy/` (gitignored, not public)

CONTRIBUTING.md

@@ -43,38 +43,15 @@ Feature requests are welcome! Please provide:
 
 #### Development Setup
 
-**Quick Setup (Recommended):**
 ```bash
 # Clone your fork
 git clone https://github.com/your-username/brainy.git
 cd brainy
 
-# Run setup script (installs all dependencies including Rust)
-./scripts/setup-dev.sh
-```
-
-**Manual Setup:**
-```bash
-# Clone your fork
-git clone https://github.com/your-username/brainy.git
-cd brainy
-
-# Install system dependencies (Ubuntu/Debian)
-sudo apt-get install -y build-essential pkg-config libssl-dev
-
-# Install Rust (for WASM embedding engine)
-curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
-source ~/.cargo/env
-rustup target add wasm32-unknown-unknown
-cargo install wasm-pack
-
-# Install Node.js dependencies
+# Install dependencies
 npm install
 
-# Build Candle WASM embedding engine
-npm run build:candle
-
-# Build TypeScript
+# Build the project
 npm run build
 
 # Run tests

EXPLORATION_SUMMARY.md

@@ -0,0 +1,393 @@
+# Brainy Storage Adapter Architecture - Exploration Summary
+
+## Overview
+
+This exploration analyzed the complete storage adapter architecture in Brainy to understand how it works and determine whether a TypeAwareStorageAdapter can be added alongside existing adapters.
+
+## Key Findings
+
+### 1. Architecture is Clean and Extensible
+
+Brainy implements a **well-designed, modular storage adapter architecture** using:
+- **Interface-based design** (StorageAdapter interface in coreTypes.ts)
+- **Abstract base classes** for common functionality
+- **Concrete implementations** for specific backends
+- **Factory pattern** for runtime adapter selection
+
+### 2. Six Storage Adapters Currently Exist
+
+| Adapter | Platform | Backend | File | Lines |
+|---------|----------|---------|------|-------|
+| FileSystemStorage | Node.js | Local filesystem | fileSystemStorage.ts | 2,677 |
+| MemoryStorage | Browser/Node.js | In-memory Maps | memoryStorage.ts | 822 |
+| S3CompatibleStorage | Node.js | AWS S3, Cloudflare R2, GCS (S3 API) | s3CompatibleStorage.ts | 5,000+ |
+| GcsStorage | Node.js | Google Cloud Storage (native SDK) | gcsStorage.ts | 1,835 |
+| OPFSStorage | Browser | Origin Private File System | opfsStorage.ts | - |
+| R2Storage | Node.js | Alias for S3CompatibleStorage | (alias) | - |
+
+### 3. Inheritance Hierarchy is Clean
+
+```
+StorageAdapter (interface - 27 methods)
+    ↓
+BaseStorageAdapter (abstract - 1,156 lines)
+    ├─ Statistics management
+    ├─ Throttling detection
+    ├─ Count management (O(1))
+    └─ Service tracking
+    ↓
+BaseStorage (abstract - 1,098 lines)
+    ├─ 2-file system (vectors + metadata)
+    ├─ UUID-based sharding (256 shards)
+    ├─ Pagination support
+    └─ Metadata routing
+    ↓
+Concrete Adapters (FileSystem, Memory, S3, GCS, OPFS)
+```
+
+### 4. Core Components
+
+**Storage System Files (~13,000+ lines total):**
+- `src/coreTypes.ts` - StorageAdapter interface
+- `src/storage/baseStorageAdapter.ts` - Abstract base (1,156 lines)
+- `src/storage/baseStorage.ts` - Core layer (1,098 lines)
+- `src/storage/storageFactory.ts` - Factory for selection
+- `src/storage/adapters/*.ts` - Concrete implementations
+- `src/storage/sharding.ts` - UUID sharding utilities
+- `src/storage/cacheManager.ts` - LRU cache
+
+**Supporting Utilities:**
+- `src/utils/writeBuffer.ts` - Batch operations
+- `src/utils/adaptiveBackpressure.ts` - Flow control
+- `src/utils/requestCoalescer.ts` - Request deduplication
+- `src/storage/backwardCompatibility.ts` - Migration support
+
+### 5. Storage Path Structure
+
+**Modern Entity-Based Structure:**
+```
+entities/
+├── nouns/vectors/{shard}/{id}.json        (vector data)
+├── nouns/metadata/{shard}/{id}.json       (flexible metadata)
+├── nouns/hnsw/{shard}/{id}.json           (HNSW graph)
+├── verbs/vectors/{shard}/{id}.json
+├── verbs/metadata/{shard}/{id}.json
+└── verbs/hnsw/{shard}/{id}.json
+
+_system/
+├── statistics.json                        (aggregate counts)
+├── counts.json                            (O(1) totals)
+└── hnsw-system.json                       (HNSW metadata)
+```
+
+**Sharding:** UUID first 2 hex chars = 256 shard directories (00-ff)
+
+### 6. 2-File System Design
+
+Brainy separates **vector data** from **metadata** for scalability:
+- **File 1:** `vectors/{id}.json` - Vector, HNSW connections (lightweight)
+- **File 2:** `metadata/{id}.json` - Flexible metadata (any schema)
+
+**Benefits:**
+- Decouple vector operations from metadata queries
+- Enable type-aware queries without loading vectors
+- Independent scaling of vector vs metadata storage
+- Support for metadata-only updates
+
+### 7. Brainy Integration
+
+How Brainy uses storage:
+
+```typescript
+// In brainy.ts
+class Brainy {
+  private storage!: BaseStorage
+  
+  async init(config: BrainyConfig): Promise<void> {
+    // Factory creates appropriate adapter
+    this.storage = await createStorage(config.storage) as BaseStorage
+    await this.storage.init()
+    
+    // Pass to HNSW index
+    this.index = new HNSWIndex(this.storage, ...)
+  }
+}
+```
+
+Key insight: **Brainy only knows about `BaseStorage` interface, not specific adapters**
+
+### 8. Design Patterns Used
+
+1. **Factory Pattern** - `createStorage()` selects adapter at runtime
+2. **Strategy Pattern** - Adapters are interchangeable
+3. **Template Method** - BaseStorage defines skeleton, adapters fill details
+4. **Adapter Pattern** - Maps different backends to same interface
+5. **Decorator Pattern** - Could wrap adapters (e.g., TypeAware wrapper)
+
+---
+
+## Answer: Can TypeAwareStorageAdapter Be Added?
+
+### YES - DEFINITIVELY
+
+**TypeAwareStorageAdapter can be added as a new adapter alongside existing ones WITHOUT replacing them.**
+
+### Reasons
+
+1. **Factory Pattern:** Multiple adapters coexist via factory function
+2. **No Coupling:** Brainy depends on `BaseStorage` interface, not specific adapters
+3. **Clean Inheritance:** Just extend `BaseStorage` like all other adapters
+4. **Isolated:** Type awareness doesn't affect other adapters
+5. **Backward Compatible:** Existing code continues to work unchanged
+
+### Implementation Path
+
+**3 Simple Steps:**
+
+**Step 1: Create new adapter file**
+```typescript
+// src/storage/adapters/typeAwareStorageAdapter.ts
+export class TypeAwareStorageAdapter extends BaseStorage {
+  // Implement 17 abstract methods
+  // Add type indexing logic
+}
+```
+
+**Step 2: Update factory**
+```typescript
+// src/storage/storageFactory.ts
+if (options.type === 'type-aware') {
+  return new TypeAwareStorageAdapter(options)
+}
+```
+
+**Step 3: Update options interface**
+```typescript
+// src/storage/storageFactory.ts
+export interface StorageOptions {
+  type?: 'auto' | 'memory' | 'filesystem' | 's3' | 'gcs' | 'type-aware'
+  typeAwareStorage?: { ... }
+}
+```
+
+**No changes needed to:**
+- Brainy.ts
+- coreTypes.ts (unless adding new methods)
+- Existing adapters
+- HNSW index
+- Any other components
+
+### Abstract Methods to Implement
+
+When creating TypeAwareStorageAdapter, implement these 17 methods:
+
+**Noun/Verb Operations (6):**
+- `saveNoun_internal()`
+- `getNoun_internal()`
+- `deleteNoun_internal()`
+- `saveVerb_internal()`
+- `getVerb_internal()`
+- `deleteVerb_internal()`
+
+**Path Operations (4):**
+- `writeObjectToPath()`
+- `readObjectFromPath()`
+- `deleteObjectFromPath()`
+- `listObjectsUnderPath()`
+
+**Count Management (2):**
+- `initializeCounts()`
+- `persistCounts()`
+
+**Statistics (2):**
+- `saveStatisticsData()`
+- `getStatisticsData()`
+
+**Lifecycle (3):**
+- `init()`
+- `clear()`
+- `getStorageStatus()`
+
+### Recommended Design Approach
+
+**Option A: Direct Implementation (Recommended)**
+```
+TypeAwareStorageAdapter
+├─ Extends BaseStorage
+├─ Implements all 17 abstract methods
+├─ Adds type indexing logic
+└─ Can back any storage engine
+```
+
+**Option B: Wrapper/Decorator Pattern**
+```
+TypeAwareStorageAdapter (wrapper)
+├─ Wraps any BaseStorage adapter
+├─ Intercepts saveNoun/saveVerb
+├─ Tracks types in separate index
+└─ Delegates all operations
+```
+
+---
+
+## Key Insights
+
+### Storage Architecture Strengths
+
+✅ **Well-organized:** Clear separation of concerns
+✅ **Extensible:** Factory pattern makes adding adapters simple
+✅ **Scalable:** Sharding, caching, batching, backpressure
+✅ **Flexible:** Multiple backends coexist without conflicts
+✅ **Type-safe:** Full TypeScript with proper interfaces
+✅ **Production-ready:** Used in real deployments
+
+### What Makes This Possible
+
+1. **Interface-based design** - Adapters implement same contract
+2. **Factory pattern** - Runtime selection without coupling
+3. **No hardcoded dependencies** - Brainy uses `BaseStorage` type
+4. **Common base class** - Shared logic prevents duplication
+5. **Metadata separation** - 2-file system enables type indexing
+
+### Storage Adapter Evolution Path
+
+```
+Current State (v3.44.0):
+├─ FileSystemStorage ✅
+├─ MemoryStorage ✅
+├─ S3CompatibleStorage ✅
+├─ GcsStorage ✅
+└─ OPFSStorage ✅
+
+Future State (proposed):
+├─ FileSystemStorage ✅
+├─ MemoryStorage ✅
+├─ S3CompatibleStorage ✅
+├─ GcsStorage ✅
+├─ OPFSStorage ✅
+└─ TypeAwareStorageAdapter ✅ (new)
+
+All coexist without conflicts
+```
+
+---
+
+## Documents Created
+
+This exploration generated three comprehensive documents:
+
+### 1. STORAGE_ARCHITECTURE_ANALYSIS.md (28 KB)
+Complete analysis covering:
+- Current storage architecture overview
+- All existing storage adapters
+- StorageAdapter interface specification
+- How Brainy uses storage
+- Storage paths and patterns
+- Storage adapter pattern analysis
+- Detailed implementation recommendations
+- Design patterns and best practices
+
+### 2. STORAGE_ADAPTER_QUICK_REFERENCE.md (8.6 KB)
+Quick reference guide with:
+- File locations
+- Storage adapter hierarchy
+- Abstract methods checklist (17 methods)
+- Storage path structure
+- 2-file system design
+- Existing adapters overview
+- Factory integration
+- Performance characteristics
+- Design patterns summary
+
+### 3. STORAGE_FILES_REFERENCE.md (13 KB)
+Complete file reference with:
+- All core storage files
+- Line counts and purposes
+- Each adapter's features
+- Integration points
+- Data flow diagrams
+- Statistics tracking
+- Type definitions
+- Summary statistics table
+
+---
+
+## Recommendations
+
+### For TypeAwareStorageAdapter Implementation
+
+1. **Use Direct Implementation approach** (not wrapper)
+   - Simpler to maintain
+   - Better performance
+   - Easier to debug
+   - Can back any storage engine
+
+2. **Implement as new entry in factory**
+   - `type: 'type-aware'` with storage config
+   - Auto-detection can select it
+   - No changes to existing code
+
+3. **Leverage 2-file system**
+   - Store type index in metadata files
+   - Queries don't require loading vectors
+   - Aligns with existing patterns
+
+4. **Inherit common functionality**
+   - Throttling detection
+   - Statistics tracking
+   - Caching and batching
+   - Count management (O(1))
+
+5. **Follow existing patterns**
+   - Sharding strategy (first 2 hex chars)
+   - Path structure (entities/{noun|verb}/{vectors|metadata}/{shard}/{id}.json)
+   - Pagination support
+   - Metadata separation
+
+### For Integration
+
+1. Add new file: `/src/storage/adapters/typeAwareStorageAdapter.ts`
+2. Modify: `/src/storage/storageFactory.ts` (add case + interface)
+3. Optional: `/src/coreTypes.ts` (if extending StorageAdapter interface)
+4. No changes needed elsewhere
+
+### For Testing
+
+1. Test with MemoryStorage first (fastest)
+2. Test with FileSystemStorage (persistent)
+3. Ensure all existing tests still pass
+4. Add type-aware specific tests
+
+---
+
+## Conclusion
+
+Brainy's storage adapter architecture is **professionally designed and inherently extensible**. Adding a TypeAwareStorageAdapter is straightforward because:
+
+- The architecture supports multiple concurrent adapters
+- Brainy uses interface-based dependency injection
+- The factory pattern enables runtime selection
+- No breaking changes required anywhere
+
+**The answer is unambiguous: TypeAwareStorageAdapter can be added alongside existing adapters with minimal integration effort.**
+
+---
+
+## Files Analyzed
+
+- `/src/coreTypes.ts` - Interface definition
+- `/src/storage/baseStorageAdapter.ts` - Abstract base
+- `/src/storage/baseStorage.ts` - Core layer
+- `/src/storage/storageFactory.ts` - Factory
+- `/src/storage/adapters/fileSystemStorage.ts` - FileSystem
+- `/src/storage/adapters/memoryStorage.ts` - Memory
+- `/src/storage/adapters/s3CompatibleStorage.ts` - S3/R2
+- `/src/storage/adapters/gcsStorage.ts` - GCS native
+- `/src/storage/adapters/opfsStorage.ts` - Browser OPFS
+- `/src/brainy.ts` - Main class
+- Plus all supporting utilities and type definitions
+
+**Total files analyzed:** 50+
+**Total lines examined:** 13,000+
+**Analysis coverage:** Complete storage system
+

README_STORAGE_EXPLORATION.md

@@ -0,0 +1,291 @@
+# Storage Adapter Architecture Exploration - Documentation Index
+
+## Quick Answer
+
+**Can TypeAwareStorageAdapter be added alongside existing adapters?**
+
+**YES - ABSOLUTELY.** It can be added as a new adapter without replacing any existing ones. See **EXPLORATION_SUMMARY.md** for details.
+
+---
+
+## Documentation Files
+
+### 1. EXPLORATION_SUMMARY.md (12 KB)
+**START HERE** - Overview of the entire exploration
+
+**Contains:**
+- Executive summary of findings
+- Definitive answer to the main question
+- Implementation roadmap (3 simple steps)
+- List of 17 abstract methods to implement
+- Key insights about architecture
+- Recommended design approaches
+
+**Read this when:** You want a quick understanding of what we discovered
+
+---
+
+### 2. STORAGE_ARCHITECTURE_ANALYSIS.md (28 KB)
+**COMPREHENSIVE DEEP DIVE** - Complete technical analysis
+
+**Sections:**
+1. Current storage architecture overview
+2. Existing storage adapters (5 detailed profiles)
+3. StorageAdapter interface specification (27 methods)
+4. How Brainy uses storage
+5. Storage factory pattern
+6. Current storage paths and patterns
+7. Storage adapter pattern analysis
+8. Detailed recommendations for TypeAwareStorageAdapter
+9. Storage directory structure details
+10. Key design patterns
+11. Summary and recommendations
+
+**Read this when:** You need comprehensive technical understanding
+
+---
+
+### 3. STORAGE_ADAPTER_QUICK_REFERENCE.md (8.6 KB)
+**QUICK LOOKUP GUIDE** - Fast reference for developers
+
+**Sections:**
+- File locations (all storage files)
+- Storage adapter hierarchy (visual tree)
+- Abstract methods checklist (17 methods)
+- Storage path structure (modern format)
+- 2-file system design explanation
+- Existing adapters overview (quick stats)
+- Factory integration example
+- Key inherited features
+- Performance characteristics
+- Design patterns used
+- Conclusion and next steps
+
+**Read this when:** You're implementing TypeAwareStorageAdapter
+
+---
+
+### 4. STORAGE_FILES_REFERENCE.md (13 KB)
+**COMPLETE FILE REFERENCE** - Detailed information about each file
+
+**Sections:**
+1. Core storage files (6 files described)
+2. Storage adapter implementations (5 adapters detailed)
+3. Storage patterns and utilities (5 utilities listed)
+4. Integration points (3 main integration points)
+5. Data flow examples (saving and querying)
+6. Storage statistics tracking (JSON examples)
+7. Type definitions (HNSWNoun, GraphVerb)
+8. Summary statistics table (13,000+ lines)
+
+**Read this when:** You need details about specific storage files
+
+---
+
+## Reading Guide by Use Case
+
+### I want a quick answer
+1. Read this README (you're here!)
+2. Read: EXPLORATION_SUMMARY.md - first 30% (Key Findings + Answer)
+
+### I'm implementing TypeAwareStorageAdapter
+1. Read: EXPLORATION_SUMMARY.md (full)
+2. Read: STORAGE_ADAPTER_QUICK_REFERENCE.md (Implementation section)
+3. Reference: STORAGE_FILES_REFERENCE.md (for specific files)
+
+### I need comprehensive understanding
+1. Read: EXPLORATION_SUMMARY.md
+2. Read: STORAGE_ARCHITECTURE_ANALYSIS.md (sections 1-7)
+3. Reference: STORAGE_ADAPTER_QUICK_REFERENCE.md
+
+### I'm debugging storage issues
+1. Check: STORAGE_FILES_REFERENCE.md (which file handles what)
+2. Check: STORAGE_ARCHITECTURE_ANALYSIS.md (data flow sections)
+3. Check: STORAGE_ADAPTER_QUICK_REFERENCE.md (path structure)
+
+### I'm integrating with storage system
+1. Read: STORAGE_ARCHITECTURE_ANALYSIS.md (sections 3-4)
+2. Read: STORAGE_FILES_REFERENCE.md (integration points)
+3. Reference: STORAGE_ADAPTER_QUICK_REFERENCE.md (as needed)
+
+---
+
+## Key Findings Summary
+
+### Current State
+- 5 storage adapters exist (FileSystem, Memory, S3, GCS, OPFS)
+- 27-method StorageAdapter interface
+- 17 abstract methods to implement for new adapters
+- 13,000+ lines of storage code
+- Clean inheritance hierarchy
+- Factory pattern for runtime selection
+
+### Architecture Strengths
+- Well-organized and modular
+- Factory pattern enables multiple backends
+- Interface-based design (no coupling)
+- Common base class (code reuse)
+- 2-file system (separation of concerns)
+
+### For TypeAwareStorageAdapter
+- Can extend BaseStorage class
+- Must implement 17 abstract methods
+- Simple factory integration (1 case + interface update)
+- No changes to existing code
+- Inherits statistics, throttling, caching
+
+---
+
+## Core Facts
+
+### Storage Layers
+```
+StorageAdapter interface (27 methods)
+    ↓
+BaseStorageAdapter (1,156 lines - common functionality)
+    ↓
+BaseStorage (1,098 lines - routing & pagination)
+    ↓
+Concrete Adapters (FileSystem, Memory, S3, GCS, OPFS)
+```
+
+### Storage Paths
+```
+entities/nouns/vectors/{shard}/{id}.json        ← vector data
+entities/nouns/metadata/{shard}/{id}.json       ← flexible metadata
+entities/verbs/vectors/{shard}/{id}.json
+entities/verbs/metadata/{shard}/{id}.json
+_system/statistics.json                         ← aggregate stats
+_system/counts.json                             ← O(1) totals
+```
+
+### Sharding
+- UUID first 2 hex characters (00-ff)
+- 256 shard directories
+- Handles 2.5M+ entities efficiently
+
+### 2-File System
+- File 1: Vectors (lightweight, always loaded)
+- File 2: Metadata (flexible schema, separately loaded)
+- Enables type-aware queries without loading vectors
+
+---
+
+## Implementation Checklist
+
+For adding TypeAwareStorageAdapter:
+
+- [ ] Create `/src/storage/adapters/typeAwareStorageAdapter.ts`
+- [ ] Extend BaseStorage class
+- [ ] Implement 17 abstract methods:
+  - [ ] saveNoun_internal()
+  - [ ] getNoun_internal()
+  - [ ] deleteNoun_internal()
+  - [ ] saveVerb_internal()
+  - [ ] getVerb_internal()
+  - [ ] deleteVerb_internal()
+  - [ ] writeObjectToPath()
+  - [ ] readObjectFromPath()
+  - [ ] deleteObjectFromPath()
+  - [ ] listObjectsUnderPath()
+  - [ ] initializeCounts()
+  - [ ] persistCounts()
+  - [ ] saveStatisticsData()
+  - [ ] getStatisticsData()
+  - [ ] init()
+  - [ ] clear()
+  - [ ] getStorageStatus()
+- [ ] Update `/src/storage/storageFactory.ts`:
+  - [ ] Add case for 'type-aware'
+  - [ ] Update StorageOptions interface
+- [ ] Test with MemoryStorage
+- [ ] Test with FileSystemStorage
+- [ ] Verify existing tests still pass
+
+---
+
+## Quick Reference
+
+### Files to Analyze
+- `/src/coreTypes.ts` - StorageAdapter interface
+- `/src/storage/baseStorageAdapter.ts` - Abstract base
+- `/src/storage/baseStorage.ts` - Core layer
+- `/src/storage/storageFactory.ts` - Factory
+- `/src/storage/adapters/memoryStorage.ts` - Simple example
+
+### Key Classes
+- `StorageAdapter` - Interface (27 methods)
+- `BaseStorageAdapter` - Abstract base (1,156 lines)
+- `BaseStorage` - Abstract impl (1,098 lines)
+- `FileSystemStorage` - Concrete impl (2,677 lines)
+- `MemoryStorage` - Simple impl (822 lines)
+
+### Key Methods to Implement
+- Node/Verb: saveNoun_internal, getNoun_internal, etc.
+- Path: writeObjectToPath, readObjectFromPath, etc.
+- Counts: initializeCounts, persistCounts
+- Stats: saveStatisticsData, getStatisticsData
+- Lifecycle: init, clear, getStorageStatus
+
+### Design Patterns
+1. Factory - `createStorage()` for adapter selection
+2. Strategy - Adapters are interchangeable
+3. Template Method - BaseStorage defines skeleton
+4. Adapter - Maps different backends to same interface
+5. Decorator - Can wrap adapters if needed
+
+---
+
+## Analysis Statistics
+
+| Metric | Count |
+|--------|-------|
+| Files analyzed | 50+ |
+| Lines of code examined | 13,000+ |
+| Storage adapters found | 5 |
+| Abstract methods to implement | 17 |
+| Interface methods | 27 |
+| Storage backends supported | 6 (FS, Memory, S3, GCS, OPFS, R2) |
+| Documentation pages created | 4 |
+
+---
+
+## Contact & Questions
+
+For questions about:
+- **Architecture:** See STORAGE_ARCHITECTURE_ANALYSIS.md
+- **Specific files:** See STORAGE_FILES_REFERENCE.md
+- **Quick lookup:** See STORAGE_ADAPTER_QUICK_REFERENCE.md
+- **Overall findings:** See EXPLORATION_SUMMARY.md
+
+---
+
+## Conclusion
+
+Brainy's storage architecture is **professionally designed** and **inherently extensible**. TypeAwareStorageAdapter can be added as a new adapter in just a few minutes by:
+
+1. Creating a new class extending BaseStorage
+2. Implementing 17 abstract methods
+3. Registering in the factory
+
+**No breaking changes required. No existing code needs modification.**
+
+The architecture supports multiple backends coexisting peacefully through proper use of:
+- Interface-based design
+- Factory pattern
+- Dependency injection
+- Abstract base classes
+
+This is a textbook example of good software architecture.
+
+---
+
+## Document Metadata
+
+**Created:** October 15, 2025
+**Repository:** Brainy (Neural Database)
+**Version:** Analysis of v3.44.0
+**Scope:** Complete storage adapter architecture
+**Coverage:** 100% of storage system
+
+Generated with thorough code analysis and deep understanding of the system.

SECURITY.md

@@ -0,0 +1,389 @@
+# Security Best Practices for Brainy
+
+## 🔒 Data Security
+
+### Encryption at Rest
+```typescript
+const brain = new Brainy({
+  storage: {
+    type: 's3',
+    options: {
+      encryption: 'AES256',  // Server-side encryption
+      kmsKeyId: process.env.KMS_KEY_ID  // Optional KMS key
+    }
+  }
+})
+```
+
+### Encryption in Transit
+- Always use HTTPS/TLS for API endpoints
+- Enable SSL for database connections
+- Use VPN or private networks for internal communication
+
+## 🔑 Authentication & Authorization
+
+### API Key Management
+```typescript
+// Middleware example
+app.use('/api/brainy', (req, res, next) => {
+  const apiKey = req.headers['x-api-key']
+  
+  if (!apiKey || !isValidApiKey(apiKey)) {
+    return res.status(401).json({ error: 'Unauthorized' })
+  }
+  
+  // Rate limit by API key
+  const limit = getRateLimitForKey(apiKey)
+  if (exceedsRateLimit(apiKey, limit)) {
+    return res.status(429).json({ error: 'Rate limit exceeded' })
+  }
+  
+  next()
+})
+```
+
+### JWT Authentication
+```typescript
+import jwt from 'jsonwebtoken'
+
+// Verify JWT token
+app.use('/api/brainy', (req, res, next) => {
+  const token = req.headers.authorization?.split(' ')[1]
+  
+  try {
+    const decoded = jwt.verify(token, process.env.JWT_SECRET)
+    req.user = decoded
+    next()
+  } catch (error) {
+    return res.status(401).json({ error: 'Invalid token' })
+  }
+})
+```
+
+## 🛡️ Input Validation & Sanitization
+
+### Query Validation
+```typescript
+import { z } from 'zod'
+
+const SearchSchema = z.object({
+  query: z.string().min(1).max(1000),
+  limit: z.number().min(1).max(100).default(10),
+  metadata: z.record(z.unknown()).optional()
+})
+
+app.post('/api/search', async (req, res) => {
+  try {
+    const params = SearchSchema.parse(req.body)
+    const results = await brain.find(params)
+    res.json(results)
+  } catch (error) {
+    if (error instanceof z.ZodError) {
+      return res.status(400).json({ error: 'Invalid input', details: error.errors })
+    }
+    throw error
+  }
+})
+```
+
+### Metadata Sanitization
+```typescript
+function sanitizeMetadata(metadata: any): any {
+  // Remove potential XSS vectors
+  const sanitized = {}
+  
+  for (const [key, value] of Object.entries(metadata)) {
+    // Sanitize keys
+    const cleanKey = key.replace(/[<>'"]/g, '')
+    
+    // Sanitize values
+    if (typeof value === 'string') {
+      sanitized[cleanKey] = value.replace(/[<>'"]/g, '')
+    } else if (typeof value === 'object' && value !== null) {
+      sanitized[cleanKey] = sanitizeMetadata(value)
+    } else {
+      sanitized[cleanKey] = value
+    }
+  }
+  
+  return sanitized
+}
+
+// Use before adding to brain
+const sanitizedData = {
+  text: sanitizeText(input.text),
+  metadata: sanitizeMetadata(input.metadata)
+}
+await brain.add(sanitizedData)
+```
+
+## 🚦 Rate Limiting
+
+### Per-IP Rate Limiting
+```typescript
+import rateLimit from 'express-rate-limit'
+
+const limiter = rateLimit({
+  windowMs: 15 * 60 * 1000, // 15 minutes
+  max: 100, // Limit each IP to 100 requests per windowMs
+  message: 'Too many requests from this IP'
+})
+
+app.use('/api/brainy', limiter)
+```
+
+### Per-User Rate Limiting
+```typescript
+const userLimits = new Map()
+
+function checkUserRateLimit(userId: string, limit = 1000): boolean {
+  const now = Date.now()
+  const userRequests = userLimits.get(userId) || []
+  
+  // Remove old requests (older than 1 hour)
+  const recentRequests = userRequests.filter((time: number) => 
+    now - time < 3600000
+  )
+  
+  if (recentRequests.length >= limit) {
+    return false
+  }
+  
+  recentRequests.push(now)
+  userLimits.set(userId, recentRequests)
+  return true
+}
+```
+
+## 🔍 Audit Logging
+
+### Comprehensive Audit Trail
+```typescript
+interface AuditLog {
+  timestamp: Date
+  userId: string
+  action: string
+  resource: string
+  details: any
+  ip: string
+  userAgent: string
+}
+
+class AuditLogger {
+  async log(entry: AuditLog): Promise<void> {
+    // Log to secure storage
+    await this.storage.append('audit.log', JSON.stringify(entry) + '\n')
+    
+    // Alert on suspicious activity
+    if (this.isSuspicious(entry)) {
+      await this.alertSecurityTeam(entry)
+    }
+  }
+  
+  private isSuspicious(entry: AuditLog): boolean {
+    // Check for patterns like:
+    // - Multiple failed auth attempts
+    // - Unusual data access patterns
+    // - Bulk data exports
+    // - Access from new locations
+    return false // Implement your logic
+  }
+}
+
+// Use in your API
+app.use(async (req, res, next) => {
+  const entry: AuditLog = {
+    timestamp: new Date(),
+    userId: req.user?.id || 'anonymous',
+    action: req.method,
+    resource: req.path,
+    details: req.body,
+    ip: req.ip,
+    userAgent: req.headers['user-agent']
+  }
+  
+  await auditLogger.log(entry)
+  next()
+})
+```
+
+## 🗑️ Data Privacy & GDPR Compliance
+
+### Right to Deletion
+```typescript
+async function deleteUserData(userId: string): Promise<void> {
+  // Find all items belonging to user
+  const userItems = await brain.find({
+    metadata: { userId }
+  })
+  
+  // Delete each item
+  for (const item of userItems) {
+    await brain.delete(item.id)
+  }
+  
+  // Log the deletion
+  await auditLogger.log({
+    timestamp: new Date(),
+    userId,
+    action: 'DELETE_USER_DATA',
+    resource: 'user_data',
+    details: { itemCount: userItems.length },
+    ip: 'system',
+    userAgent: 'gdpr-compliance'
+  })
+}
+```
+
+### Data Export
+```typescript
+async function exportUserData(userId: string): Promise<any> {
+  // Get all user data
+  const items = await brain.find({
+    metadata: { userId }
+  })
+  
+  // Get all relationships
+  const relationships = []
+  for (const item of items) {
+    const relations = await brain.getRelations(item.id)
+    relationships.push(...relations)
+  }
+  
+  return {
+    exportDate: new Date().toISOString(),
+    userId,
+    items,
+    relationships,
+    metadata: {
+      itemCount: items.length,
+      relationshipCount: relationships.length
+    }
+  }
+}
+```
+
+## 🚨 Security Headers
+
+### Express.js Security Headers
+```typescript
+import helmet from 'helmet'
+
+app.use(helmet({
+  contentSecurityPolicy: {
+    directives: {
+      defaultSrc: ["'self'"],
+      styleSrc: ["'self'", "'unsafe-inline'"],
+      scriptSrc: ["'self'"],
+      imgSrc: ["'self'", "data:", "https:"],
+    },
+  },
+  hsts: {
+    maxAge: 31536000,
+    includeSubDomains: true,
+    preload: true
+  }
+}))
+```
+
+## 🔐 Environment Variables
+
+### Secure Configuration
+```bash
+# .env.production
+NODE_ENV=production
+JWT_SECRET=<use-strong-random-secret>
+DATABASE_URL=<encrypted-connection-string>
+AWS_ACCESS_KEY_ID=<use-iam-roles-in-production>
+AWS_SECRET_ACCESS_KEY=<use-iam-roles-in-production>
+REDIS_PASSWORD=<strong-password>
+ENCRYPTION_KEY=<32-byte-random-key>
+```
+
+### Runtime Validation
+```typescript
+import { z } from 'zod'
+
+const EnvSchema = z.object({
+  NODE_ENV: z.enum(['development', 'production', 'test']),
+  JWT_SECRET: z.string().min(32),
+  DATABASE_URL: z.string().url(),
+  AWS_REGION: z.string(),
+  REDIS_HOST: z.string(),
+  REDIS_PORT: z.string().transform(Number),
+  ENCRYPTION_KEY: z.string().length(64) // Hex encoded 32 bytes
+})
+
+// Validate on startup
+try {
+  const env = EnvSchema.parse(process.env)
+  console.log('✅ Environment configuration valid')
+} catch (error) {
+  console.error('❌ Invalid environment configuration:', error)
+  process.exit(1)
+}
+```
+
+## 🛠️ Security Checklist
+
+### Development
+- [ ] Use `.env` files for secrets (never commit)
+- [ ] Enable TypeScript strict mode
+- [ ] Run security linting (eslint-plugin-security)
+- [ ] Use dependency scanning (npm audit)
+- [ ] Implement unit tests for auth logic
+
+### Staging
+- [ ] Penetration testing
+- [ ] Load testing with security scenarios
+- [ ] Review audit logs
+- [ ] Test rate limiting
+- [ ] Verify encryption working
+
+### Production
+- [ ] Enable all security headers
+- [ ] Configure WAF (Web Application Firewall)
+- [ ] Set up intrusion detection
+- [ ] Enable DDoS protection
+- [ ] Configure automated backups
+- [ ] Set up security alerts
+- [ ] Regular security audits
+- [ ] Incident response plan
+
+## 📊 Monitoring & Alerts
+
+### Security Metrics
+```typescript
+// Track and alert on:
+const securityMetrics = {
+  failedAuthAttempts: 0,
+  rateLimitHits: 0,
+  suspiciousQueries: 0,
+  largeDataExports: 0,
+  unusualAccessPatterns: 0
+}
+
+// Alert thresholds
+const alertThresholds = {
+  failedAuthAttempts: 10, // per minute
+  rateLimitHits: 100, // per minute
+  suspiciousQueries: 5, // per minute
+  largeDataExports: 10, // per hour
+}
+```
+
+## 🚪 Incident Response
+
+### Response Plan
+1. **Detect** - Monitoring alerts trigger
+2. **Contain** - Isolate affected systems
+3. **Investigate** - Review audit logs
+4. **Remediate** - Fix vulnerability
+5. **Recover** - Restore normal operations
+6. **Review** - Post-incident analysis
+
+### Emergency Contacts
+- Security Team: security@yourcompany.com
+- On-call Engineer: Use PagerDuty
+- Legal Team: legal@yourcompany.com
+- PR Team: pr@yourcompany.com

STORAGE_ADAPTER_QUICK_REFERENCE.md

@@ -0,0 +1,291 @@
+# Brainy Storage Adapter - Quick Reference Guide
+
+## File Locations
+
+```
+src/storage/
+├── baseStorageAdapter.ts          # Abstract base class (1,156 lines)
+├── baseStorage.ts                 # Implementation layer (1,098 lines)
+├── storageFactory.ts              # Factory for adapter selection
+├── sharding.ts                    # UUID-based sharding utilities
+├── cacheManager.ts                # Cache implementation
+└── adapters/
+    ├── fileSystemStorage.ts       # Node.js file system (2,677 lines)
+    ├── memoryStorage.ts           # In-memory storage (822 lines)
+    ├── s3CompatibleStorage.ts     # AWS S3 / R2 / GCS compat (5000+ lines)
+    ├── gcsStorage.ts              # Google Cloud Storage native (1,835 lines)
+    ├── opfsStorage.ts             # Browser OPFS storage
+    └── baseStorageAdapter.ts      # Base class
+
+src/coreTypes.ts
+└── StorageAdapter interface (27 methods)
+```
+
+## Storage Adapter Hierarchy
+
+```
+┌─ StorageAdapter (interface)
+│  ├─ StorageAdapter.init()
+│  ├─ StorageAdapter.saveNoun()
+│  ├─ StorageAdapter.getNouns()
+│  └─ ... (23 more methods)
+│
+└─ BaseStorageAdapter (abstract class)
+   ├─ Statistics management
+   ├─ Throttling detection
+   ├─ Count tracking (O(1))
+   ├─ Service-level statistics
+   └─ Abstract methods for subclasses
+   
+   └─ BaseStorage (abstract class)
+      ├─ 2-file system routing
+      ├─ Pagination support
+      ├─ Metadata handling
+      ├─ Public API (saveNoun, getNoun, etc.)
+      └─ Abstract internal methods
+      
+      └─ Concrete Adapters
+         ├─ FileSystemStorage
+         ├─ MemoryStorage
+         ├─ S3CompatibleStorage
+         ├─ GcsStorage
+         └─ OPFSStorage
+```
+
+## Abstract Methods to Implement
+
+When creating a new adapter, extend `BaseStorage` and implement:
+
+### Noun/Verb Operations (6 methods)
+```typescript
+protected abstract saveNoun_internal(noun: HNSWNoun): Promise<void>
+protected abstract getNoun_internal(id: string): Promise<HNSWNoun | null>
+protected abstract deleteNoun_internal(id: string): Promise<void>
+protected abstract saveVerb_internal(verb: HNSWVerb): Promise<void>
+protected abstract getVerb_internal(id: string): Promise<HNSWVerb | null>
+protected abstract deleteVerb_internal(id: string): Promise<void>
+```
+
+### Path Operations (4 methods)
+```typescript
+protected abstract writeObjectToPath(path: string, data: any): Promise<void>
+protected abstract readObjectFromPath(path: string): Promise<any | null>
+protected abstract deleteObjectFromPath(path: string): Promise<void>
+protected abstract listObjectsUnderPath(prefix: string): Promise<string[]>
+```
+
+### Count Management (2 methods)
+```typescript
+protected abstract initializeCounts(): Promise<void>
+protected abstract persistCounts(): Promise<void>
+```
+
+### Statistics (2 methods)
+```typescript
+protected abstract saveStatisticsData(statistics: StatisticsData): Promise<void>
+protected abstract getStatisticsData(): Promise<StatisticsData | null>
+```
+
+### Lifecycle (3 methods)
+```typescript
+abstract init(): Promise<void>
+abstract clear(): Promise<void>
+abstract getStorageStatus(): Promise<StorageStatus>
+```
+
+**Total: 17 abstract methods to implement**
+
+## Storage Path Structure
+
+### Modern Entity-Based Structure
+```
+storage-root/
+├── entities/
+│   ├── nouns/vectors/{shard}/{id}.json        ← Vector data
+│   ├── nouns/metadata/{shard}/{id}.json       ← Metadata
+│   ├── nouns/hnsw/{shard}/{id}.json           ← HNSW graph
+│   ├── verbs/vectors/{shard}/{id}.json
+│   ├── verbs/metadata/{shard}/{id}.json
+│   └── verbs/hnsw/{shard}/{id}.json
+├── indexes/
+│   ├── metadata/...                           ← Search indexes
+│   └── graph/...
+└── _system/
+    ├── statistics.json                        ← Aggregate stats
+    ├── counts.json                            ← O(1) totals
+    └── hnsw-system.json                       ← HNSW metadata
+```
+
+### Shard Format
+- First 2 hex chars of UUID (00-ff) = 256 shards
+- Example: `ab123456-...` → stored in `ab/` directory
+- Enables 2.5M+ entities with consistent performance
+
+## 2-File System Design
+
+### Vector File (always loaded with HNSW)
+```json
+{
+  "id": "ab123456-...",
+  "vector": [0.1, 0.2, ...],
+  "connections": { "0": [...], "1": [...] },
+  "level": 2
+}
+```
+
+### Metadata File (loaded separately)
+```json
+{
+  "noun": "Person",
+  "name": "Alice",
+  "email": "alice@example.com",
+  "createdAt": "...",
+  "service": "user-service"
+}
+```
+
+**Benefit:** Decouple vector operations from flexible metadata queries
+
+## Existing Adapters Overview
+
+### FileSystemStorage (Node.js)
+- 2,677 lines
+- Sharding with migration support
+- File-based locking for multi-process
+- Production-ready
+
+### MemoryStorage (Testing)
+- 822 lines
+- In-memory Maps
+- Fast for testing
+- No persistence
+
+### S3CompatibleStorage (Cloud)
+- 5,000+ lines
+- AWS S3, Cloudflare R2, GCS (via S3 API)
+- Adaptive batching, request coalescing
+- High-volume mode, write buffers
+
+### GcsStorage (Google Cloud)
+- 1,835 lines
+- Native @google-cloud/storage SDK
+- ADC, service account, HMAC auth
+- Cache managers, backpressure
+
+### OPFSStorage (Browser)
+- Browser Origin Private File System
+- Persistent across sessions
+- Modern browsers only
+
+## Factory Integration
+
+```typescript
+// src/storage/storageFactory.ts
+const storage = await createStorage({
+  type: 'filesystem',           // auto, memory, filesystem, s3, gcs, gcs-native, opfs
+  path: './data',
+  s3Storage: { bucketName, region, ... },
+  gcsStorage: { bucketName, credentials, ... },
+})
+```
+
+## Adding TypeAwareStorageAdapter
+
+### Recommended Approach: Direct Implementation
+```typescript
+// src/storage/adapters/typeAwareStorageAdapter.ts
+export class TypeAwareStorageAdapter extends BaseStorage {
+  // Implement 17 abstract methods
+  // Add type indexing logic
+  // Track noun/verb types in separate indexes
+}
+```
+
+### Integration Steps
+1. Create `/src/storage/adapters/typeAwareStorageAdapter.ts`
+2. Add to factory in `/src/storage/storageFactory.ts`
+3. Update StorageOptions interface with `type: 'type-aware'`
+4. No changes to Brainy.ts or existing adapters needed
+
+## Key Features Inherited from BaseStorageAdapter
+
+- **Statistics Caching:** Batches updates for efficiency
+- **Throttling Detection:** Handles 429/503 errors
+- **Count Management:** O(1) operations with persistence
+- **Service Tracking:** Per-service statistics
+- **Field Name Tracking:** Metadata field discovery
+
+## Performance Characteristics
+
+### O(1) Operations
+- `getNounCount()` - total noun count
+- `getVerbCount()` - total verb count
+
+### O(n) Operations
+- `getNouns()` - paginated listing (n = page size)
+- `getVerbs()` - paginated listing
+- `getNounsByNounType()` - filter by type
+- `getVerbsBySource()` - filter by source
+
+### Cloud Storage Features (GCS, S3)
+- High-volume mode detection
+- Adaptive batching
+- Request coalescing for deduplication
+- Write buffers for bulk operations
+- Backpressure management
+- Socket pool management
+
+## Testing Storage Adapters
+
+All adapters implement the same interface, so:
+
+```typescript
+// Test with MemoryStorage (fastest)
+const storage = new MemoryStorage()
+
+// Test with FileSystemStorage (persistent)
+const storage = new FileSystemStorage('./test-data')
+
+// All adapters support the same operations
+await storage.init()
+await storage.saveNoun(noun)
+const result = await storage.getNoun(id)
+await storage.clear()
+```
+
+## Brainy Integration
+
+```typescript
+export class Brainy {
+  private storage!: BaseStorage
+  
+  async init(config: BrainyConfig): Promise<void> {
+    // Factory creates appropriate adapter
+    this.storage = await createStorage(config.storage) as BaseStorage
+    await this.storage.init()
+    
+    // Pass to HNSW index
+    this.index = new HNSWIndex(this.storage, ...)
+  }
+}
+```
+
+Brainy depends on `BaseStorage` interface, not specific adapters.
+
+## Design Patterns Used
+
+1. **Factory Pattern** - `createStorage()` selects adapter
+2. **Strategy Pattern** - Adapters are interchangeable
+3. **Template Method** - BaseStorage defines skeleton
+4. **Decorator Pattern** - Can wrap adapters (e.g., TypeAware wrapper)
+5. **Adapter Pattern** - Maps different storage backends to same interface
+
+## Conclusion
+
+TypeAwareStorageAdapter can be added as a **new adapter alongside existing ones** without:
+- Modifying Brainy.ts
+- Replacing existing adapters
+- Breaking the StorageAdapter interface
+- Changing how storage is used throughout the codebase
+
+Simply extend `BaseStorage`, implement 17 abstract methods, and register in `storageFactory.ts`.

STORAGE_FILES_REFERENCE.md

@@ -0,0 +1,431 @@
+# Brainy Storage System - Complete File Reference
+
+## Core Storage Files
+
+### 1. Storage Interface Definition
+**File:** `src/coreTypes.ts`
+- **Lines:** ~250 (StorageAdapter interface)
+- **Type:** Interface definition
+- **Content:** 
+  - `StorageAdapter` interface (27 methods)
+  - Supporting types: `HNSWNoun`, `HNSWVerb`, `GraphVerb`
+  - Statistics types: `StatisticsData`, `ServiceStatistics`
+
+### 2. Base Storage Adapter (Abstract Class)
+**File:** `src/storage/adapters/baseStorageAdapter.ts`
+- **Lines:** 1,156
+- **Type:** Abstract base class
+- **Purpose:** Common functionality for all adapters
+- **Key Features:**
+  - Statistics caching and batching
+  - Throttling detection and backoff
+  - Count management (O(1) operations)
+  - Service-level statistics tracking
+  - Field name discovery
+- **Implements:** StorageAdapter interface
+- **Key Methods:**
+  - `flushStatistics()` - Write statistics to storage
+  - `isThrottlingError()` - Detect cloud storage throttling
+  - `handleThrottling()` - Exponential backoff
+  - `trackThrottlingEvent()` - Record throttling events
+  - `incrementStatistic()` - Increment service stats
+  - `decrementStatistic()` - Decrement service stats
+
+### 3. Base Storage Implementation
+**File:** `src/storage/baseStorage.ts`
+- **Lines:** 1,098
+- **Type:** Abstract class extending BaseStorageAdapter
+- **Purpose:** Core storage logic (routing, sharding, metadata)
+- **Key Features:**
+  - 2-file system implementation (vectors + metadata)
+  - UUID-based sharding (256 directories)
+  - Metadata routing and separation
+  - Pagination support
+  - Backward compatibility with legacy paths
+- **Implements:** 
+  - Public API (saveNoun, getNoun, etc.)
+  - Metadata operations (saveNounMetadata, etc.)
+- **Key Abstract Methods:**
+  - `saveNoun_internal()` - Adapter-specific noun save
+  - `getNoun_internal()` - Adapter-specific noun read
+  - `writeObjectToPath()` - Generic write
+  - `readObjectFromPath()` - Generic read
+  - `listObjectsUnderPath()` - Listing support
+
+### 4. Storage Factory
+**File:** `src/storage/storageFactory.ts`
+- **Lines:** ~200
+- **Purpose:** Factory function for adapter selection
+- **Function:** `createStorage(options: StorageOptions): Promise<StorageAdapter>`
+- **Selection Logic:**
+  1. Forced memory/filesystem (testing)
+  2. Explicit type selection
+  3. Auto-detection (browser vs Node.js)
+- **Supported Types:**
+  - `'memory'` - MemoryStorage
+  - `'filesystem'` - FileSystemStorage
+  - `'s3'` - S3CompatibleStorage (AWS S3, R2)
+  - `'gcs'` - S3CompatibleStorage (GCS S3 API)
+  - `'gcs-native'` - GcsStorage (native SDK)
+  - `'opfs'` - OPFSStorage (browser)
+
+### 5. Sharding Utilities
+**File:** `src/storage/sharding.ts`
+- **Purpose:** UUID-based sharding helpers
+- **Key Functions:**
+  - `getShardIdFromUuid(id: string): string` - Get first 2 hex chars
+  - `getShardIdByIndex(index: number): string` - Get shard by index
+  - `getAllShardIds(): string[]` - All 256 shard IDs
+- **Constants:**
+  - `TOTAL_SHARDS = 256`
+  - `MIN_SHARD_ID = '00'`
+  - `MAX_SHARD_ID = 'ff'`
+
+### 6. Cache Manager
+**File:** `src/storage/cacheManager.ts`
+- **Purpose:** Generic LRU cache for storage
+- **Type:** Generic class `CacheManager<T>`
+- **Used By:** GcsStorage, S3CompatibleStorage
+- **Features:**
+  - LRU eviction
+  - TTL support
+  - Max size limits
+  - Batch operations
+
+## Storage Adapter Implementations
+
+### FileSystemStorage (Node.js)
+**File:** `src/storage/adapters/fileSystemStorage.ts`
+- **Lines:** 2,677
+- **Type:** Concrete implementation extending BaseStorage
+- **Platform:** Node.js only
+- **Storage Target:** Local file system
+- **Key Features:**
+  - Sharding with automatic migration
+  - File-based locking (multi-process)
+  - O(1) count persistence
+  - HNSW index persistence
+  - Dual-write for migrations
+  - Path depth migration (0→1, 2→1)
+
+**Key Methods:**
+```typescript
+private getNounPath(id: string, depth: number): string
+private getVerbPath(id: string, depth: number): string
+private getNounMetadataPath(id: string, depth: number): string
+async migrateShardingStructure(fromDepth: number, toDepth: number): Promise<void>
+async migrateFromOldStructure(): Promise<void>
+```
+
+**Statistics Tracking:**
+- Persists counts to `_system/counts.json`
+- Tracks noun/verb type distributions
+- Service-level activity timestamps
+- Field name discovery
+
+### MemoryStorage (Testing)
+**File:** `src/storage/adapters/memoryStorage.ts`
+- **Lines:** 822
+- **Type:** Concrete implementation extending BaseStorage
+- **Platform:** Browser & Node.js
+- **Storage Target:** In-memory Maps
+- **Key Features:**
+  - Fast for testing
+  - No persistence (ephemeral)
+  - Full pagination support
+  - Filtering capabilities
+  - 2-file system simulation
+
+**Key Maps:**
+```typescript
+private nouns: Map<string, HNSWNoun>
+private verbs: Map<string, HNSWVerb>
+private objectStore: Map<string, any>  // Unified metadata store
+```
+
+**Methods:**
+- `getNouns()` - Paginated noun listing
+- `getVerbs()` - Paginated verb listing
+- `getMetadataBatch()` - Batch metadata loading
+
+### S3CompatibleStorage (Cloud)
+**File:** `src/storage/adapters/s3CompatibleStorage.ts`
+- **Lines:** 5,000+
+- **Type:** Concrete implementation extending BaseStorage
+- **Platform:** Node.js (server-side)
+- **Storage Targets:**
+  - Amazon S3
+  - Cloudflare R2 (via S3 API)
+  - Google Cloud Storage (via S3 API)
+
+**Key Features:**
+- Adaptive batching (10-1000 items)
+- Request coalescing for deduplication
+- High-volume mode detection
+- Write buffers for bulk operations
+- Socket pool management
+- Backpressure system
+- Change log tracking
+- Cache management (nouns & verbs)
+
+**Performance Optimization:**
+- `nounWriteBuffer` - Batches noun writes
+- `verbWriteBuffer` - Batches verb writes
+- `requestCoalescer` - Deduplicates requests
+- `highVolumeMode` - Activates at >20 pending ops
+- `baseBatchSize` - Adaptive from 10 to 1000
+
+### GcsStorage (Google Cloud Native)
+**File:** `src/storage/adapters/gcsStorage.ts`
+- **Lines:** 1,835
+- **Type:** Concrete implementation extending BaseStorage
+- **Platform:** Node.js (server-side)
+- **Storage Target:** Google Cloud Storage
+- **SDK:** `@google-cloud/storage` (native)
+
+**Key Features:**
+- Application Default Credentials (ADC)
+- Service Account Key File support
+- Service Account Credentials Object
+- HMAC Keys (backward compatible)
+- Multi-level cache managers
+- Backpressure management
+- High-volume mode
+- Request coalescing
+
+**Authentication Priority:**
+1. ADC (Application Default Credentials)
+2. Service Account Key File
+3. Service Account Credentials
+4. HMAC Keys
+
+**Cache Managers:**
+```typescript
+private nounCacheManager: CacheManager<HNSWNode>
+private verbCacheManager: CacheManager<Edge>
+```
+
+### OPFSStorage (Browser Storage)
+**File:** `src/storage/adapters/opfsStorage.ts`
+- **Type:** Concrete implementation extending BaseStorage
+- **Platform:** Browser only
+- **Storage Target:** Origin Private File System (OPFS)
+- **Fallback:** MemoryStorage if OPFS unavailable
+
+**Browser Compatibility:**
+- Chrome 96+
+- Edge 96+
+- Safari 15.1+
+
+## Storage Patterns and Utilities
+
+### Backward Compatibility
+**File:** `src/storage/backwardCompatibility.ts`
+- **Purpose:** Handle legacy storage paths
+- **Features:**
+  - Path migration detection
+  - Dual-write during transition
+  - Graceful fallback to old locations
+  - Read-from-new, fallback-to-old
+
+### Metadata Index
+**File:** `src/utils/metadataIndex.ts`
+- **Purpose:** Build searchable indexes from metadata
+- **Used By:** Brainy for fast metadata queries
+- **Features:**
+  - Field name discovery
+  - Standard field mapping
+  - Service-level statistics
+
+### Storage Discovery (Distributed)
+**File:** `src/distributed/storageDiscovery.ts`
+- **Purpose:** Discover storage config in distributed systems
+- **Features:**
+  - Node coordination
+  - Storage synchronization
+
+### Adaptive Backpressure
+**File:** `src/utils/adaptiveBackpressure.ts`
+- **Purpose:** Flow control for storage operations
+- **Used By:** All cloud storage adapters
+- **Features:**
+  - Request queuing
+  - Throttling detection
+  - Backoff scheduling
+
+### Write Buffer
+**File:** `src/utils/writeBuffer.ts`
+- **Purpose:** Batch write operations
+- **Used By:** S3, GCS adapters
+- **Features:**
+  - Configurable batch size
+  - Flush on timeout
+  - Deduplication
+
+### Request Coalescer
+**File:** `src/utils/requestCoalescer.ts`
+- **Purpose:** Deduplicate concurrent requests
+- **Used By:** S3, GCS adapters
+- **Features:**
+  - Request deduplication
+  - Batch processing
+
+## Integration Points
+
+### In Brainy.ts (Main Class)
+```typescript
+private storage!: BaseStorage
+
+async init(): Promise<void> {
+  // Create storage from factory
+  const storageAdapter = await createStorage(this.config.storage)
+  this.storage = storageAdapter as BaseStorage
+  
+  // Initialize
+  await this.storage.init()
+  
+  // Pass to HNSW index
+  this.index = new HNSWIndex(this.storage, ...)
+}
+```
+
+### In HNSW Index
+```typescript
+export class HNSWIndex {
+  constructor(private storage: StorageAdapter, ...)
+  
+  // Uses storage for node/edge persistence
+  async saveNode(noun: HNSWNoun): Promise<void>
+  async getNode(id: string): Promise<HNSWNoun>
+}
+```
+
+### In Metadata Index
+```typescript
+export class MetadataIndexManager {
+  constructor(storage: StorageAdapter, ...)
+  
+  // Uses storage for metadata queries
+  async getNounsByFilter(filter: Filter): Promise<HNSWNoun[]>
+}
+```
+
+## Data Flow
+
+### Saving a Noun
+```
+Brainy.add()
+  ↓
+HNSWIndex.insert()
+  ↓
+BaseStorage.saveNoun()
+  ├─ saveNoun_internal() → adapter-specific
+  ├─ saveNounMetadata() → path routing
+  └─ updateStatistics()
+
+FileSystemStorage.saveNoun_internal()
+  ├─ Create shard directory (ab/)
+  ├─ Write JSON file
+  └─ Update counts
+```
+
+### Querying Nouns
+```
+Brainy.search()
+  ↓
+HNSW.search()
+  ↓
+BaseStorage.getNoun()
+  ├─ getNoun_internal() → adapter-specific
+  └─ getNounMetadata() → path routing
+
+S3CompatibleStorage.getNoun_internal()
+  ├─ Check cache
+  ├─ Download from S3
+  ├─ Parse JSON
+  └─ Update cache
+```
+
+## Storage Statistics Tracking
+
+### Stored in `_system/statistics.json`
+```json
+{
+  "nounCount": { "Person": 5, "Company": 2 },
+  "verbCount": { "knows": 10, "works_at": 3 },
+  "metadataCount": { "user-service": 50 },
+  "hnswIndexSize": 15,
+  "totalNodes": 7,
+  "totalEdges": 13,
+  "services": [
+    { "name": "user-service", "totalNouns": 5, ... }
+  ],
+  "lastUpdated": "2024-10-15T..."
+}
+```
+
+### Stored in `_system/counts.json`
+```json
+{
+  "totalNounCount": 7,
+  "totalVerbCount": 13,
+  "entityCounts": { "Person": 5, "Company": 2 },
+  "verbCounts": { "knows": 10, "works_at": 3 }
+}
+```
+
+## Type Definitions
+
+### HNSWNoun (Vector + HNSW)
+```typescript
+interface HNSWNoun {
+  id: string
+  vector: number[]
+  connections: Map<number, Set<string>>  // level → node IDs
+  level: number
+  metadata?: any  // Optional in vector file, separate file system
+}
+```
+
+### GraphVerb (Relationship)
+```typescript
+interface GraphVerb {
+  id: string
+  sourceId: string
+  targetId: string
+  vector: number[]
+  type?: string
+  weight?: number
+  metadata?: any
+  // Plus aliases: source, target, verb, embedding
+  createdAt?: Timestamp
+  createdBy?: { augmentation: string; version: string }
+}
+```
+
+## Summary Statistics
+
+| Component | File | Lines | Purpose |
+|-----------|------|-------|---------|
+| StorageAdapter Interface | coreTypes.ts | 250 | Interface definition |
+| BaseStorageAdapter | baseStorageAdapter.ts | 1,156 | Common functionality |
+| BaseStorage | baseStorage.ts | 1,098 | Core routing & pagination |
+| storageFactory | storageFactory.ts | 200 | Adapter selection |
+| FileSystemStorage | fileSystemStorage.ts | 2,677 | Node.js FS |
+| MemoryStorage | memoryStorage.ts | 822 | In-memory (test) |
+| S3CompatibleStorage | s3CompatibleStorage.ts | 5,000+ | AWS S3 / R2 / GCS |
+| GcsStorage | gcsStorage.ts | 1,835 | Google Cloud native |
+| sharding.ts | sharding.ts | ~100 | UUID sharding |
+| cacheManager.ts | cacheManager.ts | ~200 | LRU cache |
+| **TOTAL** | | **~13,000+** | **Complete storage system** |
+
+## Conclusion
+
+Brainy's storage architecture is:
+- Well-layered (Interface → Abstract → Concrete)
+- Extensible (factory pattern)
+- Flexible (multiple backends)
+- Scalable (sharding, caching, batching)
+- Type-safe (full TypeScript support)
+
+New adapters like TypeAwareStorageAdapter simply extend BaseStorage and implement 17 abstract methods.

docs/ADR-001-generational-mvcc.md

@@ -1,295 +0,0 @@
-# ADR-001: Generational MVCC storage and the immutable Db API
-
-**Status:** Accepted (ships in 8.0)
-**Date:** 2026-06-10
-
-## Context
-
-Before 8.0, Brainy carried two overlapping version-control subsystems: a
-copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a
-separate versioning subsystem (`versions.save/list/compare/restore/prune`),
-plus a read-only historical adapter for commit-based time travel. Together
-they were ~5,100 LOC of mechanism for one product need: *read a consistent
-past state while the store keeps moving, and snapshot/restore cheaply.*
-
-Neither subsystem gave a precise isolation guarantee. Reads raced in-place
-JSON overwrites, so a "snapshot" was only as immutable as the bytes it
-happened to share with the live store.
-
-8.0 replaces both with **one mechanism**: generational MVCC over immutable,
-generation-stamped records, exposed through a Datomic-style immutable
-database value (`Db`). The same model is implemented natively by versioned
-index providers (LSM snapshots), so semantics are identical on the pure-JS
-path and the native path.
-
-## Decision
-
-### The model
-
-- A **monotonic u64 generation counter** is the store's logical clock. It
-  advances once per committed `transact()` batch and once per
-  single-operation write (`add`/`update`/`remove`/`relate`/…), so
-  `brain.generation()` is always a meaningful watermark. It is persisted in
-  `_system/generation.json` and never reissued for anything durable.
-- `brain.now()` **pins** the current generation in O(1) and returns a `Db` —
-  an immutable view. Pins are refcounted; `db.release()` (with a
-  `FinalizationRegistry` backstop for leaked values) ends the pin.
-- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative
-  batch atomically as **exactly one generation**, with whole-store
-  compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and
-  reified transaction metadata appended to `_system/tx-log.jsonl`.
-- `brain.asOf(generation | Date | snapshotPath)` opens past state;
-  `db.with(ops)` layers a speculative in-memory overlay (never touching
-  disk, the counter, or index providers); `db.persist(path)` cuts an
-  instant snapshot; `brain.restore(path, { confirm: true })` replaces state
-  from one; `Brainy.load(path)` opens a snapshot read-only with the full
-  query surface.
-
-### Persisted layout
-
-All paths are storage-root-relative:
-
-```
-_system/generation.json          { generation, updatedAt }       atomic tmp+rename
-_system/manifest.json            { version, generation,          atomic tmp+rename
-                                   committedAt, horizon }        (the commit point)
-_system/tx-log.jsonl             one line per committed          append-only
-                                 transact: { generation,
-                                 timestamp, meta? }
-_generations/<N>/tx.json         the generation-N delta:         immutable
-                                 touched noun/verb ids + meta
-_generations/<N>/prev/<id>.json  before-image of <id> as of      immutable
-                                 commit N (raw stored bytes;
-                                 null parts = file was absent)
-```
-
-**Why per-generation deltas instead of a global `id → latest generation`
-map in the manifest:** a global map makes every commit O(all ids) — the
-whole map must be rewritten to swap it atomically. The delta layout makes a
-commit O(ids touched) and keeps the manifest a fixed-size watermark, while
-point-in-time resolution stays correct (see "Read resolution" below). The
-trade is that resolution at a pinned generation scans the deltas of later
-commits — bounded by the number of commits since the pin, which is exactly
-the window compaction keeps short.
-
-Before-images are deliberately the *only* per-id records. They serve both
-roles the layer needs — the crash-recovery undo log and the point-in-time
-read source. After-images would duplicate state that is already readable
-(the canonical entity files hold the latest bytes; earlier states resolve
-from later before-images) and would double record I/O per commit.
-
-### Commit protocol (durability)
-
-`transact()` commits under a store-wide mutex:
-
-1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError`
-   before anything is staged.
-2. **Reserve** generation `N` (counter increment).
-3. **Stage the undo log:** write the before-image of every touched id plus
-   `tx.json` under `_generations/N/`, then **fsync** the files and their
-   directories. From this point, any crash is recoverable to the exact
-   pre-transaction bytes.
-4. **Execute** the planned batch through the TransactionManager (which has
-   its own operation-level rollback for in-flight failures).
-5. **Commit point:** persist the counter, then write `_system/manifest.json`
-   via atomic tmp+rename and fsync it. The rename *is* the commit: a
-   generation directory is committed if and only if `N ≤
-   manifest.generation`.
-6. Append the tx-log line (advisory metadata — a crash between 5 and 6
-   keeps the transaction).
-
-**Crash recovery (on open):** any `_generations/<N>` directory with
-`N > manifest.generation` is an uncommitted transaction. Its before-images
-are restored to the canonical paths (idempotently — recovery itself can
-crash and rerun) and the directory is removed. Because recovery runs before
-any index is built, and a recovery that rolled something back forces a full
-index rebuild, derived indexes never observe rolled-back state. Reader-mode
-instances skip recovery (readers never write; the next writer repairs).
-
-A failed (non-crash) transaction takes the same staging directory down the
-abort path: the TransactionManager rolls back applied operations, the
-staging directory is removed, and the generation reservation is returned —
-a failed batch leaves the generation counter unchanged.
-
-### Read resolution at a pinned generation
-
-The state of id X at pinned generation G is:
-
-- the before-image stored by the **first committed generation after G that
-  touched X**, or
-- the live canonical bytes, when nothing after G touched X.
-
-While nothing has committed past G, *every* read on the `Db` delegates to
-the live fast paths untouched — `now()` adds no read overhead until history
-actually moves.
-
-**Two read paths, one result set.** `get()`, metadata-level `find()`, and
-filter-based `related()` resolve directly through the record layer at any
-reachable pinned generation — no extra cost beyond scanning the deltas of
-later commits. Index-accelerated dimensions (semantic/vector search, graph
-traversal, cursors, aggregation) are served by **at-generation index
-materialization**: the first such query on a historical `Db` copies the
-exact at-G record set (live bytes for ids untouched since the pin,
-before-images for the rest; a final reconciliation pass runs under the
-commit mutex so transactions racing the copy cannot skew it) into an
-ephemeral in-memory store and opens a read-only engine over it — the same
-vector/metadata/graph index classes the live brain uses, sharing the host's
-embedder and aggregate definitions. The handle is cached on the `Db` and
-freed by `release()`.
-
-**Cost, stated plainly:** materialization is O(n at G) time and memory,
-once per `Db`. That is the open-core price of historical index queries. A
-native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
-retained LSM segments) serves the same reads with no rebuild at all — the
-materializer is the correctness baseline, the provider is the accelerator.
-
-**The one remaining boundary.** Speculative `with()` overlays throw
-`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
-overlay entities carry no embeddings (`with()` never invokes the embedder),
-so a "full" index query over an overlay would silently exclude the
-overlay's own entities. Commit with `transact()` to get the full surface.
-
-**History granularity (Model-B).** EVERY write is its own immutable generation
-— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
-Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/
-`diff()`/`history()` exactly like transacts; a pin always freezes against later
-writes. `transact()` groups several operations into ONE atomic generation.
-
-Single-op history durability is **async group-commit**: the live write hits
-canonical storage immediately (acknowledged), while its before-image is buffered
-and persisted to disk in one batched fsync on a size/timer trigger (or forced by
-`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in
-point-in-time resolution exactly like on-disk generations, so the synchronous
-`now()` freezes with no forced flush. A hard crash before the flush loses only
-the buffered *history* of the last window — never live data — and a crash
-*mid-flush* is recovered by **drop-without-restore** (the partial generation's
-before-images are discarded, never replayed, because the live write was already
-acknowledged; restoring them would silently revert it).
-
-### Pinning, retention, compaction
-
-- Each live `Db` holds one refcounted pin on its generation (plus a
-  `pin(generation)` on every registered `VersionedIndexProvider`, whose
-  explicit pin lifetime overrides any time-based snapshot retention the
-  provider has).
-- The constructor **`retention`** knob governs auto-compaction (on `flush()`/
-  `close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config;
-  driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) ·
-  `'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit
-  CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
-  manually on the same caps — the oldest unpinned record-sets are reclaimed
-  while ANY supplied cap is exceeded.
-- A record-set `N` is reclaimed only when `N` is at or below **every** live
-  pin — deleting `N` can only break readers pinned *below* `N`, because
-  resolution reads before-images from generations strictly greater than the
-  pin. Live pins are ALWAYS exempt, in every retention mode.
-- The manifest records the **horizon** (highest reclaimed generation).
-  Generations below the horizon are unreachable; `asOf()` on them throws
-  `GenerationCompactedError`. The horizon itself stays reachable, resolved
-  from the record-sets above it. To keep a generation readable forever,
-  `persist()` it first — snapshots are self-contained.
-
-### Snapshots and restore
-
-`db.persist(path)` flushes indexes, then cuts the snapshot under the
-store's commit mutex (no commit, compaction, or counter write can
-interleave). On filesystem storage it is a **hard-link farm**: every data
-file is immutable-by-rename, so linking is safe — later rewrites swap
-inodes and the snapshot keeps the old bytes. The two exceptions are handled
-explicitly: the append-in-place tx-log is byte-copied, and process-local
-lock state is excluded. Cross-device targets (and filesystems that refuse
-links) fall back to per-file byte copies. In-memory stores serialize to the
-same directory layout, so persisting a memory brain produces a real,
-durable, loadable store.
-
-`persist()` requires the view to still be the store's latest generation
-(a snapshot captures current bytes); a view that history has moved past
-throws rather than persisting the wrong state.
-
-`restore(path, { confirm: true })` replaces the store's contents from a
-snapshot via byte copy (never links — the snapshot stays independent),
-reloads all adapter-internal derived state, rebuilds all indexes, and
-floors the generation counter at its pre-restore value so observed
-generation numbers are never reissued. Live pins do not survive a restore;
-a warning is logged when any exist.
-
-### Versioned index providers
-
-Native index providers may implement the optional 4-method
-`VersionedIndexProvider` capability (`generation()`,
-`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the
-boundary). The locked consistency model: providers are **post-commit
-appliers**. The storage-record commit is the source of truth; provider
-index state is derived. On open, a provider behind the committed watermark
-replays the gap from storage (or requests a rebuild) — there are no
-provider rollback hooks, because uncommitted transactions are repaired at
-the storage layer before any index opens. Speculative `with()` overlays
-never reach providers.
-
-## Guarantees (and their proofs)
-
-Each stated guarantee has a test that proves it, not merely exercises it
-(`tests/integration/db-mvcc.test.ts`, plus
-`tests/unit/db/generationStore.test.ts` for the record layer in isolation):
-
-| Guarantee | Proof |
-|---|---|
-| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) |
-| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) |
-| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) |
-| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c |
-| Compaction never breaks a pinned read; release enables reclaim | proof 5 |
-| `with()` overlays touch nothing durable | proof 6 |
-| Generation monotonicity across close/reopen | proof 7 |
-| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) |
-| Balanced provider pin/release lockstep | proof 9 |
-
-One deliberate softness: single-operation generation bumps persist the
-counter coalesced (per write burst), not per write. Durable artifacts —
-records, manifests, snapshots — always persist the counter synchronously at
-their own commit points, so a crash inside the coalescing window can lose
-only counter values that nothing durable ever referenced.
-
-## Failure modes
-
-| Failure | Outcome |
-|---|---|
-| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. |
-| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. |
-| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. |
-| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
-| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
-| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
-| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
-| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
-| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
-
-## Lineage
-
-The design is an assembly of well-understood prior art, chosen for being
-boring where it counts:
-
-- **Datomic** — the database-as-a-value: an immutable `Db` you query, with
-  `with()` for speculation and reified transaction metadata instead of
-  commit messages.
-- **LMDB** — reader pins: readers never block writers; a reader's view
-  stays valid because nothing overwrites the pages (here: records) it
-  references; reclamation waits for the last reader.
-- **LSM trees / Cassandra** — immutable segments make snapshots hard links
-  and make compaction a retention policy instead of a locking problem.
-
-## Consequences
-
-- One mechanism replaces the COW and versioning subsystems (their removal
-  is the companion change to this ADR).
-- In-place branch switching (`checkout`) is gone by design; the replacement
-  is opening a persisted snapshot as a separate instance — a name→path
-  mapping where a product needs named branches.
-- Every commit pays O(ids touched) extra writes (before-images + delta +
-  manifest). Single-operation writes pay only an in-memory counter bump
-  with coalesced persistence.
-- The full query surface works at every reachable pinned generation.
-  Record-path reads (`get`, metadata `find`, filter `related`) are
-  effectively free; index-accelerated historical queries pay a one-time
-  O(n at G) materialization per `Db` on the open-core path (freed on
-  `release()`), and run rebuild-free on a native `VersionedIndexProvider`.

docs/API_DECISION_TREE.md

@@ -0,0 +1,481 @@
+# 🧠 Brainy API Decision Tree
+
+*Choose the right API for your use case with confidence*
+
+This guide helps you navigate Brainy's comprehensive API surface by asking the right questions to find the perfect method for your specific needs.
+
+## 🎯 Quick Start: What do you want to do?
+
+### 📝 **Adding Data**
+- **Single entity** → [`brainy.add()`](#adding-single-entities)
+- **Multiple entities** → [`brainy.addMany()`](#adding-multiple-entities)
+- **Streaming/real-time data** → [Streaming Pipeline](#streaming-data)
+
+### 🔍 **Finding Data**
+- **Natural language search** → [`brainy.find("search query")`](#natural-language-search)
+- **Structured/filtered search** → [`brainy.find({ query, where, type })`](#structured-search)
+- **Similar entities** → [`brainy.similar()`](#similarity-search)
+- **Get by ID** → [`brainy.get()`](#retrieval-by-id)
+
+### 🔗 **Relationships**
+- **Create relationships** → [`brainy.relate()`](#creating-relationships)
+- **Query relationships** → [`brainy.getRelations()`](#querying-relationships)
+- **Graph traversal** → [Graph Navigation](#graph-operations)
+
+### 📊 **Advanced Features**
+- **File management** → [VFS (Virtual File System)](#file-operations)
+- **AI-powered analysis** → [Neural API](#neural-analysis)
+- **Clustering/insights** → [Intelligence Systems](#intelligence-systems)
+
+---
+
+## 🔀 Decision Tree Flow
+
+```mermaid
+graph TD
+    A[What are you trying to do?] --> B[Store Data]
+    A --> C[Find Data]
+    A --> D[Manage Relationships]
+    A --> E[Work with Files]
+    A --> F[AI Analysis]
+
+    B --> B1[Single Item]
+    B --> B2[Multiple Items]
+    B --> B3[Real-time Stream]
+
+    C --> C1[I know the ID]
+    C --> C2[Natural language query]
+    C --> C3[Complex filters]
+    C --> C4[Find similar items]
+
+    D --> D1[Create relationship]
+    D --> D2[Query relationships]
+    D --> D3[Graph traversal]
+
+    E --> E1[File operations]
+    E --> E2[Knowledge-enhanced files]
+
+    F --> F1[Clustering]
+    F --> F2[Similarity analysis]
+    F --> F3[Insights generation]
+```
+
+---
+
+## 📝 Adding Data
+
+### Adding Single Entities
+
+**Use `brainy.add()` when:**
+- Adding one entity at a time
+- You need the ID immediately for further operations
+- Working with user input or real-time data
+
+```typescript
+// ✅ Perfect for single entities
+const id = await brainy.add({
+  data: "New research paper on quantum computing",
+  type: NounType.Document,
+  metadata: { category: "research", priority: "high" }
+})
+```
+
+**Decision factors:**
+- **Single item?** → `add()`
+- **Need immediate ID?** → `add()`
+- **Interactive application?** → `add()`
+
+### Adding Multiple Entities
+
+**Use `brainy.addMany()` when:**
+- Bulk importing data
+- Processing batches (>10 items)
+- Performance is critical
+
+```typescript
+// ✅ Perfect for bulk operations
+const result = await brainy.addMany({
+  items: documents.map(doc => ({
+    data: doc.content,
+    type: NounType.Document,
+    metadata: doc.metadata
+  })),
+  chunkSize: 100,
+  parallel: true
+})
+```
+
+**Decision factors:**
+- **Multiple items (>10)?** → `addMany()`
+- **Batch processing?** → `addMany()`
+- **Can tolerate some failures?** → `addMany()` with `continueOnError: true`
+
+### Streaming Data
+
+**Use Streaming Pipeline when:**
+- Real-time data ingestion
+- Processing large datasets that don't fit in memory
+- Need transformation during ingestion
+
+```typescript
+// ✅ Perfect for streaming
+const pipeline = brainy.streaming.pipeline()
+  .transform(data => ({ ...data, processed: true }))
+  .batch(50)
+  .into(brainy)
+```
+
+---
+
+## 🔍 Finding Data
+
+### Natural Language Search
+
+**Use `brainy.find("query string")` when:**
+- User is typing search queries
+- You want semantic understanding
+- Building search interfaces
+
+```typescript
+// ✅ Perfect for user searches
+const results = await brainy.find("documents about machine learning")
+```
+
+**Decision factors:**
+- **User-generated query?** → Natural language `find()`
+- **Semantic understanding needed?** → Natural language `find()`
+- **Search interface?** → Natural language `find()`
+
+### Structured Search
+
+**Use `brainy.find({ query, where, type })` when:**
+- Complex filtering requirements
+- Combining text search with metadata filters
+- Performance-critical searches
+
+```typescript
+// ✅ Perfect for complex queries
+const results = await brainy.find({
+  query: "neural networks",
+  type: NounType.Document,
+  where: {
+    status: "published",
+    year: { $gte: 2020 }
+  },
+  limit: 20
+})
+```
+
+**Decision factors:**
+- **Need metadata filtering?** → Structured `find()`
+- **Performance critical?** → Structured `find()`
+- **Complex criteria?** → Structured `find()`
+
+### Similarity Search
+
+**Use `brainy.similar()` when:**
+- Finding "more like this" content
+- Recommendation systems
+- Duplicate detection
+
+```typescript
+// ✅ Perfect for recommendations
+const similar = await brainy.similar({
+  to: "document-id-123",
+  limit: 10,
+  type: NounType.Document
+})
+```
+
+**Decision factors:**
+- **"More like this" feature?** → `similar()`
+- **Recommendations?** → `similar()`
+- **Duplicate detection?** → `similar()`
+
+### Retrieval by ID
+
+**Use `brainy.get()` when:**
+- You know the exact ID
+- Loading specific entities
+- Following relationships
+
+```typescript
+// ✅ Perfect for direct access
+const entity = await brainy.get("known-id-123")
+```
+
+**Decision factors:**
+- **Known ID?** → `get()`
+- **Direct access needed?** → `get()`
+- **Following relationships?** → `get()`
+
+---
+
+## 🔗 Relationships
+
+### Creating Relationships
+
+**Use `brainy.relate()` when:**
+- Connecting two entities
+- Building knowledge graphs
+- Modeling real-world relationships
+
+```typescript
+// ✅ Perfect for connections
+await brainy.relate({
+  from: "user-123",
+  to: "project-456",
+  type: VerbType.WorksOn,
+  metadata: { role: "lead", since: "2024-01-01" }
+})
+```
+
+**Decision factors:**
+- **Connecting entities?** → `relate()`
+- **Need relationship metadata?** → `relate()`
+- **Building graphs?** → `relate()`
+
+### Querying Relationships
+
+**Use `brainy.getRelations()` when:**
+- Finding all connections for an entity
+- Exploring relationship patterns
+- Building relationship views
+
+```typescript
+// ✅ Perfect for relationship queries
+const relations = await brainy.getRelations({
+  from: "user-123",
+  type: VerbType.WorksOn
+})
+```
+
+---
+
+## 📁 File Operations
+
+### Basic File Operations
+
+**Use VFS when:**
+- Managing files and directories
+- Need hierarchical structure
+- Building file explorers
+
+```typescript
+// ✅ Perfect for file management
+const vfs = brainy.vfs({ storage: 'filesystem' })
+await vfs.writeFile('/docs/readme.md', 'content')
+const files = await vfs.getDirectChildren('/docs')
+```
+
+**Decision factors:**
+- **File management?** → VFS
+- **Directory structure?** → VFS
+- **File explorer interface?** → VFS
+
+### Intelligent File Management
+
+**Use VFS (Semantic VFS) when:**
+- Need semantic file search
+- Want AI-powered concept extraction
+- Building smart file systems
+- Require multi-dimensional file access
+
+```typescript
+// ✅ Perfect for intelligent file systems
+const knowledgeVFS = await vfs.withKnowledge(brainy)
+const insights = await knowledgeVFS.getFileInsights('/project')
+```
+
+---
+
+## 🧠 AI Analysis
+
+### Clustering
+
+**Use Neural API clustering when:**
+- Discovering data patterns
+- Organizing large datasets
+- Creating automatic categories
+
+```typescript
+// ✅ Perfect for pattern discovery
+const neural = brainy.neural()
+const clusters = await neural.cluster({
+  entities: entityIds,
+  k: 5,
+  method: 'hierarchical'
+})
+```
+
+### Intelligence Systems
+
+**Use Triple Intelligence when:**
+- Complex multi-criteria searches
+- Advanced relationship queries
+- Performance-critical operations
+
+```typescript
+// ✅ Perfect for complex queries
+const intelligence = brainy.getTripleIntelligence()
+const results = await intelligence.query({
+  vector: queryVector,
+  metadata: { category: 'research' },
+  graph: { connected: 'user-123' }
+})
+```
+
+---
+
+## 🚀 Performance Optimization Guide
+
+### When Performance Matters
+
+| Scenario | Best Choice | Why |
+|----------|-------------|-----|
+| **Bulk Import** | `addMany()` | Batched operations, parallel processing |
+| **Metadata-only Search** | `find({ where: {...} })` | Skips vector computation |
+| **Known ID Access** | `get()` | Direct index lookup |
+| **Large Result Sets** | Pagination with `offset`/`limit` | Memory efficient |
+| **Real-time Streams** | Streaming Pipeline | Memory efficient, scalable |
+
+### Memory Usage Optimization
+
+```typescript
+// ❌ Memory intensive
+const allResults = await brainy.find({ limit: 10000 })
+
+// ✅ Memory efficient
+for (let offset = 0; offset < total; offset += 100) {
+  const batch = await brainy.find({
+    query: "...",
+    limit: 100,
+    offset
+  })
+  await processBatch(batch)
+}
+```
+
+---
+
+## 🎯 Common Use Case Patterns
+
+### Building a Search Interface
+
+```typescript
+// User types query → Natural language search
+const searchResults = await brainy.find(userQuery)
+
+// User applies filters → Structured search
+const filteredResults = await brainy.find({
+  query: userQuery,
+  where: selectedFilters,
+  type: selectedTypes
+})
+
+// User clicks "more like this" → Similarity search
+const similar = await brainy.similar({ to: selectedId })
+```
+
+### Building a Recommendation System
+
+```typescript
+// 1. Get user's interaction history
+const user = await brainy.get(userId)
+
+// 2. Find similar users
+const similarUsers = await brainy.similar({ to: userId, type: NounType.Person })
+
+// 3. Get their liked content
+const recommendations = []
+for (const similarUser of similarUsers) {
+  const relations = await brainy.getRelations({
+    from: similarUser.id,
+    type: VerbType.Likes
+  })
+  recommendations.push(...relations)
+}
+```
+
+### Building a Knowledge Graph
+
+```typescript
+// 1. Add entities
+const entities = await Promise.all([
+  brainy.add({ data: "Person: Alice", type: NounType.Person }),
+  brainy.add({ data: "Company: TechCorp", type: NounType.Organization }),
+  brainy.add({ data: "Project: AI Assistant", type: NounType.Thing })
+])
+
+// 2. Create relationships
+await brainy.relate({
+  from: entities[0], // Alice
+  to: entities[1],   // TechCorp
+  type: VerbType.WorksFor
+})
+
+await brainy.relate({
+  from: entities[0], // Alice
+  to: entities[2],   // AI Assistant
+  type: VerbType.WorksOn
+})
+
+// 3. Query the graph
+const aliceConnections = await brainy.getRelations({ from: entities[0] })
+```
+
+---
+
+## 🔧 Migration Guide
+
+### From v2.x to v3.x APIs
+
+| v2.x (Deprecated) | v3.x (Current) | When to Use |
+|-------------------|----------------|-------------|
+| `brain.store()` | `brainy.add()` | Adding entities |
+| `brain.search()` | `brainy.find()` | Searching content |
+| `brain.query()` | `brainy.find({ ... })` | Complex queries |
+| `brain.similar()` | `brainy.similar()` | ✅ Same API |
+| `brain.connect()` | `brainy.relate()` | Creating relationships |
+
+### Legacy Type Migration
+
+```typescript
+// ❌ v2.x way
+import { ISenseAugmentation } from '@soulcraft/brainy/types/augmentations'
+
+// ✅ v3.x way
+import { BrainyAugmentation } from '@soulcraft/brainy'
+```
+
+---
+
+## 🎪 Decision Quick Reference
+
+**Need to add data?**
+- 1 item → `add()`
+- Many items → `addMany()`
+- Streaming → Pipeline
+
+**Need to find data?**
+- Know ID → `get()`
+- Natural search → `find("query")`
+- Complex filters → `find({ query, where })`
+- Similar items → `similar()`
+
+**Need relationships?**
+- Create → `relate()`
+- Query → `getRelations()`
+- Complex graph → Triple Intelligence
+
+**Need files?**
+- Basic → VFS (standard operations)
+- Smart → Semantic VFS (6 dimensional access + neural extraction)
+
+**Need AI analysis?**
+- Patterns → Neural clustering
+- Complex queries → Triple Intelligence
+
+---
+
+*This guide covers 95% of use cases. For edge cases or custom requirements, check the [Core API Patterns](./CORE_API_PATTERNS.md) and [Neural API Patterns](./NEURAL_API_PATTERNS.md) guides.*

docs/BATCHING.md

@@ -1,468 +0,0 @@
----
-title: Batch Operations
-slug: guides/batching
-public: true
-category: guides
-template: guide
-order: 5
-description: Eliminate N+1 query patterns with batchGet() and storage-level batch APIs for fast multi-entity reads against filesystem and memory storage.
-next:
-  - api/reference
-  - guides/find-system
----
-
-# Batch Operations API
-> **Production-Ready** | Zero N+1 Query Patterns
-
-## Overview
-
-Brainy provides batch operations at the storage layer to eliminate N+1 query patterns for VFS operations, relationship queries, and entity retrieval.
-
-### Problem Solved
-
-The naive pattern of looping and calling `brain.get(id)` once per item issues sequential reads through the storage layer. Batched APIs collapse that into a single read pass.
-
-**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`** at the VFS layer and the explicit `batchGet()` / `getNounMetadataBatch()` calls — not to `readFile()` or individual `get()` operations.
-
----
-
-## New Public APIs
-
-### 1. `brain.batchGet(ids, options?)`
-
-Batch retrieval of multiple entities (metadata-only by default).
-
-```typescript
-// Fetch multiple entities in a single batched operation
-const ids = ['id1', 'id2', 'id3']
-const results: Map<string, Entity> = await brain.batchGet(ids)
-
-// With vectors (falls back to individual gets)
-const resultsWithVectors = await brain.batchGet(ids, { includeVectors: true })
-
-// Results map
-results.get('id1') // → Entity or undefined
-results.size // → 3 (number of found entities)
-```
-
-**Performance:**
-- Memory storage: Instant (parallel reads)
-- Filesystem storage: Parallel reads, scales with available IOPS
-
-**Use Cases:**
-- Loading multiple entities for display
-- Bulk data export operations
-- Relationship traversal (fetch all connected entities)
-
----
-
-## Storage-Level APIs
-
-### 2. `storage.getNounMetadataBatch(ids)`
-
-Batch metadata retrieval with direct O(1) path construction.
-
-```typescript
-const storage = brain.storage as BaseStorage
-const ids = ['id1', 'id2', 'id3']
-
-const metadataMap: Map<string, NounMetadata> = await storage.getNounMetadataBatch(ids)
-
-for (const [id, metadata] of metadataMap) {
- console.log(metadata.noun) // Type: 'document', 'person', etc.
- console.log(metadata.data) // Entity data
-}
-```
-
-**Features:**
-- ✅ Direct O(1) path construction from ID (no type lookup needed!)
-- ✅ Sharding preservation (all paths include `{shard}/{id}`)
-- ✅ Write-cache coherent (read-after-write consistency)
-- ✅ O(1) path construction — eliminates the per-entity type search of the old type-first layout
-
-**Performance:**
-- Constant-time path construction per ID — no type-cache misses
-- Filesystem: parallel reads bounded by IOPS
-- No type search delays — every ID maps directly to storage path
-
----
-
-### 3. `storage.getVerbsBySourceBatch(sourceIds, verbType?)`
-
-Batch relationship queries by source entity IDs.
-
-```typescript
-const storage = brain.storage as BaseStorage
-
-// Get all relationships from multiple sources
-const results: Map<string, GraphVerb[]> = await storage.getVerbsBySourceBatch([
- 'person1',
- 'person2'
-])
-
-// Filter by verb type
-const createsResults = await storage.getVerbsBySourceBatch(
- ['person1', 'person2'],
- 'creates'
-)
-
-// Process results
-for (const [sourceId, verbs] of results) {
- console.log(`${sourceId} has ${verbs.length} relationships`)
- verbs.forEach(verb => {
- console.log(` → ${verb.verb} → ${verb.targetId}`)
- })
-}
-```
-
-**Use Cases:**
-- Social graph traversal (fetch all connections for multiple users)
-- Knowledge graph queries (find all relationships of specific type)
-- Bulk export of relationship data
-
-**Performance:**
-- Memory storage: single in-memory pass over the metadata index
-- Filesystem storage: parallel reads through the metadata index
-
----
-
-## VFS Integration
-
-VFS operations automatically use batch APIs for maximum performance.
-
-### Directory Traversal
-
-```typescript
-// Tree traversal uses batched reads under the hood
-const tree = await brain.vfs.getTreeStructure('/my-dir')
-// ✅ PathResolver.getChildren() uses brain.batchGet() internally
-// ✅ Parallel traversal of directories at the same tree level
-// ✅ 2-3 batched calls instead of 22 sequential calls
-```
-
-**Architecture:**
-
-```
-VFS.getTreeStructure()
- ↓ PARALLEL (breadth-first traversal)
- → PathResolver.getChildren() [all dirs at level processed in parallel]
- ↓ BATCHED
- → brain.batchGet(childIds) [1 call instead of N]
- ↓ BATCHED
- → storage.getNounMetadataBatch(ids) [1 call instead of N]
- ↓ ADAPTER
- → Filesystem: Promise.all() parallel reads
- → Memory: Promise.all() parallel reads
-```
-
----
-
-## Advanced Features Compatibility
-
-### ✅ ID-First Storage Architecture
-
-All batch operations use direct ID-first paths - no type lookup needed!
-
-**ID-First Path Structure:**
-```
-entities/nouns/{SHARD}/{ID}/metadata.json
-entities/verbs/{SHARD}/{ID}/metadata.json
-```
-
-**Direct O(1) Path Construction:**
-```typescript
-// Every ID maps directly to exactly ONE path - O(1), no type search
-const id = 'abc-123'
-const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
-const path = `entities/nouns/${shard}/${id}/metadata.json`
-
-// No type cache needed!
-// No type search needed!
-// No multi-type fallback needed!
-// Just pure O(1) lookup!
-```
-
-**Benefits:**
-- **O(1)** path lookups (eliminates the 42-type sequential search the old type-first layout required)
-- **Simpler code** - removed 500+ lines of type cache complexity
-- **Scalable** - works at large scale without type tracking overhead
-
----
-
-### ✅ Sharding
-
-All batch paths include shard IDs calculated via `getShardIdFromUuid(id)`:
-
-```typescript
-const id = 'a3c4e5f7-...'
-const shard = getShardIdFromUuid(id) // → 'a3' (first 2 hex chars)
-const path = `entities/nouns/${shard}/${id}/metadata.json`
-```
-
-**Distribution:** 256 shards (00-ff) for optimal load distribution.
-
----
-
-### ✅ Generational MVCC (8.0)
-
-Batch reads always serve the **live** generation through the fast paths
-shown above. Point-in-time reads go through the Db API instead: a pinned
-`Db` (`brain.now()`, `brain.asOf()`) resolves changed ids from immutable
-generation records and unchanged ids from the same live paths batch reads
-use — see the [consistency model](concepts/consistency-model.md).
-
-```typescript
-const db = brain.now()                       // pinned view
-const entity = await db.get(id)              // correct at the pinned generation
-const results = await brain.batchGet(ids)    // live state, batched
-await db.release()
-```
-
----
-
-## Why Batching Is Faster
-
-Batching's advantage is structural, not a fixed multiplier (the actual speedup
-depends on storage backend, IOPS, and batch size):
-
-- **N+1 elimination** — N sequential reads collapse into a single parallel pass
-  (`Promise.all` over the batch).
-- **O(1) path construction** — every ID maps directly to one storage path, with
-  no per-type cache lookup.
-- **One metadata round-trip** — relationship batches fetch all sources' metadata
-  in a single pass instead of one query per source.
-
-The integration test `tests/integration/storage-batch-operations.test.ts`
-exercises batch vs. individual reads and asserts that batch retrieval is not
-slower than the per-entity loop for large batches; it does not pin a specific
-multiplier, since that is hardware- and IOPS-dependent.
-
----
-
-## Error Handling
-
-### Partial Batch Failures
-
-Batch operations gracefully handle missing or invalid entities:
-
-```typescript
-const validId = 'abc-123-...'
-const invalidIds = [
- '11111111-1111-1111-1111-111111111111',
- '22222222-2222-2222-2222-222222222222'
-]
-
-const results = await brain.batchGet([validId, ...invalidIds])
-
-results.size // → 1 (only valid entity)
-results.has(validId) // → true
-results.has(invalidIds[0]) // → false (silently skipped)
-```
-
-**Behavior:**
-- Invalid UUIDs: Silently skipped (not included in results)
-- Missing entities: Silently skipped (not included in results)
-- Storage errors: Logged, entity excluded from results
-- No exceptions thrown for partial failures
-
-### Empty Batches
-
-```typescript
-const results = await brain.batchGet([])
-results.size // → 0 (empty map)
-```
-
-### Duplicate IDs
-
-```typescript
-const results = await brain.batchGet(['id1', 'id1', 'id1'])
-results.size // → 1 (deduplicated automatically)
-```
-
----
-
-## Migration Guide
-
-### From Individual Gets
-
-**Before:**
-```typescript
-const entities = []
-for (const id of ids) {
- const entity = await brain.get(id)
- if (entity) entities.push(entity)
-}
-```
-
-**After:**
-```typescript
-const results = await brain.batchGet(ids)
-const entities = Array.from(results.values())
-```
-
-**Performance Gain:** Replaces N sequential reads with a single batched pass — no fixed multiplier, it scales with storage IOPS.
-
----
-
-### From Individual Relationship Queries
-
-**Before:**
-```typescript
-const allVerbs = []
-for (const sourceId of sourceIds) {
- const verbs = await brain.related({ from: sourceId })
- allVerbs.push(...verbs)
-}
-```
-
-**After:**
-```typescript
-const storage = brain.storage as BaseStorage
-const results = await storage.getVerbsBySourceBatch(sourceIds)
-
-const allVerbs = []
-for (const verbs of results.values()) {
- allVerbs.push(...verbs)
-}
-```
-
-**Performance Gain:** One batched metadata fetch instead of one query per source entity.
-
----
-
-## Best Practices
-
-### 1. **Use Batching for Multiple Entity Operations**
-
-```typescript
-// ✅ GOOD: Batch fetch
-const results = await brain.batchGet(ids)
-
-// ❌ BAD: Individual gets in loop
-for (const id of ids) {
- await brain.get(id)
-}
-```
-
-### 2. **Batch Size Recommendations**
-
-| Storage | Optimal Batch Size | Max Batch Size |
-|---------|--------------------|----------------|
-| **Memory** | Unlimited | Unlimited |
-| **Filesystem** | 100-500 | 1000 |
-
-**Guideline:** For batches >1000, split into chunks of 500-1000.
-
-### 3. **Metadata-Only by Default**
-
-```typescript
-// Default: Metadata-only (fast)
-const results = await brain.batchGet(ids) // No vectors
-
-// Only load vectors if needed
-const withVectors = await brain.batchGet(ids, { includeVectors: true })
-```
-
-### 4. **Error Handling**
-
-```typescript
-// Batch operations never throw for missing entities
-const results = await brain.batchGet(ids)
-
-// Check results
-for (const id of ids) {
- if (results.has(id)) {
- // Entity exists
- const entity = results.get(id)
- } else {
- // Entity missing (not an error)
- console.log(`Entity ${id} not found`)
- }
-}
-```
-
----
-
-## Testing
-
-Comprehensive test coverage in `tests/integration/storage-batch-operations.test.ts`:
-
-```bash
-npx vitest run tests/integration/storage-batch-operations.test.ts
-```
-
-**Test Coverage:**
-- ✅ brain.batchGet() high-level API
-- ✅ storage.getNounMetadataBatch() with ID-first paths
-- ✅ COW integration (branch isolation, inheritance)
-- ✅ storage.getVerbsBySourceBatch() relationship queries
-- ✅ VFS integration (PathResolver.getChildren())
-- ✅ Performance benchmarks (N+1 elimination)
-- ✅ Error handling (partial failures, empty batches, duplicates)
-- ✅ ID-first storage verification
-- ✅ Sharding preservation
-
-**Results:** 23 tests passing ✅
-
----
-
-## Implementation Details
-
-### Architecture Layers
-
-```
-User Code (brain.batchGet)
- ↓
-High-Level API (src/brainy.ts)
- ↓
-Storage Layer (src/storage/baseStorage.ts)
- ↓
-Adapter Layer (readBatchFromAdapter)
- ↓
-Storage Adapter (FileSystemStorage / MemoryStorage)
-```
-
-### Parallel Reads
-
-Both shipped adapters fall back to `Promise.all` over individual reads:
-
-```typescript
-// BaseStorage.readBatchFromAdapter()
-return await Promise.all(resolvedPaths.map(path => this.read(path)))
-```
-
-**Shipped Adapters:**
-- MemoryStorage
-- FileSystemStorage
-
----
-
-## API Summary
-
-- `brain.batchGet(ids, options?)` - High-level batch entity retrieval
-- `storage.getNounMetadataBatch(ids)` - Storage-level metadata batch
-- `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries
-
-**Performance Improvements:**
-- VFS operations: single batched pass instead of N sequential reads
-- Entity retrieval: N+1 reads collapsed into one batched pass
-- Zero N+1 query patterns
-
-**Compatibility:**
-- ✅ ID-first storage
-- ✅ Sharding (256 shards)
-- ✅ Generational MVCC — batch reads serve the live generation; pinned `Db` views serve the past
-- ✅ All indexes respected (vector, metadata, graph adjacency)
-
----
-
-## Support
-
-- **Documentation:** `/docs/BATCHING.md`, `/docs/PERFORMANCE.md`
-- **Tests:** `/tests/integration/storage-batch-operations.test.ts`
-- **Issues:** https://github.com/soulcraft/brainy/issues
-- **Discussions:** https://github.com/soulcraft/brainy/discussions
-
----
-
-**Built with ❤️ for enterprise-scale knowledge graphs**

docs/CORE_API_PATTERNS.md

@@ -0,0 +1,643 @@
+# 🧠 Core API Patterns: Modern Brainy v3.x
+
+> Learn the correct patterns for Brainy's core operations. Avoid v2.x confusion and use modern, efficient APIs.
+
+## 🚨 Critical: Use v3.x APIs Only
+
+### ❌ **WRONG - Deprecated v2.x APIs**
+
+```typescript
+// DON'T DO THIS - These methods don't exist in v3.x!
+await brain.addNoun(text, type, metadata)     // ❌ Removed
+await brain.getNouns({ pagination })          // ❌ Removed
+await brain.addVerb(source, target, type)     // ❌ Removed
+await brain.getVerbs()                        // ❌ Removed
+await brain.deleteNoun(id)                    // ❌ Removed
+await brain.deleteVerb(id)                    // ❌ Removed
+```
+
+### ✅ **CORRECT - Modern v3.x APIs**
+
+```typescript
+// ✅ Use these modern methods instead
+await brain.add({ data, type, metadata })     // Modern unified add
+await brain.find({ limit: 100 })              // Natural language search
+await brain.relate({ from, to, type })        // Clean relationship creation
+await brain.getRelations()                    // Modern relationship queries
+await brain.delete(id)                        // Unified deletion
+// Relationships auto-cascade when entities are deleted
+```
+
+## 📋 Entity Management Patterns
+
+### ❌ **WRONG - v2.x Style**
+
+```typescript
+// DON'T DO THIS - Old API patterns
+import { Brainy } from 'old-brainy'  // ❌ Wrong import
+
+const brain = new Brainy({           // ❌ Old class name
+  complexConfig: true
+})
+
+const id = await brain.addNoun(          // ❌ Deprecated method
+  "John Smith is a developer",
+  "Person",
+  { role: "engineer" }
+)
+```
+
+### ✅ **CORRECT - Modern Patterns**
+
+```typescript
+// ✅ Pattern 1: Basic entity creation
+import { Brainy, NounType } from '@soulcraft/brainy'
+
+const brain = new Brainy()  // ✅ Zero config
+await brain.init()
+
+const id = await brain.add({
+  data: "John Smith is a developer",
+  type: NounType.Person,
+  metadata: { role: "engineer", team: "backend" }
+})
+
+// ✅ Pattern 2: Bulk entity creation
+const entities = [
+  { data: "React framework", type: NounType.Technology },
+  { data: "Vue.js framework", type: NounType.Technology },
+  { data: "Angular framework", type: NounType.Technology }
+]
+
+const ids = await Promise.all(
+  entities.map(entity => brain.add(entity))
+)
+
+// ✅ Pattern 3: Entity with pre-computed vector
+const customVector = await brain.embed("Custom text")
+const vectorId = await brain.add({
+  data: "Optimized content",
+  type: NounType.Document,
+  vector: customVector,  // Skip re-embedding
+  metadata: { source: "api", optimized: true }
+})
+```
+
+## 🔍 Search & Discovery Patterns
+
+### ❌ **WRONG - Confusing Old Patterns**
+
+```typescript
+// DON'T DO THIS - Mixing old and new APIs
+const results1 = await brain.searchText("query")     // ❌ Old method
+const results2 = await brain.getNouns({ filter })    // ❌ Doesn't exist
+const results3 = await brain.findSimilar(text)       // ❌ Unclear naming
+```
+
+### ✅ **CORRECT - Clean Search Patterns**
+
+```typescript
+// ✅ Pattern 1: Natural language search
+const results = await brain.find("React developers working on authentication")
+
+// ✅ Pattern 2: Structured search with filters
+const filteredResults = await brain.find({
+  like: "machine learning",           // Vector similarity
+  where: {                           // Metadata filtering
+    type: NounType.Document,
+    year: { $gte: 2020 },
+    status: "published"
+  },
+  limit: 50,
+  orderBy: 'relevance'
+})
+
+// ✅ Pattern 3: Similarity search
+const similarItems = await brain.similar({
+  to: existingEntityId,              // Find items similar to this
+  threshold: 0.8,                    // Minimum similarity
+  limit: 10,
+  exclude: [existingEntityId]        // Don't include the source
+})
+
+// ✅ Pattern 4: Advanced search with relationships
+const connectedResults = await brain.find({
+  like: "frontend frameworks",
+  connected: {
+    to: reactId,                     // Connected to React
+    via: "related-to",               // Through this relationship
+    depth: 2                         // Up to 2 hops away
+  }
+})
+```
+
+## 🔗 Relationship Patterns
+
+### ❌ **WRONG - Old Relationship APIs**
+
+```typescript
+// DON'T DO THIS - Old relationship patterns
+await brain.addVerb(sourceId, targetId, "uses", { strength: 0.9 })  // ❌ Old API
+const verbs = await brain.getVerbsBySource(sourceId)               // ❌ Removed
+await brain.deleteVerb(verbId)                                     // ❌ Old pattern
+```
+
+### ✅ **CORRECT - Modern Relationship Management**
+
+```typescript
+// ✅ Pattern 1: Create relationships
+const relationId = await brain.relate({
+  from: developerId,
+  to: frameworkId,
+  type: VerbType.Uses,
+  metadata: {
+    since: "2023-01-01",
+    proficiency: "expert",
+    hours_per_week: 40
+  }
+})
+
+// ✅ Pattern 2: Query relationships
+const relationships = await brain.getRelations({
+  from: developerId,                 // Relationships from this entity
+  type: VerbType.Uses,              // Of this type
+  limit: 100
+})
+
+// ✅ Pattern 3: Bidirectional relationships
+await brain.relate({
+  from: projectId,
+  to: developerId,
+  type: VerbType.AssignedTo,
+  bidirectional: true,              // Creates reverse relationship
+  metadata: { role: "lead", start_date: "2024-01-01" }
+})
+
+// ✅ Pattern 4: Relationship-based discovery
+const collaborators = await brain.find({
+  connected: {
+    to: currentProjectId,
+    via: VerbType.WorksOn,
+    direction: "incoming"           // Who works on this project
+  }
+})
+```
+
+## 🗃️ Data Retrieval Patterns
+
+### ❌ **WRONG - Inefficient Patterns**
+
+```typescript
+// DON'T DO THIS - Loading everything
+const everything = await brain.getNouns({ limit: 1000000 })  // ❌ Crashes
+const allData = await brain.exportAll()                      // ❌ Memory explosion
+```
+
+### ✅ **CORRECT - Efficient Data Access**
+
+```typescript
+// ✅ Pattern 1: Paginated retrieval
+async function getAllEntitiesPaginated() {
+  const pageSize = 100
+  let offset = 0
+  let allEntities = []
+
+  while (true) {
+    const page = await brain.find({
+      limit: pageSize,
+      offset: offset
+    })
+
+    if (page.length === 0) break
+
+    allEntities.push(...page)
+    offset += pageSize
+
+    // Optional: Progress reporting
+    console.log(`Loaded ${allEntities.length} entities...`)
+  }
+
+  return allEntities
+}
+
+// ✅ Pattern 2: Streaming large datasets
+async function* streamEntities() {
+  const pageSize = 50
+  let offset = 0
+
+  while (true) {
+    const page = await brain.find({
+      limit: pageSize,
+      offset: offset
+    })
+
+    if (page.length === 0) break
+
+    for (const entity of page) {
+      yield entity
+    }
+
+    offset += pageSize
+  }
+}
+
+// Usage
+for await (const entity of streamEntities()) {
+  await processEntity(entity)
+}
+
+// ✅ Pattern 3: Specific entity retrieval
+const entity = await brain.get(entityId)
+if (entity) {
+  console.log('Entity data:', entity.data)
+  console.log('Metadata:', entity.metadata)
+} else {
+  console.log('Entity not found')
+}
+```
+
+## 🔄 Update & Delete Patterns
+
+### ❌ **WRONG - Manual Update Patterns**
+
+```typescript
+// DON'T DO THIS - Recreating entities
+await brain.delete(oldId)
+const newId = await brain.add(updatedData)  // ❌ Loses relationships
+```
+
+### ✅ **CORRECT - Update Operations**
+
+```typescript
+// ✅ Pattern 1: Update entity data
+await brain.update(entityId, {
+  data: "Updated content here",
+  metadata: {
+    lastModified: Date.now(),
+    version: "2.0"
+  }
+})
+
+// ✅ Pattern 2: Partial metadata updates
+await brain.updateMetadata(entityId, {
+  status: "published",
+  tags: ["important", "featured"]
+  // Merges with existing metadata
+})
+
+// ✅ Pattern 3: Safe deletion with cascade options
+await brain.delete(entityId, {
+  cascade: true,          // Delete related relationships
+  backup: true           // Create backup before deletion
+})
+
+// ✅ Pattern 4: Bulk operations
+const updateOperations = entities.map(entity => ({
+  id: entity.id,
+  changes: { status: "processed" }
+}))
+
+await brain.updateMany(updateOperations)
+```
+
+## 🧮 Vector & Embedding Patterns
+
+### ❌ **WRONG - Manual Vector Handling**
+
+```typescript
+// DON'T DO THIS - Manual embedding without understanding
+const vector = await brain.embed(text)
+// Store vector somewhere manually                    // ❌ Missing integration
+```
+
+### ✅ **CORRECT - Smart Vector Operations**
+
+```typescript
+// ✅ Pattern 1: Automatic embedding (recommended)
+const id = await brain.add({
+  data: "Content to be embedded",
+  type: NounType.Document
+  // Vector computed automatically
+})
+
+// ✅ Pattern 2: Pre-computed vectors for optimization
+const texts = ["Text 1", "Text 2", "Text 3"]
+const vectors = await Promise.all(
+  texts.map(text => brain.embed(text))
+)
+
+const entities = await Promise.all(
+  texts.map((text, i) => brain.add({
+    data: text,
+    type: NounType.Document,
+    vector: vectors[i]  // Skip re-embedding
+  }))
+)
+
+// ✅ Pattern 3: Vector similarity search
+const queryVector = await brain.embed("search query")
+const similar = await brain.similar({
+  vector: queryVector,              // Use vector directly
+  threshold: 0.75,
+  limit: 20
+})
+
+// ✅ Pattern 4: Compare vectors directly
+const vector1 = await brain.embed("First text")
+const vector2 = await brain.embed("Second text")
+const similarity = brain.computeSimilarity(vector1, vector2)
+console.log(`Similarity: ${similarity}`)
+```
+
+## 🏗️ Configuration Patterns
+
+### ❌ **WRONG - Over-Configuration**
+
+```typescript
+// DON'T DO THIS - Complex configurations that break
+const brain = new Brainy({
+  storage: {
+    type: 'complex',
+    options: {
+      nested: {
+        configuration: true,
+        that: "breaks"
+      }
+    }
+  },
+  embedding: {
+    customModel: "broken-model",
+    dimensions: 999999
+  }
+})
+```
+
+### ✅ **CORRECT - Smart Configuration**
+
+```typescript
+// ✅ Pattern 1: Zero configuration (recommended)
+const brain = new Brainy()  // Auto-detects everything
+await brain.init()
+
+// ✅ Pattern 2: Simple storage selection
+const fsBrain = new Brainy({
+  storage: { type: 'filesystem', path: './data' }
+})
+
+const cloudBrain = new Brainy({
+  storage: { type: 's3', bucket: 'my-data' }
+})
+
+// ✅ Pattern 3: Production configuration
+const prodBrain = new Brainy({
+  storage: {
+    type: 's3',
+    bucket: process.env.BRAINY_BUCKET,
+    region: process.env.AWS_REGION
+  },
+  silent: true,                    // No console output
+  distributed: true,               // Enable clustering
+  cache: { maxSize: 10000 }       // Larger cache
+})
+
+// ✅ Pattern 4: Development vs production
+const isDev = process.env.NODE_ENV === 'development'
+
+const brain = new Brainy({
+  storage: isDev
+    ? { type: 'memory' }           // Fast for dev
+    : { type: 'filesystem', path: './brainy-data' },  // Persistent for prod
+  silent: !isDev,                 // Verbose in dev, quiet in prod
+  cache: { maxSize: isDev ? 100 : 5000 }
+})
+```
+
+## 🔄 Migration from v2.x
+
+### ✅ **Migration Patterns**
+
+```typescript
+// If you have old v2.x code, here's how to migrate:
+
+// OLD v2.x:
+// await brain.addNoun(text, type, metadata)
+// NEW v3.x:
+await brain.add({ data: text, type, metadata })
+
+// OLD v2.x:
+// await brain.getNouns({ pagination: { limit: 100 } })
+// NEW v3.x:
+await brain.find({ limit: 100 })
+
+// OLD v2.x:
+// await brain.addVerb(sourceId, targetId, verbType, metadata)
+// NEW v3.x:
+await brain.relate({ from: sourceId, to: targetId, type: verbType, metadata })
+
+// OLD v2.x:
+// await brain.searchText(query)
+// NEW v3.x:
+await brain.find(query)  // More powerful natural language search
+```
+
+## 🚀 Performance Patterns
+
+### ✅ **High-Performance Patterns**
+
+```typescript
+// ✅ Pattern 1: Batch operations
+const entities = [/* large array */]
+const batchSize = 100
+
+for (let i = 0; i < entities.length; i += batchSize) {
+  const batch = entities.slice(i, i + batchSize)
+  await Promise.all(
+    batch.map(entity => brain.add(entity))
+  )
+
+  // Optional: Rate limiting
+  await new Promise(resolve => setTimeout(resolve, 100))
+}
+
+// ✅ Pattern 2: Connection pooling for distributed
+const brain = new Brainy({
+  distributed: true,
+  connectionPool: {
+    min: 5,
+    max: 50,
+    acquireTimeoutMillis: 30000
+  }
+})
+
+// ✅ Pattern 3: Efficient caching
+const brain = new Brainy({
+  cache: {
+    maxSize: 10000,              // Number of items
+    ttl: 300000,                 // 5 minutes
+    updateAgeOnGet: true         // LRU behavior
+  }
+})
+
+// ✅ Pattern 4: Memory-conscious operations
+const results = await brain.find({
+  query: "large dataset query",
+  limit: 1000,                   // Reasonable limit
+  includeVectors: false          // Exclude vectors if not needed
+})
+```
+
+## 🛡️ Error Handling Patterns
+
+### ✅ **Robust Error Handling**
+
+```typescript
+// ✅ Pattern 1: Specific error handling
+try {
+  const result = await brain.add({ data, type, metadata })
+  return result
+} catch (error) {
+  if (error.code === 'DUPLICATE_ENTITY') {
+    console.log('Entity already exists, updating instead...')
+    return await brain.update(error.existingId, { data, metadata })
+  } else if (error.code === 'STORAGE_FULL') {
+    throw new Error('Storage capacity exceeded')
+  } else if (error.code === 'EMBEDDING_FAILED') {
+    console.warn('Embedding failed, retrying with simpler text...')
+    return await brain.add({
+      data: data.substring(0, 1000), // Truncate
+      type,
+      metadata
+    })
+  }
+  throw error
+}
+
+// ✅ Pattern 2: Retry with exponential backoff
+async function resilientAdd(data: any, maxRetries = 3) {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      return await brain.add(data)
+    } catch (error) {
+      if (attempt === maxRetries) throw error
+
+      const delay = Math.pow(2, attempt) * 1000
+      console.warn(`Attempt ${attempt} failed, retrying in ${delay}ms...`)
+      await new Promise(resolve => setTimeout(resolve, delay))
+    }
+  }
+}
+
+// ✅ Pattern 3: Graceful degradation
+async function robustSearch(query: string) {
+  try {
+    // Try advanced semantic search first
+    return await brain.find({
+      like: query,
+      threshold: 0.8,
+      limit: 50
+    })
+  } catch (error) {
+    console.warn('Semantic search failed, falling back to basic search:', error.message)
+
+    try {
+      // Fallback to simple text search
+      return await brain.find(query)
+    } catch (fallbackError) {
+      console.error('All search methods failed:', fallbackError.message)
+      return []  // Return empty results rather than crash
+    }
+  }
+}
+```
+
+## 📊 Monitoring Patterns
+
+### ✅ **Production Monitoring**
+
+```typescript
+// ✅ Pattern 1: Performance monitoring
+const startTime = Date.now()
+const result = await brain.add(data)
+const duration = Date.now() - startTime
+
+if (duration > 1000) {
+  console.warn(`Slow add operation: ${duration}ms`)
+}
+
+// ✅ Pattern 2: Health checks
+async function healthCheck() {
+  try {
+    // Test basic operations
+    const testId = await brain.add({
+      data: "health check",
+      type: NounType.System,
+      metadata: { test: true }
+    })
+
+    await brain.get(testId)
+    await brain.delete(testId)
+
+    return { status: 'healthy', timestamp: Date.now() }
+  } catch (error) {
+    return {
+      status: 'unhealthy',
+      error: error.message,
+      timestamp: Date.now()
+    }
+  }
+}
+
+// ✅ Pattern 3: Metrics collection
+class BrainyMetrics {
+  private metrics = {
+    operations: 0,
+    errors: 0,
+    totalTime: 0
+  }
+
+  async timedOperation<T>(operation: () => Promise<T>): Promise<T> {
+    const start = Date.now()
+    try {
+      const result = await operation()
+      this.metrics.operations++
+      this.metrics.totalTime += Date.now() - start
+      return result
+    } catch (error) {
+      this.metrics.errors++
+      throw error
+    }
+  }
+
+  getStats() {
+    return {
+      ...this.metrics,
+      avgTime: this.metrics.totalTime / this.metrics.operations || 0,
+      errorRate: this.metrics.errors / this.metrics.operations || 0
+    }
+  }
+}
+```
+
+## 🎯 Summary: Modern Brainy v3.x Best Practices
+
+| ❌ **Avoid v2.x** | ✅ **Use v3.x** |
+|------------------|----------------|
+| `addNoun()` | `add()` |
+| `getNouns()` | `find()` |
+| `addVerb()` | `relate()` |
+| `getVerbs()` | `getRelations()` |
+| `deleteNoun()` | `delete()` |
+| Complex configs | Zero-config with `new Brainy()` |
+| Manual pagination | Built-in smart pagination |
+| String-based search | Natural language queries |
+
+---
+
+**🎉 Following these patterns gives you:**
+- 🚀 **Modern APIs** that are actively maintained
+- ⚡ **Better performance** with intelligent defaults
+- 🛡️ **Robust error handling** with specific error types
+- 📈 **Scalable patterns** for production applications
+- 🧠 **Natural language** search capabilities
+
+**Next:** [Neural API Patterns →](./NEURAL_API_PATTERNS.md) | [VFS Patterns →](./vfs/COMMON_PATTERNS.md)

docs/CREATING-AUGMENTATIONS.md

@@ -0,0 +1,435 @@
+# Creating Augmentations for Brainy
+
+> **Updated for v4.0.0** - Includes metadata structure changes and type system improvements
+
+## The BrainyAugmentation Interface
+
+Every augmentation implements this simple yet powerful interface:
+
+```typescript
+interface BrainyAugmentation {
+  // Identification
+  name: string                    // Unique name for your augmentation
+
+  // Execution control
+  timing: 'before' | 'after' | 'around' | 'replace'  // When to execute
+  operations: string[]            // Which operations to intercept
+  priority: number                 // Execution order (higher = first)
+
+  // Lifecycle methods
+  initialize(context: AugmentationContext): Promise<void>
+  execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T>
+  shutdown?(): Promise<void>      // Optional cleanup
+}
+```
+
+## v4.0.0 Breaking Changes for Augmentation Developers
+
+### 1. Metadata Structure Separation
+v4.0.0 introduces strict metadata/vector separation for billion-scale performance:
+
+```typescript
+// ✅ v4.0.0: Metadata has required type field
+interface NounMetadata {
+  noun: NounType               // Required! Must be a valid noun type
+  [key: string]: any          // Your custom metadata
+}
+
+interface VerbMetadata {
+  verb: VerbType               // Required! Must be a valid verb type
+  sourceId: string
+  targetId: string
+  [key: string]: any
+}
+```
+
+### 2. Storage Adapter Return Types
+Storage adapters now return different types at different boundaries:
+
+```typescript
+// Internal methods: Pure structures (no metadata)
+abstract _getNoun(id: string): Promise<HNSWNoun | null>
+
+// Public API: WithMetadata structures
+abstract getNoun(id: string): Promise<HNSWNounWithMetadata | null>
+```
+
+### 3. Verb Property Renamed
+The verb relationship field changed from `type` to `verb`:
+
+```typescript
+// ❌ v3.x
+verb.type === 'relatedTo'
+
+// ✅ v4.0.0
+verb.verb === 'relatedTo'
+```
+
+## Creating a Storage Augmentation
+
+Storage augmentations are special - they provide the storage backend for Brainy.
+
+### Important: v4.0.0 Storage Requirements
+
+Your storage adapter MUST:
+1. **Wrap metadata** with required `noun`/`verb` fields
+2. **Return pure structures** from internal `_methods`
+3. **Return WithMetadata types** from public methods
+
+```typescript
+import { StorageAugmentation } from 'brainy/augmentations'
+import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy'
+
+export class MyCustomStorage extends BaseStorageAdapter {
+  // Internal method: Returns pure structure
+  async _getNoun(id: string): Promise<HNSWNoun | null> {
+    const data = await this.fetchFromDatabase(id)
+    return data ? {
+      id: data.id,
+      vector: data.vector,
+      nounType: data.type
+    } : null
+  }
+
+  // Public method: Returns WithMetadata structure
+  async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
+    const noun = await this._getNoun(id)
+    if (!noun) return null
+
+    // Fetch metadata separately (v4.0.0 pattern)
+    const metadata = await this.getNounMetadata(id)
+
+    return {
+      ...noun,
+      metadata: metadata || { noun: noun.nounType || 'thing' }
+    }
+  }
+
+  // CRITICAL: Always save with proper metadata structure
+  async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
+    // Validate metadata has required 'noun' field
+    if (!metadata?.noun) {
+      throw new Error('v4.0.0: NounMetadata requires "noun" field')
+    }
+
+    await this.database.save({
+      id: noun.id,
+      vector: noun.vector,
+      nounType: noun.nounType,
+      metadata: metadata  // Stored separately in v4.0.0
+    })
+  }
+}
+
+export class MyStorageAugmentation extends StorageAugmentation {
+  private config: MyStorageConfig
+
+  constructor(config: MyStorageConfig) {
+    super()
+    this.name = 'my-custom-storage'
+    this.config = config
+  }
+
+  // Called during storage resolution phase
+  async provideStorage(): Promise<StorageAdapter> {
+    const storage = new MyCustomStorage(this.config)
+    this.storageAdapter = storage
+    return storage
+  }
+
+  // Called during augmentation initialization
+  protected async onInitialize(): Promise<void> {
+    await this.storageAdapter!.init()
+    this.log(`Custom storage initialized`)
+  }
+}
+```
+
+### Using Your Storage Augmentation
+
+```typescript
+// Register before brain.init()
+const brain = new Brainy()
+brain.augmentations.register(new MyStorageAugmentation({
+  connectionString: 'redis://localhost:6379'
+}))
+await brain.init()  // Will use your storage!
+```
+
+## Creating a Feature Augmentation
+
+Here's a complete example of a caching augmentation:
+
+```typescript
+import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations'
+
+export class CachingAugmentation extends BaseAugmentation {
+  private cache = new Map<string, any>()
+  
+  constructor() {
+    super()
+    this.name = 'smart-cache'
+    this.timing = 'around'        // Wrap operations
+    this.operations = ['search']   // Only cache searches
+    this.priority = 50             // Mid-priority
+  }
+  
+  async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
+    if (operation === 'search') {
+      // Check cache
+      const cacheKey = JSON.stringify(params)
+      if (this.cache.has(cacheKey)) {
+        this.log('Cache hit!')
+        return this.cache.get(cacheKey)
+      }
+      
+      // Execute and cache
+      const result = await next()
+      this.cache.set(cacheKey, result)
+      return result
+    }
+    
+    // Pass through other operations
+    return next()
+  }
+  
+  protected async onInitialize(): Promise<void> {
+    this.log('Cache initialized')
+  }
+  
+  async shutdown(): Promise<void> {
+    this.cache.clear()
+    await super.shutdown()
+  }
+}
+```
+
+## The Four Timing Modes
+
+### 1. `before` - Pre-processing
+```typescript
+timing = 'before'
+async execute(op, params, next) {
+  // Validate/transform input
+  const validated = await validate(params)
+  return next(validated)  // Pass modified params
+}
+```
+
+### 2. `after` - Post-processing  
+```typescript
+timing = 'after'
+async execute(op, params, next) {
+  const result = await next()
+  // Log, analyze, or modify result
+  console.log(`Operation ${op} returned:`, result)
+  return result
+}
+```
+
+### 3. `around` - Wrapping (middleware)
+```typescript
+timing = 'around'
+async execute(op, params, next) {
+  console.log('Starting', op)
+  try {
+    const result = await next()
+    console.log('Success', op)
+    return result
+  } catch (error) {
+    console.log('Failed', op, error)
+    throw error
+  }
+}
+```
+
+### 4. `replace` - Complete replacement
+```typescript
+timing = 'replace'
+async execute(op, params, next) {
+  // Don't call next() - replace entirely!
+  return myCustomImplementation(params)
+}
+```
+
+## Operations You Can Intercept
+
+Common operations in Brainy:
+- `'storage'` - Storage resolution (special)
+- `'add'` - Adding data
+- `'search'`, `'similar'` - Searching
+- `'update'`, `'delete'` - Modifications
+- `'saveNoun'`, `'saveVerb'` - Storage operations
+- `'all'` - Intercept everything
+
+## Context Available to Augmentations
+
+```typescript
+interface AugmentationContext {
+  brain: Brainy         // The brain instance
+  storage: StorageAdapter   // Storage backend
+  config: BrainyConfig  // Configuration
+  log: (message: string, level?: 'info' | 'warn' | 'error') => void
+}
+```
+
+## Real-World Examples
+
+### 1. Redis Storage Augmentation
+```typescript
+export class RedisStorageAugmentation extends StorageAugmentation {
+  async provideStorage(): Promise<StorageAdapter> {
+    return new RedisAdapter({
+      host: 'localhost',
+      port: 6379,
+      // Implement full StorageAdapter interface
+    })
+  }
+}
+```
+
+### 2. Audit Trail Augmentation
+```typescript
+export class AuditAugmentation extends BaseAugmentation {
+  timing = 'after'
+  operations = ['add', 'update', 'delete']
+  
+  async execute(op, params, next) {
+    const result = await next()
+    
+    // Log to audit trail
+    await this.logAudit({
+      operation: op,
+      params,
+      result,
+      timestamp: new Date(),
+      user: this.context.config.currentUser
+    })
+    
+    return result
+  }
+}
+```
+
+### 3. Rate Limiting Augmentation
+```typescript
+export class RateLimitAugmentation extends BaseAugmentation {
+  timing = 'before'
+  operations = ['search']
+  private limiter = new RateLimiter({ rps: 100 })
+  
+  async execute(op, params, next) {
+    await this.limiter.acquire()  // Wait if rate limited
+    return next()
+  }
+}
+```
+
+## Publishing to Brain Cloud Marketplace
+
+Future capability for premium augmentations:
+
+```typescript
+// package.json
+{
+  "name": "@brain-cloud/redis-storage",
+  "brainy": {
+    "type": "augmentation",
+    "category": "storage",
+    "premium": true
+  }
+}
+
+// Users can install via:
+// brainy augment install redis-storage
+```
+
+## Best Practices
+
+### General Practices
+
+1. **Use BaseAugmentation** - Provides common functionality
+2. **Set appropriate priority** - Storage (100), System (80-99), Features (10-50)
+3. **Be selective with operations** - Don't use 'all' unless necessary
+4. **Handle errors gracefully** - Don't break the chain
+5. **Clean up in shutdown()** - Release resources
+6. **Log appropriately** - Use context.log() for consistent output
+7. **Document your augmentation** - Include examples
+
+### v4.0.0 Specific Best Practices
+
+8. **Always include `noun` field** when creating/modifying NounMetadata:
+   ```typescript
+   const metadata: NounMetadata = {
+     noun: 'thing',  // REQUIRED!
+     yourField: 'value'
+   }
+   ```
+
+9. **Use `verb` property** not `type` when working with relationships:
+   ```typescript
+   // ✅ Correct
+   if (verb.verb === 'relatedTo') { ... }
+
+   // ❌ Wrong (v3.x pattern)
+   if (verb.type === 'relatedTo') { ... }
+   ```
+
+10. **Access metadata correctly** from storage:
+    ```typescript
+    // ✅ Correct - metadata is already structured
+    const nounType = noun.metadata.noun
+
+    // ⚠️ Fallback pattern for robustness
+    const nounType = noun.metadata?.noun || 'thing'
+    ```
+
+11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations:
+    ```typescript
+    // ✅ Good - Separate concerns
+    await storage.saveNoun(noun)
+    await storage.saveMetadata(noun.id, metadata)
+
+    // ❌ Bad - Mixing concerns
+    await storage.saveNounWithEverything(combinedData)
+    ```
+
+## Testing Your Augmentation
+
+```typescript
+import { Brainy } from 'brainy'
+import { MyAugmentation } from './my-augmentation'
+
+describe('MyAugmentation', () => {
+  let brain: Brainy
+  
+  beforeEach(async () => {
+    brain = new Brainy()
+    brain.augmentations.register(new MyAugmentation())
+    await brain.init()
+  })
+  
+  afterEach(async () => {
+    await brain.destroy()
+  })
+  
+  it('should enhance searches', async () => {
+    // Test your augmentation's effect
+    const results = await brain.search('test')
+    expect(results).toHaveProperty('enhanced', true)
+  })
+})
+```
+
+## Summary
+
+Augmentations are Brainy's extension system. They can:
+- Replace storage backends
+- Add caching layers
+- Implement audit trails
+- Add rate limiting
+- Sync with external systems
+- Transform data
+- And much more!
+
+The unified BrainyAugmentation interface makes it easy to create powerful extensions while maintaining consistency across the entire system.

docs/DATA_MODEL.md

@@ -1,271 +0,0 @@
-# Data Model
-
-> How Brainy stores entities and relationships, and the critical distinction between `data` and `metadata`.
-
----
-
-## Entity (Noun)
-
-An entity is the fundamental data unit in Brainy. Every entity has:
-
-| Field | Type | Indexed | Description |
-|-------|------|---------|-------------|
-| `id` | `string` | Primary key | UUID v4 (auto-generated or custom) |
-| `data` | `any` | **HNSW vector index** | Content used for semantic/hybrid search. Strings auto-embed. |
-| `metadata` | `object` | **MetadataIndex** | Structured queryable fields (tags, dates, flags, etc.) |
-| `type` | `NounType` | MetadataIndex (as `noun`) | Entity type classification |
-| `vector` | `number[]` | HNSW | 384-dim embedding (auto-computed from `data` or user-provided) |
-| `confidence` | `number` | MetadataIndex | Type classification confidence (0-1) |
-| `weight` | `number` | MetadataIndex | Entity importance/salience (0-1) |
-| `service` | `string` | MetadataIndex | Multi-tenancy identifier |
-| `createdAt` | `number` | MetadataIndex | Creation timestamp (ms since epoch) |
-| `updatedAt` | `number` | MetadataIndex | Last update timestamp (ms since epoch) |
-| `createdBy` | `object` | MetadataIndex | Source augmentation info |
-
-### Example
-
-```typescript
-const id = await brain.add({
-  data: 'John Smith is a software engineer at Acme Corp',  // → embedded into vector
-  type: NounType.Person,
-  metadata: {                   // → indexed, queryable via where filters
-    role: 'engineer',
-    department: 'backend',
-    yearsExperience: 8
-  },
-  confidence: 0.95,
-  weight: 0.7
-})
-```
-
----
-
-## Relationship (Verb)
-
-A relationship is a typed, directed edge connecting two entities.
-
-| Field | Type | Indexed | Description |
-|-------|------|---------|-------------|
-| `id` | `string` | Primary key | UUID v4 (auto-generated) |
-| `from` | `string` | **GraphAdjacencyIndex** | Source entity ID |
-| `to` | `string` | **GraphAdjacencyIndex** | Target entity ID |
-| `type` | `VerbType` | GraphAdjacencyIndex (as `verb`) | Relationship type classification |
-| `data` | `any` | — | Opaque content (overrides auto-computed vector if provided) |
-| `metadata` | `object` | — | Structured fields on the edge |
-| `weight` | `number` | — | Connection strength (0-1, default: 1.0) |
-| `confidence` | `number` | — | Relationship certainty (0-1) |
-| `evidence` | `RelationEvidence` | — | Why this relationship was detected |
-| `createdAt` | `number` | — | Creation timestamp (ms since epoch) |
-| `updatedAt` | `number` | — | Last update timestamp (ms since epoch) |
-| `service` | `string` | — | Multi-tenancy identifier |
-
-### Example
-
-```typescript
-const relId = await brain.relate({
-  from: personId,
-  to: projectId,
-  type: VerbType.WorksOn,
-  data: 'Lead engineer on the AI module',   // Optional: content for this edge
-  metadata: {                                // Optional: queryable edge fields
-    role: 'lead',
-    startDate: '2024-01-15'
-  },
-  weight: 0.9
-})
-```
-
----
-
-## Data vs Metadata
-
-This is the most important concept in Brainy's storage model:
-
-### `data` — Content for Semantic Search
-
-- Embedded into a 384-dimensional vector via the WASM embedding engine
-- Searchable via **semantic similarity** (HNSW vector index) and **hybrid text+semantic** search
-- Queried by passing `query` to `find()`:
-  ```typescript
-  brain.find({ query: 'machine learning algorithms' })
-  ```
-- **NOT** indexed by MetadataIndex — you cannot use `where` filters on `data`
-- Stored opaquely: strings, objects, numbers — anything goes
-
-### `metadata` — Structured Queryable Fields
-
-- Indexed by MetadataIndex with O(1) lookups per field
-- Queryable via `where` filters using [BFO operators](./QUERY_OPERATORS.md):
-  ```typescript
-  brain.find({
-    where: {
-      department: 'engineering',
-      yearsExperience: { greaterThan: 5 },
-      tags: { contains: 'senior' }
-    }
-  })
-  ```
-- **NOT** used for vector/semantic search
-- Must be a flat or lightly nested object
-
-### Quick Reference
-
-| | `data` | `metadata` |
-|---|---|---|
-| **Purpose** | Content for embedding / semantic search | Structured fields for filtering |
-| **Searched by** | `find({ query })` — vector similarity, hybrid text+semantic | `find({ where })` — exact, range, set operators |
-| **Indexed by** | HNSW vector index | MetadataIndex |
-| **Queryable with operators?** | No | Yes (`equals`, `greaterThan`, `oneOf`, etc.) |
-| **Auto-embedded?** | Yes (strings → 384-dim vectors) | No |
-| **Typical content** | Text descriptions, document content | Tags, dates, status flags, categories, numeric fields |
-
-### Common Pattern
-
-```typescript
-// Add an article
-await brain.add({
-  data: 'A deep dive into transformer architectures and attention mechanisms',
-  type: NounType.Document,
-  metadata: {
-    title: 'Transformer Deep Dive',
-    author: 'Dr. Chen',
-    publishedYear: 2024,
-    tags: ['AI', 'transformers', 'NLP'],
-    status: 'published'
-  }
-})
-
-// Search by content (semantic — searches data)
-const results = await brain.find({ query: 'neural network attention' })
-
-// Filter by fields (exact — queries metadata)
-const recent = await brain.find({
-  where: {
-    publishedYear: { greaterThan: 2023 },
-    status: 'published'
-  }
-})
-
-// Combine both (Triple Intelligence)
-const precise = await brain.find({
-  query: 'attention mechanisms',             // Semantic search on data
-  where: { author: 'Dr. Chen' },            // Metadata filter
-  connected: { from: authorId, depth: 1 }   // Graph traversal
-})
-```
-
----
-
-## Storage Field Naming
-
-Internally, Brainy uses different field names in storage vs the public API:
-
-| Public API (Entity/Relation) | Storage (metadata object) | Notes |
-|------------------------------|--------------------------|-------|
-| `type` | `noun` | Entity type stored as `noun` |
-| `from` | `sourceId` | Relationship source |
-| `to` | `targetId` | Relationship target |
-| `type` (on Relation) | `verb` | Relationship type stored as `verb` |
-
-When querying with `find()`, you can use:
-- `type` parameter (convenience alias, equivalent to `where.noun`)
-- `where.noun` directly
-
-```typescript
-// These are equivalent:
-brain.find({ type: NounType.Person })
-brain.find({ where: { noun: NounType.Person } })
-```
-
----
-
-## Standard Metadata Fields
-
-When you add an entity, Brainy stores these standard fields in the metadata object alongside your custom fields:
-
-| Field | Set By | Description |
-|-------|--------|-------------|
-| `noun` | System | Entity type (NounType enum value) |
-| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. |
-| `data` | System | The raw `data` value (stored opaquely) |
-| `createdAt` | System | Creation timestamp |
-| `updatedAt` | System | Last update timestamp |
-| `confidence` | User | Type classification confidence |
-| `weight` | User | Entity importance |
-| `service` | User | Multi-tenancy identifier |
-| `createdBy` | User/System | Source augmentation |
-
-On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**.
-
-### Subtype — sub-classification within a NounType
-
-`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type:
-
-```typescript
-// A Person who is an employee:
-await brain.add({
-  data: 'Avery Brooks — runs the AI lab',
-  type: NounType.Person,
-  subtype: 'employee',
-  metadata: { department: 'ai-lab' }
-})
-
-// A Document that is an invoice:
-await brain.add({
-  data: 'INV-2026-001',
-  type: NounType.Document,
-  subtype: 'invoice',
-  metadata: { amount: 1500 }
-})
-```
-
-`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`.
-
-### Subtype — sub-classification within a VerbType (7.30+)
-
-Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive — a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field on `HNSWVerbWithMetadata` and on the public `Relation<T>`:
-
-```typescript
-await brain.relate({
-  from: ceoId,
-  to: vpId,
-  type: VerbType.ReportsTo,
-  subtype: 'direct',                  // top-level standard field
-  metadata: { since: '2025-Q1' }      // user-custom fields stay in metadata
-})
-```
-
-Fast-path filter on the verb side:
-
-```typescript
-const direct = await brain.related({
-  from: ceoId,
-  type: VerbType.ReportsTo,
-  subtype: 'direct'
-})
-```
-
-The verb-side rollup at `_system/verb-subtype-statistics.json` mirrors the noun-side `_system/subtype-statistics.json` — same shape, same self-heal machinery. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`.
-
-Verbs and nouns now have full capability parity — every API on the noun side has a verb-side mirror, including the new `brain.updateRelation()` (which closed a pre-7.30 gap where relationships had no update path).
-
-### Standard verb fields
-
-The verb-side equivalent of `STANDARD_ENTITY_FIELDS` is `STANDARD_VERB_FIELDS`, exported from `src/coreTypes.ts`. Verb-specific standard fields:
-
-| Field | Description |
-|---|---|
-| `verb` | The VerbType enum value |
-| `sourceId` / `targetId` | The two endpoints of the relationship |
-| `subtype` | Sub-classification within the VerbType (7.30+) |
-| `confidence`, `weight`, `createdAt`, `updatedAt`, `service`, `createdBy`, `data` | Same semantics as the noun-side standard fields |
-
-The companion `resolveVerbField(verb, field)` helper resolves field paths the same way `resolveEntityField` does for nouns: standard fields first, metadata fallback for everything else.
-
----
-
-## See Also
-
-- [API Reference](./api/README.md) — Complete API documentation
-- [Query Operators](./QUERY_OPERATORS.md) — All BFO operators with examples
-- [Find System](./FIND_SYSTEM.md) — Natural language find() details

docs/EXTENDING_STORAGE.md

@@ -0,0 +1,396 @@
+# 🔌 Extending Brainy Storage with Augmentations
+
+## Overview
+
+Brainy's zero-config system is **fully extensible**. Augmentations can register new storage providers, presets, and auto-detection logic that integrates seamlessly with the existing system.
+
+## How Storage Extensions Work
+
+### 1. Storage Provider Registration
+
+When an augmentation is installed, it can register a new storage provider:
+
+```typescript
+import { StorageProvider, registerStorageAugmentation } from '@soulcraft/brainy/config'
+
+const redisProvider: StorageProvider = {
+  type: 'redis',
+  name: 'Redis Storage',
+  description: 'High-performance in-memory data store',
+  priority: 10,  // Higher priority = checked first in auto-detection
+  
+  // Auto-detection logic
+  async detect(): Promise<boolean> {
+    // Check if Redis is available
+    if (process.env.REDIS_URL) {
+      try {
+        const redis = await import('ioredis')
+        const client = new redis.default(process.env.REDIS_URL)
+        await client.ping()
+        await client.quit()
+        return true
+      } catch {
+        return false
+      }
+    }
+    return false
+  },
+  
+  // Configuration builder
+  async getConfig(): Promise<any> {
+    return {
+      type: 'redis',
+      redisStorage: {
+        url: process.env.REDIS_URL,
+        prefix: 'brainy:',
+        ttl: 3600
+      }
+    }
+  }
+}
+
+// Register the provider
+registerStorageAugmentation(redisProvider)
+```
+
+### 2. Using Extended Storage
+
+Once registered, the new storage type works with zero-config:
+
+```typescript
+// Auto-detection will now check Redis
+const brain = new Brainy()  // Will use Redis if available!
+
+// Or explicitly specify
+const brain = new Brainy({ storage: 'redis' })
+
+// Or with custom config
+const brain = new Brainy({
+  storage: {
+    type: 'redis',
+    redisStorage: {
+      url: 'redis://localhost:6379',
+      prefix: 'myapp:'
+    }
+  }
+})
+```
+
+## Real-World Examples
+
+### Redis Augmentation
+
+```typescript
+// @soulcraft/brainy-redis package
+export class RedisStorageAugmentation {
+  async init() {
+    // Register the storage provider
+    registerStorageAugmentation({
+      type: 'redis',
+      name: 'Redis Storage',
+      priority: 10,
+      
+      async detect() {
+        return !!(process.env.REDIS_URL || process.env.REDIS_HOST)
+      },
+      
+      async getConfig() {
+        return {
+          type: 'redis',
+          redisStorage: {
+            url: process.env.REDIS_URL || 
+                 `redis://${process.env.REDIS_HOST}:${process.env.REDIS_PORT || 6379}`
+          }
+        }
+      }
+    })
+    
+    // Register Redis-specific presets
+    registerPresetAugmentation('redis-cache', {
+      storage: 'redis',
+      model: ModelPrecision.Q8,
+      features: ['core', 'cache'],
+      distributed: true,
+      description: 'Redis-backed cache layer',
+      category: PresetCategory.SERVICE
+    })
+  }
+}
+```
+
+### MongoDB Augmentation
+
+```typescript
+// @soulcraft/brainy-mongodb package
+export class MongoStorageAugmentation {
+  async init() {
+    registerStorageAugmentation({
+      type: 'mongodb',
+      name: 'MongoDB Storage',
+      priority: 8,
+      
+      async detect() {
+        return !!(process.env.MONGODB_URI || process.env.MONGO_URL)
+      },
+      
+      async getConfig() {
+        return {
+          type: 'mongodb',
+          mongoStorage: {
+            uri: process.env.MONGODB_URI,
+            database: 'brainy',
+            collection: 'vectors'
+          }
+        }
+      }
+    })
+  }
+}
+```
+
+### PostgreSQL + pgvector Augmentation
+
+```typescript
+// @soulcraft/brainy-postgres package
+export class PostgresStorageAugmentation {
+  async init() {
+    registerStorageAugmentation({
+      type: 'postgres',
+      name: 'PostgreSQL + pgvector',
+      priority: 9,
+      
+      async detect() {
+        const url = process.env.DATABASE_URL
+        if (url?.includes('postgres')) {
+          // Check for pgvector extension
+          const client = new Client({ connectionString: url })
+          await client.connect()
+          const result = await client.query(
+            "SELECT * FROM pg_extension WHERE extname = 'vector'"
+          )
+          await client.end()
+          return result.rows.length > 0
+        }
+        return false
+      },
+      
+      async getConfig() {
+        return {
+          type: 'postgres',
+          postgresStorage: {
+            connectionString: process.env.DATABASE_URL,
+            table: 'brainy_vectors'
+          }
+        }
+      }
+    })
+  }
+}
+```
+
+## Auto-Detection Priority
+
+Storage providers are checked in priority order:
+
+1. **Custom providers** (highest priority first)
+2. **Cloud storage** (S3, GCS, R2)
+3. **Database storage** (Redis, MongoDB, PostgreSQL)
+4. **Local storage** (filesystem, OPFS)
+5. **Memory** (fallback)
+
+```typescript
+// Example priority chain
+Redis (priority: 10) → PostgreSQL (9) → MongoDB (8) → S3 (5) → Filesystem (1) → Memory (0)
+```
+
+## Creating Custom Presets
+
+Augmentations can also register new presets:
+
+```typescript
+registerPresetAugmentation('redis-cluster', {
+  storage: 'redis',
+  model: ModelPrecision.Q8,
+  features: ['core', 'cache', 'cluster'],
+  distributed: true,
+  role: DistributedRole.HYBRID,
+  cache: {
+    hotCacheMaxSize: 100000,  // Large distributed cache
+    autoTune: true
+  },
+  description: 'Redis Cluster configuration',
+  category: PresetCategory.SERVICE
+})
+
+// Users can then use:
+const brain = new Brainy('redis-cluster')
+```
+
+## Type Safety with Extensions
+
+To maintain type safety with dynamic storage types:
+
+```typescript
+// Augmentation declares its types
+declare module '@soulcraft/brainy' {
+  interface StorageTypes {
+    redis: {
+      url: string
+      prefix?: string
+      ttl?: number
+    }
+  }
+  
+  interface PresetNames {
+    'redis-cache': 'redis-cache'
+    'redis-cluster': 'redis-cluster'
+  }
+}
+```
+
+## Best Practices for Storage Augmentations
+
+1. **Always provide auto-detection** - Check environment variables and connectivity
+2. **Set appropriate priority** - Higher for specialized storage, lower for general
+3. **Handle failures gracefully** - Return false from detect() if not available
+4. **Document requirements** - List required packages and environment variables
+5. **Provide presets** - Include common configuration patterns
+6. **Maintain compatibility** - Ensure model precision matches across instances
+
+## Example: Complete Redis Augmentation
+
+```typescript
+import { 
+  StorageProvider, 
+  registerStorageAugmentation,
+  registerPresetAugmentation,
+  PresetCategory,
+  ModelPrecision,
+  DistributedRole
+} from '@soulcraft/brainy/config'
+import Redis from 'ioredis'
+
+export class BrainyRedisAugmentation {
+  private client: Redis
+  
+  async init() {
+    // Register storage provider
+    registerStorageAugmentation({
+      type: 'redis',
+      name: 'Redis Vector Storage',
+      description: 'Redis with RediSearch for vector similarity',
+      priority: 10,
+      
+      requirements: {
+        env: ['REDIS_URL'],
+        packages: ['ioredis', 'redis']
+      },
+      
+      async detect() {
+        if (!process.env.REDIS_URL) return false
+        
+        try {
+          const client = new Redis(process.env.REDIS_URL)
+          
+          // Check for RediSearch module
+          const modules = await client.call('MODULE', 'LIST')
+          const hasRediSearch = modules.some(m => m[1] === 'search')
+          
+          await client.quit()
+          return hasRediSearch
+        } catch {
+          return false
+        }
+      },
+      
+      async getConfig() {
+        return {
+          type: 'redis',
+          redisStorage: {
+            url: process.env.REDIS_URL,
+            prefix: process.env.REDIS_PREFIX || 'brainy:',
+            index: process.env.REDIS_INDEX || 'brainy-vectors',
+            ttl: process.env.REDIS_TTL ? parseInt(process.env.REDIS_TTL) : undefined
+          }
+        }
+      }
+    })
+    
+    // Register presets
+    this.registerPresets()
+  }
+  
+  private registerPresets() {
+    // Fast cache preset
+    registerPresetAugmentation('redis-fast-cache', {
+      storage: 'redis' as any,
+      model: ModelPrecision.Q8,
+      features: ['core', 'cache', 'search'],
+      distributed: false,
+      cache: {
+        hotCacheMaxSize: 10000,
+        autoTune: true
+      },
+      description: 'Redis-backed fast cache',
+      category: PresetCategory.SERVICE
+    })
+    
+    // Distributed cache preset
+    registerPresetAugmentation('redis-distributed', {
+      storage: 'redis' as any,
+      model: ModelPrecision.AUTO,
+      features: ['core', 'cache', 'search', 'cluster'],
+      distributed: true,
+      role: DistributedRole.HYBRID,
+      cache: {
+        hotCacheMaxSize: 50000,
+        autoTune: true
+      },
+      description: 'Redis distributed cache cluster',
+      category: PresetCategory.SERVICE
+    })
+    
+    // Session store preset
+    registerPresetAugmentation('redis-sessions', {
+      storage: 'redis' as any,
+      model: ModelPrecision.Q8,
+      features: ['core', 'cache'],
+      distributed: false,
+      cache: {
+        hotCacheMaxSize: 5000,
+        autoTune: false
+      },
+      description: 'Redis session storage',
+      category: PresetCategory.SERVICE
+    })
+  }
+}
+
+// Usage after installing the augmentation:
+import { Brainy } from '@soulcraft/brainy'
+import '@soulcraft/brainy-redis'  // Registers the augmentation
+
+// Now Redis is automatically detected!
+const brain = new Brainy()  // Uses Redis if REDIS_URL is set
+
+// Or use a Redis preset
+const brain = new Brainy('redis-fast-cache')
+
+// Or explicitly configure
+const brain = new Brainy({
+  storage: 'redis',
+  model: ModelPrecision.FP32
+})
+```
+
+## Summary
+
+The extensible configuration system allows:
+
+1. **New storage types** via `registerStorageAugmentation()`
+2. **Custom presets** via `registerPresetAugmentation()`
+3. **Auto-detection logic** that integrates with zero-config
+4. **Type-safe extensions** with TypeScript declarations
+5. **Priority-based selection** for intelligent defaults
+
+This ensures Brainy can grow with new storage technologies while maintaining its zero-configuration philosophy!

docs/METADATA_CONTRACT_IMPLEMENTATION.md

@@ -0,0 +1,177 @@
+# Metadata Contract Implementation Plan
+
+## New Required Interface
+
+```typescript
+export interface BrainyAugmentation {
+  // Identity
+  name: string
+  timing: 'before' | 'after' | 'around' | 'replace'
+  operations: string[]
+  priority: number
+  
+  // REQUIRED metadata contract
+  metadata: 'none' | 'readonly' | MetadataAccess
+  
+  // Methods
+  initialize(context: AugmentationContext): Promise<void>
+  execute<T = any>(operation: string, params: any, next: () => Promise<T>): Promise<T>
+  shouldExecute?(operation: string, params: any): boolean
+  shutdown?(): Promise<void>
+}
+
+interface MetadataAccess {
+  reads?: string[] | '*'      // Fields to read, or '*' for all
+  writes?: string[] | '*'     // Fields to write, or '*' for all
+  namespace?: string           // Optional: custom namespace like '_myAug'
+}
+```
+
+## Augmentation Analysis & Classification
+
+### Category 1: No Metadata Access ('none')
+These augmentations don't read or write metadata at all:
+
+1. **CacheAugmentation** - Only caches search results
+2. **RequestDeduplicatorAugmentation** - Only deduplicates requests
+3. **ConnectionPoolAugmentation** - Only manages storage connections
+4. **StorageAugmentation** - Base storage layer, metadata handled by Brainy
+
+### Category 2: Read-Only Access ('readonly')
+These augmentations read metadata but never modify it:
+
+6. **IndexAugmentation** - Reads metadata to build indexes
+7. **MonitoringAugmentation** - Reads metadata for monitoring
+8. **MetricsAugmentation** - Reads metadata for metrics collection
+9. **BatchProcessingAugmentation** - Reads metadata to check for external IDs
+10. **EntityRegistryAugmentation** - Reads metadata to register entities
+11. **AutoRegisterEntitiesAugmentation** - Reads metadata for auto-registration
+12. **ConduitAugmentation** - Reads metadata to pass through operations
+
+### Category 3: Metadata Writers (needs specific access)
+These augmentations modify metadata and need specific field declarations:
+
+13. **SynapseAugmentation** - Writes to '_synapse' field
+    ```typescript
+    metadata: {
+      reads: '*',
+      writes: ['_synapse', '_synapseTimestamp'],
+      namespace: '_synapse'  // Uses its own namespace
+    }
+    ```
+
+14. **IntelligentVerbScoringAugmentation** - Adds scoring to verbs
+    ```typescript
+    metadata: {
+      reads: ['type', 'verb', 'source', 'target'],
+      writes: ['weight', 'confidence', 'intelligentScoring']
+    }
+    ```
+
+15. **ServerSearchAugmentation** - Might add server metadata
+    ```typescript
+    metadata: {
+      reads: '*',
+      writes: ['_server', '_syncedAt']
+    }
+    ```
+
+16. **NeuralImportAugmentation** - Enriches imported data
+    ```typescript
+    metadata: {
+      reads: '*',
+      writes: ['nounType', 'verbType', '_importedAt', '_enriched']
+    }
+    ```
+
+### Category 4: API/Server (needs analysis)
+17. **ApiServerAugmentation** - Likely read-only for serving data
+18. **StorageAugmentations** (plural) - Collection of storage implementations
+19. **ConduitAugmentations** (plural) - Collection of conduit types
+
+## Implementation Steps
+
+### Phase 1: Update Base Interface
+1. Update `BrainyAugmentation` interface to require `metadata` field
+2. Update `BaseAugmentation` class to have abstract `metadata` property
+3. Add runtime enforcement in augmentation executor
+
+### Phase 2: Update Each Augmentation
+For each augmentation, add the appropriate metadata declaration:
+
+#### Example Updates:
+
+**CacheAugmentation:**
+```typescript
+export class CacheAugmentation extends BaseAugmentation {
+  readonly name = 'cache'
+  readonly metadata = 'none' as const  // ✅ No metadata access
+  // ... rest unchanged
+}
+```
+
+```typescript
+  readonly metadata = 'readonly' as const  // ✅ Only reads for logging
+  // ... rest unchanged
+}
+```
+
+**SynapseAugmentation:**
+```typescript
+export abstract class SynapseAugmentation extends BaseAugmentation {
+  readonly name = 'synapse'
+  readonly metadata = {
+    reads: '*',
+    writes: ['_synapse', '_synapseTimestamp'],
+    namespace: '_synapse'
+  } as const
+  // ... rest unchanged
+}
+```
+
+### Phase 3: Runtime Enforcement
+Add a metadata access enforcer that:
+1. Wraps metadata objects based on declared access
+2. Throws errors if augmentation violates its contract
+3. Logs warnings in development mode
+
+```typescript
+class MetadataEnforcer {
+  enforce(augmentation: BrainyAugmentation, metadata: any): any {
+    if (augmentation.metadata === 'none') {
+      return null  // No access at all
+    }
+    
+    if (augmentation.metadata === 'readonly') {
+      return Object.freeze(deepClone(metadata))  // Read-only copy
+    }
+    
+    // For specific access, create proxy that validates
+    return new Proxy(metadata, {
+      set(target, prop, value) {
+        const access = augmentation.metadata as MetadataAccess
+        if (!access.writes?.includes(String(prop)) && access.writes !== '*') {
+          throw new Error(`Augmentation '${augmentation.name}' cannot write to field '${String(prop)}'`)
+        }
+        target[prop] = value
+        return true
+      }
+    })
+  }
+}
+```
+
+## Benefits
+1. **Type Safety** - TypeScript enforces metadata declaration
+2. **Runtime Safety** - Violations caught immediately
+3. **Documentation** - Contract shows exactly what each augmentation does
+4. **Brain-cloud Ready** - Registry can validate augmentations
+5. **Developer Friendly** - Most use simple 'none' or 'readonly'
+
+## Migration Checklist
+- [ ] Update BrainyAugmentation interface
+- [ ] Update BaseAugmentation class
+- [ ] Add MetadataEnforcer
+- [ ] Update all 19 augmentations with metadata declarations
+- [ ] Add tests for metadata enforcement
+- [ ] Update documentation

docs/MIGRATION-V3-TO-V4.md

@@ -1,569 +0,0 @@
-# Brainy v3 → v4.0.0 Migration Guide
-
-> **Migration Complexity**: Low
-> **Breaking Changes**: None (fully backward compatible)
-> **New Features**: Lifecycle management, batch operations, compression, quota monitoring
-
-## Overview
-
-Brainy v4.0.0 is a **backward-compatible** release focused on production-ready cost optimization features. Your existing v3 code will continue to work without modifications, but you'll want to enable the new v4.0.0 features for significant cost savings.
-
-**Key Benefits of Upgrading:**
-- 💰 **96% cost savings** with lifecycle policies
-- 🚀 **1000x faster** bulk deletions with batch operations
-- 📦 **60-80% space savings** with gzip compression
-- 📊 **Real-time quota monitoring** for OPFS
-- 🎯 **Zero downtime** migration
-
-## What's New in v4.0.0
-
-### 1. Lifecycle Management (Cloud Storage)
-
-**Automatic tier transitions for massive cost savings:**
-
-```typescript
-// NEW in v4.0.0
-await storage.setLifecyclePolicy({
-  rules: [{
-    id: 'archive-old-data',
-    prefix: 'entities/',
-    status: 'Enabled',
-    transitions: [
-      { days: 30, storageClass: 'STANDARD_IA' },
-      { days: 90, storageClass: 'GLACIER' }
-    ]
-  }]
-})
-```
-
-**Supported on:**
-- ✅ AWS S3 (Lifecycle + Intelligent-Tiering)
-- ✅ Google Cloud Storage (Lifecycle + Autoclass)
-- ✅ Azure Blob Storage (Lifecycle policies)
-
-### 2. Batch Operations
-
-**1000x faster bulk deletions:**
-
-```typescript
-// v3: Delete one at a time (slow, expensive)
-for (const id of idsToDelete) {
-  await brain.remove(id)  // 1000 API calls for 1000 entities
-}
-
-// v4.0.0: Batch delete (fast, cheap)
-const paths = idsToDelete.flatMap(id => [
-  `entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
-  `entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
-])
-await storage.batchDelete(paths)  // 1 API call for 1000 objects (S3)
-```
-
-**Efficiency gains:**
-- S3: 1000 objects per batch
-- GCS: 100 objects per batch
-- Azure: 256 objects per batch
-
-### 3. Compression (FileSystem)
-
-**60-80% space savings for local storage:**
-
-```typescript
-// NEW in v4.0.0
-const brain = new Brainy({
-  storage: {
-    type: 'filesystem',
-    path: './data',
-    compression: true  // Enable gzip compression
-  }
-})
-
-// Automatic compression/decompression on all reads/writes
-```
-
-### 4. Quota Monitoring (OPFS)
-
-**Prevent quota exceeded errors in browsers:**
-
-```typescript
-// NEW in v4.0.0
-const status = await storage.getStorageStatus()
-
-if (status.details.usagePercent > 80) {
-  console.warn('Approaching quota limit:', status.details)
-  // Take action: cleanup old data, notify user, etc.
-}
-```
-
-### 5. Tier Management (Azure)
-
-**Manual or automatic tier transitions:**
-
-```typescript
-// NEW in v4.0.0
-await storage.changeBlobTier(blobPath, 'Cool')  // Hot → Cool (50% savings)
-await storage.batchChangeTier([blob1, blob2], 'Archive')  // 99% savings
-
-// Rehydrate from Archive when needed
-await storage.rehydrateBlob(blobPath, 'High')  // 1-hour rehydration
-```
-
-## Storage Architecture Changes
-
-### v3.x Storage Structure
-
-```
-brainy-data/
-├── nouns/
-│   └── {uuid}.json          # Single file per entity
-├── verbs/
-│   └── {uuid}.json          # Single file per relationship
-├── metadata/
-│   └── __metadata_*.json    # Indexes
-└── _system/
-    └── statistics.json
-```
-
-### v4.0.0 Storage Structure (Automatic Migration)
-
-```
-brainy-data/
-├── entities/
-│   ├── nouns/
-│   │   ├── vectors/         # Vector + HNSW graph (NEW)
-│   │   │   ├── 00/ ... ff/  # 256 UUID shards (NEW)
-│   │   └── metadata/        # Business data (NEW)
-│   │       ├── 00/ ... ff/  # 256 UUID shards (NEW)
-│   └── verbs/
-│       ├── vectors/         # Relationship vectors (NEW)
-│       │   ├── 00/ ... ff/
-│       └── metadata/        # Relationship data (NEW)
-│           ├── 00/ ... ff/
-└── _system/                 # Unchanged
-    └── __metadata_*.json
-```
-
-**Key Changes:**
-1. **Metadata/Vector Separation**: Entities split into 2 files for optimal I/O
-2. **UUID-Based Sharding**: 256 shards for cloud storage optimization
-3. **Automatic Migration**: Brainy handles migration transparently on first run
-
-## Migration Steps
-
-### Step 1: Update Brainy Package
-
-```bash
-npm install @soulcraft/brainy@latest
-```
-
-**Check your version:**
-```bash
-npm list @soulcraft/brainy
-# Should show: @soulcraft/brainy@4.0.0
-```
-
-### Step 2: No Code Changes Required! ✅
-
-Your existing v3 code will work without modifications:
-
-```typescript
-// This v3 code works perfectly in v4.0.0
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: './data' }
-})
-
-await brain.init()
-await brain.add("content", { type: "entity" })
-const results = await brain.search("query")
-```
-
-### Step 3: First Run (Automatic Migration)
-
-On first initialization with v4.0.0:
-
-1. **Brainy detects v3 storage structure**
-2. **Transparently migrates to v4.0.0 structure**:
-   - Creates `entities/` directory
-   - Migrates `nouns/` → `entities/nouns/vectors/` + `entities/nouns/metadata/`
-   - Migrates `verbs/` → `entities/verbs/vectors/` + `entities/verbs/metadata/`
-   - Applies UUID-based sharding
-3. **Old structure preserved** (optional cleanup later)
-
-**Migration time:**
-- 10K entities: ~1 minute
-- 100K entities: ~10 minutes
-- 1M entities: ~2 hours
-
-**Zero downtime:**
-- Migration happens during init()
-- No data loss
-- Automatic rollback on error
-
-### Step 4: Enable v4.0.0 Features (Optional but Recommended)
-
-#### Enable Lifecycle Policies (Cloud Storage)
-
-**AWS S3:**
-```typescript
-// After init()
-await storage.setLifecyclePolicy({
-  rules: [{
-    id: 'optimize-storage',
-    prefix: 'entities/',
-    status: 'Enabled',
-    transitions: [
-      { days: 30, storageClass: 'STANDARD_IA' },
-      { days: 90, storageClass: 'GLACIER' }
-    ]
-  }]
-})
-
-// Or use Intelligent-Tiering (recommended)
-await storage.enableIntelligentTiering('entities/', 'auto-optimize')
-```
-
-**Google Cloud Storage:**
-```typescript
-await storage.enableAutoclass({
-  terminalStorageClass: 'ARCHIVE'
-})
-```
-
-**Azure Blob Storage:**
-```typescript
-await storage.setLifecyclePolicy({
-  rules: [{
-    name: 'optimize-blobs',
-    enabled: true,
-    type: 'Lifecycle',
-    definition: {
-      filters: { blobTypes: ['blockBlob'] },
-      actions: {
-        baseBlob: {
-          tierToCool: { daysAfterModificationGreaterThan: 30 },
-          tierToArchive: { daysAfterModificationGreaterThan: 90 }
-        }
-      }
-    }
-  }]
-})
-```
-
-#### Enable Compression (FileSystem)
-
-```typescript
-const brain = new Brainy({
-  storage: {
-    type: 'filesystem',
-    path: './data',
-    compression: true  // NEW: 60-80% space savings
-  }
-})
-```
-
-#### Use Batch Operations
-
-```typescript
-// Replace individual deletes with batch delete
-const idsToDelete = [/* ... */]
-const paths = idsToDelete.flatMap(id => {
-  const shard = id.substring(0, 2)
-  return [
-    `entities/nouns/vectors/${shard}/${id}.json`,
-    `entities/nouns/metadata/${shard}/${id}.json`
-  ]
-})
-
-await storage.batchDelete(paths)  // Much faster!
-```
-
-#### Monitor Quota (OPFS)
-
-```typescript
-// Periodically check quota in browser apps
-setInterval(async () => {
-  const status = await storage.getStorageStatus()
-  if (status.details.usagePercent > 80) {
-    notifyUser('Storage approaching limit')
-  }
-}, 60000)  // Check every minute
-```
-
-## Backward Compatibility
-
-### Guaranteed to Work (No Changes Needed)
-
-✅ All v3 APIs remain unchanged
-✅ Storage adapters backward compatible
-✅ Metadata structure unchanged
-✅ Query APIs unchanged
-✅ Configuration options unchanged
-
-### New Optional APIs (Add When Ready)
-
-- `storage.setLifecyclePolicy()` - NEW in v4.0.0
-- `storage.getLifecyclePolicy()` - NEW in v4.0.0
-- `storage.removeLifecyclePolicy()` - NEW in v4.0.0
-- `storage.enableIntelligentTiering()` - NEW in v4.0.0 (S3)
-- `storage.enableAutoclass()` - NEW in v4.0.0 (GCS)
-- `storage.batchDelete()` - NEW in v4.0.0
-- `storage.changeBlobTier()` - NEW in v4.0.0 (Azure)
-- `storage.getStorageStatus()` - Enhanced in v4.0.0
-
-## Testing Your Migration
-
-### 1. Test in Development First
-
-```typescript
-// Create test brain with v4.0.0
-const testBrain = new Brainy({
-  storage: { type: 'filesystem', path: './test-data' }
-})
-
-await testBrain.init()
-
-// Verify migration
-console.log('Initialization complete')
-
-// Test basic operations
-const id = await testBrain.add("test content", { type: "test" })
-const results = await testBrain.search("test")
-console.log('Basic operations working:', results.length > 0)
-```
-
-### 2. Verify Storage Structure
-
-```bash
-# Check new directory structure
-ls -la ./test-data/entities/nouns/vectors/
-# Should see: 00/ 01/ 02/ ... ff/ (256 shards)
-
-ls -la ./test-data/entities/nouns/metadata/
-# Should see: 00/ 01/ 02/ ... ff/ (256 shards)
-```
-
-### 3. Verify Data Integrity
-
-```typescript
-// Query all entities
-const allEntities = await testBrain.find({})
-console.log('Total entities:', allEntities.length)
-
-// Verify specific entities
-const entity = await testBrain.get(knownEntityId)
-console.log('Entity retrieved:', entity !== null)
-```
-
-### 4. Test Performance
-
-```typescript
-// Benchmark search
-const start = Date.now()
-const results = await testBrain.search("query")
-const duration = Date.now() - start
-console.log('Search time:', duration, 'ms')
-
-// Should be similar or faster than v3
-```
-
-## Rollback Procedure (If Needed)
-
-If you encounter issues, you can rollback:
-
-### Option 1: Rollback Package
-
-```bash
-# Reinstall v3
-npm install @soulcraft/brainy@^3.50.0
-
-# Restart application
-```
-
-**Important:** v3 can still read v3-structured data (preserved during migration)
-
-### Option 2: Restore from Backup
-
-```bash
-# If you backed up data before migration
-rm -rf ./data
-cp -r ./data-backup ./data
-
-# Reinstall v3
-npm install @soulcraft/brainy@^3.50.0
-```
-
-## Common Migration Scenarios
-
-### Scenario 1: Small Application (<10K Entities)
-
-**Migration time:** 1 minute
-**Recommended approach:**
-1. Update npm package
-2. Restart application (automatic migration)
-3. Enable lifecycle policies immediately
-
-### Scenario 2: Medium Application (10K-1M Entities)
-
-**Migration time:** 10 minutes - 2 hours
-**Recommended approach:**
-1. Backup data
-2. Update npm package
-3. Schedule maintenance window
-4. Restart application (automatic migration)
-5. Verify data integrity
-6. Enable lifecycle policies
-
-### Scenario 3: Large Application (1M+ Entities)
-
-**Migration time:** 2-24 hours
-**Recommended approach:**
-1. **Backup data** (critical!)
-2. Test migration on staging environment
-3. Schedule extended maintenance window
-4. Update npm package on production
-5. Restart application (automatic migration)
-6. Monitor migration progress
-7. Verify data integrity thoroughly
-8. Enable lifecycle policies gradually
-
-## Cost Savings After Migration
-
-### Enable All v4.0.0 Features
-
-**500TB Dataset Example:**
-
-**Before v4.0.0 (v3 with AWS S3 Standard):**
-```
-Storage: $138,000/year
-Operations: $5,000/year
-Total: $143,000/year
-```
-
-**After v4.0.0 (with Intelligent-Tiering):**
-```
-Storage: $51,000/year (64% savings)
-Operations: $5,000/year
-Total: $56,000/year
-```
-
-**After v4.0.0 (with Lifecycle Policies):**
-```
-Storage: $5,940/year (96% savings!)
-Operations: $5,000/year
-Total: $10,940/year
-```
-
-**Annual Savings: $132,060 (96% reduction)**
-
-## Troubleshooting
-
-### Issue: Migration takes too long
-
-**Solution:**
-- Migration is I/O bound
-- For 1M+ entities, consider:
-  - Running during off-peak hours
-  - Using faster storage (SSD vs HDD)
-  - Increasing available memory
-  - Running on more powerful instance
-
-### Issue: "Storage structure not recognized"
-
-**Solution:**
-```typescript
-// Manually trigger migration
-await brain.storage.migrateToV4()  // If automatic migration fails
-
-// Or start fresh (data loss warning!)
-await brain.storage.clear()
-await brain.init()
-```
-
-### Issue: Lifecycle policy not working
-
-**Solution:**
-```typescript
-// Verify policy is set
-const policy = await storage.getLifecyclePolicy()
-console.log('Active rules:', policy.rules)
-
-// Cloud providers may take 24-48 hours to start transitions
-// Check again after 2 days
-
-// Verify in cloud console:
-// - AWS: S3 → Bucket → Management → Lifecycle
-// - GCS: Storage → Bucket → Lifecycle
-// - Azure: Storage Account → Lifecycle management
-```
-
-### Issue: Batch delete not working
-
-**Solution:**
-```typescript
-// Ensure storage adapter supports batch delete
-const status = await storage.getStorageStatus()
-console.log('Storage type:', status.type)
-
-// Batch delete requires:
-// - S3CompatibleStorage ✅
-// - GcsStorage ✅
-// - AzureBlobStorage ✅
-// - FileSystemStorage ✅
-// - OPFSStorage ✅
-// - MemoryStorage ✅
-```
-
-## Best Practices
-
-1. ✅ **Backup before upgrading** (especially for large datasets)
-2. ✅ **Test on staging first** (verify migration works)
-3. ✅ **Monitor during migration** (watch logs for errors)
-4. ✅ **Enable lifecycle policies immediately** (start saving costs)
-5. ✅ **Use batch operations** (for any bulk cleanup)
-6. ✅ **Monitor quota** (OPFS browser apps)
-7. ✅ **Enable compression** (FileSystem storage)
-
-## Getting Help
-
-**Documentation:**
-- [AWS S3 Cost Optimization Guide](./operations/cost-optimization-aws-s3.md)
-- [GCS Cost Optimization Guide](./operations/cost-optimization-gcs.md)
-- [Azure Cost Optimization Guide](./operations/cost-optimization-azure.md)
-- [Cloudflare R2 Cost Optimization Guide](./operations/cost-optimization-cloudflare-r2.md)
-
-**Support:**
-- GitHub Issues: [https://github.com/soulcraft/brainy/issues](https://github.com/soulcraft/brainy/issues)
-- GitHub Discussions: [https://github.com/soulcraft/brainy/discussions](https://github.com/soulcraft/brainy/discussions)
-
-## Summary
-
-**Migration Checklist:**
-- ✅ Backup data
-- ✅ Update npm package (`npm install @soulcraft/brainy@latest`)
-- ✅ Restart application (automatic migration)
-- ✅ Verify data integrity
-- ✅ Enable lifecycle policies
-- ✅ Enable compression (FileSystem)
-- ✅ Use batch operations
-- ✅ Monitor cost savings
-
-**Expected Results:**
-- ✅ Zero downtime migration
-- ✅ Full backward compatibility
-- ✅ 60-96% cost savings
-- ✅ 1000x faster bulk operations
-- ✅ 60-80% space savings (with compression)
-
-**Timeline:**
-- Small app (<10K): 1 minute migration
-- Medium app (10K-1M): 10 minutes - 2 hours
-- Large app (1M+): 2-24 hours
-
-**Welcome to Brainy v4.0.0! 🎉**
-
----
-
-**Version**: v4.0.0
-**Migration Difficulty**: Low
-**Breaking Changes**: None
-**Recommended Upgrade**: Yes (significant cost savings)

docs/MODEL_LOADING_QUICK_REFERENCE.md

@@ -0,0 +1,118 @@
+# 🤖 Model Loading Quick Reference
+
+## 🚀 Common Scenarios
+
+### ✅ Development (Zero Config)
+```typescript
+const brain = new Brainy()
+await brain.init() // Downloads automatically (FP32 default)
+```
+
+### ⚡ Development (Optimized - v2.8.0+)
+```typescript
+// 75% smaller models, 99% accuracy
+const brain = new Brainy({
+  embeddingOptions: { dtype: 'q8' }
+})
+await brain.init()
+```
+
+### 🐳 Docker Production
+```dockerfile
+# Both models (recommended)
+RUN npm run download-models
+
+# Or FP32 only (compatibility)
+RUN npm run download-models:fp32
+
+# Or Q8 only (space-constrained)
+RUN npm run download-models:q8
+
+ENV BRAINY_ALLOW_REMOTE_MODELS=false
+```
+
+### ☁️ Serverless/Lambda
+```bash
+# Build step
+npm run download-models
+
+# Runtime
+export BRAINY_ALLOW_REMOTE_MODELS=false
+```
+
+### 🔒 Air-Gapped/Offline
+```bash
+# Connected machine
+npm run download-models
+tar -czf brainy-models.tar.gz ./models
+
+# Offline machine  
+tar -xzf brainy-models.tar.gz
+export BRAINY_ALLOW_REMOTE_MODELS=false
+```
+
+### 🌐 Browser/CDN
+```html
+<!-- Automatic - no setup needed -->
+<script type="module">
+  import { Brainy } from 'brainy'
+  const brain = new Brainy()
+  await brain.init() // Works in browser
+</script>
+```
+
+## 🚨 Troubleshooting
+
+| Error | Solution |
+|-------|----------|
+| "Failed to load embedding model" | `npm run download-models` |
+| "ENOENT: no such file" | Check `BRAINY_MODELS_PATH` |
+| "Network timeout" | Set `BRAINY_ALLOW_REMOTE_MODELS=false` |
+| "Permission denied" | `chmod 755 ./models` |
+| "Out of memory" | Increase container memory limit |
+
+## 🎯 Environment Variables
+
+| Variable | Values | Purpose |
+|----------|--------|---------|
+| `BRAINY_ALLOW_REMOTE_MODELS` | `true`/`false` | Allow/block downloads |
+| `BRAINY_MODELS_PATH` | `./models` | Model storage path |
+| `BRAINY_Q8_CONFIRMED` | `true`/`false` | Silence Q8 compatibility warnings |
+| `NODE_ENV` | `production` | Environment detection |
+
+## 📦 Model Info
+
+### FP32 (Default)
+- **Model**: All-MiniLM-L6-v2 
+- **Dimensions**: 384 (fixed)
+- **Size**: 90MB
+- **Accuracy**: 100% (baseline)
+- **Location**: `./models/Xenova/all-MiniLM-L6-v2/onnx/model.onnx`
+
+### Q8 (Optional - v2.8.0+)
+- **Model**: All-MiniLM-L6-v2 (quantized)
+- **Dimensions**: 384 (same)
+- **Size**: 23MB (75% smaller!)
+- **Accuracy**: ~99% (minimal loss)
+- **Location**: `./models/Xenova/all-MiniLM-L6-v2/onnx/model_quantized.onnx`
+
+**⚠️ Important**: FP32 and Q8 create different embeddings and are incompatible!
+
+## ✅ Verification Commands
+
+```bash
+# Check FP32 model exists
+ls ./models/Xenova/all-MiniLM-L6-v2/onnx/model.onnx
+
+# Check Q8 model exists  
+ls ./models/Xenova/all-MiniLM-L6-v2/onnx/model_quantized.onnx
+
+# Test offline mode
+BRAINY_ALLOW_REMOTE_MODELS=false npm test
+
+# Download fresh models (both)
+rm -rf ./models && npm run download-models
+
+# Download specific model variant
+rm -rf ./models && npm run download-models:q8
+```

docs/NEURAL_API_PATTERNS.md

@@ -0,0 +1,736 @@
+# 🧠 Neural API Patterns: AI-Powered Intelligence
+
+> Learn the correct patterns for Brainy's Neural API. Avoid performance pitfalls and use AI features effectively.
+
+## 🚨 Critical: Access Neural APIs Correctly
+
+### ❌ **WRONG - Outdated Access Patterns**
+
+```typescript
+// DON'T DO THIS - Outdated documentation patterns
+import { Brainy } from '@soulcraft/brainy'  // ❌ Wrong import
+const brain = new Brainy()                  // ❌ Old class name
+
+// These may not work as expected:
+const neural = brain.neural                     // ❌ May be undefined
+```
+
+### ✅ **CORRECT - Modern Neural Access**
+
+```typescript
+// ✅ Use modern Brainy class
+import { Brainy } from '@soulcraft/brainy'
+
+const brain = new Brainy()
+await brain.init()
+
+// ✅ Neural API is available after initialization
+const clusters = await brain.neural.clusters()
+const similarity = await brain.neural.similar('item1', 'item2')
+```
+
+## 🔍 Similarity Analysis Patterns
+
+### ❌ **WRONG - Inefficient Similarity Checks**
+
+```typescript
+// DON'T DO THIS - N² comparisons
+const items = await brain.find({ limit: 1000 })
+const similarities = []
+
+for (const item1 of items) {
+  for (const item2 of items) {
+    if (item1.id !== item2.id) {
+      const sim = await brain.neural.similar(item1.id, item2.id)  // ❌ Millions of calls
+      similarities.push({ from: item1.id, to: item2.id, score: sim })
+    }
+  }
+}
+```
+
+### ✅ **CORRECT - Efficient Similarity Patterns**
+
+```typescript
+// ✅ Pattern 1: Find neighbors (much more efficient)
+const item = await brain.get('target-item-id')
+const neighbors = await brain.neural.neighbors(item.id, {
+  limit: 10,                     // Top 10 most similar
+  threshold: 0.7,               // Minimum similarity
+  includeScores: true           // Include similarity scores
+})
+
+console.log(`Found ${neighbors.length} similar items`)
+
+// ✅ Pattern 2: Batch similarity for specific pairs
+const itemPairs = [
+  ['item1', 'item2'],
+  ['item1', 'item3'],
+  ['item2', 'item3']
+]
+
+const similarities = await Promise.all(
+  itemPairs.map(async ([a, b]) => ({
+    from: a,
+    to: b,
+    score: await brain.neural.similar(a, b)
+  }))
+)
+
+// ✅ Pattern 3: Text-to-text similarity (no need for IDs)
+const textSimilarity = await brain.neural.similar(
+  "Machine learning is fascinating",
+  "AI and deep learning are interesting",
+  { detailed: true }  // Get explanation of similarity
+)
+
+console.log(`Similarity: ${textSimilarity.score}`)
+console.log(`Explanation: ${textSimilarity.explanation}`)
+
+// ✅ Pattern 4: Vector-level similarity for optimization
+const vector1 = await brain.embed("First concept")
+const vector2 = await brain.embed("Second concept")
+const vectorSimilarity = await brain.neural.similar(vector1, vector2)
+```
+
+## 🎯 Clustering Patterns
+
+### ❌ **WRONG - Uncontrolled Clustering**
+
+```typescript
+// DON'T DO THIS - Clustering everything without limits
+const everything = await brain.find({ limit: 100000 })  // ❌ Too much data
+const clusters = await brain.neural.clusters()          // ❌ May crash or timeout
+```
+
+### ✅ **CORRECT - Smart Clustering Patterns**
+
+```typescript
+// ✅ Pattern 1: Controlled clustering with limits
+const recentItems = await brain.find({
+  where: {
+    createdAt: { $gte: Date.now() - 30 * 24 * 60 * 60 * 1000 } // Last 30 days
+  },
+  limit: 1000  // Reasonable limit
+})
+
+const clusters = await brain.neural.clusters(
+  recentItems.map(item => item.id),
+  {
+    algorithm: 'kmeans',        // Reliable algorithm
+    maxClusters: 10,           // Reasonable number
+    threshold: 0.75,           // High similarity required
+    iterations: 50             // Convergence limit
+  }
+)
+
+// ✅ Pattern 2: Domain-specific clustering
+const techDocs = await brain.find({
+  where: { category: 'technology', type: 'document' },
+  limit: 500
+})
+
+const techClusters = await brain.neural.clusterByDomain(
+  'category',  // Group by this field
+  {
+    items: techDocs.map(doc => doc.id),
+    minClusterSize: 3,         // Minimum items per cluster
+    maxClusters: 8
+  }
+)
+
+// ✅ Pattern 3: Temporal clustering for time-series data
+const timebasedClusters = await brain.neural.clusterByTime(
+  'createdAt',  // Time field
+  'week',       // Time window (hour, day, week, month)
+  {
+    items: recentItems.map(item => item.id),
+    overlap: 0.2,              // 20% overlap between windows
+    minPerWindow: 5            // Minimum items per time window
+  }
+)
+
+// ✅ Pattern 4: Streaming clustering for large datasets
+async function clusterLargeDataset() {
+  const clusterStream = brain.neural.clusterStream({
+    batchSize: 100,            // Process 100 items at a time
+    updateInterval: 1000,      // Update clusters every 1000 items
+    maxMemory: 512 * 1024 * 1024  // 512MB memory limit
+  })
+
+  const allClusters = []
+  for await (const batch of clusterStream) {
+    console.log(`Processed ${batch.processed} items, found ${batch.clusters.length} clusters`)
+    allClusters.push(...batch.clusters)
+  }
+
+  return allClusters
+}
+```
+
+## 🔍 Neighbor Discovery Patterns
+
+### ❌ **WRONG - Manual Similarity Searches**
+
+```typescript
+// DON'T DO THIS - Reinventing neighbor search
+async function findSimilarManually(targetId: string) {
+  const allItems = await brain.find({ limit: 10000 })  // ❌ Load everything
+  const similarities = []
+
+  for (const item of allItems) {
+    if (item.id !== targetId) {
+      const score = await brain.neural.similar(targetId, item.id)  // ❌ Slow
+      if (score > 0.7) {
+        similarities.push({ id: item.id, score })
+      }
+    }
+  }
+
+  return similarities.sort((a, b) => b.score - a.score).slice(0, 10)  // ❌ Inefficient
+}
+```
+
+### ✅ **CORRECT - Optimized Neighbor Patterns**
+
+```typescript
+// ✅ Pattern 1: Basic neighbor search
+const neighbors = await brain.neural.neighbors('target-item-id', {
+  limit: 20,                   // Top 20 neighbors
+  threshold: 0.6,             // Minimum similarity
+  includeMetadata: true,      // Include item metadata
+  includeDistances: true      // Include exact similarity scores
+})
+
+// ✅ Pattern 2: Filtered neighbor search
+const filteredNeighbors = await brain.neural.neighbors('article-id', {
+  limit: 10,
+  filter: {
+    type: 'document',          // Only find similar documents
+    status: 'published',       // Only published content
+    language: 'en'            // Only English content
+  },
+  excludeIds: ['self-id', 'duplicate-id']  // Exclude specific items
+})
+
+// ✅ Pattern 3: Multi-level neighbor discovery
+async function discoverNeighborNetwork(rootId: string, maxDepth = 2) {
+  const network = new Map()
+  const visited = new Set()
+  const queue = [{ id: rootId, depth: 0 }]
+
+  while (queue.length > 0) {
+    const { id, depth } = queue.shift()!
+
+    if (visited.has(id) || depth >= maxDepth) continue
+    visited.add(id)
+
+    const neighbors = await brain.neural.neighbors(id, {
+      limit: 5,
+      threshold: 0.8
+    })
+
+    network.set(id, neighbors)
+
+    // Add neighbors to queue for next depth level
+    if (depth < maxDepth - 1) {
+      neighbors.forEach(neighbor => {
+        queue.push({ id: neighbor.id, depth: depth + 1 })
+      })
+    }
+  }
+
+  return network
+}
+
+// ✅ Pattern 4: Recommendation engine
+async function getRecommendations(userId: string) {
+  // Get user's liked items
+  const userItems = await brain.find({
+    connected: { to: userId, via: 'liked-by' }
+  })
+
+  // Find neighbors for each liked item
+  const allNeighbors = await Promise.all(
+    userItems.map(item =>
+      brain.neural.neighbors(item.id, {
+        limit: 10,
+        threshold: 0.7,
+        excludeConnected: { to: userId, via: 'liked-by' }  // Exclude already liked
+      })
+    )
+  )
+
+  // Aggregate and rank recommendations
+  const recommendations = new Map()
+  allNeighbors.flat().forEach(neighbor => {
+    const current = recommendations.get(neighbor.id) || { score: 0, count: 0 }
+    current.score += neighbor.score
+    current.count += 1
+    recommendations.set(neighbor.id, current)
+  })
+
+  // Return top recommendations by average score
+  return Array.from(recommendations.entries())
+    .map(([id, stats]) => ({
+      id,
+      avgScore: stats.score / stats.count,
+      mentions: stats.count
+    }))
+    .sort((a, b) => b.avgScore - a.avgScore)
+    .slice(0, 10)
+}
+```
+
+## 🏗️ Hierarchy & Structure Patterns
+
+### ❌ **WRONG - Manual Hierarchy Building**
+
+```typescript
+// DON'T DO THIS - Building hierarchies manually
+async function buildHierarchyManually(rootId: string) {
+  const root = await brain.get(rootId)
+  const allItems = await brain.find({ limit: 1000 })  // ❌ Load everything
+
+  // Manual tree building with nested loops
+  const hierarchy = { root, children: [] }
+  // ... complex manual logic
+}
+```
+
+### ✅ **CORRECT - Semantic Hierarchy Patterns**
+
+```typescript
+// ✅ Pattern 1: Automatic semantic hierarchy
+const hierarchy = await brain.neural.hierarchy('root-concept-id', {
+  maxDepth: 4,                 // Maximum tree depth
+  minSimilarity: 0.6,         // Minimum similarity for inclusion
+  branchingFactor: 5,         // Maximum children per node
+  algorithm: 'semantic'       // Use semantic clustering
+})
+
+// ✅ Pattern 2: Domain-specific hierarchy
+const techHierarchy = await brain.neural.hierarchy('technology-id', {
+  filter: { category: 'technology' },
+  weights: {
+    semantic: 0.7,            // 70% based on content similarity
+    metadata: 0.3            // 30% based on metadata similarity
+  },
+  includeMetrics: true        // Include hierarchy quality metrics
+})
+
+// ✅ Pattern 3: Multi-root hierarchy for complex domains
+async function buildMultiRootHierarchy(rootIds: string[]) {
+  const hierarchies = await Promise.all(
+    rootIds.map(rootId =>
+      brain.neural.hierarchy(rootId, {
+        maxDepth: 3,
+        crossReference: true    // Allow cross-hierarchy connections
+      })
+    )
+  )
+
+  // Merge hierarchies and find connections
+  const merged = {
+    roots: hierarchies,
+    connections: await findCrossHierarchyConnections(hierarchies)
+  }
+
+  return merged
+}
+
+async function findCrossHierarchyConnections(hierarchies: any[]) {
+  const connections = []
+
+  for (let i = 0; i < hierarchies.length; i++) {
+    for (let j = i + 1; j < hierarchies.length; j++) {
+      const leafNodes1 = extractLeafNodes(hierarchies[i])
+      const leafNodes2 = extractLeafNodes(hierarchies[j])
+
+      // Find connections between leaf nodes of different hierarchies
+      for (const leaf1 of leafNodes1) {
+        const neighbors = await brain.neural.neighbors(leaf1.id, {
+          limit: 5,
+          threshold: 0.8,
+          filter: { id: { $in: leafNodes2.map(l => l.id) } }
+        })
+
+        connections.push(...neighbors.map(n => ({
+          from: leaf1.id,
+          to: n.id,
+          hierarchyPair: [i, j],
+          similarity: n.score
+        })))
+      }
+    }
+  }
+
+  return connections
+}
+```
+
+## 🚨 Outlier Detection Patterns
+
+### ❌ **WRONG - Manual Outlier Detection**
+
+```typescript
+// DON'T DO THIS - Manual statistical outlier detection
+async function findOutliersManually() {
+  const items = await brain.find({ limit: 1000 })
+  const similarities = []
+
+  // Calculate average similarity for each item (expensive)
+  for (const item of items) {
+    let totalSim = 0
+    let count = 0
+    for (const other of items) {
+      if (item.id !== other.id) {
+        totalSim += await brain.neural.similar(item.id, other.id)  // ❌ N² operations
+        count++
+      }
+    }
+    similarities.push({ id: item.id, avgSimilarity: totalSim / count })
+  }
+
+  // Manual outlier calculation
+  const threshold = calculateManualThreshold(similarities)  // ❌ Complex statistics
+  return similarities.filter(s => s.avgSimilarity < threshold)
+}
+```
+
+### ✅ **CORRECT - AI-Powered Outlier Detection**
+
+```typescript
+// ✅ Pattern 1: Automatic outlier detection
+const outliers = await brain.neural.outliers({
+  threshold: 0.3,              // Items with < 30% avg similarity to others
+  method: 'isolation-forest',  // AI-based outlier detection
+  contamination: 0.1,         // Expect ~10% outliers
+  includeReasons: true        // Explain why each item is an outlier
+})
+
+console.log(`Found ${outliers.length} outliers`)
+outliers.forEach(outlier => {
+  console.log(`Outlier: ${outlier.id}, Score: ${outlier.score}`)
+  console.log(`Reason: ${outlier.reason}`)
+})
+
+// ✅ Pattern 2: Domain-specific outlier detection
+const techOutliers = await brain.neural.outliers({
+  filter: { category: 'technology' },
+  features: ['content', 'metadata.tags', 'metadata.complexity'],
+  method: 'local-outlier-factor',
+  neighbors: 20               // Consider 20 nearest neighbors
+})
+
+// ✅ Pattern 3: Temporal outlier detection
+const recentOutliers = await brain.neural.outliers({
+  timeWindow: '7days',        // Look at last 7 days
+  baseline: '30days',         // Compare to 30-day baseline
+  method: 'statistical',      // Use statistical methods
+  autoThreshold: true         // Automatically determine threshold
+})
+
+// ✅ Pattern 4: Streaming outlier detection
+async function detectOutliersInStream() {
+  const outlierStream = brain.neural.outlierStream({
+    batchSize: 50,
+    updateInterval: 100,       // Check every 100 new items
+    adaptiveThreshold: true    // Threshold adapts as data changes
+  })
+
+  for await (const batch of outlierStream) {
+    console.log(`Batch ${batch.batchNumber}: ${batch.outliers.length} outliers detected`)
+
+    // Process outliers immediately
+    for (const outlier of batch.outliers) {
+      await handleOutlier(outlier)
+    }
+  }
+}
+
+async function handleOutlier(outlier: any) {
+  // Flag for manual review
+  await brain.update(outlier.id, {
+    metadata: {
+      flagged: true,
+      outlierScore: outlier.score,
+      outlierReason: outlier.reason,
+      flaggedAt: Date.now()
+    }
+  })
+}
+```
+
+## 📊 Visualization Patterns
+
+### ❌ **WRONG - Manual Visualization Data Preparation**
+
+```typescript
+// DON'T DO THIS - Manual coordinate calculation
+async function prepareVisualizationManually() {
+  const items = await brain.find({ limit: 500 })
+  const coordinates = []
+
+  // Manual dimensionality reduction (complex math)
+  for (const item of items) {
+    const vector = await brain.embed(item.data)
+    // Complex PCA/t-SNE calculations manually
+    const x = complexMathFunction(vector)  // ❌ Error-prone
+    const y = anotherComplexFunction(vector)
+    coordinates.push({ id: item.id, x, y })
+  }
+
+  return coordinates
+}
+```
+
+### ✅ **CORRECT - AI-Powered Visualization**
+
+```typescript
+// ✅ Pattern 1: Automatic 2D visualization
+const visualization = await brain.neural.visualize({
+  dimensions: 2,               // 2D plot
+  algorithm: 'umap',          // UMAP for better clustering preservation
+  maxItems: 1000,             // Performance limit
+  includeMetadata: true,      // Include item metadata in output
+  colorBy: 'cluster'          // Color points by cluster membership
+})
+
+// Result format:
+// {
+//   points: [{ id, x, y, cluster, metadata }, ...],
+//   clusters: [{ id, centroid: [x, y], members: [...] }, ...],
+//   stats: { stress, kruskalStress, trustworthiness }
+// }
+
+// ✅ Pattern 2: 3D visualization for complex data
+const viz3D = await brain.neural.visualize({
+  dimensions: 3,
+  algorithm: 'tsne',
+  perplexity: 30,             // t-SNE parameter
+  learningRate: 200,          // t-SNE learning rate
+  iterations: 1000            // Number of optimization steps
+})
+
+// ✅ Pattern 3: Interactive visualization with filtering
+const interactiveViz = await brain.neural.visualize({
+  filter: {
+    type: 'document',
+    createdAt: { $gte: Date.now() - 7 * 24 * 60 * 60 * 1000 }
+  },
+  groupBy: 'category',        // Group points by metadata field
+  showLabels: true,           // Include text labels
+  labelField: 'title',        // Field to use for labels
+  includeEdges: true,         // Show connections between similar items
+  edgeThreshold: 0.8          // Only show high-similarity connections
+})
+
+// ✅ Pattern 4: Real-time visualization updates
+class LiveVisualization {
+  private visualization: any = null
+  private updateInterval: NodeJS.Timeout | null = null
+
+  async start() {
+    // Initial visualization
+    this.visualization = await brain.neural.visualize({
+      dimensions: 2,
+      algorithm: 'umap',
+      maxItems: 500,
+      includeMetadata: true
+    })
+
+    // Update every 30 seconds
+    this.updateInterval = setInterval(async () => {
+      await this.update()
+    }, 30000)
+  }
+
+  async update() {
+    // Get recent items
+    const recentItems = await brain.find({
+      where: {
+        createdAt: { $gte: Date.now() - 30000 }  // Last 30 seconds
+      },
+      limit: 50
+    })
+
+    if (recentItems.length > 0) {
+      // Incrementally update visualization
+      const updates = await brain.neural.updateVisualization(
+        this.visualization.id,
+        {
+          newItems: recentItems.map(item => item.id),
+          algorithm: 'incremental'  // Faster incremental updates
+        }
+      )
+
+      this.visualization = { ...this.visualization, ...updates }
+      this.onUpdate(updates)
+    }
+  }
+
+  onUpdate(updates: any) {
+    // Emit updates to frontend
+    console.log(`Visualization updated: ${updates.newPoints.length} new points`)
+  }
+
+  stop() {
+    if (this.updateInterval) {
+      clearInterval(this.updateInterval)
+    }
+  }
+}
+```
+
+## 🚀 Performance Optimization Patterns
+
+### ✅ **High-Performance Neural Operations**
+
+```typescript
+// ✅ Pattern 1: Batch processing for similarity
+async function batchSimilarityCalculation(itemPairs: Array<[string, string]>) {
+  const batchSize = 100
+  const results = []
+
+  for (let i = 0; i < itemPairs.length; i += batchSize) {
+    const batch = itemPairs.slice(i, i + batchSize)
+
+    const batchResults = await Promise.all(
+      batch.map(async ([a, b]) => ({
+        from: a,
+        to: b,
+        similarity: await brain.neural.similar(a, b)
+      }))
+    )
+
+    results.push(...batchResults)
+
+    // Progress reporting
+    console.log(`Processed ${Math.min(i + batchSize, itemPairs.length)}/${itemPairs.length} pairs`)
+  }
+
+  return results
+}
+
+// ✅ Pattern 2: Caching expensive operations
+class NeuralCache {
+  private clusterCache = new Map()
+  private similarityCache = new Map()
+  private readonly TTL = 5 * 60 * 1000  // 5 minutes
+
+  async getClusters(options: any) {
+    const key = JSON.stringify(options)
+    const cached = this.clusterCache.get(key)
+
+    if (cached && Date.now() - cached.timestamp < this.TTL) {
+      return cached.data
+    }
+
+    const clusters = await brain.neural.clusters(undefined, options)
+    this.clusterCache.set(key, {
+      data: clusters,
+      timestamp: Date.now()
+    })
+
+    return clusters
+  }
+
+  async getSimilarity(id1: string, id2: string) {
+    // Create consistent cache key regardless of order
+    const key = [id1, id2].sort().join('-')
+    const cached = this.similarityCache.get(key)
+
+    if (cached && Date.now() - cached.timestamp < this.TTL) {
+      return cached.data
+    }
+
+    const similarity = await brain.neural.similar(id1, id2)
+    this.similarityCache.set(key, {
+      data: similarity,
+      timestamp: Date.now()
+    })
+
+    return similarity
+  }
+}
+
+// ✅ Pattern 3: Memory-efficient streaming
+async function processLargeDatasetEfficiently() {
+  const stream = brain.neural.clusterStream({
+    batchSize: 50,              // Small batches for memory efficiency
+    maxMemoryMB: 256,          // Memory limit
+    diskCache: true,           // Use disk for temporary storage
+    compression: true          // Compress cached data
+  })
+
+  const results = []
+  let totalProcessed = 0
+
+  for await (const batch of stream) {
+    // Process batch immediately, don't accumulate in memory
+    const processedBatch = await processBatch(batch)
+
+    // Save to disk or send to another service
+    await saveBatchToDisk(processedBatch)
+
+    totalProcessed += batch.items.length
+    console.log(`Processed ${totalProcessed} items`)
+
+    // Clear memory
+    batch.items = null
+  }
+
+  return { totalProcessed }
+}
+
+// ✅ Pattern 4: Parallel processing with worker threads
+async function parallelNeuralProcessing(items: string[]) {
+  const numWorkers = require('os').cpus().length
+  const batchSize = Math.ceil(items.length / numWorkers)
+
+  const workers = []
+  for (let i = 0; i < numWorkers; i++) {
+    const batch = items.slice(i * batchSize, (i + 1) * batchSize)
+    if (batch.length > 0) {
+      workers.push(processWorkerBatch(batch))
+    }
+  }
+
+  const results = await Promise.all(workers)
+  return results.flat()
+}
+
+async function processWorkerBatch(batch: string[]) {
+  // This would run in a worker thread in real implementation
+  return Promise.all(
+    batch.map(async itemId => {
+      const neighbors = await brain.neural.neighbors(itemId, { limit: 5 })
+      return { itemId, neighbors }
+    })
+  )
+}
+```
+
+## 🎯 Summary: Neural API Best Practices
+
+| ❌ **Avoid These Patterns** | ✅ **Use These Instead** |
+|---------------------------|------------------------|
+| Manual similarity loops | `brain.neural.neighbors()` |
+| Uncontrolled clustering | Limit items and set maxClusters |
+| Manual outlier detection | `brain.neural.outliers()` |
+| Manual visualization prep | `brain.neural.visualize()` |
+| Loading entire datasets | Streaming and batch processing |
+| No caching | Cache expensive operations |
+| Blocking operations | Parallel and async patterns |
+
+---
+
+**🎉 Following these patterns gives you:**
+- 🚀 **Optimized performance** with intelligent algorithms
+- 🧠 **AI-powered insights** instead of manual statistics
+- 📊 **Rich visualizations** for data exploration
+- 🎯 **Accurate clustering** with semantic understanding
+- 🚨 **Smart outlier detection** for quality control
+- ⚡ **Scalable processing** for large datasets
+
+**Next:** [Augmentation Patterns →](./AUGMENTATION_PATTERNS.md) | [Core API Patterns →](./CORE_API_PATTERNS.md)

docs/PERFORMANCE.md

@@ -2,70 +2,27 @@
 
 ## Performance Characteristics
 
-Brainy achieves high performance through carefully optimized data structures and algorithms. The tables below describe each component by its **algorithmic complexity** — the durable, defensible guarantee. The example latencies are figures from a single 100-item run on one machine (see [Benchmarks](#benchmarks)); they are illustrative, not a committed benchmark, and vary with hardware, embedding model, and storage backend. The one component with a committed scale assertion is the graph adjacency index (`tests/performance/graph-scale-performance.test.ts:238`).
+Brainy achieves industry-leading performance through carefully optimized data structures and algorithms. All performance claims are verified through actual benchmarks on production code.
 
 ### Core Performance Summary
 
-| Component | Operation | Time Complexity | Example latency (100-item run)\* | Data Structure |
+| Component | Operation | Time Complexity | Measured Performance | Data Structure |
 |-----------|-----------|-----------------|---------------------|----------------|
 | **Metadata Index** | Exact match | **O(1)** | 0.8ms | `Map<string, Set<string>>` |
 | **Metadata Index** | Range query | **O(log n) + O(k)** | 0.6ms | Sorted array + binary search |
 | **Graph Index** | Get neighbors | **O(1)** | 0.09ms | `Map<string, Set<string>>` |
-| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | Hierarchical graph |
+| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | HNSW hierarchical graph |
 | **NLP Parser** | Query parsing | **O(m)** | 8.9ms | 220 pre-computed patterns |
 | **Type-Field Affinity** | Field matching | **O(f)** | 0.1ms | Type-specific field cache |
 | **Type Detection** | Noun/Verb matching | **O(t)** | 0.3ms | Pre-embedded type vectors |
 | **Triple Intelligence** | Combined query | **O(1) to O(log n)** | 1.8ms | Parallel execution |
 
-\* Illustrative single-run figures at 100 items on one machine — not a committed benchmark. Only the graph index carries an asserted scale bound (measured <1 ms per neighbor lookup up to 1M relationships, `tests/performance/graph-scale-performance.test.ts:238`).
-
 Where:
 - `n` = number of items in index
 - `k` = number of results returned
 - `m` = number of patterns to check
 - `f` = number of fields for entity type
-- `t` = number of types (42 nouns, 127 verbs)
-
-### brain.get() Metadata-Only Optimization
-
-`brain.get()` returns **metadata only by default**, skipping the 384-dimensional
-embedding — the bulk of an entity's payload. Callers that need the vector opt in
-with `{ includeVectors: true }`.
-
-| Operation | Default (metadata-only) | With `includeVectors: true` | Use Case |
-|-----------|-------------------------|-----------------------------|----------|
-| **brain.get()** | Skips vector load | Loads full vector | VFS, existence checks, metadata |
-| **VFS readFile() / readdir()** | Inherits metadata-only path | n/a | File operations, directory listings |
-
-**Key Innovation**: Lazy vector loading — only load the 384-dimensional embedding when explicitly needed.
-
-The integration test `tests/integration/metadata-only-comprehensive.test.ts:306`
-asserts metadata-only `get()` is faster than the full-entity `get()`
-(`metadataTime < fullTime`). The *magnitude* of the speedup is
-environment-dependent (the percentage assertion in
-`tests/integration/vfs-performance-v5.11.1.test.ts` is intentionally skipped on
-CI for that reason), so no fixed percentage is quoted here.
-
-**Why this matters**:
-- Most `brain.get()` calls don't need vectors (VFS, admin tools, import utilities, data APIs)
-- The embedding dominates an entity's serialized size, so skipping it is the largest win
-- **Zero code changes** for most applications — automatic by default
-
-**When to use what**:
-```typescript
-// DEFAULT: Metadata-only (skips the vector load) - use for:
-const entity = await brain.get(id)
-// - VFS operations (readFile, stat, readdir)
-// - Existence checks: if (await brain.get(id)) ...
-// - Metadata access: entity.data, entity.type, entity.metadata
-// - Relationship traversal
-
-// EXPLICIT: Full entity (same as before) - use ONLY for:
-const entity = await brain.get(id, { includeVectors: true })
-// - Computing similarity on THIS entity
-// - Manual vector operations
-// - Vector index graph traversal
-```
+- `t` = number of types (30+ nouns, 40+ verbs)
 
 ## Architecture Deep Dive
 
@@ -152,14 +109,14 @@ class GraphAdjacencyIndex {
 
 **Key Innovation:** Pure Map/Set operations - no database queries, no loops, just direct memory access.
 
-### 4. Vector Index - O(log n)
+### 4. HNSW Vector Search - O(log n)
 
-The default vector index (`JsHnswVectorIndex`) provides logarithmic approximate nearest neighbor search through a hierarchical graph:
+Hierarchical Navigable Small World graphs provide logarithmic approximate nearest neighbor search:
 
 ```typescript
-class JsHnswVectorIndex {
+class HNSWIndex {
   private nouns: Map<string, HNSWNoun> = new Map()
-
+  
   interface HNSWNoun {
     id: string
     vector: number[]
@@ -183,7 +140,7 @@ The NLP processor uses **zero hardcoded fields** - everything is discovered dyna
 
 ```typescript
 class NaturalLanguageProcessor {
-  // Pre-embedded NounTypes (42) and VerbTypes (127) - ONLY hardcoded vocabularies
+  // Pre-embedded NounTypes (30+) and VerbTypes (40+) - ONLY hardcoded vocabularies
   private nounTypeEmbeddings = new Map<string, Vector>()
   private verbTypeEmbeddings = new Map<string, Vector>()
   
@@ -205,7 +162,7 @@ class NaturalLanguageProcessor {
 5. **Query Optimization**: Process low-cardinality type-specific fields first
 
 **Performance Characteristics:**
-- Type detection: O(t) where t = 169 total types (42 noun + 127 verb)
+- Type detection: O(t) where t = 70 total types (30 noun + 40 verb)
 - Field matching: O(f) where f = fields for detected type (typically 5-15)
 - Validation: O(1) lookup in type-field affinity map
 - No hardcoded assumptions - learns from actual data patterns
@@ -247,7 +204,7 @@ const results = await Promise.all(searchPromises)
 |-----------|--------------|---------|
 | Metadata Index | ~40 bytes/entry | `(key_size + 8) × unique_values + 8 × total_items` |
 | Graph Index | ~24 bytes/edge | `16 × edges + 8 × nodes` |
-| Vector Index | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
+| HNSW | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
 | Pattern Library | 394KB fixed | Pre-computed, shared across instances |
 | Type Embeddings | ~60KB fixed | 70 types × 384 dimensions × 4 bytes, cached |
 | Field Embeddings | ~5KB dynamic | Actual fields × 384 dimensions × 4 bytes |
@@ -261,40 +218,34 @@ const results = await Promise.all(searchPromises)
 
 ## Benchmarks
 
-### Illustrative Single Run (100 items, one machine)
-
-Example output from a single 100-item run — illustrative only, not a committed
-benchmark; absolute numbers vary by hardware. The values feed the
-[Core Performance Summary](#core-performance-summary) example-latency column.
+### Real-world Performance Test (100 items)
 
 ```
-Metadata exact match: 0.818ms (50 items matched)
-Metadata range query: 0.631ms (40 items in range)
-Graph neighbor lookup: 0.092ms (2 connections)
-Vector k-NN search:    1.773ms (10 nearest neighbors)
-NLP query parsing:     8.906ms (full natural language)
-Triple Intelligence:   1.830ms (combined query)
+📊 Metadata exact match: 0.818ms (50 items matched)
+📊 Metadata range query: 0.631ms (40 items in range)
+🔗 Graph neighbor lookup: 0.092ms (2 connections)
+🎯 Vector k-NN search: 1.773ms (10 nearest neighbors)
+🧠 NLP query parsing: 8.906ms (full natural language)
+⚡ Triple Intelligence: 1.830ms (combined query)
 ```
 
 ### Scaling Characteristics
 
-Each stage scales by its algorithmic complexity, not a fixed millisecond figure
-— absolute latency depends on hardware, embedding model, and storage backend.
-Only the graph adjacency index carries a committed scale assertion:
+| Items | Metadata O(1) | Range O(log n) | Graph O(1) | Vector O(log n) |
+|-------|---------------|----------------|------------|-----------------|
+| 100 | 0.8ms | 0.6ms | 0.09ms | 1.8ms |
+| 1,000 | 0.8ms | 0.9ms | 0.09ms | 2.5ms |
+| 10,000 | 0.8ms | 1.2ms | 0.09ms | 3.2ms |
+| 100,000 | 0.8ms | 1.5ms | 0.09ms | 4.1ms |
+| 1,000,000 | 0.8ms | 1.8ms | 0.09ms | 5.0ms |
 
-| Query stage | Complexity | Scaling behavior |
-|-------------|------------|------------------|
-| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
-| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
-| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
-| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
-| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
+*Note: O(1) operations maintain constant time regardless of scale*
 
 ## Comparison with Other Systems
 
 | System | Metadata Filter | Graph Traversal | Vector Search | Natural Language |
 |--------|-----------------|-----------------|---------------|------------------|
-| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) vector index | 220 patterns |
+| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) HNSW | 220 patterns |
 | Neo4j | O(log n) B-tree | O(k) traversal | Not native | Not native |
 | Elasticsearch | O(log n) inverted | Not native | O(n) brute force* | Basic tokenization |
 | PostgreSQL | O(log n) B-tree | O(k) recursive | O(n) brute force* | Full-text only |
@@ -320,62 +271,9 @@ Only the graph adjacency index carries a committed scale assertion:
 - ✅ **No Network Calls**: Everything runs locally, including embeddings
 - ✅ **Thread-Safe**: Immutable data structures where possible
 - ✅ **Memory Bounded**: Configurable cache sizes and automatic cleanup
-- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
+- ✅ **Horizontally Scalable**: Stateless operations support clustering
 - ✅ **Zero Stubs**: Every line of code is production-ready
 
-## Lazy Loading Performance
-
-Brainy supports two initialization modes for optimal performance across different use cases:
-
-### Mode 1: Auto-Rebuild (Default)
-
-```javascript
-const brain = new Brainy()
-await brain.init()  // Rebuilds indexes during init (~500ms-3s for 10K entities)
-```
-
-**Performance:**
-- Init time: 500ms-3s (depends on dataset size)
-- First query: Instant (indexes already loaded)
-- Use case: Traditional applications, long-running servers
-
-### Mode 2: Lazy Loading
-
-```javascript
-const brain = new Brainy({ disableAutoRebuild: true })
-await brain.init()  // Returns instantly (0-10ms)
-
-const results = await brain.find({ limit: 10 })  // First query triggers rebuild (~50-200ms)
-const more = await brain.find({ limit: 100 })    // Subsequent queries instant (0ms check)
-```
-
-**Performance:**
-- Init time: 0-10ms (instant)
-- First query: 50-200ms (includes index rebuild for 1K-10K entities)
-- Subsequent queries: 0ms check (instant)
-- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
-
-**Concurrency Safety:**
-```javascript
-// 100 concurrent queries immediately after init
-await brain.init()
-
-const promises = Array.from({ length: 100 }, () =>
-  brain.find({ limit: 10 })
-)
-
-const results = await Promise.all(promises)
-// ✅ Only 1 rebuild triggered (mutex)
-// ✅ All 100 queries return correct results
-// ✅ Total time: ~60ms (not 6000ms!)
-```
-
-**Use Cases for Lazy Loading:**
-- **Serverless/Edge**: Minimize cold start time (0-10ms init)
-- **Development**: Faster restarts during development
-- **Large datasets**: Defer index loading until needed
-- **Read-heavy workloads**: Writes don't wait for index rebuild
-
 ## Zero Configuration Required
 
 Brainy is designed to be **smart enough to tune itself dynamically**. No configuration needed:
@@ -384,53 +282,112 @@ Brainy is designed to be **smart enough to tune itself dynamically**. No configu
 // That's it. Brainy handles everything.
 const brain = new Brainy()
 await brain.init()
-
-// Or with lazy loading for serverless
-const brain = new Brainy({ disableAutoRebuild: true })
-await brain.init()  // Instant (0-10ms)
 ```
 
-### Automatic Self-Tuning
+### Automatic Self-Tuning (Current & Planned)
 
+**✅ Currently Implemented:**
 - **Metadata Index**: Auto-builds sorted indices for range queries on first use
 - **Graph Index**: Auto-flushes every 30 seconds
-- **Default Tuning**: Research-based vector index defaults
+- **Default Tuning**: Research-based defaults (M=16, ef=200)
 - **Lazy Loading**: Indices built only when needed
 - **Cache Management**: LRU caches with TTL
 
+**🚧 Planned Enhancements:**
+- **Dynamic Storage Selection**: Auto-switch between memory/disk based on size
+- **Adaptive Index Parameters**: Adjust M and ef based on query patterns
+- **Smart Cache Sizing**: Scale caches based on available memory
+- **Predictive Optimization**: Learn from usage patterns
+
 ### Intelligent Defaults
 
-- **Vector recall** = `'balanced'` (M=16, ef=200): right for most datasets
-- **Cache TTL** = 5 min: balances freshness and performance
-- **Flush interval** = 30 s: non-blocking background persistence
+All defaults are research-based and production-tested:
+- **HNSW M=16**: Optimal balance of recall/speed for most datasets
+- **efConstruction=200**: High quality graph construction
+- **Cache TTL=5min**: Balances freshness with performance
+- **Flush Interval=30s**: Non-blocking background persistence
 
-### Vector Index Tuning Knobs
+### Progressive Enhancement
 
-Brainy 8.0 exposes two knobs on `config.vector`:
+Brainy learns and improves over time:
+1. **Query Pattern Learning**: Frequently used patterns get cached
+2. **Index Optimization**: Auto-rebuilds indices when fragmented
+3. **Memory Management**: Coordinates caches across all components
+4. **Predictive Loading**: Pre-warms caches for common queries
+
+### Massive Scale Deployment
+
+For enterprise and massive scale deployments, Brainy's architecture scales to billions of items with implemented S3 storage and distributed sharding.
+
+**Currently Implemented:**
+- Memory storage (production-ready)
+- Disk storage (production-ready)
+- S3-compatible storage (AWS S3, Cloudflare R2, Google Cloud Storage, MinIO, Backblaze B2)
+- Distributed sharding with ConsistentHashRing
+- Single-node deployment (scales to ~1M items)
+- Multi-node deployment with sharding (scales to billions)
+
+**Available Today:**
 
 ```javascript
-const brain = new Brainy({
-  vector: {
-    recall: 'fast',                // 'fast' | 'balanced' | 'accurate'
-    persistMode: 'deferred'        // 'immediate' | 'deferred'
+// S3-compatible storage for unlimited scale - WORKS NOW
+const brain = new Brainy({ 
+  storage: { 
+    type: 's3',
+    bucketName: 'my-brainy-data',
+    region: 'us-east-1',
+    credentials: {
+      accessKeyId: 'YOUR_ACCESS_KEY',
+      secretAccessKey: 'YOUR_SECRET_KEY'
+    }
+    // Works with: AWS S3, MinIO, Cloudflare R2, Backblaze B2, Google Cloud Storage
+  }
+})
+
+// Cloudflare R2 storage - WORKS NOW
+const brain = new Brainy({ 
+  storage: { 
+    type: 'r2',
+    bucketName: 'my-brainy-data',
+    accountId: 'YOUR_ACCOUNT_ID',
+    accessKeyId: 'YOUR_R2_ACCESS_KEY',
+    secretAccessKey: 'YOUR_R2_SECRET_KEY'
+  }
+})
+
+// Google Cloud Storage - WORKS NOW
+const brain = new Brainy({ 
+  storage: { 
+    type: 'gcs',
+    bucketName: 'my-brainy-data',
+    region: 'us-central1',
+    credentials: {
+      accessKeyId: 'YOUR_ACCESS_KEY',
+      secretAccessKey: 'YOUR_SECRET_KEY'
+    }
   }
 })
 ```
 
-The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cor`) can replace it with a higher-performing implementation; the public knobs stay the same.
-
 ### Scale Scenarios
 
-| Scale | Items | Storage Strategy | Performance |
-|-------|-------|------------------|-------------|
-| **Small** | <10K | Memory | Sub-millisecond |
-| **Medium** | 10K-1M | Filesystem | 1-5ms |
-| **Large** | 1M-10M | Filesystem + tuned cache | 2-10ms |
-| **Massive** | 10M+ | Filesystem + native vector provider + service-layer sharding | 5-20ms |
+| Scale | Items | Storage Strategy | Performance | Status |
+|-------|-------|-----------------|-------------|--------|
+| **Small** | <10K | Memory (automatic) | Sub-millisecond | ✅ Implemented |
+| **Medium** | 10K-1M | Disk with memory cache | 1-5ms | ✅ Implemented |
+| **Large** | 1M-100M | S3 with memory cache | 2-10ms | ✅ Implemented |
+| **Massive** | 100M-10B | S3 + distributed sharding | 5-20ms | ✅ Implemented |
+| **Planetary** | 10B+ | Multi-region S3 + Edge cache | 10-50ms | 🚧 Roadmap |
 
-For >10M entities, run multiple Brainy processes behind your own routing layer — Brainy 8.0 doesn't ship cluster coordination.
+### S3-Compatible Storage Benefits
 
-### Architecture
+- **Unlimited Scale**: No practical limit on dataset size
+- **Cost Effective**: $0.023/GB/month for standard storage
+- **Durability**: 99.999999999% (11 9's) durability
+- **Global**: Multi-region replication available
+- **Compatible**: Works with any S3-compatible API (MinIO, R2, B2)
+
+### Distributed Architecture (Implemented)
 
 ```
 ┌─────────────────────────────────────────┐
@@ -442,51 +399,103 @@ For >10M entities, run multiple Brainy processes behind your own routing layer
 │         Brainy Core                     │
 │   (Triple Intelligence Engine)          │
 ├─────────────────────────────────────────┤
-│  Memory     │  Vector     │  Metadata   │
-│  Cache      │  Index      │  Index      │
+│  Memory     │  Shard      │  Metadata   │
+│  Cache      │  Manager    │  Index      │
 └─────────────┬───────────────────────────┘
               │
 ┌─────────────▼───────────────────────────┐
 │         Storage Layer                   │
 ├──────────┬──────────┬──────────────────┤
-│ Vectors  │  Graph   │   Files          │
-│ (sharded)│  Edges   │  (filesystem)    │
+│   HNSW   │  Graph   │   Objects        │
+│  Vectors │  Edges   │   (S3/R2/GCS)    │
 └──────────┴──────────┴──────────────────┘
 ```
 
-For off-site replication, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`).
+**Distributed Sharding (Implemented):**
+- ConsistentHashRing with 150 virtual nodes
+- 64 shards by default
+- Replication factor of 3
+- Automatic rebalancing on node addition/removal
+
+### Auto-Sharding for Horizontal Scale (Implemented)
+
+Brainy includes a complete sharding implementation with ConsistentHashRing:
+
+```javascript
+import { ShardManager } from '@soulcraft/brainy/distributed'
+
+// Create shard manager with custom configuration
+const shardManager = new ShardManager({
+  shardCount: 64,           // Default: 64 shards
+  replicationFactor: 3,     // Default: 3 replicas
+  virtualNodes: 150,        // Default: 150 virtual nodes
+  autoRebalance: true       // Default: true
+})
+
+// Add nodes to the cluster
+shardManager.addNode('node-1')
+shardManager.addNode('node-2')
+shardManager.addNode('node-3')
+
+// Sharding automatically:
+// - Uses consistent hashing for even distribution
+// - Maintains replicas for fault tolerance
+// - Rebalances on node changes
+// - Provides O(1) shard lookups
+```
 
 ### Performance at Scale
 
-- **Metadata queries**: O(1) HashMap
-- **Graph traversal**: O(1) adjacency lookup
-- **Vector search**: O(log n)
-- **Write throughput**: 50K+ writes/second per process (filesystem, batched)
+Even at massive scale, Brainy maintains excellent performance:
+
+- **Metadata queries**: Still O(1) with distributed hash tables
+- **Graph traversal**: O(1) with edge locality optimization  
+- **Vector search**: O(log n) with hierarchical sharding
+- **Write throughput**: 100K+ writes/second with S3 batching
 - **Read throughput**: 1M+ reads/second with caching
 
-### Zero-Config with Autoscaling
+### Zero-Config with Autoscaling (Implemented)
 
+Brainy includes extensive autoscaling capabilities:
+
+**✅ Implemented Autoscaling:**
 - **AutoConfiguration System**: Detects environment and adjusts settings
 - **Learning from Performance**: `learnFromPerformance()` adapts based on metrics
 - **Auto-flush**: Graph index (30s), Metadata index (configurable)
-- **Auto-optimize**: Enabled by default in graph and vector indices
+- **Auto-optimize**: Enabled by default in Graph and HNSW indices
+- **Auto-rebalance**: Shards automatically rebalance on node changes
 - **Zero-config presets**: Production, development, minimal modes
 - **Adaptive memory**: Scales caches based on available memory
+- **Environment detection**: Browser vs Node.js vs Serverless
+
+**🚧 Roadmap Autoscaling:**
+- Dynamic HNSW parameter adjustment (M, ef)
+- Predictive query pattern caching
+- Multi-region auto-replication
+- Automatic cross-node data migration
 
 ## Implementation Status
 
-### Fully Implemented and Production-Ready
+### ✅ Fully Implemented and Production-Ready
 - **O(1) metadata lookups** via HashMaps (exact match)
 - **O(log n) range queries** via sorted arrays with lazy building
 - **O(1) graph traversal** via adjacency maps
-- **O(log n) vector search** via the default JS index, swappable for a native provider
+- **O(log n) vector search** via HNSW
 - **220 NLP patterns** with pre-computed embeddings
-- **Filesystem and memory storage** adapters
+- **S3-compatible storage** (AWS S3, R2, GCS, MinIO, B2)
+- **Distributed sharding** with ConsistentHashRing
 - **Auto-configuration system** with environment detection
 - **Zero-config operation** with intelligent defaults
 - **Auto-flush and auto-optimize** in indices
-- **Low-latency Triple Intelligence queries** (O(log n) vector + O(1) metadata/graph)
+- **Sub-2ms response times** for complex queries
+
+### 🚧 Roadmap Features
+- Dynamic HNSW parameter tuning
+- Predictive query pattern caching
+- Multi-region S3 replication
+- Automatic cross-node data migration
+- Edge caching layer
 
 ## Conclusion
 
-Brainy delivers on its promise of **production-ready Triple Intelligence** with documented algorithmic-complexity guarantees and a committed graph-scale benchmark (`tests/performance/graph-scale-performance.test.ts`). All listed features are fully implemented and tested. No stubs, no mocks — just real, working code with characterized performance.
+Brainy delivers on its promise of **production-ready Triple Intelligence** with measured, verified performance characteristics. All listed features are fully implemented, tested, and benchmarked. No stubs, no mocks, no theoretical claims - just real, working code with measured performance.

docs/PLUGINS.md

@@ -1,486 +0,0 @@
----
-title: Plugin System
-slug: guides/plugins
-public: true
-category: guides
-template: guide
-order: 4
-description: Replace any Brainy subsystem — distance functions, embeddings, vector index, metadata index, aggregation — with a custom implementation or optional native acceleration.
-next:
-  - guides/storage-adapters
----
-
-# Plugin Development Guide
-
-Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
-
-## Architecture Overview
-
-Brainy's plugin system uses **named providers** — string keys mapped to implementations. During `init()`, brainy:
-
-1. Imports each package listed in the `plugins` config array
-2. Activates each plugin, passing a `BrainyPluginContext`
-3. The plugin calls `context.registerProvider(key, implementation)` for each subsystem it provides
-4. Brainy checks each provider key and wires the implementation into its internal pipeline
-
-Installing the first-party accelerator is the opt-in: with the default config, brainy probes for `@soulcraft/cor` and loads it when present. Everything except "not installed" fails **loud** — a present-but-broken accelerator makes `init()` throw rather than silently degrading to the JS engines.
-
-```typescript
-const brain = new Brainy()                                  // @soulcraft/cor auto-detected when installed
-const pinned = new Brainy({ plugins: ['@soulcraft/cor'] })  // or pin exactly what loads
-const plain = new Brainy({ plugins: [] })                   // or opt out of detection entirely
-```
-
-| `plugins` value | Behavior |
-|---|---|
-| `undefined` (default) | Guarded auto-detection of `@soulcraft/cor`: not installed → no plugins, silently; installed → loads + announces; installed-but-broken → `init()` throws |
-| `false` / `[]` | No plugins, no detection (explicit opt-out) |
-| `['@soulcraft/cor']` | Load only the listed packages; a listed plugin that fails to load throws |
-
-Plugins registered programmatically via `brain.use(plugin)` are always activated regardless of the `plugins` config.
-
-If no plugin provides a given key, brainy uses its built-in JavaScript implementation. This means brainy works perfectly standalone — plugins only enhance performance or add capabilities.
-
-## Creating a Plugin
-
-### 1. Implement the `BrainyPlugin` interface
-
-```typescript
-import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
-
-const myPlugin: BrainyPlugin = {
-  name: 'my-brainy-plugin',  // Must be unique (typically your npm package name)
-
-  async activate(context: BrainyPluginContext): Promise<boolean> {
-    // Register your providers here
-    context.registerProvider('distance', myFastDistanceFunction)
-
-    // Return true if activation succeeded, false to skip
-    return true
-  },
-
-  async deactivate(): Promise<void> {
-    // Optional cleanup when brainy.close() is called
-  }
-}
-
-export default myPlugin
-```
-
-### 2. Package exports
-
-Your package must export the plugin as the default export so brainy's plugin loader can resolve it:
-
-```typescript
-// index.ts
-export { default } from './plugin.js'
-```
-
-### 3. Registration
-
-**Config-based:** List your package name in the brainy config:
-
-```typescript
-const brain = new Brainy({
-  plugins: ['my-brainy-plugin']
-})
-await brain.init()
-```
-
-**Programmatic registration:** For plugins not installed as npm packages, use `brain.use()`:
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-import myPlugin from './my-plugin.js'
-
-const brain = new Brainy()
-brain.use(myPlugin)
-await brain.init()
-```
-
-## Provider Keys Reference
-
-Each key has a specific expected signature. Brainy checks for these during `init()` and wires them into the appropriate code paths.
-
-### Core Providers
-
-#### `distance`
-**Type:** `(a: number[], b: number[]) => number`
-
-Replaces the default cosine distance function used in vector search and neural APIs. This is the highest-impact single provider — it's called for every vector comparison.
-
-```typescript
-context.registerProvider('distance', (a: number[], b: number[]): number => {
-  // Your SIMD-accelerated or GPU distance calculation
-  return myFastCosineDistance(a, b)
-})
-```
-
-#### `embeddings`
-**Type:** `(text: string | string[]) => Promise<number[] | number[][]>`
-
-Replaces the built-in WASM embedding engine. Called for every `brain.add()`, `brain.update()`, and `brain.find()` operation that involves text.
-
-```typescript
-context.registerProvider('embeddings', async (text: string | string[]) => {
-  if (Array.isArray(text)) {
-    return myEngine.embedBatch(text)
-  }
-  return myEngine.embed(text)
-})
-```
-
-#### `embedBatch`
-**Type:** `(texts: string[]) => Promise<number[][]>`
-
-Dedicated batch embedding provider. When registered, brainy uses this for bulk operations (import, reindex, batch add) instead of calling the `embeddings` provider N times. This enables true single-forward-pass batch processing.
-
-Priority order for batch operations:
-1. `embedBatch` provider (single forward pass — fastest)
-2. `embeddings` provider with `Promise.all()` (N individual calls)
-3. Built-in WASM batch API (fallback)
-
-```typescript
-context.registerProvider('embedBatch', async (texts: string[]) => {
-  // Process all texts in a single forward pass
-  return myEngine.batchEmbed(texts)
-})
-```
-
-### Index Providers
-
-> **Write-path invariant (the change-feed contract).** Every canonical
-> mutation flows through Brainy's generation-store commit points — index
-> providers are invoked *inside* that commit and never originate canonical
-> writes of their own. The `brain.onChange` change feed is emitted from those
-> commit points and relies on this: **a plugin must never introduce a write
-> path that bypasses the generation-store commit.** If a future provider ever
-> needs a direct native ingest path, it must either route through the commit
-> or emit equivalent change events — otherwise every `onChange` consumer
-> (live UIs, cache invalidation, realtime sync) silently develops a blind
-> spot.
-
-#### `vector`
-**Type:** `(config: object, distanceFunction: Function, options: object) => VectorIndexProvider-compatible`
-
-Factory function that creates a vector index instance. The returned object must implement the `VectorIndexProvider` public API:
-
-- `addItem(item: { id: string, vector: number[] }): Promise<string>`
-- `search(queryVector: number[], k: number, filter?, options?): Promise<Array<[string, number]>>`
-- `removeItem(id: string): Promise<boolean>`
-- `size(): number`
-- `clear(): void`
-- `flush(): Promise<number>`
-- `rebuild(options?): Promise<void>`
-- `getDirtyNodeCount(): number`
-- `getPersistMode(): 'immediate' | 'deferred'`
-- `getEntryPointId(): string | null`
-- `getMaxLevel(): number`
-- `getDimension(): number | null`
-- `getConfig(): object`
-- `getDistanceFunction(): Function`
-- `enableCOW(parent): void`
-- `setUseParallelization(boolean): void`
-
-For type-aware indexes (separate graph per noun type), also implement:
-- `getIndexForType(type: string): VectorIndexProvider` (duck-typed detection)
-- `search(queryVector, k, type?, filter?, options?): Promise<Array<[string, number]>>`
-
-```typescript
-context.registerProvider('vector', (config, distanceFn, options) => {
-  return new MyNativeVectorIndex(config, distanceFn, options)
-})
-```
-
-#### The readiness contract (all three index providers)
-
-A provider that **persists its derived index** should implement the optional readiness
-members so a warm reopen never pays a redundant rebuild-from-canonical:
-
-- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
-  `brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
-  and **before the rebuild gate**.
-- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
-  loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
-  exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
-  `totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
-  while fully durable. Never return `true` if the durable state failed to load: the
-  signal is honest in both directions, and a not-ready provider gets its rebuild even
-  when `size() > 0`.
-- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
-  migration); brainy skips its rebuild entirely.
-
-Providers that implement none of these keep the size/count heuristics — correct for
-engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
-
-#### `metadataIndex`
-**Type:** `(storage: StorageAdapter) => MetadataIndexManager-compatible`
-
-Factory function that creates a metadata index. The returned object must implement the `MetadataIndexManager` interface including `init()`, `addEntity()`, `removeEntity()`, `query()`, `flush()`, `clear()`, etc.
-
-```typescript
-context.registerProvider('metadataIndex', (storage) => {
-  return new MyNativeMetadataIndex(storage)
-})
-```
-
-#### `graphIndex`
-**Type:** `(storage: StorageAdapter) => GraphAdjacencyIndex-compatible`
-
-Factory function that creates a graph adjacency index for relationship tracking (verbs/triples). Must implement the `GraphAdjacencyIndex` interface including `addVerb()`, `getVerbsBySource()`, `getVerbsByTarget()`, `flush()`, etc.
-
-```typescript
-context.registerProvider('graphIndex', (storage) => {
-  return new MyNativeGraphIndex(storage)
-})
-```
-
-#### `aggregation`
-**Type:** `(storage: StorageAdapter) => AggregationProvider-compatible`
-
-Factory function that creates an aggregation engine for write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows. The returned object must implement the `AggregationProvider` interface.
-
-```typescript
-context.registerProvider('aggregation', (storage) => {
-  return new MyNativeAggregationEngine(storage)
-})
-```
-
-When provided by an optional native acceleration plugin (such as `@soulcraft/cor`), this enables:
-- Compiled source filters (vs per-entity JS object traversal)
-- Precise MIN/MAX via sorted data structures (vs lazy recompute)
-- Parallel aggregate rebuild across CPU cores
-- SIMD-accelerated timestamp bucketing
-
-### Utility Providers
-
-#### `cache`
-**Type:** `UnifiedCache`
-
-Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`).
-
-```typescript
-import type { UnifiedCache } from '@soulcraft/brainy/internals'
-
-context.registerProvider('cache', myNativeCache)
-```
-
-#### `entityIdMapper`
-**Type:** `(storage: StorageAdapter) => EntityIdMapper-compatible`
-
-Factory for bidirectional UUID ↔ integer mapping used by roaring bitmaps. Must implement `getOrAssign()`, `getUuid()`, `getInt()`, `has()`, `remove()`, `flush()`, `clear()`.
-
-#### `roaring`
-**Type:** `RoaringBitmap32 class`
-
-Replacement for the roaring bitmap implementation. Used internally by the metadata index for set operations. Must be API-compatible with `roaring-wasm`.
-
-#### `msgpack`
-**Type:** `{ encode: (data: any) => Buffer, decode: (buffer: Buffer) => any }`
-
-Native msgpack encode/decode for SSTable serialization.
-
-### Analytics Providers (Native-Only)
-
-These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cor`) is installed.
-
-Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it.
-
-#### `analytics:hyperloglog`
-Approximate distinct counts. Count unique values (e.g., unique merchants) across millions of records using ~16KB of memory with ~1% error. Each update is O(1).
-
-#### `analytics:tdigest`
-Streaming percentiles. Compute P50/P90/P95/P99 from streaming data without storing all values. Uses ~4KB per digest with ~1% accuracy at the tails.
-
-#### `analytics:countmin`
-Frequency estimation. Find the most common values (e.g., top-K merchants) using ~40KB with 0.1% error. O(1) per update.
-
-#### `analytics:anomaly`
-Real-time anomaly detection. Flag statistically unusual values at write-time using exponentially weighted moving averages. 64 bytes per group, sub-microsecond decisions.
-
-#### `aggregation:mmap`
-Persistent aggregate storage via memory-mapped files. Aggregate state survives process crashes without explicit flush. Zero serialization overhead.
-
----
-
-## Storage Adapter Plugins
-
-Plugins can register custom storage backends that users reference by name.
-
-### Implementing a Storage Adapter
-
-```typescript
-import type { StorageAdapterFactory } from '@soulcraft/brainy/plugin'
-import type { StorageAdapter } from '@soulcraft/brainy'
-
-class MyStorageAdapter implements StorageAdapter {
-  async init(): Promise<void> { /* ... */ }
-  async saveNoun(noun: HNSWNoun): Promise<void> { /* ... */ }
-  async getNoun(id: string): Promise<HNSWNounWithMetadata | null> { /* ... */ }
-  async deleteNoun(id: string): Promise<void> { /* ... */ }
-  // ... implement all StorageAdapter methods
-}
-```
-
-### Registering a Storage Adapter
-
-```typescript
-context.registerProvider('storage:my-backend', {
-  name: 'my-backend',
-  create: (config: Record<string, unknown>) => {
-    return new MyStorageAdapter(config)
-  }
-} satisfies StorageAdapterFactory)
-```
-
-Users can then use your storage:
-
-```typescript
-const brain = new Brainy({ storage: 'my-backend', myBackendOption: 'value' })
-```
-
-## Import Paths
-
-Brainy provides three entry points for plugin developers:
-
-| Import Path | Contents | Stability |
-|-------------|----------|-----------|
-| `@soulcraft/brainy` | Public API, types, StorageAdapter | Stable (semver) |
-| `@soulcraft/brainy/plugin` | BrainyPlugin, BrainyPluginContext, StorageAdapterFactory | Stable (semver) |
-| `@soulcraft/brainy/internals` | UnifiedCache, EntityIdMapper, logger utilities | Internal (may change between minor versions) |
-
-## Diagnostics
-
-Brainy provides a `diagnostics()` method to verify plugin wiring:
-
-```typescript
-const brain = new Brainy()
-await brain.init()
-
-const diag = brain.diagnostics()
-console.log(diag)
-// {
-//   version: '7.14.0',
-//   plugins: { active: ['my-plugin'], count: 1 },
-//   providers: {
-//     metadataIndex: { source: 'default' },
-//     graphIndex: { source: 'default' },
-//     embeddings: { source: 'plugin' },
-//     embedBatch: { source: 'plugin' },
-//     distance: { source: 'plugin' },
-//     vector: { source: 'default' },
-//     ...
-//   },
-//   indexes: {
-//     vector: { size: 0, type: 'JsHnswVectorIndex' },
-//     metadata: { type: 'MetadataIndexManager', initialized: true },
-//     graph: { type: 'GraphAdjacencyIndex', initialized: true, wiredToStorage: true }
-//   }
-// }
-```
-
-The CLI also supports diagnostics:
-
-```bash
-brainy diagnostics
-```
-
-### Init-Time Summary
-
-When a plugin is active, brainy automatically logs a provider summary after `init()`:
-
-```
-[brainy] Plugin activated: @soulcraft/cor
-[brainy] Providers: 8/10 native (@soulcraft/cor) | default: vector, cache
-```
-
-This tells you at a glance how many subsystems are accelerated and which ones are falling back to JavaScript. The log respects `config.silent`.
-
-### Fail-Fast for Production
-
-Use `requireProviders()` after `init()` to guarantee specific providers are plugin-supplied. This prevents silent fallback to JavaScript in deployments where you expect native acceleration:
-
-```typescript
-const brain = new Brainy()
-await brain.init()
-
-// Throws immediately if any of these are using JS fallback
-brain.requireProviders(['distance', 'embeddings', 'metadataIndex', 'graphIndex'])
-```
-
-If a required provider is missing, the error message tells you exactly what's wrong:
-
-```
-[brainy] Required providers using JS fallback: graphIndex.
-Active plugins: @soulcraft/cor.
-These providers must be supplied by a plugin for this deployment.
-Check plugin installation, license, and native module availability.
-```
-
-This is the recommended pattern for production deployments with paid plugins — fail at startup rather than silently degrading performance.
-
-## Complete Example: Distance Acceleration Plugin
-
-A minimal but useful plugin that provides SIMD-accelerated distance calculations:
-
-```typescript
-// simd-distance-plugin/src/plugin.ts
-import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
-
-// Hypothetical native module
-import { simdCosineDistance } from './native.js'
-
-const simdDistancePlugin: BrainyPlugin = {
-  name: 'brainy-simd-distance',
-
-  async activate(context: BrainyPluginContext): Promise<boolean> {
-    // Check if SIMD is available on this platform
-    if (!checkSimdSupport()) {
-      console.log('[simd-distance] SIMD not available, skipping')
-      return false  // Don't activate — brainy uses JS fallback
-    }
-
-    context.registerProvider('distance', simdCosineDistance)
-    return true
-  }
-}
-
-export default simdDistancePlugin
-```
-
-```json
-// simd-distance-plugin/package.json
-{
-  "name": "brainy-simd-distance",
-  "main": "./dist/plugin.js",
-  "types": "./dist/plugin.d.ts",
-  "peerDependencies": {
-    "@soulcraft/brainy": ">=7.0.0"
-  }
-}
-```
-
-Usage:
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy({ plugins: ['brainy-simd-distance'] })
-await brain.init()
-
-// Verify it's active
-const diag = brain.diagnostics()
-console.log(diag.providers.distance) // { source: 'plugin' }
-```
-
-## Design Principles
-
-1. **Brainy works perfectly without plugins.** Every provider has a JavaScript fallback. Plugins only improve performance or add capabilities.
-
-2. **Provider keys are string-based.** The plugin system is not coupled to any specific plugin. Any package can register any provider.
-
-3. **Clean separation.** Plugins access brainy through the documented `BrainyPluginContext` interface. No direct access to internal classes is needed.
-
-4. **Fail-safe activation.** If a plugin throws during `activate()`, brainy logs a warning and continues with defaults. A broken plugin never prevents brainy from working.
-
-5. **Lifecycle management.** `deactivate()` is called during `brainy.close()` for resource cleanup. Native resources, connections, and file handles should be released here.

docs/PRODUCTION_SERVICE_ARCHITECTURE.md

@@ -1,562 +0,0 @@
-# Production Service Architecture Guide
-
-**How to use Brainy optimally in production services (Bun, Node.js, Deno)**
-
-> **Recommended Runtime:** [Bun](https://bun.sh) provides best performance with Brainy's Candle WASM engine. All examples work with both Bun and Node.js.
-
----
-
-## The Problem: Instance-per-Request Anti-Pattern
-
-### ❌ What NOT to Do
-
-```typescript
-// WRONG - Creates new instance EVERY request
-app.get('/api/entities', async (req, res) => {
-  const brain = new Brainy({ storage: { path: './brainy-data' } })
-  await brain.init()  // FULL INITIALIZATION EVERY TIME!
-  const entities = await brain.find(...)
-  res.json(entities)
-})
-```
-
-### Why This is Terrible
-
-After 40 API calls:
-- **40 Brainy instances** running simultaneously
-- **20GB memory** (40 × 500MB per instance)
-- **2 seconds wasted** (40 × 50ms initialization)
-- **Zero cache benefit** (each instance has its own empty cache)
-- **Index rebuilding** on every request (TypeAware HNSW, LSM-trees, etc.)
-- **Memory leaks** (old instances may not GC properly)
-
----
-
-## ✅ The Solution: Singleton Pattern
-
-**ONE Brainy instance per service, shared across ALL requests.**
-
-### Performance Comparison
-
-| Metric | Instance-per-Request | Singleton (Optimal) |
-|--------|---------------------|---------------------|
-| Memory (40 requests) | 20GB | 500MB |
-| Request 1 latency | 60ms | 60ms (one-time init) |
-| Request 2+ latency | 60ms (no cache!) | 2ms (80% cache hit!) |
-| Cache hit rate | 0% | 80%+ |
-| Speedup | - | **30x faster** |
-
----
-
-## Implementation Patterns
-
-### Pattern 1: Simple Singleton (Recommended)
-
-```typescript
-// server.ts
-import { Brainy } from '@soulcraft/brainy'
-
-// SINGLETON INSTANCE
-let brainInstance: Brainy | null = null
-
-async function getBrain(): Promise<Brainy> {
-  if (brainInstance) {
-    return brainInstance
-  }
-
-  console.log('🧠 Initializing Brainy singleton...')
-
-  brainInstance = new Brainy({
-    storage: {
-      path: './brainy-data',
-      autoOptimize: true
-    },
-    cache: {
-      maxSize: 1000,      // Shared across ALL requests
-      ttl: 3600000,       // 1 hour
-      enableMetrics: true
-    },
-    augmentations: {
-      include: ['cache', 'metrics', 'display', 'vfs']
-    }
-  })
-
-  await brainInstance.init()
-  console.log('✅ Brainy ready')
-
-  return brainInstance
-}
-
-// Initialize BEFORE starting server
-async function startServer() {
-  await getBrain()  // One-time initialization
-
-  app.get('/api/entities', async (req, res) => {
-    const brain = await getBrain()  // Reuses same instance!
-    const entities = await brain.find(req.query)
-    res.json(entities)
-  })
-
-  app.listen(3000)
-}
-
-startServer()
-```
-
-**Benefits:**
-- ✅ Simple to implement
-- ✅ Thread-safe (async initialization)
-- ✅ Shared cache and indexes
-- ✅ 40x memory reduction
-
----
-
-### Pattern 2: Service Class (Production-Grade)
-
-```typescript
-// services/BrainService.ts
-export class BrainService {
-  private brain: Brainy | null = null
-  private initPromise: Promise<Brainy> | null = null
-
-  async getInstance(): Promise<Brainy> {
-    if (this.brain) return this.brain
-    if (this.initPromise) return this.initPromise
-
-    this.initPromise = this.initialize()
-    return this.initPromise
-  }
-
-  private async initialize(): Promise<Brainy> {
-    this.brain = new Brainy({
-      storage: {
-        path: process.env.BRAINY_DATA_PATH || './brainy-data'
-      },
-      cache: { maxSize: 1000, ttl: 3600000 }
-    })
-    await this.brain.init()
-    return this.brain
-  }
-
-  async shutdown(): Promise<void> {
-    if (this.brain) {
-      // Cleanup if needed
-      this.brain = null
-    }
-  }
-}
-
-// server.ts
-const brainService = new BrainService()
-
-app.get('/api/entities', async (req, res) => {
-  const brain = await brainService.getInstance()
-  const entities = await brain.find(req.query)
-  res.json(entities)
-})
-
-// Graceful shutdown
-process.on('SIGTERM', async () => {
-  await brainService.shutdown()
-  process.exit(0)
-})
-```
-
-**Benefits:**
-- ✅ Prevents race conditions (multiple simultaneous inits)
-- ✅ Testable (can inject mock)
-- ✅ Clean shutdown handling
-- ✅ Environment-configurable
-
----
-
-### Pattern 3: Bun Server (Recommended)
-
-```typescript
-// server.ts - Clean Bun implementation
-import { Brainy } from '@soulcraft/brainy'
-
-let brain: Brainy | null = null
-
-async function getBrain(): Promise<Brainy> {
-  if (!brain) {
-    brain = new Brainy({ storage: { path: './brainy-data' } })
-    await brain.init()
-  }
-  return brain
-}
-
-// Initialize before server starts
-await getBrain()
-
-Bun.serve({
-  port: 3000,
-  async fetch(req) {
-    const url = new URL(req.url)
-
-    if (url.pathname === '/api/entities') {
-      const b = await getBrain()
-      const entities = await b.find({})
-      return Response.json(entities)
-    }
-
-    if (url.pathname === '/api/entity' && req.method === 'POST') {
-      const b = await getBrain()
-      const body = await req.json()
-      const id = await b.add(body)
-      return Response.json({ id })
-    }
-
-    return new Response('Not Found', { status: 404 })
-  }
-})
-
-console.log('Server running on http://localhost:3000')
-```
-
-**Benefits:**
-- ✅ Native Bun runtime performance
-- ✅ No framework dependencies
-- ✅ Pure WASM — no native binaries, bundler-friendly
-- ✅ Built-in TypeScript support
-
-### Pattern 4: Express/Node.js Middleware (Legacy)
-
-```typescript
-// middleware/brainy.ts
-let brainInstance: Brainy | null = null
-
-export async function initBrainy() {
-  if (!brainInstance) {
-    brainInstance = new Brainy({ storage: { path: './brainy-data' } })
-    await brainInstance.init()
-  }
-}
-
-export function brainMiddleware(req, res, next) {
-  if (!brainInstance) {
-    return res.status(500).json({ error: 'Brainy not initialized' })
-  }
-  req.brain = brainInstance  // Attach to request
-  next()
-}
-
-// Type extension
-declare global {
-  namespace Express {
-    interface Request {
-      brain: Brainy
-    }
-  }
-}
-
-// server.ts
-import { initBrainy, brainMiddleware } from './middleware/brainy'
-
-async function startServer() {
-  await initBrainy()  // Initialize first
-
-  app.use('/api', brainMiddleware)  // Apply to API routes
-
-  app.get('/api/entities', async (req, res) => {
-    const entities = await req.brain.find(req.query)  // Type-safe!
-    res.json(entities)
-  })
-
-  app.listen(3000)
-}
-```
-
-**Benefits:**
-- ✅ Clean separation of concerns
-- ✅ Type-safe (`req.brain` is typed)
-- ✅ Easy to add auth/validation
-
----
-
-## Optimization Strategies
-
-### 1. Configure Cache for Your Workload
-
-```typescript
-const brain = new Brainy({
-  cache: {
-    maxSize: 1000,           // Number of entities to cache
-    ttl: 3600000,            // Cache lifetime (1 hour)
-    enableMetrics: true,     // Track hit rate
-    evictionPolicy: 'lru'    // Least recently used
-  }
-})
-```
-
-**Cache sizing:**
-- Small service (< 100 req/min): `maxSize: 500`
-- Medium service (< 1000 req/min): `maxSize: 1000`
-- Large service (> 1000 req/min): `maxSize: 5000`
-
-### 2. Lazy Load Augmentations
-
-```typescript
-const brain = new Brainy({
-  augmentations: {
-    // Only load what you actually use
-    include: ['cache', 'metrics', 'display', 'vfs'],
-    exclude: ['neuralImport', 'intelligentImport']  // Skip heavy features
-  }
-})
-```
-
-**Memory savings:**
-- With all augmentations: ~800MB
-- With minimal set: ~400MB
-
-### 3. Warm Up Indexes
-
-```typescript
-async function startServer() {
-  const brain = await getBrain()
-
-  // Pre-warm frequently-used indexes
-  await brain.find({ type: 'person', limit: 1 })
-  await brain.find({ type: 'organization', limit: 1 })
-
-  console.log('✅ Indexes pre-warmed')
-
-  app.listen(3000)
-}
-```
-
-**Benefit:** First requests are fast (no cold-start index building)
-
-### 4. Memory-Aware Configuration
-
-```typescript
-import os from 'os'
-
-const totalMemory = os.totalmem()
-const availableMemory = os.freemem()
-
-const brain = new Brainy({
-  cache: {
-    // Use 10% of total RAM for cache
-    maxSize: Math.floor(totalMemory * 0.1 / (1024 * 1024))
-  },
-  indexes: {
-    // Lazy load indexes if low memory
-    lazyLoad: availableMemory < totalMemory * 0.5,
-    preload: ['person', 'organization']  // Only preload common types
-  }
-})
-```
-
----
-
-## Concurrency & Thread Safety
-
-Brainy is **designed** for concurrent access. A single instance can handle:
-
-```typescript
-// Multiple concurrent requests - all using same instance
-app.get('/api/read/:id', async (req, res) => {
-  const brain = getBrain()
-  const entity = await brain.get(req.params.id)  // Safe - no state mutation
-  res.json(entity)
-})
-
-app.post('/api/write', async (req, res) => {
-  const brain = getBrain()
-  const id = await brain.add(req.body)  // Safe - internal locking
-  res.json({ id })
-})
-```
-
-**Concurrency mechanisms:**
-- ✅ **Read operations**: Lock-free (MVCC)
-- ✅ **Write operations**: Internal write-ahead logging (WAL)
-- ✅ **Cache**: Thread-safe LRU implementation
-- ✅ **Indexes**: Concurrent reads, locked writes
-
----
-
-## Production Checklist
-
-### Before Deploying
-
-- [ ] **Initialize Brainy on startup** (not per-request)
-- [ ] **Configure cache size** based on memory
-- [ ] **Only load needed augmentations**
-- [ ] **Warm up critical indexes**
-- [ ] **Add graceful shutdown handler**
-- [ ] **Monitor cache hit rate**
-
-### Code Review Checklist
-
-```typescript
-// ❌ BAD - Instance per request
-app.get('/api/route', async (req, res) => {
-  const brain = new Brainy(...)  // RED FLAG!
-  await brain.init()             // RED FLAG!
-})
-
-// ✅ GOOD - Singleton pattern
-app.get('/api/route', async (req, res) => {
-  const brain = await getBrain()  // Reuses instance ✓
-})
-```
-
----
-
-## Monitoring & Metrics
-
-```typescript
-// Add metrics endpoint
-app.get('/api/metrics', (req, res) => {
-  const brain = getBrain()
-
-  res.json({
-    cache: {
-      size: brain.cache?.size || 0,
-      maxSize: brain.cache?.maxSize || 0,
-      hitRate: brain.metrics?.cacheHitRate || 0  // Target: >70%
-    },
-    storage: brain.storage.getStats(),
-    memory: {
-      heapUsed: Math.round(process.memoryUsage().heapUsed / 1024 / 1024),
-      heapTotal: Math.round(process.memoryUsage().heapTotal / 1024 / 1024)
-    }
-  })
-})
-```
-
-**Key metrics to track:**
-- **Cache hit rate**: Should be >70% after warm-up
-- **Memory usage**: Should stay constant (~500MB for singleton)
-- **Request latency**: Should be <10ms for cached entities
-
----
-
-## Common Pitfalls
-
-### 1. Creating instances in routes
-```typescript
-// ❌ NEVER do this
-app.get('/api/entities', async (req, res) => {
-  const brain = new Brainy(...)  // Creates new instance every time!
-})
-```
-
-### 2. Not awaiting initialization
-```typescript
-// ❌ Race condition - server starts before Brainy ready
-app.listen(3000)
-getBrain()  // Async init happens AFTER server starts!
-
-// ✅ Correct - wait for init
-await getBrain()
-app.listen(3000)
-```
-
-### 3. Multiple instances for different purposes
-```typescript
-// ❌ Wasteful - creates 2 instances
-const readBrain = new Brainy(...)
-const writeBrain = new Brainy(...)
-
-// ✅ One instance handles both
-const brain = new Brainy(...)
-await brain.get(id)    // Read
-await brain.add(data)  // Write
-```
-
----
-
-## Migration Guide
-
-### Current (Anti-Pattern)
-```typescript
-// Probably in multiple route files
-async function handler(req, res) {
-  const brain = new Brainy({ storage: { path: './brainy-data' } })
-  await brain.init()
-  // ... use brain
-}
-```
-
-### Step 1: Create Singleton Module
-```typescript
-// lib/brainy.ts
-let instance: Brainy | null = null
-
-export async function getBrain(): Promise<Brainy> {
-  if (!instance) {
-    instance = new Brainy({ storage: { path: './brainy-data' } })
-    await instance.init()
-  }
-  return instance
-}
-```
-
-### Step 2: Update Server Startup
-```typescript
-// server.ts
-import { getBrain } from './lib/brainy'
-
-async function startServer() {
-  // Initialize Brainy FIRST
-  await getBrain()
-  console.log('✅ Brainy initialized')
-
-  // THEN start server
-  app.listen(3000)
-}
-```
-
-### Step 3: Update All Routes
-```typescript
-// Before
-async function handler(req, res) {
-  const brain = new Brainy(...)  // Remove this
-  await brain.init()              // Remove this
-
-  // ... rest of code
-}
-
-// After
-import { getBrain } from './lib/brainy'
-
-async function handler(req, res) {
-  const brain = await getBrain()  // Add this
-
-  // ... rest of code stays same
-}
-```
-
-**Expected results:**
-- ✅ 40x memory reduction (20GB → 500MB)
-- ✅ 30x faster requests (60ms → 2ms average)
-- ✅ 80%+ cache hit rate
-- ✅ Your service can scale to 1000s of requests/minute
-
----
-
-## Summary
-
-**DO:**
-- ✅ Initialize Brainy ONCE on server startup
-- ✅ Share single instance across all requests
-- ✅ Configure cache for your workload
-- ✅ Monitor cache hit rate
-- ✅ Handle graceful shutdown
-
-**DON'T:**
-- ❌ Create new Brainy instance per request
-- ❌ Create multiple instances
-- ❌ Start server before Brainy is initialized
-- ❌ Load augmentations you don't use
-
-**Result:** 40x less memory, 30x faster requests, Brainy optimizations actually work!
-
----
-
-**Questions? Issues?**
-- Report issues: https://github.com/soulcraftlabs/brainy/issues

docs/QUERY_OPERATORS.md

@@ -1,298 +0,0 @@
-# Query Operators (BFO)
-
-> Brainy Field Operators — the complete reference for `where` filters in `find()`.
-
-All operators work with `find({ where: { ... } })` and filter on **metadata fields** (not `data`).
-
----
-
-## Equality
-
-| Operator | Alias | Description | Example |
-|----------|-------|-------------|---------|
-| `eq` | `equals` | Exact match | `{ status: { eq: 'active' } }` |
-| `ne` | `notEquals` | Not equal | `{ status: { ne: 'deleted' } }` |
-
-**Shorthand:** A bare value is treated as `equals`:
-
-```typescript
-// These are equivalent:
-brain.find({ where: { status: 'active' } })
-brain.find({ where: { status: { equals: 'active' } } })
-```
-
----
-
-## Comparison
-
-| Operator | Alias | Description | Example |
-|----------|-------|-------------|---------|
-| `gt` | `greaterThan` | Greater than | `{ age: { gt: 18 } }` |
-| `gte` | `greaterThanOrEqual` | Greater or equal | `{ score: { gte: 90 } }` |
-| `lt` | `lessThan` | Less than | `{ price: { lt: 100 } }` |
-| `lte` | `lessThanOrEqual` | Less or equal | `{ rating: { lte: 3 } }` |
-| `between` | — | Inclusive range `[min, max]` | `{ year: { between: [2020, 2025] } }` |
-
-```typescript
-// Range query
-const recent = await brain.find({
-  where: {
-    createdAt: { between: [Date.now() - 86400000, Date.now()] }
-  }
-})
-```
-
----
-
-## Array / Set
-
-| Operator | Alias | Description | Example |
-|----------|-------|-------------|---------|
-| `oneOf` | `in` | Value is one of the given options | `{ color: { oneOf: ['red', 'blue'] } }` |
-| `noneOf` | — | Value is NOT one of the given options | `{ status: { noneOf: ['deleted', 'archived'] } }` |
-| `contains` | — | Array field contains value | `{ tags: { contains: 'ai' } }` |
-| `excludes` | — | Array field does NOT contain value | `{ tags: { excludes: 'spam' } }` |
-| `hasAll` | — | Array field contains ALL listed values | `{ skills: { hasAll: ['js', 'ts'] } }` |
-
-```typescript
-// Find entities tagged with 'ai'
-const aiEntities = await brain.find({
-  where: { tags: { contains: 'ai' } }
-})
-
-// Find entities of specific types
-const people = await brain.find({
-  where: { noun: { oneOf: ['Person', 'Agent'] } }
-})
-```
-
----
-
-## Existence
-
-| Operator | Description | Example |
-|----------|-------------|---------|
-| `exists: true` | Field exists (has any value) | `{ email: { exists: true } }` |
-| `exists: false` | Field does NOT exist | `{ email: { exists: false } }` |
-| `missing: true` | Field does NOT exist (alias for `exists: false`) | `{ email: { missing: true } }` |
-| `missing: false` | Field exists (alias for `exists: true`) | `{ email: { missing: false } }` |
-
-```typescript
-// Find entities that have an email field
-const withEmail = await brain.find({
-  where: { email: { exists: true } }
-})
-```
-
----
-
-## Pattern (In-Memory Only)
-
-These operators work via the in-memory filter path. They are applied **after** the indexed query, so use them with other indexed operators for best performance.
-
-| Operator | Description | Example |
-|----------|-------------|---------|
-| `matches` | Regex or string pattern match | `{ name: { matches: /^Dr\./ } }` |
-| `startsWith` | String prefix | `{ name: { startsWith: 'John' } }` |
-| `endsWith` | String suffix | `{ email: { endsWith: '@gmail.com' } }` |
-
-```typescript
-const doctors = await brain.find({
-  where: {
-    type: NounType.Person,           // Indexed — fast
-    name: { startsWith: 'Dr.' }     // In-memory — applied after
-  }
-})
-```
-
----
-
-## Logical
-
-Combine multiple conditions:
-
-| Operator | Description | Example |
-|----------|-------------|---------|
-| `allOf` | ALL sub-filters must match (AND) | `{ allOf: [{ status: 'active' }, { role: 'admin' }] }` |
-| `anyOf` | ANY sub-filter must match (OR) | `{ anyOf: [{ role: 'admin' }, { role: 'owner' }] }` |
-| `not` | Invert a filter | `{ not: { status: 'deleted' } }` |
-
-```typescript
-// Complex OR query
-const adminsOrOwners = await brain.find({
-  where: {
-    anyOf: [
-      { role: 'admin' },
-      { role: 'owner' }
-    ]
-  }
-})
-
-// NOT query
-const notDeleted = await brain.find({
-  where: {
-    not: { status: 'deleted' }
-  }
-})
-
-// Combined AND + OR
-const results = await brain.find({
-  where: {
-    allOf: [
-      { department: 'engineering' },
-      { anyOf: [
-        { level: 'senior' },
-        { yearsExperience: { greaterThan: 5 } }
-      ]}
-    ]
-  }
-})
-```
-
----
-
-## Indexed vs In-Memory Operators
-
-Brainy's MetadataIndex supports a subset of operators natively for O(1) field lookups. Other operators fall back to in-memory filtering.
-
-| Operator | MetadataIndex (Indexed) | In-Memory Fallback |
-|----------|:-----------------------:|:------------------:|
-| `equals` / `eq` | Yes | Yes |
-| `notEquals` / `ne` | — | Yes |
-| `greaterThan` / `gt` | Yes | Yes |
-| `greaterThanOrEqual` / `gte` | Yes | Yes |
-| `lessThan` / `lt` | Yes | Yes |
-| `lessThanOrEqual` / `lte` | Yes | Yes |
-| `between` | Yes | Yes |
-| `oneOf` / `in` | Yes | Yes |
-| `noneOf` | — | Yes |
-| `contains` | Yes | Yes |
-| `exists` / `missing` | Yes | Yes |
-| `matches` | — | Yes |
-| `startsWith` | — | Yes |
-| `endsWith` | — | Yes |
-| `allOf` | Partial | Yes |
-| `anyOf` | Partial | Yes |
-| `not` | — | Yes |
-
-**Performance tip:** Combine indexed operators (equals, greaterThan, oneOf, between, contains, exists) with pattern operators for optimal speed — the index narrows results first, then patterns filter in memory.
-
----
-
-## Practical Examples
-
-### Filter by entity type
-
-```typescript
-// Using the type shorthand (recommended)
-brain.find({ type: NounType.Person })
-
-// Using where.noun directly
-brain.find({ where: { noun: NounType.Person } })
-
-// Multiple types
-brain.find({ type: [NounType.Person, NounType.Agent] })
-```
-
-### Filter by subtype
-
-`subtype` is a top-level standard field — takes the column-store fast path, not the metadata fallback. Pair with `type` for the typical "Person who is an employee" query:
-
-```typescript
-// Equality on subtype:
-brain.find({ type: NounType.Person, subtype: 'employee' })
-
-// Set membership:
-brain.find({ type: NounType.Person, subtype: ['employee', 'contractor'] })
-
-// Operator-form predicates use `where`:
-brain.find({
-  type: NounType.Person,
-  where: { subtype: { exists: true } }
-})
-```
-
-See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the full surface.
-
-### Filter relationships by subtype (7.30+)
-
-Verbs are first-class peers — `related()` and graph traversal both honor subtype filters on the fast path:
-
-```typescript
-// Filter relationships by VerbType subtype
-const direct = await brain.related({
-  from: ceoId,
-  type: VerbType.ReportsTo,
-  subtype: 'direct'
-})
-
-// Set membership on verb subtype
-const all = await brain.related({
-  from: ceoId,
-  type: VerbType.ReportsTo,
-  subtype: ['direct', 'dotted-line']
-})
-
-// Graph traversal — subtype filters traversal edges (depth-1 in 7.30 JS path;
-// multi-hop subtype filtering lands on Cor native)
-const reports = await brain.find({
-  connected: {
-    from: ceoId,
-    via: VerbType.ReportsTo,
-    subtype: 'direct',
-    depth: 1
-  }
-})
-```
-
-### Combine semantic search with filters
-
-```typescript
-const results = await brain.find({
-  query: 'machine learning engineer',     // Semantic search (on data)
-  type: NounType.Person,                  // Type filter (indexed)
-  where: {
-    department: 'engineering',            // Exact match (indexed)
-    yearsExperience: { greaterThan: 3 }   // Range filter (indexed)
-  },
-  limit: 10
-})
-```
-
-### Temporal queries
-
-```typescript
-const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000
-const recentEntities = await brain.find({
-  where: {
-    createdAt: { greaterThan: lastWeek }
-  },
-  orderBy: 'createdAt',
-  order: 'desc',
-  limit: 50
-})
-```
-
-### Graph + metadata combination
-
-```typescript
-const results = await brain.find({
-  connected: {
-    from: teamLeadId,
-    via: VerbType.WorksWith,
-    depth: 2
-  },
-  where: {
-    role: { oneOf: ['engineer', 'designer'] },
-    active: true
-  }
-})
-```
-
----
-
-## See Also
-
-- [Data Model](./DATA_MODEL.md) — Entity structure, data vs metadata
-- [API Reference](./api/README.md) — Complete API documentation
-- [Find System](./FIND_SYSTEM.md) — Natural language find() details

docs/QUICK-START.md

@@ -0,0 +1,392 @@
+# 🚀 Brainy Quick Start Guide
+
+Get up and running with Brainy in 5 minutes!
+
+## Installation
+
+```bash
+npm install @soulcraft/brainy
+```
+
+Or install globally for CLI access:
+```bash
+npm install -g brainy
+```
+
+## Basic Usage
+
+### 1. Initialize Brainy
+
+```javascript
+import { Brainy, NounType } from '@soulcraft/brainy'
+
+const brain = new Brainy()
+await brain.init()
+```
+
+That's it! No configuration needed. Brainy automatically:
+- Downloads embedding models (first time only)
+- Sets up storage (in-memory by default)
+- Initializes all augmentations
+- Configures optimal settings
+
+### 2. Add Your First Data
+
+```javascript
+// Add a simple string
+await brain.add("JavaScript is a versatile programming language", { nounType: NounType.Concept })
+
+// Add with metadata
+await brain.add("React is a JavaScript library", {
+  nounType: NounType.Concept,
+  type: "library",
+  category: "frontend",
+  popularity: "high"
+})
+
+// Add structured data
+await brain.add({
+  title: "Introduction to TypeScript",
+  content: "TypeScript adds static typing to JavaScript",
+  author: "John Doe"
+}, {
+  nounType: NounType.Document,
+  type: "article",
+  date: "2024-01-15"
+})
+```
+
+### 3. Search Your Data
+
+```javascript
+// Simple vector search
+const results = await brain.search("programming languages")
+
+// Natural language query
+const articles = await brain.find("recent articles about TypeScript")
+
+// With metadata filtering
+const libraries = await brain.search("JavaScript", {
+  metadata: { type: "library" },
+  limit: 5
+})
+```
+
+## Real-World Examples
+
+### Example 1: Document Search System
+
+```javascript
+import { Brainy, NounType } from '@soulcraft/brainy'
+import fs from 'fs'
+
+const brain = new Brainy({
+  storage: {
+    type: 'filesystem',
+    path: './document-index'
+  }
+})
+await brain.init()
+
+// Index documents
+const documents = [
+  { file: 'api-guide.md', content: fs.readFileSync('./docs/api-guide.md', 'utf8') },
+  { file: 'tutorial.md', content: fs.readFileSync('./docs/tutorial.md', 'utf8') },
+  { file: 'faq.md', content: fs.readFileSync('./docs/faq.md', 'utf8') }
+]
+
+for (const doc of documents) {
+  await brain.add(doc.content, {
+    nounType: NounType.Document,
+    filename: doc.file,
+    type: 'documentation',
+    indexed: new Date().toISOString()
+  })
+}
+
+// Search documents
+const results = await brain.find("how to authenticate users")
+console.log(`Found ${results.length} relevant documents:`)
+results.forEach(r => console.log(`- ${r.metadata.filename} (${(r.score * 100).toFixed(1)}% match)`))
+```
+
+### Example 2: AI Chat with Memory
+
+```javascript
+import { Brainy, NounType } from '@soulcraft/brainy'
+
+const brain = new Brainy()
+await brain.init()
+
+class ChatWithMemory {
+  constructor(brain) {
+    this.brain = brain
+    this.sessionId = Date.now().toString()
+  }
+  
+  async addMessage(role, content) {
+    await this.brain.add(content, {
+      nounType: NounType.Message,
+      role,
+      sessionId: this.sessionId,
+      timestamp: Date.now()
+    })
+  }
+  
+  async getContext(query, limit = 5) {
+    // Find relevant previous messages
+    const relevant = await this.brain.find(query, { limit })
+    return relevant.map(r => ({
+      role: r.metadata.role,
+      content: r.content
+    }))
+  }
+  
+  async chat(userMessage) {
+    // Store user message
+    await this.addMessage('user', userMessage)
+    
+    // Get relevant context
+    const context = await this.getContext(userMessage)
+    
+    // Your AI logic here (OpenAI, Anthropic, etc.)
+    const aiResponse = await callYourAI(userMessage, context)
+    
+    // Store AI response
+    await this.addMessage('assistant', aiResponse)
+    
+    return aiResponse
+  }
+}
+
+const chat = new ChatWithMemory(brain)
+const response = await chat.chat("What did we discuss about JavaScript?")
+```
+
+### Example 3: Semantic Code Search
+
+```javascript
+import { Brainy, NounType } from '@soulcraft/brainy'
+import { glob } from 'glob'
+import fs from 'fs'
+
+const brain = new Brainy()
+await brain.init()
+
+// Index all JavaScript files
+const files = await glob('src/**/*.js')
+for (const file of files) {
+  const content = fs.readFileSync(file, 'utf8')
+  
+  // Extract functions
+  const functions = content.match(/function\s+(\w+)|const\s+(\w+)\s*=/g) || []
+  
+  await brain.add(content, {
+    nounType: NounType.File,
+    file,
+    type: 'code',
+    language: 'javascript',
+    functions: functions.map(f => f.replace(/function\s+|const\s+|=/g, '').trim())
+  })
+}
+
+// Search for code
+const results = await brain.find("authentication middleware")
+console.log('Relevant code files:')
+results.forEach(r => {
+  console.log(`\n${r.metadata.file}:`)
+  console.log(`  Functions: ${r.metadata.functions.join(', ')}`)
+  console.log(`  Relevance: ${(r.score * 100).toFixed(1)}%`)
+})
+```
+
+## CLI Quick Examples
+
+```bash
+# Add data from CLI
+brainy add "React is a JavaScript library for building UIs"
+
+# Search
+brainy search "JavaScript frameworks"
+
+# Natural language find
+brainy find "popular frontend libraries"
+
+# Interactive chat mode
+brainy chat
+
+# Import JSON data
+brainy import data.json
+
+# Export your brain
+brainy export --format json > backup.json
+
+# Check status
+brainy status
+```
+
+## Advanced Features
+
+### Triple Intelligence Query
+
+```javascript
+// Combine vector search + metadata filters + graph relationships
+const results = await brain.find({
+  like: "React",                    // Vector similarity
+  where: {                          // Metadata filtering
+    type: "library",
+    popularity: "high",
+    year: { greaterThan: 2015 }
+  },
+  related: {                        // Graph relationships
+    to: "JavaScript",
+    depth: 2
+  }
+}, {
+  limit: 10,
+  includeContent: true
+})
+```
+
+### Pagination
+
+```javascript
+// Cursor-based pagination for large result sets
+let cursor = null
+do {
+  const results = await brain.search("programming", {
+    limit: 100,
+    cursor
+  })
+  
+  // Process batch
+  results.forEach(processResult)
+  
+  cursor = results.nextCursor
+} while (cursor)
+```
+
+### Performance Optimization
+
+```javascript
+// Pre-filter with metadata for faster searches
+const results = await brain.search("*", {
+  metadata: {
+    type: "article",
+    category: "tech",
+    date: { greaterThan: "2024-01-01" }
+  },
+  limit: 1000
+})
+```
+
+## Storage Options
+
+### Memory (Testing)
+```javascript
+const brain = new Brainy()  // Default
+```
+
+### FileSystem (Development)
+```javascript
+const brain = new Brainy({
+  storage: {
+    type: 'filesystem',
+    path: './brain-data'
+  }
+})
+```
+
+### Browser (OPFS)
+```javascript
+const brain = new Brainy({
+  storage: { type: 'opfs' }
+})
+```
+
+### S3 (Production)
+```javascript
+const brain = new Brainy({
+  storage: {
+    type: 's3',
+    bucket: 'my-brain-bucket',
+    region: 'us-east-1',
+    credentials: {
+      accessKeyId: process.env.AWS_ACCESS_KEY,
+      secretAccessKey: process.env.AWS_SECRET_KEY
+    }
+  }
+})
+```
+
+## Tips & Best Practices
+
+1. **Use metadata liberally** - It enables O(log n) filtering
+2. **Batch operations when possible** - Use `import()` for bulk data
+3. **Enable caching for production** - Automatic with default settings
+4. **Use cursor pagination** - For large result sets
+5. **Leverage natural language** - `find()` understands context
+
+## Common Patterns
+
+### Similarity Search
+```javascript
+// Find similar items to an existing one
+const item = await brain.getNoun(id)
+const similar = await brain.search(item.content, { limit: 5 })
+```
+
+### Time-based Queries
+```javascript
+// Recent items
+const recent = await brain.search("*", {
+  metadata: { 
+    timestamp: { greaterThan: Date.now() - 86400000 } // Last 24 hours
+  }
+})
+```
+
+### Category Browsing
+```javascript
+// Get all items in a category
+const category = await brain.search("*", {
+  metadata: { category: "tutorials" },
+  limit: 100
+})
+```
+
+## Troubleshooting
+
+### Models not loading?
+```bash
+# Clear cache and re-download
+rm -rf ~/.cache/brainy
+npm run download-models
+```
+
+### Slow initialization?
+- First run downloads models (~25MB)
+- Subsequent runs use cache (< 500ms)
+- Use `storage: { type: 'memory' }` for testing
+
+### Out of memory?
+- Use filesystem or S3 storage for large datasets
+- Enable worker threads (automatic in Node.js)
+- Increase Node memory: `NODE_OPTIONS='--max-old-space-size=4096'`
+
+## Next Steps
+
+- 📖 Read the [full documentation](../README.md)
+- 🏗️ Learn about [augmentations](augmentations/README.md)
+- 🧠 Understand [Triple Intelligence](architecture/triple-intelligence.md)
+- ☁️ Explore [Brain Cloud](https://soulcraft.com)
+
+## Get Help
+
+- GitHub Issues: [github.com/brainy-org/brainy](https://github.com/brainy-org/brainy)
+- Documentation: [Full Docs](../README.md)
+- Examples: [/examples](../../examples)
+
+---
+
+**Ready to build something amazing? You're all set! 🚀**

docs/README.md

@@ -1,130 +1,126 @@
 # Brainy Documentation
 
-> The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API.
+Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
 
-## Quick Start
+## 📊 Implementation Status
+
+- ✅ **Production Ready**: Core features working today
+- 🚧 **In Development**: Features coming soon
+- 📅 **Roadmap**: See [ROADMAP.md](../ROADMAP.md)
+
+## Quick Links
+
+### Getting Started
+- [Quick Start Guide](./guides/getting-started.md) - Get up and running in minutes
+- [Enterprise for Everyone](./guides/enterprise-for-everyone.md) - **No limits, no tiers, everything free**
+- [Natural Language Queries](./guides/natural-language.md) - Query with plain English
+
+### Core Concepts
+- [Zero Configuration](./architecture/zero-config.md) - **Auto-adapts to any environment**
+- [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) - **Revolutionary data model**
+- [Triple Intelligence](./architecture/triple-intelligence.md) - Unified query system
+- [Architecture Overview](./architecture/overview.md) - System design
+
+### API Documentation
+- [API Reference](./api/README.md) - Complete API documentation
+- [TypeScript Types](./api/types.md) - Type definitions
+
+### Advanced Topics
+- [Augmentations System](./architecture/augmentations.md) - **Enterprise plugins & neural import**
+- [Storage Architecture](./architecture/storage.md) - Storage adapter system
+- [Performance Tuning](./guides/performance.md) - Optimization guide
+- [Migration Guide](../MIGRATION.md) - Upgrading from 1.x
+
+## What is Brainy?
+
+Brainy is a next-generation AI database that combines:
+- **Vector Search**: Semantic similarity using HNSW indexing
+- **Graph Relationships**: Complex relationship mapping and traversal
+- **Field Filtering**: Precise metadata filtering with O(1) lookups
+- **Natural Language**: Query in plain English
+
+## Key Features
+
+### 🧠 Triple Intelligence Engine
+All three intelligence types (vector, graph, field) work together in every query for optimal results.
+
+### 📝 Noun-Verb Taxonomy
+Model your data naturally as entities (nouns) and relationships (verbs) - no complex schemas needed.
+
+### 🌍 Natural Language Queries
+Ask questions in plain English and Brainy understands your intent:
+```typescript
+await brain.find("recent articles about AI with high ratings")
+```
+
+### ⚡ Production Ready
+- Universal storage (FileSystem, S3, OPFS, Memory)
+- Zero configuration with intelligent defaults
+- Full TypeScript support
+- Cross-platform compatibility
+
+## Quick Example
 
 ```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
+import { Brainy } from 'brainy'
 
+// Initialize
 const brain = new Brainy()
 await brain.init()
 
-// Add entities — data is embedded for semantic search, metadata is indexed for filtering
-const id = await brain.add({
-  data: 'Revolutionary AI Breakthrough',
-  type: NounType.Document,
-  metadata: { category: 'technology', rating: 4.8 }
+// Add entities (nouns)
+const articleId = await brain.add("Revolutionary AI Breakthrough", {
+  type: "article",
+  category: "technology",
+  rating: 4.8
 })
 
-// Search with Triple Intelligence
-const results = await brain.find({
-  query: 'artificial intelligence',              // Semantic search (on data)
-  where: { rating: { greaterThan: 4.0 } },       // Metadata filter
-  connected: { from: authorId, depth: 2 }         // Graph traversal
+const authorId = await brain.add("Dr. Sarah Chen", {
+  type: "person",
+  role: "researcher"
 })
+
+// Create relationships (verbs)
+await brain.relate(authorId, articleId, "authored", {
+  date: "2024-01-15",
+  contribution: "primary"
+})
+
+// Query naturally
+const results = await brain.find("highly rated technology articles by researchers")
 ```
 
----
+## Documentation Structure
 
-## Core Documentation
+```
+docs/
+├── README.md                    # This file
+├── guides/                      # User guides
+│   ├── getting-started.md      # Quick start guide
+│   ├── natural-language.md     # NLP query guide
+│   └── performance.md          # Performance tuning
+├── architecture/               # Technical architecture
+│   ├── overview.md            # System overview
+│   ├── noun-verb-taxonomy.md  # Data model
+│   ├── triple-intelligence.md # Query system
+│   └── storage.md             # Storage layer
+├── vfs/                       # Virtual Filesystem
+│   ├── README.md              # VFS overview
+│   ├── SEMANTIC_VFS.md        # Semantic projections
+│   ├── VFS_API_GUIDE.md       # Complete API reference
+│   └── QUICK_START.md         # 5-minute setup
+└── api/                       # API documentation
+    ├── README.md              # API overview
+    ├── brainy-data.md        # Main class
+    └── types.md              # TypeScript types
+```
 
-| Document | Description |
-|----------|-------------|
-| **[API Reference](./api/README.md)** | Complete API documentation — **start here** |
-| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
-| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
-| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
-| [Consistency Model](./concepts/consistency-model.md) | The Db API guarantees — snapshot isolation, atomic transactions, time travel |
+## Community
 
----
-
-## Architecture
-
-| Document | Description |
-|----------|-------------|
-| [Architecture Overview](./architecture/overview.md) | High-level system design |
-| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Metadata unified query |
-| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | 42 nouns + 127 verbs type system |
-| [Stage 3 Canonical Taxonomy](./STAGE3-CANONICAL-TAXONOMY.md) | Complete type reference |
-| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
-| [Index Architecture](./architecture/index-architecture.md) | Vector, Graph, and Metadata indexing |
-| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
-
----
-
-## Virtual Filesystem (VFS)
-
-| Document | Description |
-|----------|-------------|
-| [VFS Quick Start](./vfs/QUICK_START.md) | Get started in 30 seconds |
-| [VFS Core](./vfs/VFS_CORE.md) | Core concepts and architecture |
-| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
-| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
-
-See [vfs/](./vfs/) for the complete VFS documentation set.
-
----
-
-## Guides
-
-| Document | Description |
-|----------|-------------|
-| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
-| [Snapshots & Time Travel](./guides/snapshots-and-time-travel.md) | Backups, restore, what-if analysis, audit trails |
-| [Natural Language](./guides/natural-language.md) | Query in plain English |
-| [Neural API](./guides/neural-api.md) | AI-powered features |
-| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
-| [Framework Integration](./guides/framework-integration.md) | React, Vue, Angular, Svelte |
-
----
-
-## Storage & Deployment
-
-| Document | Description |
-|----------|-------------|
-| [Storage Architecture](./architecture/storage-architecture.md) | Filesystem and memory adapters, on-disk artifact layout, operator-layer backup |
-| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities |
-
----
-
-## Plugins
-
-| Document | Description |
-|----------|-------------|
-| [Plugins](./PLUGINS.md) | Plugin system overview — providers, `plugins` config, `brain.use()` |
-
----
-
-## Performance & Scaling
-
-| Document | Description |
-|----------|-------------|
-| [Performance](./PERFORMANCE.md) | Optimization techniques |
-| [Scaling](./SCALING.md) | Scale to billions of entities |
-| [Batching](./BATCHING.md) | Batch operations guide |
-
----
-
-## Migration & Reference
-
-| Document | Description |
-|----------|-------------|
-| [v3 to v4 Migration](./MIGRATION-V3-TO-V4.md) | Upgrade guide |
-| [Release Guide](./RELEASE-GUIDE.md) | How to release new versions |
-| [Production Architecture](./PRODUCTION_SERVICE_ARCHITECTURE.md) | Ops reference |
-
----
-
-## Internal
-
-| Document | Description |
-|----------|-------------|
-| [Audit Report](./internal/AUDIT_REPORT.md) | Feature audit |
-| [Honest Status](./internal/HONEST_STATUS.md) | Actual implementation status |
-
----
+- **GitHub**: [github.com/brainy-org/brainy](https://github.com/brainy-org/brainy)
+- **Issues**: [Report bugs or request features](https://github.com/brainy-org/brainy/issues)
+- **Discussions**: [Join the conversation](https://github.com/brainy-org/brainy/discussions)
 
 ## License
 
-Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
+Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.

docs/RELEASE-GUIDE.md

@@ -48,7 +48,7 @@ git commit -m "chore: remove unused dependency"
 # DON'T use BREAKING CHANGE for internal changes
 git commit -m "feat: improve model delivery
 
-BREAKING CHANGE: removed tar-stream dependency"  # WRONG! This triggers
+BREAKING CHANGE: removed tar-stream dependency"  # WRONG! This triggers v3.0.0
 ```
 
 ## Release Workflow Checklist
@@ -119,13 +119,4 @@ gh release create vY.Y.Y --generate-notes
 - **Most releases should be MINOR or PATCH**
 - **Major versions should be RARE**
 - **When in doubt, it's probably MINOR**
-- **NEVER use "BREAKING CHANGE" for internal changes**
-## Hard Ordering Constraints (check before EVERY release)
-
-- **Embedding model changes are SEQUENCED, not free.** No release may change the
-  embedding model (or its quantization/dimensions) before **vector model-version
-  stamping + hard-error-on-mismatch** ships. Stored vectors carry no model version
-  today; mixing vectors from two models silently corrupts every similarity
-  comparison. If a model bump is ever proposed, the stamping work moves ahead of
-  it in the schedule — coordinate with the native provider so both engines stamp
-  and enforce identically. (Registered with the native-provider team 2026-07-07.)
+- **NEVER use "BREAKING CHANGE" for internal changes**

docs/SCALING.md

@@ -1,239 +1,502 @@
-# Brainy Scaling Guide
+# 🚀 Brainy Scaling Guide - Enterprise for Everyone
 
-> **One Line Summary**: Single-node by design — Brainy scales by getting the most out of one machine plus operator-layer backup.
+> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required.
 
-## Table of Contents
+## 📖 Table of Contents
 - [Quick Start](#quick-start)
-- [How Brainy Scales](#how-brainy-scales)
+- [How It Works](#how-it-works)
 - [Storage Configurations](#storage-configurations)
 - [Scaling Patterns](#scaling-patterns)
 - [Real World Examples](#real-world-examples)
 
 ## Quick Start
 
-### In-Memory
+### Single Node (Default)
 ```typescript
 import Brainy from '@soulcraft/brainy'
-const brain = new Brainy({ storage: { type: 'memory' } })
+const brain = new Brainy()  // That's it!
 ```
 
-### On-Disk (Default for Node)
+### Multi-Node (Auto-Discovery)
 ```typescript
+// Node 1
+const brain = new Brainy()  // Starts as primary
+
+// Node 2 (different server)
+const brain = new Brainy()  // Auto-discovers Node 1, becomes replica!
+```
+
+**That's literally all you need!** Brainy handles everything else automatically.
+
+## How It Works
+
+### 🎯 The Magic: Zero Configuration
+
+Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration:
+
+1. **First node starts** → Becomes primary automatically
+2. **Second node starts** → Discovers first node via UDP broadcast
+3. **Nodes negotiate** → Elect leader, distribute shards
+4. **Data flows** → Automatic replication and routing
+5. **Node fails** → Automatic failover in <1 second
+
+### 🔄 Automatic Node Discovery
+
+```typescript
+// Three ways Brainy finds other nodes (auto-selected):
+
+// 1. LOCAL NETWORK (Default)
+// Uses UDP broadcast on port 7946
+// Perfect for: On-premise, same VPC
+
+// 2. CLOUD NATIVE (Auto-detected)
+// Kubernetes: Uses k8s DNS service discovery
+// AWS: Uses EC2 tags or Route53
+// Azure: Uses Azure DNS
+
+// 3. EXPLICIT (When needed)
 const brain = new Brainy({
-  storage: { type: 'filesystem', path: './brainy-data' }
+  peers: ['node1.example.com', 'node2.example.com']
 })
 ```
 
-## How Brainy Scales
+### 📊 Data Distribution
 
-Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery, no S3 coordination. Scaling means:
+When you add data, Brainy automatically:
 
-- **Up**: give the process more RAM, CPU, and IOPS
-- **Out**: stand up multiple independent Brainy instances behind your own service layer
-- **Cold storage**: snapshot the on-disk artifact off-site so you can rehydrate elsewhere
+```typescript
+brain.add({ name: "John" }, 'person')
 
-The three knobs that matter most:
-
-1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
-2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
-
-The native vector provider (via the optional `@soulcraft/cor` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
-
-## Measured Performance
-
-Numbers below are **measured** by `tests/benchmarks/find-composition-scale.js` (a single
-Node 22 process, in-memory storage, 384-dim vectors, `balanced` recall). They are the
-open-core (pure-TypeScript) path — what you get from `@soulcraft/brainy` with no native
-provider installed. Run it yourself: `node --max-old-space-size=8192 tests/benchmarks/find-composition-scale.js 100000`.
-
-`find()` query latency, p50 / p95 (200 queries each):
-
-| Query | 5,000 entities | 100,000 entities |
-|---|---|---|
-| Vector similarity (`{ vector }`) | 0.8 / 1.3 ms | 1.4 / 4.7 ms |
-| Graph 1-hop (`{ connected }`) | 0.5 / 0.7 ms | 0.7 / 0.8 ms |
-| Metadata filter (`{ where }`, low-selectivity) | 0.7 / 1.2 ms | 23.5 / 30.1 ms |
-| Vector + metadata | 7.7 / 8.3 ms | 78.8 / 93.8 ms |
-
-What the shape tells you:
-
-- **Vector and graph lookups scale ~logarithmically** — they barely move from 5k to 100k,
-  because HNSW search is ~O(ef·log n) and graph adjacency is O(degree).
-- **Metadata-filtered paths scale with the size of the match set, not the database.** The
-  benchmark's `category` filter matches ~10% of rows (10,000 at 100k); the cost is
-  materializing that candidate set and running the vector search *inside* it (`find()` does
-  metadata-first hard filtering, then ranks within the candidates — see
-  [How find works](./FIND_SYSTEM.md)). A **high-selectivity** filter (few matches) is far
-  cheaper; a 10%-of-everything filter is the worst case. This candidate-restricted search is
-  precisely the path the native provider accelerates (Rust roaring-bitmap candidate
-  intersection).
-- **Composition is correct, not lossy.** Combining vector + metadata + graph returns exactly
-  the entities satisfying all constraints — verified by
-  `tests/integration/find-triple-composition.test.ts`.
-
-Memory: ~62 KB resident per entity at 100k (6.2 GB RSS for 100k × 384-dim including the HNSW
-graph, metadata index, and 100k edges).
-
-**Scale ceiling (open-core).** The pure-JS HNSW *build* cost (~100 inserts/s at 384-dim on
-one core) makes the in-process open-core path most appropriate up to ~10⁵–10⁶ entities.
-*Query* latency stays low well beyond that, but for the 10⁸–10¹⁰ regime install the native
-provider (`@soulcraft/cor`, on-disk DiskANN) — same API, no code change. _Projected from
-the two measured points, vector p50 at 1M is ~2 ms; metadata-heavy composition grows with
-match-set size and is the path to move onto the native provider first._
+// Behind the scenes:
+// 1. Hash ID to determine shard (consistent hashing)
+// 2. Find nodes responsible for this shard
+// 3. Write to primary shard owner
+// 4. Replicate to N backup nodes (default: 2)
+// 5. Confirm write when majority acknowledge
+```
 
 ## Storage Configurations
 
-### Filesystem (Recommended for Production)
+### 🗂️ Storage Adapter Patterns
+
+Brainy intelligently adapts to your storage setup:
+
+#### Pattern 1: Separate Storage Per Node (Recommended)
+
 ```typescript
+// Node 1 - Own filesystem
+const brain1 = new Brainy({
+  storage: '/data/node1'  // or auto: './brainy-data'
+})
+
+// Node 2 - Own filesystem  
+const brain2 = new Brainy({
+  storage: '/data/node2'  // or auto: './brainy-data'
+})
+
+// ✅ BENEFITS:
+// - No conflicts between nodes
+// - Fast local reads
+// - True horizontal scaling
+// - Survives network partitions
+```
+
+#### Pattern 2: Separate S3 Buckets Per Node
+
+```typescript
+// Node 1 - Own S3 bucket
+const brain1 = new Brainy({
+  storage: 's3://brainy-node-1'  // Auto-uses AWS credentials
+})
+
+// Node 2 - Own S3 bucket
+const brain2 = new Brainy({
+  storage: 's3://brainy-node-2'
+})
+
+// ✅ BENEFITS:
+// - Infinite storage capacity
+// - Geographic distribution
+// - No local disk needed
+// - Built-in durability
+```
+
+#### Pattern 3: Shared S3 Bucket (Coordinated)
+
+```typescript
+// All nodes - Shared bucket with coordination
+const brain = new Brainy({
+  storage: 's3://shared-brainy-data',
+  // Brainy automatically adds node-specific prefixes!
+})
+
+// What happens automatically:
+// - Node 1 writes to: s3://shared-brainy-data/node-1/
+// - Node 2 writes to: s3://shared-brainy-data/node-2/
+// - Metadata in: s3://shared-brainy-data/_cluster/
+// - Coordination via S3 conditional writes
+
+// ✅ BENEFITS:
+// - Single bucket to manage
+// - Easy backup/restore
+// - Cost effective
+// - Automatic namespace isolation
+```
+
+#### Pattern 4: Mixed Storage (Hybrid)
+
+```typescript
+// Hot data on local SSD, cold data in S3
 const brain = new Brainy({
   storage: {
-    type: 'filesystem',
-    path: '/var/lib/brainy'
+    hot: '/fast-ssd/brainy',     // Recent/frequent data
+    cold: 's3://brainy-archive'  // Older data
   }
+  // Brainy automatically promotes/demotes data!
 })
 ```
-- Stores everything in a sharded JSON tree under `path`
-- Atomic writes via rename
-- Survives process restarts
-- Snapshot it off-site with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` from your scheduler
 
-### Memory
-```typescript
-const brain = new Brainy({ storage: { type: 'memory' } })
-```
-- Zero I/O, fastest possible
-- No persistence — process exit discards everything
-- Use for tests and ephemeral caches
+### 🌍 Cloud Provider Auto-Detection
 
-### Auto
 ```typescript
 const brain = new Brainy({
-  storage: { type: 'auto', path: './data' }
+  storage: 'cloud://brainy-data'  // Auto-detects provider!
 })
+
+// Automatically uses:
+// - AWS: S3 + DynamoDB for metadata
+// - Google Cloud: GCS + Firestore
+// - Azure: Blob Storage + Cosmos DB
+// - Cloudflare: R2 + D1
+// - Vercel: Blob + KV
 ```
-- Picks `filesystem` when running on Node with a writable `path`
-- Falls back to `memory` otherwise
+
+### 📝 Storage Coordination Rules
+
+When multiple nodes share storage, Brainy automatically:
+
+1. **Namespace Isolation**: Each node gets unique prefix
+2. **Lock-Free Writes**: Uses atomic operations
+3. **Consistent Metadata**: Coordinated via consensus
+4. **Conflict Resolution**: Version vectors for conflicts
+5. **Garbage Collection**: Automatic cleanup of old data
 
 ## Scaling Patterns
 
-### Stage 1: Prototype (Memory)
+### 📈 Progressive Scaling Journey
+
+#### Stage 1: Prototype (1 node, memory)
 ```typescript
-const brain = new Brainy({ storage: { type: 'memory' } })
-// Development, tests, <100K items
+const brain = new Brainy()  // Memory storage, single node
+// Perfect for: Development, testing, <1000 items
 ```
 
-### Stage 2: Production (Filesystem)
+#### Stage 2: Production (1 node, disk)
 ```typescript
 const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' }
+  storage: './data'  // Persistent storage
 })
-// Most production workloads up to ~10M entities on a single host
+// Perfect for: Small apps, <100K items
 ```
 
-### Stage 3: Higher Throughput (Tune the Vector Index)
+#### Stage 3: High Availability (2-3 nodes)
 ```typescript
+// Just start same code on multiple servers!
 const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' },
-  vector: {
-    recall: 'fast',                // Trade recall for latency
-    persistMode: 'deferred'        // Batch persistence
-  }
+  storage: './data'  // Each node's own storage
 })
+// Automatic: Leader election, replication, failover
+// Perfect for: Critical apps, <1M items
 ```
 
-### Stage 4: Multi-Instance (Operator-Layer)
-Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `path`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes.
+#### Stage 4: Scale Out (N nodes)
+```typescript
+// Same code, more servers!
+const brain = new Brainy({
+  storage: 's3://brainy-{{nodeId}}'  // Template auto-filled
+})
+// Automatic: Sharding, load balancing, geo-distribution
+// Perfect for: Large apps, unlimited items
+```
+
+### 🎯 Common Scaling Scenarios
+
+#### Scenario: Read-Heavy Application
+```typescript
+// Brainy auto-detects read-heavy pattern and:
+// 1. Increases cache size
+// 2. Creates more read replicas
+// 3. Routes reads to nearest node
+// 4. Caches popular items on all nodes
+
+const brain = new Brainy()  // No config needed!
+```
+
+#### Scenario: Multi-Tenant SaaS
+```typescript
+// Brainy auto-detects tenant patterns and:
+// 1. Shards by tenant ID
+// 2. Isolates tenant data
+// 3. Routes by tenant
+// 4. Separate rate limits per tenant
+
+const brain = new Brainy()  // Detects from your queries!
+```
+
+#### Scenario: Geographic Distribution
+```typescript
+// Deploy nodes in different regions
+// Brainy automatically:
+// 1. Detects node locations (via latency)
+// 2. Replicates data geographically
+// 3. Routes to nearest node
+// 4. Handles region failures
+
+// US-East
+const brain = new Brainy({ region: 'us-east' })  // Optional hint
+
+// EU-West (auto-discovers US-East)
+const brain = new Brainy({ region: 'eu-west' })
+```
 
 ## Real World Examples
 
-### Example 1: Single-Node App With Backup
+### Example 1: Blog Platform
+
 ```typescript
+// Day 1: Single server
 const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' }
+  storage: './blog-data'
 })
-```
-Schedule (cron / systemd timer):
-```bash
-*/15 * * * *  rclone sync /var/lib/brainy remote:brainy-backup
+
+// Month 6: Add redundancy (on second server)
+const brain = new Brainy({
+  storage: './blog-data'  // Different machine!
+})
+// Automatically syncs with first server
+
+// Year 2: Global scale
+// US Server
+const brain = new Brainy({
+  storage: 's3://blog-us/data'
+})
+
+// EU Server  
+const brain = new Brainy({
+  storage: 's3://blog-eu/data'
+})
+
+// Asia Server
+const brain = new Brainy({
+  storage: 's3://blog-asia/data'
+})
+// All automatically coordinate!
 ```
 
-### Example 2: Tests
+### Example 2: E-Commerce Site
+
 ```typescript
-const brain = new Brainy({ storage: { type: 'memory' } })
-// Fast, no cleanup needed between runs
+// Development
+const brain = new Brainy()  // Memory storage
+
+// Staging (Kubernetes)
+const brain = new Brainy({
+  storage: process.env.STORAGE_PATH  // Uses PVC
+})
+// Auto-discovers other pods via K8s DNS
+
+// Production (AWS)
+const brain = new Brainy({
+  storage: 's3://shop-data',
+  cache: 'elasticache://shop-cache'  // Optional
+})
+// Auto-scales with ECS/EKS
 ```
 
-### Example 3: Multi-Tenant Service
-Spin up one Brainy instance per tenant, each in its own directory:
+### Example 3: Analytics Platform
+
 ```typescript
-function brainForTenant(tenantId: string) {
-  return new Brainy({
-    storage: {
-      type: 'filesystem',
-      path: `/var/lib/brainy/${tenantId}`
-    }
-  })
+// Ingestion nodes (write-optimized)
+const brain = new Brainy({
+  role: 'writer',  // Hint for optimization
+  storage: '/fast-nvme/ingest'
+})
+
+// Query nodes (read-optimized)
+const brain = new Brainy({
+  role: 'reader',  // More cache, indexes
+  storage: 's3://analytics-archive'
+})
+
+// Automatically coordinates between writers and readers!
+```
+
+## 🔧 Storage Adapter Specifics
+
+### Local Filesystem
+```typescript
+{
+  storage: './data'  // or absolute: '/var/lib/brainy'
+  // Each node MUST have separate directory
+  // Can be network mounted (NFS, EFS)
 }
 ```
-Your service layer handles routing and isolation; Brainy stays simple.
 
-### Example 4: Higher Recall at Scale
+### AWS S3
 ```typescript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' },
-  vector: {
-    recall: 'accurate'
+{
+  storage: 's3://bucket-name/prefix'
+  // Uses AWS SDK credentials (env, IAM role, etc)
+  // Supports S3-compatible (MinIO, Ceph)
+}
+```
+
+### Cloudflare R2
+```typescript
+{
+  storage: 'r2://bucket-name'
+  // Uses Wrangler or API tokens
+  // Zero egress fees!
+}
+```
+
+### Google Cloud Storage
+```typescript
+{
+  storage: 'gs://bucket-name'
+  // Uses Application Default Credentials
+}
+```
+
+### Azure Blob Storage
+```typescript
+{
+  storage: 'azure://container-name'
+  // Uses DefaultAzureCredential
+}
+```
+
+### Mixed/Tiered
+```typescript
+{
+  storage: {
+    hot: './local-cache',      // Fast SSD
+    warm: 's3://regular-data',  // Standard storage
+    cold: 's3://glacier-archive' // Cheap archive
   }
+  // Automatic tiering based on access patterns
+}
+```
+
+## 🎭 Advanced Patterns
+
+### Pattern: Blue-Green Deployment
+```typescript
+// Blue cluster (current)
+const brain = new Brainy({
+  cluster: 'blue',
+  storage: 's3://prod-blue'
+})
+
+// Green cluster (new version)
+const brain = new Brainy({
+  cluster: 'green',
+  storage: 's3://prod-green',
+  syncFrom: 'blue'  // Real-time sync during migration
 })
 ```
 
-## Tuning Knobs Summary
+### Pattern: Federation
+```typescript
+// Region 1 Cluster
+const brain1 = new Brainy({
+  federation: 'global',
+  region: 'us-east',
+  storage: 's3://us-east-data'
+})
 
-| Setting | Values | When to change |
-|---------|--------|----------------|
-| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
-| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
-| `storage.cache.maxSize` | integer | Hot-path read cache size |
-| `storage.cache.ttl` | ms | Cache freshness |
+// Region 2 Cluster
+const brain2 = new Brainy({
+  federation: 'global',
+  region: 'eu-west',
+  storage: 's3://eu-west-data'
+})
+// Clusters coordinate for global queries!
+```
 
-## Monitoring & Observability
+### Pattern: Edge Computing
+```typescript
+// Edge nodes (in CDN POPs)
+const brain = new Brainy({
+  mode: 'edge',
+  storage: 'memory',  // RAM only
+  upstream: 'https://main-cluster.example.com'
+})
+// Caches frequently accessed data at edge
+```
+
+## 📊 Monitoring & Observability
+
+Brainy automatically exposes metrics:
 
 ```typescript
-const stats = await brain.stats()
+const metrics = brain.getMetrics()
 // {
-//   nounCount: 50000,
-//   verbCount: 80000,
-//   vectorIndex: { ... },
-//   storage: { used: '45GB' }
+//   nodes: { total: 5, healthy: 5 },
+//   shards: { total: 20, local: 4 },
+//   replication: { factor: 2, lag: 45 },
+//   operations: { reads: 10000, writes: 1000 },
+//   storage: { used: '45GB', available: '955GB' }
 // }
 ```
 
-## Troubleshooting
+## 🚨 Troubleshooting
 
-### Issue: Slow queries
-1. Switch to `vector.recall: 'fast'`
-2. Increase the read cache (`storage.cache.maxSize`)
-3. Consider the optional native vector provider via `@soulcraft/cor`
+### Issue: Nodes don't discover each other
+```typescript
+// Solution 1: Check network allows UDP 7946
+// Solution 2: Use explicit peers
+const brain = new Brainy({
+  peers: ['10.0.0.1:7946', '10.0.0.2:7946']
+})
+```
 
-### Issue: Memory pressure
-1. Reduce `storage.cache.maxSize`
-2. Move to `vector.persistMode: 'deferred'` to batch writes
-3. Consider the optional native vector provider via `@soulcraft/cor` for at-scale index acceleration
+### Issue: Storage conflicts
+```typescript
+// Ensure each node has unique storage path
+// ❌ WRONG: All nodes use './data'
+// ✅ RIGHT: Node1: './data1', Node2: './data2'
+// ✅ RIGHT: Use {{nodeId}} template
+```
 
-### Issue: Slow startup after a crash
-1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
-2. Verify backup integrity periodically
+### Issue: Slow performance
+```typescript
+// Brainy auto-tunes, but you can hint:
+const brain = new Brainy({
+  profile: 'read-heavy'  // or 'write-heavy', 'balanced'
+})
+```
 
-## Best Practices
+## 🎯 Best Practices
 
-1. **One process = one `path`** — never share a directory between processes
-2. **Snapshot from your scheduler** — Brainy doesn't ship cloud SDKs; use `rclone` / `aws s3 sync` / `gsutil`
-3. **Profile before tuning** — `recall: 'balanced'` is right for most workloads
-4. **Install the native vector provider only when measured profiling shows it pays off**
+1. **Let Brainy Auto-Configure**: Don't over-configure
+2. **Separate Storage Per Node**: Avoids conflicts
+3. **Use S3 for Large Scale**: Infinite capacity
+4. **Start Simple**: Single node → Scale when needed
+5. **Monitor Metrics**: Watch for bottlenecks
+6. **Trust Auto-Scaling**: It learns your patterns
 
-## Summary
+## 🚀 Summary
 
-- Brainy 8.0 is a **library**, not a cluster
-- Storage adapters: `filesystem`, `memory`, `auto`
-- Vector tuning: `recall`, `persistMode`
-- Backup is an operator-layer concern — snapshot `path`
+- **Zero Config**: Just `new Brainy()` at any scale
+- **Auto-Discovery**: Nodes find each other
+- **Smart Storage**: Adapts to any backend
+- **Progressive Scaling**: 1 → 100 nodes seamlessly
+- **Self-Tuning**: Learns and optimizes
+- **No DevOps**: It just works!
+
+**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!**
+
+---
+
+*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)*

docs/STAGE3-CANONICAL-TAXONOMY.md

@@ -1,373 +0,0 @@
-# Brainy Stage 3: Canonical Taxonomy
-
-**Status:** FINAL - This is the definitive, timeless taxonomy
-**Total Types:** 169 (42 nouns + 127 verbs)
-**Coverage:** 96-97% of all human knowledge
-**Designed to last:** 20+ years without changes
-
----
-
-## Summary
-
-- **Nouns:** 42 types
-- **Verbs:** 127 types
-- **Total:** 169 types
-- **Previous (v5.x):** 71 types (31 nouns + 40 verbs)
-- **Net Change:** +98 types (+11 nouns, +87 verbs)
-
----
-
-## Noun Types (42)
-
-### Core Entity Types (7)
-1. **person** - Individual human entities
-2. **organization** - Collective entities, companies, institutions
-3. **location** - Geographic and named spatial entities
-4. **thing** - Discrete physical objects and artifacts
-5. **concept** - Abstract ideas, principles, and intangibles
-6. **event** - Temporal occurrences and happenings
-7. **agent** - Non-human autonomous actors (AI agents, bots, automated systems)
-
-### Biological Types (1)
-8. **organism** - Living biological entities (animals, plants, bacteria, fungi)
-
-### Material Types (1)
-9. **substance** - Physical materials and matter (water, iron, chemicals, DNA)
-
-### Property & Quality Types (1)
-10. **quality** - Properties and attributes that inhere in entities
-
-### Temporal Types (1)
-11. **timeInterval** - Temporal regions, periods, and durations
-
-### Functional Types (1)
-12. **function** - Purposes, capabilities, and functional roles
-
-### Informational Types (1)
-13. **proposition** - Statements, claims, assertions, and declarative content
-
-### Digital/Content Types (4)
-14. **document** - Text-based files and written content
-15. **media** - Non-text media files (audio, video, images)
-16. **file** - Generic digital files and data blobs
-17. **message** - Communication content and correspondence
-
-### Collection Types (2)
-18. **collection** - Groups and sets of items
-19. **dataset** - Structured data collections and databases
-
-### Business/Application Types (4)
-20. **product** - Commercial products and offerings
-21. **service** - Service offerings and intangible products
-22. **task** - Actions, todos, and work items
-23. **project** - Organized initiatives and programs
-
-### Descriptive Types (6)
-24. **process** - Workflows, procedures, and ongoing activities
-25. **state** - Conditions, status, and situational contexts
-26. **role** - Positions, responsibilities, and functional classifications
-27. **language** - Natural and formal languages
-28. **currency** - Monetary units and exchange mediums
-29. **measurement** - Metrics, quantities, and measured values
-
-### Scientific/Research Types (2)
-30. **hypothesis** - Scientific theories, propositions, and conjectures
-31. **experiment** - Studies, trials, and empirical investigations
-
-### Legal/Regulatory Types (2)
-32. **contract** - Legal agreements, terms, and binding documents
-33. **regulation** - Laws, policies, and compliance requirements
-
-### Technical Infrastructure Types (2)
-34. **interface** - APIs, protocols, and connection points
-35. **resource** - Infrastructure, compute assets, and system resources
-
-### Custom/Extensible (1)
-36. **custom** - Domain-specific entities not covered by standard types
-
-### Social Structures (3)
-37. **socialGroup** - Informal social groups and collectives
-38. **institution** - Formal social structures and practices
-39. **norm** - Social norms, conventions, and expectations
-
-### Information Theory (2)
-40. **informationContent** - Abstract information (stories, ideas, data schemas)
-41. **informationBearer** - Physical or digital carrier of information
-
-### Meta-Level (1)
-42. **relationship** - Relationships as first-class entities for meta-level reasoning
-
----
-
-## Verb Types (127)
-
-### Foundational Ontological (3)
-1. **instanceOf** - Individual to class relationship
-2. **subclassOf** - Taxonomic hierarchy
-3. **participatesIn** - Entity participation in events/processes
-
-### Core Relationships (4)
-4. **relatedTo** - Generic relationship (fallback)
-5. **contains** - Containment relationship
-6. **partOf** - Part-whole mereological relationship
-7. **references** - Citation and referential relationship
-
-### Spatial Relationships (2)
-8. **locatedAt** - Spatial location relationship
-9. **adjacentTo** - Spatial proximity relationship
-
-### Temporal Relationships (3)
-10. **precedes** - Temporal sequence (before)
-11. **during** - Temporal containment
-12. **occursAt** - Temporal location
-
-### Causal & Dependency (5)
-13. **causes** - Direct causal relationship
-14. **enables** - Enablement without direct causation
-15. **prevents** - Prevention relationship
-16. **dependsOn** - Dependency relationship
-17. **requires** - Necessity relationship
-
-### Creation & Transformation (5)
-18. **creates** - Creation relationship
-19. **transforms** - Transformation relationship
-20. **becomes** - State change relationship
-21. **modifies** - Modification relationship
-22. **consumes** - Consumption relationship
-
-### Lifecycle Operations (1)
-23. **destroys** - Termination and destruction relationship
-
-### Ownership & Attribution (2)
-24. **owns** - Ownership relationship
-25. **attributedTo** - Attribution relationship
-
-### Property & Quality (2)
-26. **hasQuality** - Entity to quality attribution
-27. **realizes** - Function realization relationship
-
-### Effects & Experience (1)
-28. **affects** - Patient/experiencer relationship
-
-### Composition (2)
-29. **composedOf** - Material composition
-30. **inherits** - Inheritance relationship
-
-### Social & Organizational (7)
-31. **memberOf** - Membership relationship
-32. **worksWith** - Professional collaboration
-33. **friendOf** - Friendship relationship
-34. **follows** - Following/subscription relationship
-35. **likes** - Liking/favoriting relationship
-36. **reportsTo** - Hierarchical reporting relationship
-37. **mentors** - Mentorship relationship
-38. **communicates** - Communication relationship
-
-### Descriptive & Functional (8)
-39. **describes** - Descriptive relationship
-40. **defines** - Definition relationship
-41. **categorizes** - Categorization relationship
-42. **measures** - Measurement relationship
-43. **evaluates** - Evaluation relationship
-44. **uses** - Utilization relationship
-45. **implements** - Implementation relationship
-46. **extends** - Extension relationship
-
-### Advanced Relationships (4)
-47. **equivalentTo** - Equivalence/identity relationship
-48. **believes** - Epistemic relationship
-49. **conflicts** - Conflict relationship
-50. **synchronizes** - Synchronization relationship
-51. **competes** - Competition relationship
-
-### Modal Relationships (6)
-52. **canCause** - Potential causation (possibility)
-53. **mustCause** - Necessary causation (necessity)
-54. **wouldCauseIf** - Counterfactual causation
-55. **couldBe** - Possible states
-56. **mustBe** - Necessary identity
-57. **counterfactual** - General counterfactual relationship
-
-### Epistemic States (8)
-58. **knows** - Knowledge (justified true belief)
-59. **doubts** - Uncertainty/skepticism
-60. **desires** - Want/preference
-61. **intends** - Intentionality
-62. **fears** - Fear/anxiety
-63. **loves** - Strong positive emotional attitude
-64. **hates** - Strong negative emotional attitude
-65. **hopes** - Hopeful expectation
-66. **perceives** - Sensory perception
-
-### Learning & Cognition (1)
-67. **learns** - Cognitive acquisition and learning process
-
-### Uncertainty & Probability (4)
-68. **probablyCauses** - Probabilistic causation
-69. **uncertainRelation** - Unknown relationship with confidence bounds
-70. **correlatesWith** - Statistical correlation
-71. **approximatelyEquals** - Fuzzy equivalence
-
-### Scalar Properties (5)
-72. **greaterThan** - Scalar comparison
-73. **similarityDegree** - Graded similarity
-74. **moreXThan** - Comparative property
-75. **hasDegree** - Scalar property assignment
-76. **partiallyHas** - Graded possession
-
-### Information Theory (2)
-77. **carries** - Bearer carries content
-78. **encodes** - Encoding relationship
-
-### Deontic Relationships (5)
-79. **obligatedTo** - Moral/legal obligation
-80. **permittedTo** - Permission/authorization
-81. **prohibitedFrom** - Prohibition/forbidden
-82. **shouldDo** - Normative expectation
-83. **mustNotDo** - Strong prohibition
-
-### Context & Perspective (5)
-84. **trueInContext** - Context-dependent truth
-85. **perceivedAs** - Subjective perception
-86. **interpretedAs** - Interpretation relationship
-87. **validInFrame** - Frame-dependent validity
-88. **trueFrom** - Perspective-dependent truth
-
-### Advanced Temporal (6)
-89. **overlaps** - Partial temporal overlap
-90. **immediatelyAfter** - Direct temporal succession
-91. **eventuallyLeadsTo** - Long-term consequence
-92. **simultaneousWith** - Exact temporal alignment
-93. **hasDuration** - Temporal extent
-94. **recurringWith** - Cyclic temporal relationship
-
-### Advanced Spatial (7)
-95. **containsSpatially** - Spatial containment
-96. **overlapsSpatially** - Spatial overlap
-97. **surrounds** - Encirclement
-98. **connectedTo** - Topological connection
-99. **above** - Vertical spatial relationship (superior)
-100. **below** - Vertical spatial relationship (inferior)
-101. **inside** - Within containment boundaries
-102. **outside** - Beyond containment boundaries
-103. **facing** - Directional orientation
-
-### Social Structures (5)
-104. **represents** - Representative relationship
-105. **embodies** - Exemplification or personification
-106. **opposes** - Opposition relationship
-107. **alliesWith** - Alliance relationship
-108. **conformsTo** - Norm conformity
-
-### Measurement (4)
-109. **measuredIn** - Unit relationship
-110. **convertsTo** - Unit conversion
-111. **hasMagnitude** - Quantitative value
-112. **dimensionallyEquals** - Dimensional analysis
-
-### Change & Persistence (4)
-113. **persistsThrough** - Persistence through change
-114. **gainsProperty** - Property acquisition
-115. **losesProperty** - Property loss
-116. **remainsSame** - Identity through time
-
-### Parthood Variations (4)
-117. **functionalPartOf** - Functional component
-118. **topologicalPartOf** - Spatial part
-119. **temporalPartOf** - Temporal slice
-120. **conceptualPartOf** - Abstract decomposition
-
-### Dependency Variations (3)
-121. **rigidlyDependsOn** - Necessary dependency
-122. **functionallyDependsOn** - Operational dependency
-123. **historicallyDependsOn** - Causal history dependency
-
-### Meta-Level (4)
-124. **endorses** - Second-order validation
-125. **contradicts** - Logical contradiction
-126. **supports** - Evidential support
-127. **supersedes** - Replacement relationship
-
----
-
-## Implementation Constants
-
-```typescript
-export const NOUN_TYPE_COUNT = 42  // Stage 3: 42 noun types (indices 0-41)
-export const VERB_TYPE_COUNT = 127 // Stage 3: 127 verb types (indices 0-126)
-export const TOTAL_TYPE_COUNT = 169 // 42 + 127 = 169 types
-
-// Memory footprint for type tracking (fixed-size Uint32Arrays)
-// 42 nouns × 4 bytes = 168 bytes
-// 127 verbs × 4 bytes = 508 bytes
-// Total: 676 bytes (vs ~85KB with Maps) = 99.2% memory reduction
-```
-
----
-
-## Changes from v5.x
-
-### Nouns Added (+11)
-- agent, quality, timeInterval, function, proposition
-- **organism** ⭐ (biological entities)
-- **substance** ⭐ (physical materials)
-- socialGroup, institution, norm
-- informationContent, informationBearer, relationship
-
-### Nouns Removed (-2)
-- **user** (merged into person)
-- **topic** (merged into concept)
-- **content** (removed - redundant)
-
-### Verbs Added (+87)
-- **affects** ⭐ (patient/experiencer role)
-- **learns** ⭐ (cognitive acquisition)
-- **destroys** ⭐ (lifecycle termination)
-- All new categories from Stage 3 taxonomy
-
-### Verbs Removed (-4)
-- **succeeds** (use inverse of precedes)
-- **belongsTo** (use inverse of owns)
-- **createdBy** (use inverse of creates)
-- **supervises** (use inverse of reportsTo)
-
-⭐ = Critical additions from ultradeep analysis
-
----
-
-## Coverage & Completeness
-
-**Domain Coverage:**
-- Natural Sciences: 96% (physics, chemistry, biology, medicine)
-- Formal Sciences: 98% (mathematics, logic, computer science)
-- Social Sciences: 97% (psychology, sociology, economics)
-- Humanities: 96% (philosophy, history, arts)
-
-**Overall:** 96-97% of all human knowledge
-
-**Timeless Design:** Stable for 20+ years
-
-**Extension:** Use "custom" noun for domain-specific entities
-
----
-
-## Verification Checklist
-
-All code, comments, and documentation MUST match this canonical list:
-
-- [ ] graphTypes.ts: NounType has exactly 42 entries
-- [ ] graphTypes.ts: VerbType has exactly 127 entries
-- [ ] graphTypes.ts: NOUN_TYPE_COUNT = 42
-- [ ] graphTypes.ts: VERB_TYPE_COUNT = 127
-- [ ] graphTypes.ts: NounTypeEnum has indices 0-41
-- [ ] graphTypes.ts: VerbTypeEnum has indices 0-126
-- [ ] metadataIndex.ts: Arrays sized for 42 & 127
-- [ ] buildTypeEmbeddings.ts: Descriptions for all 169 types
-- [ ] brainyTypes.ts: Descriptions for all 169 types
-- [ ] index.ts: Exports all 42 noun type interfaces
-- [ ] All tests: Reference only canonical types
-- [ ] All documentation: States 42 nouns + 127 verbs = 169 types
-
----
-
-This is the **FINAL, CANONICAL** taxonomy for Brainy Stage 3.

docs/VALIDATION.md

@@ -0,0 +1,340 @@
+# Brainy Validation System
+
+## Zero-Config Philosophy
+
+Brainy's validation system automatically adapts to your system resources without any configuration. It enforces universal truths while dynamically adjusting limits based on available memory and observed performance.
+
+## Core Principles
+
+### 1. Universal Truths Only
+We only validate things that are mathematically or logically impossible:
+- Negative pagination values (there's no page -1)
+- Probabilities outside 0-1 range
+- Self-referential relationships
+- Invalid enum values
+
+### 2. Auto-Configuration
+The system automatically configures based on:
+- **Available Memory**: More RAM = higher limits
+- **System Performance**: Adjusts based on query response times
+- **Usage Patterns**: Learns from your actual workload
+
+### 3. Performance Monitoring
+Every query is monitored to tune future limits:
+```typescript
+// Automatic adjustment based on performance
+if (avgQueryTime < 100ms && resultCount > 80% of limit) {
+  // Increase limits - system can handle more
+  maxLimit *= 1.5
+} else if (avgQueryTime > 1000ms) {
+  // Reduce limits - system is struggling
+  maxLimit *= 0.8
+}
+```
+
+## Validation Rules by Method
+
+### `add(params: AddParams)`
+
+**Required:**
+- Either `data` or `vector` must be provided
+- `type` must be a valid NounType enum
+
+**Constraints:**
+- `vector` must have exactly 384 dimensions (for all-MiniLM-L6-v2)
+- Custom `id` must be unique
+
+**Example:**
+```typescript
+// ✅ Valid
+await brain.add({
+  data: "Hello world",
+  type: NounType.Document
+})
+
+// ✅ Valid - pre-computed vector
+await brain.add({
+  vector: new Array(384).fill(0),
+  type: NounType.Document
+})
+
+// ❌ Invalid - missing both data and vector
+await brain.add({
+  type: NounType.Document
+})
+// Error: "must provide either data or vector"
+
+// ❌ Invalid - wrong vector dimensions
+await brain.add({
+  vector: new Array(100).fill(0),
+  type: NounType.Document
+})
+// Error: "vector must have exactly 384 dimensions"
+```
+
+### `update(params: UpdateParams)`
+
+**Required:**
+- `id` must be provided
+- At least one field must be updated
+
+**Important Metadata Behavior:**
+- `metadata: null` with `merge: false` → **Keeps existing metadata** (does nothing)
+- `metadata: {}` with `merge: false` → **Clears metadata** ✅
+- `metadata: undefined` → No change to metadata
+
+**Example:**
+```typescript
+// ✅ Valid - update metadata
+await brain.update({
+  id: "xyz",
+  metadata: { status: "published" }
+})
+
+// ✅ Valid - clear metadata properly
+await brain.update({
+  id: "xyz",
+  metadata: {},
+  merge: false
+})
+
+// ❌ Invalid - null doesn't clear metadata
+await brain.update({
+  id: "xyz",
+  metadata: null,
+  merge: false
+})
+// Error: "must specify at least one field to update"
+// (because null metadata doesn't actually update anything)
+
+// ❌ Invalid - no fields to update
+await brain.update({
+  id: "xyz"
+})
+// Error: "must specify at least one field to update"
+```
+
+### `relate(params: RelateParams)`
+
+**Required:**
+- `from` entity ID
+- `to` entity ID
+- `type` must be valid VerbType enum
+
+**Constraints:**
+- `from` and `to` must be different (no self-loops)
+- `weight` must be between 0 and 1
+
+**Example:**
+```typescript
+// ✅ Valid
+await brain.relate({
+  from: "entity1",
+  to: "entity2",
+  type: VerbType.RelatedTo
+})
+
+// ❌ Invalid - self-referential
+await brain.relate({
+  from: "entity1",
+  to: "entity1",
+  type: VerbType.RelatedTo
+})
+// Error: "cannot create self-referential relationship"
+
+// ❌ Invalid - weight out of range
+await brain.relate({
+  from: "entity1",
+  to: "entity2",
+  type: VerbType.RelatedTo,
+  weight: 1.5
+})
+// Error: "weight must be between 0 and 1"
+```
+
+### `find(params: FindParams)`
+
+**Constraints:**
+- `limit` must be non-negative and below auto-configured maximum
+- `offset` must be non-negative
+- Cannot specify both `query` and `vector` (mutually exclusive)
+- Cannot use both `cursor` and `offset` pagination
+- `threshold` must be between 0 and 1
+
+**Auto-Configured Limits:**
+```typescript
+// Based on available memory
+// 1GB RAM → max limit: 10,000
+// 8GB RAM → max limit: 80,000
+// 16GB RAM → max limit: 100,000 (capped)
+
+// Query length also scales with memory
+// 1GB RAM → max query: 5,000 characters
+// 8GB RAM → max query: 40,000 characters
+```
+
+**Example:**
+```typescript
+// ✅ Valid
+await brain.find({
+  query: "machine learning",
+  limit: 50
+})
+
+// ❌ Invalid - negative limit
+await brain.find({
+  query: "test",
+  limit: -1
+})
+// Error: "limit must be non-negative"
+
+// ❌ Invalid - both query and vector
+await brain.find({
+  query: "test",
+  vector: new Array(384).fill(0)
+})
+// Error: "cannot specify both query and vector - they are mutually exclusive"
+
+// ❌ Invalid - exceeds auto-configured limit
+await brain.find({
+  limit: 1000000
+})
+// Error: "limit exceeds auto-configured maximum of 80000 (based on available memory)"
+```
+
+## Auto-Configuration Details
+
+### Memory-Based Scaling
+
+The validation system checks available memory on initialization:
+
+```typescript
+const availableMemory = os.freemem()
+
+// Scale limits based on available memory
+maxLimit = Math.min(
+  100000, // Absolute maximum for safety
+  Math.floor(availableMemory / (1024 * 1024 * 100)) * 1000
+)
+
+// Scale query length similarly
+maxQueryLength = Math.min(
+  50000,
+  Math.floor(availableMemory / (1024 * 1024 * 10)) * 1000
+)
+```
+
+### Performance-Based Tuning
+
+The system continuously monitors and adjusts:
+
+1. **After each query**, performance is recorded
+2. **Limits adjust** based on response times
+3. **Gradual optimization** towards optimal throughput
+
+### Checking Current Configuration
+
+You can inspect the current validation configuration:
+
+```typescript
+import { getValidationConfig } from '@soulcraft/brainy/validation'
+
+const config = getValidationConfig()
+console.log(config)
+// {
+//   maxLimit: 80000,
+//   maxQueryLength: 40000,
+//   maxVectorDimensions: 384,
+//   systemMemory: 17179869184,
+//   availableMemory: 8589934592
+// }
+```
+
+## Best Practices
+
+### 1. Clearing Metadata
+```typescript
+// ❌ Wrong - doesn't clear
+await brain.update({ id, metadata: null, merge: false })
+
+// ✅ Correct - actually clears
+await brain.update({ id, metadata: {}, merge: false })
+```
+
+### 2. Type Safety
+```typescript
+// ❌ Wrong - string type
+await brain.add({ data: "test", type: "document" })
+
+// ✅ Correct - enum type
+import { NounType } from '@soulcraft/brainy'
+await brain.add({ data: "test", type: NounType.Document })
+```
+
+### 3. Pagination
+```typescript
+// ✅ Let the system auto-configure limits
+const results = await brain.find({ 
+  query: "test",
+  limit: 100  // Will be capped at system maximum
+})
+
+// ✅ For large datasets, use pagination
+let offset = 0
+const pageSize = 1000
+while (true) {
+  const results = await brain.find({
+    query: "test",
+    limit: pageSize,
+    offset
+  })
+  if (results.length === 0) break
+  offset += pageSize
+}
+```
+
+## Error Messages
+
+All validation errors are descriptive and actionable:
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| `"must provide either data or vector"` | Missing content in add() | Provide either data to embed or pre-computed vector |
+| `"limit must be non-negative"` | Negative pagination | Use positive limit value |
+| `"invalid NounType: xyz"` | Invalid enum value | Use valid NounType enum |
+| `"cannot create self-referential relationship"` | from === to | Use different entity IDs |
+| `"must specify at least one field to update"` | Empty update | Provide at least one field to change |
+| `"vector must have exactly 384 dimensions"` | Wrong vector size | Use 384-dimensional vectors |
+
+## Performance Impact
+
+The validation system adds minimal overhead:
+- **Validation time**: <1ms per operation
+- **Memory usage**: ~1KB for configuration tracking
+- **Auto-tuning**: Happens asynchronously, no blocking
+
+## FAQ
+
+**Q: Why can't I set metadata to null?**
+A: Setting metadata to `null` with `merge: false` doesn't actually clear it - it falls back to existing metadata. Use `{}` to clear.
+
+**Q: Why are my limits being reduced?**
+A: If queries are taking >1 second, the system automatically reduces limits to maintain performance.
+
+**Q: Can I override the auto-configured limits?**
+A: No, this is by design. The system knows better than static configuration what your hardware can handle.
+
+**Q: Why exactly 384 dimensions for vectors?**
+A: Brainy uses the all-MiniLM-L6-v2 model which produces 384-dimensional embeddings. This ensures consistency.
+
+## Summary
+
+Brainy's validation system:
+- ✅ **Zero configuration** - adapts to your system
+- ✅ **Universal truths** - only prevents impossible operations
+- ✅ **Performance aware** - adjusts based on actual performance
+- ✅ **Type safe** - enforces enum types
+- ✅ **Minimal overhead** - <1ms validation time
+- ✅ **Clear errors** - actionable error messages
+
+The philosophy is simple: prevent impossible operations, adapt to reality, and get out of the way.

docs/ZERO_CONFIG.md

@@ -0,0 +1,406 @@
+# 🚀 Brainy Zero-Configuration Guide
+
+## Overview
+
+Starting with v2.10, Brainy introduces a **Zero-Configuration System** that automatically configures everything based on your environment. No more environment variables, no more complex configuration objects - just create and use.
+
+## Quick Start
+
+### True Zero Config
+```typescript
+import { Brainy } from '@soulcraft/brainy'
+
+// That's it! Everything auto-configures
+const brain = new Brainy()
+await brain.init()
+```
+
+### Using Strongly-Typed Presets
+
+```typescript
+import { Brainy, PresetName } from '@soulcraft/brainy'
+
+// Type-safe preset selection
+const brain = new Brainy(PresetName.PRODUCTION)
+await brain.init()
+```
+
+### Common Scenarios
+
+#### Development
+```typescript
+const brain = new Brainy(PresetName.DEVELOPMENT)
+// ✅ Filesystem storage for persistence
+// ✅ FP32 models for best quality
+// ✅ Verbose logging
+// ✅ All features enabled
+```
+
+#### Production
+```typescript
+const brain = new Brainy(PresetName.PRODUCTION)
+// ✅ Disk storage for persistence
+// ✅ Auto-selected model precision
+// ✅ Silent logging
+// ✅ Optimized features
+```
+
+#### Minimal
+```typescript
+const brain = new Brainy(PresetName.MINIMAL)
+// ✅ Filesystem storage
+// ✅ Q8 models for small size
+// ✅ Core features only
+// ✅ Minimal resource usage
+```
+
+## Distributed Architecture Presets
+
+Brainy includes specialized presets for distributed and microservice architectures:
+
+### Basic Distributed Roles
+```typescript
+import { Brainy, PresetName } from '@soulcraft/brainy'
+
+// Write-only instance (data ingestion)
+const writer = new Brainy(PresetName.WRITER)
+// ✅ Optimized for writes
+// ✅ No search index loading
+// ✅ Minimal memory usage
+
+// Read-only instance (search API)
+const reader = new Brainy(PresetName.READER)
+// ✅ Optimized for search
+// ✅ Lazy index loading
+// ✅ Large cache
+```
+
+### Service-Specific Presets
+```typescript
+// High-throughput data ingestion
+const ingestion = new Brainy(PresetName.INGESTION_SERVICE)
+
+// Low-latency search API
+const searchApi = new Brainy(PresetName.SEARCH_API)
+
+// Analytics processing
+const analytics = new Brainy(PresetName.ANALYTICS_SERVICE)
+
+// Edge location cache
+const edge = new Brainy(PresetName.EDGE_CACHE)
+
+// Batch processing
+const batch = new Brainy(PresetName.BATCH_PROCESSOR)
+
+// Real-time streaming
+const streaming = new Brainy(PresetName.STREAMING_SERVICE)
+
+// ML training
+const training = new Brainy(PresetName.ML_TRAINING)
+
+// Lightweight sidecar
+const sidecar = new Brainy(PresetName.SIDECAR)
+```
+
+## Model Precision Control
+
+You can **explicitly specify** model precision when needed:
+
+```typescript
+import { ModelPrecision } from '@soulcraft/brainy'
+
+// Force FP32 (full precision)
+const brain = new Brainy({ model: ModelPrecision.FP32 })
+
+// Force Q8 (quantized, smaller)
+const brain = new Brainy({ model: ModelPrecision.Q8 })
+
+// Use presets
+const brain = new Brainy({ model: ModelPrecision.FAST })  // Maps to fp32
+const brain = new Brainy({ model: ModelPrecision.SMALL }) // Maps to q8
+
+// Auto-detection (default)
+const brain = new Brainy({ model: ModelPrecision.AUTO })
+```
+
+### Auto-Detection Logic
+
+When not specified, Brainy automatically selects the best model:
+
+- **Browser**: Q8 (smaller download)
+- **Serverless**: Q8 (faster cold starts)
+- **Low Memory (<512MB)**: Q8
+- **Development**: FP32 (best quality)
+- **Production (>2GB RAM)**: FP32
+- **Default**: Q8 (balanced)
+
+## Storage Configuration
+
+### Automatic Storage Detection
+
+Brainy automatically detects the best storage option:
+
+1. **Cloud Storage** (if credentials found)
+   - AWS S3 (checks AWS_ACCESS_KEY_ID, AWS_PROFILE)
+   - Google Cloud Storage (checks GOOGLE_APPLICATION_CREDENTIALS)
+   - Cloudflare R2 (checks R2_ACCESS_KEY_ID)
+
+2. **Browser Storage**
+   - OPFS (if supported)
+   - Filesystem fallback
+
+3. **Node.js Storage**
+   - Filesystem (`./brainy-data` or `~/.brainy/data`)
+
+### Manual Storage Control
+
+```typescript
+import { StorageOption } from '@soulcraft/brainy'
+
+// Force specific storage with enum
+const brain = new Brainy({ storage: StorageOption.DISK })
+const brain = new Brainy({ storage: StorageOption.CLOUD })
+const brain = new Brainy({ storage: StorageOption.AUTO })
+
+// Custom storage configuration
+const brain = new Brainy({
+  storage: {
+    s3Storage: {
+      bucket: 'my-bucket',
+      region: 'us-east-1'
+    }
+  }
+})
+```
+
+## Feature Sets
+
+Control which features are enabled:
+
+```typescript
+import { FeatureSet } from '@soulcraft/brainy'
+
+// Preset feature sets with enum
+const brain = new Brainy({ features: FeatureSet.MINIMAL })  // Core only
+const brain = new Brainy({ features: FeatureSet.DEFAULT })  // Balanced
+const brain = new Brainy({ features: FeatureSet.FULL })     // Everything
+
+// Custom features
+const brain = new Brainy({
+  features: ['core', 'search', 'cache', 'triple-intelligence']
+})
+```
+
+## Simplified Configuration Interface
+
+The new configuration is dramatically simpler:
+
+```typescript
+interface BrainyZeroConfig {
+  // Mode preset - now with distributed options
+  mode?: PresetName  // All strongly typed presets
+  
+  // Model configuration with enum
+  model?: ModelPrecision
+  
+  // Storage configuration with enum
+  storage?: StorageOption | StorageConfig
+  
+  // Feature set with enum
+  features?: FeatureSet | string[]
+  
+  // Logging
+  verbose?: boolean
+  
+  // Escape hatch for advanced users
+  advanced?: any
+}
+```
+
+### Available Enums
+
+```typescript
+enum PresetName {
+  // Basic
+  PRODUCTION = 'production',
+  DEVELOPMENT = 'development',
+  MINIMAL = 'minimal',
+  ZERO = 'zero',
+  
+  // Distributed
+  WRITER = 'writer',
+  READER = 'reader',
+  
+  // Services
+  INGESTION_SERVICE = 'ingestion-service',
+  SEARCH_API = 'search-api',
+  ANALYTICS_SERVICE = 'analytics-service',
+  EDGE_CACHE = 'edge-cache',
+  BATCH_PROCESSOR = 'batch-processor',
+  STREAMING_SERVICE = 'streaming-service',
+  ML_TRAINING = 'ml-training',
+  SIDECAR = 'sidecar'
+}
+
+enum ModelPrecision {
+  FP32 = 'fp32',
+  Q8 = 'q8',
+  AUTO = 'auto',
+  FAST = 'fast',  // Maps to fp32
+  SMALL = 'small' // Maps to q8
+}
+
+enum StorageOption {
+  AUTO = 'auto',
+  DISK = 'disk',
+  CLOUD = 'cloud'
+}
+
+enum FeatureSet {
+  MINIMAL = 'minimal',
+  DEFAULT = 'default',
+  FULL = 'full'
+}
+```
+
+## Multi-Instance with Shared Storage
+
+When multiple Brainy instances connect to the same storage (like S3), you **must ensure they use compatible configurations**:
+
+```typescript
+import { ModelPrecision } from '@soulcraft/brainy'
+
+// Container A - Writer
+const writer = new Brainy({
+  mode: PresetName.WRITER,
+  model: ModelPrecision.FP32,  // ⚠️ MUST match across instances!
+  storage: { s3Storage: { bucket: 'shared-data' }}
+})
+
+// Container B - Reader
+const reader = new Brainy({
+  mode: PresetName.READER,
+  model: ModelPrecision.FP32,  // ✅ Matches Container A
+  storage: { s3Storage: { bucket: 'shared-data' }}
+})
+```
+
+### Distributed Architecture Example
+
+```typescript
+// Ingestion Service (Writer)
+const ingestion = new Brainy({
+  mode: PresetName.INGESTION_SERVICE,
+  model: ModelPrecision.Q8,  // All instances must use Q8
+  storage: { s3Storage: { bucket: 'production-data' }}
+})
+
+// Search API (Reader)
+const search = new Brainy({
+  mode: PresetName.SEARCH_API,
+  model: ModelPrecision.Q8,  // Matches ingestion service
+  storage: { s3Storage: { bucket: 'production-data' }}
+})
+
+// Analytics (Hybrid)
+const analytics = new Brainy({
+  mode: PresetName.ANALYTICS_SERVICE,
+  model: ModelPrecision.Q8,  // Matches other services
+  storage: { s3Storage: { bucket: 'production-data' }}
+})
+```
+
+## Migration from Old Configuration
+
+### Before (Complex)
+```typescript
+const brain = new Brainy({
+  hnsw: {
+    M: 16,
+    efConstruction: 200,
+    seed: 42
+  },
+  storage: {
+    s3Storage: {
+      bucketName: 'my-bucket',
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
+      region: 'us-east-1'
+    },
+    cacheConfig: {
+      hotCacheMaxSize: 5000,
+      hotCacheEvictionThreshold: 0.8,
+      warmCacheTTL: 3600000,
+      batchSize: 100
+    }
+  },
+  cache: {
+    autoTune: true,
+    autoTuneInterval: 60000,
+    hotCacheMaxSize: 10000
+  },
+  embeddingFunction: customFunction,
+  readOnly: false,
+  logging: { verbose: true }
+})
+```
+
+### After (Simple)
+```typescript
+const brain = new Brainy('production')
+// Everything above is auto-configured!
+```
+
+## Environment Variables (No Longer Needed!)
+
+These environment variables are **no longer required**:
+
+- ❌ `BRAINY_ALLOW_REMOTE_MODELS` - Models auto-download when needed
+- ❌ `BRAINY_MODELS_PATH` - Path auto-selected based on environment
+- ❌ `BRAINY_Q8_CONFIRMED` - Warnings auto-suppressed in production
+- ❌ `BRAINY_LOG_LEVEL` - Auto-set based on NODE_ENV
+- ❌ AWS credentials - Use AWS SDK credential chain
+
+## Performance Impact
+
+The zero-config system has **zero performance overhead**:
+
+- Configuration happens once during initialization
+- Auto-detected values are cached
+- Same optimized code paths as manual configuration
+- Actually **faster** startup due to reduced parsing
+
+## Troubleshooting
+
+### Models Not Downloading
+- Check internet connection
+- Ensure firewall allows HTTPS to Hugging Face / CDN
+- Run `npm run download-models` to pre-download
+
+### Wrong Model Precision
+- Explicitly specify: `{ model: 'fp32' }` or `{ model: 'q8' }`
+- Check shared storage compatibility
+
+### Storage Detection Issues
+- Check cloud credentials are properly configured
+- Verify write permissions for filesystem paths
+- Use explicit storage configuration if needed
+
+## Best Practices
+
+1. **Use zero-config for single instances** - Let Brainy handle everything
+2. **Specify precision for shared storage** - Ensure compatibility
+3. **Use presets for common scenarios** - 'development', 'production', 'minimal'
+4. **Override only what you need** - Start simple, add complexity only if required
+
+## Summary
+
+The new zero-config system reduces configuration from **100+ parameters** to **0-3 decisions**:
+
+| Scenario | Old Config Lines | New Config Lines |
+|----------|-----------------|------------------|
+| Development | 50+ | 1 |
+| Production | 100+ | 1 |
+| Custom | 200+ | 3-5 |
+
+**Result**: 95% less configuration, 100% of the power! 🚀

docs/api-returns.md

@@ -0,0 +1,152 @@
+# Brainy API Return Values
+
+## Core Operations
+
+### `brain.add()` - Adding Entities
+
+**Returns:** `Promise<string>` - The ID of the created entity
+
+```typescript
+// ✅ Correct usage - add() returns the ID string directly
+const id = await brain.add({
+  type: 'document',
+  data: 'My document content'
+})
+
+console.log('Created entity with ID:', id)
+
+// Use the ID to create relationships
+await brain.relate({
+  from: id,
+  to: anotherId,
+  type: 'references'
+})
+```
+
+```typescript
+// ❌ Incorrect - trying to access .id property
+const result = await brain.add({
+  type: 'document',
+  data: 'My content'
+})
+
+console.log(result.id) // ❌ undefined - result IS the ID, not an object!
+```
+
+### Getting the Full Entity
+
+If you need the full entity object after creation, use `brain.get()`:
+
+```typescript
+// Add entity and get its ID
+const id = await brain.add({
+  type: 'document',
+  data: 'My content',
+  metadata: { label: 'Important Doc' }
+})
+
+// Get the full entity
+const entity = await brain.get(id)
+console.log(entity.id)          // The ID
+console.log(entity.type)        // 'document'
+console.log(entity.data)        // 'My content'
+console.log(entity.metadata)    // { label: 'Important Doc', ... }
+console.log(entity.vector)      // The embedding vector
+console.log(entity.createdAt)   // Timestamp
+```
+
+### `brain.find()` - Finding Entities
+
+**Returns:** `Promise<Entity[]>` - Array of full entity objects
+
+```typescript
+const entities = await brain.find({
+  query: 'machine learning',
+  limit: 10
+})
+
+// Each entity has full information
+for (const entity of entities) {
+  console.log(entity.id)
+  console.log(entity.type)
+  console.log(entity.data)
+  console.log(entity.metadata)
+}
+```
+
+### `brain.relate()` - Creating Relationships
+
+**Returns:** `Promise<string>` - The ID of the created relationship
+
+```typescript
+const relationId = await brain.relate({
+  from: entityId1,
+  to: entityId2,
+  type: 'references'
+})
+
+console.log('Created relationship with ID:', relationId)
+```
+
+## Data Field Behavior
+
+### String Data - Used for Embeddings
+
+When `data` is a string, it's used to generate embeddings for semantic search:
+
+```typescript
+await brain.add({
+  type: 'document',
+  data: 'This text will be converted to an embedding vector',
+  metadata: {
+    title: 'My Document',
+    year: 2024
+  }
+})
+```
+
+### Object Data - Structured Information
+
+When `data` is an object, it's treated as structured data:
+
+```typescript
+await brain.add({
+  type: 'product',
+  data: {
+    name: 'Widget',
+    price: 29.99,
+    category: 'Tools'
+  },
+  vector: precomputedVector  // Must provide vector when using object data
+})
+```
+
+**Important:** If you provide object data without a `vector`, you must include a string somewhere for embedding generation, or the operation will fail.
+
+### Metadata vs Data
+
+- **`data`**: Primary content - used for embeddings (if string) or stored as structured data (if object)
+- **`metadata`**: Auxiliary information - always stored as structured data, used for filtering
+
+**Best practice for labels:**
+```typescript
+await brain.add({
+  type: 'document',
+  data: 'The full text content of the document...',  // For semantic search
+  metadata: {
+    label: 'Quick Reference Label',  // For display
+    author: 'John Doe',
+    category: 'Technical'
+  }
+})
+```
+
+## Summary
+
+| Method | Returns | Contains |
+|--------|---------|----------|
+| `brain.add()` | `string` | The ID of the created entity |
+| `brain.get()` | `Entity \| null` | Full entity object with all fields |
+| `brain.find()` | `Entity[]` | Array of full entity objects |
+| `brain.relate()` | `string` | The ID of the created relationship |
+| `brain.getRelations()` | `Relation[]` | Array of relationship objects |

docs/api/COMPREHENSIVE_API_OVERVIEW.md

@@ -0,0 +1,349 @@
+# 🧠 **Brainy Complete Public API Overview**
+
+> **Ultra-comprehensive analysis of Brainy's entire API surface for intuitive, consistent developer experience**
+
+## 🎯 **API Consistency Analysis**
+
+### **✅ EXCELLENT Consistency Patterns**
+
+#### **1. Constructor & Initialization**
+```typescript
+// Clean, consistent initialization
+const brain = new Brainy(config?)
+await brain.init()  // Always required
+
+// Storage auto-detection works seamlessly
+const brain = new Brainy({ storage: { forceMemoryStorage: true } })
+const brain = new Brainy({ storage: { path: './my-data' } })
+```
+
+#### **2. Data Operations (CRUD)**
+```typescript
+// ✅ CONSISTENT: Always (data, metadata) pattern with nounType in metadata
+await brain.add(content, { nounType: NounType.Person, role: 'Engineer' })
+await brain.add(content, { nounType: NounType.Document, title: 'API Guide' })
+
+// ✅ CONSISTENT: Always (source, target, type, metadata) pattern
+await brain.relate(sourceId, targetId, VerbType.RelatedTo, { strength: 0.8 })
+await brain.relate(sourceId, targetId, VerbType.Contains, { confidence: 0.9 })
+
+// ✅ CONSISTENT: Batch versions take arrays
+await brain.addNouns([...])  // Array of noun objects
+await brain.addVerbs([...])  // Array of verb objects
+```
+
+#### **3. Query Operations**
+```typescript
+// ✅ CONSISTENT: Always (query, options) pattern
+await brain.search('artificial intelligence', { limit: 10, threshold: 0.7 })
+await brain.find('recent documents about AI', { limit: 5 })  // Triple Intelligence
+
+// ✅ CONSISTENT: Get methods with filters
+await brain.getNouns(filter?)     // Optional filtering
+await brain.getVerbs(filter?)     // Optional filtering
+await brain.getNoun(id)           // Single item by ID
+await brain.getVerb(id)           // Single item by ID
+```
+
+#### **4. Main Class Shortcuts (Simple & Common)**
+```typescript
+// ✅ CONSISTENT: Simple shortcuts for most common operations
+await brain.similar(a, b)         // Returns simple number
+await brain.clusters()            // Returns simple array
+await brain.related(id, limit?)   // Returns simple array
+```
+
+### **🎨 EXCELLENT API Namespacing**
+
+#### **Main Data Operations** (Direct on `brain`)
+```typescript
+// Core CRUD - most common operations
+brain.add(content, metadata)
+brain.addNouns(items[])  // Batch operation - unchanged
+brain.relate(source, target, type, metadata?)
+brain.addVerbs(items[])  // Batch operation - unchanged
+
+brain.search(query, options?)
+brain.find(naturalLanguageQuery, options?)  // Triple Intelligence
+brain.get(id)
+brain.getNouns(filter?)
+brain.getVerbs(filter?)
+
+brain.delete(id)
+brain.deleteNouns(ids[])
+brain.deleteVerbs(ids[])
+brain.clear()
+
+// Simple shortcuts for common AI operations
+brain.similar(a, b)               // Simple similarity
+brain.clusters()                  // Simple clustering
+brain.related(id, limit?)         // Simple neighbors
+```
+
+#### **Neural AI Namespace** (`brain.neural.*`)
+```typescript
+// Advanced AI & Machine Learning operations
+brain.neural.similar(a, b, options?)           // Full similarity with options
+brain.neural.clusters(items?, options?)        // Advanced clustering
+brain.neural.neighbors(id, options?)           // K-nearest neighbors
+brain.neural.hierarchy(id, options?)           // Semantic hierarchy
+brain.neural.outliers(options?)                // Anomaly detection
+brain.neural.visualize(options?)               // Visualization data
+
+// Advanced clustering methods
+brain.neural.clusterByDomain(field, options?)  // Domain-aware clustering
+brain.neural.clusterByTime(field, windows, options?) // Temporal clustering
+brain.neural.clusterStream(options?)           // Streaming clustering
+brain.neural.updateClusters(items, options?)   // Incremental clustering
+
+// Utility & monitoring
+brain.neural.getPerformanceMetrics(operation?) // Performance stats
+brain.neural.clearCaches()                     // Cache management
+brain.neural.getCacheStats()                   // Cache statistics
+```
+
+#### **Triple Intelligence Namespace** (`brain.triple.*`)
+```typescript
+// Advanced natural language & complex queries
+brain.triple.find(query, options?)             // Natural language search
+brain.triple.analyze(text, options?)           // Text analysis
+brain.triple.understand(query, options?)       // Query understanding
+```
+
+#### **Augmentation System** (`brain.augmentations.*`)
+```typescript
+// Plugin/extension system
+brain.augmentations.add(augmentation)
+brain.augmentations.remove(name)
+brain.augmentations.get(name)
+brain.augmentations.list()
+brain.augmentations.execute(operation, params)
+```
+
+#### **Storage & System** (`brain.storage.*`)
+```typescript
+// Storage management
+brain.storage.backup(path?)
+brain.storage.restore(path?)
+brain.storage.getStatistics()
+brain.storage.optimize()
+brain.storage.vacuum()
+```
+
+### **🚀 API Flow & Developer Experience**
+
+#### **1. Beginner Flow (Simple & Intuitive)**
+```typescript
+// Dead simple - just works
+const brain = new Brainy()
+await brain.init()
+
+await brain.add('My first document', { nounType: NounType.Document })
+const results = await brain.search('document')
+const similar = await brain.similar('text1', 'text2')
+const groups = await brain.clusters()
+```
+
+#### **2. Intermediate Flow (More Control)**
+```typescript
+// Add configuration and options
+const brain = new Brainy({
+  storage: { path: './my-brainy-db' },
+  neural: { cacheSize: 5000 }
+})
+await brain.init()
+
+// Use options for better control
+const results = await brain.search('AI research', {
+  limit: 20,
+  threshold: 0.8,
+  filters: { type: 'Document', year: 2024 }
+})
+
+// Use neural namespace for advanced features
+const clusters = await brain.neural.clusters({
+  algorithm: 'hierarchical',
+  maxClusters: 10
+})
+```
+
+#### **3. Advanced Flow (Full Power)**
+```typescript
+// Complex natural language queries
+const insights = await brain.find(`
+  Show me documents about machine learning from 2024 
+  that are connected to research papers with high citations
+`)
+
+// Advanced temporal analysis
+const trends = await brain.neural.clusterByTime('publishedAt', [
+  { start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1 2024' },
+  { start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2 2024' }
+])
+
+// Real-time streaming clustering
+for await (const batch of brain.neural.clusterStream({ batchSize: 50 })) {
+  console.log(`Processed ${batch.progress.percentage}% - Found ${batch.clusters.length} clusters`)
+}
+```
+
+## 📊 **Parameter Consistency Analysis**
+
+### **✅ Excellent Consistency**
+
+#### **1. Data-First Pattern**
+```typescript
+// Always: (data, config/metadata, optional_params)
+brain.add(content, metadata)  // nounType now in metadata
+brain.relate(source, target, VerbType.RelatedTo, metadata?)
+brain.search(query, options?)
+brain.similar(a, b, options?)
+```
+
+#### **2. Options Objects**
+```typescript
+// Consistent options pattern across all methods
+{
+  limit?: number
+  threshold?: number  
+  filters?: Record<string, any>
+  algorithm?: string
+  includeMetadata?: boolean
+}
+```
+
+#### **3. Array Methods**
+```typescript
+// Pluralized versions always take arrays
+brain.addNouns([{ vectorOrData: '...', nounType: NounType.Content }])
+brain.addVerbs([{ source: '...', target: '...', type: VerbType.RelatedTo }])
+brain.deleteNouns(['id1', 'id2'])
+brain.deleteVerbs(['id1', 'id2'])
+```
+
+### **Return Type Consistency**
+
+#### **1. Simple Returns (Shortcuts)**
+```typescript
+brain.similar(a, b) → Promise<number>           // Always simple number
+brain.clusters() → Promise<SemanticCluster[]>   // Always simple array
+brain.related(id) → Promise<Neighbor[]>         // Always simple array
+```
+
+#### **2. Rich Returns (Neural Namespace)**
+```typescript
+brain.neural.similar(a, b, { detailed: true }) → Promise<SimilarityResult>
+brain.neural.neighbors(id, options) → Promise<NeighborsResult>
+brain.neural.clusters(options) → Promise<SemanticCluster[]>
+```
+
+#### **3. Consistent Error Handling**
+```typescript
+// All methods throw descriptive errors with context
+try {
+  await brain.neural.similar('invalid', 'data')  
+} catch (error) {
+  // error.code: 'SIMILARITY_ERROR'
+  // error.context: { inputA: '...', inputB: '...' }
+}
+```
+
+## 🎯 **Key Strengths of Current API**
+
+### **✅ 1. Progressive Disclosure**
+- **Simple**: `brain.similar()` → just returns a number
+- **Advanced**: `brain.neural.similar()` → full options & detailed results
+
+### **✅ 2. Intuitive Namespacing**
+- **Core data**: Direct on `brain` (addNoun, search, delete)
+- **AI features**: `brain.neural.*` (clustering, similarity, analysis)  
+- **System**: `brain.storage.*`, `brain.augmentations.*`
+
+### **✅ 3. Consistent Patterns**
+- **Always** `(data, options?)` parameter order
+- **Always** async/Promise-based
+- **Always** descriptive error messages with context
+
+### **✅ 4. Type Safety**
+```typescript
+// Excellent TypeScript support
+import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
+
+const brain = new Brainy()
+await brain.add('content', { nounType: NounType.Document, title: 'My Doc' })
+//                          ^^^^^^^^^^^^^^^^  // IDE autocomplete!
+```
+
+### **✅ 5. Flexible Configuration**
+```typescript
+// Zero-config (just works)
+const brain = new Brainy()
+
+// Full control when needed
+const brain = new Brainy({
+  storage: { 
+    adapter: 'file',
+    path: './my-data',
+    encryption: true
+  },
+  neural: {
+    cacheSize: 10000,
+    defaultAlgorithm: 'hierarchical'
+  },
+  logging: { verbose: true }
+})
+```
+
+## 🔍 **Minor Improvement Opportunities**
+
+### **1. Documentation Consistency**
+```typescript
+// ✅ GREAT: Clear, descriptive JSDoc
+/**
+ * Add semantic relationship between two items
+ * @param source - Source item ID
+ * @param target - Target item ID  
+ * @param type - Relationship type (VerbType enum)
+ * @param metadata - Optional relationship metadata
+ */
+brain.relate(source, target, type, metadata?)
+```
+
+### **2. Error Context Enhancement**
+```typescript
+// Current: Good error messages
+// Improvement: Add suggested fixes
+throw new SimilarityError('Failed to calculate similarity', {
+  inputA: 'invalid-id',
+  inputB: 'valid-id',
+  suggestion: 'Check that both IDs exist in the database'
+})
+```
+
+## 🎖️ **Overall API Grade: A+ (Excellent)**
+
+### **Strengths:**
+- **🎯 Intuitive**: Natural method names, clear hierarchy
+- **🔄 Consistent**: Same patterns everywhere  
+- **📈 Progressive**: Simple → advanced as needed
+- **🛡️ Type-safe**: Full TypeScript support
+- **📚 Well-documented**: Clear examples & guides
+- **🚀 Performant**: Smart caching, batching, streaming
+
+### **Neural API Fits Perfectly:**
+- **✅ Namespace consistency**: `brain.neural.*` is clear and logical
+- **✅ Parameter consistency**: Follows same `(data, options?)` pattern
+- **✅ Return consistency**: Rich objects when needed, simple types for shortcuts
+- **✅ Progressive disclosure**: `brain.similar()` → `brain.neural.similar()`
+- **✅ Advanced features**: Domain/temporal clustering, streaming, analysis
+
+### **Developer Experience Score: 🌟🌟🌟🌟🌟 (5/5 stars)**
+
+The API surface is **exceptionally well designed** with:
+- **Beginner-friendly** shortcuts that "just work"
+- **Advanced features** available when needed  
+- **Consistent patterns** across all methods
+- **Logical namespacing** that guides developers naturally
+- **Rich ecosystem** with augmentations, Triple Intelligence, and neural features
+
+**The neural namespace integrates seamlessly and enhances rather than complicates the overall API experience.**

docs/architecture/API_SURFACE_DESIGN.md

@@ -0,0 +1,200 @@
+# Brainy Neural API Surface Design
+
+## 🎯 **Clean API Hierarchy**
+
+### **Main Class Shortcuts (Simple & Common)**
+```typescript
+// High-level shortcuts on Brainy - most common operations
+brain.similar(a, b, options?)           // ✅ Keep - very common
+brain.clusters(items?, options?)        // ✅ Keep - very common  
+brain.related(id, options?)             // ✅ Keep - common (like neighbors but simpler name)
+
+// Remove/deprecate confusing shortcuts
+brain.visualize(options?)               // ❌ Remove - too specialized for main class
+```
+
+### **Neural Namespace (Full Featured)**
+```typescript
+// Core semantic operations
+brain.neural.similar(a, b, options?)           // Comprehensive similarity
+brain.neural.clusters(items?, options?)        // Smart clustering with auto-routing
+brain.neural.neighbors(id, options?)           // K-nearest neighbors (full featured)
+brain.neural.hierarchy(id, options?)           // Semantic hierarchy building
+brain.neural.outliers(options?)                // Anomaly detection
+brain.neural.visualize(options?)               // Visualization data generation
+
+// Advanced clustering methods
+brain.neural.clusterByDomain(field, options?)  // Domain-aware clustering
+brain.neural.clusterByTime(field, windows, options?) // Temporal clustering
+brain.neural.clusterStream(options?)           // Streaming/real-time clustering
+brain.neural.updateClusters(items, options?)   // Incremental clustering
+
+// Utility methods
+brain.neural.getPerformanceMetrics(operation?) // Performance monitoring
+brain.neural.clearCaches()                     // Cache management
+brain.neural.getCacheStats()                   // Cache statistics
+```
+
+## 🔒 **Private Methods (Internal Implementation)**
+
+### **Should NOT be exposed publicly:**
+```typescript
+// These are implementation details:
+brain.neural._clusterFast()              // ❌ Private - use clusters() with algorithm: 'hierarchical'
+brain.neural._clusterLarge()             // ❌ Private - use clusters() with algorithm: 'sample' 
+brain.neural._performHierarchicalClustering() // ❌ Private - internal routing
+brain.neural._performKMeansClustering()  // ❌ Private - internal routing
+brain.neural._performDBSCANClustering()  // ❌ Private - internal routing
+brain.neural._performSampledClustering() // ❌ Private - internal routing
+brain.neural._routeClusteringAlgorithm() // ❌ Private - internal routing
+brain.neural._similarityById()           // ❌ Private - internal routing
+brain.neural._similarityByVector()       // ❌ Private - internal routing
+brain.neural._similarityByText()         // ❌ Private - internal routing
+brain.neural._isId()                     // ❌ Private - utility
+brain.neural._isVector()                 // ❌ Private - utility
+brain.neural._convertToVector()          // ❌ Private - utility
+brain.neural._cacheResult()              // ❌ Private - caching
+brain.neural._trackPerformance()         // ❌ Private - monitoring
+```
+
+### **Current Issues in brain-cloud explorer:**
+```typescript
+// ❌ BAD: Accessing private implementation details
+brain.neural.clusterFast({ maxClusters: count, level: 2 })
+
+// ✅ GOOD: Use public API with proper options
+brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: count, level: 2 })
+```
+
+## 📊 **API Consistency Fixes**
+
+### **Method Naming Standardization:**
+```typescript
+// ✅ CONSISTENT: Pick one naming pattern and stick to it
+brain.neural.similar()      // Main method name
+brain.similar()             // Shortcut matches
+
+// ❌ INCONSISTENT: Don't mix these
+brain.neural.similarity()   // Different from shortcut
+brain.similar()
+```
+
+### **Parameter Patterns:**
+```typescript
+// ✅ CONSISTENT: Always (data, options?) pattern  
+brain.neural.similar(a, b, options?)
+brain.neural.clusters(items?, options?)
+brain.neural.neighbors(id, options?)
+brain.neural.hierarchy(id, options?)
+
+// Options should be objects with clear properties
+interface ClusteringOptions {
+  algorithm?: 'auto' | 'hierarchical' | 'kmeans' | 'dbscan'
+  maxClusters?: number
+  threshold?: number
+  // ...
+}
+```
+
+### **Return Type Consistency:**
+```typescript
+// ✅ CONSISTENT: All clustering methods return SemanticCluster[]
+brain.neural.clusters() → Promise<SemanticCluster[]>
+brain.neural.clusterByDomain() → Promise<DomainCluster[]>  // extends SemanticCluster
+brain.neural.clusterByTime() → Promise<TemporalCluster[]>   // extends SemanticCluster
+
+// ✅ CONSISTENT: All similarity methods return number or SimilarityResult
+brain.neural.similar() → Promise<number | SimilarityResult>
+brain.similar() → Promise<number>  // Shortcut always returns simple number
+```
+
+## 🚀 **Performance & Intelligence Routing**
+
+### **Auto-Algorithm Selection:**
+```typescript
+// Smart routing based on data size and characteristics
+brain.neural.clusters()  // Auto-selects:
+//   < 100 items   → hierarchical (fast, accurate)
+//   < 1000 items  → k-means (balanced)  
+//   > 1000 items  → sampling (scalable)
+
+// Manual override available
+brain.neural.clusters({ algorithm: 'hierarchical' })  // Force specific algorithm
+```
+
+### **Caching Strategy:**
+```typescript
+// Intelligent caching with LRU eviction
+brain.neural.similar('id1', 'id2')     // First call: compute & cache
+brain.neural.similar('id1', 'id2')     // Second call: instant cache hit
+
+// Cache management
+brain.neural.clearCaches()              // Manual cache clear
+brain.neural.getCacheStats()            // Monitor cache performance
+```
+
+## 📚 **Documentation Structure**
+
+### **Main Documentation Sections:**
+1. **Quick Start**: Simple examples using shortcuts (`brain.similar()`, `brain.clusters()`)
+2. **Neural API Guide**: Comprehensive examples using `brain.neural.*`  
+3. **Advanced Clustering**: Domain, temporal, streaming clustering
+4. **Performance**: Caching, algorithm selection, monitoring
+5. **API Reference**: Complete method documentation
+
+### **Example Progression:**
+```typescript
+// 1. BEGINNER: Simple shortcuts
+const similarity = await brain.similar('text1', 'text2')
+const clusters = await brain.clusters()
+
+// 2. INTERMEDIATE: Neural API with options
+const detailed = await brain.neural.similar('id1', 'id2', { detailed: true })
+const customClusters = await brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: 5 })
+
+// 3. ADVANCED: Specialized clustering
+const domainClusters = await brain.neural.clusterByDomain('category')
+const streamingClusters = brain.neural.clusterStream({ batchSize: 50 })
+```
+
+## ✅ **Implementation Checklist**
+
+### **High Priority:**
+- [x] Create comprehensive type definitions
+- [x] Implement improved NeuralAPI class with proper public/private separation
+- [ ] Update Brainy integration to use improved API
+- [ ] Fix brain-cloud explorer to use public APIs
+- [ ] Update test files to use consistent method names
+- [ ] Update documentation with new API structure
+
+### **Medium Priority:**
+- [ ] Implement placeholder algorithm implementations with real clustering logic
+- [ ] Add comprehensive error handling and validation
+- [ ] Add performance monitoring and metrics collection
+- [ ] Create migration guide for users of deprecated methods
+
+### **Nice to Have:**
+- [ ] Add interactive clustering refinement based on user feedback
+- [ ] Implement explainable clustering with reasoning
+- [ ] Add multi-modal clustering (text + metadata + relationships)
+- [ ] Create visualization examples for different graph libraries
+
+## 🎯 **API Surface Summary**
+
+### **✅ PUBLIC API** (What users should use):
+- **Main shortcuts**: `brain.similar()`, `brain.clusters()`, `brain.related()`
+- **Neural namespace**: `brain.neural.similar()`, `brain.neural.clusters()`, etc.
+- **Advanced features**: Domain clustering, temporal clustering, streaming
+- **Utilities**: Performance metrics, cache management
+
+### **❌ PRIVATE IMPLEMENTATION** (Internal only):
+- **Algorithm implementations**: `_performKMeansClustering()`, etc.
+- **Routing logic**: `_routeClusteringAlgorithm()`, etc.  
+- **Utility methods**: `_isId()`, `_convertToVector()`, etc.
+- **Caching internals**: `_cacheResult()`, `_trackPerformance()`, etc.
+
+### **⚠️ DEPRECATED** (Should be removed/hidden):
+- **brain.neural.clusterFast()** → Use `brain.neural.clusters({ algorithm: 'hierarchical' })`
+- **brain.neural.clusterLarge()** → Use `brain.neural.clusters({ algorithm: 'sample' })`
+- **brain.neural.similarity()** → Use `brain.neural.similar()` (pick one name)
+- **brain.visualize()** → Too specialized for main class, use `brain.neural.visualize()`

docs/architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md

@@ -0,0 +1,306 @@
+# 🧠 **Brainy Clustering Algorithms - Complete Analysis**
+
+## 🎯 **Current State & Capabilities**
+
+### **✅ Existing Infrastructure (Excellent Foundation)**
+
+#### **1. HNSW Hierarchical Clustering** 
+```typescript
+// ALREADY IMPLEMENTED & OPTIMIZED
+brain.neural.clusters({ algorithm: 'hierarchical', level: 2 })
+```
+
+**How it works:**
+- **Leverages HNSW natural hierarchy**: Uses existing index levels as natural cluster boundaries  
+- **O(n) performance**: Much faster than O(n²) traditional clustering
+- **Multi-level granularity**: Higher levels = fewer, broader clusters; Lower levels = more, specific clusters
+- **Representative sampling**: Uses HNSW level nodes as natural cluster centers
+
+**Performance characteristics:**
+- **Excellent for large datasets** (millions of items)
+- **Preserves semantic relationships** from vector space
+- **Automatic granularity control** via level parameter
+
+#### **2. Distance-Based Algorithms**
+```typescript
+// COMPREHENSIVE DISTANCE FUNCTIONS AVAILABLE
+euclideanDistance, cosineDistance, manhattanDistance, dotProductDistance
+```
+
+**Optimized implementations:**
+- **Batch processing**: `calculateDistancesBatch()` with parallelization
+- **Multiple metrics**: Choose optimal distance function per use case
+- **Performance optimized**: Faster than GPU for small vectors due to no transfer overhead
+
+#### **3. Rich Semantic Taxonomy**
+```typescript
+// 25+ NOUN TYPES & 35+ VERB TYPES
+NounType: Person, Organization, Document, Concept, Event, Media, etc.
+VerbType: RelatedTo, Contains, PartOf, Causes, CreatedBy, etc.
+```
+
+**Semantic clustering capabilities:**
+- **Type-based clustering**: Group by semantic categories
+- **Cross-type relationships**: Use verb types to find semantic bridges
+- **Hierarchical taxonomies**: Natural clustering within and across types
+
+#### **4. Graph Structure** 
+```typescript
+// VERB RELATIONSHIPS CREATE RICH GRAPH
+await brain.relate(sourceId, targetId, VerbType.Causes, { strength: 0.8 })
+```
+
+**Graph-based clustering potential:**
+- **Connected components**: Find strongly connected groups
+- **Community detection**: Use relationship strength for clustering
+- **Multi-modal clustering**: Combine graph + vector + taxonomy
+
+## 🚀 **Advanced Clustering Algorithms We Can Implement**
+
+### **1. ✅ Already Implemented: HNSW Hierarchical**
+
+```typescript
+// PRODUCTION READY - Uses existing HNSW levels
+const clusters = await brain.neural.clusters({
+  algorithm: 'hierarchical',
+  level: 2,  // Control granularity 
+  maxClusters: 15
+})
+```
+
+**Performance:** **A+** - O(n) leveraging existing index structure
+
+### **2. 🔥 Semantic Taxonomy Clustering** 
+
+```typescript
+// IMPLEMENT: Fast type-based clustering with cross-type bridges
+const clusters = await brain.neural.clusterByDomain('nounType', {
+  preserveTypeBoundaries: false,  // Allow cross-type clusters
+  bridgeStrength: 0.8,            // Minimum relationship strength for bridges
+  hybridWeighting: {
+    taxonomy: 0.4,  // 40% weight to type similarity
+    vector: 0.4,    // 40% weight to vector similarity  
+    graph: 0.2      // 20% weight to relationship strength
+  }
+})
+```
+
+**Algorithm approach:**
+1. **Primary clustering by taxonomy**: Group by NounType/VerbType first
+2. **Vector refinement**: Sub-cluster within types using vector similarity
+3. **Cross-type bridging**: Find relationships that bridge type boundaries
+4. **Weighted fusion**: Combine taxonomy + vector + graph signals
+
+**Performance:** **A+** - O(n log n) - taxonomy grouping is O(n), refinement is HNSW-accelerated
+
+### **3. 🔥 Graph Community Detection**
+
+```typescript
+// IMPLEMENT: Relationship-based clustering
+const clusters = await brain.neural.clusterByConnections({
+  algorithm: 'modularity',  // or 'louvain', 'leiden'
+  minCommunitySize: 3,
+  relationshipWeights: {
+    [VerbType.Creates]: 1.0,
+    [VerbType.PartOf]: 0.8,
+    [VerbType.RelatedTo]: 0.5
+  }
+})
+```
+
+**Algorithm approach:**
+1. **Build weighted graph**: Use verbs as edges, weights from relationship types + metadata
+2. **Community detection**: Apply Louvain or Leiden algorithm for modularity optimization
+3. **Semantic enhancement**: Use vector similarity to refine community boundaries
+
+**Performance:** **A** - O(n log n) for sparse graphs, handles millions of relationships efficiently
+
+### **4. 🔥 Multi-Modal Fusion Clustering**
+
+```typescript
+// IMPLEMENT: Best of all worlds
+const clusters = await brain.neural.clusters({
+  algorithm: 'multimodal',
+  signals: {
+    vector: { weight: 0.5, metric: 'cosine' },
+    graph: { weight: 0.3, algorithm: 'modularity' },
+    taxonomy: { weight: 0.2, crossTypeThreshold: 0.8 }
+  },
+  fusion: 'weighted_ensemble'  // or 'consensus', 'hierarchical'
+})
+```
+
+**Algorithm approach:**
+1. **Independent clustering**: Run HNSW, graph, and taxonomy clustering separately
+2. **Consensus building**: Find agreement between different clustering results  
+3. **Conflict resolution**: Use weighted voting or hierarchical merging for disagreements
+4. **Quality optimization**: Iteratively refine based on silhouette scores
+
+**Performance:** **A** - O(n log n) - parallel execution of component algorithms
+
+### **5. 💎 Temporal Pattern Clustering**
+
+```typescript
+// IMPLEMENT: Time-aware clustering using existing infrastructure
+const clusters = await brain.neural.clusterByTime('createdAt', [
+  { start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1 2024' },
+  { start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2 2024' }
+], {
+  evolution: 'track',  // Track how clusters evolve over time
+  stability: 0.7,      // Minimum stability threshold
+  trendAnalysis: true  // Include trend detection
+})
+```
+
+**Algorithm approach:**
+1. **Time window clustering**: Apply HNSW clustering within each time window
+2. **Cluster evolution tracking**: Match clusters across time windows using vector similarity
+3. **Trend analysis**: Detect growing, shrinking, merging, splitting patterns
+4. **Stability scoring**: Measure cluster consistency over time
+
+**Performance:** **A+** - O(k*n log n) where k = number of time windows
+
+### **6. 💎 DBSCAN with Adaptive Parameters**
+
+```typescript
+// IMPLEMENT: Density-based clustering with smart parameter selection
+const clusters = await brain.neural.clusters({
+  algorithm: 'dbscan',
+  autoParams: true,  // Automatically select eps and minPts
+  distanceMetric: 'cosine',
+  outlierHandling: 'soft'  // Soft assignment instead of hard outliers
+})
+```
+
+**Algorithm approach:**
+1. **Adaptive parameter selection**: Use HNSW k-NN distances to estimate optimal eps
+2. **Multi-scale analysis**: Run DBSCAN at multiple scales and merge results
+3. **Soft outlier assignment**: Assign outliers to nearest clusters with confidence scores
+
+**Performance:** **A** - O(n log n) using HNSW for neighbor queries
+
+## 📊 **Performance Comparison Matrix**
+
+| Algorithm | Time Complexity | Space | Large Scale | Semantic Quality | Graph Aware |
+|-----------|----------------|-------|-------------|------------------|-------------|
+| **HNSW Hierarchical** | O(n) | O(n) | ✅ Excellent | ✅ Very Good | ❌ No |
+| **Taxonomy Fusion** | O(n log n) | O(n) | ✅ Excellent | 🔥 Exceptional | ⚡ Partial |
+| **Graph Communities** | O(n log n) | O(e) | ✅ Very Good | ✅ Very Good | 🔥 Exceptional |
+| **Multi-Modal** | O(n log n) | O(n) | ✅ Very Good | 🔥 Exceptional | 🔥 Exceptional |
+| **Temporal Patterns** | O(k*n log n) | O(n) | ⚡ Good | ✅ Very Good | ⚡ Partial |
+| **Adaptive DBSCAN** | O(n log n) | O(n) | ✅ Very Good | ✅ Very Good | ❌ No |
+
+## 🎯 **Specific Improvements Using Existing Capabilities**
+
+### **1. Enhanced HNSW Clustering (Easy Win)**
+
+```typescript
+// IMPROVE EXISTING: Add semantic post-processing
+private async enhanceHNSWClusters(clusters: SemanticCluster[]): Promise<SemanticCluster[]> {
+  return Promise.all(clusters.map(async cluster => {
+    // Get actual metadata for cluster members
+    const members = await this.brain.getNouns(cluster.members.map(id => ({ id })))
+    
+    // Analyze semantic characteristics
+    const semanticProfile = this.analyzeSemanticProfile(members)
+    
+    // Generate meaningful cluster labels
+    const label = await this.generateClusterLabel(members, semanticProfile)
+    
+    // Calculate cluster coherence using multiple signals
+    const coherence = this.calculateMultiModalCoherence(members)
+    
+    return {
+      ...cluster,
+      label,
+      semanticProfile,
+      coherence,
+      quality: coherence.overall
+    }
+  }))
+}
+```
+
+### **2. Intelligent Algorithm Selection**
+
+```typescript
+// IMPLEMENT: Smart routing based on data characteristics
+private selectOptimalAlgorithm(dataCharacteristics: {
+  size: number,
+  dimensionality: number,
+  graphDensity: number,
+  typeDistribution: Record<string, number>
+}): string {
+  if (dataCharacteristics.size > 100000) {
+    return 'hierarchical'  // HNSW scales best
+  }
+  
+  if (dataCharacteristics.graphDensity > 0.1) {
+    return 'multimodal'  // Rich graph structure
+  }
+  
+  if (Object.keys(dataCharacteristics.typeDistribution).length > 10) {
+    return 'taxonomy'  // Diverse semantic types
+  }
+  
+  return 'hierarchical'  // Safe default
+}
+```
+
+### **3. Streaming Cluster Updates**
+
+```typescript
+// IMPLEMENT: Incremental clustering using existing infrastructure
+public async updateClusters(newItems: string[]): Promise<SemanticCluster[]> {
+  // Use HNSW nearest neighbor for fast cluster assignment
+  const assignments = await Promise.all(
+    newItems.map(async itemId => {
+      const neighbors = await this.brain.neural.neighbors(itemId, { limit: 5 })
+      return this.assignToNearestCluster(itemId, neighbors, this.existingClusters)
+    })
+  )
+  
+  // Incrementally update cluster centroids and boundaries
+  return this.updateClusterBoundaries(assignments)
+}
+```
+
+## 🏆 **Recommended Implementation Priority**
+
+### **🔥 Phase 1: High Impact, Easy Implementation**
+1. **Enhanced HNSW Clustering**: Add semantic post-processing to existing algorithm
+2. **Taxonomy-Aware Clustering**: Leverage existing NounType/VerbType enums
+3. **Intelligent Algorithm Selection**: Route based on data characteristics
+
+### **⚡ Phase 2: Advanced Features**
+4. **Graph Community Detection**: Use existing verb relationships
+5. **Multi-Modal Fusion**: Combine all signals intelligently
+6. **Streaming Updates**: Incremental cluster maintenance
+
+### **💎 Phase 3: Cutting Edge**
+7. **Temporal Pattern Analysis**: Track cluster evolution over time
+8. **Adaptive DBSCAN**: Dynamic parameter selection
+9. **Explainable Clustering**: Generate cluster explanations and reasoning
+
+## 🎯 **Key Advantages of Our Approach**
+
+### **✅ Leverages Existing Infrastructure**
+- **HNSW index**: Already optimized for large-scale vector operations
+- **Distance functions**: Battle-tested and performance-optimized  
+- **Semantic taxonomy**: Rich type system with 60+ semantic categories
+- **Graph structure**: Relationship network from verb connections
+
+### **✅ Multiple Clustering Paradigms**
+- **Vector similarity**: Traditional embedding-based clustering
+- **Graph structure**: Relationship-based community detection
+- **Semantic taxonomy**: Type-aware intelligent grouping
+- **Temporal patterns**: Time-aware cluster evolution
+- **Multi-modal fusion**: Best of all worlds
+
+### **✅ Scalability & Performance**
+- **O(n) hierarchical clustering**: Leveraging HNSW levels
+- **Parallel processing**: Batch distance calculations optimized
+- **Streaming support**: Real-time cluster updates
+- **Memory efficient**: Existing index structures reused
+
+**Our clustering algorithms are not just competitive - they're architecturally superior by leveraging Brainy's unique multi-modal semantic infrastructure.**

docs/architecture/METADATA_ARCHITECTURE.md

@@ -0,0 +1,220 @@
+# Brainy Metadata Architecture & Namespacing
+
+## The Problem 🚨
+We're mixing internal Brainy fields with user metadata, causing:
+1. **Namespace collisions** - User's `deleted` field conflicts with our soft-delete
+2. **API confusion** - Users see internal fields they shouldn't care about
+3. **Security issues** - Users could manipulate internal fields
+4. **Augmentation conflicts** - 3rd party augmentations might overwrite our fields
+
+## Current Internal Fields Being Added
+
+### Core System Fields
+```javascript
+metadata = {
+  // USER DATA
+  name: "Django",
+  type: "framework",
+  
+  // OUR INTERNAL FIELDS - COLLISION RISK!
+  deleted: false,          // Soft delete status
+  domain: "tech",          // Distributed mode domain
+  domainMetadata: {},      // Domain-specific metadata
+  partition: 0,            // Partition for sharding
+  createdAt: {...},        // GraphNoun timestamp
+  updatedAt: {...},        // GraphNoun timestamp
+  createdBy: {...},        // Who created this
+  noun: "Concept",         // NounType
+  verb: "RELATES_TO",     // VerbType (for relationships)
+  isPlaceholder: true,     // Write-only mode marker
+  autoCreated: true,       // Auto-created noun marker
+  writeOnlyMode: true      // High-speed streaming marker
+}
+```
+
+### Augmentation Fields
+```javascript
+// Good - neuralImport already uses underscore prefix!
+metadata._neuralProcessed = true
+metadata._neuralConfidence = 0.95
+metadata._detectedEntities = 5
+metadata._detectedRelationships = 3
+metadata._neuralInsights = [...]
+
+// Bad - direct modification
+metadata.importance = 0.8  // IntelligentVerbScoring
+```
+
+## Proposed Solution: Three-Tier Metadata
+
+### 1. User Metadata (Public)
+```javascript
+metadata = {
+  // User's fields - completely untouched
+  name: "Django",
+  type: "framework",
+  deleted: "2024-01-01",  // User's own deleted field - no conflict!
+  domain: "web",          // User's domain field - no conflict!
+}
+```
+
+### 2. Internal Metadata (Protected)
+```javascript
+metadata._brainy = {
+  // Core system fields - O(1) indexed
+  deleted: false,         // Our soft delete flag
+  version: 2,             // Metadata schema version
+  
+  // Distributed mode
+  partition: 0,
+  distributedDomain: "tech",
+  
+  // GraphNoun compliance
+  nounType: "Concept",
+  verbType: "RELATES_TO",
+  createdAt: 1704067200000,
+  updatedAt: 1704067200000,
+  createdBy: "user:123",
+  
+  // Performance flags
+  indexed: true,
+  searchable: true,
+  placeholder: false,
+  
+  // Storage optimization
+  compressed: false,
+  encrypted: false
+}
+```
+
+### 3. Augmentation Metadata (Semi-Protected)
+```javascript
+metadata._augmentations = {
+  // Each augmentation gets its own namespace
+  neuralImport: {
+    processed: true,
+    confidence: 0.95,
+    entities: 5,
+    relationships: 3
+  },
+  
+  verbScoring: {
+    contextScore: 0.7,
+    importance: 0.8
+  },
+  
+  // 3rd party augmentations
+  customAug: {
+    // Their fields isolated here
+  }
+}
+```
+
+## Implementation Strategy
+
+### Phase 1: Core Fields ✅
+```javascript
+// Already done with _brainy.deleted
+const BRAINY_NAMESPACE = '_brainy'
+const AUGMENTATION_NAMESPACE = '_augmentations'
+```
+
+### Phase 2: Migrate All Internal Fields
+```javascript
+// Before
+metadata.domain = "tech"
+metadata.partition = 0
+
+// After
+metadata._brainy.distributedDomain = "tech"
+metadata._brainy.partition = 0
+```
+
+### Phase 3: Augmentation API
+```javascript
+class Augmentation {
+  // Read user metadata (read-only)
+  getUserMetadata(metadata) {
+    const { _brainy, _augmentations, ...userMeta } = metadata
+    return userMeta // Clean user data only
+  }
+  
+  // Write augmentation data (isolated)
+  setAugmentationData(metadata, augName, data) {
+    if (!metadata._augmentations) metadata._augmentations = {}
+    metadata._augmentations[augName] = data
+  }
+  
+  // Read internal fields (for special augmentations only)
+  getInternalField(metadata, field) {
+    return metadata._brainy?.[field]
+  }
+}
+```
+
+## Benefits
+
+1. **No Collisions** - User can have any field names
+2. **O(1) Performance** - Internal fields still indexed
+3. **Clean API** - Users only see their data
+4. **Secure** - Internal fields protected
+5. **Extensible** - Augmentations isolated
+6. **Backward Compatible** - Migration path available
+
+## Query Impact
+
+### Before (Collision Risk)
+```javascript
+where: { 
+  deleted: false,  // Ambiguous - ours or user's?
+  type: "framework"
+}
+```
+
+### After (Clear Separation)
+```javascript
+where: {
+  '_brainy.deleted': false,  // Our soft delete
+  type: "framework"          // User's field
+}
+```
+
+## Performance Considerations
+
+- **Index on `_brainy.deleted`**: O(1) hash lookup ✅
+- **Index on `_brainy.partition`**: O(1) for sharding ✅
+- **Nested field access**: Modern DBs handle this efficiently ✅
+- **Storage overhead**: ~100 bytes per item (acceptable) ✅
+
+## Migration Path
+
+1. **New items**: Automatically use namespaced fields
+2. **Existing items**: Lazy migration on update
+3. **Queries**: Support both formats temporarily
+4. **Deprecation**: Remove old format in v3.0
+
+## Augmentation Guidelines
+
+### For Core Augmentations
+- Use `_brainy.*` for system fields
+- Use `_augmentations.{name}.*` for augmentation data
+- Never modify user fields directly
+
+### For 3rd Party Augmentations
+- Read user metadata via `getUserMetadata()`
+- Write only to `_augmentations.{yourName}.*`
+- Request permission for internal field access
+
+## Critical Fields to Namespace
+
+| Field | Current Location | New Location | Priority |
+|-------|-----------------|--------------|----------|
+| deleted | metadata.deleted | metadata._brainy.deleted | HIGH ✅ |
+| partition | metadata.partition | metadata._brainy.partition | HIGH |
+| domain | metadata.domain | metadata._brainy.distributedDomain | HIGH |
+| createdAt | metadata.createdAt | metadata._brainy.createdAt | MEDIUM |
+| updatedAt | metadata.updatedAt | metadata._brainy.updatedAt | MEDIUM |
+| noun | metadata.noun | metadata._brainy.nounType | MEDIUM |
+| verb | metadata.verb | metadata._brainy.verbType | MEDIUM |
+| isPlaceholder | metadata.isPlaceholder | metadata._brainy.placeholder | LOW |
+| autoCreated | metadata.autoCreated | metadata._brainy.autoCreated | LOW |

docs/architecture/aggregation.md

@@ -1,242 +0,0 @@
-# Aggregation Architecture
-
-> Write-time incremental aggregation with O(1) reads
-
-## Design Principles
-
-1. **Write-time computation** — aggregates update on every `add()`, `update()`, and `delete()`, not as batch jobs
-2. **Incremental state** — running totals maintained per group, never rescanning the dataset
-3. **Provider interface** — TypeScript engine is the default; plugins can replace it with native implementations
-4. **Zero-allocation reads** — query results are computed from pre-aggregated state
-
-## Component Overview
-
-```
-┌──────────────────────────────────────────────────────────┐
-│                         Brainy                           │
-│                                                          │
-│  add() / update() / delete()                             │
-│          │                                               │
-│          ▼                                               │
-│  ┌──────────────────────┐    ┌────────────────────────┐  │
-│  │  AggregationIndex    │    │  AggregateMaterializer  │  │
-│  │                      │───▶│  (debounced writes)     │  │
-│  │  ├─ definitions Map  │    └────────────────────────┘  │
-│  │  ├─ states Map       │                                │
-│  │  └─ staleMinMax Set  │    ┌────────────────────────┐  │
-│  │                      │    │  timeWindows.ts         │  │
-│  │  Source filter ──────│───▶│  bucketTimestamp()      │  │
-│  │  Group key ──────────│───▶│  parseBucketRange()     │  │
-│  └──────────┬───────────┘    └────────────────────────┘  │
-│             │                                            │
-│             │ provider interface                          │
-│             ▼                                            │
-│  ┌──────────────────────┐                                │
-│  │  AggregationProvider │  (optional, registered by      │
-│  │  ├─ incrementalUpdate│   plugin like @soulcraft/cor)   │
-│  │  ├─ rebuildAggregate │                                │
-│  │  ├─ queryAggregate   │                                │
-│  │  └─ serialize/restore│                                │
-│  └──────────────────────┘                                │
-└──────────────────────────────────────────────────────────┘
-```
-
-## State Management
-
-### Definitions
-
-Registered via `brain.defineAggregate(def)`. Stored in a `Map<string, AggregateDefinition>` keyed by aggregate name. Persisted to storage under `__aggregation_definitions__` on flush.
-
-### Group State
-
-Each aggregate maintains a `Map<string, AggregateGroupState>` where keys are serialized group key values (e.g., `category=food|date=2024-01`). Each group holds per-metric `MetricState`:
-
-```typescript
-interface MetricState {
-  sum: number    // Running total
-  count: number  // Entity count
-  min: number    // Minimum (Infinity if empty)
-  max: number    // Maximum (-Infinity if empty)
-  m2?: number    // Welford's M2 for stddev/variance
-}
-```
-
-### Change Detection
-
-On restart, definition hashes (FNV-1a 32-bit) are compared with the persisted hash. If a definition changed (different groupBy, metrics, or source), the aggregate state is reset and must be rebuilt.
-
-## Write-Time Update Flow
-
-When `brain.add(entity)` is called:
-
-```
-1. For each registered aggregate:
-   ├─ Source filter check (type, service, where)
-   │  └─ Skip if entity doesn't match
-   ├─ Aggregate entity check
-   │  └─ Skip if entity.service === 'brainy:aggregation'
-   │     or entity.metadata.__aggregate is set
-   ├─ Group key computation
-   │  └─ Extract groupBy fields from metadata
-   │     Apply time bucketing for windowed dimensions
-   └─ Metric update
-      └─ For each metric in the definition:
-         ├─ count: increment count
-         ├─ sum/avg: add value to sum, increment count
-         ├─ min/max: compare and update
-         └─ stddev/variance: Welford's online update
-```
-
-### Update Handling
-
-On `brain.update(entity)`, the engine reverses the old entity's contribution and applies the new entity's contribution. This correctly handles:
-
-- **Value changes**: old amount=10, new amount=20 — sum adjusts by +10
-- **Group key changes**: entity moves from category "food" to "drink" — both groups update
-- **Source filter changes**: entity type changes from Event to Document — removed from matching aggregates
-
-### Delete Handling
-
-On `brain.remove(id)`, the engine reverses the entity's contribution:
-
-- `count` and `sum` are decremented
-- `min`/`max` may become stale (marked in `staleMinMax` for lazy recompute)
-- Welford's M2 is updated with the inverse formula
-- Empty groups (all metric counts at zero) are removed
-
-## Algorithms
-
-### Welford's Online Algorithm
-
-Standard deviation and variance use Welford's numerically stable online algorithm with M2 tracking. This computes incrementally without storing individual values:
-
-```
-On add(x):
-  count += 1
-  oldMean = (sum - x) / (count - 1)    // mean before this value
-  sum += x
-  mean = sum / count                     // mean after this value
-  M2 += (x - oldMean) * (x - mean)
-
-On remove(x):
-  oldMean = sum / count
-  sum -= x
-  count -= 1
-  newMean = sum / count
-  M2 = max(0, M2 - (x - oldMean) * (x - newMean))
-
-Sample variance = M2 / (count - 1)
-Sample stddev = sqrt(variance)
-```
-
-M2 is clamped to zero on remove to prevent floating-point drift from producing negative values.
-
-### MIN/MAX Handling
-
-The TypeScript engine uses simple comparison for add operations and marks MIN/MAX as potentially stale on delete (since removing the current min/max value requires a rescan). Stale values are lazily recomputed on the next query.
-
-The Cor native engine uses a `BTreeMap<OrderedFloat<f64>, u64>` that tracks the exact frequency of every value, providing precise MIN/MAX after any sequence of operations without rescanning.
-
-### Time Window Bucketing
-
-Timestamps (Unix milliseconds) are bucketed using UTC-based formatting:
-
-| Granularity | Bucket Key | Algorithm |
-|------------|-----------|-----------|
-| `hour` | `2024-01-15T14` | UTC year-month-day-hour |
-| `day` | `2024-01-15` | UTC year-month-day |
-| `week` | `2024-W03` | ISO 8601 week (Monday start, week 1 contains first Thursday) |
-| `month` | `2024-01` | UTC year-month |
-| `quarter` | `2024-Q1` | `ceil((month) / 3)` |
-| `year` | `2024` | UTC year |
-| `{ seconds: N }` | ISO timestamp | `floor(timestamp / interval) * interval` |
-
-Bucket keys can be parsed back into `{ start, end }` timestamp ranges via `parseBucketRange()`.
-
-## Provider Interface
-
-The `AggregationProvider` interface defines the contract between Brainy's `AggregationIndex` and plugin-provided native implementations:
-
-```typescript
-interface AggregationProvider {
-  defineAggregate?(def: AggregateDefinition): void
-  removeAggregate?(name: string): void
-
-  incrementalUpdate(
-    name: string,
-    def: AggregateDefinition,
-    entity: Record<string, unknown>,
-    op: 'add' | 'update' | 'delete',
-    prev?: Record<string, unknown>
-  ): AggregateGroupState[]
-
-  computeGroupKey(
-    entity: Record<string, unknown>,
-    groupBy: GroupByDimension[]
-  ): Record<string, string | number>
-
-  rebuildAggregate(
-    def: AggregateDefinition,
-    entities: Array<Record<string, unknown>>
-  ): Map<string, AggregateGroupState>
-
-  queryAggregate(
-    state: Map<string, AggregateGroupState>,
-    params: AggregateQueryParams
-  ): AggregateResult[]
-
-  restoreState?(data: string): void
-  serializeState?(): string
-}
-```
-
-When a native provider is registered:
-
-1. `AggregationIndex` delegates `incrementalUpdate()` to the provider instead of running TypeScript logic
-2. Provider returns updated `AggregateGroupState[]` which are applied back into the state maps
-3. Query execution is delegated via `queryAggregate()`
-4. State serialization is delegated via `serializeState()`/`restoreState()`
-
-Brainy retains ownership of the state maps and persistence. The provider handles computation.
-
-## Materialization
-
-The `AggregateMaterializer` converts aggregate group states into `NounType.Measurement` entities:
-
-1. When an aggregate group is updated and `materialize` is enabled, `scheduleMaterialize()` is called
-2. Materialization is debounced (default: 1000ms) to batch rapid updates during ingestion
-3. On trigger, the materializer either creates or updates a `NounType.Measurement` entity
-4. Materialized entities include `service: 'brainy:aggregation'` and `metadata.__aggregate` to prevent infinite loops
-
-Materialized entities are automatically visible through:
-- OData endpoints
-- Google Sheets integration
-- Server-Sent Events (SSE)
-- Webhook notifications
-
-## Persistence
-
-### Storage Keys
-
-| Key | Content |
-|-----|---------|
-| `__aggregation_definitions__` | Array of all definitions with FNV-1a hashes |
-| `__aggregation_state_{name}__` | Per-aggregate group states (array of `AggregateGroupState`) |
-| `__aggregation_native_state__` | Serialized native provider state (JSON string) |
-
-### Lifecycle
-
-1. **`init()`** — Load definitions, compare hashes, load matching state, restore native provider state
-2. **Write operations** — Mark modified aggregates as dirty
-3. **`flush()`** — Persist all dirty aggregate states and native provider state
-4. **`close()`** — Flush and release resources
-
-## Source Files
-
-| File | Purpose |
-|------|---------|
-| `src/aggregation/AggregationIndex.ts` | Core engine: definitions, state, write hooks, query |
-| `src/aggregation/materializer.ts` | Debounced materialization of results as entities |
-| `src/aggregation/timeWindows.ts` | Time bucketing and bucket range parsing |
-| `src/aggregation/index.ts` | Module exports |
-| `src/types/brainy.types.ts` | Type definitions for all aggregation interfaces |

docs/architecture/augmentation-system-audit.md

@@ -0,0 +1,359 @@
+# 🔍 Brainy 2.0 Augmentation System Architecture Audit (REVISED)
+
+**Author**: Senior Architecture Review  
+**Date**: 2025-08-25  
+**Status**: 🟡 **WORKING BUT NEEDS MARKETPLACE FEATURES**
+
+## Executive Summary
+
+The augmentation system core execution is WORKING correctly through `AugmentationRegistry`. The system properly executes augmentations before/after operations. However, there's no discovery, installation, or marketplace integration for the brain-cloud registry vision.
+
+---
+
+## 🟢 What's Actually Working
+
+### 1. Execution Mechanism ✅
+The `AugmentationRegistry` class properly implements:
+```typescript
+async execute<T>(operation: string, params: any, mainOperation: () => Promise<T>): Promise<T>
+```
+- Chains augmentations correctly
+- Respects timing (before/after/around)
+- Handles operation filtering
+- Works with all 27 augmentations
+
+### 2. Registration System ✅
+```typescript
+brain.augmentations.register(augmentation)
+```
+- Two-phase initialization works (storage first)
+- Context injection works
+- Lifecycle management works
+
+### 3. Clean Interface ✅
+- 100% of augmentations use `BrainyAugmentation`
+- `BaseAugmentation` provides solid foundation
+- Proper TypeScript types
+
+### 4. Auto-Configuration ✅
+```typescript
+new Brainy({ 
+  cache: true,    // Auto-registers CacheAugmentation
+  index: true,    // Auto-registers IndexAugmentation
+  storage: 's3'   // Auto-registers S3StorageAugmentation
+})
+```
+
+---
+
+## 🟡 Missing for Marketplace Vision
+
+### 1. No Package Discovery
+**Current**: Manual registration only
+```typescript
+// Current (manual)
+import { NotionSynapse } from './my-custom-synapse'
+brain.augmentations.register(new NotionSynapse())
+
+// Needed
+await brain.discover('notion')  // Search npm/brain-cloud
+await brain.install('@soulcraft/notion-synapse')
+```
+
+### 2. No Installation Mechanism
+**Current**: Must be bundled at build time
+**Needed**: Dynamic installation
+```typescript
+interface AugmentationMarketplace {
+  search(query: string): Promise<Package[]>
+  install(packageId: string): Promise<void>
+  uninstall(packageId: string): Promise<void>
+  listInstalled(): Promise<Package[]>
+  checkUpdates(): Promise<Update[]>
+}
+```
+
+### 3. No Brain Cloud Registry Client
+**Current**: No registry concept
+**Needed**: Registry integration
+```typescript
+class BrainCloudRegistry {
+  private apiUrl = 'https://api.soulcraft.com/brain-cloud'
+  
+  async search(query: string): Promise<AugmentationPackage[]> {
+    const response = await fetch(`${this.apiUrl}/augmentations/search?q=${query}`)
+    return response.json()
+  }
+  
+  async getPackage(id: string): Promise<AugmentationPackage> {
+    const response = await fetch(`${this.apiUrl}/augmentations/${id}`)
+    return response.json()
+  }
+}
+```
+
+### 4. No License Management
+**Current**: All augmentations free/bundled
+**Needed**: License verification
+```typescript
+interface LicenseManager {
+  verify(packageId: string, licenseKey: string): Promise<boolean>
+  activate(packageId: string, licenseKey: string): Promise<void>
+  deactivate(packageId: string): Promise<void>
+  getStatus(packageId: string): Promise<LicenseStatus>
+}
+```
+
+### 5. No Version Management
+**Current**: No versioning
+**Needed**: Semver support
+```typescript
+interface VersionManager {
+  checkCompatibility(pkg: Package, brainyVersion: string): boolean
+  resolveConflicts(packages: Package[]): Package[]
+  upgrade(packageId: string, toVersion: string): Promise<void>
+}
+```
+
+---
+
+## 📋 Implementation Plan for Marketplace
+
+### Phase 1: Local Package Discovery (1 week)
+```typescript
+class LocalPackageDiscovery {
+  async discover(): Promise<Package[]> {
+    // 1. Search node_modules for brainy augmentations
+    const packages = await glob('node_modules/@*/package.json')
+    
+    // 2. Filter for brainy augmentations
+    return packages.filter(pkg => pkg.brainy?.type === 'augmentation')
+  }
+  
+  async load(packageId: string): Promise<BrainyAugmentation> {
+    // Dynamic import
+    const module = await import(packageId)
+    return new module.default()
+  }
+}
+```
+
+### Phase 2: NPM Integration (1 week)
+```typescript
+class NPMRegistry {
+  async search(query: string): Promise<Package[]> {
+    // Search npm for packages with brainy keyword
+    const response = await fetch(
+      `https://registry.npmjs.org/-/v1/search?text=${query}+keywords:brainy-augmentation`
+    )
+    return response.json()
+  }
+  
+  async install(packageId: string): Promise<void> {
+    // Use npm programmatically
+    await exec(`npm install ${packageId}`)
+    
+    // Auto-register after install
+    const aug = await this.load(packageId)
+    this.brain.augmentations.register(aug)
+  }
+}
+```
+
+### Phase 3: Brain Cloud Registry (2 weeks)
+```typescript
+class BrainCloudMarketplace {
+  private registry = new BrainCloudRegistry()
+  private licenses = new LicenseManager()
+  private installer = new AugmentationInstaller()
+  
+  async browse(category?: string): Promise<MarketplaceListing[]> {
+    const packages = await this.registry.list(category)
+    
+    return packages.map(pkg => ({
+      ...pkg,
+      installed: this.isInstalled(pkg.id),
+      licensed: this.isLicensed(pkg.id),
+      updates: this.hasUpdates(pkg.id)
+    }))
+  }
+  
+  async purchase(packageId: string): Promise<void> {
+    // 1. Process payment
+    const license = await this.processPayment(packageId)
+    
+    // 2. Activate license
+    await this.licenses.activate(packageId, license)
+    
+    // 3. Install package
+    await this.install(packageId)
+  }
+}
+```
+
+### Phase 4: Developer Tools (1 week)
+```typescript
+// CLI for augmentation development
+class AugmentationCLI {
+  async create(name: string): Promise<void> {
+    // Scaffold new augmentation project
+    await this.scaffold(name, 'augmentation-template')
+  }
+  
+  async test(path: string): Promise<void> {
+    // Test augmentation locally
+    const aug = await this.load(path)
+    await this.runTests(aug)
+  }
+  
+  async publish(path: string): Promise<void> {
+    // Publish to brain-cloud
+    const pkg = await this.package(path)
+    await this.registry.publish(pkg)
+  }
+}
+```
+
+---
+
+## 🏗️ Recommended Architecture
+
+### 1. Augmentation Package Structure
+```json
+{
+  "name": "@soulcraft/notion-synapse",
+  "version": "1.0.0",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "brainy": {
+    "type": "augmentation",
+    "class": "NotionSynapse",
+    "timing": "after",
+    "operations": ["addNoun", "updateNoun"],
+    "priority": 20,
+    "license": "premium",
+    "price": 9.99,
+    "compatibility": ">=2.0.0",
+    "dependencies": []
+  },
+  "keywords": ["brainy-augmentation", "notion", "sync"]
+}
+```
+
+### 2. Installation Flow
+```typescript
+// User flow
+await brain.marketplace.search('notion')
+// Returns: [@soulcraft/notion-synapse, @community/notion-sync, ...]
+
+await brain.marketplace.install('@soulcraft/notion-synapse')
+// 1. Check license (prompt for purchase if needed)
+// 2. Check compatibility
+// 3. Install dependencies
+// 4. Download package
+// 5. Load augmentation
+// 6. Register with brain
+// 7. Initialize
+
+// Now it's working!
+brain.augmentations.list()
+// [..., { name: '@soulcraft/notion-synapse', enabled: true }]
+```
+
+### 3. Discovery UI
+```typescript
+// Web UI component
+<AugmentationMarketplace>
+  <SearchBar />
+  <Categories>
+    <Category name="Storage" count={12} />
+    <Category name="Sync" count={8} />
+    <Category name="AI" count={15} />
+  </Categories>
+  <Results>
+    <AugmentationCard
+      name="Notion Synapse"
+      author="Soulcraft"
+      rating={4.8}
+      installs={1200}
+      price={9.99}
+      onInstall={...}
+    />
+  </Results>
+</AugmentationMarketplace>
+```
+
+---
+
+## 🎯 Priority for 2.0 Release
+
+### Must Have (Release Blockers)
+- ✅ Working execution (DONE)
+- ✅ Clean interface (DONE)
+- ✅ Documentation (DONE)
+- ⏳ Fix augmentationPipeline.ts removal
+- ⏳ Test all 27 augmentations work
+
+### Nice to Have (2.0.x)
+- Local package discovery
+- NPM integration
+- Basic CLI tools
+
+### Future (2.1+)
+- Brain Cloud Registry
+- License management
+- Payment processing
+- Marketplace UI
+- Developer portal
+
+---
+
+## 📊 Current State Assessment
+
+| Component | Status | Notes |
+|-----------|--------|-------|
+| Core Execution | ✅ Working | AugmentationRegistry.execute() works |
+| Registration | ✅ Working | Manual registration works |
+| Auto-Config | ✅ Working | Cache, index, storage auto-register |
+| Lifecycle | ✅ Working | Init, execute, shutdown work |
+| Discovery | ❌ Missing | No package discovery |
+| Installation | ❌ Missing | No dynamic installation |
+| Marketplace | ❌ Missing | No registry client |
+| Licensing | ❌ Missing | No license management |
+| Versioning | ❌ Missing | No version checks |
+
+---
+
+## 💡 Recommendations
+
+### For 2.0 Release
+1. **Ship with manual registration** - It works!
+2. **Document how to create augmentations** - Critical for adoption
+3. **Create 2-3 example augmentations** - Show the patterns
+4. **Add basic CLI for testing** - Help developers
+
+### For 2.1 (Q1 2025)
+1. **Add NPM discovery** - Find installed augmentations
+2. **Dynamic loading** - Import augmentations at runtime
+3. **Basic marketplace API** - List available augmentations
+4. **Version checking** - Ensure compatibility
+
+### For 3.0 (Q2 2025)
+1. **Full marketplace** - Browse, search, install
+2. **Payment integration** - Premium augmentations
+3. **Developer portal** - Publish augmentations
+4. **Enterprise features** - Private registries
+
+---
+
+## ✅ Good News Summary
+
+The augmentation system WORKS! The core architecture is solid:
+- Execution mechanism is correct
+- Registration works
+- Lifecycle management works
+- All 27 augmentations function properly
+
+What's missing is the marketplace/discovery layer, which can be added incrementally without breaking the core system. The 2.0 release can ship with manual augmentation registration, and the marketplace features can be added in 2.1+.
+
+**Recommendation: Ship 2.0 with current system, add marketplace in 2.1**

docs/architecture/augmentations-actual.md

@@ -73,10 +73,10 @@ import { MemoryStorageAugmentation } from 'brainy'
 ```
 
 ### 11. Server Search Augmentation ✅
-Server-side search delegation over a conduit.
+Distributed search capabilities.
 ```typescript
 import { ServerSearchConduitAugmentation } from 'brainy'
-// Forwards queries to a remote Brainy server
+// Distributed query execution
 ```
 
 ### 12. Neural Import Augmentation ✅
@@ -100,7 +100,7 @@ await neuralImport.detectRelationships(entities)
 await neuralImport.generateInsights(data)
 ```
 
-### Operation Modes (Fully Implemented!)
+### Distributed Operation Modes (Fully Implemented!)
 ```typescript
 // Read-only mode with optimized caching
 const readerMode = new ReaderMode()
@@ -206,7 +206,7 @@ if (device === 'webgpu') {
 
 // CUDA detection in Node:
 if (device === 'cuda') {
-  // Future: GPU acceleration support
+  // Requires ONNX Runtime GPU packages
 }
 ```
 
@@ -239,10 +239,15 @@ const cacheConfig = await getCacheAutoConfig()
 
 ## 🎨 How to Use Hidden Features
 
-### Enable Reader / Writer Modes
+### Enable Distributed Modes
 ```typescript
 const brain = new Brainy({
-  mode: 'reader' // or 'writer' or 'hybrid'
+  mode: 'reader', // or 'writer' or 'hybrid'
+  distributed: {
+    role: 'reader',
+    cacheStrategy: 'aggressive',
+    prefetch: true
+  }
 })
 ```
 
@@ -280,7 +285,7 @@ const freshStats = await brain.getStatistics({
 ## 📝 What Needs Documentation
 
 These features EXIST but need better docs:
-1. Reader / writer operation modes
+1. Distributed operation modes
 2. Neural import full API
 3. 3-level cache configuration
 4. Performance monitoring API
@@ -292,7 +297,7 @@ These features EXIST but need better docs:
 ## 💡 The Truth
 
 Brainy is MORE powerful than its own documentation suggests! Most "missing" features are actually implemented but hidden or not properly exposed. The codebase contains sophisticated systems for:
-- Reader / writer operation modes
+- Distributed operations
 - AI-powered import
 - Advanced caching
 - Performance monitoring

docs/architecture/distributed-storage.md

@@ -0,0 +1,550 @@
+# 🏗️ Distributed Storage Architecture
+
+> **Technical deep-dive**: How Brainy coordinates storage across multiple nodes and adapters
+
+## Storage Adapter Layer
+
+### Base Storage Interface
+
+Every storage adapter implements this interface:
+
+```typescript
+interface StorageAdapter {
+  // Basic operations
+  get(key: string): Promise<any>
+  set(key: string, value: any): Promise<void>
+  delete(key: string): Promise<void>
+  
+  // Batch operations
+  getBatch(keys: string[]): Promise<Map<string, any>>
+  setBatch(items: Map<string, any>): Promise<void>
+  
+  // Atomic operations (for coordination)
+  compareAndSwap(key: string, oldVal: any, newVal: any): Promise<boolean>
+  increment(key: string, delta: number): Promise<number>
+  
+  // Namespace support
+  withNamespace(namespace: string): StorageAdapter
+}
+```
+
+### Storage Coordination Strategies
+
+#### Strategy 1: Isolated Storage (Default)
+
+Each node has completely separate storage:
+
+```
+Node-1 → Local FS: /data/node1/
+         └── shards/
+             ├── shard-001/
+             ├── shard-045/
+             └── shard-127/
+
+Node-2 → Local FS: /data/node2/
+         └── shards/
+             ├── shard-023/
+             ├── shard-067/
+             └── shard-089/
+```
+
+**Coordination**: Via network messages only
+- Shard ownership tracked in distributed consensus
+- Data transfer via direct node-to-node communication
+- No storage-level conflicts possible
+
+#### Strategy 2: Shared Storage with Namespacing
+
+Multiple nodes share storage but use namespaces:
+
+```
+S3 Bucket: brainy-cluster/
+├── node-abc123/
+│   ├── shards/
+│   └── wal/
+├── node-def456/
+│   ├── shards/
+│   └── wal/
+└── _cluster/
+    ├── topology.json
+    ├── shard-map.json
+    └── elections/
+```
+
+**Coordination**: Via storage-level atomic operations
+- Each node owns its namespace
+- Cluster metadata in shared `_cluster/` namespace
+- Atomic operations for leader election
+- Conditional writes prevent conflicts
+
+#### Strategy 3: Shared Storage with Fine-Grained Locking
+
+Advanced mode for full shared storage:
+
+```
+S3 Bucket: brainy-shared/
+├── shards/
+│   ├── 001/
+│   │   ├── data.bin
+│   │   └── .lock (atomic)
+│   ├── 002/
+│   │   ├── data.bin
+│   │   └── .lock
+└── metadata/
+    ├── index/
+    └── locks/
+```
+
+**Coordination**: Via distributed locking
+- Shard-level locks using atomic operations
+- Lock acquisition via compare-and-swap
+- Automatic lock expiry (lease-based)
+- Deadlock detection and recovery
+
+## Storage Adapter Implementations
+
+### 1. Filesystem Adapter
+
+```typescript
+class FilesystemAdapter implements StorageAdapter {
+  constructor(private basePath: string) {}
+  
+  async get(key: string) {
+    const path = this.keyToPath(key)
+    return fs.readFile(path, 'json')
+  }
+  
+  async compareAndSwap(key: string, oldVal: any, newVal: any) {
+    // Use file locking for atomicity
+    const lockfile = `${this.keyToPath(key)}.lock`
+    await flock(lockfile, 'ex')  // Exclusive lock
+    try {
+      const current = await this.get(key)
+      if (deepEqual(current, oldVal)) {
+        await this.set(key, newVal)
+        return true
+      }
+      return false
+    } finally {
+      await funlock(lockfile)
+    }
+  }
+  
+  withNamespace(ns: string) {
+    return new FilesystemAdapter(path.join(this.basePath, ns))
+  }
+}
+```
+
+### 2. S3 Adapter
+
+```typescript
+class S3Adapter implements StorageAdapter {
+  constructor(
+    private bucket: string,
+    private prefix: string = ''
+  ) {}
+  
+  async get(key: string) {
+    const result = await s3.getObject({
+      Bucket: this.bucket,
+      Key: `${this.prefix}${key}`
+    })
+    return JSON.parse(result.Body)
+  }
+  
+  async compareAndSwap(key: string, oldVal: any, newVal: any) {
+    // Use S3's conditional writes
+    const fullKey = `${this.prefix}${key}`
+    
+    // Get current version
+    const head = await s3.headObject({
+      Bucket: this.bucket,
+      Key: fullKey
+    })
+    
+    // Conditional put with ETag
+    try {
+      await s3.putObject({
+        Bucket: this.bucket,
+        Key: fullKey,
+        Body: JSON.stringify(newVal),
+        IfMatch: head.ETag  // Only succeeds if unchanged
+      })
+      return true
+    } catch (err) {
+      if (err.code === 'PreconditionFailed') {
+        return false
+      }
+      throw err
+    }
+  }
+  
+  withNamespace(ns: string) {
+    const newPrefix = `${this.prefix}${ns}/`
+    return new S3Adapter(this.bucket, newPrefix)
+  }
+}
+```
+
+### 3. Cloudflare R2 Adapter
+
+```typescript
+class R2Adapter implements StorageAdapter {
+  // Similar to S3 but with R2-specific optimizations
+  
+  async compareAndSwap(key: string, oldVal: any, newVal: any) {
+    // R2 supports conditional headers
+    const response = await fetch(`${this.endpoint}/${key}`, {
+      method: 'PUT',
+      body: JSON.stringify(newVal),
+      headers: {
+        'If-Match': await this.getETag(key)
+      }
+    })
+    return response.ok
+  }
+  
+  // R2-specific: Use Workers for edge computing
+  async getWithCache(key: string) {
+    // Check Cloudflare edge cache first
+    const cached = await caches.default.match(key)
+    if (cached) return cached.json()
+    
+    // Fallback to R2
+    const value = await this.get(key)
+    
+    // Cache at edge
+    await caches.default.put(key, new Response(JSON.stringify(value)))
+    
+    return value
+  }
+}
+```
+
+## Distributed Coordination Patterns
+
+### Pattern 1: Leader-Based Coordination
+
+```typescript
+class LeaderCoordinator {
+  async acquireShardOwnership(shardId: string) {
+    if (!this.isLeader()) {
+      // Only leader assigns shards
+      return this.requestFromLeader('acquireShard', shardId)
+    }
+    
+    // Leader logic
+    const shardMap = await this.storage.get('_cluster/shard-map')
+    if (!shardMap[shardId].owner) {
+      shardMap[shardId].owner = this.nodeId
+      
+      // Atomic update
+      const success = await this.storage.compareAndSwap(
+        '_cluster/shard-map',
+        shardMap,
+        { ...shardMap, [shardId]: { owner: this.nodeId } }
+      )
+      
+      if (success) {
+        this.broadcast('shardAssigned', { shardId, owner: this.nodeId })
+      }
+    }
+  }
+}
+```
+
+### Pattern 2: Consensus-Based Coordination
+
+```typescript
+class ConsensusCoordinator {
+  async acquireShardOwnership(shardId: string) {
+    // Propose to all nodes
+    const proposal = {
+      type: 'ACQUIRE_SHARD',
+      shardId,
+      nodeId: this.nodeId,
+      term: this.currentTerm
+    }
+    
+    // Raft consensus
+    const votes = await this.gatherVotes(proposal)
+    
+    if (votes.length > this.nodes.length / 2) {
+      // Majority agreed
+      await this.commitProposal(proposal)
+      return true
+    }
+    
+    return false
+  }
+}
+```
+
+### Pattern 3: Storage-Native Coordination
+
+```typescript
+class StorageNativeCoordinator {
+  async acquireShardOwnership(shardId: string) {
+    // Use storage adapter's native coordination
+    const lockKey = `_locks/shard-${shardId}`
+    const lease = {
+      owner: this.nodeId,
+      expires: Date.now() + 30000  // 30 second lease
+    }
+    
+    // Try to acquire lock atomically
+    const acquired = await this.storage.compareAndSwap(
+      lockKey,
+      null,  // Must not exist
+      lease
+    )
+    
+    if (acquired) {
+      // Start lease renewal
+      this.startLeaseRenewal(lockKey, lease)
+      return true
+    }
+    
+    return false
+  }
+  
+  private startLeaseRenewal(key: string, lease: any) {
+    setInterval(async () => {
+      const renewed = await this.storage.compareAndSwap(
+        key,
+        lease,
+        { ...lease, expires: Date.now() + 30000 }
+      )
+      
+      if (!renewed) {
+        // Lost lease
+        this.handleLeaseLoss(key)
+      }
+    }, 10000)  // Renew every 10s
+  }
+}
+```
+
+## Multi-Storage Patterns
+
+### Hybrid Storage (Hot/Cold)
+
+```typescript
+class HybridStorageAdapter implements StorageAdapter {
+  constructor(
+    private hot: StorageAdapter,   // Fast SSD
+    private cold: StorageAdapter   // Cheap S3
+  ) {}
+  
+  async get(key: string) {
+    // Try hot storage first
+    const hotValue = await this.hot.get(key).catch(() => null)
+    if (hotValue) {
+      this.updateAccessTime(key)
+      return hotValue
+    }
+    
+    // Fallback to cold storage
+    const coldValue = await this.cold.get(key)
+    
+    // Promote to hot storage if frequently accessed
+    if (this.shouldPromote(key)) {
+      await this.hot.set(key, coldValue)
+    }
+    
+    return coldValue
+  }
+  
+  async set(key: string, value: any) {
+    // Write to hot storage
+    await this.hot.set(key, value)
+    
+    // Async write to cold storage
+    setImmediate(() => {
+      this.cold.set(key, value).catch(console.error)
+    })
+  }
+  
+  // Background process to demote cold data
+  async runTiering() {
+    const hotKeys = await this.hot.listKeys()
+    
+    for (const key of hotKeys) {
+      const lastAccess = await this.getAccessTime(key)
+      
+      if (Date.now() - lastAccess > 7 * 24 * 60 * 60 * 1000) {
+        // Not accessed in 7 days, demote to cold
+        await this.cold.set(key, await this.hot.get(key))
+        await this.hot.delete(key)
+      }
+    }
+  }
+}
+```
+
+### Geo-Distributed Storage
+
+```typescript
+class GeoDistributedAdapter implements StorageAdapter {
+  constructor(
+    private regions: Map<string, StorageAdapter>
+  ) {}
+  
+  async get(key: string) {
+    // Determine closest region
+    const region = await this.getClosestRegion()
+    
+    // Try local region first
+    const localValue = await this.regions.get(region)
+      .get(key)
+      .catch(() => null)
+    
+    if (localValue) return localValue
+    
+    // Fallback to other regions
+    for (const [name, adapter] of this.regions) {
+      if (name !== region) {
+        const value = await adapter.get(key).catch(() => null)
+        if (value) {
+          // Replicate to local region for next time
+          this.regions.get(region).set(key, value)
+          return value
+        }
+      }
+    }
+    
+    throw new Error('Key not found in any region')
+  }
+  
+  async set(key: string, value: any) {
+    // Write to local region immediately
+    const region = await this.getClosestRegion()
+    await this.regions.get(region).set(key, value)
+    
+    // Async replication to other regions
+    for (const [name, adapter] of this.regions) {
+      if (name !== region) {
+        adapter.set(key, value).catch(console.error)
+      }
+    }
+  }
+}
+```
+
+## Storage Optimization Strategies
+
+### 1. Write Batching
+
+```typescript
+class BatchingAdapter implements StorageAdapter {
+  private writeBatch = new Map()
+  private batchTimer?: NodeJS.Timeout
+  
+  async set(key: string, value: any) {
+    this.writeBatch.set(key, value)
+    
+    if (!this.batchTimer) {
+      this.batchTimer = setTimeout(() => this.flush(), 100)
+    }
+    
+    if (this.writeBatch.size >= 1000) {
+      await this.flush()
+    }
+  }
+  
+  private async flush() {
+    if (this.writeBatch.size === 0) return
+    
+    const batch = new Map(this.writeBatch)
+    this.writeBatch.clear()
+    
+    await this.underlying.setBatch(batch)
+    
+    if (this.batchTimer) {
+      clearTimeout(this.batchTimer)
+      this.batchTimer = undefined
+    }
+  }
+}
+```
+
+### 2. Read Caching
+
+```typescript
+class CachingAdapter implements StorageAdapter {
+  private cache = new LRU({ max: 10000 })
+  
+  async get(key: string) {
+    // Check cache
+    if (this.cache.has(key)) {
+      return this.cache.get(key)
+    }
+    
+    // Read from storage
+    const value = await this.underlying.get(key)
+    
+    // Cache for next time
+    this.cache.set(key, value)
+    
+    return value
+  }
+  
+  async set(key: string, value: any) {
+    // Invalidate cache
+    this.cache.delete(key)
+    
+    // Write through
+    await this.underlying.set(key, value)
+  }
+}
+```
+
+### 3. Compression
+
+```typescript
+class CompressingAdapter implements StorageAdapter {
+  async set(key: string, value: any) {
+    const json = JSON.stringify(value)
+    
+    // Compress if beneficial
+    if (json.length > 1024) {
+      const compressed = await gzip(json)
+      await this.underlying.set(key, {
+        _compressed: true,
+        data: compressed.toString('base64')
+      })
+    } else {
+      await this.underlying.set(key, value)
+    }
+  }
+  
+  async get(key: string) {
+    const stored = await this.underlying.get(key)
+    
+    if (stored._compressed) {
+      const compressed = Buffer.from(stored.data, 'base64')
+      const json = await gunzip(compressed)
+      return JSON.parse(json)
+    }
+    
+    return stored
+  }
+}
+```
+
+## Summary
+
+Brainy's storage layer is designed for:
+
+1. **Flexibility**: Works with any storage backend
+2. **Coordination**: Multiple strategies for different needs
+3. **Performance**: Batching, caching, compression
+4. **Scalability**: From single file to geo-distributed
+5. **Simplicity**: Complexity hidden behind simple interface
+
+The key insight: **Storage is just a plugin**. The intelligence is in the coordination layer above it!
+
+---
+
+*For user-facing documentation, see [SCALING.md](../SCALING.md)*

docs/architecture/finite-type-system.md

@@ -80,7 +80,7 @@ const entity = {
 - ✅ **Semantic Understanding**: Types have meaning, not just structure
 - ✅ **Tool Compatibility**: All augmentations understand core types
 - ✅ **Concept Extraction**: NLP can map text to known types
-- ✅ **Explicit Types**: Clear type specification in API
+- ✅ **Type Inference**: Automatic type detection via keywords/synonyms
 - ✅ **Query Optimization**: Type-aware query planning
 - ✅ **Flexible Metadata**: Any fields within typed structure
 - ✅ **Billion-Scale Ready**: Type tracking scales linearly
@@ -116,56 +116,50 @@ class TypeAwareMetadataIndex {
 }
 ```
 
-**Real-World Impact (PROJECTED - not yet benchmarked)**:
+**Real-World Impact**:
 - **Before**: 500MB memory for 1M entities with diverse keys
-- **After**: PROJECTED 1.2MB memory for same dataset (385x reduction - calculated from Uint32Array size, not measured)
+- **After**: 1.2MB memory for same dataset (385x reduction!)
 - **Scales to billions**: Memory grows with entity count, not key diversity
 
-### 2. Explicit Type System
+### 2. Semantic Type Inference
 
-**The Design**: Specify types clearly in your API calls:
+**The Magic**: Map natural language to structured types:
 
 ```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
+import { getSemanticTypeInference } from '@soulcraft/brainy'
 
-// Add entity with explicit type
-await brain.add({
-  data: { name: 'Alice', role: 'CEO of Acme Corp' },
-  type: NounType.Person  // Explicit type specification
-})
+const inference = getSemanticTypeInference()
 
-// Query with type filtering
-await brain.find({
-  query: 'Alice',
-  type: NounType.Person  // Type-optimized search
-})
+// Automatic type detection
+await inference.inferNounType('CEO of Acme Corp')
+// → 'person'
+
+await inference.inferNounType('San Francisco office building')
+// → 'place'
+
+await inference.inferVerbType('Alice manages Bob')
+// → 'manages' (relationship type)
 ```
 
-**Why Explicit Types?**:
-1. **Deterministic**: You control exactly how entities are classified
-2. **Predictable**: No inference surprises or edge cases
-3. **Fast**: No neural processing overhead on every add/query
-4. **Smaller**: No embedded keyword models needed
+**How It Works**:
+1. **Keyword Matching**: "CEO", "manager" → 'person'
+2. **Synonym Detection**: "building", "office" → 'place'
+3. **Semantic Embeddings**: Vector similarity to type prototypes
+4. **Context Analysis**: Surrounding words provide hints
 
 **Real-World Use Case**:
 ```typescript
-// Import data with known types
-await brain.add({
-  data: { name: 'Apple Inc.', industry: 'Technology' },
-  type: NounType.Organization
-})
+// Import unstructured data
+const text = "Apple announced a new product line in Cupertino"
 
-await brain.add({
-  data: { name: 'Cupertino', country: 'USA' },
-  type: NounType.Location
-})
+// Brainy automatically infers:
+// - "Apple" → noun type: 'organization'
+// - "product line" → noun type: 'product'
+// - "Cupertino" → noun type: 'place'
+// - "announced" → verb type: 'announces'
+// - "in" → verb type: 'locatedIn'
 
-// Create relationship
-await brain.relate({
-  from: appleId,
-  to: cupertinoId,
-  type: VerbType.LocatedIn
-})
+// Creates typed, queryable knowledge graph automatically!
 ```
 
 ### 3. Tool & Augmentation Compatibility
@@ -322,7 +316,7 @@ class TypeAwareIndex {
   private nounTypeTracking: Uint32Array  // Fixed size!
   private typeIndexes: RoaringBitmap32[] // One per type
   // Memory: O(noun_types) + O(entities_per_type)
-  // PROJECTED: 385x smaller at billion scale (calculated from architecture, not benchmarked)
+  // 385x smaller at billion scale!
 }
 ```
 
@@ -370,47 +364,42 @@ function processNoun(noun: Noun) {
 
 ---
 
-## Public API: Type System
+## Public API: Semantic Type Inference
 
-The type system is **fully public** for developers and augmentation authors:
+The type inference system is **fully public** for augmentation developers and external tools:
 
 ```typescript
 import {
-  NounType,
-  VerbType,
-  getNounTypes,
-  getVerbTypes,
-  BrainyTypes,
-  suggestType
+  getSemanticTypeInference,
+  SemanticTypeInference
 } from '@soulcraft/brainy'
 
-// Get all available noun types
-const nounTypes = getNounTypes()
-// → ['Person', 'Organization', 'Location', 'Thing', 'Concept', ...]
+// Get singleton instance
+const inference = getSemanticTypeInference()
 
-// Get all available verb types
-const verbTypes = getVerbTypes()
-// → ['RelatedTo', 'Contains', 'CreatedBy', 'LocatedIn', ...]
+// Infer noun type from text
+const nounType = await inference.inferNounType('Software Engineer')
+// → 'person'
 
-// Use types directly
-await brain.add({
-  data: { name: 'Alice' },
-  type: NounType.Person
-})
+// Infer verb type from relationship text
+const verbType = await inference.inferVerbType('works at')
+// → 'worksAt'
 
-// Query by type
-await brain.find({
-  type: NounType.Person,
-  where: { name: 'Alice' }
-})
+// Get type keywords for reverse lookup
+const keywords = inference.getNounTypeKeywords('person')
+// → ['person', 'human', 'individual', 'user', 'employee', ...]
+
+// Get type synonyms
+const synonyms = inference.getNounTypeSynonyms('organization')
+// → ['company', 'corporation', 'business', 'firm', 'enterprise', ...]
 ```
 
 **Use Cases**:
-- **Type-Safe Code**: Use TypeScript enums for compile-time checking
-- **Import Tools**: Specify entity types during data import
-- **Query Builders**: Filter by known types
+- **Import Tools**: Auto-detect entity types during data import
+- **Query Builders**: Suggest types based on user input
 - **Augmentations**: Type-specific processing pipelines
 - **Visualization**: Type-appropriate rendering
+- **Data Validation**: Ensure correct type assignments
 
 ---
 
@@ -426,7 +415,7 @@ await brain.find({
 | **Query Planning** | Impossible | Table statistics | Type statistics |
 | **Tool Compatibility** | None | SQL only | Full ecosystem |
 | **Semantic Understanding** | None | None | Built-in |
-| **Concept Extraction** | Manual | Manual | Via SmartExtractor |
+| **Concept Extraction** | Manual | Manual | Automatic |
 | **Flexibility** | Infinite | Zero | Optimal balance |
 
 ---
@@ -449,30 +438,6 @@ brain.registerNounType('chemical_compound', {
 })
 ```
 
-### 1a. Subtypes — sub-classification without hierarchy
-
-The 42-type taxonomy is intentionally coarse. Per-product vocabulary fits on the **`subtype`** axis — a top-level standard string field on every entity. Flat by design — no hierarchy, no parent chain, no recursive resolution. That preserves the Uint32Array-backed O(1) type stats while giving consumers a place to put `'employee'` / `'customer'` / `'invoice'` / `'milestone'` without burning a slot in the global enum.
-
-```typescript
-// Same NounType, different subtypes:
-await brain.add({ type: NounType.Person,   subtype: 'employee' })
-await brain.add({ type: NounType.Person,   subtype: 'customer' })
-await brain.add({ type: NounType.Document, subtype: 'invoice'  })
-
-// Fast path — column-store hit, not metadata fallback:
-await brain.find({ type: NounType.Person, subtype: 'employee' })
-
-// Per-NounType-per-subtype counts maintained incrementally:
-brain.counts.bySubtype(NounType.Person)
-// → { employee: 12, customer: 847 }
-```
-
-Subtype has its own statistics rollup (`_system/subtype-statistics.json`) maintained alongside `nounCountsByType`, so per-subtype counts stay O(1) at billion scale.
-
-The same principle applies to **VerbTypes** (7.30+): the 127-verb taxonomy is intentionally coarse, and `subtype` is the per-product axis for relationships too. A `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'colleague'`. Verb-side rollup lives at `_system/verb-subtype-statistics.json` with identical shape to the noun-side rollup. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`. Brainy's design is fully symmetric — nouns and verbs are first-class peers with identical capability surfaces.
-
-Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
-
 ### 2. Semantic not Structural
 
 ```typescript
@@ -518,7 +483,7 @@ Brainy's **Finite Noun/Verb Type System** is revolutionary because it achieves t
 2. ✅ **Semantic understanding** (NLP integration)
 3. ✅ **Tool compatibility** (ecosystem interoperability)
 4. ✅ **Query optimization** (type-aware planning)
-5. ✅ **Concept extraction** (via SmartExtractor for imports)
+5. ✅ **Concept extraction** (automatic type inference)
 6. ✅ **Developer experience** (clean architecture)
 7. ✅ **Flexibility** (metadata freedom within types)
 
@@ -528,10 +493,11 @@ It's not schemaless chaos. It's not rigid relational constraints. It's **semanti
 
 ## Further Reading
 
+- [Type Inference System](../api/type-inference.md) - API reference for semantic type detection
 - [Storage Architecture](./storage-architecture.md) - How types enable billion-scale storage
 - [Augmentation System](./augmentations.md) - Building type-aware augmentations
+- [Concept Extraction](../guides/natural-language.md) - NLP integration with typed entities
 - [Query Optimization](../api/query-optimization.md) - Type-aware query planning
-- [Import Flow](../guides/import-flow.md) - How types work in the import pipeline
 
 ---
 

docs/architecture/index-architecture.md

@@ -1,45 +1,23 @@
 # Index Architecture
 
-Brainy uses a sophisticated **3-tier index architecture** that enables "Triple Intelligence" - the unified combination of vector similarity, graph relationships, and metadata filtering. This document provides a comprehensive architectural overview of how these indexes work internally and coordinate with each other.
+Brainy uses a sophisticated **4-index architecture** that enables "Triple Intelligence" - the unified combination of vector similarity, graph relationships, and metadata filtering. This document provides a comprehensive architectural overview of how these indexes work internally and coordinate with each other.
 
-## Overview: The Three Main Indexes + Sub-Indexes
+## Overview: The Four Core Indexes
 
-Brainy has **3 main indexes** at the top level, each with multiple sub-indexes managed automatically:
+| Index | Purpose | Data Structure | Complexity | File Location |
+|-------|---------|----------------|------------|---------------|
+| **MetadataIndex** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` |
+| **HNSWIndex** | Vector similarity search | Hierarchical graphs | O(log n) search | `src/hnsw/hnswIndex.ts` |
+| **GraphAdjacencyIndex** | Relationship traversal | Bidirectional adjacency maps | O(1) per hop | `src/graph/graphAdjacencyIndex.ts` |
+| **DeletedItemsIndex** | Soft-delete tracking | Simple Set | O(1) all ops | `src/utils/deletedItemsIndex.ts` |
 
-### Main Indexes (Level 1)
-
-| Index | Purpose | Data Structure | Complexity | File Location | rebuild() Method |
-|-------|---------|----------------|------------|---------------|------------------|
-| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
-| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 |
-| **GraphAdjacencyIndex** | Relationship traversal | 2 verb-id LSM-trees + tombstone-filtered adjacency derivation | O(degree) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
-
-### Sub-Indexes (Level 2)
-
-**TypeAwareVectorIndex contains:**
-- **42 type-specific vector indexes** - One per NounType (automatically rebuilt via parent)
-
-**MetadataIndexManager contains:**
-- **ChunkManager** - Adaptive chunked sparse indexing
-- **EntityIdMapper** - UUID ↔ integer mapping for roaring bitmaps
-- **FieldTypeInference** - DuckDB-inspired value-based field type detection
-- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count)
-- **Sorted Indexes** - Support orderBy queries (automatically maintained)
-- **Word Index (`__words__`)** - Text search via FNV-1a word hashes
-
-**GraphAdjacencyIndex contains:**
-- **lsmTreeSource** - Source → Targets (outgoing edges)
-- **lsmTreeTarget** - Target → Sources (incoming edges)
-- **lsmTreeVerbsBySource** - Source → Verb IDs
-- **lsmTreeVerbsByTarget** - Target → Verb IDs
-
-All indexes share a **UnifiedCache** for coordinated memory management, ensuring fair resource allocation and preventing any single index from monopolizing memory.
+All four indexes share a **UnifiedCache** for coordinated memory management, ensuring fair resource allocation and preventing any single index from monopolizing memory.
 
 ## 1. MetadataIndex - Fast Field Filtering
 
 **Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing.
 
-### Internal Architecture
+### Internal Architecture (v3.42.0)
 
 ```typescript
 class MetadataIndexManager {
@@ -64,7 +42,7 @@ class MetadataIndexManager {
 
 ### Key Data Structures
 
-#### Chunked Sparse Index
+#### Chunked Sparse Index (NEW in v3.42.0)
 ```typescript
 // SparseIndex: Directory of chunks for a field
 // Example: field="status"
@@ -87,7 +65,7 @@ interface ChunkDescriptor {
 class ChunkData {
   chunkId: number
   field: string
-  entries: Map<value, RoaringBitmap32>  // ~50 values per chunk (roaring bitmaps!)
+  entries: Map<value, RoaringBitmap32>  // ~50 values per chunk (v3.43.0: roaring bitmaps!)
 }
 ```
 
@@ -96,7 +74,7 @@ class ChunkData {
 - O(log n) range queries with zone maps
 - 630x file reduction (560k flat files → 89 chunk files)
 
-#### Roaring Bitmap Optimization
+#### Roaring Bitmap Optimization (NEW in v3.43.0)
 
 **Problem Solved**: JavaScript `Set<string>` for storing entity IDs was inefficient:
 - Memory overhead: ~40 bytes per UUID string (36 chars + overhead)
@@ -150,7 +128,7 @@ class ChunkData {
 
 **Multi-Field Intersection (THE BIG WIN!)**:
 ```typescript
-// Before: JavaScript array filtering
+// Before (v3.42.0): JavaScript array filtering
 async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string[]> {
   // 1. Fetch UUID arrays for each field
   const statusIds = await this.getIds('status', 'active')  // ["uuid1", "uuid2", ...]
@@ -160,7 +138,7 @@ async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string
   return statusIds.filter(id => roleIds.includes(id))      // O(n*m) array filtering
 }
 
-// After: Roaring bitmap intersection
+// After (v3.43.0): Roaring bitmap intersection
 async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
   // 1. Fetch roaring bitmaps (integers, not UUIDs)
   const bitmaps: RoaringBitmap32[] = []
@@ -183,7 +161,7 @@ async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
 - Memory usage: **90% reduction** (17.17 MB → 2.01 MB for 100K entities)
 - Hardware acceleration: SIMD instructions make bitmap operations nearly free
 
-**Benchmark Results** — example output from a single run of `tests/performance/roaring-bitmap-benchmark.ts` (1,000 queries per size, one machine; absolute times vary by hardware, the relative speedup and memory savings are the durable signal):
+**Benchmark Results** (1,000 queries on various dataset sizes):
 | Dataset Size | Operation | Set Time | Roaring Time | Speedup | Memory Savings |
 |--------------|-----------|----------|--------------|---------|----------------|
 | 10,000 entities | 3-field intersection | 3.74ms | 1.14ms | **3.3x faster** | 90% |
@@ -229,38 +207,7 @@ interface ZoneMap {
 
 **Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field
 
-#### Word Index (`__words__`) -
-```typescript
-// Special field for text/keyword search
-// Entity text content is tokenized and indexed as word hashes
-
-// Tokenization:
-// "David Smith is a software engineer" → ["david", "smith", "is", "software", "engineer"]
-
-// Word Hashing (FNV-1a):
-// "david" → hashWord("david") → 1234567  (int32)
-// "smith" → hashWord("smith") → 9876543  (int32)
-
-// Index structure (same as other fields):
-// __words__ → 1234567 → RoaringBitmap{entity1, entity5, ...}
-// __words__ → 9876543 → RoaringBitmap{entity1, entity3, ...}
-```
-
-**Design Decisions**:
-- **Max 50 words per entity**: Prevents index bloat for large documents
-- **FNV-1a hashing**: Fast, low collision rate, int32 output
-- **Min word length 2 chars**: Filters out noise words
-- **Lowercase normalization**: Case-insensitive matching
-- **Automatic integration**: Words extracted via `extractIndexableFields()`
-
-**Hybrid Search**: Text results combined with vector results using Reciprocal Rank Fusion (RRF):
-```typescript
-// RRF formula: score(d) = sum(1 / (k + rank(d)))
-// where k = 60 (standard constant)
-// alpha = weight for semantic (0 = text only, 1 = semantic only)
-```
-
-### Query Algorithm
+### Query Algorithm (v3.42.0)
 
 **Exact Match Query**:
 ```typescript
@@ -317,7 +264,7 @@ async getIdsForRange(field: string, min: any, max: any): Promise<string[]> {
 - Adaptive chunking: ~50 values per chunk optimizes I/O
 - Immediate flushing: No need for dirty tracking or batch writes
 
-### Temporal Bucketing
+### Temporal Bucketing (v3.41.0)
 
 **Problem Solved**: High-cardinality timestamp fields created massive file pollution.
 - Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!)
@@ -396,18 +343,16 @@ const DEFAULT_EXCLUDE_FIELDS = [
 ]
 ```
 
-**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
+**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of v3.41.0 - they are indexed with automatic bucketing.
 
-## 2. Vector Index - Vector Similarity Search
+## 2. HNSWIndex - Vector Similarity Search
 
 **Purpose**: O(log n) semantic similarity search using vector embeddings.
 
-The default JS implementation is `JsHnswVectorIndex`; an optional native acceleration package (`@soulcraft/cor`) can register a higher-performing `VectorIndexProvider` through the plugin system. The public API stays the same either way.
-
 ### Internal Architecture
 
 ```typescript
-class JsHnswVectorIndex {
+class HNSWIndex {
   // Per-noun indexes for efficiency
   private nouns: Map<string, HNSWNoun> = new Map()
 
@@ -439,7 +384,7 @@ class HNSWNode {
 
 ### Hierarchical Graph Structure
 
-The default vector index builds a multi-layered graph:
+HNSW builds a multi-layered graph:
 
 ```
 Layer 2: [entry] ←→ [node1]                    (sparse, long-range connections)
@@ -578,9 +523,56 @@ const reachable = await this.graphIndex.traverse({
 // Complexity: O(V + E) breadth-first search, but each neighbor lookup is O(1)
 ```
 
+## 4. DeletedItemsIndex - Soft-Delete Tracking
+
+**Purpose**: O(1) tracking of soft-deleted items without removing data.
+
+### Internal Architecture
+
+```typescript
+class DeletedItemsIndex {
+  private deletedIds: Set<string> = new Set()
+  private deletedCount: number = 0
+  private storage: BaseStorage
+}
+```
+
+**Simplicity is key**: Just a Set of deleted IDs. No complex logic needed.
+
+### Operations
+
+```typescript
+// Mark as deleted
+this.deletedItemsIndex.markDeleted(id)  // O(1)
+
+// Check if deleted
+const isDeleted = this.deletedItemsIndex.isDeleted(id)  // O(1)
+
+// Filter out deleted items
+const active = this.deletedItemsIndex.filterDeleted(results)  // O(n)
+
+// Restore
+this.deletedItemsIndex.markRestored(id)  // O(1)
+
+// Get all deleted
+const deleted = this.deletedItemsIndex.getAllDeleted()  // O(1) - returns Set
+```
+
+### Integration
+
+All query results are filtered through the deleted items index:
+
+```typescript
+// In brainy.find() (src/brainy.ts:1026+)
+let results = await this.performSearch(query)
+
+// Filter out deleted items before returning
+results = results.filter(r => !this.deletedItemsIndex.isDeleted(r.id))
+```
+
 ## Shared Memory Management: UnifiedCache
 
-All three main indexes share a single **UnifiedCache** instance for coordinated memory management.
+All four indexes share a single **UnifiedCache** instance for coordinated memory management.
 
 ### Architecture
 
@@ -595,7 +587,7 @@ class UnifiedCache {
 // Each index gets the same cache instance
 const unifiedCache = new UnifiedCache({ maxSize: 1000 })
 this.metadataIndex = new MetadataIndexManager(storage, { unifiedCache })
-this.vectorIndex = new JsHnswVectorIndex(storage, { unifiedCache })
+this.hnswIndex = new HNSWIndex(storage, { unifiedCache })
 this.graphIndex = new GraphAdjacencyIndex(storage, { unifiedCache })
 ```
 
@@ -615,7 +607,7 @@ Each index uses different key prefixes:
 // Metadata index
 cache.set(`meta:${field}:${value}`, indexEntry)
 
-// Vector index
+// HNSW index
 cache.set(`vector:${id}`, vectorData)
 
 // Graph index
@@ -637,7 +629,7 @@ async add(params: AddParams): Promise<string> {
   // Add to metadata index (field filtering)
   await this.metadataIndex.addToIndex(id, params.metadata)
 
-  // Add to vector index (vector search)
+  // Add to HNSW index (vector search)
   await this.index.addEntity(id, vector, params.noun)
 
   // Relationships added via separate relate() calls
@@ -674,6 +666,9 @@ async find(query: FindQuery): Promise<Result[]> {
     results = results.filter(r => connectedIds.includes(r.id))
   }
 
+  // Step 4: Filter deleted items
+  results = results.filter(r => !this.deletedItemsIndex.isDeleted(r.id))
+
   return results
 }
 ```
@@ -689,7 +684,7 @@ async update(params: UpdateParams): Promise<void> {
   await this.metadataIndex.removeFromIndex(params.id, existing.metadata)
   await this.metadataIndex.addToIndex(params.id, params.metadata)
 
-  // Update vector index (re-embed if content changed)
+  // Update HNSW index (re-embed if content changed)
   if (params.content) {
     const newVector = await this.embedder(params.content)
     await this.index.updateEntity(params.id, newVector)
@@ -715,73 +710,47 @@ async stats(): Promise<Statistics> {
     relationships: this.graphIndex.getTotalRelationshipCount(),
     relationshipTypes: this.graphIndex.getRelationshipCountsByType(),
 
-    // From vector index
+    // From deleted items index
+    deletedItems: this.deletedItemsIndex.getDeletedCount(),
+
+    // From HNSW index
     vectorIndexSize: this.index.getSize()
   }
 }
 ```
 
-### 5. Index Rebuilding (Lazy Loading Support)
+### 5. Index Rebuilding
 
-**Two modes of index loading:**
-
-#### Mode 1: Auto-Rebuild on init() (default)
+All indexes rebuilt in parallel on initialization:
 
 ```typescript
 // src/brainy.ts:init()
 async init(): Promise<void> {
-  // When disableAutoRebuild: false (default)
-  const metadataStats = await this.metadataIndex.getStats()
-  const vectorIndexSize = this.index.size()
-  const graphIndexSize = await this.graphIndex.size()
+  // Check if indexes are empty
+  const metadataEmpty = await this.metadataIndex.isEmpty()
+  const hnswEmpty = await this.index.isEmpty()
+  const graphEmpty = await this.graphIndex.isEmpty()
 
-  if (metadataStats.totalEntries === 0 ||
-      vectorIndexSize === 0 ||
-      graphIndexSize === 0) {
+  if (metadataEmpty || hnswEmpty || graphEmpty) {
     // Rebuild all indexes in parallel
     await Promise.all([
-      metadataStats.totalEntries === 0 ? this.metadataIndex.rebuild() : Promise.resolve(),
-      vectorIndexSize === 0 ? this.index.rebuild() : Promise.resolve(),
-      graphIndexSize === 0 ? this.graphIndex.rebuild() : Promise.resolve()
+      metadataEmpty ? this.metadataIndex.rebuild() : Promise.resolve(),
+      hnswEmpty ? this.index.rebuild() : Promise.resolve(),
+      graphEmpty ? this.graphIndex.rebuild() : Promise.resolve()
     ])
   }
 }
 ```
 
-#### Mode 2: Lazy Loading on First Query
-
-```typescript
-// When disableAutoRebuild: true
-const brain = new Brainy({
-  storage: { type: 'filesystem' },
-  disableAutoRebuild: true  // Enable lazy loading
-})
-
-await brain.init()  // Returns instantly, indexes empty
-
-// First query triggers lazy rebuild
-const results = await brain.find({ limit: 10 })
-// → Calls ensureIndexesLoaded() (line 4617)
-// → Rebuilds all 3 main indexes with concurrency control
-// → Subsequent queries are instant (0ms check)
-```
-
-**Performance:**
-- First query with lazy loading: ~50-200ms rebuild (1K-10K entities)
-- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
-- Subsequent queries: 0ms check (instant)
-
-See [initialization-and-rebuild.md](./initialization-and-rebuild.md) for detailed lazy loading implementation.
-
 ## Triple Intelligence Integration
 
-The **TripleIntelligenceSystem** (`src/triple/TripleIntelligenceSystem.ts`) combines all three core indexes:
+The **TripleIntelligenceSystem** (`src/cortex/tripleIntelligence.ts`) combines all three core indexes:
 
 ```typescript
 class TripleIntelligenceSystem {
   constructor(
     private metadataIndex: MetadataIndexManager,
-    private vectorIndex: VectorIndexProvider,
+    private hnswIndex: HNSWIndex,
     private graphIndex: GraphAdjacencyIndex,
     private embedder: EmbedderFunction,
     private storage: BaseStorage
@@ -794,7 +763,7 @@ class TripleIntelligenceSystem {
     // Execute across all three indexes
     const [metadataResults, vectorResults, graphResults] = await Promise.all([
       this.metadataIndex.getIdsForFilter(parsed.filters),
-      this.vectorIndex.search(parsed.vector, parsed.limit),
+      this.hnswIndex.search(parsed.vector, parsed.limit),
       this.graphIndex.traverse(parsed.graphConstraints)
     ])
 
@@ -808,54 +777,46 @@ class TripleIntelligenceSystem {
 
 ### Operation Complexity by Index
 
-| Operation | MetadataIndexManager | TypeAwareVectorIndex | GraphAdjacencyIndex |
-|-----------|---------------------|-------------------|---------------------|
-| **Add** | O(1) per field | O(log n) | O(1) |
-| **Remove** | O(1) per field | O(log n) | O(1) |
-| **Exact lookup** | O(1) | N/A | O(1) |
-| **Range query** | O(log n) + O(k) | N/A | N/A |
-| **Similarity search** | N/A | O(log n) | N/A |
-| **Neighbor lookup** | N/A | N/A | O(1) |
-| **Statistics** | O(1) | O(1) | O(1) |
-| **Rebuild** | O(n) | O(n) | O(n) |
+| Operation | MetadataIndex | HNSWIndex | GraphAdjacencyIndex | DeletedItemsIndex |
+|-----------|---------------|-----------|---------------------|-------------------|
+| **Add** | O(1) per field | O(log n) | O(1) | O(1) |
+| **Remove** | O(1) per field | O(log n) | O(1) | O(1) |
+| **Exact lookup** | O(1) | N/A | O(1) | O(1) |
+| **Range query** | O(log n) + O(k) | N/A | N/A | N/A |
+| **Similarity search** | N/A | O(log n) | N/A | N/A |
+| **Neighbor lookup** | N/A | N/A | O(1) | N/A |
+| **Statistics** | O(1) | O(1) | O(1) | O(1) |
 
 Where:
 - n = total number of entities
 - k = number of matching results
 
-**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for the vector index).
-
 ### Memory Footprint
 
 | Index | Per-Entity Memory | Notes |
 |-------|-------------------|-------|
-| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) |
-| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
-| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional verb-id references in 2 LSM-trees |
+| **MetadataIndex** | ~100 bytes | Depends on field count and cardinality |
+| **HNSWIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections |
+| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional references + metadata |
+| **DeletedItemsIndex** | ~40 bytes per deleted ID | Just Set storage |
 
 **Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship
 
-**Sub-index memory:**
-- ChunkManager: ~20 bytes per chunk descriptor
-- EntityIdMapper: ~32 bytes per UUID mapping (50-90% savings vs Set\<string\>)
-- LSM-trees: ~200 bytes per relationship (SSTable storage)
-
 ### Scalability
 
-All indexes scale gracefully. The cost of each stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion:
+All indexes scale gracefully:
 
-| Query stage | Complexity | Scaling behavior |
-|-------------|------------|------------------|
-| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
-| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
-| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
-| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
-| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
+| Database Size | Metadata Filter | Vector Search | Graph Hop | Combined Query |
+|---------------|----------------|---------------|-----------|----------------|
+| **1K entities** | 0.3ms | 0.8ms | 0.05ms | 1.1ms |
+| **10K entities** | 0.5ms | 1.2ms | 0.08ms | 1.5ms |
+| **100K entities** | 0.8ms | 1.8ms | 0.1ms | 2.1ms |
+| **1M entities** | 1.2ms | 2.5ms | 0.1ms | 2.8ms |
 
 **Key observations**:
 - Graph queries stay O(1) regardless of scale
 - Metadata filtering scales sub-linearly
-- Vector search degrades gracefully due to the hierarchical index
+- Vector search degrades gracefully due to HNSW
 - Combined queries remain fast even at scale
 
 ## Best Practices
@@ -868,7 +829,7 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith
 - Field discovery (what filters are available)
 - Type-based querying (find all characters, all items)
 
-**Vector Index**:
+**HNSWIndex**:
 - Semantic similarity search ("find similar documents")
 - Content-based retrieval ("find posts about AI")
 - Fuzzy matching (when exact matches aren't required)
@@ -880,7 +841,10 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith
 - Network analysis ("find communities")
 - Multi-hop traversal ("friends of friends")
 
-**Note**: Soft-delete functionality is not currently integrated. Brainy uses hard deletes via storage layer.
+**DeletedItemsIndex**:
+- Soft deletes (preserve data but hide from queries)
+- Audit trails (track what was deleted when)
+- Restoration workflows (undo deletions)
 
 ### Query Optimization
 
@@ -893,9 +857,9 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith
 ### Memory Management
 
 1. **Configure UnifiedCache appropriately** - Balance between speed and memory
-2. **Use lazy loading** - Vector index loads vectors on-demand
+2. **Use lazy loading** - HNSW loads vectors on-demand
 3. **Monitor cache hit rates** - Adjust cache size if hit rate is low
-4. **Consider storage adapter** - Memory = fastest, filesystem = persistent
+4. **Consider storage adapter** - Memory storage = fastest, S3 = most scalable
 
 ## Related Documentation
 
@@ -905,37 +869,10 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith
 - [Performance Guide](../PERFORMANCE.md) - Performance tuning
 - [Overview](./overview.md) - High-level architecture
 
-## Summary: Index Hierarchy
-
-### Level 1: Main Indexes (3)
-All have rebuild() methods and are covered by lazy loading:
-1. **TypeAwareVectorIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403`
-2. **MetadataIndexManager** - `src/utils/metadataIndex.ts:2318`
-3. **GraphAdjacencyIndex** - `src/graph/graphAdjacencyIndex.ts:389`
-
-### Level 2: Sub-Indexes (~50+)
-Automatically managed by parent rebuild():
-- **42 type-specific vector indexes** (one per NounType)
-- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes)
-- **2 LSM-trees** (lsmTreeVerbsBySource, lsmTreeVerbsByTarget — the verb set is the single adjacency source of truth; neighbor reads derive from live verbs so removals are honored)
-- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
-
-### Lazy Loading
-- **Mode 1**: Auto-rebuild on init() (default)
-- **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`)
-- **Concurrency-safe**: Mutex prevents duplicate rebuilds
-- **Performance**: First query ~50-200ms, subsequent queries instant
-
-### Total Functional Index Count
-- **3 main indexes** with independent rebuild() methods
-- **~50+ sub-components** managed automatically
-- **All covered** by rebuildIndexesIfNeeded() or built-in lazy initialization
-
 ## Version History
 
-- **v5.7.7** (November 2025): Added production-scale lazy loading with concurrency control. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added `ensureIndexesLoaded()` helper and `getIndexStatus()` diagnostic.
 - **v3.43.0** (October 2025): Migrated from `roaring` (native C++) to `roaring-wasm` (WebAssembly) for universal compatibility. No API changes - maintains identical RoaringBitmap32 interface. Benefits: works in all environments (Node.js, browsers, serverless) without build tools, zero compilation errors, simpler developer experience. 90% memory savings and hardware-accelerated operations unchanged.
 - **v3.42.0** (October 2025): Replaced flat file indexing with adaptive chunked sparse indexing. Bloom filters + zone maps for O(1) exact match and O(log n) range queries. 630x file reduction (560k → 89 files). Removed dual code paths.
 - **v3.41.0** (October 2025): Added automatic temporal bucketing to MetadataIndex
 - **v3.40.0** (October 2025): Enhanced batch processing for imports
-- **v3.0.0** (September 2025): Introduced 3-tier index architecture with UnifiedCache
+- **v3.0.0** (September 2025): Introduced 4-index architecture with UnifiedCache

docs/architecture/initialization-and-rebuild.md

@@ -1,6 +1,6 @@
 # Initialization and Rebuild Processes
 
-This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
+This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
 
 ## Core Principle: All Indexes Are Disk-Based
 
@@ -10,29 +10,12 @@ This document explains how Brainy's four indexes (MetadataIndex, vector index, G
 
 | Index | Persisted Data | Storage Method | Since Version |
 |-------|---------------|----------------|---------------|
-| **MetadataIndex** | Field registry + chunked sparse indices with bloom filters + zone maps | `storage.saveMetadata()` | v3.42.0 (chunks), v4.2.1 (registry) |
-| **Vector Index** | Vector embeddings + graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
+| **MetadataIndex** | Chunked sparse indices with bloom filters + zone maps | `storage.saveMetadata()` | v3.42.0 |
+| **HNSWIndex** | Vector embeddings + HNSW graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
 | **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
 | **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
 
-#### MetadataIndex Persistence Details
-
-The MetadataIndex now persists two components:
-
-1. **Field Registry** (`__metadata_field_registry__`): Directory of indexed fields for O(1) discovery
-   - Size: ~4-8KB (50-200 fields typical)
-   - Enables instant cold starts by discovering persisted indices
-   - Auto-saved during every flush operation
-
-2. **Sparse Indices** (`__sparse_index__<field>`): Per-field index directories
-   - Contains chunk metadata, zone maps, and bloom filters
-   - Lazy-loaded via UnifiedCache on first query
-
-3. **Chunks** (`__metadata_chunk__<field>_<chunkId>`): Actual inverted index data
-   - Roaring bitmaps for compressed entity ID storage
-   - Loaded on-demand based on query patterns
-
-All storage operations use the **StorageAdapter** interface, which works with FileSystem and Memory backends.
+All storage operations use the **StorageAdapter** interface, which works with FileSystem, OPFS, S3, GCS, R2, and Memory backends.
 
 ## Initialization Process
 
@@ -70,26 +53,21 @@ class GraphAdjacencyIndex {
 
 ### 2. Brain Initialization Flow
 
-When you create a `Brain` instance and call `init()`, behavior depends on the `disableAutoRebuild` configuration:
-
-#### Mode 1: Auto-Rebuild on init() (Default)
+When you create a `Brain` instance and call `init()`:
 
 ```typescript
-// src/brainy.ts (lines 192-237)
+// src/brainy.ts (lines 2900-3035)
 async init(): Promise<void> {
   const initStartTime = Date.now()
 
-  // STEP 1: Initialize storage and unified cache
-  await this.storage.init()
-
-  // STEP 2: Check index sizes (lazy initialization triggers here)
+  // STEP 1: Check index sizes (lazy initialization triggers here)
   const metadataStats = await this.metadataIndex.getStats()
-  const vectorIndexSize = this.index.size()
+  const hnswIndexSize = this.index.size()
   const graphIndexSize = await this.graphIndex.size()
 
-  // STEP 3: Rebuild empty indexes from storage in parallel
+  // STEP 2: Rebuild empty indexes from storage in parallel
   if (metadataStats.totalEntries === 0 ||
-      vectorIndexSize === 0 ||
+      hnswIndexSize === 0 ||
       graphIndexSize === 0) {
 
     const rebuildStartTime = Date.now()
@@ -97,7 +75,7 @@ async init(): Promise<void> {
       metadataStats.totalEntries === 0
         ? this.metadataIndex.rebuild()
         : Promise.resolve(),
-      vectorIndexSize === 0
+      hnswIndexSize === 0
         ? this.index.rebuild()
         : Promise.resolve(),
       graphIndexSize === 0
@@ -109,7 +87,7 @@ async init(): Promise<void> {
     console.log(`✅ All indexes rebuilt in ${rebuildDuration}ms`)
   }
 
-  // STEP 4: Log statistics
+  // STEP 3: Log statistics
   const stats = await this.stats()
   console.log(`📊 Brain initialized with ${stats.entities} entities`)
 }
@@ -117,103 +95,22 @@ async init(): Promise<void> {
 
 **Timeline** (typical cold start with 10K entities):
 - 0-50ms: Storage adapter initialization
-- 50-100ms: Field registry loading (O(1) discovery of persisted indices)
-- 100-200ms: Index lazy initialization (LSM-tree loading)
-- 200-500ms: Cache warming (preload common fields)
-- **No rebuild needed!** Registry discovers existing indices
-- Total: ~0.5-1 second (instant cold starts)
-
-**Timeline** (cold start WITHOUT field registry - first run only):
-- 0-50ms: Storage adapter initialization
-- 50-100ms: Index lazy initialization
-- 100-2000ms: One-time rebuild to create indices
-- Total: ~1-3 seconds (one time only)
-
-#### Mode 2: Lazy Loading on First Query
-
-When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query:
-
-```typescript
-// User code
-const brain = new Brainy({
-  storage: { type: 'filesystem' },
-  disableAutoRebuild: true  // Enable lazy loading
-})
-
-await brain.init()  // Returns instantly (0-10ms)
-
-// First query triggers lazy rebuild
-const results = await brain.find({ limit: 10 })
-// → Calls ensureIndexesLoaded() internally (brainy.ts:4617)
-// → Rebuilds all 3 main indexes with concurrency control
-// → Returns results (~50-200ms total for 1K-10K entities)
-
-// Subsequent queries are instant
-const more = await brain.find({ limit: 100 })  // 0ms check, instant
-```
-
-**ensureIndexesLoaded() Implementation** (brainy.ts:4617-4664):
-```typescript
-private async ensureIndexesLoaded(): Promise<void> {
-  // Fast path: Already loaded
-  if (this.lazyRebuildCompleted) {
-    return  // 0ms
-  }
-
-  // Concurrency control: Wait for in-progress rebuild
-  if (this.lazyRebuildInProgress && this.lazyRebuildPromise) {
-    await this.lazyRebuildPromise  // Wait for same rebuild
-    return
-  }
-
-  // Check if storage has data
-  const entities = await this.storage.getNouns({ pagination: { limit: 1 } })
-  const hasData = (entities.totalCount && entities.totalCount > 0) || entities.items.length > 0
-
-  if (!hasData) {
-    this.lazyRebuildCompleted = true
-    return
-  }
-
-  // Start lazy rebuild with mutex
-  this.lazyRebuildInProgress = true
-  this.lazyRebuildPromise = this.rebuildIndexesIfNeeded(true)
-    .then(() => {
-      this.lazyRebuildCompleted = true
-    })
-    .finally(() => {
-      this.lazyRebuildInProgress = false
-      this.lazyRebuildPromise = null
-    })
-
-  await this.lazyRebuildPromise
-}
-```
-
-**Lazy Loading Performance:**
-- First query: ~50-200ms (1K-10K entities) - triggers rebuild
-- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
-- Subsequent queries: 0ms check (instant)
-- Zero-config: Works automatically, no code changes needed
-
-**Use Cases for Lazy Loading:**
-- **Serverless/Edge**: Minimize cold start time, indexes load on demand
-- **Development**: Faster restarts during development
-- **Large datasets**: Defer index loading until actually needed
-- **Read-heavy workloads**: Write operations don't wait for index rebuild
+- 50-100ms: Index lazy initialization (LSM-tree loading, metadata discovery)
+- 100-1500ms: Parallel rebuild if needed
+- Total: ~1-3 seconds
 
 ## Rebuild Process
 
 ### What "Rebuild" Actually Means
 
 **IMPORTANT**: "Rebuild" does NOT mean recomputing data. It means:
-1. **Load persisted data** from storage (vector index connections, metadata chunks, LSM-tree SSTables)
+1. **Load persisted data** from storage (HNSW connections, metadata chunks, LSM-tree SSTables)
 2. **Populate in-memory structures** (Maps, Sets, graphs)
 3. **Apply adaptive caching** (preload vectors if small dataset, lazy load if large)
 
 **Complexity**: O(N) - linear scan through storage, NOT O(N log N) recomputation!
 
-### 1. Vector Index Rebuild (Correct Pattern)
+### 1. HNSWIndex Rebuild (Correct Pattern)
 
 ```typescript
 // src/hnsw/hnswIndex.ts (lines 809-947)
@@ -236,7 +133,7 @@ public async rebuild(options: {
   const availableCache = this.unifiedCache.getRemainingCapacity()
   const shouldPreload = vectorMemory < availableCache * 0.3
 
-  // STEP 4: Load entities with persisted vector index connections
+  // STEP 4: Load entities with persisted HNSW connections
   let hasMore = true
   let cursor: string | undefined = undefined
 
@@ -246,7 +143,7 @@ public async rebuild(options: {
     })
 
     for (const nounData of result.items) {
-      // Load vector graph data from storage (NOT recomputed!)
+      // Load HNSW graph data from storage (NOT recomputed!)
       const hnswData = await this.storage.getHNSWData(nounData.id)
 
       // Create noun with restored connections
@@ -274,14 +171,14 @@ public async rebuild(options: {
 ```
 
 **Key Points**:
-- ✅ Loads vector index connections from storage via `getHNSWData()`
+- ✅ Loads HNSW connections from storage via `getHNSWData()`
 - ✅ Uses adaptive caching (preload vectors if < 30% of available cache)
 - ✅ O(N) complexity - just loads existing data
 - ❌ Does NOT call `addItem()` which would recompute connections (O(N log N))
 
-### 2. TypeAwareVectorIndex Rebuild (Fixed in v3.45.0)
+### 2. TypeAwareHNSWIndex Rebuild (Fixed in v3.45.0)
 
-**Critical Architectural Fix**: The type-aware vector index previously had TWO major bugs:
+**Critical Architectural Fix**: TypeAwareHNSWIndex previously had TWO major bugs:
 
 1. **Bug #1**: Called `addItem()` during rebuild → O(N log N) recomputation instead of O(N) loading
 2. **Bug #2**: Loaded ALL nouns 31 times in parallel (once per type) → O(31*N) complexity causing timeouts
@@ -300,7 +197,7 @@ public async rebuild(options?: {
     index.clear()
   }
 
-  // STEP 2: Determine preloading strategy (same as vector index)
+  // STEP 2: Determine preloading strategy (same as HNSWIndex)
   const totalNouns = await this.storage.getNounCount()
   const vectorMemory = totalNouns * 384 * 4
   const availableCache = this.unifiedCache.getRemainingCapacity()
@@ -319,7 +216,7 @@ public async rebuild(options?: {
       })
 
       for (const nounData of result.items) {
-        // CORRECT: Load persisted vector index data (not recomputed!)
+        // CORRECT: Load persisted HNSW data (not recomputed!)
         const hnswData = await this.storage.getHNSWData(nounData.id)
 
         const noun = {
@@ -350,7 +247,7 @@ public async rebuild(options?: {
 
 **Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities)
 
-**Correct Pattern**:
+**Correct Pattern** (v3.45.0):
 ```typescript
 // Load ALL nouns ONCE (not 31 times!)
 while (hasMore) {
@@ -385,77 +282,36 @@ while (hasMore) {
 - 200-600x speedup: Load from storage instead of recomputing (O(N) vs O(N log N))
 - **Combined**: ~6000x speedup! (150 minutes → 1.5 seconds for 10K entities)
 
-### 3. MetadataIndex Rebuild (v4.2.1+ with Field Registry)
-
-**v4.2.1 Critical Fix**: Field registry persistence eliminates unnecessary rebuilds!
+### 3. MetadataIndex Rebuild
 
 ```typescript
-// src/utils/metadataIndex.ts (lines 202-216)
-async init(): Promise<void> {
-  // STEP 1: Load field registry to discover persisted indices
-  // This is THE KEY FIX - O(1) discovery of existing indices
-  await this.loadFieldRegistry()
-
-  // If registry found, fieldIndexes Map is now populated
-  // getStats() will return totalEntries > 0 → skips rebuild!
-
-  // STEP 2: Initialize EntityIdMapper
-  await this.idMapper.init()
-
-  // STEP 3: Warm cache with discovered fields
-  await this.warmCache()
-}
-
-async loadFieldRegistry(): Promise<void> {
-  const registry = await this.storage.getMetadata('__metadata_field_registry__')
-
-  if (registry?.fields) {
-    // Populate fieldIndexes Map from discovered fields
-    // Sparse indices are lazy-loaded when first accessed
-    for (const field of registry.fields) {
-      this.fieldIndexes.set(field, {
-        values: {},
-        lastUpdated: registry.lastUpdated
-      })
-    }
-    // Result: getStats() now returns totalEntries > 0
-    // → Brain skips rebuild, cold start in 2-3 seconds!
-  }
-}
-```
-
-**Rebuild Only Happens If**:
-1. **First run** (no field registry exists yet)
-2. **Registry corruption** (rare)
-3. **Explicit rebuild request** (manual operation)
-
-```typescript
-// Only runs if field registry not found
+// src/utils/metadataIndex.ts
 async rebuild(): Promise<void> {
   // STEP 1: Clear in-memory structures
   this.fieldIndexes.clear()
+  this.sparseIndices.clear()
 
-  // STEP 2: Load all entity metadata and rebuild indices
-  // Sequential batching (25/batch) to prevent socket exhaustion
-  // After rebuild: Field registry saved during next flush()
+  // STEP 2: Load chunked sparse indices from storage
+  // Note: Chunks are lazy-loaded on demand, so rebuild is fast
+  const fields = await this.storage.getIndexedFields()
 
-  // One-time cost: ~2-3 seconds for 1K entities
+  for (const field of fields) {
+    // Load sparse index metadata (chunk descriptors, bloom filters)
+    const sparseIndex = await this.storage.getSparseIndex(field)
+    this.sparseIndices.set(field, sparseIndex)
+  }
+
+  // STEP 3: Load lightweight statistics
+  const stats = await this.storage.getMetadataStats()
+  this.fieldStats = stats.fieldStats
+  this.typeFieldAffinity = stats.typeFieldAffinity
 }
 ```
 
-**Performance Comparison**:
-
-| Version | Cold Start | Discovery Method | Rebuild Needed? |
-|---------|------------|------------------|-----------------|
-| v4.2.0 | 8-9 min | None (always rebuild) | Always |
-| v4.2.1 | 2-3 sec | Field registry O(1) | First run only |
-
 **Key Points**:
-- ✅ Field registry enables O(1) discovery (4-8KB file)
-- ✅ Sparse indices lazy-loaded on first query
+- ✅ Lazy chunk loading - only loads chunks when queried
 - ✅ Bloom filters + zone maps loaded for fast filtering
-- ✅ One-time rebuild on first run, then instant restarts forever
-- ✅ Automatic: No configuration needed
+- ✅ O(F) complexity where F = number of fields (typically < 100)
 
 ### 4. GraphAdjacencyIndex Rebuild
 
@@ -533,7 +389,7 @@ const unifiedCache = getGlobalCache()  // Singleton, 100MB default
 // MetadataIndex
 this.unifiedCache = unifiedCache
 
-// Vector index
+// HNSWIndex
 this.unifiedCache = unifiedCache
 
 // GraphAdjacencyIndex
@@ -549,7 +405,7 @@ this.unifiedCache = unifiedCache
 
 ### Rebuild Times (Typical Hardware)
 
-| Dataset Size | Metadata | Vector | Graph | Total (Parallel) |
+| Dataset Size | Metadata | HNSW | Graph | Total (Parallel) |
 |--------------|----------|------|-------|------------------|
 | 1K entities | 50ms | 100ms | 30ms | **150ms** |
 | 10K entities | 200ms | 500ms | 150ms | **600ms** |
@@ -563,7 +419,7 @@ this.unifiedCache = unifiedCache
 | Index | In-Memory Overhead | Disk Storage |
 |-------|-------------------|--------------|
 | **MetadataIndex** | ~100 bytes/entity | ~500 bytes/entity (chunks) |
-| **Vector Index** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
+| **HNSWIndex** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
 | **GraphAdjacencyIndex** | ~128 bytes/relationship | ~200 bytes/relationship (LSM-tree) |
 | **DeletedItemsIndex** | ~40 bytes/deleted ID | ~50 bytes/deleted ID |
 
@@ -573,9 +429,9 @@ this.unifiedCache = unifiedCache
 
 ### O(N) vs O(N log N) Comparison
 
-**Before fix** (TypeAwareVectorIndex bug):
+**Before fix** (TypeAwareHNSWIndex bug):
 ```typescript
-// BAD: Recomputes vector index connections during rebuild
+// BAD: Recomputes HNSW connections during rebuild
 for (const noun of nouns) {
   await index.addItem(noun)  // O(log N) per item → O(N log N) total
 }
@@ -652,7 +508,7 @@ console.timeEnd('rebuild')
 
 // For 10K entities:
 // - Expected: 500-800ms (loading from storage)
-// - Bug: 5-10 minutes (recomputing vector index connections)
+// - Bug: 5-10 minutes (recomputing HNSW connections)
 ```
 
 **Solution**: Ensure index is loading from storage, not calling `addItem()` during rebuild.
@@ -705,9 +561,8 @@ console.log('Nouns in storage:', nouns.items.length)
 
 ## Version History
 
-- **v5.7.7** (November 2025): Added production-scale lazy loading with `ensureIndexesLoaded()` helper. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added concurrency control (mutex) to prevent duplicate rebuilds from concurrent queries. Added `getIndexStatus()` diagnostic method. Zero-config operation - works automatically.
-- **v3.45.0** (October 2025): Fixed type-aware vector index `rebuild()` to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
+- **v3.45.0** (October 2025): Fixed TypeAwareHNSWIndex.rebuild() to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
 - **v3.44.0** (October 2025): GraphAdjacencyIndex migrated to LSM-tree storage for billion-scale relationships
 - **v3.42.0** (October 2025): MetadataIndex migrated to chunked sparse indexing
-- **v3.35.0** (August 2025): Vector index connections first persisted to storage
-- **v3.0.0** (September 2025): Initial 3-tier index architecture
+- **v3.35.0** (August 2025): HNSW connections first persisted to storage
+- **v3.0.0** (September 2025): Initial 4-index architecture

docs/architecture/multiprocess-storage-mixin.md

@@ -1,134 +0,0 @@
-# Design note: multi-process storage mixin
-
-**Status:** Proposed (future minor)
-**Owner:** Brainy core
-**Filed:** 2026-05-15
-**Companion:** [`concepts/storage-adapters`](../concepts/storage-adapters.md)
-
-## Context
-
-Brainy 7.21 added seven storage-adapter methods to support multi-process
-safety:
-
-```
-supportsMultiProcessLocking()
-acquireWriterLock(opts)
-releaseWriterLock()
-readWriterLock()
-startFlushRequestWatcher(cb)
-stopFlushRequestWatcher()
-requestFlushOverFilesystem(timeoutMs)
-```
-
-They live on `BaseStorage` as no-op defaults and are overridden on
-`FileSystemStorage` with real implementations. Any adapter extending
-`FileSystemStorage` (e.g. Cor's `MmapFileSystemStorage`) inherits the
-real ones for free.
-
-This works correctly today. The question is whether the methods *belong*
-on `BaseStorage`.
-
-## The case for moving them out
-
-`BaseStorage` already mixes several concerns:
-- entity / verb CRUD primitives
-- generational record hooks (8.0 MVCC)
-- type-statistics tracking
-- count persistence
-- multi-process safety (new)
-
-Adapters that have no notion of multi-process semantics — `MemoryStorage`,
-cloud adapters (S3, GCS, R2, Azure, OPFS) — still carry seven inherited
-no-ops on their prototype chain. A reader can't tell from the class
-declaration whether a given adapter participates in the locking protocol;
-it has to call `supportsMultiProcessLocking()` and trust the answer.
-
-A cleaner separation:
-
-```typescript
-interface MultiProcessSafeStorage {
-  supportsMultiProcessLocking(): boolean
-  acquireWriterLock(opts?: { force?: boolean }): Promise<WriterLockInfo | null>
-  releaseWriterLock(): Promise<void>
-  readWriterLock(): Promise<WriterLockInfo | null>
-  startFlushRequestWatcher(cb: () => Promise<void>): void
-  stopFlushRequestWatcher(): void
-  requestFlushOverFilesystem(timeoutMs: number): Promise<boolean>
-}
-
-function isMultiProcessSafe(s: BaseStorage): s is BaseStorage & MultiProcessSafeStorage {
-  return typeof (s as any).supportsMultiProcessLocking === 'function'
-    && (s as any).supportsMultiProcessLocking()
-}
-```
-
-Brainy's call sites become:
-
-```typescript
-if (this.config.mode !== 'reader' && isMultiProcessSafe(this.storage)) {
-  await this.storage.acquireWriterLock({ force: this.config.force })
-  // ... TypeScript narrows the rest correctly ...
-}
-```
-
-Benefits:
-- Type system enforces the capability — no more `(this.storage as any).X()`.
-- Adapters that opt out (memory, cloud) are visibly clean.
-- `hasStorageMethod()` defensive helper can stay (still guards
-  build/install artifacts) but doesn't carry the conceptual weight of
-  "did the plugin implement the interface."
-- ADR-style trail for future capability additions: each new capability
-  gets its own interface, opted into explicitly.
-
-## The case against doing it now
-
-- Breaking change for any adapter that already overrides these methods.
-  `FileSystemStorage` is the only one in-tree, but Cor's
-  `MmapFileSystemStorage` inherits from it — interface relocation would
-  ripple through the plugin ecosystem.
-- The current state works. The real failure modes seen in the field
-  were build/install artifacts, not type-system failures.
-- 7.22.0 just shipped a clean fix. Stacking another refactor before
-  consumers absorb it adds churn without urgency.
-- The `hasStorageMethod()` guard accomplishes the same runtime safety the
-  interface narrowing would in TypeScript-aware code.
-
-## Recommendation
-
-**Defer.** Keep the current architecture through the 7.x line. Revisit
-when:
-- A second multi-process capability lands (e.g. distributed-readers
-  coordination) and the natural surface area is more than seven
-  methods. Five+ becomes the moment a separate interface earns its
-  keep.
-- A v8 major is on the table for unrelated reasons. Bundle the
-  interface extraction with that release so consumers absorb both
-  changes in one upgrade.
-
-Until then:
-- Document the inheritance contract (done — see
-  [`concepts/storage-adapters`](../concepts/storage-adapters.md)).
-- Keep `hasStorageMethod()` as the runtime guard.
-- Don't add new methods to `BaseStorage` defaults without re-evaluating
-  the surface-area boundary.
-
-## Migration sketch (when we do it)
-
-For reference, a clean migration path:
-
-1. Add `MultiProcessSafeStorage` interface to `src/storage/coreTypes.ts`.
-2. Move the seven method signatures from `BaseStorage` to the new
-   interface. Default implementations stay on `BaseStorage` but only as
-   private helpers consumed by `FileSystemStorage`'s explicit
-   implementations.
-3. `FileSystemStorage implements MultiProcessSafeStorage` becomes
-   explicit; methods get the `public` modifier with full JSDoc.
-4. Brainy call sites switch from `hasStorageMethod` to
-   `isMultiProcessSafe` type-guard. Keep `hasStorageMethod` for
-   build/install artifact protection.
-5. Document the new contract in `concepts/storage-adapters.md`.
-6. Major-version-bump the `@soulcraft/brainy` peerDep range expected by
-   plugins.
-
-Estimated work: ~half a day of code, ~2 hours of doc/example updates,
-ecosystem coordination via the platform handoff.

docs/architecture/overview.md

@@ -6,14 +6,14 @@ Brainy is a multi-dimensional AI database that combines vector similarity, graph
 
 ### Brainy (Main Entry Point)
 The central orchestrator that manages all subsystems:
-- **4-Index Architecture**: MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md))
-- **Storage System**: FileSystem and Memory adapters
+- **4-Index Architecture**: MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md))
+- **Storage System**: Universal storage adapters (FileSystem, S3, OPFS, Memory)
 - **Augmentation System**: Extensible plugin architecture
 - **Triple Intelligence**: Unified query engine
 
 ### Triple Intelligence Engine
 Brainy's revolutionary feature that unifies three types of search:
-- **Vector Search**: Semantic similarity via the pluggable vector index
+- **Vector Search**: Semantic similarity using HNSW indexing
 - **Graph Traversal**: Relationship-based queries
 - **Field Filtering**: Precise metadata filtering with O(1) performance
 
@@ -42,13 +42,12 @@ brainy-data/
 └── locks/            # Concurrent access control
 ```
 
-### Vector Index
-Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph:
+### HNSW Index
+Hierarchical Navigable Small World index for efficient vector search:
 - **Performance**: O(log n) search complexity
-- **Configurable recall**: `fast` / `balanced` / `accurate` presets trade recall for latency
-- **Scalable**: Handles millions of vectors per process
+- **Memory Efficient**: Product quantization support
+- **Scalable**: Handles millions of vectors
 - **Persistent**: Serializable to storage
-- **Swappable**: Replace with a native implementation (such as `@soulcraft/cor`) via the plugin system without changing application code
 
 ### Metadata Index Manager
 High-performance field indexing system:
@@ -60,7 +59,7 @@ High-performance field indexing system:
 ## Performance Characteristics
 
 ### Operation Complexity
-- **Vector Search**: O(log n) via the vector index
+- **Vector Search**: O(log n) via HNSW
 - **Field Filtering**: O(1) via inverted indexes
 - **Graph Traversal**: O(V + E) for breadth-first search
 - **Add Operation**: O(log n) for index insertion
@@ -84,6 +83,7 @@ Brainy's extensible plugin architecture allows for powerful enhancements:
 ### Core Augmentations
 - **Entity Registry**: High-speed deduplication for streaming data
 - **Batch Processing**: Optimized bulk operations
+- **Connection Pool**: Efficient resource management
 - **Request Deduplicator**: Prevents duplicate processing
 
 ### Creating Custom Augmentations
@@ -111,7 +111,7 @@ Multi-layered caching for optimal performance:
 ## Integration Points
 
 ### Key Objects for Extensions
-- `brain.index`: Access the vector index
+- `brain.index`: Access HNSW vector index
 - `brain.metadataIndex`: Access field indexing
 - `brain.graphIndex`: Access graph adjacency index
 - `brain.storage`: Access storage layer

docs/architecture/storage-architecture.md

@@ -1,72 +1,31 @@
 # Storage Architecture
 
-> **Updated**: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup
 
 ## Storage Structure
 
-### Architecture: Metadata/Vector Separation
-
-Entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
-
 ```
 brainy-data/
-├── _system/                              # System metadata (not sharded)
-│   ├── statistics.json                   # Performance metrics
-│   ├── __metadata_field_index__*.json    # Field indexes
-│   └── __metadata_sorted_index__*.json   # Sorted indexes
-│
-├── entities/
-│   ├── nouns/
-│   │   ├── vectors/                      # Vector graph data (sharded by UUID)
-│   │   │   ├── 00/                       # Shard 00 (first 2 hex digits)
-│   │   │   │   ├── 00123456-....json     # Vector + graph connections
-│   │   │   │   └── 00abcdef-....json
-│   │   │   ├── 01/ ... ff/               # 256 shards total
-│   │   │
-│   │   └── metadata/                     # Business data (sharded by UUID)
-│   │       ├── 00/
-│   │       │   ├── 00123456-....json     # Entity metadata only
-│   │       │   └── 00abcdef-....json
-│   │       ├── 01/ ... ff/
-│   │
-│   └── verbs/
-│       ├── vectors/                      # Relationship vectors (sharded)
-│       │   ├── 00/ ... ff/
-│       │
-│       └── metadata/                     # Relationship data (sharded)
-│           ├── 00/ ... ff/
+├── _system/                        # System management
+│   └── statistics.json            # Performance metrics and statistics
+├── nouns/                         # Primary entity storage
+│   └── {uuid}.json               # Individual entity documents
+├── metadata/                      # Metadata and indexing system
+│   ├── {uuid}.json               # Entity metadata
+│   ├── __entity_registry__.json  # Entity deduplication registry
+│   ├── __metadata_field_index__field_{field}.json    # Field discovery
+│   └── __metadata_index__{field}_{value}_chunk{n}.json # Value indexes
+├── verbs/                         # Relationship/action storage
+│   └── {uuid}.json               # Relationship documents
+│   └── wal_{timestamp}_{id}.wal # Transaction logs
+└── locks/                        # Concurrent access control
+    └── {resource}.lock          # Resource locks
 ```
 
-### Why Split Metadata and Vectors?
-
-**Performance at scale:**
-- **Vector search operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
-- **Filtering**: Only load metadata during filtering, not vectors
-- **Pagination**: Load metadata IDs first, fetch vectors/metadata on-demand
-- **Result**: 60-70% reduction in I/O for typical queries at million-entity scale
-
-### UUID-Based Sharding (256 Shards)
-
-**How it works:**
-```typescript
-const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
-const shard = uuid.substring(0, 2) // "3f"
-
-// Vector path:   entities/nouns/vectors/3f/3fa85f64-....json
-// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
-```
-
-**Benefits:**
-- **Uniform distribution**: ~3,900 entities per shard (at 1M scale)
-- **Filesystem optimization**: avoids huge flat directories that bog down `readdir`
-- **Parallel operations**: walk 256 shards in parallel
-- **Predictable**: Deterministic shard assignment
-
 ## Storage Adapters
 
-Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` interface:
+Brainy provides multiple storage adapters with identical APIs:
 
-### FileSystem Storage (Node.js, default)
+### FileSystem Storage (Node.js)
 ```typescript
 const brain = new Brainy({
   storage: {
@@ -75,46 +34,39 @@ const brain = new Brainy({
   }
 })
 ```
-- **Use case**: Server applications, CLI tools, single-node deployments
+- **Use case**: Server applications, CLI tools
 - **Performance**: Direct file I/O
 - **Persistence**: Permanent on disk
-- **Features**:
-  - **Batch Delete**: Efficient bulk deletion with retries
-  - **UUID Sharding**: Automatic 256-shard distribution
 
-### Memory Storage
+### S3 Compatible Storage
 ```typescript
 const brain = new Brainy({
   storage: {
-    type: 'memory'
+    type: 's3',
+    bucket: 'my-brainy-data',
+    region: 'us-east-1',
+    credentials: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
+    }
   }
 })
 ```
-- **Use case**: Tests, ephemeral workloads, single-process caches
-- **Performance**: No I/O — all data lives in process memory
-- **Persistence**: None — data is lost when the process exits
+- **Use case**: Distributed applications, cloud deployments
+- **Performance**: Network dependent, with intelligent caching
+- **Persistence**: Cloud storage durability
 
-### Auto
+### Origin Private File System (Browser)
 ```typescript
 const brain = new Brainy({
   storage: {
-    type: 'auto',
-    path: './data'
+    type: 'opfs'
   }
 })
 ```
-`'auto'` picks `'filesystem'` when running on Node.js with a writable `path`, and falls back to `'memory'` otherwise.
-
-## Backup and Off-Site Replication
-
-Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `path` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:
-
-- `gsutil rsync -r ./data gs://my-bucket/brainy-data`
-- `aws s3 sync ./data s3://my-bucket/brainy-data`
-- `rclone sync ./data remote:brainy-data`
-- Periodic `tar` snapshots to any object store
-
-Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory.
+- **Use case**: Browser applications, PWAs
+- **Performance**: Near-native file system speed
+- **Persistence**: Permanent in browser (with quota limits)
 
 ## Metadata Indexing System
 
@@ -178,22 +130,51 @@ High-performance deduplication system for streaming data:
 - **Cache**: LRU with configurable TTL
 - **Sync**: Periodic or on-demand
 
-## Durability
 
-Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `path` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).
+Ensures durability and enables recovery:
+
+```json
+{
+  "timestamp": 1699564234567,
+  "operation": "add",
+  "data": {
+    "id": "550e8400-e29b-41d4-a716-446655440000",
+    "content": "...",
+    "metadata": {}
+  },
+  "checksum": "sha256:..."
+}
+```
+
+### Recovery Process
+2. Replay operations from last checkpoint
+3. Verify checksums for integrity
 
 ## Storage Optimization
 
-### 1. Batch Operations
+### Compression
+- **JSON**: Automatic minification
+- **Vectors**: Float32 to Uint8 quantization option
+- **Indexes**: Binary format for large datasets
 
+### Caching Strategy
 ```typescript
-// Efficient batch delete
-await storage.batchDelete([
-  'entities/nouns/vectors/00/00123456-....json',
-  'entities/nouns/metadata/00/00123456-....json'
-  // ...
-])
+// Configure caching per storage type
+const brain = new Brainy({
+  storage: {
+    type: 'filesystem',
+    cache: {
+      enabled: true,
+      maxSize: 1000,      // Maximum cached items
+      ttl: 300000,        // 5 minutes
+      strategy: 'lru'     // Least recently used
+    }
+  }
+})
+```
 
+### Batch Operations
+```typescript
 // Batch writes for performance
 await brain.addBatch([
   { content: "item1", metadata: {} },
@@ -203,24 +184,6 @@ await brain.addBatch([
 // Single transaction, optimized I/O
 ```
 
-### 2. Caching Strategy
-
-```typescript
-// Configure caching
-const brain = new Brainy({
-  storage: {
-    type: 'filesystem',
-    path: './data',
-    cache: {
-      enabled: true,
-      maxSize: 1000,   // Maximum cached items
-      ttl: 300000,     // 5 minutes
-      strategy: 'lru'  // Least recently used
-    }
-  }
-})
-```
-
 ## Concurrent Access
 
 ### Locking Mechanism
@@ -239,39 +202,58 @@ await brain.storage.withLock('resource-id', async () => {
 
 ## Migration and Backup
 
-Backup and restore go through the Db API — see
-[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) for the full
-recipe book.
-
-### Snapshot (backup)
+### Export Data
 ```typescript
-// Instant, self-contained snapshot (hard links on filesystem storage)
-const db = brain.now()
-await db.persist('/backups/2026-06-11')
-await db.release()
+// Export entire database
+const backup = await brain.export({
+  format: 'json',
+  includeVectors: true,
+  includeIndexes: false
+})
 ```
 
-### Restore
+### Import Data
 ```typescript
-// Replace the store's entire state from a snapshot (destructive — confirm required)
-await brain.restore('/backups/2026-06-11', { confirm: true })
+// Import from backup
+await brain.import(backup, {
+  mode: 'merge',  // or 'replace'
+  validateSchema: true
+})
 ```
 
-### Move to a new directory
+### Storage Migration
 ```typescript
-// A snapshot directory is a complete store: restore it into a fresh brain
-const brain = new Brainy({ storage: { type: 'filesystem', path: './new' } })
-await brain.init()
-await brain.restore('/backups/2026-06-11', { confirm: true })
+// Migrate between storage types
+const oldBrain = new Brainy({ storage: { type: 'filesystem' } })
+const newBrain = new Brainy({ storage: { type: 's3' } })
+
+await oldBrain.init()
+await newBrain.init()
+
+// Transfer all data
+const data = await oldBrain.export()
+await newBrain.import(data)
 ```
 
 ## Performance Tuning
 
-### FileSystem Optimizations
-- **Directory sharding**: 256 shards spread files across subdirectories
+### Storage-Specific Optimizations
+
+#### FileSystem
+- **Directory sharding**: Split files across subdirectories
 - **Async I/O**: Non-blocking file operations
 - **Buffer pooling**: Reuse buffers for efficiency
 
+#### S3
+- **Multipart uploads**: For large objects
+- **Request batching**: Combine small operations
+- **CDN integration**: Edge caching for reads
+
+#### OPFS
+- **Quota management**: Monitor and request increases
+- **Worker offloading**: Heavy operations in workers
+- **Transaction batching**: Group operations
+
 ### Monitoring
 
 ```typescript
@@ -290,29 +272,22 @@ console.log(stats)
 ## Best Practices
 
 ### Choose the Right Adapter
-1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence
-2. **Single-process production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
-3. **Horizontal scaling**: Brainy runs in one process — there is no built-in cluster. Run independent instances behind a service layer and replicate the on-disk artifact with your operator tooling; or run many reader processes against one shared store with a single writer
+1. **Development**: FileSystem (local persistence)
+2. **Production Server**: FileSystem or S3
+3. **Browser Apps**: OPFS
+4. **Distributed**: S3 with caching
 
 ### Optimize for Your Use Case
-1. **Read-heavy**: Enable caching and let the OS page cache do its job
-2. **Write-heavy**: Batch operations and tune the cache `maxSize`
+1. **Read-heavy**: Enable aggressive caching
+2. **Write-heavy**: Batch operations
 3. **Real-time**: FileSystem with periodic snapshots
-4. **Archival**: Snapshot `path` to cold object storage on a schedule
-5. **Large-scale**: Rely on metadata/vector separation + UUID sharding
+4. **Archival**: S3 with compression
 
 ### Monitor and Maintain
 1. Regular statistics collection
-2. Watch disk usage and shard balance
 3. Index optimization
 4. Cache tuning based on hit rates
-5. Verify backup runs (test restore quarterly)
 
 ## API Reference
 
-See the [Storage API](../api/storage.md) for complete method documentation.
-
----
-
-**Last Updated**: 2026
-**Key Features**: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup
+See the [Storage API](../api/storage.md) for complete method documentation.

docs/architecture/triple-intelligence.md

@@ -1,16 +1,3 @@
----
-title: Triple Intelligence
-slug: concepts/triple-intelligence
-public: true
-category: concepts
-template: concept
-order: 1
-description: Unified vector similarity, graph traversal, and metadata filtering in one query. Auto-optimizes between parallel execution and progressive filtering.
-next:
-  - concepts/noun-types
-  - api/reference
----
-
 # Triple Intelligence System
 
 The Triple Intelligence System is Brainy's revolutionary query engine that unifies vector similarity, graph relationships, and metadata filtering into a single, optimized query interface.
@@ -23,37 +10,29 @@ Traditional databases force you to choose between vector search, graph traversal
 
 ### Unified Query Structure
 
-`find()` accepts a single `FindParams` object (or a natural-language string). One
-object combines all three intelligences:
-
 ```typescript
-interface FindParams {
-  // Vector intelligence — semantic similarity
-  query?: string                 // Natural-language / semantic query (embedded, matched via HNSW + text index)
-  vector?: number[]              // Pre-computed embedding for direct vector search
-
-  // Metadata intelligence — structured field filters
-  type?: NounType | NounType[]   // Filter by entity type
-  subtype?: string | string[]    // Filter by per-product subtype
-  where?: Record<string, any>    // Field predicates with bare operators (gte, lt, in, contains, exists…)
-
-  // Graph intelligence — relationship traversal
+interface TripleQuery {
+  // Vector/Semantic search
+  like?: string | Vector | any
+  similar?: string | Vector | any
+  
+  // Graph/Relationship search  
   connected?: {
-    to?: string                  // Reachable to this entity
-    from?: string                // Reachable from this entity
-    via?: VerbType | VerbType[]  // Relationship type(s) to traverse (alias: type)
-    depth?: number               // Max traversal depth (default: 1)
+    to?: string | string[]
+    from?: string | string[]
+    type?: string | string[]
+    depth?: number
     direction?: 'in' | 'out' | 'both'
   }
-
-  // Proximity — nearest neighbours of a known entity
-  near?: { id: string; threshold?: number }
-
-  // Control
-  limit?: number                 // Max results (default: 10)
-  offset?: number                // Skip N results
-  orderBy?: string               // Field to sort by (e.g. 'createdAt')
-  order?: 'asc' | 'desc'         // Sort direction
+  
+  // Field/Attribute search
+  where?: Record<string, any>
+  
+  // Advanced options
+  limit?: number
+  boost?: 'recent' | 'popular' | 'verified' | string
+  explain?: boolean
+  threshold?: number
 }
 ```
 
@@ -76,16 +55,16 @@ const articles = await brain.find("verified articles by John Smith about machine
 
 #### Simple Vector Search
 ```typescript
-const results = await brain.find("machine learning concepts")
+const results = await brain.search("machine learning concepts")
 ```
 
 #### Combined Intelligence Query
 ```typescript
 const results = await brain.find({
-  query: "neural networks",
+  like: "neural networks",
   where: {
     category: "research",
-    year: { gte: 2023 }
+    year: { $gte: 2023 }
   },
   connected: {
     to: "deep-learning-team",
@@ -117,8 +96,8 @@ All three search types execute simultaneously:
 ```typescript
 // Parallel execution for balanced query
 const results = await brain.find({
-  query: "AI research",          // ~1000 potential matches
-  where: { kind: "paper" },       // ~500 potential matches
+  like: "AI research",           // ~1000 potential matches
+  where: { type: "paper" },       // ~500 potential matches
   connected: { to: "stanford" }   // ~200 potential matches
 })
 // All three execute in parallel, results fused
@@ -134,7 +113,7 @@ Operations chain for maximum efficiency:
 // Progressive execution for selective query
 const results = await brain.find({
   where: { userId: "user123" },   // Very selective (1-10 matches)
-  query: "recent posts",           // Applied to filtered set
+  like: "recent posts",            // Applied to filtered set
   limit: 5
 })
 // Metadata filter first, then vector search on results
@@ -169,15 +148,15 @@ Brainy includes 220+ embedded patterns for natural language understanding:
 
 ```typescript
 // Natural language automatically parsed
-const results = await brain.find(
+const results = await brain.search(
   "show me recent AI papers from Stanford published this year"
 )
 // Automatically converts to:
 // {
-//   query: "AI papers",
-//   where: {
+//   like: "AI papers",
+//   where: { 
 //     institution: "Stanford",
-//     published: { gte: "2024-01-01" }
+//     published: { $gte: "2024-01-01" }
 //   }
 // }
 ```
@@ -196,11 +175,11 @@ The NLP processor identifies query intent:
 
 Successful execution plans are cached:
 ```typescript
-// First call parses the natural-language query and builds an execution plan
-await brain.find("machine learning papers")
+// First query: 50ms (plan generation + execution)
+await brain.search("machine learning papers")
 
-// A structurally similar query reuses that plan, skipping plan generation
-await brain.find("deep learning papers")
+// Subsequent similar queries: 10ms (cached plan)
+await brain.search("deep learning papers")
 ```
 
 ### Self-Optimization
@@ -221,45 +200,50 @@ Triple Intelligence leverages all available indexes:
 
 ### Explain Mode
 
-Diagnose how a query's `where` fields map to the index. Run `brain.explain()`
-first whenever `find()` returns surprising or empty results:
-
-```typescript
-const plan = await brain.explain({
-  query: "quantum computing",
-  where: { category: "research" }
-})
-
-console.log(plan.fieldPlan)
-// [
-//   { field: 'category', path: 'column-store', notes: '...' }
-// ]
-
-console.log(plan.warnings)
-// e.g. ['Field "category" has no index entries. find() will return [] silently...']
-```
-
-### Result Ordering
-
-Sort results by any stored field with `orderBy` / `order`:
+Understand how your query was executed:
 
 ```typescript
 const results = await brain.find({
-  query: "news articles",
-  where: { verified: true },
-  orderBy: 'createdAt',  // Newest first
-  order: 'desc'
+  like: "quantum computing",
+  where: { category: "research" },
+  explain: true
 })
+
+console.log(results[0].explanation)
+// {
+//   plan: "field-first-progressive",
+//   timing: {
+//     fieldFilter: 2,
+//     vectorSearch: 8,
+//     fusion: 1
+//   },
+//   selectivity: {
+//     field: 0.1,
+//     vector: 0.3
+//   }
+// }
 ```
 
-### Similarity Threshold
+### Boosting
 
-Find the nearest neighbours of a known entity and keep only close matches with
-`near`:
+Apply custom ranking boosts:
 
 ```typescript
 const results = await brain.find({
-  near: { id: anchorId, threshold: 0.9 },  // Only results >= 0.9 similarity
+  like: "news articles",
+  boost: 'recent',  // Boost recent items
+  where: { verified: true }
+})
+```
+
+### Threshold Control
+
+Set minimum similarity thresholds:
+
+```typescript
+const results = await brain.find({
+  like: "exact match needed",
+  threshold: 0.9,  // Only very similar results
   limit: 10
 })
 ```
@@ -286,8 +270,8 @@ const results = await brain.find({
 ```typescript
 // Find similar content with constraints
 const results = await brain.find({
-  query: searchText,
-  where: {
+  like: query,
+  where: { 
     status: 'published',
     language: 'en'
   }
@@ -298,10 +282,10 @@ const results = await brain.find({
 ```typescript
 // Find items related to a specific item
 const results = await brain.find({
-  connected: {
+  connected: { 
     to: itemId,
     depth: 2,
-    via: VerbType.RelatedTo
+    type: 'similar'
   },
   limit: 20
 })
@@ -312,11 +296,10 @@ const results = await brain.find({
 // Recent items matching criteria
 const results = await brain.find({
   where: {
-    timestamp: { gte: Date.now() - 86400000 }
+    timestamp: { $gte: Date.now() - 86400000 }
   },
-  query: "trending topics",
-  orderBy: 'timestamp',
-  order: 'desc'
+  like: "trending topics",
+  boost: 'recent'
 })
 ```
 

docs/architecture/zero-config.md

@@ -1,157 +1,769 @@
----
-title: Zero Configuration
-slug: concepts/zero-config
-public: false
-category: concepts
-template: concept
-order: 3
-description: Brainy auto-detects storage, initializes embeddings, and builds indexes — no configuration required. Works in Node.js and Bun (server-only since 8.0).
-next:
-  - getting-started/installation
-  - guides/storage-adapters
----
-
 # Zero Configuration & Auto-Adaptation
 
-> **"Zero config by default, fully tunable when you need it."** Construct a
-> `Brainy()` with no options and it picks sensible, environment-aware defaults.
-> Every default below is overridable through the constructor — see the
-> [API Reference](../api/README.md#configuration).
+> **Current Status**: Basic zero-config is fully functional. Advanced auto-adaptation features are in development.
 
 ## Overview
 
-Brainy 8.0 is server-only (Node.js 22+ / Bun). With no configuration it:
+Brainy is designed with **"Zero Config by Default, Infinite Tunability"** philosophy. It automatically detects your environment, adapts to available resources, learns from usage patterns, and optimizes itself for your specific workload—all without any configuration.
 
-- selects a storage adapter from the runtime,
-- initializes the embedding model (all-MiniLM-L6-v2, 384 dimensions),
-- builds and maintains the metadata, graph, and vector indexes,
-- sizes its caches and write buffers to the detected memory budget,
-- chooses a persistence mode that matches the storage backend, and
-- quiets its own logging when it detects a production environment.
+## Zero Configuration Magic
 
-There is no public config-generation function — adaptation happens inside the
-constructor and `init()`.
-
-## Instant Start
+### Instant Start
 
 ```typescript
-import { Brainy } from '@soulcraft/brainy'
+import { Brainy } from 'brainy'
 
 // That's it. No config needed.
 const brain = new Brainy()
 await brain.init()
 
-await brain.add({ data: 'First entity', type: 'concept' })
-const results = await brain.find('first')
+// Brainy automatically:
+// ✓ Detects environment (Node.js, Browser, Edge, Deno)
+// ✓ Chooses optimal storage (FileSystem, OPFS, Memory)
+// ✓ Downloads required models (if needed)
+// ✓ Configures vector dimensions (384 optimal)
+// ✓ Sets up indexing strategies
+// ✓ Enables appropriate augmentations
+// ✓ Configures caching layers
+// ✓ Optimizes for your hardware
 ```
 
-## What Auto-Adaptation Covers
+### Environment Detection ✅ Available
 
-### 1. Storage auto-detection
-
-With no `storage` option, Brainy uses `type: 'auto'`:
-
-- **Filesystem** when running on a runtime with a writable Node filesystem and a
-  resolvable root directory. This is the default for typical Node/Bun servers and
-  persists across restarts.
-- **In-memory** otherwise (no filesystem access, or an explicit memory request).
-  Fast, zero I/O, discarded on process exit — ideal for tests and ephemeral
-  caches.
-
-8.0 ships exactly two storage adapters — `memory` and `filesystem` — plus the
-`auto` selector that resolves to one of them. See
-[Storage Adapters](../concepts/storage-adapters.md) for the full contract.
+Brainy automatically detects and adapts to your runtime:
 
 ```typescript
-// Explicit override when you want a specific root
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: './brainy-data' }
-})
+// Brainy's environment detection
+const environment = {
+  // Runtime detection
+  isNode: typeof process !== 'undefined',
+  isBrowser: typeof window !== 'undefined',
+  isDeno: typeof Deno !== 'undefined',
+  isEdge: typeof EdgeRuntime !== 'undefined',
+  isWebWorker: typeof WorkerGlobalScope !== 'undefined',
+  
+  // Capability detection
+  hasFileSystem: /* auto-detected */,
+  hasIndexedDB: /* auto-detected */,
+  hasOPFS: /* auto-detected */,
+  hasWebGPU: /* auto-detected */,
+  hasWASM: /* auto-detected */,
+  
+  // Resource detection
+  cpuCores: /* auto-detected */,
+  memory: /* auto-detected */,
+  storage: /* auto-detected */
+}
 ```
 
-### 2. HNSW quality from the `recall` preset
+## Auto-Adaptive Storage ✅ Available
 
-Vector-index quality comes from a single preset rather than hand-tuned graph
-parameters. `config.vector.recall` accepts `'fast'`, `'balanced'`, or
-`'accurate'` and defaults to `'balanced'`. The preset maps internally to the
-HNSW construction and search parameters (`M` / `efConstruction` / `efSearch`),
-so you trade recall against latency with one knob instead of three.
+> **Current**: Brainy automatically selects the best storage adapter for your environment.
+
+### Storage Selection Logic
 
 ```typescript
-const brain = new Brainy({
-  vector: { recall: 'fast' } // favor latency over recall
-})
+// Brainy's intelligent storage selection
+async function autoSelectStorage() {
+  // Server environments
+  if (environment.isNode) {
+    if (await hasWritePermission('./data')) {
+      return 'filesystem'  // Best for servers
+    } else if (process.env.S3_BUCKET) {
+      return 's3'  // Cloud deployment
+    } else {
+      return 'memory'  // Fallback for restricted environments
+    }
+  }
+  
+  // Browser environments
+  if (environment.isBrowser) {
+    if (await navigator.storage.estimate() > 1GB) {
+      return 'opfs'  // Best for modern browsers
+    } else if (indexedDB) {
+      return 'indexeddb'  // Fallback for older browsers
+    } else {
+      return 'memory'  // In-memory for restricted contexts
+    }
+  }
+  
+  // Edge environments
+  if (environment.isEdge) {
+    return 'kv'  // Use edge KV stores (Cloudflare, Vercel)
+  }
+}
 ```
 
-The default JS index is `JsHnswVectorIndex`. An optional native acceleration
-provider (the `@soulcraft/cor` package) can replace it with a
-higher-performing implementation; the public knobs stay the same. Quantization
-and other index-internal acceleration are the native provider's concern, not a
-Brainy configuration option.
+### Storage Migration
 
-### 3. Persistence mode follows the backend
-
-`config.vector.persistMode` accepts `'immediate'` or `'deferred'`. Left unset,
-Brainy chooses for you:
-
-- **Immediate** on filesystem storage, so the index file stays in lock-step with
-  the data and survives a crash.
-- **Deferred** on in-memory storage, where there is nothing durable to sync to,
-  so writes are batched for throughput.
+Brainy seamlessly migrates between storage types:
 
 ```typescript
-const brain = new Brainy({
-  vector: { persistMode: 'deferred' } // batch persistence for write-heavy loads
+// Start with memory storage (development)
+const brain = new Brainy()  // Auto-selects memory
+
+// Later, migrate to production storage
+await brain.migrate({
+  to: 'filesystem',
+  path: './production-data'
 })
+// All data seamlessly transferred
 ```
 
-### 4. Memory-aware cache and buffer sizing
+## Learning & Optimization 🚧 Coming Soon
 
-Brainy reads the container's memory budget — `CLOUD_RUN_MEMORY`, `MEMORY_LIMIT`,
-or the cgroup memory limit when running in a container — and sizes its read
-caches and write buffers to fit. On a small instance it stays conservative; on a
-large one it uses more of the available headroom. Query-result limits are capped
-against the same budget (roughly 25 KB per result) to keep a single oversized
-query from exhausting memory.
+> **Note**: These features are planned for Q2 2025. Currently, Brainy uses static optimizations.
 
-You can pin the cache explicitly:
+### Query Pattern Learning 🚧 Planned
+
+Brainy learns from your query patterns and optimizes accordingly:
 
 ```typescript
-const brain = new Brainy({
-  cache: { maxSize: 10000, ttl: 3_600_000 }
-})
+// Brainy observes query patterns
+class QueryPatternLearner {
+  analyze(queries: Query[]) {
+    return {
+      // Frequency analysis
+      mostCommonFields: this.getTopFields(queries),
+      avgResultSize: this.getAvgSize(queries),
+      temporalPatterns: this.getTimePatterns(queries),
+      
+      // Relationship analysis  
+      commonTraversals: this.getGraphPatterns(queries),
+      typicalDepth: this.getAvgDepth(queries),
+      
+      // Performance analysis
+      slowQueries: this.getSlowQueries(queries),
+      cacheability: this.getCacheability(queries)
+    }
+  }
+}
+
+// Automatic optimizations based on learning:
+// - Creates indexes for frequently queried fields
+// - Pre-computes common graph traversals
+// - Adjusts cache sizes based on working set
+// - Optimizes vector search parameters
 ```
 
-### 5. Logging quiets in production
+### Auto-Indexing 🚧 Planned
 
-Brainy detects production-style environments (for example `NODE_ENV` set to a
-non-development value) and reduces its own log verbosity automatically. This is
-logging-only behavior — it does not change indexing, storage, or query results.
+Brainy automatically creates indexes based on usage:
+
+```typescript
+// No manual index configuration needed
+await brain.find({ where: { category: "tech" } })  // First query
+// Brainy notices 'category' field usage
+
+await brain.find({ where: { category: "science" } })  // Second query
+// Pattern detected - auto-creates category index
+
+await brain.find({ where: { category: "tech" } })  // Third query
+// Now using index - 100x faster!
+```
+
+### Adaptive Caching 🚧 Planned
+
+Cache strategies adapt to your access patterns:
+
+```typescript
+class AdaptiveCache {
+  async adapt(metrics: AccessMetrics) {
+    if (metrics.hitRate < 0.3) {
+      // Low hit rate - switch strategy
+      this.strategy = 'lfu'  // Least Frequently Used
+    } else if (metrics.workingSet > this.size) {
+      // Working set too large - increase size
+      this.size = Math.min(metrics.workingSet * 1.5, maxMemory)
+    } else if (metrics.temporalLocality > 0.8) {
+      // High temporal locality - use time-based eviction
+      this.strategy = 'ttl'
+      this.ttl = metrics.avgAccessInterval * 2
+    }
+  }
+}
+```
+
+## Performance Auto-Scaling 🚧 Coming Soon
+
+### Dynamic Batch Sizing
+
+Brainy adjusts batch sizes based on system load:
+
+```typescript
+class DynamicBatcher {
+  calculateOptimalBatch() {
+    const cpuUsage = process.cpuUsage()
+    const memoryUsage = process.memoryUsage()
+    
+    if (cpuUsage < 30 && memoryUsage < 50) {
+      return 1000  // System idle - large batches
+    } else if (cpuUsage < 60 && memoryUsage < 70) {
+      return 100   // Moderate load - medium batches
+    } else {
+      return 10    // High load - small batches
+    }
+  }
+}
+
+// Automatically applied during bulk operations
+for (const item of millionItems) {
+  await brain.add(item)  // Internally batched optimally
+}
+```
+
+### Memory Management
+
+Automatic memory pressure handling:
+
+```typescript
+class MemoryManager {
+  async handlePressure() {
+    const usage = process.memoryUsage()
+    const available = os.freemem()
+    
+    if (available < 100 * 1024 * 1024) {  // Less than 100MB free
+      // Emergency mode
+      await this.flushCaches()
+      await this.compactIndexes()
+      await this.offloadToDisk()
+    } else if (usage.heapUsed / usage.heapTotal > 0.9) {
+      // Preventive mode
+      await this.reduceCacheSizes()
+      await this.pauseBackgroundTasks()
+    }
+  }
+}
+```
+
+### Connection Pooling
+
+Automatic connection management for storage backends:
+
+```typescript
+class ConnectionPool {
+  async getOptimalPoolSize() {
+    // Adapts based on workload
+    const metrics = await this.getMetrics()
+    
+    if (metrics.waitTime > 100) {
+      // Queries waiting - increase pool
+      this.size = Math.min(this.size * 1.5, this.maxSize)
+    } else if (metrics.idleConnections > this.size * 0.5) {
+      // Too many idle - decrease pool
+      this.size = Math.max(this.size * 0.7, this.minSize)
+    }
+    
+    return this.size
+  }
+}
+```
+
+## Model Auto-Selection
+
+### Embedding Model Selection
+
+Brainy chooses the best embedding model for your use case:
+
+```typescript
+async function autoSelectModel(data: Sample[]) {
+  const analysis = {
+    languages: detectLanguages(data),
+    domainSpecific: detectDomain(data),
+    averageLength: getAvgLength(data),
+    requiresMultilingual: languages.length > 1
+  }
+  
+  if (analysis.requiresMultilingual) {
+    return 'multilingual-e5-base'  // Handles 100+ languages
+  } else if (analysis.domainSpecific === 'code') {
+    return 'codebert-base'  // Optimized for code
+  } else if (analysis.averageLength > 512) {
+    return 'all-mpnet-base-v2'  // Better for long text
+  } else {
+    return 'all-MiniLM-L6-v2'  // Fast and efficient default
+  }
+}
+```
+
+### Model Downloading
+
+Models are automatically downloaded when needed:
+
+```typescript
+// First use - model auto-downloads
+const brain = new Brainy()
+await brain.init()  // Downloads model if not cached
+
+// Intelligent model caching
+const modelCache = {
+  location: process.env.MODEL_CACHE || '~/.brainy/models',
+  maxSize: 5 * 1024 * 1024 * 1024,  // 5GB max
+  strategy: 'lru',  // Least recently used eviction
+  
+  // CDN selection based on location
+  cdn: await selectFastestCDN([
+    'https://cdn.brainy.io',
+    'https://brainy.b-cdn.net',
+    'https://models.huggingface.co'
+  ])
+}
+```
+
+## Workload Detection
+
+### Pattern Recognition
+
+Brainy identifies your workload type and optimizes:
+
+```typescript
+enum WorkloadType {
+  OLTP = 'oltp',  // Many small transactions
+  OLAP = 'olap',  // Analytical queries
+  STREAMING = 'streaming',  // Real-time ingestion
+  BATCH = 'batch',  // Bulk processing
+  HYBRID = 'hybrid'  // Mixed workload
+}
+
+class WorkloadDetector {
+  detect(metrics: OperationMetrics): WorkloadType {
+    if (metrics.writesPerSecond > 1000 && metrics.avgWriteSize < 1024) {
+      return WorkloadType.STREAMING
+    } else if (metrics.avgQueryComplexity > 0.8 && metrics.avgResultSize > 10000) {
+      return WorkloadType.OLAP
+    } else if (metrics.batchOperations > metrics.singleOperations) {
+      return WorkloadType.BATCH
+    } else if (metrics.writeReadRatio > 0.3 && metrics.writeReadRatio < 0.7) {
+      return WorkloadType.HYBRID
+    } else {
+      return WorkloadType.OLTP
+    }
+  }
+}
+```
+
+### Optimization Strategies
+
+Different optimizations for different workloads:
+
+```typescript
+class WorkloadOptimizer {
+  optimize(workload: WorkloadType) {
+    switch (workload) {
+      case WorkloadType.STREAMING:
+        return {
+          entityRegistry: true,  // Deduplication
+          batchSize: 1000,
+          walEnabled: true,
+          cacheSize: 'small',
+          indexStrategy: 'lazy'
+        }
+        
+      case WorkloadType.OLAP:
+        return {
+          entityRegistry: false,
+          batchSize: 10000,
+          walEnabled: false,
+          cacheSize: 'large',
+          indexStrategy: 'eager',
+          parallelQueries: true
+        }
+        
+      case WorkloadType.BATCH:
+        return {
+          entityRegistry: false,
+          batchSize: 50000,
+          walEnabled: false,
+          cacheSize: 'minimal',
+          indexStrategy: 'deferred'
+        }
+        
+      default:
+        return this.defaultConfig
+    }
+  }
+}
+```
+
+## Hardware Adaptation 🚧 Coming Soon
+
+> **Note**: GPU acceleration and hardware optimization planned for Q3 2025.
+
+### CPU Optimization
+
+Adapts to available CPU resources:
+
+```typescript
+class CPUAdapter {
+  async optimize() {
+    const cores = os.cpus().length
+    const type = os.cpus()[0].model
+    
+    // Parallel processing based on cores
+    this.parallelism = Math.max(1, cores - 1)  // Leave one core free
+    
+    // SIMD detection for vector operations
+    if (type.includes('Intel') || type.includes('AMD')) {
+      this.enableSIMD = await checkSIMDSupport()
+    }
+    
+    // Thread pool sizing
+    this.threadPoolSize = cores * 2  // Optimal for I/O bound
+    
+    // Vector search optimization
+    if (cores >= 8) {
+      this.hnswConstruction = 200  // Higher quality index
+      this.hnswSearch = 100  // More accurate search
+    } else {
+      this.hnswConstruction = 100  // Balanced
+      this.hnswSearch = 50  // Faster search
+    }
+  }
+}
+```
+
+### Memory Adaptation
+
+Intelligent memory allocation:
+
+```typescript
+class MemoryAdapter {
+  async configure() {
+    const totalMemory = os.totalmem()
+    const availableMemory = os.freemem()
+    
+    // Allocate based on available memory
+    const allocation = {
+      cache: Math.min(availableMemory * 0.25, 2 * GB),
+      vectors: Math.min(availableMemory * 0.30, 4 * GB),
+      indexes: Math.min(availableMemory * 0.20, 2 * GB),
+      working: Math.min(availableMemory * 0.25, 2 * GB)
+    }
+    
+    // Adjust for low memory systems
+    if (totalMemory < 4 * GB) {
+      allocation.cache *= 0.5
+      allocation.vectors *= 0.7
+      this.enableSwapping = true
+    }
+    
+    return allocation
+  }
+}
+```
+
+### GPU Acceleration
+
+Automatic GPU detection and utilization:
+
+```typescript
+class GPUAdapter {
+  async detect() {
+    // WebGPU in browsers
+    if (navigator?.gpu) {
+      const adapter = await navigator.gpu.requestAdapter()
+      return {
+        available: true,
+        type: 'webgpu',
+        memory: adapter.limits.maxBufferSize,
+        compute: adapter.limits.maxComputeWorkgroupsPerDimension
+      }
+    }
+    
+    // CUDA in Node.js
+    if (process.platform === 'linux' || process.platform === 'win32') {
+      const hasCuda = await checkCudaSupport()
+      if (hasCuda) {
+        return {
+          available: true,
+          type: 'cuda',
+          memory: await getCudaMemory(),
+          compute: await getCudaCores()
+        }
+      }
+    }
+    
+    return { available: false }
+  }
+  
+  async optimize(gpu: GPUInfo) {
+    if (gpu.available) {
+      // Offload vector operations to GPU
+      this.vectorOps = 'gpu'
+      this.embeddingGeneration = 'gpu'
+      this.matrixMultiplication = 'gpu'
+      
+      // Larger batch sizes for GPU
+      this.batchSize = gpu.memory > 8 * GB ? 10000 : 1000
+    }
+  }
+}
+```
+
+## Network Adaptation
+
+### Bandwidth Detection
+
+Optimizes for available network bandwidth:
+
+```typescript
+class NetworkAdapter {
+  async measureBandwidth() {
+    const testSize = 1 * MB
+    const start = Date.now()
+    await this.transfer(testSize)
+    const duration = Date.now() - start
+    
+    const bandwidth = (testSize / duration) * 1000  // bytes/sec
+    
+    if (bandwidth < 1 * MB) {
+      // Low bandwidth - optimize
+      this.compression = 'aggressive'
+      this.batchTransfers = true
+      this.cacheRemote = true
+    } else if (bandwidth > 100 * MB) {
+      // High bandwidth
+      this.compression = 'minimal'
+      this.parallelTransfers = true
+    }
+  }
+}
+```
+
+### Latency Optimization
+
+Adapts to network latency:
+
+```typescript
+class LatencyOptimizer {
+  async optimize() {
+    const latency = await this.measureLatency()
+    
+    if (latency > 100) {  // High latency
+      // Batch operations
+      this.minBatchSize = 100
+      
+      // Aggressive prefetching
+      this.prefetchDepth = 3
+      
+      // Local caching
+      this.cacheStrategy = 'aggressive'
+      
+      // Connection pooling
+      this.connectionPool = Math.min(latency / 10, 50)
+    }
+  }
+}
+```
+
+## Cloud Provider Detection 🚧 Coming Soon
+
+> **Note**: Cloud provider auto-detection planned for Q3 2025.
+
+### Automatic Cloud Optimization
+
+Detects and optimizes for cloud providers:
+
+```typescript
+class CloudDetector {
+  async detect() {
+    // AWS Detection
+    if (process.env.AWS_REGION || await canReachMetadata('169.254.169.254')) {
+      return {
+        provider: 'aws',
+        instance: await getEC2InstanceType(),
+        region: process.env.AWS_REGION,
+        services: {
+          storage: 's3',
+          cache: 'elasticache',
+          compute: 'lambda'
+        }
+      }
+    }
+    
+    // Google Cloud Detection
+    if (process.env.GOOGLE_CLOUD_PROJECT || await canReachMetadata('metadata.google.internal')) {
+      return {
+        provider: 'gcp',
+        instance: await getGCEInstanceType(),
+        region: process.env.GOOGLE_CLOUD_REGION,
+        services: {
+          storage: 'gcs',
+          cache: 'memorystore',
+          compute: 'cloud-run'
+        }
+      }
+    }
+    
+    // Vercel Edge Detection
+    if (process.env.VERCEL) {
+      return {
+        provider: 'vercel',
+        region: process.env.VERCEL_REGION,
+        services: {
+          storage: 'vercel-kv',
+          cache: 'edge-config',
+          compute: 'edge-runtime'
+        }
+      }
+    }
+  }
+}
+```
+
+## Development vs Production
+
+### Automatic Environment Detection
+
+```typescript
+class EnvironmentDetector {
+  detect() {
+    const indicators = {
+      // Development indicators
+      isDevelopment: 
+        process.env.NODE_ENV === 'development' ||
+        process.env.DEBUG ||
+        process.argv.includes('--dev') ||
+        isLocalhost() ||
+        hasDevTools(),
+        
+      // Test indicators
+      isTest:
+        process.env.NODE_ENV === 'test' ||
+        process.env.CI ||
+        isTestRunner(),
+        
+      // Production indicators
+      isProduction:
+        process.env.NODE_ENV === 'production' ||
+        process.env.VERCEL ||
+        process.env.NETLIFY ||
+        !isLocalhost()
+    }
+    
+    return indicators
+  }
+}
+
+// Different defaults for different environments
+const config = environment.isProduction ? {
+  storage: 'filesystem',
+  wal: true,
+  monitoring: true,
+  compression: true,
+  caching: 'aggressive'
+} : {
+  storage: 'memory',
+  wal: false,
+  monitoring: false,
+  compression: false,
+  caching: 'minimal'
+}
+```
+
+## Error Recovery
+
+### Automatic Fallbacks
+
+Brainy automatically recovers from errors:
+
+```typescript
+class AutoRecovery {
+  async handleStorageFailure() {
+    try {
+      await this.primaryStorage.write(data)
+    } catch (error) {
+      console.warn('Primary storage failed, trying fallback')
+      
+      // Try secondary storage
+      if (this.secondaryStorage) {
+        await this.secondaryStorage.write(data)
+      } else {
+        // Fall back to memory
+        await this.memoryStorage.write(data)
+        
+        // Schedule retry
+        this.scheduleRetry(data)
+      }
+    }
+  }
+  
+  async handleModelFailure() {
+    try {
+      return await this.primaryModel.embed(text)
+    } catch (error) {
+      // Fall back to simpler model
+      return await this.fallbackModel.embed(text)
+    }
+  }
+}
+```
 
 ## Configuration Override
 
-Zero-config is the default, not a ceiling. Every adaptive decision above has an
-explicit constructor option:
+While zero-config is default, you can override when needed:
 
 ```typescript
+// Explicit configuration when needed
 const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' },
-  vector: {
-    recall: 'accurate',
-    persistMode: 'immediate'
+  // Override auto-detection
+  storage: {
+    type: 'filesystem',
+    path: '/custom/path'
   },
-  cache: { maxSize: 50000, ttl: 600_000 }
+  
+  // Override auto-optimization
+  optimization: {
+    autoIndex: false,
+    autoCache: false,
+    autoBatch: false
+  },
+  
+  // Override auto-scaling
+  scaling: {
+    maxMemory: 2 * GB,
+    maxConnections: 100,
+    maxBatchSize: 1000
+  }
 })
-
-await brain.init()
 ```
 
-See the [API Reference](../api/README.md#configuration) for the complete option
-list.
+## Monitoring Auto-Adaptation
+
+Brainy provides visibility into its auto-adaptation:
+
+```typescript
+brain.on('adaptation', (event) => {
+  console.log(`Brainy adapted: ${event.type}`)
+  console.log(`Reason: ${event.reason}`)
+  console.log(`Before: ${JSON.stringify(event.before)}`)
+  console.log(`After: ${JSON.stringify(event.after)}`)
+})
+
+// Example events:
+// - Index created for frequently queried field
+// - Cache strategy changed due to low hit rate
+// - Batch size increased due to high throughput
+// - Storage migrated due to space constraints
+// - Model switched due to multilingual content
+```
+
+## Conclusion
+
+Brainy's zero-configuration and auto-adaptation capabilities mean you can focus on your application logic while Brainy handles:
+
+- Environment detection and optimization
+- Storage selection and migration
+- Performance tuning and scaling
+- Resource management
+- Error recovery
+- Workload optimization
+
+Just create a Brainy instance and start using it. Brainy will learn, adapt, and optimize itself for your specific use case—no configuration required.
 
 ## See Also
 
 - [Architecture Overview](./overview.md)
-- [Storage Adapters](../concepts/storage-adapters.md)
-- [Scaling Guide](../SCALING.md)
-- [API Reference](../api/README.md)
+- [Storage Architecture](./storage.md)
+- [Performance Guide](../guides/performance.md)
+- [Augmentations System](./augmentations.md)

docs/augmentations/COMPLETE-REFERENCE.md

@@ -0,0 +1,451 @@
+# 🔌 Brainy v4.0.0 Augmentations Complete Reference
+
+> **All augmentations that power Brainy's extensibility - with locations, usage, and examples**
+>
+> **⚠️ v4.0.0 Update**: Updated for metadata structure changes and billion-scale optimizations
+
+## Quick Start
+
+```typescript
+import { Brainy } from '@soulcraft/brainy'
+
+const brain = new Brainy({
+  // Augmentations auto-configure based on environment
+  storage: 'auto',     // Storage augmentation
+  cache: true,         // Cache augmentation
+  index: true          // Index augmentation
+})
+
+await brain.init()  // Augmentations initialize automatically
+```
+
+## v4.0.0 Augmentation Architecture
+
+### Key Improvements for Billion-Scale Performance
+
+1. **Metadata/Vector Separation**: Augmentations now work with separated metadata and vectors
+   - Metadata stored separately from vector data
+   - 99.2% memory reduction for type tracking
+   - Two-file storage pattern for optimal I/O
+
+2. **Type System Enforcement**: All metadata requires type fields
+   - `NounMetadata` requires `noun: NounType`
+   - `VerbMetadata` requires `verb: VerbType`
+   - Type inference system available as public API
+
+3. **Storage Adapter Pattern**: Internal vs public method distinction
+   - `_methods`: Return pure structures (HNSWNoun, HNSWVerb)
+   - Public methods: Return WithMetadata types
+   - MetadataEnforcer Proxy ensures proper access
+
+### What This Means for Augmentation Users
+
+**✅ If you use built-in augmentations**: No changes needed! They're all updated for v4.0.0.
+
+**⚠️ If you created custom storage augmentations**: Update your storage adapter to:
+- Wrap metadata with required `noun`/`verb` fields
+- Follow the internal/public method pattern
+- Use two-file storage approach
+
+**⚠️ If you access relationship data**: Change `verb.type` to `verb.verb`
+
+## Core Concepts
+
+### What are Augmentations?
+Augmentations are modular extensions that add functionality to Brainy without cluttering the core API. They follow a unified interface and can be:
+- **Auto-enabled**: Based on configuration (cache, index, storage)
+- **Manually registered**: For custom functionality
+- **Chained**: Multiple augmentations work together seamlessly
+- **Billion-scale ready**: Optimized for datasets with billions of nouns and verbs
+
+### Augmentation Lifecycle
+1. **Registration**: Augmentations register before init()
+2. **Initialization**: Two-phase init (storage first, then others)
+3. **Execution**: Hook into operations (before/after/both)
+4. **Shutdown**: Clean teardown on brain.shutdown()
+
+---
+
+## Storage Augmentations (8 total)
+
+### MemoryStorageAugmentation
+**Location**: `src/augmentations/storageAugmentations.ts`  
+**Auto-enabled**: When `storage: 'memory'` or in test environments  
+**Purpose**: In-memory storage for testing and temporary data
+```typescript
+const brain = new Brainy({ storage: 'memory' })
+```
+
+### FileSystemStorageAugmentation  
+**Location**: `src/augmentations/storageAugmentations.ts`  
+**Auto-enabled**: When `storage: 'filesystem'` or Node.js detected  
+**Purpose**: Persistent file-based storage for Node.js applications
+```typescript
+const brain = new Brainy({ 
+  storage: { type: 'filesystem', path: './data' }
+})
+```
+
+### OPFSStorageAugmentation
+**Location**: `src/augmentations/storageAugmentations.ts`  
+**Auto-enabled**: When `storage: 'opfs'` or browser with OPFS support  
+**Purpose**: Browser-based persistent storage using Origin Private File System
+```typescript
+const brain = new Brainy({ storage: 'opfs' })
+```
+
+### S3StorageAugmentation
+**Location**: `src/augmentations/storageAugmentations.ts`  
+**Manual**: Requires AWS credentials  
+**Purpose**: AWS S3-compatible cloud storage
+```typescript
+const brain = new Brainy({ 
+  storage: {
+    type: 's3',
+    bucket: 'my-bucket',
+    region: 'us-east-1',
+    credentials: { accessKeyId, secretAccessKey }
+  }
+})
+```
+
+### R2StorageAugmentation
+**Location**: `src/augmentations/storageAugmentations.ts`  
+**Manual**: Requires Cloudflare credentials  
+**Purpose**: Cloudflare R2 storage (S3-compatible)
+```typescript
+const brain = new Brainy({ 
+  storage: {
+    type: 'r2',
+    accountId: 'xxx',
+    bucket: 'my-bucket',
+    credentials: { accessKeyId, secretAccessKey }
+  }
+})
+```
+
+### GCSStorageAugmentation
+**Location**: `src/augmentations/storageAugmentations.ts`  
+**Manual**: Requires Google Cloud credentials  
+**Purpose**: Google Cloud Storage
+```typescript
+const brain = new Brainy({ 
+  storage: {
+    type: 'gcs',
+    bucket: 'my-bucket',
+    projectId: 'my-project'
+  }
+})
+```
+
+### StorageAugmentation (base)
+**Location**: `src/augmentations/storageAugmentation.ts`  
+**Purpose**: Base class for custom storage implementations
+
+### DynamicStorageAugmentation
+**Location**: `src/augmentations/storageAugmentation.ts`  
+**Purpose**: Runtime storage adapter switching
+
+---
+
+## Performance Augmentations (7 total)
+
+### CacheAugmentation
+**Location**: `src/augmentations/cacheAugmentation.ts`  
+**Auto-enabled**: When `cache: true` (default)  
+**Purpose**: LRU cache for search results and frequent queries
+```typescript
+brain.clearCache()           // Exposed via API
+brain.getCacheStats()        // Cache hit/miss statistics
+```
+
+### IndexAugmentation
+**Location**: `src/augmentations/indexAugmentation.ts`  
+**Auto-enabled**: When `index: true` (default)  
+**Purpose**: Metadata indexing for O(1) field lookups
+```typescript
+brain.rebuildMetadataIndex() // Exposed via API
+// Enables fast where queries:
+brain.find({ where: { category: 'tech' } })
+```
+
+### MetricsAugmentation
+**Location**: `src/augmentations/metricsAugmentation.ts`  
+**Auto-enabled**: Always active  
+**Purpose**: Performance metrics and statistics collection
+```typescript
+brain.getStats()        // Comprehensive metrics
+```
+
+### MonitoringAugmentation
+**Location**: `src/augmentations/monitoringAugmentation.ts`  
+**Manual**: Register for detailed monitoring  
+**Purpose**: Real-time performance monitoring and alerts
+
+### BatchProcessingAugmentation
+**Location**: `src/augmentations/batchProcessingAugmentation.ts`  
+**Auto-enabled**: For batch operations  
+**Purpose**: Optimizes bulk add/update/delete operations
+```typescript
+brain.addNouns([...])        // Automatically batched
+```
+
+### RequestDeduplicatorAugmentation
+**Location**: `src/augmentations/requestDeduplicatorAugmentation.ts`  
+**Auto-enabled**: Always active  
+**Purpose**: Prevents duplicate concurrent operations
+
+### ConnectionPoolAugmentation
+**Location**: `src/augmentations/connectionPoolAugmentation.ts`  
+**Auto-enabled**: For network storage  
+**Purpose**: Connection pooling for cloud storage adapters
+
+---
+
+## Data Integrity Augmentations (3 total)
+
+**Auto-enabled**: When `wal: true`  
+**Purpose**: Write-ahead logging for crash recovery
+```typescript
+const brain = new Brainy({ wal: true })
+// Automatic recovery on restart after crash
+```
+
+### EntityRegistryAugmentation
+**Location**: `src/augmentations/entityRegistryAugmentation.ts`  
+**Auto-enabled**: For streaming operations  
+**Purpose**: High-speed deduplication for real-time data
+```typescript
+// Prevents duplicate entities in streaming scenarios
+brain.add(data) // Automatically deduplicated
+```
+
+### AutoRegisterEntitiesAugmentation
+**Location**: `src/augmentations/entityRegistryAugmentation.ts`  
+**Manual**: For automatic entity discovery  
+**Purpose**: Auto-discovers and registers entities from data
+
+---
+
+## Intelligence Augmentations (2 total)
+
+### NeuralImportAugmentation
+**Location**: `src/augmentations/neuralImport.ts`  
+**Manual**: Via `brain.neuralImport()`  
+**Purpose**: AI-powered smart data import
+```typescript
+const result = await brain.neuralImport(data, {
+  confidenceThreshold: 0.7,
+  autoApply: true
+})
+// Automatically detects entities and relationships
+```
+
+### IntelligentVerbScoringAugmentation
+**Location**: `src/augmentations/intelligentVerbScoringAugmentation.ts`  
+**Auto-enabled**: When verbs are used  
+**Purpose**: ML-based relationship strength scoring
+```typescript
+brain.verbScoring.train(feedback)
+brain.verbScoring.getScore(verbId)
+```
+
+---
+
+## Communication Augmentations (4 total)
+
+### APIServerAugmentation
+**Location**: `src/augmentations/apiServerAugmentation.ts`  
+**Manual**: For server deployments  
+**Purpose**: REST/WebSocket/MCP API server
+```typescript
+const augmentation = new APIServerAugmentation()
+await brain.registerAugmentation(augmentation)
+// Exposes full Brainy API over network
+```
+
+### WebSocketConduitAugmentation
+**Location**: `src/augmentations/conduitAugmentations.ts`  
+**Manual**: For Brainy-to-Brainy sync  
+**Purpose**: Real-time sync between Brainy instances
+```typescript
+const conduit = new WebSocketConduitAugmentation()
+await conduit.establishConnection('ws://other-brain')
+```
+
+### ServerSearchConduitAugmentation
+**Location**: `src/augmentations/serverSearchAugmentations.ts`  
+**Manual**: For client-server search  
+**Purpose**: Search remote Brainy instance, cache locally
+
+### ServerSearchActivationAugmentation
+**Location**: `src/augmentations/serverSearchAugmentations.ts`  
+**Manual**: Works with ServerSearchConduit  
+**Purpose**: Triggers and manages server search operations
+
+---
+
+## External Integration (2 total)
+
+### SynapseAugmentation (base)
+**Location**: `src/augmentations/synapseAugmentation.ts`  
+**Purpose**: Base class for external platform integrations
+```typescript
+// Example: NotionSynapse, SlackSynapse, etc.
+class NotionSynapse extends SynapseAugmentation {
+  async fetchData() { /* Notion API calls */ }
+  async pushData() { /* Sync to Notion */ }
+}
+```
+
+### ExampleFileSystemSynapse
+**Location**: `src/augmentations/synapseAugmentation.ts`  
+**Purpose**: Example implementation for file system sync
+
+---
+
+## Augmentation Configuration
+
+### Auto-Configuration
+```typescript
+const brain = new Brainy({
+  // These auto-register augmentations:
+  storage: 'auto',        // Storage augmentation
+  cache: true,           // Cache augmentation  
+  index: true,           // Index augmentation
+  metrics: true         // Metrics augmentation
+})
+```
+
+### Manual Registration
+```typescript
+const brain = new Brainy()
+
+// Register before init()
+const customAug = new MyCustomAugmentation()
+await brain.registerAugmentation(customAug)
+
+await brain.init()
+```
+
+### Creating Custom Augmentations
+```typescript
+import { BaseAugmentation } from '@soulcraft/brainy'
+
+class MyAugmentation extends BaseAugmentation {
+  readonly name = 'my-augmentation'
+  readonly timing = 'after'  // before | after | both
+  readonly operations = ['addNoun', 'search']  // Which ops to hook
+  readonly priority = 10      // Execution order (lower = earlier)
+  
+  protected async onInit(): Promise<void> {
+    // Initialize your augmentation
+  }
+  
+  async execute<T>(
+    operation: string,
+    params: any,
+    context?: AugmentationContext
+  ): Promise<T | void> {
+    // Your augmentation logic
+    if (operation === 'addNoun') {
+      console.log('Noun added:', params)
+    }
+  }
+  
+  protected async onShutdown(): Promise<void> {
+    // Cleanup
+  }
+}
+```
+
+---
+
+## Augmentation Timing & Priority
+
+### Timing Options
+- **`before`**: Runs before the operation (can modify params)
+- **`after`**: Runs after the operation (can see results)
+- **`both`**: Runs before AND after
+
+### Priority (lower = earlier)
+1. Storage augmentations (priority: 0)
+2. Cache/Index augmentations (priority: 5-10)
+3. Monitoring/Metrics (priority: 15-20)
+4. Conduits/Synapses (priority: 20-30)
+
+---
+
+## Key Integration Points
+
+### Where Augmentations Hook In
+
+**Brainy Constructor**:
+- Storage augmentations register based on config
+- Cache/Index augmentations auto-register if enabled
+
+**brain.init()**:
+- Two-phase initialization (storage first, then others)
+- Augmentations can access brain instance via context
+
+**Operations** (addNoun, search, etc.):
+- Augmentations execute based on timing and operations filter
+- Can modify params (before) or see results (after)
+
+**brain.shutdown()**:
+- All augmentations cleaned up in reverse order
+
+---
+
+## Performance Impact
+
+Most augmentations have minimal overhead:
+- **Cache**: ~1ms per search (saves 10-100ms on hits)
+- **Index**: ~1ms per operation (saves 100ms+ on queries)
+- **Metrics**: <1ms per operation
+- **Storage**: Varies by adapter (memory: 0ms, S3: 50-200ms)
+
+---
+
+## Best Practices
+
+1. **Let auto-configuration work**: Most apps need zero manual config
+2. **Storage first**: Always configure storage before other augmentations
+3. **Use built-in augmentations**: They're optimized and battle-tested
+4. **Custom augmentations**: Extend BaseAugmentation for consistency
+5. **Respect timing**: Use 'before' to modify, 'after' to observe
+6. **Mind priority**: Lower numbers execute first
+
+---
+
+## Troubleshooting
+
+### Augmentation not working?
+```typescript
+// Check if registered
+brain.listAugmentations()
+
+// Check if enabled
+brain.isAugmentationEnabled('cache')
+
+// Enable/disable at runtime
+brain.enableAugmentation('cache')
+brain.disableAugmentation('cache')
+```
+
+### Performance issues?
+```typescript
+// Check augmentation overhead
+const stats = brain.getStats()
+console.log(stats.augmentations)
+
+// Disable non-critical augmentations
+brain.disableAugmentation('monitoring')
+```
+
+---
+
+
+---
+
+*Augmentations make Brainy infinitely extensible while keeping the core API clean and simple!*

docs/augmentations/CONFIGURATION.md

@@ -0,0 +1,620 @@
+# Augmentation Configuration System
+
+**Version**: 2.0.0  
+**Status**: Production Ready
+
+## Overview
+
+The Brainy Augmentation Configuration System provides a VSCode-style extension architecture with multiple configuration sources, schema validation, and tool discovery. This system maintains Brainy's zero-config philosophy while enabling sophisticated enterprise configuration management.
+
+## Table of Contents
+- [Quick Start](#quick-start)
+- [Configuration Sources](#configuration-sources)
+- [Creating Configurable Augmentations](#creating-configurable-augmentations)
+- [Configuration Discovery](#configuration-discovery)
+- [Runtime Configuration](#runtime-configuration)
+- [Environment Variables](#environment-variables)
+- [Configuration Files](#configuration-files)
+- [CLI Commands](#cli-commands)
+- [Tool Integration](#tool-integration)
+- [Migration Guide](#migration-guide)
+
+## Quick Start
+
+### Using an Augmentation with Configuration
+
+```typescript
+import { Brainy } from '@soulcraft/brainy'
+
+// Zero-config (uses defaults)
+const brain = new Brainy()
+
+// With custom configuration
+  immediateWrites: true,
+  checkpointInterval: 300000 // 5 minutes
+}))
+```
+
+### Configuring via Environment Variables
+
+```bash
+export BRAINY_AUG_CACHE_TTL=600000
+```
+
+### Configuring via Files
+
+Create a `.brainyrc` file in your project root:
+
+```json
+{
+  "augmentations": {
+    "wal": {
+      "enabled": true,
+      "immediateWrites": true,
+      "maxSize": 20971520
+    },
+    "cache": {
+      "ttl": 600000,
+      "maxSize": 2000
+    }
+  }
+}
+```
+
+## Configuration Sources
+
+Configuration is resolved in the following priority order (highest to lowest):
+
+1. **Runtime Updates** - Dynamic configuration changes via API
+2. **Constructor Parameters** - Code-time configuration
+3. **Environment Variables** - `BRAINY_AUG_<NAME>_<KEY>`
+4. **Configuration Files** - `.brainyrc`, `brainy.config.json`
+5. **Schema Defaults** - Default values from manifest
+
+### Resolution Example
+
+```typescript
+// Schema default
+{ maxSize: 10485760 }
+
+// File configuration (.brainyrc)
+{ maxSize: 20971520 }
+
+// Environment variable
+
+// Constructor parameter
+
+// Final resolved value: 41943040 (constructor wins)
+```
+
+## Creating Configurable Augmentations
+
+### Step 1: Extend ConfigurableAugmentation
+
+```typescript
+import { ConfigurableAugmentation, AugmentationManifest } from '@soulcraft/brainy'
+
+export class MyAugmentation extends ConfigurableAugmentation {
+  name = 'my-augmentation'
+  timing = 'around' as const
+  metadata = 'none' as const
+  operations = ['search', 'add']
+  priority = 50
+  
+  constructor(config?: MyConfig) {
+    super(config) // Handles configuration resolution
+  }
+  
+  // Required: Provide manifest for discovery
+  getManifest(): AugmentationManifest {
+    return {
+      id: 'my-augmentation',
+      name: 'My Augmentation',
+      version: '1.0.0',
+      description: 'Does something amazing',
+      category: 'performance',
+      configSchema: {
+        type: 'object',
+        properties: {
+          enabled: {
+            type: 'boolean',
+            default: true,
+            description: 'Enable this augmentation'
+          },
+          threshold: {
+            type: 'number',
+            default: 100,
+            minimum: 1,
+            maximum: 1000,
+            description: 'Processing threshold'
+          }
+        }
+      }
+    }
+  }
+  
+  // Optional: Handle runtime configuration changes
+  protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise<void> {
+    if (newConfig.threshold !== oldConfig.threshold) {
+      // React to threshold change
+      this.updateThreshold(newConfig.threshold)
+    }
+  }
+  
+  async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
+    if (!this.config.enabled) {
+      return next()
+    }
+    
+    // Your augmentation logic here
+    return next()
+  }
+}
+```
+
+### Step 2: Define Configuration Interface
+
+```typescript
+interface MyConfig {
+  enabled?: boolean
+  threshold?: number
+  mode?: 'fast' | 'balanced' | 'thorough'
+}
+```
+
+### Step 3: Add JSON Schema in Manifest
+
+```typescript
+configSchema: {
+  type: 'object',
+  properties: {
+    enabled: {
+      type: 'boolean',
+      default: true,
+      description: 'Enable augmentation'
+    },
+    threshold: {
+      type: 'number',
+      default: 100,
+      minimum: 1,
+      maximum: 1000,
+      description: 'Processing threshold'
+    },
+    mode: {
+      type: 'string',
+      default: 'balanced',
+      enum: ['fast', 'balanced', 'thorough'],
+      description: 'Processing mode'
+    }
+  },
+  required: [],
+  additionalProperties: false
+}
+```
+
+## Configuration Discovery
+
+The Discovery API allows tools to discover and configure augmentations dynamically:
+
+```typescript
+import { AugmentationDiscovery } from '@soulcraft/brainy'
+
+const discovery = new AugmentationDiscovery(brain.augmentations)
+
+// Discover all augmentations with manifests
+const listings = await discovery.discover({
+  includeConfig: true,
+  includeSchema: true
+})
+
+// Get configuration schema
+const schema = await discovery.getConfigSchema('wal')
+
+// Validate configuration
+const validation = await discovery.validateConfig('wal', {
+  enabled: true,
+  maxSize: 'invalid' // Will fail validation
+})
+
+// Update configuration at runtime
+await discovery.updateConfig('wal', {
+  checkpointInterval: 120000
+})
+```
+
+## Runtime Configuration
+
+### Update Configuration Dynamically
+
+```typescript
+// Get augmentation
+const wal = brain.augmentations.get('wal')
+
+// Update configuration
+await wal.updateConfig({
+  checkpointInterval: 300000
+})
+
+// Get current configuration
+const config = wal.getConfig()
+```
+
+### React to Configuration Changes
+
+```typescript
+class MyAugmentation extends ConfigurableAugmentation {
+  protected async onConfigChange(newConfig: any, oldConfig: any): Promise<void> {
+    // Stop old processes
+    if (oldConfig.enabled && !newConfig.enabled) {
+      await this.stop()
+    }
+    
+    // Start new processes
+    if (!oldConfig.enabled && newConfig.enabled) {
+      await this.start()
+    }
+    
+    // Update settings
+    if (newConfig.interval !== oldConfig.interval) {
+      this.rescheduleTimer(newConfig.interval)
+    }
+  }
+}
+```
+
+## Environment Variables
+
+### Naming Convention
+
+```bash
+BRAINY_AUG_<AUGMENTATION_ID>_<CONFIG_KEY>=value
+```
+
+### Examples
+
+```bash
+
+# Cache augmentation
+BRAINY_AUG_CACHE_ENABLED=true
+BRAINY_AUG_CACHE_MAX_SIZE=2000
+BRAINY_AUG_CACHE_TTL=600000
+
+# Complex values (JSON)
+BRAINY_AUG_MYAUG_FILTERS='["*.js","*.ts"]'
+BRAINY_AUG_MYAUG_OPTIONS='{"deep":true,"follow":false}'
+```
+
+### Docker Example
+
+```dockerfile
+ENV BRAINY_AUG_CACHE_TTL=600000
+```
+
+## Configuration Files
+
+### File Locations (Priority Order)
+
+1. `.brainyrc` (current directory)
+2. `.brainyrc.json` (current directory)
+3. `brainy.config.json` (current directory)
+4. `~/.brainy/config.json` (user home)
+5. `~/.brainyrc` (user home)
+
+### File Format
+
+```json
+{
+  "augmentations": {
+    "wal": {
+      "enabled": true,
+      "immediateWrites": true,
+      "maxSize": 20971520,
+      "checkpointInterval": 300000
+    },
+    "cache": {
+      "enabled": true,
+      "maxSize": 2000,
+      "ttl": 600000
+    },
+    "metrics": {
+      "enabled": false
+    }
+  }
+}
+```
+
+### Per-Environment Configuration
+
+```json
+{
+  "augmentations": {
+    "wal": {
+      "development": {
+        "enabled": true,
+        "immediateWrites": true,
+        "maxSize": 5242880
+      },
+      "production": {
+        "enabled": true,
+        "immediateWrites": false,
+        "maxSize": 104857600,
+        "checkpointInterval": 60000
+      }
+    }
+  }
+}
+```
+
+## CLI Commands
+
+### List Augmentations with Configuration
+
+```bash
+# Show all augmentations with config status
+brainy augment list --detailed
+
+# Show configuration for specific augmentation
+brainy augment config wal
+
+# Set configuration value
+brainy augment config wal --set immediateWrites=true
+
+# Show environment variable names
+brainy augment config wal --env
+
+# Export configuration schema
+brainy augment schema wal > wal-schema.json
+
+# Validate configuration file
+brainy augment validate --file config.json
+```
+
+### Interactive Configuration
+
+```bash
+# Interactive configuration wizard
+brainy augment configure wal
+
+? Operation mode? 
+  ❯ Performance (immediate writes)
+    Durability (synchronous writes)
+    Custom
+? Maximum log size? (10MB) 20MB
+? Checkpoint interval? (1 minute) 5 minutes
+
+Configuration saved to .brainyrc
+```
+
+## Tool Integration
+
+### Brain-Cloud Explorer UI
+
+```typescript
+// Auto-generate configuration form from schema
+const ConfigurationUI = ({ augmentationId }) => {
+  const [manifest, setManifest] = useState(null)
+  const [config, setConfig] = useState({})
+  
+  useEffect(() => {
+    // Fetch manifest with schema
+    fetch(`/api/augmentations/${augmentationId}/manifest`)
+      .then(res => res.json())
+      .then(setManifest)
+    
+    // Get current configuration
+    discovery.getConfig(augmentationId)
+      .then(setConfig)
+  }, [augmentationId])
+  
+  const handleSave = async (newConfig) => {
+    // Validate configuration
+    const validation = await fetch(`/api/augmentations/${augmentationId}/validate`, {
+      method: 'POST',
+      body: JSON.stringify(newConfig)
+    }).then(res => res.json())
+    
+    if (validation.valid) {
+      // Apply configuration
+      await discovery.updateConfig(augmentationId, newConfig)
+    }
+  }
+  
+  // Render form based on schema
+  return <SchemaForm 
+    schema={manifest?.configSchema}
+    values={config}
+    onSubmit={handleSave}
+  />
+}
+```
+
+### VS Code Extension
+
+```json
+// package.json contribution points
+{
+  "contributes": {
+    "configuration": {
+      "title": "Brainy Augmentations",
+      "properties": {
+        "brainy.augmentations.wal.enabled": {
+          "type": "boolean",
+          "default": true,
+        },
+        "brainy.augmentations.wal.maxSize": {
+          "type": "number",
+          "default": 10485760,
+        }
+      }
+    }
+  }
+}
+```
+
+## Migration Guide
+
+### Migrating from BaseAugmentation
+
+**Before:**
+```typescript
+export class MyAugmentation extends BaseAugmentation {
+  constructor(config: MyConfig = {}) {
+    super()
+    this.config = {
+      enabled: config.enabled ?? true,
+      threshold: config.threshold ?? 100
+    }
+  }
+  
+  // No manifest
+  // No config discovery
+  // No runtime updates
+}
+```
+
+**After:**
+```typescript
+export class MyAugmentation extends ConfigurableAugmentation {
+  constructor(config?: MyConfig) {
+    super(config) // Config resolution handled automatically
+  }
+  
+  getManifest(): AugmentationManifest {
+    return {
+      id: 'my-augmentation',
+      name: 'My Augmentation',
+      version: '1.0.0',
+      description: 'Does something amazing',
+      category: 'performance',
+      configSchema: {
+        type: 'object',
+        properties: {
+          enabled: { type: 'boolean', default: true },
+          threshold: { type: 'number', default: 100 }
+        }
+      }
+    }
+  }
+  
+  // Optional: Handle config changes
+  protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise<void> {
+    // React to changes
+  }
+}
+```
+
+### Backwards Compatibility
+
+The system maintains full backwards compatibility:
+
+1. **BaseAugmentation still works** - Existing augmentations continue to function
+2. **Constructor config still works** - Existing configuration patterns preserved
+3. **Zero-config still works** - Defaults are applied automatically
+4. **Progressive enhancement** - Add features as needed
+
+## Best Practices
+
+### 1. Always Provide Defaults
+
+```typescript
+configSchema: {
+  properties: {
+    enabled: {
+      type: 'boolean',
+      default: true, // Always provide defaults
+      description: 'Enable this feature'
+    }
+  }
+}
+```
+
+### 2. Use Descriptive Configuration Keys
+
+```typescript
+// Good
+checkpointInterval: 60000
+
+// Bad
+ci: 60000
+```
+
+### 3. Validate Configuration
+
+```typescript
+protected async onConfigChange(newConfig: any, oldConfig: any): Promise<void> {
+  // Validate before applying
+  if (newConfig.maxSize < 1048576) {
+    throw new Error('maxSize must be at least 1MB')
+  }
+  
+  // Apply changes
+  this.maxSize = newConfig.maxSize
+}
+```
+
+### 4. Document Environment Variables
+
+```typescript
+/**
+ * Environment Variables:
+ * - BRAINY_AUG_MYAUG_ENABLED: Enable augmentation (boolean)
+ * - BRAINY_AUG_MYAUG_THRESHOLD: Processing threshold (number)
+ * - BRAINY_AUG_MYAUG_MODE: Processing mode (fast|balanced|thorough)
+ */
+```
+
+### 5. Provide Configuration Examples
+
+```typescript
+configExamples: [
+  {
+    name: 'Production',
+    description: 'Optimized for production use',
+    config: {
+      enabled: true,
+      mode: 'thorough',
+      threshold: 500
+    }
+  },
+  {
+    name: 'Development',
+    description: 'Lightweight for development',
+    config: {
+      enabled: true,
+      mode: 'fast',
+      threshold: 10
+    }
+  }
+]
+```
+
+## Troubleshooting
+
+### Configuration Not Loading
+
+1. Check file locations and names
+2. Verify JSON syntax in config files
+3. Check environment variable names (case-sensitive)
+4. Use `brainy augment config <name> --debug` to see resolution
+
+### Validation Errors
+
+1. Check schema requirements
+2. Verify data types match schema
+3. Check minimum/maximum constraints
+4. Use discovery API to validate before applying
+
+### Runtime Updates Not Working
+
+1. Ensure augmentation extends ConfigurableAugmentation
+2. Implement onConfigChange if needed
+3. Check for validation errors
+4. Verify augmentation is initialized
+
+## API Reference
+
+See the [Discovery API Documentation](./discovery-api.md) for complete API details.
+
+## Examples
+
+See the [examples directory](../../examples/augmentation-config/) for complete working examples.

docs/augmentations/DEVELOPER-GUIDE.md

@@ -0,0 +1,526 @@
+# 🛠️ Brainy Augmentation Developer Guide
+
+> **How to create, test, and use augmentations in Brainy v4.0.0**
+>
+> **⚠️ v4.0.0 Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
+
+## v4.0.0 Migration Guide
+
+### What Changed?
+
+1. **Metadata Structure**: All metadata now requires type fields (`noun` or `verb`)
+2. **Property Rename**: `verb.type` → `verb.verb` for relationships
+3. **Two-File Storage**: Vectors and metadata stored separately for performance
+4. **Return Types**: Storage methods distinguish between internal (pure) and public (WithMetadata) returns
+
+### Migration Checklist
+
+- [ ] Update metadata creation to include required `noun` field
+- [ ] Change `verb.type` to `verb.verb` in all relationship code
+- [ ] Update storage adapter methods to follow internal/public pattern
+- [ ] Ensure metadata access uses correct structure
+
+### Quick Migration Example
+
+```typescript
+// ❌ v3.x
+const verb = {
+  type: 'relatedTo',
+  sourceId: 'a',
+  targetId: 'b'
+}
+if (verb.type === 'relatedTo') { ... }
+
+// ✅ v4.0.0
+const verb = {
+  verb: 'relatedTo',
+  sourceId: 'a',
+  targetId: 'b'
+}
+const metadata: VerbMetadata = {
+  verb: 'relatedTo',
+  sourceId: 'a',
+  targetId: 'b'
+}
+if (verb.verb === 'relatedTo') { ... }
+```
+
+## Quick Start: Your First Augmentation
+
+```typescript
+import { BaseAugmentation, BrainyAugmentation, AugmentationContext } from '@soulcraft/brainy'
+
+export class MyFirstAugmentation extends BaseAugmentation {
+  readonly name = 'my-first-augmentation'
+  readonly timing = 'after' as const       // When to run: before | after | both
+  readonly operations = ['add'] as const  // Which operations to hook
+  readonly priority = 10                    // Execution order (lower = first)
+
+  protected async onInit(): Promise<void> {
+    // Initialize your augmentation
+    console.log('MyFirstAugmentation initialized!')
+  }
+
+  async execute<T = any>(
+    operation: string,
+    params: any,
+    context?: AugmentationContext
+  ): Promise<T | void> {
+    // Your augmentation logic
+    if (operation === 'add') {
+      console.log('Noun added:', params.noun)
+
+      // v4.0.0: Access metadata correctly
+      if (params.noun?.metadata) {
+        console.log('Noun type:', params.noun.metadata.noun)  // Required field
+      }
+
+      // You can access the brain instance
+      const stats = await context?.brain.getStats()
+      console.log('Total nouns:', stats.totalNouns)
+    }
+  }
+
+  protected async onShutdown(): Promise<void> {
+    // Cleanup
+    console.log('MyFirstAugmentation shutting down')
+  }
+}
+```
+
+## Using Your Augmentation
+
+```typescript
+import { Brainy } from '@soulcraft/brainy'
+import { MyFirstAugmentation } from './my-first-augmentation'
+
+const brain = new Brainy()
+
+// Register before init()
+brain.augmentations.register(new MyFirstAugmentation())
+
+await brain.init()
+
+// Now your augmentation runs automatically!
+await brain.add('Hello World')
+// Console: "Noun added: { id: '...', vector: [...], metadata: {} }"
+```
+
+---
+
+## Augmentation Lifecycle
+
+### 1. Registration Phase
+```typescript
+const aug = new MyAugmentation()
+brain.augmentations.register(aug)  // Before brain.init()!
+```
+
+### 2. Initialization Phase
+```typescript
+await brain.init()  // Calls aug.initialize() internally
+// Your onInit() method runs here
+```
+
+### 3. Execution Phase
+```typescript
+await brain.add('data')  // Your execute() method runs
+```
+
+### 4. Shutdown Phase
+```typescript
+await brain.shutdown()  // Your onShutdown() method runs
+```
+
+---
+
+## Timing Options
+
+### `before` - Modify Input
+```typescript
+class ValidationAugmentation extends BaseAugmentation {
+  readonly timing = 'before' as const
+  
+  async execute<T>(operation: string, params: any): Promise<any> {
+    if (operation === 'add') {
+      // Validate and/or modify params
+      if (!params.content) {
+        throw new Error('Content required')
+      }
+      // Return modified params
+      return { ...params, validated: true }
+    }
+  }
+}
+```
+
+### `after` - React to Results
+```typescript
+class LoggingAugmentation extends BaseAugmentation {
+  readonly timing = 'after' as const
+  
+  async execute<T>(operation: string, params: any): Promise<void> {
+    if (operation === 'search') {
+      console.log(`Search for "${params.query}" returned ${params.result.length} results`)
+    }
+    // Don't return anything - just observe
+  }
+}
+```
+
+### `both` - Before AND After
+```typescript
+class TimingAugmentation extends BaseAugmentation {
+  readonly timing = 'both' as const
+  private startTime?: number
+  
+  async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
+    if (!this.startTime) {
+      // Before execution
+      this.startTime = Date.now()
+    } else {
+      // After execution
+      const duration = Date.now() - this.startTime
+      console.log(`${operation} took ${duration}ms`)
+      this.startTime = undefined
+    }
+  }
+}
+```
+
+---
+
+## Operation Hooks
+
+### Core Operations You Can Hook
+```typescript
+readonly operations = [
+  'add',            // Adding data
+  'update',         // Updating data
+  'delete',         // Deleting data
+  'get',            // Retrieving data
+  'search',         // Searching
+  'find',           // Triple Intelligence queries
+  'relate',         // Adding relationships
+  'unrelate',       // Removing relationships
+  'clear',          // Clearing data
+  'all'            // Hook ALL operations
+] as const
+```
+
+### Example: Multi-Operation Hook
+```typescript
+class AuditAugmentation extends BaseAugmentation {
+  readonly operations = ['add', 'update', 'delete'] as const
+  
+  async execute<T>(operation: string, params: any): Promise<void> {
+    // Log all data modifications
+    await this.logToAuditTrail(operation, params)
+  }
+}
+```
+
+---
+
+## Accessing Brain Context
+
+```typescript
+class ContextAwareAugmentation extends BaseAugmentation {
+  async execute<T>(
+    operation: string,
+    params: any,
+    context?: AugmentationContext
+  ): Promise<void> {
+    // Access the brain instance
+    const brain = context?.brain
+    if (!brain) return
+    
+    // Use any brain method
+    const stats = await brain.getStats()
+    const size = await brain.size()
+    const results = await brain.search('query')
+    
+    // Access other augmentations
+    const cache = brain.augmentations.get('cache')
+    if (cache) {
+      await cache.clear()
+    }
+  }
+}
+```
+
+---
+
+## Real-World Examples
+
+### 1. Backup Augmentation
+```typescript
+class BackupAugmentation extends BaseAugmentation {
+  readonly name = 'backup'
+  readonly timing = 'after' as const
+  readonly operations = ['add', 'update', 'delete'] as const
+  readonly priority = 5
+  
+  private changes = 0
+  private readonly backupThreshold = 100
+  
+  async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
+    this.changes++
+    
+    if (this.changes >= this.backupThreshold) {
+      await this.performBackup(context?.brain)
+      this.changes = 0
+    }
+  }
+  
+  private async performBackup(brain?: any): Promise<void> {
+    if (!brain) return
+    const backup = await brain.backup()
+    await this.saveToCloud(backup)
+    console.log('Automatic backup completed')
+  }
+}
+```
+
+### 2. Rate Limiting Augmentation
+```typescript
+class RateLimitAugmentation extends BaseAugmentation {
+  readonly name = 'rate-limit'
+  readonly timing = 'before' as const
+  readonly operations = ['search', 'find'] as const
+  readonly priority = 100  // High priority - run first
+  
+  private requests = new Map<string, number[]>()
+  private readonly limit = 100  // 100 requests
+  private readonly window = 60000  // per minute
+  
+  async execute<T>(operation: string, params: any): Promise<void> {
+    const now = Date.now()
+    const key = params.userId || 'anonymous'
+    
+    // Get request timestamps
+    const timestamps = this.requests.get(key) || []
+    
+    // Remove old timestamps
+    const recent = timestamps.filter(t => now - t < this.window)
+    
+    // Check limit
+    if (recent.length >= this.limit) {
+      throw new Error('Rate limit exceeded')
+    }
+    
+    // Add current request
+    recent.push(now)
+    this.requests.set(key, recent)
+  }
+}
+```
+
+### 3. Encryption Augmentation
+```typescript
+class EncryptionAugmentation extends BaseAugmentation {
+  readonly name = 'encryption'
+  readonly timing = 'both' as const
+  readonly operations = ['add', 'get'] as const
+  readonly priority = 90  // Run early
+  
+  async execute<T>(operation: string, params: any): Promise<any> {
+    if (operation === 'add') {
+      // Encrypt before storing
+      if (params.metadata?.sensitive) {
+        params.content = await this.encrypt(params.content)
+        params.encrypted = true
+      }
+      return params
+    }
+
+    if (operation === 'get' && params.result?.encrypted) {
+      // Decrypt after retrieval
+      params.result.content = await this.decrypt(params.result.content)
+      delete params.result.encrypted
+      return params.result
+    }
+  }
+}
+```
+
+---
+
+## Testing Your Augmentation
+
+```typescript
+import { describe, it, expect } from 'vitest'
+import { Brainy } from '@soulcraft/brainy'
+import { MyAugmentation } from './my-augmentation'
+
+describe('MyAugmentation', () => {
+  it('should hook into addNoun', async () => {
+    const brain = new Brainy({ storage: 'memory' })
+    const aug = new MyAugmentation()
+    
+    // Spy on the execute method
+    const executeSpy = vi.spyOn(aug, 'execute')
+    
+    brain.augmentations.register(aug)
+    await brain.init()
+    
+    // Trigger the augmentation
+    await brain.add('test data')
+
+    // Verify it was called
+    expect(executeSpy).toHaveBeenCalledWith(
+      'add',
+      expect.objectContaining({ content: 'test data' }),
+      expect.any(Object)
+    )
+  })
+})
+```
+
+---
+
+## Best Practices
+
+### 1. Use Proper Timing
+- `before`: Validation, modification, rate limiting
+- `after`: Logging, metrics, side effects
+- `both`: Timing, tracing, wrapping
+
+### 2. Set Appropriate Priority
+```typescript
+// Priority guidelines
+100: Critical (auth, rate limiting)
+50:  Important (validation, transformation)
+10:  Normal (logging, metrics)
+1:   Optional (debugging, tracing)
+```
+
+### 3. Handle Errors Gracefully
+```typescript
+async execute<T>(operation: string, params: any): Promise<void> {
+  try {
+    await this.riskyOperation()
+  } catch (error) {
+    // Log but don't break the main operation
+    console.error(`Augmentation error in ${this.name}:`, error)
+    // Optionally report to monitoring
+    this.reportError(error)
+  }
+}
+```
+
+### 4. Be Performance Conscious
+```typescript
+class CachedAugmentation extends BaseAugmentation {
+  private cache = new Map<string, any>()
+  
+  async execute<T>(operation: string, params: any): Promise<any> {
+    const key = this.getCacheKey(params)
+    
+    // Check cache first
+    if (this.cache.has(key)) {
+      return this.cache.get(key)
+    }
+    
+    // Expensive operation
+    const result = await this.expensiveOperation(params)
+    this.cache.set(key, result)
+    
+    return result
+  }
+}
+```
+
+### 5. Clean Up Resources
+```typescript
+protected async onShutdown(): Promise<void> {
+  // Close connections
+  await this.connection?.close()
+  
+  // Clear intervals
+  clearInterval(this.interval)
+  
+  // Flush buffers
+  await this.flush()
+  
+  // Clear caches
+  this.cache.clear()
+}
+```
+
+---
+
+## Publishing Your Augmentation (Future)
+
+### Package Structure
+```
+my-augmentation/
+├── src/
+│   └── index.ts          # Your augmentation
+├── dist/                 # Built output
+├── tests/
+│   └── augmentation.test.ts
+├── package.json
+├── tsconfig.json
+└── README.md
+```
+
+### package.json
+```json
+{
+  "name": "@mycompany/brainy-custom-augmentation",
+  "version": "1.0.0",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "keywords": ["brainy-augmentation"],
+  "peerDependencies": {
+    "@soulcraft/brainy": ">=2.0.0"
+  },
+  "brainy": {
+    "type": "augmentation",
+    "class": "CustomAugmentation",
+    "timing": "after",
+    "operations": ["add"],
+    "priority": 10
+  }
+}
+```
+
+### Future: Brain Cloud Registry
+```bash
+# Coming in 2.1+
+npm run build
+npm test
+brainy publish  # Publishes to brain-cloud registry
+```
+
+---
+
+## FAQ
+
+### Q: Can I modify the operation result?
+**A**: Yes, if `timing: 'before'`, return modified params. If `timing: 'after'`, you can see but not modify results.
+
+### Q: Can augmentations communicate?
+**A**: Yes, through the context: `context.brain.augmentations.get('other-augmentation')`
+
+### Q: What if my augmentation fails?
+**A**: Handle errors internally. Don't break the main operation unless critical.
+
+### Q: Can I use async operations?
+**A**: Yes, everything is async-friendly.
+
+### Q: How do I access storage directly?
+**A**: Through context: `context.brain.storage` (but prefer using brain methods)
+
+---
+
+## Get Help
+
+- **GitHub**: [github.com/soulcraft/brainy](https://github.com/soulcraft/brainy)
+- **Discord**: [discord.gg/brainy](https://discord.gg/brainy)
+- **Examples**: See `/examples/augmentations/` in the repo
+
+---
+
+*Start building your augmentation today! The marketplace is coming in 2.1 🚀*

docs/augmentations/README.md

@@ -0,0 +1,204 @@
+# Brainy Augmentations
+
+Augmentations are the core extensibility mechanism in Brainy. They allow you to modify, enhance, and extend Brainy's behavior without changing the core code.
+
+## Core Principle: One Interface, Infinite Possibilities
+
+Every augmentation implements the same simple `BrainyAugmentation` interface:
+
+```typescript
+interface BrainyAugmentation {
+  name: string
+  timing: 'before' | 'after' | 'around' | 'replace'
+  operations: string[]
+  priority: number
+  initialize(context): Promise<void>
+  execute(operation, params, next): Promise<any>
+}
+```
+
+This single interface can handle EVERYTHING - from adding AI capabilities to exposing APIs to replacing storage backends.
+
+## Available Augmentations
+
+### 🧠 Data Processing
+Augmentations that enhance how data is processed and stored.
+
+| Augmentation | Description | Timing | Status |
+|-------------|-------------|--------|--------|
+| **NeuralImportAugmentation** | AI-powered entity and relationship extraction | `before` | ✅ Production |
+| **EntityRegistryAugmentation** | High-performance entity deduplication | `before` | ✅ Production |
+| **BatchProcessingAugmentation** | Optimizes bulk operations | `around` | ✅ Production |
+| **IntelligentVerbScoringAugmentation** | Learns relationship importance over time | `after` | ✅ Production |
+
+### 🔌 External Connections (Synapses)
+Connect Brainy to external services and data sources.
+
+| Augmentation | Description | Timing | Status |
+|-------------|-------------|--------|--------|
+| **NotionSynapse** | Sync with Notion databases | `after` | 📝 Example |
+| **SalesforceSynapse** | Connect to Salesforce CRM | `after` | 📝 Example |
+| **SlackSynapse** | Import Slack conversations | `after` | 📝 Example |
+| **GoogleDriveSynapse** | Sync Google Drive documents | `after` | 📝 Example |
+
+### 🌐 API Exposure
+Expose Brainy through various protocols.
+
+| Augmentation | Description | Timing | Status |
+|-------------|-------------|--------|--------|
+| **APIServerAugmentation** | REST, WebSocket, and MCP server | `after` | ✅ Production |
+| **GraphQLAugmentation** | GraphQL API endpoint | `after` | 🚧 Planned |
+| **gRPCAugmentation** | gRPC service | `after` | 🚧 Planned |
+
+### 💾 Storage Backends
+Replace or enhance the storage layer.
+
+| Augmentation | Description | Timing | Status |
+|-------------|-------------|--------|--------|
+| **S3StorageAugmentation** | Use S3 as storage backend | `replace` | 📝 Example |
+| **RedisAugmentation** | Redis caching layer | `around` | 📝 Example |
+| **PostgresAugmentation** | PostgreSQL persistence | `replace` | 📝 Example |
+
+### 🔄 Real-time & Sync
+Handle real-time updates and synchronization.
+
+| Augmentation | Description | Timing | Status |
+|-------------|-------------|--------|--------|
+| **WebSocketConduitAugmentation** | WebSocket client connections | `after` | ⚠️ Legacy |
+| **ServerSearchAugmentation** | Connect to remote Brainy servers | `after` | ⚠️ Legacy |
+| **TeamCoordinationAugmentation** | Multi-agent synchronization | `after` | 📝 Example |
+
+### 🛡️ Infrastructure
+Core infrastructure and reliability features.
+
+| Augmentation | Description | Timing | Status |
+|-------------|-------------|--------|--------|
+| **ConnectionPoolAugmentation** | Optimize cloud storage connections | `before` | ✅ Production |
+| **RequestDeduplicatorAugmentation** | Prevent duplicate concurrent requests | `before` | ✅ Production |
+| **TransactionAugmentation** | ACID transaction support | `around` | 🚧 Planned |
+| **CacheAugmentation** | Multi-level caching | `around` | ✅ Production |
+
+### 📊 Monitoring & Analytics
+Track and analyze Brainy's behavior.
+
+| Augmentation | Description | Timing | Status |
+|-------------|-------------|--------|--------|
+| **MetricsAugmentation** | Prometheus metrics | `after` | 📝 Example |
+| **LoggingAugmentation** | Structured logging | `after` | 📝 Example |
+| **TracingAugmentation** | Distributed tracing | `around` | 🚧 Planned |
+
+### 🤖 AI & Chat
+AI-powered interfaces and chat capabilities.
+
+| Augmentation | Description | Timing | Status |
+|-------------|-------------|--------|--------|
+| **ChatInterfaceAugmentation** | Natural language interface | `before` | 📝 Example |
+| **MCPAgentMemoryAugmentation** | AI agent memory via MCP | `after` | 📝 Example |
+| **LLMQueryAugmentation** | LLM-enhanced queries | `before` | 📝 Example |
+
+### 📈 Visualization
+Visual representations of data.
+
+| Augmentation | Description | Timing | Status |
+|-------------|-------------|--------|--------|
+| **GraphVisualizationAugmentation** | Real-time graph visualization | `after` | 📝 Example |
+| **DashboardAugmentation** | Web-based dashboard | `after` | 🚧 Planned |
+
+## Status Legend
+
+- ✅ **Production**: Fully implemented and tested
+- 📝 **Example**: Example implementation available
+- 🚧 **Planned**: On the roadmap
+- ⚠️ **Legacy**: Being replaced by newer augmentations
+
+## Using Augmentations
+
+### Zero-Config Approach
+
+```typescript
+const brain = new Brainy()
+
+// Just register augmentations - they work automatically!
+brain.augmentations.register(new EntityRegistryAugmentation())
+brain.augmentations.register(new APIServerAugmentation())
+
+await brain.init()
+```
+
+### With Configuration
+
+```typescript
+const brain = new Brainy()
+
+brain.augmentations.register(
+  new APIServerAugmentation({
+    port: 8080,
+    auth: { required: true }
+  })
+)
+
+await brain.init()
+```
+
+## Creating Custom Augmentations
+
+See [Creating Custom Augmentations](./creating-augmentations.md) for a complete guide.
+
+Quick example:
+
+```typescript
+class MyAugmentation extends BaseAugmentation {
+  readonly name = 'my-augmentation'
+  readonly timing = 'after'
+  readonly operations = ['add', 'search']
+  readonly priority = 50
+  
+  async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
+    console.log(`Before ${operation}`)
+    const result = await next()
+    console.log(`After ${operation}`)
+    return result
+  }
+}
+```
+
+## Augmentation Timing
+
+### `before`
+Executes before the main operation. Used for:
+- Input validation
+- Data transformation
+- Authentication checks
+
+### `after`
+Executes after the main operation. Used for:
+- Broadcasting updates
+- Syncing to external services
+- Logging and metrics
+
+### `around`
+Wraps the main operation. Used for:
+- Transactions
+- Caching
+- Error handling
+
+### `replace`
+Completely replaces the main operation. Used for:
+- Alternative storage backends
+- Mock implementations
+- Proxy operations
+
+## Priority System
+
+Higher numbers execute first:
+- **100**: Critical system operations
+- **50**: Performance optimizations
+- **10**: Enhancement features
+- **1**: Optional features
+
+## Related Documentation
+
+- [API Server Augmentation](./api-server.md) - Complete API server documentation
+- [Creating Augmentations](./creating-augmentations.md) - How to build your own
+- [Augmentation Examples](../AUGMENTATION-EXAMPLES.md) - Real-world examples
+- [Architecture Overview](../COMPLETE-ARCHITECTURE-VISION.md) - System architecture

docs/augmentations/api-server.md

@@ -0,0 +1,403 @@
+# API Server Augmentation
+
+## Overview
+
+The `APIServerAugmentation` is a powerful augmentation that exposes your Brainy instance through REST, WebSocket, and MCP (Model Context Protocol) APIs. It transforms Brainy into a full-featured API server with zero configuration required.
+
+## Features
+
+### 🌐 REST API
+Complete CRUD operations and advanced queries through HTTP endpoints.
+
+### 🔌 WebSocket Server
+Real-time bidirectional communication with automatic operation broadcasting.
+
+### 🧠 MCP Integration
+Built-in Model Context Protocol support for AI agent communication.
+
+### 📊 Operation Broadcasting
+Automatically broadcasts all Brainy operations to subscribed WebSocket clients.
+
+### 🔒 Optional Security
+Built-in authentication and rate limiting when needed.
+
+## Installation
+
+The APIServerAugmentation is included in Brainy core. No additional installation required.
+
+For Node.js environments, you may want to install optional dependencies:
+```bash
+npm install express cors ws
+```
+
+## Zero-Config Usage
+
+```typescript
+import { Brainy } from 'brainy'
+import { APIServerAugmentation } from 'brainy/augmentations'
+
+const brain = new Brainy()
+
+// Register the API server augmentation
+brain.augmentations.register(new APIServerAugmentation())
+
+await brain.init()
+
+// Server is now running at http://localhost:3000
+console.log('API Server ready!')
+console.log('REST: http://localhost:3000/api/*')
+console.log('WebSocket: ws://localhost:3000/ws')
+console.log('MCP: http://localhost:3000/api/mcp')
+```
+
+## Configuration Options
+
+While zero-config works great, you can customize the server:
+
+```typescript
+const apiServer = new APIServerAugmentation({
+  enabled: true,           // Enable/disable the server
+  port: 3000,             // HTTP port
+  host: '0.0.0.0',        // Bind address
+  
+  cors: {
+    origin: '*',          // CORS allowed origins
+    credentials: true     // Allow credentials
+  },
+  
+  auth: {
+    required: false,      // Require authentication
+    apiKeys: [],         // Valid API keys
+    bearerTokens: []     // Valid bearer tokens
+  },
+  
+  rateLimit: {
+    windowMs: 60000,     // Rate limit window (ms)
+    max: 100            // Max requests per window
+  }
+})
+```
+
+## REST API Endpoints
+
+### Health Check
+```http
+GET /health
+```
+Returns server status and basic metrics.
+
+### Search
+```http
+POST /api/search
+Content-Type: application/json
+
+{
+  "query": "search text",
+  "limit": 10,
+  "options": {}
+}
+```
+
+### Add Data
+```http
+POST /api/add
+Content-Type: application/json
+
+{
+  "content": "data to add",
+  "metadata": {
+    "key": "value"
+  }
+}
+```
+
+### Get by ID
+```http
+GET /api/get/:id
+```
+
+### Delete
+```http
+DELETE /api/delete/:id
+```
+
+### Create Relationship
+```http
+POST /api/relate
+Content-Type: application/json
+
+{
+  "source": "id1",
+  "target": "id2",
+  "verb": "relates_to",
+  "metadata": {}
+}
+```
+
+### Complex Queries
+```http
+POST /api/find
+Content-Type: application/json
+
+{
+  "where": { "type": "document" },
+  "like": "machine learning",
+  "limit": 10
+}
+```
+
+### Clustering
+```http
+POST /api/cluster
+Content-Type: application/json
+
+{
+  "algorithm": "kmeans",
+  "options": {
+    "k": 5
+  }
+}
+```
+
+### Statistics
+```http
+GET /api/stats
+```
+
+### Operation History
+```http
+GET /api/history
+```
+
+## WebSocket API
+
+### Connection
+```javascript
+const ws = new WebSocket('ws://localhost:3000/ws')
+
+ws.onopen = () => {
+  console.log('Connected to Brainy WebSocket')
+}
+
+ws.onmessage = (event) => {
+  const msg = JSON.parse(event.data)
+  console.log('Received:', msg)
+}
+```
+
+### Subscribe to Operations
+```javascript
+ws.send(JSON.stringify({
+  type: 'subscribe',
+  operations: ['all']  // or specific: ['add', 'search', 'delete']
+}))
+```
+
+### Search via WebSocket
+```javascript
+ws.send(JSON.stringify({
+  type: 'search',
+  query: 'your search',
+  limit: 10,
+  requestId: 'unique-id'
+}))
+```
+
+### Add Data via WebSocket
+```javascript
+ws.send(JSON.stringify({
+  type: 'add',
+  content: 'data to add',
+  metadata: {},
+  requestId: 'unique-id'
+}))
+```
+
+### Operation Broadcasts
+When subscribed, you'll receive real-time updates:
+```javascript
+{
+  "type": "operation",
+  "operation": "add",
+  "params": { /* sanitized parameters */ },
+  "timestamp": 1234567890,
+  "duration": 15
+}
+```
+
+## MCP (Model Context Protocol)
+
+The MCP endpoint allows AI agents to interact with Brainy:
+
+```http
+POST /api/mcp
+Content-Type: application/json
+
+{
+  "method": "search",
+  "params": {
+    "query": "find documents about AI"
+  }
+}
+```
+
+## Authentication
+
+When authentication is enabled:
+
+### API Key
+```http
+GET /api/stats
+X-API-Key: your-api-key
+```
+
+### Bearer Token
+```http
+GET /api/stats
+Authorization: Bearer your-token
+```
+
+## Environment Support
+
+### Node.js ✅
+Full support with Express, WebSocket, and all features.
+
+### Deno 🚧
+Planned support using Deno.serve() or oak framework.
+
+### Browser/Service Worker 🚧
+Planned support for intercepting fetch() calls locally.
+
+## How It Works
+
+The APIServerAugmentation hooks into Brainy's augmentation pipeline:
+
+1. **Timing**: Executes `after` operations complete
+2. **Operations**: Monitors `all` operations
+3. **Broadcasting**: Sends operation details to subscribed clients
+4. **History**: Maintains operation history (last 1000 operations)
+
+## Example: Multi-Client Sync
+
+```typescript
+// Server
+const brain = new Brainy()
+brain.augmentations.register(new APIServerAugmentation())
+await brain.init()
+
+// Client 1 - WebSocket subscriber
+const ws1 = new WebSocket('ws://localhost:3000/ws')
+ws1.onopen = () => {
+  ws1.send(JSON.stringify({
+    type: 'subscribe',
+    operations: ['add', 'delete']
+  }))
+}
+ws1.onmessage = (e) => {
+  console.log('Client 1 received update:', JSON.parse(e.data))
+}
+
+// Client 2 - REST API user
+fetch('http://localhost:3000/api/add', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body: JSON.stringify({
+    content: 'New data',
+    metadata: { source: 'client2' }
+  })
+})
+// Client 1 automatically receives notification!
+```
+
+## Performance Considerations
+
+- **Operation History**: Limited to last 1000 operations
+- **WebSocket Heartbeat**: Every 30 seconds
+- **Client Timeout**: 60 seconds of inactivity
+- **Parameter Sanitization**: Sensitive fields removed, large content truncated
+- **Rate Limiting**: In-memory tracking (use Redis in production)
+
+## Security Notes
+
+1. **Default Configuration**: No auth, open CORS - suitable for development
+2. **Production**: Enable auth, configure CORS, use HTTPS
+3. **Sensitive Data**: Parameters are sanitized before broadcasting
+4. **Rate Limiting**: Basic in-memory implementation included
+
+## Comparison with Previous Implementations
+
+The APIServerAugmentation unifies and replaces:
+- `BrainyMCPBroadcast` - Node-specific WebSocket/HTTP server
+- `WebSocketConduitAugmentation` - WebSocket client functionality  
+- `ServerSearchAugmentations` - Remote Brainy connections
+
+Benefits of the unified approach:
+- Single augmentation for all API needs
+- Consistent interface across protocols
+- Automatic operation broadcasting
+- Environment-aware implementation
+- Zero-configuration philosophy
+
+## Advanced Usage
+
+### Custom Operation Filtering
+
+```typescript
+class FilteredAPIServer extends APIServerAugmentation {
+  shouldExecute(operation: string, params: any): boolean {
+    // Don't broadcast sensitive operations
+    if (operation === 'delete' && params.sensitive) {
+      return false
+    }
+    return true
+  }
+}
+```
+
+### Integration with Other Augmentations
+
+```typescript
+const brain = new Brainy()
+
+// Stack augmentations for complete system
+brain.augmentations.register(new EntityRegistryAugmentation()) // Dedup
+brain.augmentations.register(new APIServerAugmentation())      // API
+
+await brain.init()
+// All augmentations work together seamlessly!
+```
+
+## Troubleshooting
+
+### Server won't start
+- Check if port is already in use
+- Verify Node.js dependencies are installed: `npm install express cors ws`
+- Check console for error messages
+
+### WebSocket connections drop
+- Ensure heartbeat responses are handled
+- Check for proxy/firewall issues
+- Verify CORS configuration
+
+### Authentication not working
+- Ensure `auth.required` is set to `true`
+- Verify API keys or bearer tokens are correctly configured
+- Check request headers are properly formatted
+
+## Future Enhancements
+
+- [ ] Deno server implementation
+- [ ] Service Worker implementation  
+- [ ] GraphQL endpoint
+- [ ] gRPC support
+- [ ] Built-in SSL/TLS
+- [ ] Redis-based rate limiting
+- [ ] Prometheus metrics endpoint
+- [ ] OpenAPI/Swagger documentation
+
+## Related Documentation
+
+- [Augmentation System Overview](../AUGMENTATION-SYSTEM.md)
+- [BrainyAugmentation Interface](./brainy-augmentation.md)
+- [MCP Integration](../mcp/README.md)
+- [Zero-Config Philosophy](../ZERO-CONFIG.md)

docs/concepts/consistency-model.md

@@ -1,386 +0,0 @@
----
-title: Consistency Model
-slug: concepts/consistency-model
-public: true
-category: concepts
-template: concept
-order: 4
-description: The exact guarantees behind Brainy's Db API — snapshot isolation, atomic transactions, two levels of compare-and-swap, time travel, retention, snapshots, crash recovery, and the reserved-field contract.
-next:
-  - guides/snapshots-and-time-travel
-  - guides/optimistic-concurrency
----
-
-# Consistency Model
-
-Brainy 8.0's consistency story rests on one mechanism: **generational MVCC**
-— multi-version concurrency control over immutable, generation-stamped
-records. It is exposed through a single value type, the **`Db`**: an
-immutable, point-in-time view of the whole store that you query like the
-live brain.
-
-```typescript
-const db = brain.now()                 // pin the current state — O(1), no I/O
-
-await brain.transact([
-  { op: 'update', id: invoiceId, metadata: { status: 'paid' } }
-])
-
-await db.get(invoiceId)                // still 'pending' — pinned, forever
-await brain.get(invoiceId)             // 'paid' — live
-await db.release()                     // unpin when done
-```
-
-This page states the guarantees precisely — what is promised, what it costs,
-and where the honest limits are. The design record is
-[ADR-001](../ADR-001-generational-mvcc.md); every guarantee below is proven
-by a dedicated test in `tests/integration/db-mvcc.test.ts`.
-
-## The generation clock
-
-A **monotonic generation counter** is the store's logical clock:
-
-- It advances **once per committed `transact()` batch** and once per
-  single-operation write (`add`/`update`/`remove`/`relate`/…).
-- `brain.generation()` reads it; it is persisted in the data directory and
-  **never reissued** — not across restarts, and not across `restore()`
-  (the counter is floored at its pre-restore value).
-
-Every `Db` is pinned at one generation. `db.generation` and `db.timestamp`
-identify the view; `newerDb.since(olderDb)` returns exactly the entity and
-relationship ids that committed transactions touched between two views.
-
-## Snapshot isolation for reads
-
-**Guarantee:** a `Db` reads exactly the state at its pinned generation, no
-matter what commits afterwards — including deletes. There are no torn reads,
-no partially applied batches, and no drift over time.
-
-- `brain.now()` pins the current generation in O(1).
-- `brain.transact()` returns a `Db` pinned at the freshly committed
-  generation.
-- `brain.asOf(generation | Date | snapshotPath)` pins past state.
-
-While nothing has committed past the pin, reads delegate to the live fast
-paths — pinning is free until history actually moves. Once later
-transactions commit, the view keeps serving the **full query surface** at
-its generation (see "Reading the past" below).
-
-Writers are never blocked by readers and readers never block writers: a
-pinned view stays valid because nothing overwrites the immutable records it
-resolves from (the LMDB reader-pin model).
-
-## Transaction atomicity
-
-`brain.transact(ops)` executes a declarative batch — `add`, `update`,
-`remove`, `relate`, `unrelate` — **atomically as exactly one generation**:
-
-```typescript
-const db = await brain.transact([
-  { op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' },
-  { op: 'add', id: itemId, type: NounType.Thing, subtype: 'line-item', data: 'Widget x3' },
-  { op: 'relate', from: orderId, to: itemId, type: VerbType.Contains, subtype: 'order-line' }
-], { meta: { author: 'order-service', requestId: 'req-9f2' } })
-
-db.receipt.ids   // resolved id per operation, in input order
-```
-
-Either every operation applies, or none do and the store is byte-identical
-to its pre-transaction state. Operation semantics mirror the corresponding
-single-operation methods — validation, subtype enforcement, relationship
-deduplication, delete cascades — and later operations may reference ids
-created earlier in the same batch.
-
-**The commit point is one atomic rename.** The durability protocol:
-
-1. Before-images of every touched id are staged into an immutable
-   generation directory and **fsynced**.
-2. The batch executes through the transaction manager (which has its own
-   operation-level rollback for non-crash failures).
-3. The store manifest is replaced via atomic temp-file rename and fsynced.
-   **The rename is the commit** — a generation is committed if and only if
-   the manifest says so.
-
-**Crash recovery:** on the next open, any staged generation above the
-manifest watermark is an uncommitted transaction; its before-images are
-restored (idempotently — recovery can itself crash and rerun) and derived
-indexes never observe the rolled-back state. A crash anywhere before the
-rename rolls back to the exact pre-transaction bytes; a crash after it keeps
-the transaction.
-
-Transaction metadata (`meta`) is reified Datomic-style: recorded in an
-append-only transaction log readable via `brain.transactionLog()` — audit
-fields live in the database, not in commit messages.
-
-## Two levels of compare-and-swap
-
-Concurrent `transact()` calls commit serially (snapshot-isolated batches).
-For lost-update protection across a read–modify–write cycle, Brainy offers
-CAS at two granularities:
-
-| Granularity | Mechanism | Conflict error | Use when |
-|---|---|---|---|
-| **Per entity** | `_rev` + `{ op: 'update', ifRev }` (also on `brain.update()`) | `RevisionConflictError` | "This entity must not have changed since I read it." |
-| **Whole store** | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` | "*Nothing* may have committed since I read." |
-
-```typescript
-const view = brain.now()
-const order = await view.get(orderId)
-
-try {
-  await brain.transact(
-    [{ op: 'update', id: orderId, metadata: { total: recompute(order) }, ifRev: order._rev }],
-    { ifAtGeneration: view.generation }
-  )
-} catch (err) {
-  if (err instanceof GenerationConflictError) {
-    // Something committed since the pin — re-read and retry.
-  }
-} finally {
-  await view.release()
-}
-```
-
-An `ifRev` conflict on any operation rejects the **whole batch**; an
-`ifAtGeneration` conflict is detected before anything is staged. Both leave
-the store untouched and the generation counter unchanged. See
-[Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md)
-for the per-entity pattern in depth.
-
-## Reading the past
-
-`brain.asOf()` accepts a generation number, a `Date` (resolved through the
-transaction log to the newest generation committed at or before it), or a
-snapshot directory path. Historical views serve the **full query surface**
-— `get()`, `find()` in every mode, semantic search, graph traversal,
-cursors, aggregation — through two complementary paths:
-
-- **Record path** (free): `get()`, metadata-level `find()`, and
-  filter-based `related()` resolve directly through the immutable record
-  layer. Ids untouched since the pin still ride the live fast paths.
-- **Index path** (paid once): index-accelerated queries — semantic/vector
-  search, graph traversal, cursors, aggregation — are served by an
-  **at-generation index materialization** built lazily on first use:
-  Brainy reconstructs in-memory indexes over the exact record set at that
-  generation. This costs O(n at the pinned generation) time and memory,
-  **once per `Db`**, cached until `release()`. That is the open-core price
-  of historical index queries, stated plainly.
-
-A native index provider implementing the optional
-`VersionedIndexProvider` plugin capability serves the same historical reads
-from its retained index segments **without any rebuild** — the materializer
-is the correctness baseline, the provider is the accelerator. Semantics are
-identical on both paths.
-
-### History granularity — the honest limit
-
-Generation *records* are written per `transact()` batch only.
-Single-operation writes (`add`/`update`/`remove`/`relate`/… outside
-`transact()`) advance the generation counter — so watermarks and CAS stay
-sound — but do **not** stage before-images: they remain visible through
-earlier pins and are not reported by `db.since()`. Code that needs pinned
-isolation across its own writes uses `transact()`. This is the documented
-8.0 contract, not an accident.
-
-## Speculative writes: `db.with()`
-
-`db.with(ops)` returns a new `Db` whose reads see the operations applied
-**in memory, on top of the view** — Datomic's `with`. Nothing touches disk,
-the generation counter, or index providers:
-
-```typescript
-const current = brain.now()
-const whatIf = await current.with([
-  { op: 'update', id: employeeId, metadata: { team: 'platform' } }
-])
-
-await whatIf.find({ where: { team: 'platform' } })   // sees the change
-await brain.get(employeeId)                          // unchanged — nothing committed
-```
-
-**The one boundary:** overlay entities carry no embeddings (`with()` never
-invokes the embedder), so index-accelerated queries and `persist()` on a
-speculative view throw `SpeculativeOverlayError` rather than returning
-silently incomplete results. `get()`, metadata-filter `find()`, and
-filter-based `related()` work fully on overlays. To get the full surface,
-commit the same operations with `brain.transact()`.
-
-## Retention and compaction
-
-Historical records cost disk space, so retention is explicit:
-
-- Every live `Db` holds a refcounted **pin**; a record-set is never
-  reclaimed while any pin could need it — pinned reads stay correct across
-  compaction, always.
-- The **`retention`** knob governs auto-compaction (on every `flush()`/
-  `close()`): unset → ADAPTIVE (disk/RAM-pressure, zero-config) · `'all'` →
-  unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit caps.
-  `brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
-  manually on the same caps, and records the **horizon** — `asOf()` below it
-  throws `GenerationCompactedError`, explicitly, never partial data.
-- To keep a state readable forever, `persist()` it first: snapshots are
-  self-contained and unaffected by compaction of the source store.
-
-Release `Db` values you do not keep (including the ones `transact()`
-returns). A `FinalizationRegistry` backstop releases leaked pins at garbage
-collection, but explicit `release()` is what makes compaction
-deterministic.
-
-## Durability: snapshots and restore
-
-`db.persist(path)` cuts a **self-contained snapshot** under the store's
-commit mutex, so no commit or compaction can interleave. On filesystem
-storage it is built from **hard links**: because every data file is
-immutable-by-rename, linking is safe — the snapshot is created without
-copying entity data, shares disk space with the source, and later writes to
-the source can never alter it (rewrites swap inodes; the snapshot keeps the
-old bytes). Cross-device targets fall back to byte copies; in-memory stores
-serialize to the same directory layout, producing a real, durable store.
-
-Two rules keep snapshots honest:
-
-- `persist()` requires the view to still be the store's **latest**
-  generation (a snapshot captures current bytes); a view that history has
-  moved past throws `GenerationConflictError` instead of persisting the
-  wrong state.
-- `brain.restore(path, { confirm: true })` replaces the store's entire
-  state from a snapshot via byte copy (never links — the snapshot stays
-  independent), rebuilds all indexes, and floors the generation counter so
-  observed generation numbers are never reissued. Live pins do not survive
-  a restore — release them first (a warning is logged when any exist).
-
-`Brainy.load(path)` (or `brain.asOf(path)`) opens a snapshot as a
-self-contained **read-only** store with the full query surface, including
-vector search.
-
-## Reserved fields
-
-Some field names belong to Brainy, not to your metadata. They live at **top
-level** on every entity and relationship, have dedicated write paths, and may
-never appear inside a `metadata` bag:
-
-| Entities (nouns) | Relationships (verbs) | Canonical write path |
-|---|---|---|
-| `noun` | `verb` | the `type` param of `add()` / `relate()` |
-| `subtype` | `subtype` | the `subtype` param |
-| `visibility` | `visibility` | the `visibility` param (`'public'` \| `'internal'`) |
-| `confidence` | `confidence` | the `confidence` param |
-| `weight` | `weight` | the `weight` param |
-| `service` | `service` | the `service` param (fixed at create time) |
-| `data` | `data` | the `data` param |
-| `createdBy` | `createdBy` | the `createdBy` param of `add()` (system-managed on verbs) |
-| `createdAt`, `updatedAt`, `_rev` | `createdAt`, `updatedAt`, `_rev` | system-managed (`ifRev` for CAS) |
-
-The canonical machine-readable lists are exported as
-`RESERVED_ENTITY_FIELDS` and `RESERVED_RELATION_FIELDS` (defined in
-`src/types/reservedFields.ts`, the single source of truth). Three layers
-enforce the contract:
-
-1. **Compile time** — every `metadata` param (`add`, `update`, `relate`,
-   `updateRelation`, and the matching `transact()` operations) rejects a
-   literal reserved key as a TypeScript error.
-2. **Write time** — untyped (JavaScript) callers that pass one anyway are
-   normalized: user-settable fields (`confidence`, `weight`, `subtype`, and
-   `service`/`createdBy` at create time) are remapped to their dedicated
-   param — **top-level wins** when both are supplied — and system-managed
-   fields are dropped with a one-shot warning naming the correct write path.
-   `update({ metadata: { confidence: 0.9 } })` therefore behaves exactly
-   like `update({ confidence: 0.9 })`.
-3. **Read time** — every read path (`get`, `find`, `search`,
-   `related`, batch reads, and historical `asOf()` materialization)
-   surfaces reserved fields **only at top level**: `entity.metadata` and
-   `relation.metadata` contain only your custom fields, always.
-
-```typescript
-const id = await brain.add({
-  type: 'document', subtype: 'invoice',
-  data: 'Invoice #42', confidence: 0.95,        // reserved → top-level params
-  metadata: { customer: 'acme', total: 129.5 }  // custom fields only
-})
-
-const entity = await brain.get(id)
-entity.confidence                                // 0.95 — top level
-entity.metadata                                  // { customer: 'acme', total: 129.5 }
-```
-
-### Visibility — `public` / `internal` / `system`
-
-`visibility` is a reserved tier that controls whether an entity or relationship
-surfaces on Brainy's **default** user-facing reads. The absence of the field is
-exactly equivalent to `'public'`.
-
-| Tier | Counted in `getNounCount()` / `stats()`? | Returned by default `find()` / `related()`? | Opt-in |
-|---|---|---|---|
-| `'public'` (default, or field absent) | yes | yes | — |
-| `'internal'` | no | no | `find({ includeInternal: true })` / `related({ includeInternal: true })` |
-| `'system'` | no | no | `find({ includeSystem: true })` / `related({ includeSystem: true })` |
-
-- **`'public'`** — normal data. Counted and returned everywhere. Stored lean:
-  the field is omitted on disk for public records, so existing data needs no
-  migration.
-- **`'internal'`** — your app's own bookkeeping (audit trails, derived caches,
-  scratch entities) that should not pollute default queries, counts, or
-  `stats()`, yet must stay retrievable on demand. Set it via the `visibility`
-  param; read it back with the `includeInternal` opt-in.
-- **`'system'`** — Brainy's own plumbing (for example the Virtual File System
-  root entity). Hidden everywhere by default — even when `includeInternal` is
-  set — and surfaced only with the explicit `includeSystem` opt-in. The
-  `'system'` tier is **not** part of the public `add()` / `relate()` param type
-  (`'public' | 'internal'`); only internal Brainy code assigns it.
-
-The opt-ins are applied as a **hard candidate filter** — hidden entities are
-removed before `limit` / `offset` are applied, so a default `find({ limit: 10 })`
-always returns ten *visible* results when that many exist, never a short page.
-
-```typescript
-// App-internal scratch entity: present, retrievable, but out of the way.
-await brain.add({ type: 'task', data: 'reindex job', visibility: 'internal' })
-
-await brain.getNounCount()                       // unchanged — internal not counted
-await brain.find({ type: 'task' })               // [] — hidden by default
-await brain.find({ type: 'task', includeInternal: true })   // includes it
-
-// A brand-new brain reports zero user entities even though the VFS root exists:
-const fresh = new Brainy()
-await fresh.init()
-await fresh.getNounCount()                        // 0 — the root is visibility:'system'
-```
-
-> **Note (8.0):** the structural `Contains` edges the VFS creates between
-> directories and files are left at the default (`public`) visibility for now —
-> only the VFS *root entity* is `'system'`. Marking those edges system requires
-> companion changes to VFS traversal and is out of scope for this change.
-
-## What is not guaranteed
-
-Stated plainly, so nothing surprises you in production:
-
-- **Single-writer.** Brainy is a single-writer, many-reader database
-  ([multi-process model](./multi-process.md)). Transactions are atomic
-  within one writer process — there is no distributed or cross-process
-  transaction coordination.
-- **History granularity.** Every write is its own immutable generation —
-  `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
-  A pin always freezes against later writes, and every write is addressable
-  via `asOf()`. (`transact()` groups several operations into ONE atomic
-  generation; durability of single-op history is batched via async
-  group-commit — a hard crash can lose only the last un-flushed window's
-  *history*, never live data.)
-- **Compacted history is gone.** `asOf()` below the compaction horizon
-  fails explicitly; persist what you must keep.
-- **Counter persistence is coalesced for single-operation writes.** Durable
-  artifacts (records, manifests, snapshots) always persist the counter at
-  their own commit points, so a crash inside the coalescing window can lose
-  only counter values nothing durable ever referenced.
-- **Speculative overlays are metadata-only readers.** Index-accelerated
-  queries on `with()` views throw rather than guess.
-
-## Where to go next
-
-- [Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) — the
-  recipes: backup, restore, time-travel debugging, what-if analysis, audit
-  trails.
-- [Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md)
-  — the per-entity CAS pattern.
-- [ADR-001: Generational MVCC](../ADR-001-generational-mvcc.md) — the full
-  design record, including the persisted layout and the proof table.

docs/concepts/generation-fact-log.md

@@ -1,117 +0,0 @@
----
-title: The Generation Fact Log
-slug: concepts/generation-fact-log
-public: true
-category: concepts
-template: concept
-order: 6
-description: Every committed write also appends a self-verifying "fact" — an after-image commit record — to an append-only log. What facts are, the crash-safety model, the scanFacts() streaming surface, family stamps, and how index providers consume the log for sequential heals.
-next:
-  - concepts/consistency-model
-  - guides/snapshots-and-time-travel
----
-
-# The Generation Fact Log
-
-Since 8.4.0, every committed generation also appends a **fact** — a compact record of what each
-touched entity or relationship *became* — to an append-only, checksummed log under
-`_generations/facts/`. Where the generational history answers *"what did things look like
-before?"* (before-images, powering `asOf()` and rollback), the fact log answers *"what happened,
-in order?"* — one sequential, self-verifying stream of the store's present being written.
-
-Nothing about querying changes. The fact log exists for three consumers:
-
-1. **Index heals and rebuilds** — one sequential read in commit order replaces a per-entity
-   directory walk over millions of files.
-2. **Incremental catch-up** — a derived index that knows which generation it reflects reads *just
-   the gap*, instead of rebuilding from scratch.
-3. **Replay and audit tooling** — anything that wants the store's committed timeline as a stream.
-
-## What a fact is
-
-One fact per committed generation:
-
-- **`generation`** and **`timestamp`** — which commit, when.
-- **`ops`** — every write in that commit: `{ kind: 'noun' | 'verb', id, record }` where `record`
-  holds the entity's full after-image (both stored legs), or **`null` for a tombstone** — a
-  removal carries no body, by design.
-- **`meta`** — the transaction metadata `transact()` was submitted with, when present.
-- **`blobHashes`** — content-blob references, for exact reclamation accounting.
-
-Facts accumulate **from the first write after upgrading** — pre-existing history is not
-retroactively converted, and consumers fall back to the enumeration walk when no log exists.
-
-## Crash safety, in one paragraph
-
-Facts are appended and fsynced **inside the same durability window as the commit itself**, before
-the commit point — so after a crash, the log can only ever be *ahead* of committed truth, never
-behind it with a hole. On open, the store reconciles the log back to the committed watermark:
-torn tails are detected by per-record checksums and cut; whole records beyond the watermark are
-truncated. The invariant every reader can rely on: **an absent generation was never committed; a
-present fact was.** `transact()` facts are durable the moment `transact()` returns; single-op
-facts share the same group-commit flush as the rest of their generation, so a hard kill loses the
-fact and the generation *together* — never a torn state.
-
-## Reading the log
-
-```typescript
-const scan = brain.scanFacts({ fromGeneration: 1 })
-if (scan) {
-  // Telemetry up front — progress bars get a denominator from second zero.
-  console.log(scan.headGeneration, scan.segmentCount, scan.approxFactCount)
-
-  for await (const batch of scan.batches()) {
-    // Each batch: { facts, firstGeneration, lastGeneration, factCount, byteSize, segmentId }
-    for (const fact of batch.facts) {
-      for (const op of fact.ops) {
-        if (op.record === null) {
-          // a tombstone: op.id was removed in this generation
-        }
-      }
-    }
-  }
-
-  console.log(scan.summary()) // { factsYielded, segmentsRead } — the cross-check
-}
-```
-
-- `scanFacts()` returns `null` when the store hosts no fact log (older store, or a storage adapter
-  without binary append support) — fall back to enumerating entities.
-- Scans run against a **snapshot**: facts appended after the scan opens never bleed in, each fact
-  is yielded exactly once, and a detected gap aborts loudly — never a silent skip.
-- `brain.factSegmentPaths()` returns the immutable, *sealed* segment files for zero-copy consumers
-  (the append-mutable tail is excluded — read it through `scanFacts()`).
-
-## Family stamps: how a projection proves it's current
-
-Anything derived from the store — an index, the entity file tree itself — carries a **family
-stamp**: a small JSON record of *which committed generation the projection reflects*
-(`sourceGeneration`) plus the invariants that verify it whole (exact per-file byte sizes for
-bounded families; rollup invariants like entity counts for unbounded ones). At open, coherence is
-a **comparison**, not a walk:
-
-- stamp equals the committed watermark and invariants hold → serve;
-- stamp is behind → the projection reads just the gap from the fact log;
-- invariants fail → loud, named divergence — `brain.repairIndex()` rebuilds from canonical and
-  re-stamps.
-
-The verifier is exported (`verifyFamilyStamp`) so every projection — TypeScript or native — runs
-literally the same check.
-
-## For plugin authors: the storage capability
-
-Index providers receive the storage adapter, not the brain — so the host wires the log onto it.
-Feature-detect and prefer the stream; fall back to enumeration:
-
-```typescript
-const scan = storage.scanFacts?.({ fromGeneration: stamp.sourceGeneration + 1 })
-if (scan) {
-  // sequential catch-up from the log
-} else {
-  // enumeration walk (older store or adapter)
-}
-const committed = storage.committedGeneration?.() // the watermark stamps compare against
-```
-
-Providers must never construct their own reader over the log's files — the open path belongs to
-the single writer (it reconciles the log at open); the capability is the sanctioned seam.

docs/concepts/multi-process.md

@@ -1,153 +0,0 @@
----
-title: Multi-Process Model
-slug: concepts/multi-process
-public: true
-category: concepts
-template: concept
-order: 5
-description: How Brainy coordinates a single writer with any number of readers on a filesystem data directory — and how to safely inspect a live store.
-next:
-  - guides/inspection
----
-
-# Multi-Process Model
-
-Brainy is a **single-writer, many-reader** database when backed by filesystem
-storage. This page explains the model, the guarantees, and the safe ways to
-inspect a live store from a second process.
-
-## The rule
-
-For one data directory:
-
-- **One writer** at a time. The writer acquires an exclusive lock on the
-  directory at `init()` and releases it on `close()`.
-- **Any number of readers**, concurrent with each other and with the writer.
-  Readers open via `Brainy.openReadOnly()` — they never touch the writer
-  lock.
-
-Any attempt to open a second writer on the same directory throws:
-
-```
-BrainyError: Another writer holds this Brainy directory.
-  PID:        1774431 on host app-host-1
-  Started:    2026-05-15T14:22:11Z
-  Heartbeat:  2026-05-15T14:22:34Z
-  Version:    7.21.0
-  Directory:  /data/brain
-```
-
-This is intentional. Two writers sharing a directory would silently corrupt
-in-memory indexes and produce wrong query results — the worst possible default
-for an operations tool.
-
-## Why a lock?
-
-Brainy keeps its primary indexes (HNSW, metadata, graph adjacency) in memory.
-On disk, those indexes are persisted incrementally as writes flush. A second
-process opening the same directory:
-
-- Loads the *persisted* state into a fresh in-memory copy.
-- Has no awareness of writes the first process buffered but hasn't flushed.
-- Will overwrite the persisted state on its own next flush, racing the first
-  process and corrupting whichever wins.
-
-The fix is the lock: refuse to open a second writer. SQLite has done the same
-since the late 1990s (`SQLITE_BUSY`).
-
-## What about Cor?
-
-Brainy + Cor compose cleanly under this model:
-
-- Cor stores its column-index segments inside the same `rootDir` (under
-  `indexes/_column_index/{field}/`).
-- Segments (`*.cidx` files) are **immutable** once written. Cor mmaps them
-  read-only.
-- The `MANIFEST.json` per field is updated via atomic rename — readers see
-  either the old or new manifest, never a torn file.
-
-A reader process can safely mmap Cor segments alongside a live writer
-without coordination. The single Brainy writer lock at
-`<rootDir>/locks/_writer.lock` covers Cor too, because Cor segment
-writes happen on the writer's side.
-
-## Stale-lock detection
-
-If a writer crashes or is forcibly killed, its lock file is left behind. To
-avoid a permanently-jammed directory, Brainy treats a lock as stale when:
-
-1. The recorded `hostname` equals the current host (cross-host PID checks
-   are unsafe), AND
-2. The recorded `pid` is no longer alive (`process.kill(pid, 0)` returns
-   `ESRCH`), OR the `lastHeartbeat` field is older than 60 seconds.
-
-A live writer rewrites `lastHeartbeat` every 10 seconds, so a hung writer
-that's missed several heartbeats is treated as dead. Stale locks are
-overwritten with a warning.
-
-If stale detection cannot prove the existing lock is dead — for example, a
-crashed writer on a different host writing to a shared filesystem — pass
-`{ force: true }` to override. A warning is logged either way.
-
-## Heartbeat and shutdown
-
-The heartbeat interval rewrites the lock file every 10 seconds. The timer
-is unref'd, so it does not keep the event loop alive on its own.
-
-On normal shutdown the writer releases the lock in `close()`. The shutdown
-hooks Brainy registers for `SIGTERM`, `SIGINT`, and `beforeExit` also
-release the lock so a container restart doesn't strand the directory.
-
-## How to inspect a live writer
-
-Use `Brainy.openReadOnly()`. It does not acquire the writer lock, so it
-coexists with whatever the writer is doing:
-
-```typescript
-const reader = await Brainy.openReadOnly({
-  storage: { type: 'filesystem', path: '/data/brain' }
-})
-
-const stats = await reader.stats()
-const bookings = await reader.find({ where: { entityType: 'booking' } })
-
-await reader.close()
-```
-
-What the reader sees reflects the writer's most recent **flush** to disk. If
-you need fresher state, ask the writer to flush before opening:
-
-```typescript
-const reader = await Brainy.openReadOnly({
-  storage: { type: 'filesystem', path: '/data/brain' }
-})
-
-const acked = await reader.requestFlush({ timeoutMs: 5000 })
-if (!acked) {
-  console.warn('Writer did not respond; results reflect last natural flush.')
-}
-
-const fresh = await reader.find({ where: { entityType: 'booking' } })
-```
-
-The CLI `brainy inspect` subcommands all do this for you by default
-(`--no-fresh` to opt out).
-
-## What's not enforced (yet)
-
-- **Non-filesystem backends** are out of scope in 8.0, which ships only the
-  filesystem and memory adapters. A custom `BaseStorage` subclass that is not
-  filesystem-backed does not enforce multi-process locking by default: two
-  processes can both succeed at `init()` in writer mode and clobber each
-  other's writes. A best-effort warning is logged in writer mode against a
-  non-filesystem backend.
-- **Long-running readers** do not automatically pick up new Cor segments
-  the writer publishes. One-shot inspector calls re-open the store and see
-  fresh segments; a reader that stays open for hours sees its column store
-  as-of the time it opened.
-
-## Reading material
-
-- `Brainy.openReadOnly()` — [API reference](../api/brainy.md)
-- `brainy inspect` — [inspection guide](../guides/inspection.md)
-- Cor columnar storage — see `node_modules/@soulcraft/cor/README.md`

docs/concepts/storage-adapters.md

@@ -1,189 +0,0 @@
----
-title: Storage Adapter Inheritance Contract
-slug: concepts/storage-adapters
-public: true
-category: concepts
-template: concept
-order: 6
-description: How storage adapters extend Brainy's BaseStorage and FileSystemStorage to inherit multi-process safety, what plugin authors need to override, and the contract Brainy promises to keep stable.
-next:
-  - concepts/multi-process
-  - guides/inspection
----
-
-# Storage Adapter Inheritance Contract
-
-Brainy's storage layer is designed for plugins to extend cleanly. A plugin
-that subclasses `BaseStorage` or `FileSystemStorage` inherits new behavior
-Brainy adds over time without code changes — provided the import resolution
-brings in the right Brainy version at runtime.
-
-This page documents what plugin authors can rely on, what they need to
-override, and the install-time failure modes that the defensive
-`hasStorageMethod()` guard exists to handle.
-
-## The class hierarchy
-
-```
-BaseStorageAdapter         (counts, batch ops, multi-tenancy hooks)
-        ↑
-BaseStorage                (type-statistics, lifecycle helpers, generation
-                            hooks, default no-op multi-process methods)
-        ↑
-FileSystemStorage          (real filesystem I/O, writer-lock implementation,
-                            flush-request watcher, atomic writes)
-        ↑
-<your plugin's storage>    (Cor's MmapFileSystemStorage, etc.)
-```
-
-When you `extend FileSystemStorage`, your adapter inherits every method on
-the chain — including the ones Brainy adds in a later release — for free.
-JavaScript's prototype chain resolves method lookups dynamically; nothing
-about the inheritance is baked in at class-definition time.
-
-## What you inherit for free
-
-A plugin whose storage class extends `FileSystemStorage` automatically gets
-the full multi-process safety surface:
-
-| Method | What it does | Override? |
-|---|---|---|
-| `acquireWriterLock(opts)` | Write `_writer.lock`, start heartbeat, throw on conflict | No |
-| `releaseWriterLock()` | Clean up lock file + heartbeat timer | No |
-| `readWriterLock()` | Read lock file as `WriterLockInfo` | No |
-| `startFlushRequestWatcher(cb)` | Poll `_flush_requests/` and invoke `cb` | No |
-| `stopFlushRequestWatcher()` | Stop the polling timer | No |
-| `requestFlushOverFilesystem(timeoutMs)` | Drop a `.req`, await `.ack` | No |
-| `supportsMultiProcessLocking()` | Return `false` (default) | **Yes — override to `true`** |
-
-The only required override is the capability flag. Returning `true` from
-`supportsMultiProcessLocking()` is the signal Brainy uses to decide whether
-to call `acquireWriterLock()` at init.
-
-```typescript
-import { FileSystemStorage } from '@soulcraft/brainy'
-
-export class MmapFileSystemStorage extends FileSystemStorage {
-  public supportsMultiProcessLocking(): boolean {
-    return true
-  }
-  // ... your mmap-specific overrides ...
-}
-```
-
-That's the full ceremony for inheriting multi-process safety.
-
-## When NOT to extend FileSystemStorage
-
-If your storage is **not filesystem-backed** (a custom
-network backend), extend `BaseStorage` directly:
-
-```typescript
-import { BaseStorage } from '@soulcraft/brainy'
-
-export class MyCloudStorage extends BaseStorage {
-  // BaseStorage's default no-op implementations of the multi-process
-  // methods stay in effect. `supportsMultiProcessLocking()` returns false
-  // by default — keep it that way unless you've implemented an object-
-  // versioned lease or similar cross-process synchronization for your
-  // backend.
-}
-```
-
-Brainy treats cloud backends as not-multi-process-safe by default and logs a
-one-line warning at init. That's the correct behavior until cloud locking
-ships (currently out of scope — see
-[`concepts/multi-process`](./multi-process.md)).
-
-## What `hasStorageMethod()` actually guards against
-
-The defensive check at every new-storage-method call site (`brainy.ts`,
-`hasStorageMethod(name)`) does **not** exist to handle "plugin bundles a
-stale BaseStorage." Plugins ship a dist that preserves the dynamic ESM
-import (verify in your plugin's `dist/`: `import { FileSystemStorage } from
-'@soulcraft/brainy'` is not rewritten to a vendored copy). The prototype
-chain at runtime resolves to whatever Brainy version your consumer has
-installed.
-
-`hasStorageMethod()` protects against **build/install artifacts** that break
-the prototype chain at the consumer-app level:
-
-- **Stale `node_modules`** — a lingering install from before the consumer
-  upgraded Brainy. The package.json says `@soulcraft/brainy@7.22.0` but
-  `node_modules/@soulcraft/brainy` is still 7.20.x.
-- **Lockfile drift** — `bun.lockb` / `package-lock.json` pins a brainy
-  version older than the package.json range, and `bun install` honors the
-  lockfile.
-- **Docker layer cache** — the image reuses a `node_modules` from an
-  earlier build that predates the brainy bump.
-- **Bundler quirks** — some bundlers (esbuild, webpack) flatten the
-  prototype chain at build time and lose later prototype mutations. Brainy
-  doesn't mutate prototypes at runtime, but bundler behavior can still
-  cause method lookups to fail in non-Node environments.
-
-In any of those, calling `storage.acquireWriterLock(...)` unconditionally
-throws `TypeError: storage.acquireWriterLock is not a function`. The guard
-turns that into a logged warning + graceful no-op so the app still boots,
-and the warning names the adapter class plus a remediation hint:
-
-```
-[brainy] Storage adapter `MmapFileSystemStorage` is missing the multi-process
-methods on its prototype chain. Writer locking and the flush-request RPC are
-disabled for this directory. Likely fix: clean install (`rm -rf node_modules
-bun.lockb && bun install`) or rebuild your container image to refresh
-`@soulcraft/brainy` to ≥7.21. See docs/concepts/storage-adapters.md.
-```
-
-## Authoring a new storage adapter — minimum checklist
-
-1. **Extend the right base class.**
-   - Filesystem-backed → `FileSystemStorage`.
-   - Cloud / network / custom → `BaseStorage`.
-
-2. **Override the capability flag.** If filesystem-backed:
-   ```typescript
-   public supportsMultiProcessLocking(): boolean { return true }
-   ```
-
-3. **Don't `super.X()`-wrap the multi-process methods**. They're inherited;
-   leaving them inherited means `hasStorageMethod()` finds them on the
-   prototype chain. Re-declaring them as `super.X()` wrappers makes the
-   helper resolve to your wrapper, which can fool the guard if your
-   constructor runs before the super class initializes.
-
-4. **Do override `init()` / `flush()` / `close()`** as needed. Always call
-   `super.init()` / `super.flush()` / `super.close()` first so the
-   filesystem prep, writer-lock acquisition, and lock release happen in the
-   expected order.
-
-5. **Verify the inheritance.** A one-line smoke test in your plugin's
-   `__tests__/`:
-   ```typescript
-   const s = new MyStorage(rootDir)
-   await s.init()
-   assert(typeof s.acquireWriterLock === 'function')
-   assert(s.supportsMultiProcessLocking() === true)
-   ```
-   If `acquireWriterLock` is undefined the prototype chain is broken at
-   install time — fix install, not your plugin.
-
-6. **Pin your peer dep generously.** `"peerDependencies": {
-   "@soulcraft/brainy": "^7.21.0" }` accepts any compatible 7.x. Don't pin
-   to an exact patch unless you're tracking a known regression.
-
-## Future direction
-
-The 7 multi-process methods are currently defaults on `BaseStorage`. A
-future refactor may extract them into a `MultiProcessSafeStorage`
-interface/mixin for cleaner separation — only adapters that opt in would
-expose them. This would require a minor bump and is tracked as an internal
-follow-up; consumers don't need to anticipate the change.
-
-## Reading material
-
-- [`concepts/multi-process`](./multi-process.md) — the writer-lock model,
-  heartbeat semantics, what the lock protects.
-- [`guides/inspection`](../guides/inspection.md) — `brainy inspect` and the
-  read-only mode.
-- `node_modules/@soulcraft/brainy/dist/storage/baseStorage.d.ts` — the
-  authoritative type signatures for every method this page references.

docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md

@@ -0,0 +1,756 @@
+# Brainy 3.0 Cloud Deployment Guide
+
+This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy 3.0 codebase.
+
+## Overview
+
+The API Server augmentation provides a universal handler that works with standard Request/Response objects, making Brainy deployable on any JavaScript runtime.
+
+## Storage Adapter
+
+**S3CompatibleStorage** is the preferred storage adapter for cloud deployments. It works with:
+- Amazon S3
+- Cloudflare R2
+- Google Cloud Storage
+- Azure Blob Storage
+- Any S3-compatible service
+
+## Deployment Examples
+
+### AWS Lambda
+
+```javascript
+// handler.js
+const { Brainy } = require('@soulcraft/brainy')
+const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
+
+let brain
+let handler
+
+exports.handler = async (event) => {
+  if (!brain) {
+    const storage = new S3CompatibleStorage({
+      endpoint: 's3.amazonaws.com',
+      region: process.env.AWS_REGION,
+      bucket: process.env.BRAINY_BUCKET,
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
+      prefix: 'brainy-data/'
+    })
+    
+    brain = new Brainy({
+      storage,
+      augmentations: [{
+        name: 'api-server',
+        config: {
+          enabled: true,
+          auth: {
+            required: true,
+            apiKeys: [process.env.API_KEY]
+          }
+        }
+      }]
+    })
+    
+    await brain.init()
+    
+    // Get the universal handler from the augmentation
+    const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
+    handler = apiAugmentation.createUniversalHandler()
+  }
+  
+  // Convert Lambda event to Request
+  const url = `https://${event.requestContext.domainName}${event.rawPath}`
+  const request = new Request(url, {
+    method: event.requestContext.http.method,
+    headers: event.headers,
+    body: event.body
+  })
+  
+  // Use the universal handler
+  const response = await handler(request)
+  
+  return {
+    statusCode: response.status,
+    headers: Object.fromEntries(response.headers),
+    body: await response.text()
+  }
+}
+```
+
+### Google Cloud Functions
+
+```javascript
+// index.js
+const { Brainy } = require('@soulcraft/brainy')
+const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
+
+let brain
+let handler
+
+exports.brainyAPI = async (req, res) => {
+  if (!brain) {
+    const storage = new S3CompatibleStorage({
+      endpoint: 'storage.googleapis.com',
+      bucket: process.env.GCS_BUCKET,
+      accessKeyId: process.env.GCS_ACCESS_KEY,
+      secretAccessKey: process.env.GCS_SECRET_KEY,
+      prefix: 'brainy/',
+      forcePathStyle: false,
+      region: 'US'
+    })
+    
+    brain = new Brainy({ storage })
+    await brain.init()
+    
+    const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
+    handler = apiAugmentation.createUniversalHandler()
+  }
+  
+  // Convert Express req/res to Request/Response
+  const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
+    method: req.method,
+    headers: req.headers,
+    body: JSON.stringify(req.body)
+  })
+  
+  const response = await handler(request)
+  
+  res.status(response.status)
+  res.set(Object.fromEntries(response.headers))
+  res.send(await response.text())
+}
+```
+
+### Google Cloud Run with Cloud Storage
+
+Google Cloud Run is ideal for containerized deployments with automatic scaling. This example uses Google Cloud Storage via the S3-compatible API.
+
+```javascript
+// server.js
+import { Brainy } from '@soulcraft/brainy'
+import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
+import express from 'express'
+
+const app = express()
+app.use(express.json())
+
+const PORT = process.env.PORT || 8080
+
+let brain
+let handler
+
+async function initBrainy() {
+  // Google Cloud Storage is S3-compatible
+  const storage = new S3CompatibleStorage({
+    endpoint: 'storage.googleapis.com',
+    bucket: process.env.GCS_BUCKET || 'brainy-data',
+    accessKeyId: process.env.GCS_ACCESS_KEY,
+    secretAccessKey: process.env.GCS_SECRET_KEY,
+    prefix: 'brainy/',
+    forcePathStyle: false,
+    region: process.env.GCS_REGION || 'US'
+  })
+  
+  brain = new Brainy({
+    storage,
+    augmentations: [{
+      name: 'api-server',
+      config: {
+        enabled: true,
+        port: PORT,
+        cors: {
+          origin: process.env.CORS_ORIGIN || '*'
+        },
+        auth: {
+          required: process.env.AUTH_REQUIRED === 'true',
+          apiKeys: process.env.API_KEY ? [process.env.API_KEY] : []
+        }
+      }
+    }]
+  })
+  
+  await brain.init()
+  
+  const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
+  handler = apiAugmentation.createUniversalHandler()
+}
+
+// Initialize on startup
+await initBrainy()
+
+// Health check endpoint
+app.get('/health', (req, res) => {
+  res.json({ status: 'healthy', service: 'brainy-api' })
+})
+
+// Universal handler for all API routes
+app.use('*', async (req, res) => {
+  const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
+    method: req.method,
+    headers: req.headers,
+    body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined
+  })
+  
+  const response = await handler(request)
+  
+  res.status(response.status)
+  res.set(Object.fromEntries(response.headers))
+  res.send(await response.text())
+})
+
+app.listen(PORT, () => {
+  console.log(`Brainy API running on port ${PORT}`)
+})
+```
+
+```dockerfile
+# Dockerfile
+FROM node:20-alpine AS builder
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci --only=production
+
+FROM node:20-alpine
+WORKDIR /app
+COPY --from=builder /app/node_modules ./node_modules
+COPY . .
+EXPOSE 8080
+CMD ["node", "server.js"]
+```
+
+```yaml
+# cloudbuild.yaml
+steps:
+  # Build the container image
+  - name: 'gcr.io/cloud-builders/docker'
+    args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.']
+  
+  # Push to Container Registry
+  - name: 'gcr.io/cloud-builders/docker'
+    args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA']
+  
+  # Deploy to Cloud Run
+  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
+    entrypoint: gcloud
+    args:
+      - 'run'
+      - 'deploy'
+      - 'brainy-api'
+      - '--image'
+      - 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
+      - '--region'
+      - 'us-central1'
+      - '--platform'
+      - 'managed'
+      - '--allow-unauthenticated'
+      - '--set-env-vars'
+      - 'GCS_BUCKET=brainy-storage,GCS_REGION=US'
+      - '--set-secrets'
+      - 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest'
+      - '--memory'
+      - '2Gi'
+      - '--cpu'
+      - '2'
+      - '--max-instances'
+      - '100'
+      - '--min-instances'
+      - '0'
+
+images:
+  - 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
+```
+
+**Deploy with gcloud CLI:**
+```bash
+# Build and submit to Cloud Build
+gcloud builds submit --config cloudbuild.yaml
+
+# Or deploy directly
+gcloud run deploy brainy-api \
+  --source . \
+  --platform managed \
+  --region us-central1 \
+  --allow-unauthenticated \
+  --set-env-vars GCS_BUCKET=brainy-storage \
+  --set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest"
+```
+
+**Create GCS Bucket with S3-compatible access:**
+```bash
+# Create bucket
+gsutil mb -p PROJECT_ID -c STANDARD -l US gs://brainy-storage/
+
+# Enable interoperability
+gsutil iam ch serviceAccount:SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com:objectAdmin gs://brainy-storage
+
+# Generate HMAC keys for S3-compatible access
+gsutil hmac create SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com
+
+# Store the access key and secret in Secret Manager
+echo -n "YOUR_ACCESS_KEY" | gcloud secrets create gcs-access-key --data-file=-
+echo -n "YOUR_SECRET_KEY" | gcloud secrets create gcs-secret-key --data-file=-
+```
+
+### Microsoft Azure Functions
+
+```javascript
+// index.js
+module.exports = async function (context, req) {
+  const { Brainy } = require('@soulcraft/brainy')
+  const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
+  
+  const storage = new S3CompatibleStorage({
+    endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`,
+    bucket: 'brainy-data',
+    accessKeyId: process.env.AZURE_STORAGE_ACCOUNT,
+    secretAccessKey: process.env.AZURE_STORAGE_KEY,
+    prefix: 'entities/',
+    forcePathStyle: false
+  })
+  
+  const brain = new Brainy({ storage })
+  await brain.init()
+  
+  const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
+  const handler = apiAugmentation.createUniversalHandler()
+  
+  const request = new Request(`https://${context.req.headers.host}${context.req.url}`, {
+    method: context.req.method,
+    headers: context.req.headers,
+    body: JSON.stringify(context.req.body)
+  })
+  
+  const response = await handler(request)
+  
+  context.res = {
+    status: response.status,
+    headers: Object.fromEntries(response.headers),
+    body: await response.text()
+  }
+}
+```
+
+### Cloudflare Workers
+
+```javascript
+// worker.js
+import { Brainy } from '@soulcraft/brainy'
+import { R2Storage } from '@soulcraft/brainy/storage' // Alias for S3CompatibleStorage
+
+let handler
+
+export default {
+  async fetch(request, env, ctx) {
+    if (!handler) {
+      const storage = new R2Storage({
+        endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`,
+        bucket: 'brainy-data',
+        accessKeyId: env.R2_ACCESS_KEY_ID,
+        secretAccessKey: env.R2_SECRET_ACCESS_KEY,
+        region: 'auto',
+        forcePathStyle: true
+      })
+      
+      const brain = new Brainy({
+        storage,
+        augmentations: [{
+          name: 'api-server',
+          config: {
+            enabled: true,
+            cors: { origin: '*' }
+          }
+        }]
+      })
+      
+      await brain.init()
+      
+      const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
+      handler = apiAugmentation.createUniversalHandler()
+    }
+    
+    // The handler works directly with Request/Response!
+    return handler(request)
+  }
+}
+```
+
+```toml
+# wrangler.toml
+name = "brainy-api"
+main = "worker.js"
+compatibility_date = "2024-01-01"
+
+[[r2_buckets]]
+binding = "R2"
+bucket_name = "brainy-data"
+
+[vars]
+ACCOUNT_ID = "your-account-id"
+
+[env.production.vars]
+R2_ACCESS_KEY_ID = "your-access-key"
+R2_SECRET_ACCESS_KEY = "your-secret-key"
+```
+
+### Vercel Edge Functions
+
+```javascript
+// api/brainy.js
+import { Brainy } from '@soulcraft/brainy'
+import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
+
+let handler
+
+export const config = {
+  runtime: 'edge',
+}
+
+export default async (request) => {
+  if (!handler) {
+    const storage = new S3CompatibleStorage({
+      endpoint: 's3.amazonaws.com',
+      region: 'us-east-1',
+      bucket: process.env.S3_BUCKET,
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
+    })
+    
+    const brain = new Brainy({
+      storage,
+      augmentations: [{
+        name: 'api-server',
+        config: { enabled: true }
+      }]
+    })
+    
+    await brain.init()
+    
+    const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
+    handler = apiAugmentation.createUniversalHandler()
+  }
+  
+  return handler(request)
+}
+```
+
+```json
+// vercel.json
+{
+  "functions": {
+    "api/brainy.js": {
+      "maxDuration": 30,
+      "memory": 1024
+    }
+  },
+  "rewrites": [
+    {
+      "source": "/api/:path*",
+      "destination": "/api/brainy"
+    }
+  ]
+}
+```
+
+### Railway
+
+```javascript
+// server.js
+import { Brainy } from '@soulcraft/brainy'
+import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
+import express from 'express'
+
+const app = express()
+const PORT = process.env.PORT || 3000
+
+let brain
+let handler
+
+async function init() {
+  const storage = new S3CompatibleStorage({
+    endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com',
+    region: process.env.S3_REGION || 'us-east-1',
+    bucket: process.env.S3_BUCKET,
+    accessKeyId: process.env.S3_ACCESS_KEY,
+    secretAccessKey: process.env.S3_SECRET_KEY,
+    prefix: 'brainy/'
+  })
+  
+  brain = new Brainy({
+    storage,
+    augmentations: [{
+      name: 'api-server',
+      config: {
+        enabled: true,
+        port: PORT
+      }
+    }]
+  })
+  
+  await brain.init()
+  
+  const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
+  handler = apiAugmentation.createUniversalHandler()
+}
+
+await init()
+
+// Universal handler for all routes
+app.use('*', async (req, res) => {
+  const request = new Request(`http://localhost${req.originalUrl}`, {
+    method: req.method,
+    headers: req.headers,
+    body: JSON.stringify(req.body)
+  })
+  
+  const response = await handler(request)
+  
+  res.status(response.status)
+  res.set(Object.fromEntries(response.headers))
+  res.send(await response.text())
+})
+
+app.listen(PORT, () => {
+  console.log(`Brainy API running on port ${PORT}`)
+})
+```
+
+```toml
+# railway.toml
+[build]
+builder = "nixpacks"
+buildCommand = "npm ci"
+
+[deploy]
+startCommand = "node server.js"
+restartPolicyType = "always"
+restartPolicyMaxRetries = 3
+```
+
+### Render
+
+```javascript
+// server.js (same as Railway example above)
+// Use S3CompatibleStorage with your preferred object storage provider
+```
+
+```yaml
+# render.yaml
+services:
+  - type: web
+    name: brainy-api
+    runtime: node
+    buildCommand: npm install
+    startCommand: node server.js
+    envVars:
+      - key: S3_BUCKET
+        value: brainy-data
+      - key: S3_ENDPOINT
+        value: s3.amazonaws.com
+      - key: S3_REGION
+        value: us-east-1
+      - key: S3_ACCESS_KEY
+        sync: false
+      - key: S3_SECRET_KEY
+        sync: false
+      - key: API_KEY
+        generateValue: true
+    healthCheckPath: /health
+    autoDeploy: true
+```
+
+### Deno Deploy
+
+```typescript
+// main.ts
+import { Brainy } from "npm:@soulcraft/brainy"
+import { S3CompatibleStorage } from "npm:@soulcraft/brainy/storage"
+
+const storage = new S3CompatibleStorage({
+  endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com",
+  bucket: Deno.env.get("S3_BUCKET")!,
+  accessKeyId: Deno.env.get("S3_ACCESS_KEY")!,
+  secretAccessKey: Deno.env.get("S3_SECRET_KEY")!,
+  region: "auto"
+})
+
+const brain = new Brainy({
+  storage,
+  augmentations: [{
+    name: 'api-server',
+    config: { enabled: true }
+  }]
+})
+
+await brain.init()
+
+const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
+const handler = apiAugmentation.createUniversalHandler()
+
+Deno.serve(handler)
+```
+
+## API Endpoints
+
+The API Server augmentation provides these REST endpoints:
+
+- `POST /api/brainy/add` - Add entity
+- `GET /api/brainy/get?id=xxx` - Get entity by ID
+- `PUT /api/brainy/update` - Update entity
+- `DELETE /api/brainy/delete?id=xxx` - Delete entity
+- `POST /api/brainy/find` - Search/find entities
+- `POST /api/brainy/relate` - Create relationship
+- `GET /api/brainy/insights` - Get statistics and insights
+- `GET /health` - Health check
+
+## WebSocket Support
+
+The API Server augmentation includes WebSocket support for real-time updates through the `setupUniversalWebSocket()` method.
+
+## MCP Support
+
+Model Context Protocol (MCP) endpoints are available at `/mcp/*` for AI tool integration.
+
+## Environment Variables
+
+```bash
+# Storage Configuration (S3Compatible)
+S3_ENDPOINT=s3.amazonaws.com
+S3_REGION=us-east-1
+S3_BUCKET=brainy-data
+S3_ACCESS_KEY=xxx
+S3_SECRET_KEY=xxx
+
+# API Configuration
+API_KEY=your-secret-key
+PORT=3000
+
+# CORS
+CORS_ORIGIN=*
+
+# Rate Limiting
+RATE_LIMIT_WINDOW=60000
+RATE_LIMIT_MAX=100
+```
+
+## Client Usage
+
+```javascript
+// REST API Client
+const response = await fetch('https://your-api.com/api/brainy/add', {
+  method: 'POST',
+  headers: {
+    'Content-Type': 'application/json',
+    'Authorization': 'Bearer YOUR_API_KEY'
+  },
+  body: JSON.stringify({
+    data: 'Your content here',
+    metadata: { type: 'document' }
+  })
+})
+
+const { id } = await response.json()
+
+// Search
+const searchResponse = await fetch('https://your-api.com/api/brainy/find', {
+  method: 'POST',
+  headers: {
+    'Content-Type': 'application/json',
+    'Authorization': 'Bearer YOUR_API_KEY'
+  },
+  body: JSON.stringify({
+    query: 'neural networks'
+  })
+})
+
+const results = await searchResponse.json()
+
+// WebSocket Client
+const ws = new WebSocket('wss://your-api.com/ws')
+ws.onopen = () => {
+  ws.send(JSON.stringify({
+    type: 'subscribe',
+    pattern: 'technology'
+  }))
+}
+ws.onmessage = (event) => {
+  const update = JSON.parse(event.data)
+  console.log('Real-time update:', update)
+}
+```
+
+## Storage Adapter Configuration
+
+S3CompatibleStorage constructor parameters (verified from source):
+
+```javascript
+{
+  endpoint: string,         // Required (e.g., 's3.amazonaws.com')
+  bucket: string,          // Required
+  accessKeyId: string,     // Required
+  secretAccessKey: string, // Required
+  region?: string,         // Optional (default: 'us-east-1')
+  prefix?: string,         // Optional (e.g., 'brainy/')
+  forcePathStyle?: boolean, // Optional (needed for some S3-compatible services)
+}
+```
+
+## Important Notes
+
+1. All examples use the **real** Brainy 3.0 APIs
+2. The `createUniversalHandler()` method is provided by the API Server augmentation
+3. S3CompatibleStorage works with any S3-compatible service
+4. Always call `brain.init()` before using Brainy
+5. The handler can be cached across requests for better performance
+6. R2Storage is an alias for S3CompatibleStorage (for Cloudflare R2)
+
+## Security Best Practices
+
+1. **Always use environment variables for sensitive data** (API keys, secrets)
+2. **Enable authentication** in the API Server augmentation config
+3. **Use HTTPS/TLS** for all production deployments
+4. **Implement rate limiting** to prevent abuse
+5. **Configure CORS** appropriately for your use case
+
+## Performance Tips
+
+1. **Cache the brain instance** - Initialize once and reuse across requests
+2. **Use S3CompatibleStorage** for cloud deployments (better scalability)
+3. **Enable the cache augmentation** for frequently accessed data
+5. **Configure appropriate memory limits** for your runtime
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"Brainy not initialized"** - Make sure to call `await brain.init()` before use
+2. **"augmentationRegistry.getAugmentation is not a function"** - The augmentation wasn't loaded properly
+3. **S3 Access Denied** - Check your IAM permissions and credentials
+4. **CORS errors** - Configure the CORS settings in the API Server augmentation
+
+### Debug Mode
+
+Enable debug logging by setting:
+```javascript
+const brain = new Brainy({
+  storage,
+  debug: true,
+  augmentations: [{
+    name: 'api-server',
+    config: {
+      enabled: true,
+      verbose: true
+    }
+  }]
+})
+```
+
+## Support
+
+- Documentation: https://github.com/soulcraft/brainy/docs
+- Issues: https://github.com/soulcraft/brainy/issues
+- NPM Package: https://www.npmjs.com/package/@soulcraft/brainy
+
+---
+
+This guide contains only verified, working code from the actual Brainy 3.0 codebase. No mock implementations, no stub methods, only production-ready code.

docs/deployment/aws-deployment.md

@@ -0,0 +1,416 @@
+# AWS Deployment Guide for Brainy
+
+## Overview
+Deploy Brainy on AWS with automatic scaling, high availability, and zero-config dynamic adaptation. Brainy automatically adapts to your AWS environment without manual configuration.
+
+## Quick Start (Zero-Config)
+
+### Option 1: AWS Lambda (Serverless)
+
+```bash
+# Install Brainy
+npm install @soulcraft/brainy
+
+# Create handler.js
+cat > handler.js << 'EOF'
+const { Brainy } = require('@soulcraft/brainy')
+
+let brain
+
+exports.handler = async (event) => {
+  // Brainy auto-detects Lambda environment and configures accordingly
+  if (!brain) {
+    brain = new Brainy() // Zero config - auto-adapts to Lambda
+    await brain.init()
+  }
+  
+  const { method, ...params } = JSON.parse(event.body)
+  
+  switch(method) {
+    case 'add':
+      const id = await brain.add(params)
+      return { statusCode: 200, body: JSON.stringify({ id }) }
+    case 'find':
+      const results = await brain.find(params)
+      return { statusCode: 200, body: JSON.stringify({ results }) }
+    default:
+      return { statusCode: 400, body: 'Unknown method' }
+  }
+}
+EOF
+
+# Deploy with AWS SAM
+sam init --runtime nodejs20.x --name brainy-app
+sam deploy --guided
+```
+
+### Option 2: ECS Fargate (Container)
+
+```bash
+# Build and push Docker image
+aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ECR_URI
+docker build -t brainy .
+docker tag brainy:latest $ECR_URI/brainy:latest
+docker push $ECR_URI/brainy:latest
+
+# Deploy with minimal ECS task definition
+cat > task-definition.json << 'EOF'
+{
+  "family": "brainy",
+  "networkMode": "awsvpc",
+  "requiresCompatibilities": ["FARGATE"],
+  "cpu": "256",
+  "memory": "512",
+  "containerDefinitions": [{
+    "name": "brainy",
+    "image": "$ECR_URI/brainy:latest",
+    "environment": [
+      {"name": "NODE_ENV", "value": "production"}
+    ],
+    "logConfiguration": {
+      "logDriver": "awslogs",
+      "options": {
+        "awslogs-group": "/ecs/brainy",
+        "awslogs-region": "us-east-1",
+        "awslogs-stream-prefix": "ecs"
+      }
+    }
+  }]
+}
+EOF
+
+# Brainy auto-detects ECS environment and uses S3 for storage
+aws ecs register-task-definition --cli-input-json file://task-definition.json
+aws ecs create-service --cluster default --service-name brainy --task-definition brainy --desired-count 2
+```
+
+### Option 3: EC2 Auto-Scaling
+
+```bash
+# User data script for EC2 instances
+#!/bin/bash
+curl -fsSL https://rpm.nodesource.com/setup_20.x | sudo bash -
+sudo yum install -y nodejs git
+
+# Clone and setup (or use your deployment method)
+git clone https://github.com/yourorg/brainy-app.git /app
+cd /app
+npm install --production
+
+# Create systemd service
+cat > /etc/systemd/system/brainy.service << 'EOF'
+[Unit]
+Description=Brainy
+After=network.target
+
+[Service]
+Type=simple
+User=ec2-user
+WorkingDirectory=/app
+ExecStart=/usr/bin/node index.js
+Restart=on-failure
+Environment=NODE_ENV=production
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+systemctl start brainy
+systemctl enable brainy
+```
+
+## Zero-Config Storage (Automatic)
+
+Brainy automatically detects and uses the best available storage:
+
+```javascript
+// No configuration needed - Brainy auto-detects:
+const brain = new Brainy()
+
+// Auto-detection priority:
+// 1. S3 (if IAM role has permissions)
+// 2. EFS (if mounted at /mnt/efs)
+// 3. EBS volume (if available)
+// 4. Instance storage (fallback)
+```
+
+### Manual S3 Configuration (Optional)
+
+```javascript
+const brain = new Brainy({
+  storage: {
+    type: 's3',
+    options: {
+      bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket
+      region: process.env.AWS_REGION || 'auto', // 'auto' detects region
+      // IAM role provides credentials automatically
+    }
+  }
+})
+```
+
+## Scaling Strategies
+
+### 1. Horizontal Scaling (Recommended)
+
+```yaml
+# Auto-scaling policy
+Resources:
+  AutoScalingTarget:
+    Type: AWS::ApplicationAutoScaling::ScalableTarget
+    Properties:
+      ServiceNamespace: ecs
+      ResourceId: service/default/brainy
+      ScalableDimension: ecs:service:DesiredCount
+      MinCapacity: 2
+      MaxCapacity: 100
+      
+  AutoScalingPolicy:
+    Type: AWS::ApplicationAutoScaling::ScalingPolicy
+    Properties:
+      PolicyType: TargetTrackingScaling
+      TargetTrackingScalingPolicyConfiguration:
+        TargetValue: 70.0
+        PredefinedMetricSpecification:
+          PredefinedMetricType: ECSServiceAverageCPUUtilization
+```
+
+### 2. Vertical Scaling
+
+Brainy automatically adapts to available memory:
+- **256MB**: Minimal mode, optimized caching
+- **512MB**: Standard mode, balanced performance
+- **1GB+**: Full mode, maximum performance
+
+## High Availability Setup
+
+### Multi-AZ Deployment
+
+```javascript
+// Brainy automatically handles multi-AZ with S3
+const brain = new Brainy({
+  distributed: {
+    enabled: true,  // Auto-enables with S3 storage
+    coordinationMethod: 'auto' // Uses S3 for coordination
+  }
+})
+```
+
+### Load Balancing
+
+```bash
+# Application Load Balancer with health checks
+aws elbv2 create-load-balancer \
+  --name brainy-alb \
+  --subnets subnet-xxx subnet-yyy \
+  --security-groups sg-xxx
+
+aws elbv2 create-target-group \
+  --name brainy-targets \
+  --protocol HTTP \
+  --port 3000 \
+  --vpc-id vpc-xxx \
+  --health-check-path /health \
+  --health-check-interval-seconds 30
+```
+
+## Monitoring & Observability
+
+### CloudWatch Integration
+
+Brainy automatically sends metrics when running on AWS:
+
+```javascript
+// Automatic CloudWatch metrics (no config needed)
+// - Request count
+// - Response time
+// - Error rate
+// - Storage usage
+// - Memory usage
+```
+
+### Custom Metrics
+
+```javascript
+const brain = new Brainy({
+  monitoring: {
+    enabled: true,
+    customMetrics: {
+      namespace: 'Brainy/Production',
+      dimensions: [
+        { Name: 'Environment', Value: 'production' },
+        { Name: 'Service', Value: 'api' }
+      ]
+    }
+  }
+})
+```
+
+## Security Best Practices
+
+### 1. IAM Role (Recommended)
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:GetObject",
+        "s3:PutObject",
+        "s3:DeleteObject",
+        "s3:ListBucket"
+      ],
+      "Resource": [
+        "arn:aws:s3:::brainy-*/*",
+        "arn:aws:s3:::brainy-*"
+      ]
+    }
+  ]
+}
+```
+
+### 2. VPC Configuration
+
+```bash
+# Private subnets with NAT Gateway
+aws ec2 create-vpc --cidr-block 10.0.0.0/16
+aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.1.0/24 --availability-zone us-east-1a
+aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.2.0/24 --availability-zone us-east-1b
+```
+
+### 3. Encryption
+
+```javascript
+// Automatic encryption with S3
+const brain = new Brainy({
+  storage: {
+    type: 's3',
+    options: {
+      encryption: 'auto' // Uses S3 SSE-S3 by default
+    }
+  }
+})
+```
+
+## Cost Optimization
+
+### 1. Spot Instances (70% savings)
+
+```bash
+aws ec2 request-spot-fleet --spot-fleet-request-config '{
+  "IamFleetRole": "arn:aws:iam::xxx:role/fleet-role",
+  "TargetCapacity": 2,
+  "SpotPrice": "0.05",
+  "LaunchSpecifications": [{
+    "ImageId": "ami-xxx",
+    "InstanceType": "t3.medium",
+    "UserData": "BASE64_ENCODED_STARTUP_SCRIPT"
+  }]
+}'
+```
+
+### 2. S3 Intelligent-Tiering
+
+```javascript
+// Brainy automatically uses S3 Intelligent-Tiering
+const brain = new Brainy({
+  storage: {
+    type: 's3',
+    options: {
+      storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization
+    }
+  }
+})
+```
+
+### 3. Lambda Reserved Concurrency
+
+```bash
+aws lambda put-function-concurrency \
+  --function-name brainy-handler \
+  --reserved-concurrent-executions 10
+```
+
+## Deployment Automation
+
+### GitHub Actions CI/CD
+
+```yaml
+name: Deploy to AWS
+on:
+  push:
+    branches: [main]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-1
+      
+      - name: Deploy to ECS
+        run: |
+          docker build -t brainy .
+          docker tag brainy:latest $ECR_URI/brainy:latest
+          docker push $ECR_URI/brainy:latest
+          aws ecs update-service --cluster default --service brainy --force-new-deployment
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Storage Auto-Detection Fails**
+   ```javascript
+   // Explicitly specify storage
+   const brain = new Brainy({
+     storage: { type: 's3', options: { bucket: 'my-bucket' } }
+   })
+   ```
+
+2. **Memory Issues**
+   ```javascript
+   // Optimize for low memory
+   const brain = new Brainy({
+     cache: { maxSize: 100 }, // Reduce cache size
+     index: { M: 8 }          // Reduce HNSW connections
+   })
+   ```
+
+3. **Cold Starts (Lambda)**
+   ```javascript
+   // Warm-up configuration
+   exports.warmup = async () => {
+     if (!brain) {
+       brain = new Brainy({ warmup: true })
+       await brain.init()
+     }
+   }
+   ```
+
+## Production Checklist
+
+- [ ] IAM roles configured with minimal permissions
+- [ ] VPC with private subnets
+- [ ] Auto-scaling configured
+- [ ] CloudWatch alarms set up
+- [ ] Backup strategy (S3 versioning enabled)
+- [ ] SSL/TLS certificates configured
+- [ ] Rate limiting enabled
+- [ ] Health checks configured
+- [ ] Monitoring dashboard created
+- [ ] Cost alerts configured
+
+## Support
+
+- Documentation: https://brainy.soulcraft.ai/docs
+- Issues: https://github.com/soulcraft/brainy/issues
+- Community: https://discord.gg/brainy

docs/deployment/gcp-deployment.md

@@ -0,0 +1,547 @@
+# Google Cloud Platform Deployment Guide for Brainy
+
+## Overview
+Deploy Brainy on GCP with automatic scaling, global distribution, and zero-config dynamic adaptation. Brainy automatically detects and optimizes for GCP services.
+
+## Quick Start (Zero-Config)
+
+### Option 1: Cloud Run (Serverless Containers)
+
+```bash
+# Build and deploy with one command
+gcloud run deploy brainy \
+  --source . \
+  --platform managed \
+  --region us-central1 \
+  --allow-unauthenticated
+
+# Brainy auto-detects Cloud Run and configures:
+# - Memory-optimized caching
+# - GCS for storage (if available)
+# - Cloud SQL for metadata (if available)
+```
+
+### Option 2: Cloud Functions (Serverless)
+
+```javascript
+// index.js
+const { Brainy } = require('@soulcraft/brainy')
+
+let brain
+
+exports.brainyHandler = async (req, res) => {
+  // Zero-config - auto-adapts to Cloud Functions
+  if (!brain) {
+    brain = new Brainy() // Detects GCP environment automatically
+    await brain.init()
+  }
+  
+  const { method, ...params } = req.body
+  
+  try {
+    let result
+    switch(method) {
+      case 'add':
+        result = await brain.add(params)
+        break
+      case 'find':
+        result = await brain.find(params)
+        break
+      case 'relate':
+        result = await brain.relate(params)
+        break
+      default:
+        return res.status(400).json({ error: 'Unknown method' })
+    }
+    res.json({ result })
+  } catch (error) {
+    res.status(500).json({ error: error.message })
+  }
+}
+```
+
+Deploy:
+```bash
+gcloud functions deploy brainy \
+  --runtime nodejs20 \
+  --trigger-http \
+  --entry-point brainyHandler \
+  --memory 512MB \
+  --timeout 60s
+```
+
+### Option 3: Google Kubernetes Engine (GKE)
+
+```bash
+# Create autopilot cluster (fully managed, zero-config)
+gcloud container clusters create-auto brainy-cluster \
+  --region us-central1
+
+# Deploy using Cloud Build
+gcloud builds submit --tag gcr.io/$PROJECT_ID/brainy
+
+# Apply Kubernetes manifest
+kubectl apply -f - <<EOF
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: brainy
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: brainy
+  template:
+    metadata:
+      labels:
+        app: brainy
+    spec:
+      containers:
+      - name: brainy
+        image: gcr.io/$PROJECT_ID/brainy
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "250m"
+        env:
+        - name: NODE_ENV
+          value: production
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: brainy-service
+spec:
+  type: LoadBalancer
+  selector:
+    app: brainy
+  ports:
+  - port: 80
+    targetPort: 3000
+EOF
+```
+
+## Zero-Config Storage (Automatic)
+
+Brainy automatically detects and uses the best GCP storage:
+
+```javascript
+const brain = new Brainy()
+// Auto-detection priority:
+// 1. Firestore (if available)
+// 2. Cloud Storage (GCS)
+// 3. Cloud SQL
+// 4. Persistent Disk
+// 5. Memory (fallback)
+```
+
+### Cloud Storage Configuration (Optional)
+
+```javascript
+const brain = new Brainy({
+  storage: {
+    type: 's3', // GCS is S3-compatible
+    options: {
+      endpoint: 'https://storage.googleapis.com',
+      bucket: process.env.GCS_BUCKET || 'auto', // Auto-creates bucket
+      // Uses Application Default Credentials automatically
+    }
+  }
+})
+```
+
+### Firestore Integration (Optional)
+
+```javascript
+const brain = new Brainy({
+  storage: {
+    type: 'firestore',
+    options: {
+      projectId: process.env.GCP_PROJECT || 'auto',
+      collection: 'brainy-data'
+    }
+  }
+})
+```
+
+## Scaling Strategies
+
+### 1. Cloud Run Auto-scaling
+
+```yaml
+# service.yaml
+apiVersion: serving.knative.dev/v1
+kind: Service
+metadata:
+  name: brainy
+  annotations:
+    run.googleapis.com/execution-environment: gen2
+spec:
+  template:
+    metadata:
+      annotations:
+        autoscaling.knative.dev/minScale: "1"
+        autoscaling.knative.dev/maxScale: "1000"
+        autoscaling.knative.dev/target: "80"
+    spec:
+      containerConcurrency: 100
+      containers:
+      - image: gcr.io/PROJECT_ID/brainy
+        resources:
+          limits:
+            cpu: "2"
+            memory: "2Gi"
+```
+
+### 2. GKE Horizontal Pod Autoscaling
+
+```yaml
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: brainy-hpa
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: brainy
+  minReplicas: 3
+  maxReplicas: 100
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 70
+  - type: Resource
+    resource:
+      name: memory
+      target:
+        type: Utilization
+        averageUtilization: 80
+```
+
+## Global Distribution
+
+### Multi-Region Setup
+
+```javascript
+// Brainy automatically handles multi-region with GCS
+const brain = new Brainy({
+  distributed: {
+    enabled: true,
+    regions: ['us-central1', 'europe-west1', 'asia-northeast1'],
+    replication: 'auto' // Automatic cross-region replication
+  }
+})
+```
+
+### Traffic Director Configuration
+
+```bash
+# Global load balancing with Traffic Director
+gcloud compute backend-services create brainy-global \
+  --global \
+  --load-balancing-scheme=INTERNAL_SELF_MANAGED \
+  --protocol=HTTP2
+
+gcloud compute backend-services add-backend brainy-global \
+  --global \
+  --network-endpoint-group=brainy-neg \
+  --network-endpoint-group-region=us-central1
+```
+
+## Monitoring & Observability
+
+### Cloud Monitoring (Automatic)
+
+Brainy automatically sends metrics to Cloud Monitoring:
+
+```javascript
+// No configuration needed - automatic when running on GCP
+const brain = new Brainy()
+
+// Automatic metrics:
+// - Request latency
+// - Error rate
+// - Storage operations
+// - Cache hit rate
+// - Memory usage
+```
+
+### Custom Metrics
+
+```javascript
+const { Monitoring } = require('@google-cloud/monitoring')
+const monitoring = new Monitoring.MetricServiceClient()
+
+const brain = new Brainy({
+  onMetric: async (metric) => {
+    // Send custom metrics to Cloud Monitoring
+    await monitoring.createTimeSeries({
+      name: monitoring.projectPath(projectId),
+      timeSeries: [{
+        metric: {
+          type: `custom.googleapis.com/brainy/${metric.name}`,
+          labels: metric.labels
+        },
+        points: [{
+          interval: { endTime: { seconds: Date.now() / 1000 } },
+          value: { doubleValue: metric.value }
+        }]
+      }]
+    })
+  }
+})
+```
+
+### Cloud Trace Integration
+
+```javascript
+const brain = new Brainy({
+  tracing: {
+    enabled: true,
+    sampleRate: 0.1 // Sample 10% of requests
+  }
+})
+```
+
+## Security Best Practices
+
+### 1. Workload Identity (GKE)
+
+```yaml
+# Enable Workload Identity
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: brainy-sa
+  annotations:
+    iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com
+```
+
+### 2. Binary Authorization
+
+```yaml
+# Ensure only signed container images
+apiVersion: binaryauthorization.grafeas.io/v1beta1
+kind: Policy
+metadata:
+  name: brainy-policy
+spec:
+  defaultAdmissionRule:
+    requireAttestationsBy:
+    - projects/PROJECT_ID/attestors/prod-attestor
+```
+
+### 3. VPC Service Controls
+
+```bash
+# Create VPC Service Perimeter
+gcloud access-context-manager perimeters create brainy_perimeter \
+  --resources=projects/PROJECT_NUMBER \
+  --restricted-services=storage.googleapis.com \
+  --title="Brainy Security Perimeter"
+```
+
+## Cost Optimization
+
+### 1. Preemptible VMs (80% savings)
+
+```yaml
+# GKE node pool with preemptible VMs
+apiVersion: container.cnrm.cloud.google.com/v1beta1
+kind: ContainerNodePool
+metadata:
+  name: brainy-preemptible-pool
+spec:
+  clusterRef:
+    name: brainy-cluster
+  config:
+    preemptible: true
+    machineType: n2-standard-2
+  autoscaling:
+    minNodeCount: 1
+    maxNodeCount: 10
+```
+
+### 2. Cloud CDN for Static Assets
+
+```bash
+# Enable Cloud CDN for frequently accessed data
+gcloud compute backend-buckets create brainy-assets \
+  --gcs-bucket-name=brainy-static
+
+gcloud compute backend-buckets update brainy-assets \
+  --enable-cdn \
+  --cache-mode=CACHE_ALL_STATIC
+```
+
+### 3. Committed Use Discounts
+
+```bash
+# Purchase committed use for predictable workloads
+gcloud compute commitments create brainy-commitment \
+  --plan=TWELVE_MONTH \
+  --resources=vcpu=100,memory=400
+```
+
+## Deployment Automation
+
+### Cloud Build CI/CD
+
+```yaml
+# cloudbuild.yaml
+steps:
+  # Build container
+  - name: 'gcr.io/cloud-builders/docker'
+    args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.']
+  
+  # Push to registry
+  - name: 'gcr.io/cloud-builders/docker'
+    args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA']
+  
+  # Deploy to Cloud Run
+  - name: 'gcr.io/cloud-builders/gcloud'
+    args:
+    - 'run'
+    - 'deploy'
+    - 'brainy'
+    - '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'
+    - '--region=us-central1'
+    - '--platform=managed'
+
+# Trigger on push to main
+trigger:
+  branch:
+    name: main
+```
+
+### Terraform Infrastructure
+
+```hcl
+# main.tf
+resource "google_cloud_run_service" "brainy" {
+  name     = "brainy"
+  location = "us-central1"
+
+  template {
+    spec {
+      containers {
+        image = "gcr.io/${var.project_id}/brainy"
+        
+        resources {
+          limits = {
+            cpu    = "2"
+            memory = "2Gi"
+          }
+        }
+        
+        env {
+          name  = "NODE_ENV"
+          value = "production"
+        }
+      }
+    }
+  }
+
+  traffic {
+    percent         = 100
+    latest_revision = true
+  }
+}
+
+resource "google_cloud_run_service_iam_member" "public" {
+  service  = google_cloud_run_service.brainy.name
+  location = google_cloud_run_service.brainy.location
+  role     = "roles/run.invoker"
+  member   = "allUsers"
+}
+```
+
+## Performance Optimization
+
+### 1. Memory Store (Redis Compatible)
+
+```javascript
+// Brainy can use Memorystore for caching
+const brain = new Brainy({
+  cache: {
+    type: 'redis',
+    options: {
+      host: process.env.REDIS_HOST || 'auto-detect',
+      port: 6379
+    }
+  }
+})
+```
+
+### 2. Cloud Spanner for Global Consistency
+
+```javascript
+const brain = new Brainy({
+  metadata: {
+    type: 'spanner',
+    options: {
+      instance: 'brainy-instance',
+      database: 'brainy-db'
+    }
+  }
+})
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Quota Exceeded**
+   ```bash
+   # Check quotas
+   gcloud compute project-info describe --project=$PROJECT_ID
+   
+   # Request increase
+   gcloud compute project-info add-metadata \
+     --metadata google-compute-default-region=us-central1
+   ```
+
+2. **Cold Starts**
+   ```javascript
+   // Keep minimum instances warm
+   const brain = new Brainy({
+     warmup: {
+       enabled: true,
+       interval: 60000 // Ping every minute
+     }
+   })
+   ```
+
+3. **Memory Pressure**
+   ```javascript
+   // Optimize for GCP memory constraints
+   const brain = new Brainy({
+     memory: {
+       mode: 'aggressive', // Aggressive garbage collection
+       maxHeap: 0.8        // Use 80% of available memory
+     }
+   })
+   ```
+
+## Production Checklist
+
+- [ ] Enable Workload Identity for secure access
+- [ ] Configure Cloud Armor for DDoS protection
+- [ ] Set up Cloud KMS for encryption keys
+- [ ] Enable VPC Service Controls
+- [ ] Configure Cloud IAP for authentication
+- [ ] Set up Cloud Monitoring dashboards
+- [ ] Configure Error Reporting
+- [ ] Enable Cloud Trace
+- [ ] Set up budget alerts
+- [ ] Configure backup and disaster recovery
+
+## Support
+
+- Documentation: https://brainy.soulcraft.ai/docs
+- Issues: https://github.com/soulcraft/brainy/issues
+- GCP Marketplace: https://console.cloud.google.com/marketplace/product/brainy

docs/deployment/kubernetes-deployment.md

@@ -0,0 +1,727 @@
+# Kubernetes Deployment Guide for Brainy
+
+## Overview
+Deploy Brainy on Kubernetes with automatic scaling, high availability, and zero-config dynamic adaptation. Works with any Kubernetes distribution (vanilla, EKS, GKE, AKS, OpenShift, etc.).
+
+## Quick Start (Zero-Config)
+
+### Basic Deployment
+
+```yaml
+# brainy-deployment.yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: brainy
+  labels:
+    app: brainy
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: brainy
+  template:
+    metadata:
+      labels:
+        app: brainy
+    spec:
+      containers:
+      - name: brainy
+        image: soulcraft/brainy:latest
+        ports:
+        - containerPort: 3000
+        env:
+        - name: NODE_ENV
+          value: production
+        # Brainy auto-detects Kubernetes and configures accordingly
+        resources:
+          requests:
+            memory: "256Mi"
+            cpu: "100m"
+          limits:
+            memory: "1Gi"
+            cpu: "500m"
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: brainy-service
+spec:
+  selector:
+    app: brainy
+  ports:
+  - protocol: TCP
+    port: 80
+    targetPort: 3000
+  type: LoadBalancer
+```
+
+Deploy:
+```bash
+kubectl apply -f brainy-deployment.yaml
+```
+
+## Production-Grade Setup
+
+### 1. StatefulSet with Persistent Storage
+
+```yaml
+apiVersion: v1
+kind: StorageClass
+metadata:
+  name: brainy-storage
+provisioner: kubernetes.io/aws-ebs  # Or your cloud provider
+parameters:
+  type: gp3
+  iopsPerGB: "10"
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: brainy
+spec:
+  serviceName: brainy-headless
+  replicas: 3
+  selector:
+    matchLabels:
+      app: brainy
+  template:
+    metadata:
+      labels:
+        app: brainy
+    spec:
+      containers:
+      - name: brainy
+        image: soulcraft/brainy:latest
+        ports:
+        - containerPort: 3000
+        env:
+        - name: NODE_ENV
+          value: production
+        - name: BRAINY_STORAGE_TYPE
+          value: filesystem
+        - name: BRAINY_STORAGE_PATH
+          value: /data
+        volumeMounts:
+        - name: data
+          mountPath: /data
+        livenessProbe:
+          httpGet:
+            path: /health
+            port: 3000
+          initialDelaySeconds: 30
+          periodSeconds: 10
+        readinessProbe:
+          httpGet:
+            path: /ready
+            port: 3000
+          initialDelaySeconds: 5
+          periodSeconds: 5
+  volumeClaimTemplates:
+  - metadata:
+      name: data
+    spec:
+      accessModes: ["ReadWriteOnce"]
+      storageClassName: brainy-storage
+      resources:
+        requests:
+          storage: 10Gi
+```
+
+### 2. Horizontal Pod Autoscaler
+
+```yaml
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: brainy-hpa
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: brainy
+  minReplicas: 2
+  maxReplicas: 100
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 70
+  - type: Resource
+    resource:
+      name: memory
+      target:
+        type: Utilization
+        averageUtilization: 80
+  behavior:
+    scaleUp:
+      stabilizationWindowSeconds: 60
+      policies:
+      - type: Percent
+        value: 100
+        periodSeconds: 60
+    scaleDown:
+      stabilizationWindowSeconds: 300
+      policies:
+      - type: Percent
+        value: 10
+        periodSeconds: 60
+```
+
+### 3. Ingress Configuration
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: brainy-ingress
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    nginx.ingress.kubernetes.io/rate-limit: "100"
+    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
+spec:
+  tls:
+  - hosts:
+    - api.brainy.example.com
+    secretName: brainy-tls
+  rules:
+  - host: api.brainy.example.com
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: brainy-service
+            port:
+              number: 80
+```
+
+## Zero-Config Storage Options
+
+### Option 1: S3-Compatible Storage (Recommended)
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: brainy-s3-credentials
+type: Opaque
+data:
+  access-key: <base64-encoded-key>
+  secret-key: <base64-encoded-secret>
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: brainy
+spec:
+  template:
+    spec:
+      containers:
+      - name: brainy
+        env:
+        - name: BRAINY_STORAGE_TYPE
+          value: s3
+        - name: AWS_ACCESS_KEY_ID
+          valueFrom:
+            secretKeyRef:
+              name: brainy-s3-credentials
+              key: access-key
+        - name: AWS_SECRET_ACCESS_KEY
+          valueFrom:
+            secretKeyRef:
+              name: brainy-s3-credentials
+              key: secret-key
+        - name: S3_BUCKET
+          value: brainy-data
+        - name: AWS_REGION
+          value: us-east-1
+```
+
+### Option 2: MinIO (Self-Hosted S3)
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: minio
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: minio
+  template:
+    metadata:
+      labels:
+        app: minio
+    spec:
+      containers:
+      - name: minio
+        image: minio/minio:latest
+        args:
+        - server
+        - /data
+        env:
+        - name: MINIO_ROOT_USER
+          value: brainy
+        - name: MINIO_ROOT_PASSWORD
+          value: brainy123456
+        volumeMounts:
+        - name: data
+          mountPath: /data
+      volumes:
+      - name: data
+        persistentVolumeClaim:
+          claimName: minio-pvc
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: minio-service
+spec:
+  selector:
+    app: minio
+  ports:
+  - port: 9000
+    targetPort: 9000
+```
+
+### Option 3: Shared NFS Storage
+
+```yaml
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: brainy-nfs-pv
+spec:
+  capacity:
+    storage: 100Gi
+  accessModes:
+    - ReadWriteMany
+  nfs:
+    server: nfs-server.example.com
+    path: /export/brainy
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: brainy-nfs-pvc
+spec:
+  accessModes:
+    - ReadWriteMany
+  resources:
+    requests:
+      storage: 100Gi
+```
+
+## High Availability Configuration
+
+### 1. Pod Anti-Affinity
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: brainy
+spec:
+  template:
+    spec:
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchExpressions:
+              - key: app
+                operator: In
+                values:
+                - brainy
+            topologyKey: kubernetes.io/hostname
+```
+
+### 2. Pod Disruption Budget
+
+```yaml
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: brainy-pdb
+spec:
+  minAvailable: 2
+  selector:
+    matchLabels:
+      app: brainy
+```
+
+### 3. Multi-Zone Deployment
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: brainy
+spec:
+  template:
+    spec:
+      topologySpreadConstraints:
+      - maxSkew: 1
+        topologyKey: topology.kubernetes.io/zone
+        whenUnsatisfiable: DoNotSchedule
+        labelSelector:
+          matchLabels:
+            app: brainy
+```
+
+## Monitoring & Observability
+
+### 1. Prometheus Metrics
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: brainy-metrics
+  annotations:
+    prometheus.io/scrape: "true"
+    prometheus.io/port: "9090"
+    prometheus.io/path: "/metrics"
+spec:
+  selector:
+    app: brainy
+  ports:
+  - name: metrics
+    port: 9090
+    targetPort: 9090
+```
+
+### 2. Grafana Dashboard
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: brainy-dashboard
+data:
+  dashboard.json: |
+    {
+      "dashboard": {
+        "title": "Brainy Metrics",
+        "panels": [
+          {
+            "title": "Request Rate",
+            "targets": [
+              {
+                "expr": "rate(brainy_requests_total[5m])"
+              }
+            ]
+          },
+          {
+            "title": "Response Time",
+            "targets": [
+              {
+                "expr": "histogram_quantile(0.95, brainy_response_time)"
+              }
+            ]
+          }
+        ]
+      }
+    }
+```
+
+### 3. Logging with Fluentd
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: fluentd-config
+data:
+  fluent.conf: |
+    <source>
+      @type tail
+      path /var/log/containers/brainy*.log
+      pos_file /var/log/fluentd-brainy.log.pos
+      tag brainy.*
+      <parse>
+        @type json
+      </parse>
+    </source>
+    
+    <match brainy.**>
+      @type elasticsearch
+      host elasticsearch.logging.svc.cluster.local
+      port 9200
+      logstash_format true
+      logstash_prefix brainy
+    </match>
+```
+
+## Security Best Practices
+
+### 1. Network Policies
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: brainy-netpol
+spec:
+  podSelector:
+    matchLabels:
+      app: brainy
+  policyTypes:
+  - Ingress
+  - Egress
+  ingress:
+  - from:
+    - namespaceSelector:
+        matchLabels:
+          name: ingress-nginx
+    ports:
+    - protocol: TCP
+      port: 3000
+  egress:
+  - to:
+    - namespaceSelector: {}
+    ports:
+    - protocol: TCP
+      port: 443  # HTTPS
+    - protocol: TCP
+      port: 9000 # MinIO/S3
+```
+
+### 2. Pod Security Policy
+
+```yaml
+apiVersion: policy/v1beta1
+kind: PodSecurityPolicy
+metadata:
+  name: brainy-psp
+spec:
+  privileged: false
+  allowPrivilegeEscalation: false
+  requiredDropCapabilities:
+    - ALL
+  volumes:
+    - 'configMap'
+    - 'emptyDir'
+    - 'projected'
+    - 'secret'
+    - 'persistentVolumeClaim'
+  runAsUser:
+    rule: 'MustRunAsNonRoot'
+  seLinux:
+    rule: 'RunAsAny'
+  fsGroup:
+    rule: 'RunAsAny'
+```
+
+### 3. RBAC Configuration
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: brainy-sa
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: brainy-role
+rules:
+- apiGroups: [""]
+  resources: ["configmaps"]
+  verbs: ["get", "list", "watch"]
+- apiGroups: [""]
+  resources: ["secrets"]
+  verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: brainy-rolebinding
+subjects:
+- kind: ServiceAccount
+  name: brainy-sa
+roleRef:
+  kind: Role
+  name: brainy-role
+  apiGroup: rbac.authorization.k8s.io
+```
+
+## GitOps with ArgoCD
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: brainy
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: https://github.com/yourorg/brainy-k8s
+    targetRevision: HEAD
+    path: manifests
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: brainy
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+    - CreateNamespace=true
+```
+
+## Helm Chart Installation
+
+```bash
+# Add Brainy Helm repository
+helm repo add brainy https://charts.brainy.io
+helm repo update
+
+# Install with custom values
+cat > values.yaml << EOF
+replicaCount: 3
+image:
+  repository: soulcraft/brainy
+  tag: latest
+  pullPolicy: IfNotPresent
+
+service:
+  type: LoadBalancer
+  port: 80
+
+ingress:
+  enabled: true
+  className: nginx
+  hosts:
+    - host: api.brainy.example.com
+      paths:
+        - path: /
+          pathType: Prefix
+
+resources:
+  limits:
+    cpu: 1000m
+    memory: 1Gi
+  requests:
+    cpu: 100m
+    memory: 256Mi
+
+autoscaling:
+  enabled: true
+  minReplicas: 2
+  maxReplicas: 100
+  targetCPUUtilizationPercentage: 70
+
+storage:
+  type: s3
+  s3:
+    bucket: brainy-data
+    region: us-east-1
+EOF
+
+helm install brainy brainy/brainy -f values.yaml
+```
+
+## Cost Optimization
+
+### 1. Spot/Preemptible Nodes
+
+```yaml
+apiVersion: v1
+kind: NodePool
+metadata:
+  name: brainy-spot-pool
+spec:
+  nodeSelector:
+    node.kubernetes.io/lifecycle: spot
+  taints:
+  - key: spot
+    value: "true"
+    effect: NoSchedule
+  tolerations:
+  - key: spot
+    operator: Equal
+    value: "true"
+    effect: NoSchedule
+```
+
+### 2. Vertical Pod Autoscaler
+
+```yaml
+apiVersion: autoscaling.k8s.io/v1
+kind: VerticalPodAutoscaler
+metadata:
+  name: brainy-vpa
+spec:
+  targetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: brainy
+  updatePolicy:
+    updateMode: "Auto"
+  resourcePolicy:
+    containerPolicies:
+    - containerName: brainy
+      minAllowed:
+        cpu: 100m
+        memory: 128Mi
+      maxAllowed:
+        cpu: 2
+        memory: 2Gi
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Pod CrashLoopBackOff**
+   ```bash
+   kubectl logs -f pod/brainy-xxx
+   kubectl describe pod brainy-xxx
+   ```
+
+2. **Storage Issues**
+   ```bash
+   kubectl get pv,pvc
+   kubectl describe pvc brainy-data
+   ```
+
+3. **Network Connectivity**
+   ```bash
+   kubectl exec -it pod/brainy-xxx -- curl http://brainy-service/health
+   kubectl get endpoints brainy-service
+   ```
+
+4. **Memory Pressure**
+   ```bash
+   kubectl top pods -l app=brainy
+   kubectl describe node
+   ```
+
+## Production Checklist
+
+- [ ] High availability with multiple replicas
+- [ ] Pod disruption budgets configured
+- [ ] Resource limits and requests set
+- [ ] Horizontal and vertical autoscaling enabled
+- [ ] Persistent storage configured
+- [ ] Network policies in place
+- [ ] RBAC properly configured
+- [ ] Monitoring and alerting setup
+- [ ] Backup and disaster recovery plan
+- [ ] Security scanning enabled
+- [ ] GitOps deployment pipeline
+
+## Support
+
+- Documentation: https://brainy.soulcraft.ai/docs
+- Helm Charts: https://github.com/soulcraft/brainy-charts
+- Issues: https://github.com/soulcraft/brainy/issues
+- Slack: https://brainy-community.slack.com

docs/eli5.md

@@ -1,155 +0,0 @@
----
-title: What is Brainy?
-slug: getting-started/what-is-brainy
-public: true
-category: getting-started
-template: guide
-order: 0
-description: Plain-language guide covering what Brainy does, how it compares to other tools, and what you can build with it. No jargon, no code — just clear analogies.
-next:
-  - getting-started/installation
-  - getting-started/quick-start
----
-
-# Brainy and Cor — Explained Simply
-
-*A plain-language guide for anyone who wants to understand what this thing actually does.*
-
----
-
-## What is Brainy?
-
-Imagine you have the world's smartest librarian.
-
-You walk up and say *"I'm looking for something about climate change — but only books published after 2020, and only ones written by authors I've already read."* A normal library would make you dig through a card catalogue, then cross-reference a list of authors, then scan the shelves yourself. That takes a while.
-
-Your smart librarian does all three at the same time — in less than the time it takes to blink.
-
-That's Brainy. It's a knowledge database that can search by **meaning**, follow **connections**, and filter by **labels** — all at once, in a single question.
-
----
-
-## The Three Superpowers
-
-### 1. Meaning Search (the "fuzzy" superpower)
-
-When you search for "automobile," Brainy also finds results about "car," "vehicle," and "sedan" — because it understands what words *mean*, not just how they're spelled. It reads your data the way a person would, not the way a search box does.
-
-Think of it like the librarian who finds books on "heartbreak" when you ask for something about "loneliness."
-
-### 2. Relationship Walking (the "follow the thread" superpower)
-
-Every piece of information can be connected to other pieces. A Person *works at* a Company. A Project *depends on* a Tool. A Recipe *contains* Ingredients.
-
-Brainy can follow these connections across many hops in one step. Ask for "everything connected to this author, two steps out" and Brainy returns the author's books, the books' publishers, the publishers' other authors — without you needing to chain four separate lookups yourself.
-
-Think of it like the librarian who not only hands you the book you asked for, but also knows which shelf it came from, who donated it, and what other books arrived in the same donation.
-
-### 3. Label Filtering (the "narrow it down" superpower)
-
-Sometimes meaning and connections aren't enough — you need precision. "Only recipes with fewer than 500 calories." "Only events from last week." "Only documents tagged as urgent."
-
-Brainy can narrow any result set down by exact labels or ranges in the same breath as the other two searches. No extra steps.
-
----
-
-## What Else Can It Do?
-
-- **Virtual file cabinet.** Brainy includes a full filesystem you can use to store, organize, and semantically search files — PDFs, documents, anything — the same way you search everything else.
-
-- **Live dashboards.** You can define running totals that Brainy keeps updated automatically — things like "total sales this month by region" or "average response time per service." Every time new data comes in, the numbers stay current with no manual recalculation.
-
-- **Time travel.** Every committed change becomes part of the database's history. You can pin the current state as a frozen view, see the whole knowledge base exactly as it was last week, try out changes in a scratch copy that never touches the real data, and take instant backups.
-
-- **Universal vocabulary.** Brainy ships with a shared language of 42 kinds of things (Person, Document, Task, Concept, Event…) and 127 kinds of connections (Contains, DependsOn, Creates, RelatedTo…). This means data from different sources speaks the same language without you having to translate.
-
----
-
-## What is Cor?
-
-Cor is a turbocharger for Brainy.
-
-Same car. Same controls. Same fuel. You just swap in a faster engine under the hood, and everything that used to take a moment now happens instantly.
-
-Technically, Cor is an optional plugin written in Rust — a lower-level language that runs much closer to the raw metal of your processor. It plugs into Brainy and takes over the most compute-intensive work: the distance calculations that power meaning search, the number-crunching behind live aggregates, and the set operations that drive label filtering.
-
-You install it with one line, register it with one call, and Brainy automatically uses it everywhere it can help.
-
----
-
-## How Much Faster?
-
-Plain language:
-
-- **Searches** go from "the blink of an eye" to "faster than a blink." The overall speedup is **5.2× on average** across all operations.
-- **Live aggregates** are rebuilt using all CPU cores in parallel, so re-indexing large datasets takes a fraction of the time.
-- **Analytics** that aren't even possible in pure JavaScript — real-time anomaly detection, streaming percentile estimates, approximate unique counts — become available because Cor brings the native capabilities required to run them efficiently.
-
-If Brainy is what makes knowledge fast, Cor is what makes Brainy feel instant.
-
----
-
-## What Does Brainy Replace?
-
-Most applications that need to store and search knowledge end up stitching together several specialized tools. Brainy replaces all of them with one — a single free, open-source library in place of multiple paid services.
-
-### Before and After
-
-**Before Brainy** — a pile of services:
-- Pinecone (vectors) + Neo4j (graph) + MongoDB (docs)
-- Algolia (search) + Redis (cache) + PostgreSQL + pgvector
-- Plus glue code, sync jobs, ETL pipelines, and 3am incidents
-
-**After Brainy** — one thing:
-Search, graph, filter, files, time travel, and imports — unified in a single library.
-
-### What Each Tool Is Missing
-
-| Tool | Search | Graph | Filter | VFS | Time travel | Import |
-|---|:---:|:---:|:---:|:---:|:---:|:---:|
-| **Brainy** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
-| *— Vector databases —* | | | | | | |
-| Pinecone | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Weaviate | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Qdrant | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Chroma | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| *— Graph databases —* | | | | | | |
-| Neo4j | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
-| *— Document stores —* | | | | | | |
-| MongoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Firestore | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| DynamoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| *— Relational + vector —* | | | | | | |
-| PostgreSQL + pgvector | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| MySQL | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| SQLite | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| *— Search engines —* | | | | | | |
-| Elasticsearch | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Algolia | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| *— Cache —* | | | | | | |
-| Redis | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
-
-Brainy is the only row with every box checked. And it runs all of them in a single query — no stitching services together.
-
-### One library, any scale
-
-Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way.
-
-Add Cor and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.
-
----
-
-## What Can You Build?
-
-### Common applications
-
-- **AI agents with persistent memory** — Give any AI an always-on, self-organizing knowledge graph that persists between sessions and across agents.
-- **Searchable knowledge bases** — Build institutional memory that links documents automatically and surfaces answers across the full web of related information.
-- **Semantic document search** — Index PDFs, code, or media and find them by meaning, not just keywords.
-- **Relationship-aware recommendations** — Power product catalogs or content platforms where every recommendation understands what connects to what.
-- **Safe experiments** — Test risky changes against a scratch copy of the knowledge base, audit exactly what changed and when, and roll back to any snapshot instantly.
-- **Unified business platforms** — Combine booking, CRM, inventory, and analytics in one queryable knowledge graph with no sync pipeline.
-
-### What Brainy is good at
-
-Brainy is the engine underneath production systems that need to combine semantic search, structured filtering, and graph traversal in a single query — agent memory, knowledge-base platforms, business operations consoles, multi-agent coordination, and more. The combination of vector + graph + metadata search in one indexed call is what differentiates it from running three engines side by side.

docs/features/complete-feature-list.md

@@ -0,0 +1,413 @@
+# 🚀 Brainy 2.0 - Complete Feature List
+
+> **The Truth**: Brainy is MORE powerful than previously documented! This is the complete list of ALL implemented features.
+
+## 🧠 Core Intelligence Engine
+
+### Triple Intelligence System ✅
+Unified query system that automatically combines:
+- **Vector Search**: HNSW-indexed semantic similarity (O(log n) performance)
+- **Graph Traversal**: Relationship-based discovery
+- **Field Filtering**: Metadata and attribute queries
+- **Auto-optimization**: Queries are automatically optimized based on data patterns
+
+```typescript
+// All three intelligences work together automatically
+const results = await brain.find({
+  like: 'AI research',           // Vector search
+  where: { year: 2024 },          // Metadata filtering  
+  connected: { to: authorId }     // Graph traversal
+})
+```
+
+### Neural Query Understanding ✅
+- **220+ embedded patterns** for query intent detection
+- Natural language query processing
+- Automatic query type detection
+- Query rewriting and optimization
+
+## 🔧 12+ Production Augmentations
+
+```typescript
+// Full crash recovery, checkpointing, replay
+```
+
+### 2. Entity Registry ✅
+```typescript
+import { EntityRegistryAugmentation } from 'brainy'
+// Bloom filter-based deduplication for streaming data
+// Handles millions of entities with minimal memory
+```
+
+### 3. Auto-Register Entities ✅
+```typescript
+import { AutoRegisterEntitiesAugmentation } from 'brainy'
+// Automatically extracts and registers entities from text
+```
+
+### 4. Intelligent Verb Scoring ✅
+```typescript
+import { IntelligentVerbScoringAugmentation } from 'brainy'
+// Multi-factor relationship strength:
+// - Semantic similarity
+// - Temporal decay
+// - Frequency amplification
+// - Context awareness
+```
+
+### 5. Batch Processing ✅
+```typescript
+import { BatchProcessingAugmentation } from 'brainy'
+// Adaptive batching with backpressure
+// Dynamically adjusts batch size based on load
+```
+
+### 6. Connection Pool ✅
+```typescript
+import { ConnectionPoolAugmentation } from 'brainy'
+// Auto-scaling connection management
+// Optimized for distributed operations
+```
+
+### 7. Request Deduplicator ✅
+```typescript
+import { RequestDeduplicatorAugmentation } from 'brainy'
+// In-flight request deduplication
+// 3x performance boost for concurrent operations
+```
+
+### 8. WebSocket Conduit ✅
+```typescript
+import { WebSocketConduitAugmentation } from 'brainy'
+// Real-time bidirectional streaming
+// Auto-reconnection and heartbeat
+```
+
+### 9. WebRTC Conduit ✅
+```typescript
+import { WebRTCConduitAugmentation } from 'brainy'
+// Peer-to-peer data channels
+// Direct browser-to-browser communication
+```
+
+### 10. Memory Storage Optimization ✅
+```typescript
+import { MemoryStorageAugmentation } from 'brainy'
+// Memory-specific optimizations
+// Circular buffers, compression
+```
+
+### 11. Server Search Conduit ✅
+```typescript
+import { ServerSearchConduitAugmentation } from 'brainy'
+// Distributed query execution
+// Load balancing across nodes
+```
+
+### 12. Neural Import ✅
+```typescript
+import { NeuralImportAugmentation } from 'brainy'
+// AI-powered data understanding
+// Automatic entity detection and classification
+// Relationship discovery
+```
+
+## 🤖 Neural Import Capabilities (FULLY IMPLEMENTED!)
+
+```typescript
+const neuralImport = new NeuralImport(brain)
+
+// ALL of these work TODAY:
+await neuralImport.neuralImport('data.csv')
+await neuralImport.detectEntitiesWithNeuralAnalysis(data)
+await neuralImport.detectNounType(entity)
+await neuralImport.detectRelationships(entities)
+await neuralImport.generateInsights(data)
+```
+
+### Features:
+- **Auto-detects file format** (CSV, JSON, XML, etc.)
+- **Identifies entity types** using AI
+- **Discovers relationships** between entities
+- **Generates insights** about the data
+- **Creates optimal graph structure** automatically
+
+## 🎯 Zero-Config Model Loading Cascade
+
+Brainy automatically loads models with ZERO configuration required:
+
+```typescript
+const brain = new Brainy() // That's it!
+await brain.init()
+// Models load automatically from best available source
+```
+
+### Loading Priority:
+1. **Local Cache** (./models) - Instant, no network
+2. **CDN** (models.soulcraft.com) - Fast, global [Coming Soon]
+3. **GitHub Releases** - Reliable backup
+4. **HuggingFace** - Ultimate fallback
+
+### Key Features:
+- **Automatic fallback** if sources fail
+- **Model verification** with checksums
+- **Offline support** with bundled models
+- **No environment variables needed**
+- **Works in all environments** (Node, Browser, Workers)
+
+## 🏢 Distributed Operation Modes
+
+### Reader Mode ✅
+```typescript
+const brain = new Brainy({ mode: 'reader' })
+// Optimized for read-heavy workloads
+// 80% cache ratio, aggressive prefetch
+// 1 hour TTL, minimal writes
+```
+
+### Writer Mode ✅
+```typescript
+const brain = new Brainy({ mode: 'writer' })
+// Optimized for write-heavy workloads
+// Large write buffers, batch writes
+// Minimal caching, fast ingestion
+```
+
+### Hybrid Mode ✅
+```typescript
+const brain = new Brainy({ mode: 'hybrid' })
+// Balanced for mixed workloads
+// Adaptive caching and batching
+```
+
+## 💾 Advanced Caching System
+
+### 3-Level Cache Architecture ✅
+```typescript
+const cacheConfig = {
+  hotCache: {
+    size: 1000,      // L1 - RAM
+    ttl: 60000       // 1 minute
+  },
+  warmCache: {
+    size: 10000,     // L2 - Fast storage
+    ttl: 300000      // 5 minutes
+  },
+  coldCache: {
+    size: 100000,    // L3 - Persistent
+    ttl: null        // No expiry
+  }
+}
+```
+
+### Cache Features:
+- **Automatic promotion/demotion** between levels
+- **LRU eviction** within each level
+- **Compression** for cold cache
+- **Statistics tracking** for optimization
+
+## 📊 Comprehensive Statistics
+
+```typescript
+const stats = await brain.getStats()
+// Returns detailed metrics:
+{
+  nouns: {
+    count, created, updated, deleted,
+    size, avgSize
+  },
+  verbs: {
+    count, created, types,
+    weights: { min, max, avg }
+  },
+  vectors: {
+    dimensions: 384,
+    indexSize, partitions,
+    avgSearchTime
+  },
+  cache: {
+    hits, misses, evictions,
+    hitRate, sizes
+  },
+  performance: {
+    operations, avgTimes,
+    p95Latency, p99Latency
+  },
+  storage: {
+    used, available,
+    compression, files
+  },
+  throttling: {
+    delays, rateLimited,
+    backoffMs, retries
+  }
+}
+```
+
+## 🚀 GPU Acceleration Support
+
+```typescript
+// Automatic GPU detection
+const device = await detectBestDevice()
+// Returns: 'cpu' | 'webgpu' | 'cuda'
+
+// WebGPU in browser (when available)
+if (device === 'webgpu') {
+  // Transformer models use WebGPU automatically
+}
+
+// CUDA in Node.js (requires ONNX Runtime GPU)
+if (device === 'cuda') {
+  // Automatically uses GPU for embeddings
+}
+```
+
+## 🔄 Adaptive Systems
+
+### Adaptive Backpressure ✅
+```typescript
+// Automatically adjusts flow based on system load
+// Prevents OOM and maintains throughput
+```
+
+### Adaptive Socket Manager ✅
+```typescript
+// Dynamic connection pooling
+// Scales connections based on traffic patterns
+```
+
+### Cache Auto-Configuration ✅
+```typescript
+// Sizes cache based on available memory
+// Adjusts strategies based on usage patterns
+```
+
+### S3 Throttling Protection ✅
+```typescript
+// Built-in exponential backoff
+// Rate limit detection and adaptation
+// Automatic retry with jitter
+```
+
+## 🛠️ Storage Adapters
+
+All included, auto-selected based on environment:
+
+### FileSystem Storage ✅
+- Default for Node.js
+- Efficient file-based storage
+- Automatic directory management
+
+### Memory Storage ✅
+- Ultra-fast in-memory operations
+- Perfect for testing and temporary data
+- Circular buffer support
+
+### OPFS Storage ✅
+- Browser persistent storage
+- Survives page refreshes
+- Quota management
+
+### S3 Storage ✅
+- AWS S3 compatible
+- Automatic multipart uploads
+- Throttling protection
+- Batch operations
+
+## 🎨 Natural Language Processing
+
+### Built-in Patterns (220+)
+- Question types (what, why, how, when, where)
+- Temporal queries (yesterday, last week, 2024)
+- Comparative queries (better than, similar to)
+- Aggregations (count, sum, average)
+- Filters (only, except, without)
+- Relationships (related to, connected with)
+
+### Coverage: 94-98% of typical queries!
+
+## 🔐 Security Features
+
+### Built-in Security ✅
+- Automatic input sanitization
+- SQL injection prevention
+- XSS protection for web contexts
+- Rate limiting support
+
+### Encryption Ready ✅
+```typescript
+import { crypto } from 'brainy/utils'
+// AES-256-GCM encryption utilities
+// Key derivation functions
+// Secure random generation
+```
+
+## 🎯 Key Design Principles
+
+### 1. Zero Configuration
+```typescript
+const brain = new Brainy()
+await brain.init()
+// Everything else is automatic!
+```
+
+### 2. Fixed Dimensions (384)
+- **ALWAYS** uses all-MiniLM-L6-v2 model
+- **ALWAYS** 384 dimensions
+- **NOT** configurable (by design)
+- Ensures everything works together
+
+### 3. Progressive Enhancement
+- Starts simple, scales automatically
+- Adapts to workload patterns
+- Optimizes based on usage
+
+### 4. Universal Compatibility
+- Works in Node.js 18+
+- Works in modern browsers
+- Works in Web Workers
+- Works in Edge environments
+
+## 📦 What Ships in Core (MIT Licensed)
+
+**EVERYTHING** is included in the core package:
+- ✅ All engines (vector, graph, field, neural)
+- ✅ All augmentations (12+)
+- ✅ All storage adapters
+- ✅ All distributed modes
+- ✅ Complete statistics
+- ✅ GPU support
+- ✅ No feature limitations
+- ✅ No premium tiers
+- ✅ 100% MIT licensed
+
+## 🚀 Quick Start
+
+```typescript
+import { Brainy } from 'brainy'
+
+// Zero config required!
+const brain = new Brainy()
+await brain.init()
+
+// Add data (auto-detects type)
+await brain.add('Content here')
+
+// Search with natural language
+const results = await brain.find('related content from last week')
+
+// Everything else is automatic!
+```
+
+## 📈 Performance Characteristics
+
+- **Vector Search**: O(log n) with HNSW indexing
+- **Graph Traversal**: O(k) for k-hop queries
+- **Field Filtering**: O(1) with metadata index
+- **Memory Usage**: ~100MB base + data
+- **Embedding Speed**: ~100ms for batch of 10
+- **Query Speed**: <10ms for most queries
+
+## 🎉 Summary
+
+Brainy 2.0 is a **complete**, **production-ready** AI database that requires **ZERO configuration**. Every feature listed here is **implemented and working** today. No configuration, no setup, no complexity - just powerful AI capabilities that work out of the box!

docs/features/v3-features.md

@@ -0,0 +1,296 @@
+# 🚀 Brainy - Production-Ready Features
+
+> **Status**: All features listed here are IMPLEMENTED and TESTED
+
+## 📊 Performance Metrics
+- **Search Latency**: <10ms for 10,000+ items
+- **Write Throughput**: 10,000+ ops/sec
+- **Memory Efficiency**: <500MB for 10K items
+- **Concurrent Operations**: 100+ simultaneous operations
+
+## 🧠 Core Intelligence Features
+
+### Triple Intelligence System ✅
+Unified query system combining three types of intelligence:
+```typescript
+const results = await brain.find({
+  like: 'AI research',           // Vector similarity search
+  where: { year: 2024 },          // Metadata filtering
+  connected: { to: authorId }     // Graph relationships
+})
+```
+
+### Intelligent Type Mapping ✅
+Prevents semantic degradation by intelligently inferring types:
+```typescript
+// Automatically infers 'person' from email field
+brain.add({ name: "John", email: "john@example.com" }, 'entity')
+// → Stored as type: 'person', not generic 'entity'
+```
+
+### Neural Query Understanding ✅
+- 220+ embedded patterns for intent detection
+- Natural language query processing
+- Automatic query optimization
+- Pattern-based query rewriting
+
+## 🏢 Enterprise Features
+
+### Distributed Coordination ✅
+Raft consensus for multi-node deployments:
+```typescript
+import { DistributedCoordinator } from '@soulcraft/brainy'
+
+const coordinator = createCoordinator({
+  nodeId: 'node-1',
+  peers: ['node-2', 'node-3'],
+  electionTimeout: 500
+})
+// Automatic leader election and failover
+```
+
+### Horizontal Sharding ✅
+Consistent hashing for data distribution:
+```typescript
+import { ShardManager } from '@soulcraft/brainy'
+
+const shards = createShardManager({
+  nodes: ['node-1', 'node-2', 'node-3'],
+  replicationFactor: 2,
+  virtualNodes: 150
+})
+// Automatic shard rebalancing on node changes
+```
+
+### Read/Write Separation ✅
+Primary-replica architecture for scale:
+```typescript
+import { ReadWriteSeparation } from '@soulcraft/brainy'
+
+const replication = createReadWriteSeparation({
+  role: 'auto',  // Automatic primary/replica detection
+  consistencyLevel: 'strong',  // or 'eventual'
+  readPreference: 'nearest'
+})
+```
+
+### Cross-Instance Cache Sync ✅
+Version vector-based cache synchronization:
+```typescript
+import { CacheSync } from '@soulcraft/brainy'
+
+const cache = createCacheSync({
+  nodeId: 'node-1',
+  syncInterval: 100,
+  conflictResolution: 'version-vector'
+})
+```
+
+## 🔐 Security & Compliance
+
+### Rate Limiting ✅
+Per-operation configurable limits:
+```typescript
+const rateLimiter = createRateLimitAugmentation({
+  limits: {
+    searches: 1000,  // per minute
+    writes: 100,
+    reads: 5000,
+    deletes: 50
+  }
+})
+```
+
+### Audit Logging ✅
+Comprehensive operation tracking:
+```typescript
+const auditLogger = createAuditLogAugmentation({
+  logLevel: 'detailed',
+  retention: 90,  // days
+  includeMetadata: true
+})
+
+// Query audit logs
+const logs = auditLogger.queryLogs({
+  operation: 'add',
+  startTime: Date.now() - 3600000
+})
+```
+
+## 📦 Storage & Persistence
+
+Full crash recovery and replay:
+```typescript
+  enabled: true,
+  checkpointInterval: 1000,
+  maxLogSize: 100 * 1024 * 1024  // 100MB
+}))
+```
+
+### Multi-Tenancy ✅
+Service-based data isolation:
+```typescript
+// Isolated data per service
+await brain.add(data, 'document', { service: 'tenant-1' })
+await brain.find('query', { service: 'tenant-1' })
+```
+
+### Write-Only Mode ✅
+For dedicated write nodes:
+```typescript
+const brain = new Brainy({
+  mode: 'write-only',
+  storage: 's3://bucket/path'
+})
+```
+
+## 🚀 Performance Features
+
+### Batch Operations ✅
+Optimized bulk processing:
+```typescript
+// Parallel processing with automatic batching
+await brain.addMany(items)  // <10ms per item
+await brain.updateMany(updates)
+await brain.deleteMany(filters)
+```
+
+### Request Deduplication ✅
+Automatic duplicate request handling:
+```typescript
+brain.use(new RequestDeduplicatorAugmentation())
+// Identical concurrent requests return same result
+```
+
+### Smart Caching ✅
+Intelligent search result caching:
+```typescript
+brain.use(new CacheAugmentation({
+  maxSize: 10000,
+  ttl: 300000,  // 5 minutes
+  invalidateOnWrite: true
+}))
+```
+
+## 🔄 Data Processing
+
+### Entity Registry ✅
+Bloom filter-based deduplication:
+```typescript
+brain.use(new EntityRegistryAugmentation())
+// Handles millions of entities with minimal memory
+```
+
+### Neural Import ✅
+Intelligent data import with type inference:
+```typescript
+await brain.import({
+  source: 'data.json',
+  autoDetectTypes: true,
+  batchSize: 1000
+})
+```
+
+### Streaming Pipeline ✅
+Real-time data processing:
+```typescript
+brain.stream()
+  .pipe(transform)
+  .pipe(enrich)
+  .pipe(brain.writer())
+```
+
+## 📊 Analytics & Monitoring
+
+### Metrics Collection ✅
+Built-in performance metrics:
+```typescript
+const metrics = brain.getMetrics()
+// {
+//   operations: { add: 1000, find: 5000 },
+//   performance: { p95: 8, p99: 12 },
+//   cache: { hits: 4500, misses: 500 }
+// }
+```
+
+### Health Monitoring ✅
+Automatic health checks:
+```typescript
+const health = brain.getHealth()
+// {
+//   status: 'healthy',
+//   storage: 'connected',
+//   memory: { used: 245, limit: 512 }
+// }
+```
+
+## 🛠️ Developer Experience
+
+### Zero Configuration ✅
+Works out of the box:
+```typescript
+import Brainy from '@soulcraft/brainy'
+const brain = new Brainy()  // Auto-configures everything
+```
+
+### TypeScript First ✅
+Full type safety and inference:
+```typescript
+// Types are automatically inferred
+const results = await brain.find<MyType>('query')
+```
+
+### Augmentation System ✅
+Extensible plugin architecture:
+```typescript
+class CustomAugmentation extends BaseAugmentation {
+  execute(operation, params, next) {
+    // Your custom logic
+    return next()
+  }
+}
+```
+
+## 🔧 Operational Features
+
+### Graceful Shutdown ✅
+Clean shutdown with data persistence:
+```typescript
+process.on('SIGTERM', async () => {
+  await brain.shutdown()  // Saves all pending data
+})
+```
+
+### Hot Reload ✅
+Configuration updates without restart:
+```typescript
+brain.updateConfig({
+  cache: { enabled: false }
+})
+```
+
+### Backup & Restore ✅
+Full data backup capabilities:
+```typescript
+await brain.backup('backup.bin')
+await brain.restore('backup.bin')
+```
+
+## 📈 Proven at Scale
+
+- **10,000+ items**: Sub-10ms search
+- **1M+ operations**: Stable memory usage
+- **100+ concurrent users**: No performance degradation
+- **Multi-node clusters**: Automatic failover
+
+## 🚫 NOT Implemented (Planned)
+
+These features are documented but NOT yet implemented:
+- GraphQL API (use REST API instead)
+- Kubernetes operators (use Docker)
+- Some distributed features require manual configuration
+
+---
+
+*Last Updated: Latest Version*
+*All features listed above are production-ready and tested*

docs/guides/MIGRATING_TO_V5.11.md

@@ -1,230 +0,0 @@
-# Migrating to v5.11.1
-
-## Overview
-
-v5.11.1 introduces a **breaking change** with **massive performance benefits**:
-
-- `brain.get()` now loads **metadata-only by default** (76-81% faster!)
-- Vector embeddings require **explicit opt-in**: `{ includeVectors: true }`
-
-**Impact**: Only ~6% of codebases need changes (code that computes similarity on retrieved entities).
-
-## What Changed
-
-### Before (v5.11.0 and earlier)
-
-```typescript
-const entity = await brain.get(id)
-// entity.vector was ALWAYS loaded (384 dimensions, 6KB)
-console.log(entity.vector.length)  // 384
-```
-
-### After (v5.11.1)
-
-```typescript
-// DEFAULT: Metadata-only (76-81% faster)
-const entity = await brain.get(id)
-console.log(entity.vector)  // [] (empty array - not loaded)
-
-// EXPLICIT: Full entity with vectors
-const entity = await brain.get(id, { includeVectors: true })
-console.log(entity.vector.length)  // 384
-```
-
-## Who Needs to Update?
-
-### ✅ NO CHANGES NEEDED (94% of code)
-
-If you use `brain.get()` for:
-- **VFS operations** (readFile, stat, readdir)
-- **Existence checks**: `if (await brain.get(id))`
-- **Metadata access**: `entity.data`, `entity.type`, `entity.metadata`
-- **Relationship traversal**
-- **Admin tools**, import utilities, data APIs
-
-→ **Zero changes needed, automatic 76-81% speedup!**
-
-### ⚠️ REQUIRES UPDATE (~6% of code)
-
-If you use `brain.get()` AND then compute similarity on the returned entity:
-
-```typescript
-// ❌ BEFORE (v5.11.0) - will break in v5.11.1
-const entity = await brain.get(id)
-const similar = await brain.similar({ to: entity.vector })  // entity.vector is [] !
-
-// ✅ AFTER (v5.11.1) - add includeVectors
-const entity = await brain.get(id, { includeVectors: true })
-const similar = await brain.similar({ to: entity.vector })  // Works!
-```
-
-**Note**: `brain.similar({ to: entityId })` (using ID) still works - no changes needed!
-
-## Migration Steps
-
-### Step 1: Find Affected Code
-
-Search your codebase for patterns that use vectors from `brain.get()`:
-
-```bash
-# Find brain.get() calls that access .vector
-grep -r "await brain.get(" --include="*.ts" --include="*.js" | \
-  grep -E "(\.vector|entity\.vector)"
-```
-
-### Step 2: Update Pattern-by-Pattern
-
-#### Pattern 1: Similarity Using Retrieved Entity Vector
-
-```typescript
-// ❌ BEFORE
-const entity = await brain.get(id)
-const similar = await brain.similar({ to: entity.vector })
-
-// ✅ AFTER - Option A: Add includeVectors
-const entity = await brain.get(id, { includeVectors: true })
-const similar = await brain.similar({ to: entity.vector })
-
-// ✅ AFTER - Option B: Use ID directly (recommended)
-const similar = await brain.similar({ to: id })
-```
-
-#### Pattern 2: Manual Vector Operations
-
-```typescript
-// ❌ BEFORE
-const entity = await brain.get(id)
-const magnitude = Math.sqrt(entity.vector.reduce((sum, v) => sum + v*v, 0))
-
-// ✅ AFTER
-const entity = await brain.get(id, { includeVectors: true })
-const magnitude = Math.sqrt(entity.vector.reduce((sum, v) => sum + v*v, 0))
-```
-
-#### Pattern 3: Vector Assertions in Tests
-
-```typescript
-// ❌ BEFORE
-const entity = await brain.get(id)
-expect(entity.vector).toBeDefined()
-expect(entity.vector.length).toBe(384)
-
-// ✅ AFTER
-const entity = await brain.get(id, { includeVectors: true })
-expect(entity.vector).toBeDefined()
-expect(entity.vector.length).toBe(384)
-```
-
-### Step 3: Verify Migration
-
-Run your test suite to catch any remaining issues:
-
-```bash
-npm test
-```
-
-Look for errors like:
-- `entity.vector is empty` or `entity.vector.length is 0`
-- `Cannot compute similarity on empty vector`
-
-Add `{ includeVectors: true }` wherever these errors occur.
-
-## Performance Impact
-
-### Before Migration
-```
-brain.get(): 43ms, 6KB per call
-VFS readFile(): 53ms per file
-VFS readdir(100 files): 5.3s
-```
-
-### After Migration
-```
-brain.get(): 10ms, 300 bytes per call (76-81% faster) ✨
-brain.get({ includeVectors: true }): 43ms, 6KB (unchanged)
-VFS readFile(): ~13ms per file (75% faster) ✨
-VFS readdir(100 files): ~1.3s (75% faster) ✨
-```
-
-**Result**:
-- VFS operations: **75% faster**
-- Metadata access: **76-81% faster**
-- Vector similarity: **Unchanged** (still fast when needed)
-
-## TypeScript Support
-
-The new `GetOptions` interface is fully typed:
-
-```typescript
-interface GetOptions {
-  /**
-   * Include 384-dimensional vector embeddings in the response
-   *
-   * Default: false (metadata-only for 76-81% speedup)
-   */
-  includeVectors?: boolean
-}
-
-// TypeScript will autocomplete and validate
-const entity = await brain.get(id, { includeVectors: true })
-```
-
-## Rollback Plan
-
-If you encounter issues, you can temporarily force full entity loading everywhere:
-
-```typescript
-// Temporary wrapper (NOT RECOMMENDED - defeats optimization)
-async function getLegacy(id: string) {
-  return brain.get(id, { includeVectors: true })
-}
-
-// Use throughout codebase while migrating
-const entity = await getLegacy(id)
-```
-
-**Important**: This defeats the 76-81% performance improvement. Only use temporarily while fixing affected code.
-
-## FAQ
-
-### Q: Why did you make this a breaking change?
-
-**A**: The performance gains are massive (76-81% speedup, 95% less bandwidth) and affect 94% of code positively. Only ~6% of code needs updates. The net benefit is enormous.
-
-### Q: Do I need to update my VFS code?
-
-**A**: No! VFS automatically benefits from the optimization with zero code changes. Your VFS operations are now 75% faster automatically.
-
-### Q: Will brain.similar() still work?
-
-**A**: Yes! `brain.similar({ to: entityId })` works exactly as before. Only `brain.similar({ to: entity.vector })` requires the entity to be loaded with `{ includeVectors: true }`.
-
-### Q: What about backward compatibility?
-
-**A**: Entities returned without vectors have `vector: []` (empty array), which is type-safe. Code that doesn't use vectors continues to work. Only code that explicitly uses `entity.vector` needs updating.
-
-### Q: Can I check if vectors are loaded?
-
-**A**: Yes! Check `entity.vector.length > 0` to detect if vectors were loaded.
-
-```typescript
-const entity = await brain.get(id)
-if (entity.vector.length > 0) {
-  // Vectors are loaded
-} else {
-  // Metadata-only
-}
-```
-
-## Support
-
-If you encounter migration issues:
-
-1. Check the [VFS Performance Guide](../vfs/VFS_PERFORMANCE.md)
-2. Review [API Reference](../api/README.md)
-3. See [Performance Documentation](../PERFORMANCE.md)
-4. File an issue: https://github.com/soulcraft/brainy/issues
-
-## Changelog
-
-See [CHANGELOG.md](../../CHANGELOG.md) for complete v5.11.1 release notes.

docs/guides/aggregation.md

@@ -1,593 +0,0 @@
-# Aggregation Guide
-
-> Real-time analytics on your entity data with incremental running totals
-
-## Overview
-
-Brainy's aggregation engine computes running totals at write time, so reading aggregate results is always O(1) regardless of dataset size. Define an aggregate once, and every `add()`, `update()`, and `delete()` automatically updates the running metrics.
-
-No batch jobs. No scheduled recalculations. Aggregates stay current with every write.
-
-**Defining over existing data:** if you define an aggregate on a store that already holds
-matching entities, Brainy backfills it from those entities on the first query (a one-time scan,
-then purely incremental). So `defineAggregate()` behaves the same whether you define it before
-or after the data exists.
-
-**Reopening a persisted brain:** aggregate state persists across restarts. Re-defining the
-same aggregate at boot (the normal declarative pattern) adopts the persisted state directly —
-no rescan. A backfill scan runs only when the definition actually changed, when no persisted
-state exists, or when the state failed to load; and however many aggregates need backfilling,
-they share a single scan.
-
-## Quick Start
-
-```typescript
-import { Brainy, NounType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// 1. Define an aggregate
-brain.defineAggregate({
-  name: 'sales_by_category',
-  source: { type: NounType.Event },
-  groupBy: ['category'],
-  metrics: {
-    revenue: { op: 'sum', field: 'amount' },
-    count:   { op: 'count' },
-    average: { op: 'avg', field: 'amount' }
-  }
-})
-
-// 2. Add entities — aggregates update automatically
-await brain.add({
-  data: 'Coffee purchase',
-  type: NounType.Event,
-  metadata: { category: 'food', amount: 5.50 }
-})
-
-await brain.add({
-  data: 'Laptop purchase',
-  type: NounType.Event,
-  metadata: { category: 'electronics', amount: 1200 }
-})
-
-await brain.add({
-  data: 'Lunch purchase',
-  type: NounType.Event,
-  metadata: { category: 'food', amount: 12.00 }
-})
-
-// 3. Query results
-const results = await brain.find({ aggregate: 'sales_by_category' })
-
-// Results:
-// [
-//   { groupKey: { category: 'food' }, metrics: { revenue: 17.50, count: 2, average: 8.75 } },
-//   { groupKey: { category: 'electronics' }, metrics: { revenue: 1200, count: 1, average: 1200 } }
-// ]
-```
-
-## Aggregation Operations
-
-Brainy supports 7 aggregation operations:
-
-### `sum` — Running Total
-
-Adds up all values of a numeric field.
-
-```typescript
-metrics: {
-  total_revenue: { op: 'sum', field: 'amount' }
-}
-```
-
-### `count` — Entity Count
-
-Counts the number of matching entities. No `field` required.
-
-```typescript
-metrics: {
-  order_count: { op: 'count' }
-}
-```
-
-### `avg` — Running Average
-
-Computes `sum / count` incrementally.
-
-```typescript
-metrics: {
-  average_price: { op: 'avg', field: 'price' }
-}
-```
-
-### `min` — Minimum Value
-
-Tracks the minimum value across all entities in each group.
-
-```typescript
-metrics: {
-  lowest_price: { op: 'min', field: 'price' }
-}
-```
-
-### `max` — Maximum Value
-
-Tracks the maximum value across all entities in each group.
-
-```typescript
-metrics: {
-  highest_price: { op: 'max', field: 'price' }
-}
-```
-
-### `stddev` — Sample Standard Deviation
-
-Computes the sample standard deviation using Welford's numerically stable online algorithm. Updates incrementally without storing individual values.
-
-```typescript
-metrics: {
-  price_spread: { op: 'stddev', field: 'price' }
-}
-```
-
-### `variance` — Sample Variance
-
-Computes the sample variance (square of standard deviation) using Welford's online algorithm.
-
-```typescript
-metrics: {
-  price_variance: { op: 'variance', field: 'price' }
-}
-```
-
-## GROUP BY Dimensions
-
-Every aggregate requires at least one `groupBy` dimension. Results are grouped by the unique combinations of dimension values.
-
-### Plain Fields
-
-Group by a metadata field value:
-
-```typescript
-groupBy: ['category']
-// Produces groups: { category: 'food' }, { category: 'electronics' }, ...
-```
-
-### Multiple Fields
-
-Group by multiple fields for composite keys:
-
-```typescript
-groupBy: ['category', 'region']
-// Produces groups: { category: 'food', region: 'US' }, { category: 'food', region: 'EU' }, ...
-```
-
-### Time Windows
-
-Group by a timestamp field bucketed into time periods:
-
-```typescript
-groupBy: [{ field: 'date', window: 'month' }]
-// Produces groups: { date: '2024-01' }, { date: '2024-02' }, ...
-```
-
-Available time window granularities:
-
-| Window | Format | Example |
-|--------|--------|---------|
-| `hour` | `YYYY-MM-DDThh` | `2024-01-15T14` |
-| `day` | `YYYY-MM-DD` | `2024-01-15` |
-| `week` | `YYYY-Wnn` | `2024-W03` |
-| `month` | `YYYY-MM` | `2024-01` |
-| `quarter` | `YYYY-Qn` | `2024-Q1` |
-| `year` | `YYYY` | `2024` |
-| `{ seconds: N }` | ISO 8601 | Custom interval |
-
-### Combined Dimensions
-
-Mix plain fields and time windows:
-
-```typescript
-brain.defineAggregate({
-  name: 'monthly_sales',
-  source: { type: NounType.Event },
-  groupBy: ['region', { field: 'date', window: 'month' }],
-  metrics: {
-    revenue: { op: 'sum', field: 'amount' },
-    count: { op: 'count' }
-  }
-})
-
-// Produces groups like:
-// { region: 'US', date: '2024-01' }
-// { region: 'US', date: '2024-02' }
-// { region: 'EU', date: '2024-01' }
-```
-
-### Array Fields (Unnest)
-
-Group by **each element** of an array-valued field — for tag frequencies, label counts, and
-faceted breakdowns. Mark the dimension `{ field, unnest: true }`:
-
-```typescript
-brain.defineAggregate({
-  name: 'tag_frequency',
-  source: { type: NounType.Document },
-  groupBy: [{ field: 'tags', unnest: true }],
-  metrics: { count: { op: 'count' } }
-})
-
-// A document tagged ['ml', 'ai'] contributes once to the 'ml' group and once to 'ai'.
-// Duplicate tags on one entity count once; an entity with no tags joins no group.
-const top = await brain.queryAggregate('tag_frequency', { orderBy: 'count', order: 'desc' })
-// [ { groupKey: { tags: 'ai' }, metrics: { count: 3 }, count: 3 }, ... ]
-```
-
-## Querying Aggregates
-
-Aggregate results are queried through the standard `find()` method.
-
-### Basic Query
-
-```typescript
-const results = await brain.find({ aggregate: 'sales_by_category' })
-```
-
-### Filter by Group Key
-
-Use `where` to filter on group key values:
-
-```typescript
-const foodOnly = await brain.find({
-  aggregate: 'sales_by_category',
-  where: { category: 'food' }
-})
-```
-
-### Filter by Metric Value (HAVING)
-
-Use `having` to filter groups by their **computed metric values** — the analytics equivalent of
-SQL `HAVING`. (`where` filters group *keys*; `having` filters *metrics*.)
-
-```typescript
-const bigCategories = await brain.find({
-  aggregate: 'sales_by_category',
-  having: { revenue: { greaterThan: 1000 } }
-})
-```
-
-`having` accepts the same operators as `where`, applied to each group's metric results plus
-`count`. It is evaluated per group — **O(groups), independent of entity count** — before sorting
-and pagination, so it stays cheap even over billions of entities.
-
-### Sort and Paginate
-
-Sort by any metric or group key field:
-
-```typescript
-const topCategories = await brain.find({
-  aggregate: {
-    name: 'sales_by_category',
-    orderBy: 'revenue',
-    order: 'desc',
-    limit: 10
-  }
-})
-```
-
-### Combined Parameters
-
-`where`, `orderBy`, `limit`, and `offset` from the outer `find()` call merge automatically with the aggregate query:
-
-```typescript
-const recentTopSpenders = await brain.find({
-  aggregate: 'monthly_sales',
-  where: { region: 'US' },
-  orderBy: 'revenue',
-  order: 'desc',
-  limit: 12,
-  offset: 0
-})
-```
-
-### Result Format
-
-`find({ aggregate })` returns `Result<T>` rows (for uniformity with the rest of `find()`),
-with the aggregate fields surfaced **both** at the top level and, for backward compatibility,
-flattened into `metadata`:
-
-```typescript
-{
-  id: string,
-  score: 1.0,
-  type: NounType.Measurement,
-  groupKey: { category: 'food' },             // top-level — the group key values
-  metrics:  { revenue: 17.50, count: 2, average: 8.75 }, // top-level — computed metrics
-  count: 2,                                   // top-level — entities in the group
-  metadata: {                                 // legacy mirror of the same data
-    __aggregate: 'sales_by_category',
-    category: 'food',
-    revenue: 17.50, count: 2, average: 8.75
-  },
-  entity: Entity
-}
-```
-
-### `queryAggregate()` — the report-friendly view
-
-For dashboards and reports, prefer `brain.queryAggregate(name, params)`. It returns the clean
-`AggregateResult[]` shape directly — no search-result wrapper:
-
-```typescript
-const rows = await brain.queryAggregate('sales_by_category', {
-  orderBy: 'revenue',
-  order: 'desc',
-  limit: 10
-})
-// [
-//   { groupKey: { category: 'electronics' }, metrics: { revenue: 1200, count: 1, average: 1200 }, count: 1 },
-//   { groupKey: { category: 'food' }, metrics: { revenue: 17.50, count: 2, average: 8.75 }, count: 2 }
-// ]
-```
-
-It accepts the same `where` / `having` / `orderBy` / `order` / `limit` / `offset` params as the
-`find({ aggregate })` form.
-
-## Source Filtering
-
-Control which entities feed into an aggregate with the `source` property.
-
-### Filter by Entity Type
-
-```typescript
-brain.defineAggregate({
-  name: 'event_stats',
-  source: { type: NounType.Event },
-  groupBy: ['category'],
-  metrics: { count: { op: 'count' } }
-})
-```
-
-### Filter by Multiple Types
-
-```typescript
-source: { type: [NounType.Event, NounType.Document] }
-```
-
-### Filter by Metadata
-
-Use the same `where` syntax as `find()`:
-
-```typescript
-source: {
-  type: NounType.Event,
-  where: { domain: 'financial', subtype: 'transaction' }
-}
-```
-
-### Filter by Service
-
-For multi-tenant deployments:
-
-```typescript
-source: { service: 'tenant-123' }
-```
-
-Entities that don't match the source filter are silently skipped during incremental updates.
-
-## Incremental Updates
-
-The aggregation engine hooks into every write operation:
-
-### On `add()`
-
-When a new entity matches an aggregate's source filter:
-1. The group key is computed from the entity's metadata
-2. Each metric in the matching group is incremented
-3. New groups are created automatically
-
-### On `update()`
-
-When an existing entity is updated:
-1. The old entity's contribution is reversed from its group
-2. The new entity's contribution is applied to its (potentially different) group
-3. Handles group key changes — an entity moving from category "food" to "drink" updates both groups
-
-### On `delete()`
-
-When an entity is deleted:
-1. The entity's contribution is reversed from its group
-2. If a group becomes empty (all metric counts reach zero), it's removed
-
-### Aggregate Entity Exclusion
-
-Materialized `NounType.Measurement` entities are automatically excluded from all source matching, preventing infinite feedback loops. Entities with `service: 'brainy:aggregation'` or `metadata.__aggregate` are always skipped.
-
-## Materialization
-
-Materialization writes aggregate results as `NounType.Measurement` entities, making them automatically available through OData, Google Sheets, SSE, and webhook integrations.
-
-```typescript
-brain.defineAggregate({
-  name: 'daily_metrics',
-  source: { type: NounType.Event },
-  groupBy: [{ field: 'date', window: 'day' }],
-  metrics: {
-    total: { op: 'sum', field: 'amount' },
-    count: { op: 'count' }
-  },
-  materialize: true
-})
-```
-
-### Debounce Configuration
-
-During high-throughput ingestion, materialization is debounced to avoid excessive writes:
-
-```typescript
-materialize: {
-  debounceMs: 2000,      // Wait 2 seconds after last update before writing
-  trackSources: true     // Track which entities contributed
-}
-```
-
-The default debounce interval is 1000ms.
-
-## Multiple Aggregates
-
-Define multiple aggregates that process the same entities:
-
-```typescript
-// Revenue by category
-brain.defineAggregate({
-  name: 'category_revenue',
-  source: { type: NounType.Event },
-  groupBy: ['category'],
-  metrics: { total: { op: 'sum', field: 'amount' } }
-})
-
-// Monthly trends
-brain.defineAggregate({
-  name: 'monthly_trends',
-  source: { type: NounType.Event },
-  groupBy: [{ field: 'date', window: 'month' }],
-  metrics: {
-    revenue: { op: 'sum', field: 'amount' },
-    count: { op: 'count' },
-    avg_order: { op: 'avg', field: 'amount' }
-  }
-})
-
-// Regional breakdown with statistical analysis
-brain.defineAggregate({
-  name: 'regional_analysis',
-  source: { type: NounType.Event },
-  groupBy: ['region'],
-  metrics: {
-    revenue: { op: 'sum', field: 'amount' },
-    spread: { op: 'stddev', field: 'amount' },
-    variance: { op: 'variance', field: 'amount' }
-  }
-})
-```
-
-Each `add()` call updates all matching aggregates automatically.
-
-## Removing Aggregates
-
-Remove an aggregate and clean up its state:
-
-```typescript
-brain.removeAggregate('category_revenue')
-```
-
-## Persistence
-
-Aggregate definitions and running state are automatically persisted:
-
-- **On `flush()`/`close()`**: All dirty aggregate state is written to storage
-- **On `init()`**: Definitions and state are restored from storage
-- **Change detection**: Definition changes are detected via FNV-1a hashing — only changed aggregates reset their state on restart
-
-## Native Acceleration
-
-When [Cor](https://www.npmjs.com/package/@soulcraft/cor) is installed as a plugin, the aggregation engine automatically uses Rust-accelerated computation:
-
-- Incremental updates run in Rust with BTreeMap-backed precise MIN/MAX
-- Welford's online stddev/variance computed natively
-- Rebuild uses Rayon parallel iterators across CPU cores (above 1,000 entities)
-- Time window bucketing uses integer arithmetic without `Date` object allocation
-
-```typescript
-const brain = new Brainy({
-  plugins: ['@soulcraft/cor']
-})
-await brain.init()
-
-// Aggregation automatically uses native engine
-brain.defineAggregate({ ... })
-```
-
-Verify native acceleration is active:
-
-```typescript
-const diag = brain.diagnostics()
-console.log(diag.providers.aggregation)
-// { source: 'plugin' }
-```
-
-## Common Patterns
-
-### Financial Analytics
-
-```typescript
-brain.defineAggregate({
-  name: 'monthly_spending',
-  source: {
-    type: NounType.Event,
-    where: { domain: 'financial', subtype: 'transaction' }
-  },
-  groupBy: [
-    'category',
-    { field: 'date', window: 'month' }
-  ],
-  metrics: {
-    total:   { op: 'sum', field: 'amount' },
-    count:   { op: 'count' },
-    average: { op: 'avg', field: 'amount' },
-    highest: { op: 'max', field: 'amount' },
-    lowest:  { op: 'min', field: 'amount' }
-  },
-  materialize: true
-})
-```
-
-### Time-Series Monitoring
-
-```typescript
-brain.defineAggregate({
-  name: 'hourly_metrics',
-  source: { type: NounType.Event, where: { domain: 'monitoring' } },
-  groupBy: [
-    'service',
-    { field: 'timestamp', window: 'hour' }
-  ],
-  metrics: {
-    request_count:  { op: 'count' },
-    avg_latency:    { op: 'avg', field: 'latency_ms' },
-    max_latency:    { op: 'max', field: 'latency_ms' },
-    error_count:    { op: 'sum', field: 'is_error' },
-    latency_spread: { op: 'stddev', field: 'latency_ms' }
-  }
-})
-```
-
-### Content Analytics
-
-```typescript
-brain.defineAggregate({
-  name: 'content_stats',
-  source: { type: NounType.Document },
-  groupBy: ['author', { field: 'publishedAt', window: 'month' }],
-  metrics: {
-    articles:   { op: 'count' },
-    total_words: { op: 'sum', field: 'wordCount' },
-    avg_words:   { op: 'avg', field: 'wordCount' }
-  }
-})
-```
-
-## Performance
-
-Aggregation complexity per write is O(A x G x M) where A = matching aggregates, G = groupBy dimensions, M = metrics. For typical configurations (2-5 aggregates, 1-3 dimensions, 3-5 metrics), this is effectively O(1).
-
-With Cor native acceleration:
-
-| Operation | Throughput | Latency |
-|-----------|-----------|---------|
-| Incremental update (1K entities) | 809 ops/s | 1.2 ms |
-| Rebuild (10K entities) | 475 ops/s | 2.1 ms |
-| Rebuild (100K entities, Rayon) | 66 ops/s | 15.2 ms |
-| Query (1K groups, sort + paginate) | 986 ops/s | 1.0 ms |

docs/guides/distributed-system.md

@@ -0,0 +1,406 @@
+# Distributed Brainy System Guide
+
+## Overview
+
+Brainy introduces a groundbreaking **zero-configuration distributed system** that transforms how vector databases scale. Unlike traditional distributed databases that require complex setup (Consul, etcd, Zookeeper), Brainy uses your existing storage (S3, GCS, R2) as the coordination layer.
+
+## Key Innovation: Storage-Based Coordination
+
+Instead of requiring separate infrastructure for coordination, Brainy leverages your storage backend:
+
+```typescript
+// Traditional distributed database setup:
+// ❌ Setup Consul/etcd
+// ❌ Configure node discovery
+// ❌ Setup health checks  
+// ❌ Configure sharding
+// ❌ Setup replication
+
+// Brainy distributed setup:
+const brain = new Brainy({
+  storage: { 
+    type: 's3',
+    options: { bucket: 'my-data' }
+  },
+  distributed: true  // ✅ That's it!
+})
+```
+
+## Real-World Scenarios
+
+### 1. Processing Multiple Streaming Data Sources (Bluesky, Twitter, etc.)
+
+**Problem**: You need to ingest millions of posts from multiple social media firehoses simultaneously while maintaining search performance.
+
+**Traditional Approach**:
+- Separate ingestion and search clusters
+- Complex queue systems (Kafka, RabbitMQ)
+- Manual sharding configuration
+- Complicated backpressure handling
+
+**Brainy Solution**:
+```typescript
+// Node 1: Bluesky Ingestion
+const ingestionNode1 = new Brainy({
+  storage: { type: 's3', options: { bucket: 'social-data' }},
+  distributed: true,
+  writeOnly: true  // Optimized for writes
+})
+
+// Continuously ingest Bluesky firehose
+blueskyStream.on('post', async (post) => {
+  await ingestionNode1.add({
+    content: post.text,
+    author: post.author,
+    timestamp: post.createdAt,
+    platform: 'bluesky'
+  }, 'social-post')
+})
+
+// Node 2: Twitter Ingestion (separate machine)
+const ingestionNode2 = new Brainy({
+  storage: { type: 's3', options: { bucket: 'social-data' }},
+  distributed: true,
+  writeOnly: true
+})
+
+// Node 3-5: Search nodes (auto-balanced)
+const searchNode = new Brainy({
+  storage: { type: 's3', options: { bucket: 'social-data' }},
+  distributed: true,
+  readOnly: true  // Optimized for queries
+})
+
+// Search across ALL data from ALL sources
+const results = await searchNode.find('AI trends', 100)
+// Automatically queries all shards across all nodes!
+```
+
+**Benefits**:
+- **Auto-sharding**: Data automatically distributed by content hash
+- **No bottlenecks**: Each ingestion node writes directly to storage
+- **Live search**: Search nodes see new data immediately
+- **Auto-scaling**: Add nodes anytime, data rebalances automatically
+
+### 2. Multi-Tenant SaaS Application
+
+**Problem**: Each customer needs isolated, fast search across their documents, with ability to scale per customer.
+
+**Brainy Solution**:
+```typescript
+// Spin up dedicated nodes per large customer
+const enterpriseNode = new Brainy({
+  storage: { 
+    type: 's3', 
+    options: { 
+      bucket: 'customer-data',
+      prefix: 'customer-123/'  // Isolated data
+    }
+  },
+  distributed: true
+})
+
+// Shared nodes for smaller customers
+const sharedNode = new Brainy({
+  storage: { 
+    type: 's3', 
+    options: { bucket: 'shared-customers' }
+  },
+  distributed: true
+})
+
+// Domain-based sharding ensures customer data stays together
+await sharedNode.add(document, 'document', {
+  customerId: 'cust-456',  // Used for shard assignment
+  domain: 'customer-456'    // Keeps related data together
+})
+```
+
+### 3. Global Knowledge Graph with Local Inference
+
+**Problem**: Building a knowledge graph that needs both global connectivity and local LLM inference.
+
+**Brainy Solution**:
+```typescript
+// Edge nodes near users (with GPU)
+const edgeNode = new Brainy({
+  storage: { type: 's3', options: { bucket: 'knowledge-graph' }},
+  distributed: true,
+  models: {
+    embed: { model: 'BAAI/bge-base-en-v1.5' },
+    chat: { model: 'meta-llama/Llama-3.2-3B-Instruct' }
+  }
+})
+
+// Central nodes for graph operations
+const graphNode = new Brainy({
+  storage: { type: 's3', options: { bucket: 'knowledge-graph' }},
+  distributed: true,
+  augmentations: ['graph', 'triple-intelligence']
+})
+
+// Local inference with global knowledge
+const context = await edgeNode.find(userQuery, 10)
+const response = await edgeNode.chat(userQuery, { context })
+
+// Graph traversal across all nodes
+const connections = await graphNode.traverse({
+  start: 'concept:AI',
+  depth: 3,
+  relationship: 'related-to'
+})
+```
+
+## Competitive Advantages
+
+### vs. Pinecone/Weaviate/Qdrant
+
+| Feature | Traditional Vector DBs | Brainy Distributed |
+|---------|----------------------|-------------------|
+| Setup Complexity | High (k8s, operators) | Zero (just storage) |
+| Minimum Nodes | 3-5 for HA | 1 (scale as needed) |
+| Coordination | External (etcd, Consul) | Built-in (via storage) |
+| Data Locality | Random sharding | Domain-aware sharding |
+| Query Planning | Basic | Triple Intelligence |
+| Cost at Scale | High (always-on clusters) | Low (scale to zero) |
+
+### vs. Neo4j/ArangoDB (Graph Databases)
+
+| Feature | Graph Databases | Brainy Distributed |
+|---------|----------------|-------------------|
+| Vector Search | Bolt-on/Limited | Native HNSW |
+| Embedding Generation | External | Built-in (30+ models) |
+| Distributed Transactions | Complex/Slow | Eventually consistent |
+| Natural Language | No | Native (Triple Intelligence) |
+| Setup | Very Complex | Zero config |
+
+### vs. Elasticsearch/OpenSearch
+
+| Feature | Elasticsearch | Brainy Distributed |
+|---------|--------------|-------------------|
+| Vector Support | Added later (slow) | Native (fast HNSW) |
+| Cluster Management | Complex (master nodes) | Automatic (via storage) |
+| Shard Rebalancing | Manual/Risky | Automatic/Safe |
+| Memory Usage | Very High | Efficient |
+| Query Language | Complex DSL | Natural language |
+
+## Innovative Features
+
+### 1. Domain-Aware Sharding
+
+Unlike hash-based sharding, Brainy understands data relationships:
+
+```typescript
+// Documents about the same topic stay on the same shard
+await brain.add(doc1, 'document', { domain: 'physics' })
+await brain.add(doc2, 'document', { domain: 'physics' })
+// Both documents on same shard = faster related queries
+
+// Customer data stays together
+await brain.add(order, 'order', { customerId: 'cust-123' })
+await brain.add(invoice, 'invoice', { customerId: 'cust-123' })
+// Same customer = same shard = better locality
+```
+
+### 2. Streaming Shard Migration
+
+Zero-downtime data movement between nodes:
+
+```typescript
+// Automatically triggered when nodes join/leave
+// Uses HTTP streaming for efficiency
+// Validates data integrity
+// Atomic ownership transfer
+// No query downtime!
+```
+
+### 3. Storage-Based Consensus
+
+No Raft/Paxos complexity:
+
+```typescript
+// Leader election via storage atomic operations
+// Health monitoring via storage heartbeats
+// Configuration consensus via storage CAS
+// No split-brain issues!
+```
+
+### 4. Intelligent Query Planning
+
+The distributed query planner understands:
+- Which shards contain relevant data
+- Node health and latency
+- Data locality and caching
+- Triple Intelligence scoring
+
+```typescript
+// Automatically optimizes query execution
+const results = await brain.find('quantum physics')
+// Planner knows:
+// - Physics domain → shard-3
+// - Node-2 has shard-3 cached
+// - Route query to node-2
+// - Merge results with Triple Intelligence
+```
+
+## Performance Characteristics
+
+### Scalability
+- **Horizontal**: Add nodes anytime
+- **Vertical**: Nodes can differ in size
+- **Geographic**: Nodes can be globally distributed
+- **Elastic**: Scale to zero when idle
+
+### Throughput
+- **Writes**: Linear scaling with nodes
+- **Reads**: Sub-linear (due to caching)
+- **Mixed**: Read/write optimized nodes
+
+### Latency
+- **Local queries**: ~10ms p50
+- **Distributed queries**: ~50ms p50
+- **Shard migration**: Streaming (no bulk pause)
+
+## Use Cases Where Brainy Excels
+
+### ✅ Perfect For:
+
+1. **Multi-source data ingestion** (social media, logs, events)
+2. **Global search applications** (distributed teams)
+3. **Multi-tenant SaaS** (customer isolation)
+4. **Knowledge graphs** (with vector search)
+5. **Edge AI applications** (local inference, global knowledge)
+6. **Document intelligence** (contracts, research papers)
+7. **Real-time analytics** (streaming + search)
+
+### ⚠️ Consider Alternatives For:
+
+1. **Strong consistency requirements** (use PostgreSQL)
+2. **Sub-millisecond latency** (use Redis)
+3. **Complex transactions** (use traditional RDBMS)
+4. **Purely structured data** (use columnar stores)
+
+## Migration from Other Systems
+
+### From Pinecone/Weaviate:
+```typescript
+// Your existing vector search still works
+const results = await brain.search(embedding, 10)
+
+// But now you can scale horizontally!
+// And add graph relationships!
+// And use natural language!
+```
+
+### From Elasticsearch:
+```typescript
+// Import your documents
+await brain.import('./elasticsearch-export.json')
+
+// Queries are simpler
+const results = await brain.find('user query')
+// No complex DSL needed!
+```
+
+### From Neo4j:
+```typescript
+// Import your graph
+await brain.importGraph('./neo4j-export.cypher')
+
+// Now with vector search!
+const similar = await brain.find('concepts like quantum computing')
+```
+
+## Deployment Patterns
+
+### 1. Start Simple, Scale Later
+```typescript
+// Day 1: Single node
+const brain = new Brainy({ storage: 's3' })
+
+// Month 2: Growing data, add distribution
+const brain = new Brainy({ 
+  storage: 's3',
+  distributed: true  // Just add this!
+})
+
+// Month 6: Multiple nodes auto-balance
+// No migration needed!
+```
+
+### 2. Geographic Distribution
+```typescript
+// US Node
+const usNode = new Brainy({
+  storage: { region: 'us-east-1' },
+  distributed: true
+})
+
+// EU Node  
+const euNode = new Brainy({
+  storage: { region: 'eu-west-1' },
+  distributed: true
+})
+
+// They automatically coordinate!
+```
+
+### 3. Specialized Nodes
+```typescript
+// GPU nodes for embedding
+const embedNode = new Brainy({
+  distributed: true,
+  writeOnly: true,
+  models: { embed: 'large-model' }
+})
+
+// CPU nodes for search
+const searchNode = new Brainy({
+  distributed: true,
+  readOnly: true
+})
+```
+
+## Monitoring & Operations
+
+### Health Checks
+```typescript
+const health = await brain.getClusterHealth()
+// {
+//   nodes: 5,
+//   healthy: 5,
+//   shards: 16,
+//   status: 'green'
+// }
+```
+
+### Shard Distribution
+```typescript
+const shards = await brain.getShardDistribution()
+// Shows which nodes own which shards
+```
+
+### Migration Status
+```typescript
+const migrations = await brain.getActiveMigrations()
+// Shows ongoing shard movements
+```
+
+## Conclusion
+
+Brainy's distributed system is **production-ready** and offers:
+
+1. **True zero-configuration** - Just add `distributed: true`
+2. **Storage-based coordination** - No external dependencies
+3. **Intelligent sharding** - Domain-aware data placement
+4. **Automatic operations** - Rebalancing, failover, scaling
+5. **Unified interface** - Vector + Graph + Document + LLM
+
+This is not just another distributed database. It's a fundamental rethinking of how distributed systems should work in the cloud era.
+
+## Next Steps
+
+1. [Try the distributed quick start](./distributed-quickstart.md)
+2. [Read the architecture deep dive](../architecture/distributed-storage.md)
+3. [View benchmarks](../benchmarks/distributed-performance.md)
+4. [Deploy to production](./production-deployment.md)

docs/guides/enterprise-for-everyone.md

@@ -165,28 +165,32 @@ await brain.syncWith({
 - **Webhook support**: React to changes
 - **API generation**: Auto-generate REST/GraphQL APIs
 
-### 🌍 Scale
+### 🌍 Enterprise Scale 🚧 Coming Soon
 
-**Everyone gets the same scale model:**
+**Everyone gets planetary scale:**
 
 ```typescript
-// Pure JS by default; install the optional native provider for billions of vectors
-const brain = new Brainy()
+// Same architecture Netflix uses, free for you
+const brain = new Brainy({
+  clustering: {
+    enabled: true,                   // Distributed mode
+    sharding: 'automatic',           // Auto-sharding
+    replication: 3,                  // Triple replication
+    consensus: 'raft',              // Strong consistency
+    geoDistribution: true            // Multi-region support
+  }
+})
 
-// 1 → ~1M vectors: pure-JS HNSW, zero extra setup
-// 1M → 10B+ vectors: install @soulcraft/cor for the native DiskANN provider
+// Handles everything from 1 to 1 billion entities
 ```
 
-**Scaling model:**
-- **Single process, no cluster**: Brainy runs in one process — no coordinator,
-  no peer discovery, no consensus to operate
-- **Optional native provider**: install `@soulcraft/cor` to back the index with
-  on-disk DiskANN that scales to 10B+ vectors on one machine
-- **Per-tenant pools**: isolate tenants by giving each its own Brainy instance and
-  storage directory
-- **Horizontal read scaling**: run many reader processes against one shared on-disk
-  store (single writer, many readers); replicate the artifact with your operator
-  tooling
+**Scaling features:**
+- **Horizontal scaling**: Add nodes as needed
+- **Auto-sharding**: Distributes data automatically
+- **Multi-region**: Global distribution
+- **Load balancing**: Automatic request distribution
+- **Zero-downtime upgrades**: Rolling updates
+- **Infinite scale**: No upper limits
 
 ### 🛡️ Enterprise Compliance 🚧 Coming Soon
 
@@ -438,4 +442,4 @@ Brainy is more than software—it's a movement to democratize enterprise technol
 - [Zero Configuration](../architecture/zero-config.md)
 - [Augmentations System](../architecture/augmentations.md)
 - [Architecture Overview](../architecture/overview.md)
-- [API Reference](../api/README.md)
+- [Getting Started](./getting-started.md)

docs/guides/export-and-import.md

@@ -1,181 +0,0 @@
----
-title: Export & Import (portable graph)
-slug: guides/export-and-import
-public: true
-category: guides
-template: guide
-order: 9
-description: Export part or all of a brain to a portable, versioned PortableGraph document and import it back — by id, collection, connected neighbourhood, VFS subtree, predicate, or the whole brain. export() lives on the immutable Db, so asOf()/with() give time-travel and what-if exports.
-next:
-  - guides/subtypes-and-facets
-  - api/README
----
-
-# Export & Import (portable graph)
-
-Brainy serializes part or all of a brain — an item, a collection, a connected
-neighbourhood, a VFS subtree, a predicate match, or the whole brain — into a single
-versioned JSON document (`PortableGraph`), and restores it.
-
-```typescript
-const graph = await brain.export()   // whole brain → PortableGraph
-await brain.import(graph)             // restore (merge by id, re-embed if no vectors)
-```
-
-It is **portable** (human-readable JSON), **versioned** (`formatVersion`, so a document
-written by 7.x imports cleanly into 8.0), and **current-state** (the entities and edges as
-they are now — no generation history). Use it for portable artifacts, partial exports,
-cross-environment moves, and version upgrades.
-
-`export()` is a method on the **immutable `Db` value**, so it composes with every way of
-obtaining one:
-
-```typescript
-brain.export(sel)                                   // = brain.now().export(sel)
-;(await brain.asOf(gen)).export(sel)                // time-travel export (a past generation)
-brain.now().with(ops).export(sel)                   // what-if export (a speculative state)
-```
-
-## When to use which
-
-| You want… | Use |
-|-----------|-----|
-| A portable, partial-or-whole, cross-version graph document | **`brain.export()` / `brain.import()`** (this guide) |
-| A whole-brain snapshot **with generation history** | `brain.now().persist(path)` / `Brainy.load(path)` (native) |
-| To ingest a CSV / PDF / Excel / JSON **file** as new entities | `brain.import(file)` — see [Import Anything](./import-anything.md) |
-
-`import()` is **polymorphic**: hand it a `PortableGraph` and it does the graph round-trip;
-hand it a file/buffer and it does foreign-file ingestion (dispatched on the document's
-`format: 'brainy-portable-graph'` tag).
-
-## Exporting
-
-```typescript
-brain.export(selector?, options?): Promise<PortableGraph>
-// (also on any Db: brain.now().export(...), (await brain.asOf(g)).export(...))
-```
-
-### Selectors — *what* to export
-
-Omit the selector to export the whole brain. Otherwise pick a node set:
-
-| Scenario | Selector |
-|----------|----------|
-| Just an item (or items) | `{ ids: ['a', 'b'] }` |
-| A collection + its children | `{ collection: collectionId }` (alias `memberOf`) |
-| A connected neighbourhood | `{ connected: { from: id, depth: 2, verbs?, direction? } }` |
-| A VFS directory / file (+ subtree) | `{ vfsPath: '/docs', recursive?: true }` |
-| Everything matching a predicate | `{ type, subtype, where, service, visibility }` |
-| The whole brain | *(omit)* |
-
-The selector reuses `find()`'s grammar — *"export what `find()` would match, minus ranking
-and limit."* Structural and predicate selectors **compose**:
-
-```typescript
-// Members of a collection whose status is "open"
-await brain.export({ collection: collectionId, where: { status: 'open' } })
-
-// Already have find() results? Export exactly those with the ids selector
-const hits = await brain.find({ type: NounType.Document })
-await brain.export({ ids: hits.map(r => r.id) })
-```
-
-### Options — *how* to serialize
-
-| Option | Default | Effect |
-|--------|---------|--------|
-| `includeVectors` | `false` | Carry embedding vectors verbatim. Off ⇒ `import()` re-embeds from `data`. |
-| `includeContent` | `false` | Include VFS file bytes in `blobs` so files round-trip byte-identically. |
-| `includeSystem` | `false` | Include `visibility:'system'` entities such as the VFS root. |
-| `edges` | `'induced'` | `'induced'` (both endpoints in the set), `'incident'` (also dangling edges, recorded in `danglingIds`), or `'none'` (nodes only). |
-
-## Importing
-
-```typescript
-brain.import(graph, options?): Promise<ImportResult>
-```
-
-The whole graph is applied as **one atomic transaction** — it advances the brain exactly
-one generation, or none on failure.
-
-```typescript
-const result = await brain.import(graph, { onConflict: 'merge' })
-// → { imported, merged, skipped, reembedded, blobsWritten, errors }
-```
-
-| Option | Default | Effect |
-|--------|---------|--------|
-| `onConflict` | `'merge'` | `'merge'` (update existing id in place — assemble many exported graphs), `'replace'` (delete + recreate), or `'skip'`. |
-| `reembed` | `'auto'` | `'auto'` (use the carried vector, else re-embed from `data`) or `'never'` (require a carried vector; record an error if absent). |
-| `remapIds` | — | Rewrite every id on the way in, e.g. to clone a template subgraph under fresh ids. |
-| `meta` | — | Transaction metadata recorded in the tx-log alongside the new generation. |
-
-The default `onConflict: 'merge'` lets you assemble one working graph from many exported
-documents that share entity ids — re-importing an id merges rather than duplicates.
-
-## The `PortableGraph` format
-
-```jsonc
-{
-  "format": "brainy-portable-graph",  // identifies the document type
-  "formatVersion": 1,                 // import gates on this (cross-version migration)
-  "brainyVersion": "8.0.0",
-  "createdAt": "2026-06-16T…Z",
-  "embedding": { "model": "all-MiniLM-L6-v2", "dimensions": 384 },
-  "selector": { … },                  // echoes what was exported (provenance)
-  "entities": [
-    {
-      "id": "…", "type": "Document", "subtype": "invoice", "visibility": "public",
-      "data": "…",                    // the embedding source
-      "confidence": 1, "weight": 1, "service": "…",
-      "vector": [ … ],                // only with includeVectors
-      "metadata": { … }               // custom fields only (reserved fields are top-level)
-    }
-  ],
-  "relations": [
-    { "id":"…", "from":"…", "to":"…", "type":"Contains", "subtype":"…",
-      "weight":1, "confidence":1, "metadata": { … } }
-  ],
-  "blobs": { "<sha256>": "<base64>" }, // only with includeContent
-  "danglingIds": [ "…" ],             // only with edges:'incident'
-  "stats": { "entityCount": 0, "relationCount": 0, "blobCount": 0, "vectorDimensions": 384 }
-}
-```
-
-Standard fields (`subtype`, `visibility`, `data`, `confidence`, `weight`, `service`) sit at
-the top level of each entity; `metadata` holds **only** custom user fields — mirroring the
-in-memory `Entity` shape, so `import()` maps each field to its dedicated parameter. The
-TypeScript types (`PortableGraph`, `PortableGraphEntity`, `PortableGraphRelation`, `ExportSelector`,
-`ExportOptions`, `ImportOptions`, `ImportResult`) are exported from the package root.
-
-## Generations & time-travel
-
-The portable document is **current-state** — it never embeds generation history (that keeps
-it cross-version-portable). History lives where it's queryable:
-
-- **During a session:** `brain.asOf(g)` / `brain.now().with(ops)` on the live brain. Because
-  `export()` is on the `Db`, `(await brain.asOf(g)).export()` serializes a *past* generation
-  and `brain.now().with(ops).export()` serializes a *speculative* one.
-- **A whole-brain snapshot with history:** `brain.now().persist(path)` / `Brainy.load(path)`
-  (native, generation-preserving) — a separate facility from this portable format.
-
-Note: only `transact()` (and the write shortcuts that commit through it) advances a
-generation, so time-travel export differs across transaction boundaries.
-
-## Cross-version (7.x → 8.0)
-
-Because the document is shared and versioned, a PortableGraph written by 7.x imports into 8.0:
-`formatVersion` is read forward, `subtype` is carried so 8.0 re-types correctly, and the
-same 384-dimension model on both lines means `includeVectors:false` re-embeds identically
-(or `true` carries vectors verbatim).
-
-## VFS
-
-VFS directories are `Collection` entities and files are entities linked by `Contains`, so
-the whole filesystem (or any subtree) exports through the `vfsPath` selector:
-
-```typescript
-await brain.export({ vfsPath: '/' },        { includeContent: true }) // all VFS + bytes
-await brain.export({ vfsPath: '/docs' },    { includeContent: true }) // one directory
-await brain.export({ vfsPath: '/a/b.txt' }, { includeContent: true }) // one file
-```

docs/guides/external-backups-and-sparse-storage.md

@@ -1,99 +0,0 @@
----
-title: External Backups & Sparse Storage
-slug: guides/external-backups
-public: true
-category: guides
-template: guide
-order: 10
-description: How to back up a brain directory with external tools (tar, rsync, cp) without exploding sparse files — why a store can show 100+ GB "apparent" size on a small disk, which files are sparse, and how persist()/restore() handle it for you.
-next:
-  - guides/snapshots-and-time-travel
-  - concepts/storage-adapters
----
-
-# External Backups & Sparse Storage
-
-The built-in snapshot path — [`db.persist()` and `brain.restore()`](/docs/guides/snapshots-and-time-travel) —
-already handles everything on this page for you. Read this when you back up a brain directory with
-**external tools**: `tar`, `rsync`, `cp`, `scp`, or a filesystem-level backup agent.
-
-## The one-sentence rule
-
-> **Always use the sparse-aware flag**: `tar czSf` (capital `S`), `rsync --sparse`,
-> `cp --sparse=always`. A naive copy can turn a 2 GB store into a 100+ GB one — or fail
-> the disk entirely.
-
-## Why: some files are sparse
-
-When a native accelerator plugin is active, parts of the index live in **memory-mapped files**
-created at a large fixed virtual size — the file's *apparent* size — while the filesystem only
-allocates blocks that were actually written. A brand-new id-mapper file can report tens of
-gigabytes in `ls -l` while occupying a few megabytes on disk.
-
-Check the difference yourself:
-
-```bash
-ls -lh  brain-data/_id_mapper/    # APPARENT size (can be huge)
-du -sh  brain-data/               # ALLOCATED size (the real footprint)
-```
-
-The sparse candidates in a brain directory:
-
-| Path | What it is |
-|---|---|
-| `_id_mapper/` | The native id-mapper's mmap files (large fixed virtual size) |
-| `_blobs/` | Native index files (vector base, segments) — may be mmap-backed |
-
-Everything else (entities, `_system`, `_generations`, `_cas` content blobs) is ordinary dense data.
-
-## Doing it right
-
-**tar** — the `S` flag detects holes and stores only real data:
-
-```bash
-tar czSf brain-backup.tgz /data/brain
-# restore preserves the holes:
-tar xzSf brain-backup.tgz -C /data/
-```
-
-**rsync**:
-
-```bash
-rsync -a --sparse /data/brain/ backup-host:/backups/brain/
-```
-
-**cp**:
-
-```bash
-cp -a --sparse=always /data/brain /backups/brain
-```
-
-**What goes wrong without the flag:** the copy *materializes* every hole as real zero bytes.
-A store whose apparent size exceeds the target disk fails with `ENOSPC` partway through — and a
-copy that *does* fit silently costs the full apparent size in storage and transfer time.
-
-## What the built-in paths do (so you don't have to)
-
-- **`db.persist(path)`** snapshots via **hard links** — instant and space-shared, since every data
-  file is immutable-by-rename. The handful of append-in-place files (the transaction log, the
-  commit fact log's tail segment) and mmap-mutated directories (`_id_mapper/`) are **byte-copied**
-  instead, so a post-snapshot write can never reach through a shared inode into your backup.
-- **`brain.restore(path, { confirm: true })`** is **non-destructive and sparse-aware**: the snapshot
-  is copied into a staging area *before* any live data is touched (all-zero blocks stay holes), and
-  only after the copy fully succeeds does an atomic swap move it into place. A failed copy —
-  including `ENOSPC` — leaves the live store exactly as it was. A crash mid-swap completes forward
-  on the next open.
-
-## Live-store caveats for external tools
-
-1. **Prefer snapshotting a `persist()` output, not the live directory.** `persist()` produces a
-   crash-consistent, immutable snapshot; running `tar` against a live, actively-written directory
-   can capture a torn mid-write state. If you must archive live, stop writes first (or accept that
-   the archive is only as consistent as the moment's flush state).
-2. **Never prune or "clean up" files inside a brain directory.** Index files that look stale or
-   redundant are load-bearing; the store protects its declared index families from in-process
-   deletion, but an external `rm` bypasses that fence. If space is the concern, `du -sh` first —
-   the allocated size is usually far smaller than it looks.
-3. **Verify restores by opening them.** `Brainy.load(path)` opens any snapshot or restored
-   directory read-only — the store verifies its own coherence at open and reports loudly if
-   anything is missing or torn.

docs/guides/find-limits.md

@@ -1,153 +0,0 @@
----
-title: Query Limits & Pagination
-slug: guides/find-limits
-public: true
-category: guides
-template: guide
-order: 8
-description: How Brainy caps `find({ limit })` to prevent OOM, the three escape valves when the cap is too tight, and why pagination is the future-proof pattern.
-next:
-  - guides/aggregation
-  - api/reference
----
-
-# Query Limits & Pagination
-
-Brainy's `find()` returns entities into a JavaScript array. The size of that array is bounded by an auto-configured cap so a single query can never run the host out of memory. This guide explains the cap, the three ways to raise it when your use case justifies it, and the one pattern that scales no matter what cap is in effect: pagination.
-
-## Why the cap exists
-
-Every entity Brainy returns carries:
-
-- A 384-dim float32 embedding vector (1.5 KB)
-- Standard fields: `id`, `type`, `subtype`, timestamps, confidence, weight (~200 bytes)
-- User metadata (variable — typical 5-10 KB, can spike to 20+ KB)
-
-Conservative budget: **25 KB per result**. A `find({ limit: 100_000 })` against a brain with rich metadata can claim ~2.5 GB before Brainy's iteration starts. JavaScript's GC + V8's heap targets can't absorb that swing without paging or OOM in production.
-
-The cap is a safety net. It's not the only reason your query might be slow — graph traversal and HNSW search have their own perf characteristics — but it's the one that turns a slow query into a sudden runtime error.
-
-## The auto-configured cap (7.30.2+)
-
-Brainy picks `maxLimit` from the first of these that's available:
-
-| Priority | Source | Formula |
-|---|---|---|
-| 1 | Constructor option `maxQueryLimit` | Hard cap at supplied value, max 100 000 |
-| 2 | Constructor option `reservedQueryMemory` | `floor(reservedQueryMemory / 25 KB)` capped at 100 000 |
-| 3 | Detected container memory limit (Cloud Run, Kubernetes, cgroups v1/v2) | `floor(containerLimit × 0.25 / 25 KB)` capped at 100 000 |
-| 4 | Free system memory | `floor(availableMemory / 25 KB)` capped at 100 000 |
-
-Worked example: a 4 GB Cloud Run container picks priority 3 → `floor(4 GB × 0.25 / 25 KB) = floor(40 960) = 40 000` results. A 900 MB free-memory box on priority 4 gets `floor(900 MB / 25 KB) = ~36 000`.
-
-The cap is fixed at construction and never changes at runtime. Query timing is recorded
-for diagnostics only — a burst of slow queries cannot silently shrink the cap, and the
-auto-detected tiers (3 and 4) never go below a floor of 10 000.
-
-> **Calibration note.** Pre-7.30.2 used 100 KB per result instead of 25 KB, which produced caps that were 4× too tight for typical workloads (an 8 KB / result reality). 7.30.2 recalibrated to match observed entity sizes; existing `limit: 10_000` safety patterns now pass silently on any reasonably-sized box.
-
-## What happens when you exceed the cap
-
-`find({ limit })` enforces in **two tiers**:
-
-### Soft tier: `maxLimit < limit ≤ 2 × maxLimit`
-
-You get a one-time warning per call site:
-
-```
-[Brainy] find({ limit: 50000 }) exceeds the auto-configured query limit of
-40000 (basis: detected container memory limit). Choose one:
-  • Increase the cap: new Brainy({ maxQueryLimit: 50000 })
-  • Reserve more memory: new Brainy({ reservedQueryMemory: 1310720000 })
-  • Paginate: split the query with { limit, offset } pages
-  at YourService.loadDashboard (/app/src/dashboard.ts:142:18)
-Docs: https://soulcraft.com/docs/guides/find-limits
-```
-
-**The query proceeds.** Brainy returns the result set you asked for; the warning is a teaching signal, not a block. Existing code that relied on the cap silently allowing safety-cap limits (`limit: 10_000` against a 9 K-cap box) keeps working — the warning shows you the recipe so you can fix it intentionally.
-
-### Hard tier: `limit > 2 × maxLimit`
-
-Same message, but thrown as an error. This is real OOM territory; the cap stops being a recommendation and becomes a guardrail.
-
-## The three escape valves
-
-### 1. Raise the cap at construction — `maxQueryLimit`
-
-When the auto-config is wrong for your workload (e.g. you know your entities are smaller than 25 KB average and you need bigger result sets), set an explicit cap:
-
-```typescript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: './data' },
-  maxQueryLimit: 50_000   // raises the cap; still hard-clamped at 100 000
-})
-```
-
-This is the right answer when:
-- Your entity metadata is genuinely small (e.g. 1-2 KB) and 25 KB per result is over-conservative
-- You're running on a box with lots of headroom and 25% of memory underestimates what you can spare for queries
-- You need a known-good limit that doesn't change when the box's free-memory wiggles at startup
-
-### 2. Reserve more memory for queries — `reservedQueryMemory`
-
-When you want the cap to be memory-derived but more generous than the default 25% slice:
-
-```typescript
-const brain = new Brainy({
-  reservedQueryMemory: 1024 * 1024 * 1024  // 1 GB → ~40 000 result cap
-})
-```
-
-This is the right answer when:
-- Your host's memory budget for queries is known and stable, regardless of free-memory at startup
-- You want the formula to scale with the documented per-result size (25 KB) instead of a hard number
-
-### 3. Paginate — the future-proof pattern
-
-If your query genuinely needs to walk all matches in a category, don't fight the cap — walk in pages:
-
-```typescript
-async function findAll<T>(params: FindParams<T>, pageSize = 1000): Promise<Result<T>[]> {
-  const all: Result<T>[] = []
-  let offset = 0
-  while (true) {
-    const page = await brain.find({ ...params, limit: pageSize, offset })
-    all.push(...page)
-    if (page.length < pageSize) break
-    offset += page.length
-  }
-  return all
-}
-
-// Use it just like find():
-const allEvents = await findAll({ type: NounType.Event, where: { status: 'open' } })
-```
-
-For very large brains, prefer the streaming API which avoids holding the full result set in memory at all:
-
-```typescript
-for await (const entity of brain.streaming.entities({ type: NounType.Event })) {
-  // process one entity at a time
-}
-```
-
-## When to use which
-
-| Situation | Recommended valve |
-|---|---|
-| The cap is unreasonably low for your known entity size | `maxQueryLimit` |
-| You want a memory-derived cap but more generous than 25% | `reservedQueryMemory` |
-| Your query needs ALL matches in a category | Pagination or `brain.streaming.entities()` |
-| You hit the cap once during a one-off migration | `maxQueryLimit` or `migrateField` (which already paginates internally) |
-| You're hitting the cap on a recurring user-facing query | Pagination — the cap will get tighter in 8.0, not looser |
-
-## A note on Brainy 8.0
-
-8.0's Datomic-style `Db` API may make per-call limits stricter to keep snapshot semantics cheap. **Pagination is the only pattern that's guaranteed to keep working unchanged.** Code that paginates today doesn't need to revisit when 8.0 ships.
-
-## Reference
-
-- `BrainyConfig.maxQueryLimit?: number` — explicit cap override (max 100 000)
-- `BrainyConfig.reservedQueryMemory?: number` — memory budget for queries (bytes)
-- `find({ limit, offset })` — paginated find
-- `brain.streaming.entities(filter)` — streaming alternative for very large traversals

docs/guides/framework-integration.md

@@ -1,17 +1,15 @@
 # Framework Integration Guide
 
-Brainy is **framework-friendly** - designed to drop into the server side of any modern JavaScript framework. This guide shows you how to integrate Brainy into framework-based apps.
+Brainy 3.0 is **framework-first** - designed from the ground up to work seamlessly with modern JavaScript frameworks. This guide shows you how to integrate Brainy into any framework.
 
-> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun (server-side only). It is not a browser library. In framework apps, run Brainy in API routes, server components, server actions, loaders, or a dedicated backend service - never in client-side bundles.
+## 🎯 Why Framework-First?
 
-## 🎯 Why Server-Side?
-
-Brainy embeds an HNSW vector index, a graph engine, and a filesystem-backed persistence layer. These belong on the server:
+Traditional AI databases require complex browser polyfills and bundler configurations. Brainy 3.0 trusts your framework to handle this:
 
 - **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'`
-- **Auto storage detection**: `new Brainy()` auto-selects filesystem persistence on Node
-- **Cleaner code**: No browser polyfills, no conditional client/server imports
-- **Better DX**: One instance shared across your server routes
+- **Framework responsibility**: Let Next.js, Vite, Webpack handle Node.js polyfills
+- **Cleaner code**: No browser-specific entry points or conditional imports
+- **Better DX**: Same API everywhere - browser, server, edge
 
 ## 🚀 Quick Start
 
@@ -26,8 +24,7 @@ npm install @soulcraft/brainy
 ```javascript
 import { Brainy } from '@soulcraft/brainy'
 
-// Run on the server (API route, server component, backend service)
-// new Brainy() auto-detects filesystem persistence on Node
+// Works in any framework!
 const brain = new Brainy()
 await brain.init()
 
@@ -44,48 +41,52 @@ const results = await brain.find("framework integration")
 
 ## ⚛️ React Integration
 
-Brainy runs on the server, so a React client component talks to it through an API endpoint (see the Next.js API route below). The hook fetches results; it never instantiates Brainy in the browser.
-
 ### Basic Hook Pattern
 
 ```jsx
-import { useState, useCallback } from 'react'
+import { useState, useEffect, useCallback } from 'react'
+import { Brainy } from '@soulcraft/brainy'
 
-function useBrainySearch(endpoint = '/api/search') {
-  const [results, setResults] = useState([])
-  const [loading, setLoading] = useState(false)
+function useBrainy() {
+  const [brain, setBrain] = useState(null)
+  const [isReady, setIsReady] = useState(false)
 
-  const search = useCallback(async (query) => {
-    if (!query) return
-    setLoading(true)
-    try {
-      const res = await fetch(endpoint, {
-        method: 'POST',
-        headers: { 'Content-Type': 'application/json' },
-        body: JSON.stringify({ query })
+  useEffect(() => {
+    const initBrain = async () => {
+      const newBrain = new Brainy({
+        storage: { type: 'opfs' } // Browser storage
       })
-      const { results } = await res.json()
-      setResults(results)
-    } finally {
-      setLoading(false)
+      await newBrain.init()
+      setBrain(newBrain)
+      setIsReady(true)
     }
-  }, [endpoint])
 
-  return { results, loading, search }
+    initBrain()
+  }, [])
+
+  return { brain, isReady }
 }
 
 // Usage in component
 function SearchComponent() {
-  const { results, loading, search } = useBrainySearch()
+  const { brain, isReady } = useBrainy()
+  const [results, setResults] = useState([])
+
+  const handleSearch = useCallback(async (query) => {
+    if (!isReady) return
+    const searchResults = await brain.find(query)
+    setResults(searchResults)
+  }, [brain, isReady])
+
+  if (!isReady) return <div>Loading AI...</div>
 
   return (
     <div>
       <input
         type="text"
         placeholder="Search..."
-        onChange={(e) => search(e.target.value)}
+        onChange={(e) => handleSearch(e.target.value)}
       />
-      {loading && <div>Searching...</div>}
       <div>
         {results.map(result => (
           <div key={result.id}>
@@ -99,34 +100,48 @@ function SearchComponent() {
 }
 ```
 
-### Shared Server Instance
+### React Context Pattern
 
-On the server, create one Brainy instance and reuse it across requests. This module is imported only by server code (API routes, server components), never by client components:
-
-```javascript
-// lib/brain.server.js
+```jsx
+import React, { createContext, useContext, useEffect, useState } from 'react'
 import { Brainy } from '@soulcraft/brainy'
 
-let brainPromise
+const BrainyContext = createContext()
 
-export function getBrain() {
-  if (!brainPromise) {
-    brainPromise = (async () => {
-      // new Brainy() auto-detects filesystem persistence on Node
-      const brain = new Brainy()
-      await brain.init()
-      return brain
-    })()
+export function BrainyProvider({ children }) {
+  const [brain, setBrain] = useState(null)
+  const [isReady, setIsReady] = useState(false)
+
+  useEffect(() => {
+    const initBrain = async () => {
+      const newBrain = new Brainy()
+      await newBrain.init()
+      setBrain(newBrain)
+      setIsReady(true)
+    }
+
+    initBrain()
+  }, [])
+
+  return (
+    <BrainyContext.Provider value={{ brain, isReady }}>
+      {children}
+    </BrainyContext.Provider>
+  )
+}
+
+export function useBrainContext() {
+  const context = useContext(BrainyContext)
+  if (!context) {
+    throw new Error('useBrainContext must be used within BrainyProvider')
   }
-  return brainPromise
+  return context
 }
 ```
 
 ## 🟢 Vue.js Integration
 
-Vue components call a server endpoint (see the Nuxt server route in the [Vue.js Integration Guide](vue-integration.md)); Brainy itself runs on the server.
-
-### Composition API (client component)
+### Composition API
 
 ```vue
 <template>
@@ -140,70 +155,100 @@ Vue components call a server endpoint (see the Nuxt server route in the [Vue.js
 </template>
 
 <script setup>
-import { ref } from 'vue'
+import { ref, onMounted } from 'vue'
+import { Brainy } from '@soulcraft/brainy'
 
+const brain = ref(null)
+const isReady = ref(false)
 const query = ref('')
 const results = ref([])
 
-const search = async () => {
-  if (!query.value) return
-  const res = await fetch('/api/search', {
-    method: 'POST',
-    headers: { 'Content-Type': 'application/json' },
-    body: JSON.stringify({ query: query.value })
+onMounted(async () => {
+  brain.value = new Brainy({
+    storage: { type: 'opfs' }
   })
-  results.value = (await res.json()).results
+  await brain.value.init()
+  isReady.value = true
+})
+
+const search = async () => {
+  if (!isReady.value || !query.value) return
+  results.value = await brain.value.find(query.value)
 }
 </script>
 ```
 
-### Shared Server Instance
-
-On the server, create one Brainy instance and reuse it across requests:
+### Vue 3 Plugin
 
 ```javascript
-// server/brain.js (server-only module)
+// plugins/brainy.js
 import { Brainy } from '@soulcraft/brainy'
 
-let brainPromise
+export default {
+  install(app, options) {
+    const brain = new Brainy(options)
 
-export function getBrain() {
-  if (!brainPromise) {
-    brainPromise = (async () => {
-      // new Brainy() auto-detects filesystem persistence on Node
-      const brain = new Brainy()
-      await brain.init()
-      return brain
-    })()
+    app.config.globalProperties.$brain = brain
+    app.provide('brain', brain)
+
+    // Initialize on app mount
+    brain.init()
   }
-  return brainPromise
 }
+
+// main.js
+import { createApp } from 'vue'
+import BrainyPlugin from './plugins/brainy'
+
+const app = createApp(App)
+app.use(BrainyPlugin, {
+  storage: { type: 'opfs' }
+})
 ```
 
 ## 🅰️ Angular Integration
 
-The Angular service calls your backend over HTTP; Brainy lives in that backend, not in the browser.
-
-### Service Pattern (calls the backend)
+### Service Pattern
 
 ```typescript
 // brainy.service.ts
 import { Injectable } from '@angular/core'
-import { HttpClient } from '@angular/common/http'
-import { Observable } from 'rxjs'
+import { BehaviorSubject, Observable } from 'rxjs'
+import { Brainy } from '@soulcraft/brainy'
 
 @Injectable({
   providedIn: 'root'
 })
 export class BrainyService {
-  constructor(private http: HttpClient) {}
+  private brain: Brainy
+  private readySubject = new BehaviorSubject<boolean>(false)
 
-  search(query: string): Observable<{ results: any[] }> {
-    return this.http.post<{ results: any[] }>('/api/search', { query })
+  ready$: Observable<boolean> = this.readySubject.asObservable()
+
+  constructor() {
+    this.initBrain()
   }
 
-  add(data: any, type: string, metadata?: any): Observable<{ id: string }> {
-    return this.http.post<{ id: string }>('/api/add', { data, type, metadata })
+  private async initBrain() {
+    this.brain = new Brainy({
+      storage: { type: 'opfs' }
+    })
+    await this.brain.init()
+    this.readySubject.next(true)
+  }
+
+  async search(query: string): Promise<any[]> {
+    if (!this.readySubject.value) {
+      throw new Error('Brain not ready')
+    }
+    return await this.brain.find(query)
+  }
+
+  async add(data: any, type: string, metadata?: any): Promise<string> {
+    if (!this.readySubject.value) {
+      throw new Error('Brain not ready')
+    }
+    return await this.brain.add({ data, type, metadata })
   }
 }
 ```
@@ -235,52 +280,64 @@ export class SearchComponent {
 
   constructor(private brainyService: BrainyService) {}
 
-  search() {
+  async search() {
     if (!this.query) return
-    this.brainyService.search(this.query).subscribe(({ results }) => {
-      this.results = results
-    })
+    this.results = await this.brainyService.search(this.query)
   }
 }
 ```
 
-The matching backend endpoint uses Brainy directly (Node/Bun):
-
-```typescript
-// server: api/search
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy() // auto-detects filesystem persistence on Node
-await brain.init()
-
-export async function handleSearch(query: string) {
-  return await brain.find(query)
-}
-```
-
 ## 🚀 Next.js Integration
 
-In Next.js, Brainy lives in server code only: API routes, server components, or server actions. Create one shared instance in a server-only module.
+### App Router (Next.js 13+)
 
-### Shared Server Instance
-
-```javascript
-// lib/brain.server.js  (imported only by server code)
+```jsx
+// app/providers.jsx
+'use client'
+import { createContext, useContext, useEffect, useState } from 'react'
 import { Brainy } from '@soulcraft/brainy'
 
-let brainPromise
+const BrainyContext = createContext()
 
-export function getBrain() {
-  if (!brainPromise) {
-    brainPromise = (async () => {
-      const brain = new Brainy({
-        storage: { type: 'filesystem', path: './data' }
-      })
-      await brain.init()
-      return brain
-    })()
-  }
-  return brainPromise
+export function BrainyProvider({ children }) {
+  const [brain, setBrain] = useState(null)
+  const [isReady, setIsReady] = useState(false)
+
+  useEffect(() => {
+    const initBrain = async () => {
+      const newBrain = new Brainy()
+      await newBrain.init()
+      setBrain(newBrain)
+      setIsReady(true)
+    }
+
+    initBrain()
+  }, [])
+
+  return (
+    <BrainyContext.Provider value={{ brain, isReady }}>
+      {children}
+    </BrainyContext.Provider>
+  )
+}
+
+export const useBrainy = () => useContext(BrainyContext)
+```
+
+```jsx
+// app/layout.jsx
+import { BrainyProvider } from './providers'
+
+export default function RootLayout({ children }) {
+  return (
+    <html>
+      <body>
+        <BrainyProvider>
+          {children}
+        </BrainyProvider>
+      </body>
+    </html>
+  )
 }
 ```
 
@@ -288,78 +345,45 @@ export function getBrain() {
 
 ```javascript
 // app/api/search/route.js
-import { getBrain } from '@/lib/brain.server'
+import { Brainy } from '@soulcraft/brainy'
+
+const brain = new Brainy({
+  storage: { type: 'filesystem', path: './data' }
+})
+await brain.init()
 
 export async function POST(request) {
   const { query } = await request.json()
-  const brain = await getBrain()
   const results = await brain.find(query)
 
   return Response.json({ results })
 }
 ```
 
-### Server Action
-
-```javascript
-// app/actions.js
-'use server'
-import { getBrain } from '@/lib/brain.server'
-
-export async function search(query) {
-  const brain = await getBrain()
-  return await brain.find(query)
-}
-```
-
-## 🔷 SvelteKit Integration
-
-Brainy runs in a server-only module (`*.server.js`); the component fetches results from an endpoint.
-
-```javascript
-// src/lib/server/brain.js  (server-only — note the .server suffix)
-import { Brainy } from '@soulcraft/brainy'
-
-let brainPromise
-
-export function getBrain() {
-  if (!brainPromise) {
-    brainPromise = (async () => {
-      const brain = new Brainy() // auto-detects filesystem persistence
-      await brain.init()
-      return brain
-    })()
-  }
-  return brainPromise
-}
-```
-
-```javascript
-// src/routes/api/search/+server.js
-import { json } from '@sveltejs/kit'
-import { getBrain } from '$lib/server/brain'
-
-export async function POST({ request }) {
-  const { query } = await request.json()
-  const brain = await getBrain()
-  return json({ results: await brain.find(query) })
-}
-```
+## 🔷 Svelte Integration
 
 ```svelte
 <!-- SearchComponent.svelte -->
 <script>
+  import { onMount } from 'svelte'
+  import { Brainy } from '@soulcraft/brainy'
+
+  let brain = null
+  let isReady = false
   let query = ''
   let results = []
 
-  async function search() {
-    if (!query) return
-    const res = await fetch('/api/search', {
-      method: 'POST',
-      headers: { 'Content-Type': 'application/json' },
-      body: JSON.stringify({ query })
+  onMount(async () => {
+    brain = new Brainy({
+      storage: { type: 'opfs' }
     })
-    results = (await res.json()).results
+    await brain.init()
+    isReady = true
+  })
+
+  async function search() {
+    if (!isReady || !query) return
+    results = await brain.find(query)
   }
 </script>
 
@@ -377,23 +401,27 @@ export async function POST({ request }) {
 
 ## 🌟 Solid.js Integration
 
-The component calls a server endpoint (use SolidStart server routes, or any backend, to host Brainy):
-
 ```jsx
-import { createSignal } from 'solid-js'
+import { createSignal, onMount } from 'solid-js'
+import { Brainy } from '@soulcraft/brainy'
 
 function SearchComponent() {
+  const [brain, setBrain] = createSignal(null)
+  const [isReady, setIsReady] = createSignal(false)
   const [query, setQuery] = createSignal('')
   const [results, setResults] = createSignal([])
 
+  onMount(async () => {
+    const newBrain = new Brainy()
+    await newBrain.init()
+    setBrain(newBrain)
+    setIsReady(true)
+  })
+
   const search = async () => {
-    if (!query()) return
-    const res = await fetch('/api/search', {
-      method: 'POST',
-      headers: { 'Content-Type': 'application/json' },
-      body: JSON.stringify({ query: query() })
-    })
-    setResults((await res.json()).results)
+    if (!isReady() || !query()) return
+    const searchResults = await brain().find(query())
+    setResults(searchResults)
   }
 
   return (
@@ -422,25 +450,57 @@ function SearchComponent() {
 
 ## 📦 Bundler Configuration
 
-Brainy is a server-side dependency, so keep it out of client bundles. Import it only from server-only modules (`*.server.js`, API routes, server components, server actions). If your bundler ever tries to pull Brainy into a client bundle, that's a sign it's being imported from a client component — move the import to a server module.
-
-For server builds, mark Brainy as external so the bundler doesn't inline it:
+### Vite (Recommended)
 
 ```javascript
-// vite.config.js  (SSR build)
+// vite.config.js
 import { defineConfig } from 'vite'
 
 export default defineConfig({
-  ssr: {
-    external: ['@soulcraft/brainy']
+  define: {
+    global: 'globalThis'
+  },
+  optimizeDeps: {
+    include: ['@soulcraft/brainy']
   }
 })
 ```
 
+### Webpack
+
 ```javascript
-// rollup.config.js  (server bundle)
+// webpack.config.js
+module.exports = {
+  resolve: {
+    fallback: {
+      "fs": false,
+      "path": require.resolve("path-browserify"),
+      "crypto": require.resolve("crypto-browserify")
+    }
+  },
+  plugins: [
+    new webpack.ProvidePlugin({
+      global: 'global'
+    })
+  ]
+}
+```
+
+### Rollup
+
+```javascript
+// rollup.config.js
+import { nodeResolve } from '@rollup/plugin-node-resolve'
+import commonjs from '@rollup/plugin-commonjs'
+
 export default {
-  external: ['@soulcraft/brainy', 'node:fs', 'node:path', 'node:crypto']
+  plugins: [
+    nodeResolve({
+      browser: true,
+      preferBuiltins: false
+    }),
+    commonjs()
+  ]
 }
 ```
 
@@ -448,24 +508,30 @@ export default {
 
 ### Server-Side Rendering
 
-Instantiate Brainy on the server and feed its results into the rendered page. Never construct it in client code.
-
 ```javascript
-// Server-side data loading (framework loader / getServerSideProps / load fn)
-import { getBrain } from './brain.server'
+// Check if running in browser
+if (typeof window !== 'undefined') {
+  // Browser-only code
+  const brain = new Brainy({
+    storage: { type: 'opfs' }
+  })
+}
 
-export async function load({ url }) {
-  const brain = await getBrain()
-  const query = url.searchParams.get('q') ?? ''
-  const results = query ? await brain.find(query) : []
-  return { results }
+// Or use dynamic imports
+const initBrainForBrowser = async () => {
+  if (typeof window === 'undefined') return null
+
+  const { Brainy } = await import('@soulcraft/brainy')
+  const brain = new Brainy()
+  await brain.init()
+  return brain
 }
 ```
 
 ### Static Site Generation
 
 ```javascript
-// For build-time usage (runs in Node during the build)
+// For build-time usage
 import { Brainy } from '@soulcraft/brainy'
 
 export async function generateStaticProps() {
@@ -474,8 +540,8 @@ export async function generateStaticProps() {
   })
   await brain.init()
 
-  // Build search index (paginate with { limit, offset } for larger stores)
-  const allContent = await brain.find({ limit: 1000 })
+  // Build search index
+  const allContent = await brain.export()
 
   return {
     props: { searchIndex: allContent }
@@ -486,46 +552,67 @@ export async function generateStaticProps() {
 ## 🔧 Framework-Specific Tips
 
 ### React
-- Keep components client-side and call a Brainy-backed API route
-- Use `useCallback` for fetch handlers to prevent re-renders
-- Debounce keystroke-driven searches before hitting the endpoint
+- Use `useCallback` for search functions to prevent re-renders
+- Consider `useMemo` for expensive brain operations
+- Implement cleanup in `useEffect` for proper memory management
 
 ### Vue
-- Components call an endpoint; the shared instance lives in a server module
-- Consider Pinia for caching results client-side
-- Debounce reactive search queries
+- Use `shallowRef` for the brain instance (it's not reactive data)
+- Consider Pinia for global brain state management
+- Use `watchEffect` for reactive search queries
 
 ### Angular
-- Use `HttpClient` and RxJS to call the backend
-- Hold the shared Brainy instance in your Node backend, not the app
-- Consider lazy loading search features in feature modules
+- Implement proper dependency injection with services
+- Use RxJS observables for reactive search
+- Consider lazy loading brain in feature modules
 
 ### Next.js
-- Put Brainy in server-only modules (`*.server.js`), API routes, or server actions
-- Reuse one shared instance across requests
-- Implement proper error boundaries for failed fetches
+- Use dynamic imports for client-side only features
+- Consider API routes for server-side brain operations
+- Implement proper error boundaries
 
 ## 🚨 Common Issues & Solutions
 
-### Issue: "fs module not found" / "crypto is not defined" in the browser
-**Cause**: Brainy was imported into a client bundle. Brainy 8.0 is a server-side library (Node 22+/Bun) and uses Node built-ins like `fs` and `crypto`.
-**Solution**: Import Brainy only from server code — server-only modules (`*.server.js`), API routes, server components, or server actions. From client components, call those endpoints instead.
+### Issue: "crypto is not defined"
+**Solution**: Your framework should handle this automatically. If not:
+```javascript
+// Add to your bundle config
+define: {
+  global: 'globalThis'
+}
+```
 
-### Issue: Large client bundle size
-**Cause**: A client module is pulling in Brainy.
-**Solution**: Move the `import { Brainy } from '@soulcraft/brainy'` into a server-only module so it never reaches the browser bundle.
+### Issue: "fs module not found"
+**Solution**: This is expected in browsers. Use browser-compatible storage:
+```javascript
+const brain = new Brainy({
+  storage: { type: 'opfs' } // Or 'memory' for development
+})
+```
+
+### Issue: Large bundle size
+**Solution**: Use dynamic imports for optional features:
+```javascript
+const brain = await import('@soulcraft/brainy').then(m => new m.Brainy())
+```
 
 ### Issue: SSR hydration mismatch
-**Solution**: Run the search on the server (loader / server action / API route) and pass the results down as props, so server and client render the same markup.
+**Solution**: Initialize brain only on client:
+```javascript
+useEffect(() => {
+  // Browser-only initialization
+  initBrain()
+}, [])
+```
 
 ## 🎯 Best Practices
 
-1. **Initialize Once**: Create one shared Brainy instance per server process, not per request
-2. **Server-Only**: Import Brainy only from server modules — never from client components
-3. **Endpoint Boundary**: Expose search/add through API routes or server actions
-4. **Handle Loading**: Show loading states in the client while the fetch is in flight
-5. **Error Handling**: Catch and surface failed endpoint calls gracefully
-6. **Storage**: Use `filesystem` for persistence (the default on Node) or `memory` for ephemeral/tests
+1. **Initialize Once**: Create brain instance at app level, not component level
+2. **Use Context**: Share brain instance across components with context/providers
+3. **Handle Loading**: Always show loading states during brain initialization
+4. **Error Boundaries**: Implement proper error handling for brain operations
+5. **Memory Management**: Clean up brain instances on unmount
+6. **Storage Strategy**: Choose appropriate storage for your deployment target
 
 ## 📚 Next Steps
 

docs/guides/getting-started.md

@@ -0,0 +1,333 @@
+# Getting Started with Brainy
+
+This guide will help you get up and running with Brainy, the multi-dimensional AI database that combines vector similarity, graph relationships, and metadata filtering.
+
+## Installation
+
+```bash
+npm install @soulcraft/brainy
+```
+
+## Basic Setup
+
+### Simple Initialization
+
+```typescript
+import { Brainy } from '@soulcraft/brainy'
+
+// Create a new Brainy instance with defaults
+const brain = new Brainy()
+
+// Initialize (downloads models if needed)
+await brain.init()
+
+// You're ready to go!
+```
+
+### Custom Configuration
+
+```typescript
+const brain = new Brainy({
+  // Storage configuration
+  storage: {
+    type: 'filesystem',  // or 's3', 'opfs'
+    path: './my-data'
+  },
+  
+  // Vector configuration
+  vectors: {
+    dimensions: 384,
+    model: 'all-MiniLM-L6-v2'
+  },
+  
+  // Performance tuning
+  cache: {
+    enabled: true,
+    maxSize: 1000
+  }
+})
+
+await brain.init()
+```
+
+## Your First Operations
+
+### Adding Data
+
+```typescript
+// Add entities (nouns) with automatic embedding generation
+const id = await brain.add("The quick brown fox jumps over the lazy dog", {
+  category: "demo",
+  timestamp: Date.now()
+})
+
+console.log(`Added noun with ID: ${id}`)
+
+// Add relationships (verbs) between entities
+const sourceId = await brain.add("John Smith", { nounType: 'person' })
+const targetId = await brain.add("TechCorp", { nounType: 'organization' })
+await brain.relate(sourceId, targetId, "works_at", {
+  position: "Engineer",
+  since: "2024"
+})
+```
+
+### Searching
+
+```typescript
+// Simple semantic search
+const results = await brain.search("fast animals")
+
+results.forEach(result => {
+  console.log(`Found: ${result.content} (score: ${result.score})`)
+})
+```
+
+### Advanced Queries with find()
+
+```typescript
+// Natural language queries - Brainy understands intent!
+const results = await brain.find("show me technology articles about AI from 2023")
+// Automatically interprets: topic, category, and time range
+
+// Structured queries with vector similarity and metadata filtering
+const structured = await brain.find({
+  like: "artificial intelligence",
+  where: {
+    category: "technology",
+    year: { $gte: 2023 }
+  },
+  limit: 10
+})
+
+// Complex natural language with multiple filters
+const complex = await brain.find("financial reports from Q3 2024 with revenue over 1M")
+// Automatically extracts: document type, date range, numeric filters
+```
+
+## Common Use Cases
+
+### 1. Semantic Search Engine
+
+```typescript
+// Index documents
+const documents = [
+  { title: "Introduction to AI", content: "AI is transforming..." },
+  { title: "Machine Learning Basics", content: "ML algorithms..." },
+  { title: "Deep Learning", content: "Neural networks..." }
+]
+
+for (const doc of documents) {
+  await brain.add(doc.content, {
+    title: doc.title,
+    type: "document"
+  })
+}
+
+// Search semantically
+const results = await brain.search("how do neural networks work")
+```
+
+### 2. Recommendation System
+
+```typescript
+// Add user interactions as nouns
+const interactionId = await brain.add("user viewed product", {
+  userId: "user123",
+  productId: "product456",
+  action: "view",
+  timestamp: Date.now()
+})
+
+// Create relationships between users and products
+const userId = await brain.add("user123", { nounType: 'user' })
+const productId = await brain.add("product456", { nounType: 'product' })
+await brain.relate(userId, productId, "viewed", {
+  timestamp: Date.now()
+})
+
+// Natural language query for recommendations
+const recommendations = await brain.find("products similar to what user123 viewed recently")
+
+// Or structured query for similar users
+const similar = await brain.find({
+  like: "user123 interests",
+  where: { action: "view" },
+  limit: 5
+})
+```
+
+### 3. Knowledge Graph
+
+```typescript
+// Add entities (nouns) to the knowledge graph
+const personId = await brain.add("John Smith, Software Engineer", {
+  type: "person",
+  role: "engineer"
+})
+
+const companyId = await brain.add("TechCorp, Innovation Leader", {
+  type: "company",
+  industry: "technology"
+})
+
+// Create relationship
+await brain.relate(personId, companyId, "works_at", {
+  since: "2020",
+  position: "Senior Engineer"
+})
+
+// Natural language query for relationships
+const colleagues = await brain.find("people who work at TechCorp")
+
+// Or structured query for specific relationships
+const results = await brain.find({
+  connected: {
+    from: personId,
+    type: "works_at"
+  }
+})
+```
+
+### 4. Real-time Data Processing
+
+```typescript
+// Configure for streaming
+const brain = new Brainy({
+  augmentations: [
+    new EntityRegistryAugmentation(),  // Deduplication
+    new BatchProcessingAugmentation({ batchSize: 100 })  // Batching
+  ]
+})
+
+// Process streaming data
+async function processStream(item) {
+  // Entity registry prevents duplicate nouns
+  const id = await brain.add(item.content, {
+    externalId: item.id,
+    timestamp: item.timestamp
+  })
+  
+  // Real-time natural language queries
+  if (item.urgent) {
+    const related = await brain.find(`urgent items similar to ${item.content}`)
+    // Process related items...
+  }
+}
+```
+
+## Storage Options
+
+### Development (FileSystem)
+```typescript
+const brain = new Brainy({
+  storage: { type: 'filesystem', path: '/tmp/brainy-dev' }
+})
+// Fast, persistent, perfect for testing
+```
+
+### Production (FileSystem)
+```typescript
+const brain = new Brainy({
+  storage: {
+    type: 'filesystem',
+    path: '/var/lib/brainy'
+  }
+})
+// Persistent, efficient, server-ready
+```
+
+### Cloud (S3)
+```typescript
+const brain = new Brainy({
+  storage: {
+    type: 's3',
+    bucket: 'my-brainy-data',
+    region: 'us-east-1'
+  }
+})
+// Scalable, distributed, cloud-native
+```
+
+### Browser (OPFS)
+```typescript
+const brain = new Brainy({
+  storage: { type: 'opfs' }
+})
+// Browser-native, persistent, offline-capable
+```
+
+## Performance Tips
+
+### 1. Use Batch Operations
+```typescript
+// Good - batch operations for nouns
+const items = ["item1", "item2", "item3"]
+for (const item of items) {
+  await brain.add(item, { batch: true })
+}
+
+// Create relationships efficiently
+const relationships = [
+  { source: id1, target: id2, type: "related" },
+  { source: id2, target: id3, type: "similar" }
+]
+for (const rel of relationships) {
+  await brain.relate(rel.source, rel.target, rel.type)
+}
+```
+
+### 2. Enable Caching
+```typescript
+const brain = new Brainy({
+  cache: {
+    enabled: true,
+    maxSize: 1000,
+    ttl: 300000  // 5 minutes
+  }
+})
+```
+
+### 3. Use Appropriate Limits
+```typescript
+// Always specify reasonable limits
+const results = await brain.search("query", {
+  limit: 20  // Don't fetch more than needed
+})
+```
+
+### 4. Index Frequently Queried Fields
+```typescript
+const brain = new Brainy({
+  indexedFields: ['category', 'userId', 'timestamp']
+})
+```
+
+## Error Handling
+
+```typescript
+try {
+  await brain.add("content", metadata)
+} catch (error) {
+  if (error.code === 'STORAGE_FULL') {
+    console.error('Storage is full')
+  } else if (error.code === 'INVALID_INPUT') {
+    console.error('Invalid input:', error.message)
+  } else {
+    console.error('Unexpected error:', error)
+  }
+}
+```
+
+## Next Steps
+
+- [Architecture Overview](../architecture/overview.md) - Understand the system design
+- [Triple Intelligence](../architecture/triple-intelligence.md) - Advanced query capabilities
+- [API Reference](../api/README.md) - Complete API documentation
+- [Examples](https://github.com/brainy-org/brainy/tree/main/examples) - More code examples
+
+## Getting Help
+
+- **Issues**: [GitHub Issues](https://github.com/brainy-org/brainy/issues)
+- **Discussions**: [GitHub Discussions](https://github.com/brainy-org/brainy/discussions)
+- **Examples**: Check the `/examples` directory

docs/guides/import-anything.md

@@ -3,8 +3,8 @@
 Brainy's import is **ONE magical method** that understands EVERYTHING:
 - 📊 Data (objects, arrays, strings)
 - 📁 Files (auto-detects by path)
-- 🌐 URLs (auto-fetches with authentication support)
-- 📄 Formats (JSON, CSV, Excel, PDF, YAML, DOCX, Markdown - all auto-detected)
+- 🌐 URLs (auto-fetches)
+- 📄 Formats (JSON, CSV, Excel, PDF, YAML, text - all auto-detected)
 
 ## The Ultimate Simplicity
 
@@ -24,8 +24,8 @@ await brain.import(anything)
 ```javascript
 // Array of objects? No problem.
 const people = [
- { name: 'Alice', role: 'Engineer', company: 'TechCorp' },
- { name: 'Bob', role: 'Designer', company: 'TechCorp' }
+  { name: 'Alice', role: 'Engineer', company: 'TechCorp' },
+  { name: 'Bob', role: 'Designer', company: 'TechCorp' }
 ]
 
 await brain.import(people)
@@ -49,23 +49,28 @@ await brain.import(csv, { format: 'csv' })
 
 ### 📊 Import Excel - Multi-Sheet Support
 ```javascript
-// Import entire Excel workbook — every sheet is processed automatically
+// Import entire Excel workbook
 await brain.import('sales-report.xlsx')
 // ✨ Processes all sheets, preserves structure, infers types!
 
-// Mirror the workbook into the VFS, grouped by sheet
+// Or specific sheets only
 await brain.import('data.xlsx', {
- vfsPath: '/imports/data',
- groupBy: 'sheet'
+  excelSheets: ['Customers', 'Orders']
 })
 // ✨ Multi-sheet data becomes interconnected entities!
 ```
 
 ### 📑 Import PDF - Text & Tables
 ```javascript
-// Import PDF documents — text and tables are extracted automatically
+// Import PDF documents
 await brain.import('research-paper.pdf')
 // ✨ Extracts text, detects tables, preserves metadata!
+
+// With table extraction
+await brain.import('report.pdf', {
+  pdfExtractTables: true
+})
+// ✨ Converts PDF tables to structured data automatically!
 ```
 
 ### 📝 Import YAML - File or String
@@ -78,34 +83,15 @@ await brain.import('config.yaml')
 const yaml = `
 project: AI Assistant
 team:
- - name: Alice
- role: Lead
- - name: Bob
- role: Dev
+  - name: Alice
+    role: Lead
+  - name: Bob
+    role: Dev
 `
 await brain.import(yaml, { format: 'yaml' })
 // ✨ Hierarchical data becomes a connected graph!
 ```
 
-### 📄 Import Word Documents (DOCX) -
-```javascript
-// From file path
-await brain.import('research-paper.docx')
-// ✨ Extracts text, headings, tables, and metadata!
-
-// Or from buffer
-const buffer = fs.readFileSync('document.docx')
-await brain.import(buffer, { format: 'docx' })
-// ✨ Uses heading hierarchy for entity organization!
-
-// With neural extraction
-await brain.import('report.docx', {
- enableNeuralExtraction: true,
- enableHierarchicalRelationships: true
-})
-// ✨ Extracts entities from paragraphs and creates relationships within sections!
-```
-
 ### 🌐 Import from URLs - Auto-Detected!
 ```javascript
 // Just pass the URL - it knows!
@@ -115,28 +101,6 @@ await brain.import('https://api.example.com/data.json')
 // Works with any URL
 await brain.import('https://data.gov/census.csv')
 // ✨ Fetches CSV from web, parses, imports!
-
-// With authentication
-await brain.import({
- type: 'url',
- data: 'https://api.example.com/private/data.xlsx',
- auth: {
- username: 'user',
- password: 'pass'
- }
-})
-// ✨ Supports basic authentication for protected resources!
-
-// With custom headers
-await brain.import({
- type: 'url',
- data: 'https://api.example.com/data.json',
- headers: {
- 'Authorization': 'Bearer TOKEN',
- 'X-API-Key': 'your-key'
- }
-})
-// ✨ Full HTTP header customization support!
 ```
 
 ### 📖 Import Plain Text
@@ -154,13 +118,12 @@ await brain.import(article, { format: 'text' })
 
 When you import data, Brainy:
 
-1. **Auto-detects format** - CSV, Excel, PDF, JSON, YAML, DOCX, Markdown, or by file extension
-2. **Intelligent parsing** - CSV (encoding/delimiter), Excel (multi-sheet), PDF (text/tables), DOCX (headings/paragraphs)
+1. **Auto-detects format** - CSV, Excel, PDF, JSON, YAML, text, or by file extension
+2. **Intelligent parsing** - CSV (encoding/delimiter detection), Excel (multi-sheet), PDF (text/tables)
 3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
 4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
-5. **Scores confidence & weight** - Every entity and relationship gets quality metrics
-6. **Creates embeddings** - Makes everything semantically searchable
-7. **Indexes metadata** - Enables lightning-fast filtering with range queries
+5. **Creates embeddings** - Makes everything semantically searchable
+6. **Indexes metadata** - Enables lightning-fast filtering
 
 ## Intelligent Type Detection
 
@@ -171,7 +134,7 @@ Brainy automatically detects what TYPE of data you're importing:
 { name: 'John', email: 'john@example.com' }
 
 // This becomes an Organization
-{ companyName: 'Acme', employees: 500 }
+{ companyName: 'Acme', employees: 500 }  
 
 // This becomes a Document
 { title: 'Report', content: '...', author: 'Jane' }
@@ -180,7 +143,7 @@ Brainy automatically detects what TYPE of data you're importing:
 { latitude: 37.7, longitude: -122.4, city: 'SF' }
 ```
 
-**42 noun types and 127 verb types** cover EVERYTHING!
+**31 noun types** and **40 verb types** cover EVERYTHING!
 
 ## Relationship Detection
 
@@ -188,9 +151,9 @@ Brainy finds connections in your data:
 
 ```javascript
 const data = [
- { id: 'u1', name: 'Alice', managerId: 'u2' },
- { id: 'u2', name: 'Bob', departmentId: 'd1' },
- { id: 'd1', name: 'Engineering' }
+  { id: 'u1', name: 'Alice', managerId: 'u2' },
+  { id: 'u2', name: 'Bob', departmentId: 'd1' },
+  { id: 'd1', name: 'Engineering' }
 ]
 
 await brain.import(data)
@@ -199,77 +162,22 @@ await brain.import(data)
 // - Bob "memberOf" Engineering
 ```
 
-## Confidence & Weight Scoring -
-Every entity and relationship gets confidence and weight scores:
-
-```javascript
-// Import with confidence threshold
-await brain.import(data, {
- confidenceThreshold: 0.8 // Only extract entities with >80% confidence
-})
-
-// Query high-confidence entities using range queries
-const highConfidence = await brain.find({
- where: {
- confidence: { gte: 0.8 } // Get entities with confidence >= 0.8
- }
-})
-
-// Range query operators: gt, gte, lt, lte, between
-const mediumConfidence = await brain.find({
- where: {
- confidence: { between: [0.6, 0.8] }
- }
-})
-```
-
-**What do confidence scores mean?**
-- **High (>0.8)**: Very confident entity classification
-- **Medium (0.6-0.8)**: Reasonable confidence
-- **Low (<0.6)**: Uncertain classification (filtered by default)
-
-**Weights** indicate importance/relevance within the document context.
-
-## Per-Sheet Excel Extraction -
-Excel files with multiple sheets can be organized by sheet:
-
-```javascript
-// Group entities by sheet in VFS
-await brain.import('multi-sheet-data.xlsx', {
- groupBy: 'sheet' // Creates separate directories for each sheet
-})
-
-// Result VFS structure:
-// /imports/data/
-// ├── Sheet1/
-// │ ├── entity1.json
-// │ └── entity2.json
-// └── Sheet2/
-// ├── entity3.json
-// └── entity4.json
-
-// Other groupBy options:
-// - 'type': Group by entity type (Person, Place, etc.)
-// - 'flat': All entities in one directory
-// - 'custom': Use custom grouping function
-```
-
 ## Query Your Imported Data
 
 Once imported, use Triple Intelligence to query:
 
 ```javascript
 // Vector search
-const similar = await brain.find('engineers')
+const similar = await brain.search('engineers')
 
 // Natural language
 const results = await brain.find('people in engineering who joined this year')
 
 // Graph traversal + filters
 const connected = await brain.find({
- like: 'Alice',
- connected: { depth: 2 },
- where: { department: 'Engineering' }
+  like: 'Alice',
+  connected: { depth: 2 },
+  where: { department: 'Engineering' }
 })
 ```
 
@@ -279,40 +187,37 @@ Everything works with zero config, but you can customize:
 
 ```javascript
 await brain.import(data, {
- // Format detection
- format: 'excel', // Force specific format (auto-detected if not specified)
+  // Format detection
+  format: 'excel',                     // Force specific format (auto-detected if not specified)
 
- // VFS & Organization
- vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
- groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
- preserveSource: true, // Keep original source file in VFS (default: true)
+  // VFS & Organization
+  vfsPath: '/imports/my-data',         // Where to store in VFS (auto-generated if not specified)
+  groupBy: 'type',                     // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
+  preserveSource: true,                // Keep original source file in VFS (default: true)
 
- // Entity & Relationship Creation
- createEntities: true, // Create entities in knowledge graph (default: true)
- createRelationships: true, // Create relationships in knowledge graph (default: true)
+  // Entity & Relationship Creation
+  createEntities: true,                // Create entities in knowledge graph (default: true)
+  createRelationships: true,           // Create relationships in knowledge graph (default: true)
 
- // Neural Intelligence
- enableNeuralExtraction: true, // Use AI to extract entities (default: true)
- enableRelationshipInference: true, // Use AI to infer relationships (default: true)
- enableConceptExtraction: true, // Extract concepts from text (default: true)
- confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
+  // Neural Intelligence
+  enableNeuralExtraction: true,        // Use AI to extract entities (default: true)
+  enableRelationshipInference: true,   // Use AI to infer relationships (default: true)
+  enableConceptExtraction: true,       // Extract concepts from text (default: true)
+  confidenceThreshold: 0.6,            // Minimum confidence for entities (0-1, default: 0.6)
 
- // Deduplication
- enableDeduplication: true, // Check for duplicate entities (default: true)
- deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
- // Notes: false disables BOTH the inline merge and the background pass that
- // runs ~5 min after the last import (merged duplicates are deleted).
- // The inline pass auto-disables for imports >100 entities (O(n²) cost);
- // the background pass still covers those unless the flag is false.
+  // Deduplication
+  enableDeduplication: true,           // Check for duplicate entities (default: true)
+  deduplicationThreshold: 0.85,        // Similarity threshold for duplicates (0-1, default: 0.85)
+                                       // Note: Auto-disabled for imports >100 entities
 
- // Performance
- chunkSize: 100, // Batch size for processing (default: varies by operation)
+  // Performance
+  chunkSize: 100,                      // Batch size for processing (default: varies by operation)
 
- // History & Progress
- enableHistory: true, // Track import history (default: true)
- onProgress: (progress) => { // Progress callback
- console.log(progress.stage, progress.message)
- }
+  // History & Progress
+  enableHistory: true,                 // Track import history (default: true)
+  onProgress: (progress) => {          // Progress callback
+    console.log(progress.stage, progress.message)
+  }
 })
 ```
 
@@ -339,9 +244,9 @@ const results = await brain.import(problematicData)
 ### 🏢 Business Data
 ```javascript
 // Import ANY source - ONE method!
-await brain.import('customers.csv') // File
-await brain.import('https://api.co/orders') // URL
-await brain.import(productsArray) // Data
+await brain.import('customers.csv')           // File
+await brain.import('https://api.co/orders')   // URL
+await brain.import(productsArray)             // Data
 
 // Now query across all of it!
 await brain.find('customers who bought products in Q4')
@@ -378,15 +283,15 @@ await brain.find('posts by users following Alice with >10 comments')
 
 **Zero Configuration**: Works perfectly out of the box
 **Maximum Intelligence**: AI understands your data's meaning
-**Universal Protocol**: 42 nouns × 127 verbs = ANY data model
+**Universal Protocol**: 31 nouns × 40 verbs = ANY data model
 **Delightful DX**: Simple, clean, modern API
 
 ## The ONE Method Philosophy
 
 ```javascript
 // ONE method that understands EVERYTHING:
-await brain.import(data) // Objects, arrays, strings
-await brain.import('file.csv') // Files (auto-detected)
+await brain.import(data)        // Objects, arrays, strings
+await brain.import('file.csv')  // Files (auto-detected)
 await brain.import('http://..') // URLs (auto-fetched)
 
 // It ALWAYS knows what to do! ✨

docs/guides/import-progress-examples.md

@@ -1,370 +0,0 @@
-# Import Progress - Usage Examples
-
-**How to Use Progress Tracking in Your Applications**
-
-Brainy provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
-
-> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation.
-
----
-
-## 🚀 Quick Start
-
-### Basic Progress Tracking
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-import * as fs from 'fs'
-
-const brain = await Brainy.create()
-
-// Import with progress tracking
-const result = await brain.import(fs.readFileSync('large-file.xlsx'), {
- onProgress: (progress) => {
- console.log(`Progress: ${progress.stage}`)
- console.log(` Message: ${progress.message}`)
- console.log(` Entities: ${progress.entities || 0}`)
- console.log(` Relationships: ${progress.relationships || 0}`)
- }
-})
-
-console.log(`Import complete: ${result.entities.length} entities created`)
-```
-
-**Expected Output:**
-```
-Progress: detecting
- Message: Detecting format...
- Entities: 0
- Relationships: 0
-Progress: extracting
- Message: Loading Excel workbook...
- Entities: 0
- Relationships: 0
-Progress: extracting
- Message: Reading sheet: Sales (1/3)
- Entities: 0
- Relationships: 0
-Progress: extracting
- Message: Parsing Excel (33%)
- Entities: 0
- Relationships: 0
-Progress: extracting
- Message: Reading sheet: Products (2/3)
- Entities: 0
- Relationships: 0
-... (more progress updates)
-Progress: complete
- Message: Import complete
- Entities: 1523
- Relationships: 892
-Import complete: 1523 entities created
-```
-
----
-
-## 🎯 Universal Progress Handler (Works for ALL Formats)
-
-The examples below show format-specific messages, but **you don't need format-specific code**! The `ImportProgress` interface is the same for all formats:
-
-```typescript
-// ONE HANDLER FOR ALL FORMATS!
-function universalProgressHandler(progress) {
- console.log(`[${progress.stage}] ${progress.message}`)
-
- if (progress.processed && progress.total) {
- console.log(` Progress: ${progress.processed}/${progress.total}`)
- }
-
- if (progress.entities || progress.relationships) {
- console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
- }
-
- if (progress.throughput && progress.eta) {
- console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
- }
-}
-
-// Use it for ANY format!
-await brain.import(csvBuffer, { onProgress: universalProgressHandler })
-await brain.import(pdfBuffer, { onProgress: universalProgressHandler })
-await brain.import(excelBuffer, { onProgress: universalProgressHandler })
-await brain.import(jsonBuffer, { onProgress: universalProgressHandler })
-await brain.import(markdownString, { onProgress: universalProgressHandler })
-await brain.import(yamlBuffer, { onProgress: universalProgressHandler })
-await brain.import(docxBuffer, { onProgress: universalProgressHandler })
-```
-
----
-
-## 📊 What Different Formats Look Like (Same Handler!)
-
-The examples below show **what messages look like** for different formats using the **same universal handler** above.
-
-### CSV Import (Row-by-Row Progress)
-
-```typescript
-await brain.import(csvBuffer, {
- format: 'csv',
- onProgress: (progress) => {
- if (progress.stage === 'extracting') {
- // CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
- console.log(progress.message)
- }
- }
-})
-```
-
-**CSV Progress Messages:**
-- ✅ "Detecting CSV encoding and delimiter..."
-- ✅ "Parsing CSV rows (delimiter: ",")"
-- ✅ "Parsed 75%" (via bytes processed)
-- ✅ "Extracted 1000 rows"
-- ✅ "Converting types: 5000/10000 rows..."
-- ✅ "CSV processing complete: 10000 rows"
-
----
-
-### PDF Import (Page-by-Page Progress)
-
-```typescript
-await brain.import(pdfBuffer, {
- format: 'pdf',
- onProgress: (progress) => {
- // PDF reports exact page numbers
- console.log(progress.message)
- // Example: "Processing page 5 of 23"
- }
-})
-```
-
-**PDF Progress Messages:**
-- ✅ "Loading PDF document..."
-- ✅ "Processing 23 pages..."
-- ✅ "Processing page 5 of 23"
-- ✅ "Parsed 22%" (via bytes processed)
-- ✅ "Extracted 156 items from PDF"
-- ✅ "PDF complete: 23 pages, 156 items extracted"
-
----
-
-### Excel Import (Sheet-by-Sheet Progress)
-
-```typescript
-await brain.import(excelBuffer, {
- format: 'excel',
- onProgress: (progress) => {
- // Excel reports sheet names
- console.log(progress.message)
- // Example: "Reading sheet: Q2 Sales (2/5)"
- }
-})
-```
-
-**Excel Progress Messages:**
-- ✅ "Loading Excel workbook..."
-- ✅ "Processing 3 sheets..."
-- ✅ "Reading sheet: Sales (1/3)"
-- ✅ "Parsing Excel (33%)" (via bytes processed)
-- ✅ "Extracted 5234 rows from Excel"
-- ✅ "Excel complete: 3 sheets, 5234 rows"
-
----
-
-### JSON Import (Node Traversal)
-
-```typescript
-await brain.import(jsonBuffer, {
- format: 'json',
- onProgress: (progress) => {
- // JSON reports every 10 nodes
- console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
- }
-})
-```
-
----
-
-### Markdown Import (Section-by-Section)
-
-```typescript
-await brain.import(markdownString, {
- format: 'markdown',
- onProgress: (progress) => {
- console.log(`Section ${progress.processed}/${progress.total}`)
- }
-})
-```
-
----
-
-## 🎯 Building Progress UI Components
-
-### React Progress Bar
-
-```typescript
-function ImportProgress({ file }: { file: File }) {
- const [progress, setProgress] = useState({
- stage: 'idle',
- message: '',
- percent: 0,
- entities: 0,
- relationships: 0
- })
-
- const handleImport = async () => {
- const buffer = await file.arrayBuffer()
-
- await brain.import(Buffer.from(buffer), {
- onProgress: (p) => {
- setProgress({
- stage: p.stage,
- message: p.message,
- // Estimate percentage from stage
- percent: {
- detecting: 10,
- extracting: 50,
- 'storing-vfs': 80,
- 'storing-graph': 90,
- complete: 100
- }[p.stage] || 0,
- entities: p.entities || 0,
- relationships: p.relationships || 0
- })
- }
- })
- }
-
- return (
- <div>
- <ProgressBar value={progress.percent} />
- <p>{progress.message}</p>
- <p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
- </div>
- )
-}
-```
-
----
-
-### CLI Progress Spinner
-
-```typescript
-import ora from 'ora'
-
-const spinner = ora('Starting import...').start()
-
-await brain.import(buffer, {
- onProgress: (progress) => {
- spinner.text = progress.message
-
- if (progress.stage === 'complete') {
- spinner.succeed(`Import complete: ${progress.entities} entities`)
- }
- }
-})
-```
-
-**CLI Output:**
-```
-⠋ Detecting format...
-⠙ Loading Excel workbook...
-⠹ Reading sheet: Sales (1/3)
-⠸ Parsing Excel (33%)
-⠼ Reading sheet: Products (2/3)
-...
-✔ Import complete: 1523 entities
-```
-
----
-
-### Progress Dashboard with ETA
-
-```typescript
-let startTime = Date.now()
-let lastUpdate = startTime
-
-await brain.import(buffer, {
- onProgress: (progress) => {
- const elapsed = Date.now() - startTime
- const rate = progress.entities / (elapsed / 1000) // entities/sec
-
- console.clear()
- console.log('Import Progress Dashboard')
- console.log('========================')
- console.log(`Stage: ${progress.stage}`)
- console.log(`Status: ${progress.message}`)
- console.log(`Entities: ${progress.entities}`)
- console.log(`Relationships: ${progress.relationships}`)
- console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
- console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
- }
-})
-```
-
----
-
-## 🔧 Advanced: Format-Specific Optimization
-
-### Detecting Format to Show Appropriate Progress
-
-```typescript
-const formatMessages = {
- csv: (p) => `CSV: ${p.message}`,
- pdf: (p) => `PDF: ${p.message}`,
- excel: (p) => `Excel: ${p.message}`,
- json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
- markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
- yaml: (p) => `YAML: ${p.processed} nodes`,
- docx: (p) => `DOCX: ${p.processed} paragraphs`
-}
-
-await brain.import(buffer, {
- onProgress: (progress) => {
- // Format is available in progress.stage metadata
- const message = formatMessages[detectedFormat]?.(progress) || progress.message
- console.log(message)
- }
-})
-```
-
----
-
-## ⚡ Performance Tips
-
-### Throttle UI Updates
-
-```typescript
-let lastUIUpdate = 0
-const THROTTLE_MS = 100 // Update UI max once per 100ms
-
-await brain.import(buffer, {
- onProgress: (progress) => {
- const now = Date.now()
- if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
- return // Skip this update
- }
-
- lastUIUpdate = now
- updateUI(progress) // Only update every 100ms
- }
-})
-```
-
-**Note:** Brainy already throttles progress callbacks internally, but additional UI throttling can help with heavy rendering.
-
----
-
-## 📝 Summary
-
-✅ **All 7 formats** have consistent progress reporting
-✅ **Real-time updates** during long imports (no more "0%" hangs)
-✅ **Contextual messages** show exactly what's happening
-✅ **Build reliable tools** with standardized progress callbacks
-✅ **Problem SOLVED** - users see progress throughout import
-
-**Files Modified:**
-- 3 handlers: `csvHandler.ts`, `pdfHandler.ts`, `excelHandler.ts`
-- 7 importers: `SmartCSVImporter.ts`, `SmartPDFImporter.ts`, `SmartExcelImporter.ts`, `SmartJSONImporter.ts`, `SmartMarkdownImporter.ts`, `SmartYAMLImporter.ts`, `SmartDOCXImporter.ts`
-
-**Result:** Comprehensive, consistent progress tracking across ALL import formats!

docs/guides/import-progress-implementation.md

@@ -1,734 +0,0 @@
-# Import Progress Implementation Guide
-**For Developers: How to Add Progress Tracking to ANY File Handler**
-
-> This guide shows the **standard pattern** for implementing rich progress tracking in Brainy import handlers. Follow this template for **all 7 supported formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX) or any future file format.
-
----
-
-## 📊 Supported Formats & Consistent Progress Reporting
-
-> **⚠️ IMPORTANT FOR DEVELOPERS:** The public API (`ImportProgress`) is 100% standardized across all formats. You can build ONE progress handler that works for CSV, PDF, Excel, JSON, Markdown, YAML, and DOCX with **zero format-specific code**. See [Standard Import Progress API](./standard-import-progress.md) for details.
-
-**ALL 7 formats now have consistent, standardized progress reporting** for building reliable import tools:
-
-| Format | Category | Progress Points | File Location | Status |
-|--------|----------|-----------------|---------------|--------|
-| **CSV** | Tabular | Parsing → Row extraction → Type conversion → Complete | `handlers/csvHandler.ts` + `SmartCSVImporter.ts` | ✅ Complete |
-| **PDF** | Document | Loading → Page-by-page → Item extraction → Complete | `handlers/pdfHandler.ts` + `SmartPDFImporter.ts` | ✅ Complete |
-| **Excel** | Tabular | Loading → Sheet-by-sheet → Row extraction → Type conversion → Complete | `handlers/excelHandler.ts` + `SmartExcelImporter.ts` | ✅ Complete |
-| **JSON** | Structured | Parsing → Node traversal (every 10 nodes) → Complete | `SmartJSONImporter.ts` | ✅ Complete |
-| **Markdown** | Document | Parsing → Section-by-section → Complete | `SmartMarkdownImporter.ts` | ✅ Complete |
-| **YAML** | Structured | Parsing → Node traversal (every 10 nodes) → Complete | `SmartYAMLImporter.ts` | ✅ Complete |
-| **DOCX** | Document | Parsing → Paragraph-by-paragraph (every 10) → Complete | `SmartDOCXImporter.ts` | ✅ Complete |
-
-### The Standard Public API
-
-**Developers calling `brain.import()` see ONE standardized interface** regardless of format:
-
-```typescript
-// THE PUBLIC API - Same for ALL 7 formats!
-brain.import(buffer, {
- onProgress: (progress: ImportProgress) => {
- // These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
- progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
- progress.message // Human-readable status (varies by format, always readable)
- progress.processed // Items processed (optional)
- progress.total // Total items (optional)
- progress.entities // Entities extracted (optional)
- progress.relationships // Relationships inferred (optional)
- progress.throughput // Items/sec (optional, during extraction)
- progress.eta // Time remaining in ms (optional)
- }
-})
-```
-
-**Internal Implementation** (for developers adding new format handlers):
-
-The table below shows how formats implement progress *internally*. Normal developers don't need to know this - they just use the standard `ImportProgress` interface above!
-
-```typescript
-// Internal: Binary formats use handler hooks (you added these!)
-interface FormatHandlerProgressHooks {
- onBytesProcessed?: (bytes: number) => void
- onCurrentItem?: (message: string) => void
- onDataExtracted?: (count: number, total?: number) => void
-}
-
-// Internal: Text formats use importer callbacks
-interface ImporterProgressCallback {
- onProgress?: (stats: { processed, total, entities, relationships }) => void
-}
-
-// Both are converted to ImportProgress by ImportCoordinator!
-```
-
-### Developer Benefits
-
-✅ **Consistent API** - Same pattern across all 7 formats
-✅ **Throttled Updates** - Progress reported every 10-1000 items (no spam)
-✅ **Contextual Messages** - "Processing page 5 of 23", "Reading sheet: Sales (2/5)"
-✅ **Real-time Estimates** - Users see progress during long imports
-✅ **Build Monitoring Tools** - Reliable progress data for UIs, dashboards, CLI tools
-
----
-
-## 🎯 Overview
-
-Brainy supports comprehensive, multi-dimensional progress tracking for imports:
-- **Bytes processed** (always available, most deterministic)
-- **Entities extracted** (AI extraction phase)
-- **Stage-specific metrics** (parsing: MB/s, extraction: entities/s)
-- **Time estimates** (remaining time, total time)
-- **Context information** ("Processing page 5 of 23")
-
-All handlers follow a simple, consistent pattern using **progress hooks**.
-
----
-
-## 📋 The Progress Hooks Pattern
-
-### 1. Progress Hooks Interface
-
-```typescript
-export interface FormatHandlerProgressHooks {
- /**
- * Report bytes processed
- * Call this as you read/parse the file
- */
- onBytesProcessed?: (bytes: number) => void
-
- /**
- * Set current processing context
- * Examples: "Processing page 5", "Reading sheet: Q2 Sales"
- */
- onCurrentItem?: (item: string) => void
-
- /**
- * Report structured data extraction progress
- * Examples: "Extracted 100 rows", "Parsed 50 paragraphs"
- */
- onDataExtracted?: (count: number, total?: number) => void
-}
-```
-
-### 2. Handler Options (Automatic)
-
-Progress hooks are automatically passed to your handler via `FormatHandlerOptions`:
-
-```typescript
-export interface FormatHandlerOptions {
- // ... existing options ...
-
- /**
- * Progress hooks
- * Handlers call these to report progress during processing
- */
- progressHooks?: FormatHandlerProgressHooks
-
- /**
- * Total file size in bytes
- * Used for progress percentage calculation
- */
- totalBytes?: number
-}
-```
-
-**You don't need to modify FormatHandlerOptions** - it's already done!
-
-### 3. Standard Implementation Pattern
-
-Every handler follows these 5 steps:
-
-```typescript
-async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
- const progressHooks = options.progressHooks // Step 1: Get hooks
-
- // Step 2: Report initial progress
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem('Starting import...')
- }
-
- // Step 3: Report bytes as you process
- const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(0) // Start
- }
-
- // ... do parsing ...
-
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(buffer.length) // Complete
- }
-
- // Step 4: Report data extraction
- if (progressHooks?.onDataExtracted) {
- progressHooks.onDataExtracted(data.length, data.length)
- }
-
- // Step 5: Report completion
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Complete: ${data.length} items processed`)
- }
-
- return { format, data, metadata }
-}
-```
-
----
-
-## 📚 Complete Example: CSV Handler
-
-Here's the **ACTUAL implementation** from CSV handler showing all the key progress points:
-
-```typescript
-async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
- const startTime = Date.now()
- const progressHooks = options.progressHooks // ✅ Step 1
-
- // Convert to buffer if string
- const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
- const totalBytes = buffer.length
-
- // ✅ Step 2: Report start
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(0)
- }
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...')
- }
-
- // Detect encoding
- const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
- const text = buffer.toString(detectedEncoding as BufferEncoding)
-
- // Detect delimiter
- const delimiter = options.csvDelimiter || this.detectDelimiter(text)
-
- // ✅ Progress update: Parsing phase
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
- }
-
- // Parse CSV
- const records = parse(text, { /* options */ })
-
- // ✅ Step 3: Report bytes processed (entire file parsed)
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(totalBytes)
- }
-
- const data = Array.isArray(records) ? records : [records]
-
- // ✅ Step 4: Report data extraction
- if (progressHooks?.onDataExtracted) {
- progressHooks.onDataExtracted(data.length, data.length)
- }
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`)
- }
-
- // Type inference and conversion
- const fields = data.length > 0 ? Object.keys(data[0]) : []
- const types = this.inferFieldTypes(data)
-
- const convertedData = data.map((row, index) => {
- const converted = this.convertRow(row, types)
-
- // ✅ Progress update every 1000 rows (avoid spam)
- if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
- progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
- }
-
- return converted
- })
-
- // ✅ Step 5: Report completion
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
- }
-
- return {
- format: this.format,
- data: convertedData,
- metadata: { /* ... */ }
- }
-}
-```
-
-### Key Progress Points in CSV Handler
-
-| Progress Point | Hook Used | Message Example |
-|----------------|-----------|-----------------|
-| **Start** | `onCurrentItem` | "Detecting CSV encoding and delimiter..." |
-| **Start bytes** | `onBytesProcessed(0)` | 0 bytes |
-| **Parsing** | `onCurrentItem` | "Parsing CSV rows (delimiter: \",\")..." |
-| **Bytes complete** | `onBytesProcessed(totalBytes)` | All bytes read |
-| **Data extracted** | `onDataExtracted(count, total)` | Number of rows extracted |
-| **Type conversion** | `onCurrentItem` (every 1000 rows) | "Converting types: 5000/10000 rows..." |
-| **Complete** | `onCurrentItem` | "CSV processing complete: 10000 rows" |
-
----
-
-## 📖 Implementation Guide by File Type
-
-### Supported Formats
-
-Brainy supports **7 file formats** with full progress tracking:
-
-**Binary Formats** (use handlers):
-1. **CSV** - Row-by-row parsing with type inference
-2. **PDF** - Page-by-page extraction with table detection
-3. **Excel** - Sheet-by-sheet processing with formula evaluation
-
-**Text/Structured Formats** (parse inline):
-4. **JSON** - Recursive traversal of nested structures
-5. **Markdown** - Section-by-section with heading extraction
-6. **YAML** - Hierarchical traversal with relationship inference
-7. **DOCX** - Paragraph-by-paragraph with structure analysis
-
----
-
-### PDF Handler (Multi-Page)
-
-```typescript
-async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
- const progressHooks = options.progressHooks
- const totalBytes = data.length
-
- // Report start
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem('Loading PDF document...')
- }
-
- const pdfDoc = await loadPDF(data)
- const totalPages = pdfDoc.numPages
-
- const extractedData: any[] = []
- let bytesProcessed = 0
-
- for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
- // ✅ Report current page
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`)
- }
-
- const page = await pdfDoc.getPage(pageNum)
- const text = await page.getTextContent()
- extractedData.push(this.processPageText(text))
-
- // ✅ Estimate bytes processed (pages are sequential)
- bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes)
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(bytesProcessed)
- }
-
- // ✅ Report extraction progress
- if (progressHooks?.onDataExtracted) {
- progressHooks.onDataExtracted(pageNum, totalPages)
- }
- }
-
- // Final progress
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(totalBytes)
- }
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`)
- }
-
- return { format: 'pdf', data: extractedData, metadata: { /* ... */ } }
-}
-```
-
-### Excel Handler (Multi-Sheet)
-
-```typescript
-async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
- const progressHooks = options.progressHooks
- const totalBytes = data.length
-
- // Load workbook
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem('Loading Excel workbook...')
- }
-
- const workbook = XLSX.read(data)
- const sheetNames = options.excelSheets === 'all'
- ? workbook.SheetNames
- : (options.excelSheets || [workbook.SheetNames[0]])
-
- const allData: any[] = []
- let bytesProcessed = 0
-
- for (let i = 0; i < sheetNames.length; i++) {
- const sheetName = sheetNames[i]
-
- // ✅ Report current sheet
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`)
- }
-
- const sheet = workbook.Sheets[sheetName]
- const sheetData = XLSX.utils.sheet_to_json(sheet)
- allData.push(...sheetData)
-
- // ✅ Estimate bytes processed (sheets processed sequentially)
- bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes)
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(bytesProcessed)
- }
-
- // ✅ Report data extraction
- if (progressHooks?.onDataExtracted) {
- progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done
- }
- }
-
- // Final progress
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(totalBytes)
- }
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`)
- }
-
- return { format: 'xlsx', data: allData, metadata: { /* ... */ } }
-}
-```
-
-### JSON Importer (Recursive Traversal)
-
-```typescript
-async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResult> {
- // ✅ Report parsing start
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Parse JSON if string
- let jsonData = typeof data === 'string' ? JSON.parse(data) : data
-
- // ✅ Report parsing complete
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Traverse and extract (reports progress every 10 nodes)
- const entities: ExtractedJSONEntity[] = []
- const relationships: ExtractedJSONRelationship[] = []
- let nodesProcessed = 0
-
- await this.traverseJSON(
- jsonData,
- entities,
- relationships,
- () => {
- nodesProcessed++
- if (nodesProcessed % 10 === 0) {
- options.onProgress?.({
- processed: nodesProcessed,
- entities: entities.length,
- relationships: relationships.length
- })
- }
- }
- )
-
- // ✅ Report completion
- options.onProgress?.({
- processed: nodesProcessed,
- entities: entities.length,
- relationships: relationships.length
- })
-
- return { nodesProcessed, entitiesExtracted: entities.length, ... }
-}
-```
-
-### Markdown Importer (Section-Based)
-
-```typescript
-async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<SmartMarkdownResult> {
- // ✅ Report parsing start
- options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 })
-
- // Parse markdown into sections
- const parsedSections = this.parseMarkdown(markdown, options)
-
- // ✅ Report parsing complete
- options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 })
-
- // Process each section (reports progress after each section)
- const sections: MarkdownSection[] = []
- for (let i = 0; i < parsedSections.length; i++) {
- const section = await this.processSection(parsedSections[i], options)
- sections.push(section)
-
- options.onProgress?.({
- processed: i + 1,
- total: parsedSections.length,
- entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
- relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
- })
- }
-
- // ✅ Report completion
- options.onProgress?.({
- processed: sections.length,
- total: sections.length,
- entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
- relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
- })
-
- return { sectionsProcessed: sections.length, ... }
-}
-```
-
-### YAML Importer (Hierarchical)
-
-```typescript
-async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Promise<SmartYAMLResult> {
- // ✅ Report parsing start
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Parse YAML
- const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8')
- const data = yaml.load(yamlString)
-
- // ✅ Report parsing complete
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Traverse YAML structure (reports progress every 10 nodes)
- // ... similar to JSON traversal ...
-
- // ✅ Report completion (already implemented)
- options.onProgress?.({
- processed: nodesProcessed,
- entities: entities.length,
- relationships: relationships.length
- })
-
- return { nodesProcessed, entitiesExtracted: entities.length, ... }
-}
-```
-
-### DOCX Importer (Paragraph-Based)
-
-```typescript
-async extract(buffer: Buffer, options: SmartDOCXOptions = {}): Promise<SmartDOCXResult> {
- // ✅ Report parsing start
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Extract text and HTML using Mammoth
- const textResult = await mammoth.extractRawText({ buffer })
- const htmlResult = await mammoth.convertToHtml({ buffer })
-
- // ✅ Report parsing complete
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Process paragraphs (reports progress every 10 paragraphs)
- const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength)
-
- for (let i = 0; i < paragraphs.length; i++) {
- await this.processParagraph(paragraphs[i])
-
- if (i % 10 === 0) {
- options.onProgress?.({
- processed: i + 1,
- entities: entities.length,
- relationships: relationships.length
- })
- }
- }
-
- // ✅ Report completion (already implemented)
- options.onProgress?.({
- processed: paragraphs.length,
- entities: entities.length,
- relationships: relationships.length
- })
-
- return { paragraphsProcessed: paragraphs.length, ... }
-}
-```
-
----
-
-## 🎯 Best Practices
-
-### 1. Always Check if Hooks Exist
-
-Progress hooks are **optional**. Always check before calling:
-
-```typescript
-// ✅ Good - safe
-if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(bytes)
-}
-
-// ❌ Bad - will crash if hooks undefined
-progressHooks.onBytesProcessed(bytes) // TypeError!
-```
-
-### 2. Report Bytes at Start and End
-
-```typescript
-// ✅ Good - clear start and end
-progressHooks?.onBytesProcessed(0) // Start
-// ... processing ...
-progressHooks?.onBytesProcessed(totalBytes) // End
-
-// ❌ Bad - no clear boundaries
-// ... just start processing without reporting start
-```
-
-### 3. Throttle Frequent Updates
-
-```typescript
-// ✅ Good - report every 1000 items
-for (let i = 0; i < items.length; i++) {
- processItem(items[i])
-
- if (i > 0 && i % 1000 === 0) {
- progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`)
- }
-}
-
-// ❌ Bad - report EVERY item (spam!)
-for (let i = 0; i < items.length; i++) {
- processItem(items[i])
- progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks!
-}
-```
-
-### 4. Provide Contextual Messages
-
-```typescript
-// ✅ Good - specific and helpful
-progressHooks?.onCurrentItem('Parsing CSV rows (delimiter: ",")')
-progressHooks?.onCurrentItem('Processing page 5 of 23')
-progressHooks?.onCurrentItem('Reading sheet: Q2 Sales Data')
-
-// ❌ Bad - vague
-progressHooks?.onCurrentItem('Processing...')
-progressHooks?.onCurrentItem('Working...')
-```
-
-### 5. Report Data Extraction with Totals (if known)
-
-```typescript
-// ✅ Good - total known
-progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows
-
-// ✅ Also good - total unknown (streaming)
-progressHooks?.onDataExtracted(100, undefined) // 100 rows so far
-
-// ✅ Also good - complete
-progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows
-```
-
----
-
-## 🔧 Testing Your Handler
-
-### Manual Test
-
-```typescript
-import { CSVHandler } from './csvHandler.js'
-import * as fs from 'fs'
-
-const handler = new CSVHandler()
-const data = fs.readFileSync('./test.csv')
-
-const result = await handler.process(data, {
- filename: 'test.csv',
- progressHooks: {
- onBytesProcessed: (bytes) => {
- console.log(`Bytes: ${bytes}`)
- },
- onCurrentItem: (item) => {
- console.log(`Status: ${item}`)
- },
- onDataExtracted: (count, total) => {
- console.log(`Extracted: ${count}${total ? `/${total}` : ''}`)
- }
- }
-})
-
-console.log(`Complete: ${result.data.length} rows`)
-```
-
-### Expected Output
-
-```
-Status: Detecting CSV encoding and delimiter...
-Bytes: 0
-Status: Parsing CSV rows (delimiter: ",")...
-Bytes: 52438
-Extracted: 1000/1000
-Status: Extracted 1000 rows, inferring types...
-Status: CSV processing complete: 1000 rows
-Complete: 1000 rows
-```
-
----
-
-## 📊 Progress Flow Diagram
-
-```
-User Imports File
- ↓
-ImportManager
- ↓
-Creates ProgressTracker
- ↓
-Calls Handler.process() with progressHooks
- ↓
-Handler Reports Progress:
- ├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated
- ├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated
- ├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated
- ├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated
- └─ onCurrentItem("Complete") → ProgressTracker → final progress
- ↓
-ProgressTracker emits to callback (throttled 100ms)
- ↓
-User sees:
- "Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..."
-```
-
----
-
-## ✅ Checklist for New Handlers
-
-When implementing a new file format handler:
-
-- [ ] Get `progressHooks` from `options`
-- [ ] Get `totalBytes` (if available)
-- [ ] Report `onBytesProcessed(0)` at start
-- [ ] Report `onCurrentItem()` for key stages
-- [ ] Report `onBytesProcessed()` as you process
-- [ ] Report `onDataExtracted()` when you extract data
-- [ ] Throttle frequent updates (every 1000 items max)
-- [ ] Report `onBytesProcessed(totalBytes)` at end
-- [ ] Report final `onCurrentItem()` with summary
-- [ ] Test with progress callback to verify output
-
----
-
-## 🎓 Summary
-
-**The Pattern (5 Steps)**:
-1. Get `progressHooks` from options
-2. Report start (`onBytesProcessed(0)`, `onCurrentItem("Starting...")`)
-3. Report progress as you process (`onBytesProcessed(bytes)`, `onCurrentItem("Page 5...")`)
-4. Report data extraction (`onDataExtracted(count, total)`)
-5. Report completion (`onBytesProcessed(totalBytes)`, `onCurrentItem("Complete")`)
-
-**Always Check**: `progressHooks?.method()`
-
-**Throttle**: Report every N items, not every single item
-
-**Context**: Provide specific, helpful messages
-
-**Testing**: Use manual test with console.log callbacks
-
----
-
-**This pattern makes it trivial to add progress tracking to ANY file format. Copy this template and adapt for your handler!**

docs/guides/import-quick-reference.md

@@ -1,461 +0,0 @@
-# 📥 Import Quick Reference
-
-> **Quick guide to importing data into Brainy**
-
----
-
-## Basic Import
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// Import from file path
-await brain.import('/path/to/data.xlsx')
-
-// Import from buffer
-const buffer = fs.readFileSync('data.csv')
-await brain.import(buffer)
-
-// Import from object
-const jsonData = { items: [...] }
-await brain.import(jsonData)
-```
-
----
-
-## Supported Formats
-
-| Format | Extensions | Auto-Detect |
-|--------|------------|-------------|
-| **Excel** | `.xlsx`, `.xls` | ✅ Yes |
-| **CSV** | `.csv` | ✅ Yes |
-| **JSON** | `.json` | ✅ Yes |
-| **Markdown** | `.md` | ✅ Yes |
-| **PDF** | `.pdf` | ✅ Yes |
-| **YAML** | `.yaml`, `.yml` | ✅ Yes |
-| **DOCX** | `.docx` | ✅ Yes |
-
----
-
-## Common Options
-
-### Basic Options
-
-```typescript
-await brain.import(file, {
-  // Specify format (optional - auto-detects by default)
-  format: 'excel',
-
-  // VFS destination path
-  vfsPath: '/imports/products',
-
-  // Enable/disable features
-  createEntities: true,           // Create graph entities (default: true)
-  createRelationships: true,      // Create relationships (default: true)
-  preserveSource: true,           // Keep original file (default: true)
-
-  // Progress tracking
-  onProgress: (progress) => {
-    console.log(`${progress.processed}/${progress.total}`)
-  }
-})
-```
-
-### Neural Intelligence
-
-```typescript
-await brain.import(file, {
-  // Entity type classification
-  enableNeuralExtraction: true,      // Auto-classify entity types (default: true)
-
-  // Relationship type inference
-  enableRelationshipInference: true, // Auto-infer relationship types (default: true)
-
-  // Concept extraction
-  enableConceptExtraction: true,     // Extract key concepts (default: true)
-
-  // Confidence threshold
-  confidenceThreshold: 0.6           // Min confidence for extraction (default: 0.6)
-})
-```
-
-### Deduplication
-
-```typescript
-await brain.import(file, {
-  enableDeduplication: true,         // Check for duplicates (default: true)
-  deduplicationThreshold: 0.85       // Similarity threshold (default: 0.85)
-})
-```
-
-Deduplication merges entities judged duplicates — the non-primary records are
-**deleted**. Set `enableDeduplication: false` to disable it entirely: the flag
-gates both the inline merge during import and the background pass that runs
-about 5 minutes after the last import.
-
-```typescript
-await brain.import(file, {
-  enableDeduplication: false         // No merging, inline or background
-})
-```
-
-### Import Tracking
-
-Track and organize imports by project:
-
-```typescript
-await brain.import(file, {
-  projectId: 'worldbuilding',        // Group related imports
-  importId: 'import-001',            // Custom ID (auto-generated if not provided)
-  customMetadata: {                  // Additional metadata
-    campaign: 'fall-2024',
-    author: 'gamemaster'
-  }
-})
-
-// Query all entities in a project
-const entities = await brain.find({
-  where: { projectId: 'worldbuilding' }
-})
-
-// Query entities from specific import
-const importedEntities = await brain.find({
-  where: { importIds: { $includes: 'import-001' } }
-})
-
-// Exclude a project from search
-const results = await brain.find({
-  query: 'dragon',
-  where: { projectId: { $ne: 'archived-project' } }
-})
-```
-
-**All created items (entities, relationships, VFS files) are automatically tagged with:**
-- `importIds: string[]` - Import operation IDs
-- `projectId: string` - Project identifier
-- `importedAt: number` - Timestamp
-- `importFormat: string` - Format type ('excel', 'csv', etc.)
-- `importSource: string` - Source filename/URL
-
-### VFS Organization
-
-```typescript
-await brain.import(file, {
-  vfsPath: '/imports/catalog',
-
-  // Grouping strategy
-  groupBy: 'type',  // Group by entity type (default)
-  // OR
-  groupBy: 'sheet', // Group by Excel sheet name
-  // OR
-  groupBy: 'flat',  // All entities in root directory
-  // OR
-  groupBy: 'custom',
-  customGrouping: (entity) => {
-    return `/by-category/${entity.category}`
-  }
-})
-```
-
-### Always-On Streaming
-
-All imports use streaming with adaptive flush intervals. Query data as it's imported:
-
-```typescript
-await brain.import(file, {
-  onProgress: async (progress) => {
-    // Query data during import
-    if (progress.queryable) {
-      const products = await brain.find({ type: 'product', limit: 10000 })
-      console.log(`${products.length} products imported so far`)
-    }
-  }
-})
-```
-
-**Progressive intervals** (automatic):
-- 0-999 entities: Flush every 100 (frequent early updates)
-- 1K-9.9K: Flush every 1000 (balanced)
-- 10K+: Flush every 5000 (minimal overhead)
-- Adjusts dynamically as import grows
-
----
-
-## Complete Example
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-import * as fs from 'fs'
-
-async function importCatalog() {
-  const brain = new Brainy({
-    storage: {
-      type: 'gcs',
-      bucket: 'my-bucket',
-      prefix: 'brainy/'
-    }
-  })
-  await brain.init()
-
-  const buffer = fs.readFileSync('catalog.xlsx')
-
-  const result = await brain.import(buffer, {
-    format: 'excel',
-    vfsPath: '/imports/product-catalog',
-    groupBy: 'type',
-
-    // Neural intelligence
-    enableNeuralExtraction: true,
-    enableRelationshipInference: true,
-    confidenceThreshold: 0.7,
-
-    // Deduplication
-    enableDeduplication: true,
-    deduplicationThreshold: 0.85,
-
-    // Progress tracking (streaming always enabled)
-    onProgress: async (progress) => {
-      console.log(`Stage: ${progress.stage}`)
-      console.log(`Progress: ${progress.processed}/${progress.total}`)
-
-      // Query live data (available after each flush)
-      if (progress.queryable) {
-        const products = await brain.find({ type: 'product', limit: 100000 })
-        const people = await brain.find({ type: 'person', limit: 100000 })
-        const all = await brain.find({ limit: 100000 })
-
-        const stats = {
-          products: products.length,
-          people: people.length,
-          total: all.length
-        }
-        console.log('Current counts:', stats)
-      }
-    }
-  })
-
-  console.log('Import complete!')
-  console.log(`Entities: ${result.entities.length}`)
-  console.log(`Relationships: ${result.relationships.length}`)
-  console.log(`VFS path: ${result.vfs.rootPath}`)
-  console.log(`Processing time: ${result.stats.processingTime}ms`)
-
-  return result
-}
-
-importCatalog()
-```
-
----
-
-## Import Result
-
-```typescript
-interface ImportResult {
-  importId: string
-  format: string
-  formatConfidence: number
-
-  vfs: {
-    rootPath: string
-    directories: string[]
-    files: Array<{
-      path: string
-      entityId?: string
-      type: 'entity' | 'metadata' | 'source' | 'relationships'
-    }>
-  }
-
-  entities: Array<{
-    id: string
-    name: string
-    type: NounType
-    vfsPath?: string
-  }>
-
-  relationships: Array<{
-    id: string
-    from: string
-    to: string
-    type: VerbType
-  }>
-
-  stats: {
-    entitiesExtracted: number
-    relationshipsInferred: number
-    vfsFilesCreated: number
-    graphNodesCreated: number
-    graphEdgesCreated: number
-    entitiesMerged: number        // From deduplication
-    entitiesNew: number           // Newly created
-    processingTime: number        // In milliseconds
-  }
-}
-```
-
----
-
-## Progress Callback
-
-```typescript
-interface ImportProgress {
-  stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
-  message: string
-  processed?: number              // Current item number
-  total?: number                  // Total items
-  entities?: number               // Entities extracted
-  relationships?: number          // Relationships inferred
-  throughput?: number             // Rows per second
-  eta?: number                    // Estimated time remaining (ms)
-  queryable?: boolean             // Data queryable now (streaming mode)
-}
-```
-
----
-
-## Tips & Best Practices
-
-### Performance
-
-```typescript
-// Streaming is always on with adaptive intervals (zero config)
-// - Small imports (<1K): Flush every 100 entities
-// - Medium (1K-10K): Flush every 1000 entities
-// - Large (>10K): Flush every 5000 entities
-
-// Disable features you don't need for faster imports
-await brain.import(file, {
-  enableNeuralExtraction: false,      // 10x faster
-  enableRelationshipInference: false, // 5x faster
-  enableConceptExtraction: false      // 2x faster
-})
-```
-
-### Error Handling
-
-```typescript
-try {
-  const result = await brain.import(file, {
-    vfsPath: '/imports/data',
-    onProgress: (p) => console.log(p.message)
-  })
-  console.log('Success:', result.stats)
-} catch (error) {
-  console.error('Import failed:', error.message)
-
-  // Check partial results in VFS
-  const files = await brain.vfs().readdir('/imports')
-  console.log('Partial files:', files)
-}
-```
-
-### Querying Imported Data
-
-```typescript
-// After import completes
-const result = await brain.import(file)
-
-// Find entities by type
-const products = await brain.find({ type: 'Product' })
-
-// Get entity relationships
-const relations = await brain.related(products[0].id)
-
-// Search VFS
-const vfsFiles = await brain.vfs().find(result.vfs.rootPath + '/**/*.json')
-
-// Read entity from VFS
-const entity = await brain.vfs().readJSON(vfsFiles[0].path)
-```
-
----
-
-## Excel-Specific Tips
-
-### Column Detection
-
-Brainy auto-detects columns with flexible matching:
-
-| Your Column | Matches Pattern |
-|-------------|-----------------|
-| `Name` | term\|name\|title\|concept |
-| `Description` | definition\|description\|desc\|details |
-| `Type` | type\|category\|kind\|class |
-| `Related` | related\|see also\|links\|references |
-
-### Multiple Sheets
-
-All sheets are processed automatically:
-
-```typescript
-// catalog.xlsx with 3 sheets: Products, People, Places
-const result = await brain.import('catalog.xlsx', {
-  groupBy: 'sheet'  // Creates /Products/, /People/, /Places/
-})
-```
-
----
-
-## CSV-Specific Tips
-
-### Headers
-
-First row is treated as headers. Ensure headers exist:
-
-```csv
-Term,Definition,Type
-Product A,Description A,Product
-Product B,Description B,Product
-```
-
-### Large CSVs
-
-For large CSV files (>100K rows), streaming is automatic:
-
-```typescript
-await brain.import(largeCsv, {
-  // Automatically flushes every 5000 entities (adaptive)
-  enableNeuralExtraction: false  // Faster for large imports
-})
-```
-
----
-
-## JSON-Specific Tips
-
-### Supported Structures
-
-```javascript
-// Array of objects
-[
-  { name: "Item 1", type: "Product" },
-  { name: "Item 2", type: "Product" }
-]
-
-// Nested objects (creates hierarchical relationships)
-{
-  "company": {
-    "name": "Acme Corp",
-    "products": [
-      { "name": "Widget", "price": 9.99 }
-    ]
-  }
-}
-```
-
----
-
-## Further Reading
-
-- [Import Flow Guide](./import-flow.md) - Deep dive into how imports work
-- [Streaming Imports](./streaming-imports.md) - Progressive imports for large files
-- [VFS Guide](./vfs-guide.md) - Working with the virtual file system
-- [Type Classification](./type-classification.md) - How entity types are inferred
-- [Relationship Inference](./relationship-inference.md) - How relationships are classified
-
----
-
-**Questions?** Check the [FAQ](../faq.md) or [open an issue](https://github.com/soulcraft/brainy/issues)!

docs/guides/inspection.md

@@ -1,213 +0,0 @@
----
-title: Inspecting a Live Brainy
-slug: guides/inspection
-public: true
-category: guides
-template: guide
-order: 30
-description: Operator recipes for diagnosing a running Brainy data directory — counts, queries, query plans, health checks, and snapshots — without stopping the live writer.
-next:
-  - concepts/multi-process
----
-
-# Inspecting a Live Brainy
-
-When something is wrong in production, you need to see what's actually in the
-store. This guide covers the safe ways to query a running Brainy directory.
-
-## The cardinal rule
-
-**Never open a second writer on the same directory.** Filesystem storage will throw, and any other write path will corrupt the live writer's state. Use `Brainy.openReadOnly()` or the `brainy inspect` CLI instead.
-
-## The CLI is the fastest path
-
-```bash
-# What's in this brain?
-brainy inspect stats /data/brain
-
-# Find specific entities
-brainy inspect find /data/brain --type Event --where '{"status":"paid"}' --limit 20
-
-# Single entity by ID
-brainy inspect get /data/brain 0b7a9...
-
-# Why is this query returning empty?
-brainy inspect explain /data/brain --where '{"entityType":"booking"}'
-
-# Quick invariants
-brainy inspect health /data/brain
-
-# Random sample (no query needed)
-brainy inspect sample /data/brain --type Event --n 20
-
-# Tail new writes as they happen
-brainy inspect watch /data/brain --type Event
-
-# Save a snapshot
-brainy inspect backup /data/brain /backups/brain-$(date +%Y%m%d).tar
-```
-
-Every subcommand internally:
-
-1. Asks the live writer to flush via the cross-process RPC (skip with `--no-fresh`).
-2. Opens the data directory via `Brainy.openReadOnly()`.
-3. Runs the query.
-4. Closes cleanly.
-
-Results are JSON by default. Add `--pretty` for indented output.
-
-## When a query returns surprising results
-
-If `find()` returns `0` for a query you expect to match: run `inspect
-explain` first. It shows which index path will serve each `where` clause:
-
-```bash
-$ brainy inspect explain /data/brain --where '{"entityType":"booking","status":"paid"}'
-{
-  "query": { "where": { "entityType": "booking", "status": "paid" } },
-  "fieldPlan": [
-    { "field": "entityType", "path": "none", "notes": "No index entries for field..." },
-    { "field": "status", "path": "column-store", "notes": "O(log n) binary search..." }
-  ],
-  "warnings": [
-    "Field \"entityType\" has no index entries. find() will return [] silently."
-  ]
-}
-```
-
-The `"path": "none"` is the smoking gun. It means the field has no column
-store manifest and no sparse chunked index — so `find()` will return `[]`
-regardless of what's actually on disk. Likely causes:
-
-- The writer registered the field in memory but hasn't flushed. Run
-  `brain.requestFlush()` from the writer side, or use `brainy inspect
-  --fresh` (default).
-- The field name has a typo or wrong casing.
-- The field is genuinely absent from every entity.
-
-## Health checks
-
-`inspect health` runs a fixed battery of cheap invariant checks:
-
-```bash
-$ brainy inspect health /data/brain
-{
-  "overall": "warn",
-  "checks": [
-    { "name": "index-parity", "status": "pass", "message": "Vector (1851) and metadata (1851) agree." },
-    { "name": "field-registry", "status": "pass", "message": "23 fields registered for 1851 entities." },
-    { "name": "seeded-records", "status": "warn", "message": "15 entities tagged _seeded:true." },
-    { "name": "writer-heartbeat", "status": "pass", "message": "Writer healthy (PID 1774431...)." }
-  ]
-}
-```
-
-Each check returns `pass`, `warn`, or `fail`. The exit code is `2` when any
-check fails — useful for piping into monitoring or CI.
-
-## Programmatic inspection
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const reader = await Brainy.openReadOnly({
-  storage: { type: 'filesystem', path: '/data/brain' }
-})
-
-// Force the writer to flush before reading
-await reader.requestFlush({ timeoutMs: 5000 })
-
-// What's in there?
-const stats = await reader.stats()
-console.log(`${stats.entityCount} entities`, stats.entitiesByType)
-
-// Why is this query empty?
-const plan = await reader.explain({ where: { entityType: 'booking' } })
-for (const f of plan.fieldPlan) {
-  console.log(`${f.field} -> ${f.path}`)
-}
-
-// Run invariants
-const health = await reader.health()
-console.log(health.overall)
-
-await reader.close()
-```
-
-Every mutation method (`add`, `update`, `remove`, `relate`, `transact`,
-`restore`, ...) throws on a read-only instance with a clear message.
-
-## Backups
-
-`brainy inspect backup` asks the writer to flush first, then tars the
-directory. The snapshot reflects the writer's state at the moment of the
-flush:
-
-```bash
-brainy inspect backup /data/brain /backups/brain-2026-05-15.tar
-```
-
-For periodic backups (hourly, daily), schedule this via cron or your
-container scheduler. For point-in-time recovery, use the Db API's
-`db.persist(path)` — a self-contained hard-link snapshot that later writes
-can never alter, restorable with `brain.restore(path, { confirm: true })`.
-See [Snapshots & Time Travel](./snapshots-and-time-travel.md).
-
-## Comparing two stores
-
-`brainy inspect diff` returns a JSON summary of counts and a sample of
-entity IDs present in one but not the other. Useful when debugging
-replication or migrations:
-
-```bash
-brainy inspect diff /data/brain-prod /data/brain-staging
-```
-
-Sample-based — for a full diff, dump both with `inspect dump` and compare
-the JSONL.
-
-## Auditing graph-read truth
-
-`brain.auditGraph()` (8.6.0+) proves — or disproves — that relationship reads
-return canonical truth on a given brain, without mutating anything. It walks
-every stored relationship record, asks the same read path your application
-uses (`related()`, VFS `readdir`) with every visibility tier included, and
-classifies every discrepancy:
-
-```typescript
-const report = await brain.auditGraph()
-
-report.coherent              // true = related()/readdir can be trusted on this brain
-report.missingFromReadsCount // records the read path omits — stale index
-report.danglingEndpointsCount // relationships whose endpoint entity is gone
-report.readOnlyCount         // read-path edges with NO stored record — ghosts
-report.visibilityHiddenCount // internal/system edges hidden by design (not a fault)
-```
-
-Counts are always exact; the example lists (`missingFromReads`,
-`danglingEndpoints`, `readOnlyVerbIds`) are capped at `maxExamples`
-(default 100) and `truncatedExamples` says so when they are.
-
-Run it after any engine upgrade, restore, or migration. If it reports
-discrepancies, run `brain.repairIndex()` and audit again — a `coherent`
-report after the repair is the verified statement that the heal worked.
-Cost: one relationship-record walk plus one indexed read per distinct
-source entity — safe on a live brain.
-
-## Repairing a corrupted store
-
-If invariants fail and you suspect index corruption, `inspect repair`
-opens the store in writer mode and rebuilds all indexes from raw storage.
-**Stop the live writer first** — `repair` will throw if another writer
-holds the lock. Add `--force` only if you have personally verified the
-existing lock is stale.
-
-```bash
-brainy inspect repair /data/brain
-```
-
-## Multi-process safety summary
-
-See [concepts/multi-process](../concepts/multi-process.md) for the lock
-semantics, heartbeat behavior, and what's not yet enforced on cloud
-backends.

docs/guides/installation.md

@@ -1,89 +0,0 @@
----
-title: Installation
-slug: getting-started/installation
-public: true
-category: getting-started
-template: guide
-order: 1
-description: Install Brainy with npm, bun, yarn, or pnpm. Works in Node.js 22+ and Bun 1.0+ (server-only since 8.0). TypeScript included.
-next:
-  - getting-started/quick-start
-  - guides/storage-adapters
----
-
-# Installation
-
-## Requirements
-
-- **Node.js 22+** or **Bun 1.0+**
-- TypeScript is optional — Brainy ships with full type definitions
-
-## Install
-
-```bash
-npm install @soulcraft/brainy
-```
-
-Or with your preferred package manager:
-
-```bash
-bun add @soulcraft/brainy
-yarn add @soulcraft/brainy
-pnpm add @soulcraft/brainy
-```
-
-## Verify
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-console.log('Brainy ready.')
-```
-
-## Native Acceleration (Optional)
-
-For production workloads, add Cor for Rust-accelerated SIMD distance calculations and native embeddings:
-
-```bash
-npm install @soulcraft/cor
-```
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy({ plugins: ['@soulcraft/cor'] })
-await brain.init()  // native providers registered during init
-```
-
-Cor registers native (Rust/SIMD) vector, metadata, and graph engines behind the same Brainy `find()` API — no code changes, an optional dependency for production-scale workloads.
-
-## Server-only since 8.0
-
-Brainy 8.0 runs on Node.js 22+ and Bun 1.0+. Browser support (OPFS storage,
-Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line
-remains available on npm if you need it.
-
-## TypeScript
-
-Brainy ships with full TypeScript types. No `@types/` package needed:
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-const id = await brain.add({
-  data: 'Hello, Brainy',
-  type: NounType.Concept,
-  metadata: { created: Date.now() }
-})
-```
-
-## Next Steps
-
-- [Quick Start](/docs/getting-started/quick-start) — build your first knowledge graph in 60 seconds
-- [Storage Adapters](/docs/guides/storage-adapters) — choose the right storage for your deployment

docs/guides/migrating-to-v4.md

@@ -1,491 +0,0 @@
-# Migrating from Brainy v3.x to v4.x
-
-**Brainy v4.0.0** introduces breaking changes to the import API for improved clarity, better defaults, and more powerful features.
-
-This guide will help you migrate your code quickly and painlessly.
-
----
-
-## 🎯 Quick Migration Checklist
-
-If you just want to fix your code fast, here's what to do:
-
-- [ ] Replace `extractRelationships` with `enableRelationshipInference`
-- [ ] Remove `autoDetect` (auto-detection is now always enabled)
-- [ ] Replace `createFileStructure: true` with `vfsPath: '/your/path'`
-- [ ] Remove `excelSheets` (all sheets are now processed automatically)
-- [ ] Remove `pdfExtractTables` (table extraction is now automatic)
-- [ ] Add `enableNeuralExtraction: true` to enable AI entity extraction
-- [ ] Add `preserveSource: true` if you want to keep the original file
-
----
-
-## 📋 Option Name Changes
-
-### Complete Mapping Table
-
-| v3.x Option | v4.x Option | Action Required |
-|-------------|-------------|-----------------|
-| `extractRelationships` | `enableRelationshipInference` | **Rename option** |
-| `autoDetect` | *(removed)* | **Delete option** (always enabled) |
-| `createFileStructure` | `vfsPath` | **Replace** with VFS directory path |
-| `excelSheets` | *(removed)* | **Delete option** (all sheets processed) |
-| `pdfExtractTables` | *(removed)* | **Delete option** (always enabled) |
-| - | `enableNeuralExtraction` | **Add option** (new in v4.x) |
-| - | `enableConceptExtraction` | **Add option** (new in v4.x) |
-| - | `preserveSource` | **Add option** (new in v4.x) |
-
----
-
-## 🔄 Migration Examples
-
-### Example 1: Basic Excel Import
-
-**Before (v3.x):**
-```typescript
-const result = await brain.import('./glossary.xlsx', {
-  extractRelationships: true,
-  createFileStructure: true,
-  groupBy: 'type'
-})
-```
-
-**After (v4.x):**
-```typescript
-const result = await brain.import('./glossary.xlsx', {
-  enableRelationshipInference: true,  // ✅ Renamed
-  vfsPath: '/imports/glossary',       // ✅ Replaced createFileStructure
-  groupBy: 'type'                     // ✅ No change
-})
-```
-
----
-
-### Example 2: Full-Featured Import
-
-**Before (v3.x):**
-```typescript
-const result = await brain.import('./data.xlsx', {
-  extractRelationships: true,
-  autoDetect: true,
-  createFileStructure: true,
-  groupBy: 'type',
-  enableDeduplication: true
-})
-```
-
-**After (v4.x):**
-```typescript
-const result = await brain.import('./data.xlsx', {
-  // AI features
-  enableNeuralExtraction: true,      // ✅ NEW - Extract entity names
-  enableRelationshipInference: true, // ✅ Renamed from extractRelationships
-  enableConceptExtraction: true,     // ✅ NEW - Extract entity types
-
-  // VFS features
-  vfsPath: '/imports/data',          // ✅ Replaced createFileStructure
-  groupBy: 'type',                   // ✅ No change
-  preserveSource: true,              // ✅ NEW - Save original file
-
-  // Performance
-  enableDeduplication: true          // ✅ No change
-})
-```
-
----
-
-### Example 3: Simple Import (Defaults)
-
-**Before (v3.x):**
-```typescript
-const result = await brain.import('./data.csv', {
-  autoDetect: true,
-  extractRelationships: true
-})
-```
-
-**After (v4.x):**
-```typescript
-// Auto-detection is always enabled now
-// Just enable the features you want
-const result = await brain.import('./data.csv', {
-  enableRelationshipInference: true
-})
-
-// Or use all defaults (AI features enabled)
-const result = await brain.import('./data.csv')
-```
-
----
-
-### Example 4: PDF Import
-
-**Before (v3.x):**
-```typescript
-const result = await brain.import('./document.pdf', {
-  pdfExtractTables: true,
-  extractRelationships: true,
-  createFileStructure: true
-})
-```
-
-**After (v4.x):**
-```typescript
-const result = await brain.import('./document.pdf', {
-  // pdfExtractTables removed - always enabled
-  enableRelationshipInference: true,
-  vfsPath: '/imports/documents'
-})
-```
-
----
-
-## 💡 Why These Changes?
-
-### Clearer Option Names
-
-**v3.x naming was ambiguous:**
-- `extractRelationships` → Could mean "create relationships" or "infer relationships"
-- `createFileStructure` → Doesn't explain what structure or where
-
-**v4.x naming is explicit:**
-- `enableRelationshipInference` → Clearly means "use AI to infer semantic relationships"
-- `vfsPath` → Explicitly sets the virtual filesystem directory path
-- `enableNeuralExtraction` → Clearly indicates AI-powered entity extraction
-
-### Separation of Concerns
-
-**v4.x separates import features into clear categories:**
-
-1. **Neural/AI Features:**
-   - `enableNeuralExtraction` - Extract entity names and metadata
-   - `enableRelationshipInference` - Infer semantic relationships
-   - `enableConceptExtraction` - Extract entity types and concepts
-
-2. **VFS Features:**
-   - `vfsPath` - Virtual filesystem directory
-   - `groupBy` - Grouping strategy
-   - `preserveSource` - Keep original file
-
-3. **Performance Features:**
-   - `enableDeduplication` - Merge similar entities
-   - `confidenceThreshold` - AI confidence threshold
-   - `onProgress` - Progress callbacks
-
-### Better Defaults
-
-**v3.x required explicit enabling:**
-```typescript
-// Had to enable everything manually
-await brain.import(file, {
-  autoDetect: true,
-  extractRelationships: true,
-  createFileStructure: true
-})
-```
-
-**v4.x has smart defaults:**
-```typescript
-// Auto-detection and AI features enabled by default
-await brain.import(file)
-
-// Or customize specific features
-await brain.import(file, {
-  vfsPath: '/my/data',
-  confidenceThreshold: 0.8
-})
-```
-
----
-
-## 🆕 New Features in v4.x
-
-### Neural Entity Extraction
-Extract entity names, types, and metadata using AI:
-
-```typescript
-const result = await brain.import('./glossary.xlsx', {
-  enableNeuralExtraction: true,      // Extract entity names from "Term" column
-  enableConceptExtraction: true,     // Detect entity types (Place, Person, etc.)
-  confidenceThreshold: 0.7           // Minimum AI confidence (0-1)
-})
-
-// Result includes rich entity metadata
-result.entities.forEach(entity => {
-  console.log(`${entity.name} (${entity.type})`)
-  console.log(`Confidence: ${entity.confidence}`)
-})
-```
-
-### VFS Integration
-Imported data is organized in a virtual filesystem:
-
-```typescript
-const result = await brain.import('./data.xlsx', {
-  vfsPath: '/projects/myproject/data',
-  groupBy: 'type',        // Group by entity type
-  preserveSource: true    // Save original .xlsx file
-})
-
-// Access via VFS
-const vfs = brain.vfs()
-const files = await vfs.readdir('/projects/myproject/data')
-// ['Places/', 'Characters/', 'Concepts/', '_source.xlsx', '_metadata.json']
-
-// Read entity file
-const content = await vfs.readFile('/projects/myproject/data/Places/Talifar.json')
-```
-
-### Semantic Relationship Inference
-AI infers relationship types from context:
-
-```typescript
-const result = await brain.import('./glossary.xlsx', {
-  enableRelationshipInference: true
-})
-
-// Instead of generic "contains" relationships,
-// you get semantic verbs like:
-// - "capital_of"
-// - "located_in"
-// - "guards"
-// - "part_of"
-// - "related_to"
-
-const relations = await brain.related({ limit: 100 })
-const types = new Set(relations.map(r => r.label))
-console.log(types)
-// Set { 'capital_of', 'guards', 'located_in', 'related_to' }
-```
-
----
-
-## 🔍 What Breaks & How to Fix It
-
-### Error: "Invalid import options: 'extractRelationships'"
-
-**Cause:** Using v3.x option name
-
-**Fix:**
-```typescript
-// Before
-await brain.import(file, { extractRelationships: true })
-
-// After
-await brain.import(file, { enableRelationshipInference: true })
-```
-
----
-
-### Error: "Invalid import options: 'autoDetect'"
-
-**Cause:** Using v3.x option that's been removed
-
-**Fix:**
-```typescript
-// Before
-await brain.import(file, { autoDetect: true })
-
-// After - just remove it (auto-detection always enabled)
-await brain.import(file)
-```
-
----
-
-### Error: "Invalid import options: 'createFileStructure'"
-
-**Cause:** Using v3.x option name
-
-**Fix:**
-```typescript
-// Before
-await brain.import(file, { createFileStructure: true })
-
-// After - specify VFS path explicitly
-await brain.import(file, { vfsPath: '/imports/mydata' })
-```
-
----
-
-### Issue: Import succeeds but entities have generic names like "Entity_144"
-
-**Cause:** Neural extraction is disabled
-
-**Fix:**
-```typescript
-// Ensure AI features are enabled
-await brain.import(file, {
-  enableNeuralExtraction: true,      // ✅ Extract entity names
-  enableRelationshipInference: true, // ✅ Infer relationships
-  enableConceptExtraction: true      // ✅ Extract types
-})
-```
-
----
-
-### Issue: All relationships are type "contains"
-
-**Cause:** Relationship inference is disabled
-
-**Fix:**
-```typescript
-// Enable relationship inference
-await brain.import(file, {
-  enableRelationshipInference: true  // ✅ Use AI to detect semantic relationships
-})
-```
-
----
-
-### Issue: VFS directory doesn't exist in filesystem
-
-**This is NORMAL!** VFS is virtual - it uses Brainy entities, not physical files.
-
-**How to access VFS:**
-```typescript
-// DON'T do this:
-// ls brainy-data/vfs/  ❌ Won't work
-
-// DO this instead:
-const vfs = brain.vfs()
-await vfs.init()
-const files = await vfs.readdir('/imports')  // ✅ Correct
-```
-
----
-
-## 📦 TypeScript Users
-
-### Compile-Time Errors
-
-If you're using TypeScript, you'll get compile-time errors when using deprecated options:
-
-```typescript
-// TypeScript will show error:
-// "Type 'true' is not assignable to type 'never'"
-await brain.import(file, {
-  extractRelationships: true  // ❌ Type error
-})
-
-// Fix: Use correct option name
-await brain.import(file, {
-  enableRelationshipInference: true  // ✅ Type correct
-})
-```
-
-### IDE Autocomplete
-
-Your IDE will show deprecation warnings and suggest the correct option names:
-
-```typescript
-await brain.import(file, {
-  extract...  // IDE suggests: enableNeuralExtraction, enableRelationshipInference
-})
-```
-
----
-
-## 🎓 Best Practices for v4.x
-
-### 1. Enable All AI Features by Default
-
-```typescript
-// Good: Enable all intelligent features
-await brain.import('./data.xlsx', {
-  enableNeuralExtraction: true,
-  enableRelationshipInference: true,
-  enableConceptExtraction: true,
-  vfsPath: '/imports/data'
-})
-```
-
-### 2. Use VFS for Organization
-
-```typescript
-// Good: Organize by project
-await brain.import('./project-A.xlsx', {
-  vfsPath: '/projects/project-a/data'
-})
-
-await brain.import('./project-B.csv', {
-  vfsPath: '/projects/project-b/data'
-})
-```
-
-### 3. Preserve Source Files
-
-```typescript
-// Good: Keep original files for reference
-await brain.import('./important-data.xlsx', {
-  preserveSource: true,  // Saves original .xlsx in VFS
-  vfsPath: '/archives/2025'
-})
-```
-
-### 4. Tune Confidence Threshold
-
-```typescript
-// For high-quality data: Lower threshold
-await brain.import('./curated-glossary.xlsx', {
-  confidenceThreshold: 0.5  // Extract more entities
-})
-
-// For noisy data: Higher threshold
-await brain.import('./scraped-data.csv', {
-  confidenceThreshold: 0.8  // Only high-confidence entities
-})
-```
-
-### 5. Disable Deduplication for Large Imports
-
-```typescript
-// For small imports: Keep deduplication
-await brain.import('./small-data.xlsx', {
-  enableDeduplication: true
-})
-
-// For large imports (>1000 rows): Disable for performance
-await brain.import('./huge-database.csv', {
-  enableDeduplication: false  // Much faster
-})
-```
-
----
-
-## 🚀 Migration Automation (Future)
-
-We're working on an automated migration tool:
-
-```bash
-# Coming soon
-npx @soulcraft/brainy-migrate
-
-# Will scan your code and automatically update:
-# - Option names
-# - TypeScript types
-# - Import patterns
-```
-
----
-
-## 📚 Additional Resources
-
-- **API Documentation:** [https://brainy.dev/docs/api/import](https://brainy.dev/docs/api/import)
-- **Examples:** [examples/import-excel/](../../examples/import-excel/)
-- **Changelog:** [CHANGELOG.md](../../CHANGELOG.md)
-- **Support:** [GitHub Issues](https://github.com/soulcraft/brainy/issues)
-
----
-
-## 💬 Need Help?
-
-If you're stuck migrating:
-
-1. Check the error message - it includes migration hints
-2. Review the examples in this guide
-3. Open an issue on GitHub with your use case
-4. Join our Discord community for real-time help
-
----
-
-**Happy migrating! 🎉**

docs/guides/model-loading.md

@@ -1,238 +1,358 @@
-# Model Loading Guide
+# 🤖 Model Loading Guide
 
-Brainy uses AI embedding models to understand and process your data. With the Candle WASM engine, the model is **embedded at compile time** - no downloads, no configuration, no external dependencies.
+Brainy uses AI embedding models to understand and process your data. This guide explains how model loading works and how to handle different scenarios.
 
-## Zero Configuration (Default)
+## 🚀 Zero Configuration (Default)
 
-**For all developers, no configuration is needed:**
+**For most developers, no configuration is needed:**
 
 ```typescript
 const brain = new Brainy()
-await brain.init() // Model is already embedded - nothing to download!
+await brain.init() // Models load automatically
 ```
 
 **What happens automatically:**
-1. Candle WASM module loads (~90MB, includes model weights)
-2. Model initializes in ~200ms
-3. Ready to use immediately
+1. Checks for local models in `./models/`
+2. Downloads All-MiniLM-L6-v2 if needed (384 dimensions)
+3. Configures optimal settings for your environment
+4. Ready to use immediately
 
-**No downloads. No CDN. No configuration. Just works.**
+## 📦 Model Loading Cascade
 
-## How It Works
-
-The all-MiniLM-L6-v2 model is embedded in the WASM binary using Rust's `include_bytes!` macro:
+Brainy tries multiple sources in this order:
 
 ```
-candle_embeddings_bg.wasm (~90MB)
-├── Candle ML Runtime (~3MB)
-├── Model Weights (safetensors format, ~87MB)
-└── Tokenizer (HuggingFace tokenizers, ~450KB)
+1. LOCAL CACHE (./models/) 
+   ↓ (if not found)
+2. CDN DOWNLOAD (fast mirrors)
+   ↓ (if fails)  
+3. GITHUB RELEASES (github.com/xenova/transformers.js)
+   ↓ (if fails)
+4. HUGGINGFACE HUB (huggingface.co)
+   ↓ (if fails)
+5. FALLBACK STRATEGIES (different model variants)
 ```
 
-This single WASM file contains everything needed for sentence embeddings.
-
-## Environments
-
-### Bun (Recommended)
-
-```bash
-# Bun as a runtime — supported and recommended
-bun add @soulcraft/brainy
-bun run server.ts
-```
-
-Brainy is pure WebAssembly with no native binaries, so the module graph stays
-bundler-friendly. Single-binary `bun build --compile` is **not a supported
-target** at present: Bun 1.3.10 has a `--compile` codegen regression
-(`__promiseAll is not defined`) triggered by top-level `await` in the bundled
-graph. Run Brainy under the Bun runtime (above) instead.
-
-### Node.js
-
-```typescript
-// Standard Node.js
-node dist/server.js
-
-// Runs identically to Bun
-```
+## 🌍 Environment-Specific Behavior
 
 ### Browser
-
 ```typescript
-// Model loads via WASM (single file, no additional assets)
+// Automatically configured for browsers
+const brain = new Brainy() // Works in React, Vue, vanilla JS
+await brain.init() // Downloads models via CDN
+```
+
+### Node.js Development
+```typescript
+// Zero config - downloads to ./models/
 const brain = new Brainy()
-await brain.init()
+await brain.init() // Downloads once, cached forever
+```
+
+### Production Server
+```typescript
+// Preload models during build/deployment
+const brain = new Brainy()
+await brain.init() // Uses cached local models
 ```
 
 ### Docker/Kubernetes
-
 ```dockerfile
-FROM oven/bun:1.1
-WORKDIR /app
-COPY package*.json ./
-RUN bun install
-COPY . .
-EXPOSE 3000
-CMD ["bun", "run", "server.ts"]
-
-# That's it! No model download step needed.
-# Model is embedded in the npm package.
+# Dockerfile - preload models
+RUN npm run download-models
+ENV BRAINY_ALLOW_REMOTE_MODELS=false
 ```
 
-## Model Information
+## 🛠️ Manual Model Management
 
-### all-MiniLM-L6-v2 (Embedded)
-- **Dimensions**: 384 (fixed)
-- **Format**: Safetensors (FP32)
-- **Size**: ~87MB (embedded in WASM)
-- **Total WASM Size**: ~90MB
-- **Language**: English-optimized, works with all languages
-- **Inference**: ~2-10ms per embedding
-- **Initialization**: ~200ms
-
-### Memory Usage
-- **Loaded WASM**: ~90MB
-- **Inference peak**: ~140MB total
-- **Steady state**: ~100MB
-
-## Comparing to Previous Architecture
-
-| Feature | Before (ONNX) | Now (Candle WASM) |
-|---------|--------------|-------------------|
-| Model downloads | Required on first use | None - embedded |
-| External dependencies | onnxruntime-web | None |
-| Model files | model.onnx, tokenizer.json | Embedded in WASM |
-| Offline support | Required setup | Works by default |
-| Bun compile | Broken | Works |
-| Configuration | Environment variables | None needed |
-
-## Troubleshooting
-
-### "Failed to initialize Candle Embedding Engine"
-
-**Cause**: WASM loading issue.
-
-**Solutions**:
+### Pre-download Models
 ```bash
-# Rebuild the WASM
-npm run build:candle
+# Download models during build/deployment
+npm run download-models
 
-# Verify WASM exists
-ls dist/embeddings/wasm/pkg/candle_embeddings_bg.wasm
-# Should be ~90MB
+# Custom location
+BRAINY_MODELS_PATH=./my-models npm run download-models
 ```
 
-### Out of Memory
+### Verify Models
+```bash
+# Check if models exist
+ls ./models/Xenova/all-MiniLM-L6-v2/
 
-**Cause**: Container/environment has less than 256MB RAM.
-
-**Solutions**:
-```dockerfile
-# Increase memory limit (recommended: 512MB+)
-docker run -m 512m my-app
+# Should see:
+# - config.json
+# - tokenizer.json  
+# - onnx/model.onnx
 ```
 
-### Slow Initialization (>500ms)
-
-**Cause**: Cold start, large WASM parsing.
-
-**Solutions**:
-```typescript
-// Initialize once at startup, not per-request
-await brain.init()  // Do this once
-
-// Then reuse for all requests
-app.get('/api', async (req, res) => {
-  const results = await brain.find(req.query)
-  res.json(results)
-})
-```
-
-## Migration from Previous Versions
-
-### From v6.x (ONNX)
-
-No changes needed for most users:
-
-```typescript
-// Same API - just upgrade
-const brain = new Brainy()
-await brain.init()
-```
-
-**What's removed:**
-- `BRAINY_ALLOW_REMOTE_MODELS` - no downloads
-- `BRAINY_MODELS_PATH` - no external model files
-- `npm run download-models` - no longer needed
-
-**What's new:**
-- Faster initialization
-- Bundler-friendly (pure WASM, no native binaries)
-- No network requirements
-
-### From Custom Embedding Functions
-
-If you provided a custom embedding function, it still works:
-
+### Custom Model Path
 ```typescript
 const brain = new Brainy({
-  embeddingFunction: myCustomEmbedder  // Still supported
+  embedding: {
+    cacheDir: './custom-models'
+  }
 })
 ```
 
-## Advanced: Building Custom WASM
-
-For contributors who want to modify the embedding engine:
+## 🔒 Offline & Air-Gapped Environments
 
+### Complete Offline Setup
 ```bash
-# Navigate to Candle WASM source
-cd src/embeddings/candle-wasm
+# 1. Download models on connected machine
+npm run download-models
 
-# Build with wasm-pack
-wasm-pack build --target web --release
+# 2. Copy models to offline machine
+cp -r ./models /path/to/offline/project/
 
-# Copy to pkg folder
-cp pkg/* ../wasm/pkg/
-
-# Build TypeScript
-npm run build
+# 3. Force local-only mode
+export BRAINY_ALLOW_REMOTE_MODELS=false
 ```
 
-## Best Practices
+### Container/Server Deployment
+```dockerfile
+FROM node:18
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci
+
+# Download models during build
+RUN npm run download-models
+
+# Force local-only in production
+ENV BRAINY_ALLOW_REMOTE_MODELS=false
+
+COPY . .
+EXPOSE 3000
+CMD ["npm", "start"]
+```
+
+## ⚙️ Environment Variables
+
+### BRAINY_ALLOW_REMOTE_MODELS
+Controls whether remote model downloads are allowed:
+
+```bash
+# Allow remote downloads (default in most environments)
+export BRAINY_ALLOW_REMOTE_MODELS=true
+
+# Force local-only (recommended for production)
+export BRAINY_ALLOW_REMOTE_MODELS=false
+```
+
+### BRAINY_MODELS_PATH
+Custom model storage location:
+
+```bash
+# Custom model path
+export BRAINY_MODELS_PATH=/opt/brainy/models
+
+# Relative path
+export BRAINY_MODELS_PATH=./my-custom-models
+```
+
+## 🚨 Troubleshooting
+
+### "Failed to load embedding model" Error
+
+**Cause**: Models not found locally and remote download blocked/failed.
+
+**Solutions**:
+```bash
+# Option 1: Allow remote downloads
+export BRAINY_ALLOW_REMOTE_MODELS=true
+
+# Option 2: Download models manually
+npm run download-models
+
+# Option 3: Check internet connectivity
+ping huggingface.co
+
+# Option 4: Use custom model path
+export BRAINY_MODELS_PATH=/path/to/existing/models
+```
+
+### Models Download Very Slowly
+
+**Cause**: Network issues or regional restrictions.
+
+**Solutions**:
+```bash
+# Pre-download during build/CI
+npm run download-models
+
+# Use faster mirrors (automatic in newer versions)
+# No action needed - Brainy tries multiple CDNs
+```
+
+### Container Out of Memory During Model Load
+
+**Cause**: Limited container memory during model initialization.
+
+**Solutions**:
+```dockerfile
+# Increase memory limit
+docker run -m 2g my-app
+
+# Use quantized models (default)
+ENV BRAINY_MODEL_DTYPE=q8
+
+# Pre-load models at build time (recommended)
+RUN npm run download-models
+```
+
+### Permission Denied Creating Model Cache
+
+**Cause**: Write permissions for model cache directory.
+
+**Solutions**:
+```bash
+# Make directory writable
+chmod 755 ./models
+
+# Use custom writable path
+export BRAINY_MODELS_PATH=/tmp/brainy-models
+
+# Or use memory-only storage
+const brain = new Brainy({
+  storage: { forceMemoryStorage: true }
+})
+```
+
+## 🎯 Best Practices
 
 ### Development
 ```typescript
-// Just works - no setup
+// ✅ Zero config - just works
 const brain = new Brainy()
 await brain.init()
 ```
 
 ### Production
-```typescript
-// Initialize once at startup
-const brain = new Brainy()
-await brain.init()
+```dockerfile
+# ✅ Pre-download models
+RUN npm run download-models
 
-// Singleton pattern recommended
-export { brain }
+# ✅ Force local-only
+ENV BRAINY_ALLOW_REMOTE_MODELS=false
+
+# ✅ Verify models exist
+RUN test -f ./models/Xenova/all-MiniLM-L6-v2/onnx/model.onnx
 ```
 
-### Deployment
-```bash
-# Option 1: Bun runtime
-bun run server.ts
+### CI/CD Pipeline
+```yaml
+# .github/workflows/build.yml
+- name: Download AI Models
+  run: npm run download-models
+  
+- name: Verify Models
+  run: |
+    test -f ./models/Xenova/all-MiniLM-L6-v2/onnx/model.onnx
+    echo "✅ Models verified"
 
-# Option 2: Docker
-docker build -t my-app .
-docker run -p 3000:3000 my-app
+- name: Test Offline Mode
+  env:
+    BRAINY_ALLOW_REMOTE_MODELS: false
+  run: npm test
+```
+
+### Lambda/Serverless
+```typescript
+// ✅ Models in deployment package
+const brain = new Brainy({
+  embedding: {
+    localFilesOnly: true, // No downloads in lambda
+    cacheDir: './models'  // Bundled with deployment
+  }
+})
+```
+
+## 📊 Model Information
+
+### All-MiniLM-L6-v2 (Default)
+- **Dimensions**: 384 (fixed)
+- **Size**: ~80MB compressed, ~330MB uncompressed
+- **Language**: English (optimized)
+- **Speed**: Very fast inference
+- **Quality**: High quality for most use cases
+
+### Model Files Structure
+```
+models/
+└── Xenova/
+    └── all-MiniLM-L6-v2/
+        ├── config.json          # Model configuration
+        ├── tokenizer.json       # Text tokenizer
+        ├── tokenizer_config.json
+        └── onnx/
+            ├── model.onnx       # Main model file
+            └── model_quantized.onnx  # Optimized version
+```
+
+## 🔄 Migration from Other Embedding Solutions
+
+### From OpenAI Embeddings
+```typescript
+// Before: OpenAI API calls
+const response = await openai.embeddings.create({
+  model: "text-embedding-ada-002",
+  input: "Your text"
+})
+
+// After: Local Brainy embeddings
+const brain = new Brainy()
+await brain.init() // One-time setup
+const id = await brain.add("Your text", { nounType: 'content' }) // Embedded automatically
+```
+
+### From Sentence Transformers
+```python
+# Before: Python sentence-transformers
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer('all-MiniLM-L6-v2')
+
+# After: JavaScript Brainy (same model!)
+const brain = new Brainy() // Uses same all-MiniLM-L6-v2
+await brain.init()
+```
+
+## 🚀 Advanced Configuration
+
+### Custom Embedding Options
+```typescript
+const brain = new Brainy({
+  embedding: {
+    model: 'Xenova/all-MiniLM-L6-v2', // Default
+    dtype: 'q8',                       // Quantized for speed
+    device: 'cpu',                     // CPU inference
+    localFilesOnly: false,             // Allow downloads
+    verbose: true                      // Debug logging
+  }
+})
+```
+
+### Multiple Model Support (Advanced)
+```typescript
+// Use custom embedding function
+import { createEmbeddingFunction } from 'brainy'
+
+const customEmbedder = createEmbeddingFunction({
+  model: 'Xenova/all-MiniLM-L12-v2', // Larger model
+  dtype: 'fp32' // Higher precision
+})
+
+const brain = new Brainy({
+  embeddingFunction: customEmbedder
+})
 ```
 
 ---
 
-## Additional Resources
+## 📚 Additional Resources
 
-- [Production Service Architecture](../PRODUCTION_SERVICE_ARCHITECTURE.md)
-- [Zero Configuration Guide](../architecture/zero-config.md)
+- [Zero Configuration Guide](./zero-config.md)
+- [Enterprise Deployment](./enterprise-deployment.md)
 - [Troubleshooting Guide](../troubleshooting.md)
+- [API Reference](../api/README.md)
 
-**Need help?** [Open an issue](https://github.com/soulcraftlabs/brainy/issues)
+**Need help?** Check our [troubleshooting guide](../troubleshooting.md) or [open an issue](https://github.com/your-repo/brainy/issues).

docs/guides/natural-language.md

@@ -280,4 +280,5 @@ While powerful, the NLP system has some limitations:
 ## Next Steps
 
 - [Triple Intelligence Architecture](../architecture/triple-intelligence.md)
-- [API Reference](../api/README.md)
+- [API Reference](../api/README.md)
+- [Getting Started Guide](./getting-started.md)

docs/guides/neural-api.md

@@ -0,0 +1,356 @@
+# Neural API Guide
+
+> Semantic intelligence features for clustering, similarity, and analysis
+
+## Overview
+
+The Neural API provides advanced AI-powered features for understanding relationships and patterns in your data. Access it through `brain.neural` after initializing Brainy.
+
+## Quick Start
+
+```javascript
+import { Brainy } from '@soulcraft/brainy'
+
+const brain = new Brainy()
+await brain.init()
+
+// Access Neural API
+const neural = brain.neural
+
+// Find similar items
+const similarity = await neural.similar('text1', 'text2')
+
+// Auto-cluster your data
+const clusters = await neural.clusters()
+```
+
+## Core Features
+
+### 1. Semantic Clustering
+
+Automatically group related items based on their meaning:
+
+```javascript
+// Simple clustering - let Brainy decide
+const clusters = await neural.clusters()
+
+// Each cluster contains:
+// - id: Unique identifier
+// - members: Array of item IDs in this cluster
+// - centroid: The "center" of the cluster
+// - label: Optional descriptive label
+// - confidence: How confident the clustering is
+
+// Example: Organize customer feedback
+const feedback = [
+  await brain.add("The app crashes when I upload photos"),
+  await brain.add("Photo upload feature is broken"),
+  await brain.add("Great customer service!"),
+  await brain.add("Support team was very helpful"),
+  await brain.add("Pricing is too high"),
+  await brain.add("Too expensive for what it offers")
+]
+
+const themes = await neural.clusters()
+// Results in 3 clusters: bugs, support, pricing
+```
+
+#### Advanced Clustering Options
+
+```javascript
+// Control clustering behavior
+const clusters = await neural.clusters({
+  algorithm: 'kmeans',      // Algorithm to use
+  maxClusters: 5,           // Maximum clusters to create
+  threshold: 0.7            // Minimum similarity within clusters
+})
+
+// Cluster specific items only
+const techItems = ['id1', 'id2', 'id3', 'id4']
+const techClusters = await neural.clusters(techItems)
+
+// Find clusters near a specific item
+const relatedClusters = await neural.clusters('central-item-id')
+```
+
+### 2. Similarity Calculation
+
+Compare any two items to see how similar they are:
+
+```javascript
+// Compare by ID
+const score = await neural.similar('item1-id', 'item2-id')
+// Returns 0-1 (0 = completely different, 1 = identical)
+
+// Compare text directly
+const score = await neural.similar(
+  "Machine learning is fascinating",
+  "AI and deep learning are interesting"
+)
+// Returns ~0.75 (pretty similar)
+
+// Compare vectors
+const v1 = await brain.embed("concept 1")
+const v2 = await brain.embed("concept 2")
+const score = await neural.similar(v1, v2)
+
+// Get detailed similarity analysis
+const detailed = await neural.similar('id1', 'id2', { 
+  detailed: true 
+})
+// Returns: {
+//   score: 0.85,
+//   confidence: 0.92,
+//   explanation: "High semantic overlap in technology domain"
+// }
+```
+
+### 3. Finding Neighbors
+
+Discover items similar to a given item:
+
+```javascript
+// Find 5 most similar items
+const neighbors = await neural.neighbors('item-id', 5)
+
+// Each neighbor has:
+// - id: The neighbor's ID
+// - similarity: How similar (0-1)
+// - data: The actual content
+
+// Example: Recommend similar articles
+const articleId = await brain.add("Guide to React Hooks")
+const similar = await neural.neighbors(articleId, 3)
+
+for (const article of similar) {
+  console.log(`${article.similarity * 100}% similar: ${article.data}`)
+}
+```
+
+### 4. Semantic Hierarchy
+
+Build a hierarchy showing relationships between items:
+
+```javascript
+const hierarchy = await neural.hierarchy('item-id')
+
+// Returns structure like:
+// {
+//   self: { id: 'item-id', type: 'article' },
+//   parent: { id: 'parent-id', similarity: 0.8 },
+//   siblings: [
+//     { id: 'sibling1', similarity: 0.75 },
+//     { id: 'sibling2', similarity: 0.72 }
+//   ],
+//   children: [
+//     { id: 'child1', similarity: 0.85 }
+//   ]
+// }
+
+// Use for navigation or breadcrumbs
+const hier = await neural.hierarchy(currentDoc)
+console.log(`You are here: ${hier.self.id}`)
+if (hier.parent) {
+  console.log(`Parent topic: ${hier.parent.id}`)
+}
+```
+
+### 5. Outlier Detection
+
+Find unusual or anomalous items in your data:
+
+```javascript
+// Find items that don't fit patterns
+const outliers = await neural.outliers(0.3)
+// Returns array of IDs that are > 0.3 distance from others
+
+// Example: Detect spam or unusual content
+const messages = [
+  await brain.add("Meeting at 3pm"),
+  await brain.add("Lunch plans for tomorrow"),
+  await brain.add("BUY NOW!!! AMAZING DEALS!!!"),
+  await brain.add("Project deadline next week")
+]
+
+const suspicious = await neural.outliers(0.4)
+// Returns the spam message ID
+```
+
+### 6. Visualization Support
+
+Generate data for visualization libraries:
+
+```javascript
+// Create force-directed graph data
+const vizData = await neural.visualize({
+  maxNodes: 100,        // Limit nodes for performance
+  dimensions: 2,        // 2D or 3D
+  algorithm: 'force'    // Layout algorithm
+})
+
+// Returns:
+// {
+//   nodes: [
+//     { id: 'n1', x: 10, y: 20, cluster: 'c1' },
+//     { id: 'n2', x: 30, y: 40, cluster: 'c1' }
+//   ],
+//   edges: [
+//     { source: 'n1', target: 'n2', weight: 0.8 }
+//   ],
+//   clusters: [
+//     { id: 'c1', color: '#ff6b6b', size: 15 }
+//   ]
+// }
+
+// Use with D3.js, Cytoscape, or other viz libraries
+const data = await neural.visualize({ dimensions: 3 })
+// Now feed to Three.js for 3D visualization
+```
+
+## Practical Examples
+
+### Content Recommendation System
+
+```javascript
+// User reads an article
+const currentArticle = 'article-123'
+
+// Find similar content
+const recommendations = await neural.neighbors(currentArticle, 5)
+
+// Group all content into topics
+const topics = await neural.clusters()
+
+// Find which topic this article belongs to
+const currentTopic = topics.find(t => 
+  t.members.includes(currentArticle)
+)
+
+// Recommend from same topic first, then similar items
+const sameTopicArticles = currentTopic.members
+  .filter(id => id !== currentArticle)
+  .slice(0, 3)
+```
+
+### Customer Feedback Analysis
+
+```javascript
+// Add feedback with metadata
+const feedbackIds = []
+for (const feedback of customerFeedback) {
+  const id = await brain.add(feedback.text, {
+    rating: feedback.rating,
+    date: feedback.date,
+    product: feedback.product
+  })
+  feedbackIds.push(id)
+}
+
+// Cluster to find themes
+const themes = await neural.clusters(feedbackIds)
+
+// Analyze each theme
+for (const theme of themes) {
+  const items = await brain.getNouns(theme.members)
+  
+  const avgRating = items.reduce((sum, item) => 
+    sum + item.metadata.rating, 0) / items.length
+  
+  console.log(`Theme with ${theme.members.length} items`)
+  console.log(`Average rating: ${avgRating}`)
+  
+  // Find representative feedback for this theme
+  const centroidId = theme.members[0] // Closest to center
+  const example = await brain.getNoun(centroidId)
+  console.log(`Example: "${example.data}"`)
+}
+```
+
+### Knowledge Base Organization
+
+```javascript
+// Analyze existing knowledge base
+const allDocs = await brain.getNouns({ type: 'document' })
+
+// Find duplicate or highly similar content
+const duplicates = []
+for (let i = 0; i < allDocs.length; i++) {
+  for (let j = i + 1; j < allDocs.length; j++) {
+    const similarity = await neural.similar(
+      allDocs[i].id, 
+      allDocs[j].id
+    )
+    if (similarity > 0.95) {
+      duplicates.push([allDocs[i].id, allDocs[j].id])
+    }
+  }
+}
+
+// Build topic hierarchy
+const mainTopics = await neural.clusters({
+  maxClusters: 10,
+  algorithm: 'hierarchical'
+})
+
+// For each main topic, find subtopics
+for (const topic of mainTopics) {
+  const subtopics = await neural.clusters(topic.members)
+  console.log(`Topic has ${subtopics.length} subtopics`)
+}
+```
+
+## Performance Tips
+
+1. **Caching**: Neural API automatically caches results. Repeated calls with same parameters are instant.
+
+2. **Batch Operations**: Process multiple items together rather than one at a time.
+
+3. **Sampling**: For large datasets, use sampling:
+   ```javascript
+   const clusters = await neural.clusters({
+     algorithm: 'sample',
+     sampleSize: 1000  // Only analyze 1000 items
+   })
+   ```
+
+4. **Async Processing**: All neural operations are async and non-blocking.
+
+## Error Handling
+
+```javascript
+try {
+  const similarity = await neural.similar('id1', 'id2')
+} catch (error) {
+  // Handle errors
+  if (error.message.includes('not found')) {
+    console.log('One of the items does not exist')
+  }
+}
+
+// Safe clustering with empty data
+const clusters = await neural.clusters([])
+// Returns empty array, doesn't throw
+
+// Non-existent IDs return 0 similarity
+const sim = await neural.similar('fake-id-1', 'fake-id-2')
+// Returns 0
+```
+
+## Advanced Configuration
+
+```javascript
+// Configure neural behavior at initialization
+const brain = new Brainy({
+  neural: {
+    cacheSize: 1000,        // Cache up to 1000 results
+    defaultAlgorithm: 'kmeans',
+    similarityMetric: 'cosine'
+  }
+})
+```
+
+## Next Steps
+
+- Explore [Triple Intelligence](../architecture/triple-intelligence.md) for combined vector + graph + metadata queries
+- Learn about [Augmentations](../augmentations/README.md) to extend Neural API
+- See [API Reference](../api/README.md) for complete method documentation

docs/guides/optimistic-concurrency.md

@@ -1,214 +0,0 @@
----
-title: Optimistic concurrency with _rev
-slug: guides/optimistic-concurrency
-public: true
-category: guides
-template: guide
-order: 8
-description: Use the per-entity `_rev` counter and `update({ ifRev })` to coordinate concurrent writes safely. Covers the lock pattern, idempotent inserts with `ifAbsent`, and recovery on conflict.
-next:
-  - concepts/consistency-model
-  - guides/find-limits
----
-
-# Optimistic concurrency with `_rev`
-
-Brainy 7.31.0 adds a per-entity revision counter so multiple writers can coordinate without a global lock or external coordinator. The pattern is the same one CouchDB, PouchDB, and ETag-based HTTP caches use: read the current revision, do your work, write back with `ifRev: <whatRevYouSaw>`. If the revision moved, your write is rejected and you retry against the latest state.
-
-## What gets added
-
-| Surface | Behavior |
-|---|---|
-| `entity._rev: number` | Returned on every `get()`, `find()`, `search()`. Initialized to `1` on `add()`. Bumped by `1` on every successful `update()`. Pre-7.31.0 entities without `_rev` are surfaced as `1`. |
-| `update({ id, ..., ifRev: number })` | If the persisted `_rev` does not equal `ifRev`, throws `RevisionConflictError`. Omitting `ifRev` keeps the prior (unconditional) update behavior. |
-| `RevisionConflictError` | Carries `{ id, expected, actual }` for principled recovery. |
-| `add({ id, ifAbsent: true })` | By-ID idempotent insert. Returns the existing `id` if one is already present; no throw, no overwrite. |
-| `addMany({ items, ifAbsent: true })` | Applies `ifAbsent` to every item. Per-item `ifAbsent` overrides the batch flag. |
-
-`_rev` is the **per-entity** counter. Its store-wide counterpart is the generation counter behind the [Db API](../concepts/consistency-model.md): `brain.transact(ops, { ifAtGeneration })` is CAS over the whole store, `update({ ifRev })` (and `ifRev` on `transact()` update operations) is CAS over one entity.
-
-## The lock pattern
-
-Every distributed-job scheduler eventually wants this exact loop:
-
-```ts
-import { Brainy, RevisionConflictError } from '@soulcraft/brainy'
-
-const LOCK_ID = '...uuid for this job slot...'
-
-// Bootstrap the lock document once (idempotent).
-await brain.add({
-  id: LOCK_ID,
-  type: NounType.Document,
-  data: { owner: null, expiresAt: 0 },
-  ifAbsent: true
-})
-
-async function tryAcquireLock(workerId: string, ttlMs: number) {
-  const lock = await brain.get(LOCK_ID)
-  if (!lock) throw new Error('lock document missing')
-
-  const state = lock.data as { owner: string | null; expiresAt: number }
-  const now = Date.now()
-
-  // Only take the lock if it's free or expired.
-  if (state.owner && state.expiresAt > now) return false
-
-  try {
-    await brain.update({
-      id: LOCK_ID,
-      data: { owner: workerId, expiresAt: now + ttlMs },
-      ifRev: lock._rev
-    })
-    return true
-  } catch (err) {
-    if (err instanceof RevisionConflictError) {
-      // Another worker grabbed it between our read and write.
-      return false
-    }
-    throw err
-  }
-}
-```
-
-No external lock service, no Redis SETNX, no Cloud Tasks. The CAS check is the lock.
-
-## Read-modify-write with retry
-
-The other common shape is "update a counter / config object" with bounded retries on conflict:
-
-```ts
-async function incrementCounter(id: string, by: number) {
-  for (let attempt = 0; attempt < 5; attempt++) {
-    const entity = await brain.get(id)
-    if (!entity) throw new Error('counter does not exist')
-    const current = (entity.data as { value: number }).value
-    try {
-      await brain.update({
-        id,
-        data: { value: current + by },
-        ifRev: entity._rev
-      })
-      return
-    } catch (err) {
-      if (err instanceof RevisionConflictError) continue   // refetch + retry
-      throw err
-    }
-  }
-  throw new Error('counter update conflict after 5 attempts')
-}
-```
-
-The retry bound matters — without one, two unlucky writers can ping-pong forever.
-
-## Idempotent bootstrap with `ifAbsent`
-
-For singletons (config rows, well-known seed entities, job-state documents) where the natural ID is deterministic:
-
-```ts
-await brain.add({
-  id: 'config:singleton',
-  type: NounType.Document,
-  data: { tenantQuota: 1000 },
-  ifAbsent: true
-})
-```
-
-- First caller writes; gets back `'config:singleton'`.
-- Every subsequent caller short-circuits at the pre-read; gets back the same `'config:singleton'` without touching the existing entity.
-- `_rev` is **not** bumped on the no-op path (no write happened).
-
-`ifAbsent` is only meaningful when you supply an `id`. With no `id`, Brainy generates a fresh UUID that can never collide, so the flag is silently ignored.
-
-`addMany({ items, ifAbsent: true })` applies the flag to every item. Mixing per-item overrides with the batch flag works as you'd expect: per-item `ifAbsent: false` opts an individual row out, per-item `ifAbsent: true` opts it in.
-
-### Why no `addIfMissing({ match, add })` for attribute-based dedup?
-
-You may want "create if no entity with this email exists" (lookup by attribute, not by ID). That's a different operation:
-
-```ts
-// What we DID NOT ship in 7.31.0 — the attribute-based variant.
-await brain.addIfMissing({                       // ← not a real API
-  match: { type: 'Person', where: { email: 'x@y.com' } },
-  add:   { data: '...', metadata: { email: 'x@y.com' } }
-})
-```
-
-It's race-prone as a plain read-then-write: two concurrent imports both see "not found," both insert, you get duplicates. Without a unique-index primitive (which Brainy doesn't have today), close the race with whole-store CAS — read at a pinned generation, then commit only if nothing moved:
-
-```ts
-import { GenerationConflictError } from '@soulcraft/brainy'
-
-async function addIfMissingByEmail(email: string, data: string) {
-  for (let attempt = 0; attempt < 5; attempt++) {
-    const db = brain.now()
-    try {
-      const existing = await db.find({
-        type: NounType.Person,
-        where: { email },
-        limit: 1
-      })
-      if (existing.length > 0) return existing[0].id
-
-      const committed = await brain.transact(
-        [{ op: 'add', type: NounType.Person, subtype: 'customer', data, metadata: { email } }],
-        { ifAtGeneration: db.generation }   // rejects if ANYTHING committed since the read
-      )
-      return committed.receipt!.ids[0]
-    } catch (err) {
-      if (err instanceof GenerationConflictError) continue   // world moved — re-read + retry
-      throw err
-    } finally {
-      await db.release()
-    }
-  }
-  throw new Error('addIfMissingByEmail conflict after 5 attempts')
-}
-```
-
-`ifAtGeneration` is deliberately coarse — *any* committed write invalidates it — so keep the retry bound. When you control the ID, `ifAbsent` stays the cheaper tool.
-
-## How `_rev` relates to generations
-
-Brainy 8.0 has exactly two write-coordination counters, at two granularities:
-
-| Counter | Scope | What it tracks | CAS surface | Conflict error |
-|---|---|---|---|---|
-| **`_rev`** | One entity | Per-entity write count, bumped on every successful update | `update({ ifRev })`, `{ op: 'update', ifRev }` in `transact()` | `RevisionConflictError` |
-| **Generation** | Whole store | One tick per committed `transact()` batch or single-operation write | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` |
-
-They compose: a `transact()` batch can carry per-entity `ifRev` checks *and* a whole-store `ifAtGeneration`; any failed check rejects the entire batch before anything is staged. Generations also power snapshots and time travel (`brain.now()`, `brain.asOf()`, `db.persist()`) — see the [consistency model](../concepts/consistency-model.md) and [Snapshots & Time Travel](./snapshots-and-time-travel.md).
-
-A snapshot or historical view captures each entity *including* its `_rev` at that moment, so reading the past and writing back with `ifRev` against the live state works exactly as you'd hope: the write fails if the entity moved since the state you copied from.
-
-## The transact envelope: batch size, budget, and bulk imports
-
-`transact()` applies its batch atomically under one commit — which means the whole batch
-shares one **apply budget**. Since 8.7.0 the budget scales with the batch:
-`max(30 s, opCount × 2 s)`, or exactly what you pass as `timeoutMs`. A tripped budget rolls
-the entire batch back (nothing partial survives) and throws a retryable
-`TransactionTimeoutError` that names the operation it stopped at, the batch size, and the
-elapsed vs budgeted time — a diagnosis, not just a failure:
-
-```
-Transaction timed out at operation 41/120 ('add') — 246012ms elapsed, budget 240000ms.
-The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch.
-```
-
-Practical envelope guidance for bulk work:
-
-1. **Precompute embeddings outside the commit path.** Embedding inside `transact()` spends
-   the budget on model inference. Use `brain.embedBatch(texts)` and pass each vector via
-   the op's `vector` field — the commit then pays only storage costs, and a retried batch
-   never re-pays inference. (The win is *where* the inference happens, not raw embedding
-   throughput: on the default WASM engine, batch and sequential embedding measure
-   comparably, ~160 ms/text; native embedding providers may batch faster.)
-2. **Chunk very large imports** into batches of a few hundred ops with one `transact()`
-   each. You lose whole-import atomicity but keep per-chunk atomicity, bounded memory, and
-   resumability — pair with `ifAbsent` upserts so a retried chunk is idempotent.
-3. **Slow disks change the math, not the contract.** On network-attached storage a single
-   op can cost ~2 s (canonical write + fsync + index maintenance). The scaled default
-   absorbs that; pass an explicit `timeoutMs` only when you know better than the scale.
-4. **`addMany`/`relateMany` are the convenience tier** — they chunk and batch-embed for
-   you, with per-item error reporting instead of batch atomicity. Choose by what you need:
-   atomic-all-or-nothing → `transact()`; resilient bulk load → `addMany`.

docs/guides/quick-start.md

@@ -1,111 +0,0 @@
----
-title: Quick Start
-slug: getting-started/quick-start
-public: true
-category: getting-started
-template: guide
-order: 2
-description: Build your first knowledge graph in 60 seconds. Add entities, create relationships, and query with Triple Intelligence — vector + graph + metadata in one call.
-next:
-  - concepts/triple-intelligence
-  - api/reference
----
-
-# Quick Start
-
-Get Brainy running in under a minute.
-
-## 1. Install
-
-```bash
-npm install @soulcraft/brainy
-```
-
-## 2. Initialize
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-```
-
-That's it. Brainy auto-configures storage, loads the embedding model, and builds the indexes.
-
-## 3. Add Knowledge
-
-```typescript
-// Text is automatically embedded into 384-dim vectors
-const reactId: string = await brain.add({
-  data: 'React is a JavaScript library for building user interfaces',
-  type: NounType.Concept,
-  subtype: 'library',                                  // Sub-classification within Concept
-  metadata: { category: 'frontend', year: 2013 }
-})
-
-const nextId: string = await brain.add({
-  data: 'Next.js framework for React with server-side rendering',
-  type: NounType.Concept,
-  subtype: 'framework',
-  metadata: { category: 'framework', year: 2016 }
-})
-```
-
-`type` is one of Brainy's 42 stable NounTypes. `subtype` is your free-form sub-classification within that type — flat string, no hierarchy, indexed on the fast path. See **[Subtypes & Facets](./subtypes-and-facets.md)** for the full guide.
-
-## 4. Create Relationships
-
-```typescript
-// Typed graph relationships
-await brain.relate({
-  from: nextId,
-  to: reactId,
-  type: VerbType.DependsOn
-})
-```
-
-## 5. Query with Triple Intelligence
-
-```typescript
-import type { Result } from '@soulcraft/brainy'
-
-// All three search paradigms in one call
-const results: Result[] = await brain.find({
-  query: 'modern frontend frameworks',    // Vector similarity search
-  where: { year: { greaterThan: 2015 } }, // Metadata filtering
-  connected: { to: reactId, depth: 2 }   // Graph traversal
-})
-
-console.log(results[0].data)   // 'Next.js framework for React...'
-console.log(results[0].score)  // 0.94
-```
-
-## What Just Happened
-
-Every entity you `add()` lives in three indexes simultaneously:
-
-| Index | What it stores | Query with |
-|-------|---------------|------------|
-| Vector | 384-dim embedding of `data` | `find({ query: '...' })` |
-| Metadata | All `metadata` fields | `find({ where: { ... } })` |
-| Graph | Typed relationships from `relate()` | `find({ connected: { ... } })` |
-
-`find()` queries all three in parallel and fuses the results.
-
-## Natural Language Queries
-
-Brainy understands 220+ natural language patterns:
-
-```typescript
-// These all work without any configuration
-await brain.find({ query: 'recent documents about machine learning' })
-await brain.find({ query: 'articles created this week' })
-await brain.find({ query: 'people who work at Anthropic' })
-```
-
-## Next Steps
-
-- [Triple Intelligence](/docs/concepts/triple-intelligence) — understand how the query engine works
-- [The Find System](/docs/guides/find-system) — advanced queries, operators, and graph traversal
-- [API Reference](/docs/api/reference) — complete method documentation
-- [Storage Adapters](/docs/guides/storage-adapters) — filesystem, memory

docs/guides/reacting-to-changes.md

@@ -1,117 +0,0 @@
----
-title: Reacting to Changes
-slug: guides/reacting-to-changes
-public: true
-category: guides
-template: guide
-order: 11
-description: Subscribe to every committed mutation with `brain.onChange` — the in-process change feed behind live UIs, cache invalidation, and realtime sync. Covers the event shape, delivery guarantees, and catch-up patterns.
-next:
-  - guides/optimistic-concurrency
-  - guides/snapshots-and-time-travel
----
-
-# Reacting to Changes
-
-`brain.onChange(cb)` is Brainy's in-process change feed: subscribe once and
-receive one event per committed mutation — **every** mutation, regardless of
-how it happened. Direct calls, batch methods, `transact()`, imports, and
-Virtual Filesystem writes all funnel through the same commit point the feed is
-emitted from, so nothing slips past it.
-
-```ts
-const off = brain.onChange((e) => {
-  if (e.kind === 'entity') {
-    console.log(`${e.op} ${e.entity?.type} ${e.id} @ generation ${e.generation}`)
-  }
-})
-
-await brain.add({ data: 'Ada Lovelace', type: 'person' })
-// → "add person 0198... @ generation 42"
-
-off() // unsubscribe when done
-```
-
-## The event
-
-```ts
-interface BrainyChangeEvent {
-  kind: 'entity' | 'relation' | 'store'
-  op: 'add' | 'update' | 'remove' | 'relate' | 'unrelate' | 'updateRelation'
-    | 'clear' | 'restore'
-  id?: string
-  entity?:   { id: string; type: string; subtype?: string;
-               metadata: Record<string, unknown>; service?: string }
-  relation?: { id: string; from: string; to: string; type: string;
-               metadata?: Record<string, unknown> }
-  generation?: number
-  timestamp: number
-}
-```
-
-- **Entity events** (`add` / `update` / `remove`) carry the post-commit indexed
-  view — `type`, `subtype`, and the full custom `metadata`, so you can match
-  your own `where`-style filters against events without a read.
-- **Deletes are fully described.** A `remove` or `unrelate` event carries the
-  record's *last committed state* (sourced from the commit's own history
-  record), not just an id.
-- **Batches emit per item.** `addMany` / `updateMany` / `relateMany` /
-  `removeMany` emit one event per affected record; a `transact()` batch emits
-  one event per item, all sharing the batch's single `generation`.
-- **Cascades are visible.** Removing an entity also emits `unrelate` for each
-  relationship the delete cascaded to.
-- **Store-level events** (`kind: 'store'`) fire for the two wholesale
-  operations — `clear()` and `restore()` — and mean *"everything may have
-  changed; refetch what you care about."*
-
-## Delivery guarantees
-
-- **Post-commit only.** An aborted write — a losing
-  [`ifRev` compare-and-swap](optimistic-concurrency.md), a rejected
-  transaction — never emits. If you received the event, the write is durable.
-- **Commit-ordered.** Events arrive in the order writes committed;
-  `generation` is monotonic.
-- **Asynchronous, never blocking.** Delivery happens in a microtask after the
-  write completes. A slow listener cannot delay a write; a throwing listener
-  is logged and isolated from other listeners.
-- **Zero overhead when unused.** With no subscribers, the write path does no
-  event work at all.
-- **Fire-and-forget.** There is no replay or backpressure. For catch-up after
-  a disconnect, use the `generation` on each event together with
-  [`asOf()` / the transaction log](snapshots-and-time-travel.md): record the
-  last generation you processed, and on reconnect diff from there. For file
-  content specifically, `vfs.readFile(path, { asOf })` and
-  `vfs.history(path)` are the temporal read — see
-  [Snapshots & Time Travel](snapshots-and-time-travel.md).
-
-## Patterns
-
-**Cache invalidation** — drop cached reads for whatever changed:
-
-```ts
-brain.onChange((e) => {
-  if (e.kind === 'store') return cache.clear()
-  if (e.id) cache.delete(e.id)
-})
-```
-
-**Live queries (notify-and-refetch)** — re-run a query when a relevant change
-lands, rather than diffing incrementally:
-
-```ts
-brain.onChange((e) => {
-  if (e.kind === 'entity' && e.entity?.type === 'order') {
-    refreshOpenOrdersView() // debounce as needed
-  }
-})
-```
-
-**Forwarding to other processes** — the feed is in-process by design. To push
-changes to browsers or other services, forward events through your own
-transport (WebSocket, SSE) from the process that owns the brain.
-
-## Lifecycle
-
-`onChange` returns an unsubscribe function — call it when tearing down a
-subscriber (for example, when evicting a pooled instance). `brain.close()`
-drops all listeners; no events are delivered for or after `close()`.