.aiignore

@@ -1,12 +0,0 @@
-# An .aiignore file follows the same syntax as a .gitignore file.
-# .gitignore documentation: https://git-scm.com/docs/gitignore
-
-# you can ignore files
-.DS_Store
-*.log
-*.tmp
-
-# or folders
-dist/
-build/
-out/

.claude/skills/architecture.md

@@ -1,170 +0,0 @@
-# Brainy Architecture Reference
-
-## What Is Brainy
-
-@soulcraft/brainy (v7.17.0) is a Universal Knowledge Protocol -- a Triple Intelligence database combining vector search, graph traversal, and metadata filtering in a single library. Published to npm as a public MIT-licensed package.
-
-## Core Architecture
-
-### Storage Layer (`src/storage/`)
-- **StorageAdapter interface** (`src/coreTypes.ts:576`): The contract ALL storage backends implement. ALWAYS check this interface before adding storage methods.
-- **BaseStorage** (`src/storage/baseStorage.ts`): Base implementation with built-in type-aware partitioning (TypeAwareStorageAdapter was removed -- functionality merged into BaseStorage).
-- **Adapters** (`src/storage/adapters/`):
-  - `fileSystemStorage.ts` -- local filesystem
-  - `memoryStorage.ts` -- in-memory
-  - `baseStorageAdapter.ts` -- shared adapter base (counts, batch ops)
-  - Cloud + OPFS adapters were removed in 8.0 (cloud backup is operator tooling)
-- **Generational MVCC / Db API** (`src/db/`): immutable `Db` values over generation-stamped records
-  - `db.ts` (the `Db` value), `generationStore.ts` (record layer + commit protocol), `types.ts`, `errors.ts`, `whereMatcher.ts`
-  - Design record: `docs/ADR-001-generational-mvcc.md`; replaced the pre-8.0 COW branching + versioning subsystems
-
-### Vector Search (`src/hnsw/`)
-- `hnswIndex.ts` -- HNSW-based approximate nearest neighbor search
-- `typeAwareHNSWIndex.ts` -- type-partitioned vector search
-- NOT in `src/intelligence/` (that directory does not exist)
-
-### Graph Engine (`src/graph/`)
-- `graphAdjacencyIndex.ts` -- adjacency-based graph representation
-- `pathfinding.ts` -- relationship traversal and pathfinding
-- `lsm/` -- LSM tree implementation for graph storage
-
-### Metadata Index (`src/utils/metadataIndex.ts`)
-- O(1) exact match via hash indexes
-- O(log n) range queries via sorted indexes
-- Roaring bitmap set operations for efficient filtering
-- Adaptive chunking strategy (`metadataIndexChunking.ts`)
-- Caching layer (`metadataIndexCache.ts`)
-
-### Triple Intelligence (`src/triple/`)
-- `TripleIntelligenceSystem.ts` -- combines vector + graph + metadata into unified queries
-- Lazy-loaded indexes (loaded on first use, not at startup)
-
-### Neural/AI Components (`src/neural/`)
-- Smart Importers (`src/importers/`): CSV, Excel, PDF, DOCX, YAML, JSON, Markdown, Orchestrator
-- `SmartExtractor.ts` -- entity extraction from unstructured data
-- `SmartRelationshipExtractor.ts` -- relationship detection
-- `NeuralEntityExtractor.ts` -- ML-based entity recognition
-- Natural language processing utilities
-
-### Distributed Systems (`src/distributed/`)
-- Distributed Coordinator for multi-node operation
-- Shard Manager for data partitioning
-- Cache Synchronization across nodes
-- Read/Write separation
-- Network and HTTP transport layers
-- Storage discovery and shard migration
-
-### Transaction Management (`src/transaction/`)
-- TransactionManager for ACID operations
-- Operations: SaveNoun, AddToHNSW, UpdateMetadata, etc.
-- Distributed transaction support
-
-### Integration Hub (`src/integrations/`)
-- Google Sheets integration
-- OData (Open Data Protocol)
-- Server-Sent Events (SSE)
-- Webhooks
-- Event bus system
-
-### Virtual Filesystem (`src/vfs/`)
-- `VirtualFileSystem.ts` -- full VFS implementation (87 KB)
-- `PathResolver.ts`, `FSCompat.ts`, `MimeTypeDetector.ts`, `TreeUtils.ts`
-- Subdirectories: `semantic/` (semantic search), `streams/` (streaming), `importers/`
-
-### MCP Support (`src/mcp/`)
-- BrainyMCPAdapter, MCPAugmentationToolset, BrainyMCPService
-- Model Control Protocol request/response handling
-
-### Aggregation Engine (`src/aggregation/`)
-- **AggregationIndex** (`AggregationIndex.ts`): Write-time incremental aggregation — SUM, COUNT, AVG, MIN, MAX with GROUP BY and time windows
-- **Time Windows** (`timeWindows.ts`): ISO 8601 bucketing — hour, day, week, month, quarter, year, custom intervals
-- **Materializer** (`materializer.ts`): Debounced writes of aggregate results as `NounType.Measurement` entities
-- Integrates into `brain.find({ aggregate })` for unified query API
-- Write hooks in `add()`, `update()`, `delete()` for O(1) incremental updates
-- `'aggregation'` provider key enables native plugin acceleration
-
-### Additional Systems
-- **CLI** (`src/cli/`): Complete command-line tool with interactive mode and catalog system
-- **Migration** (`src/migration/`): MigrationRunner for database schema migrations
-- **Embeddings** (`src/embeddings/`): Embedding manager with Candle-WASM Rust source
-- **Streaming** (`src/streaming/`): Pipeline support with adaptive backpressure
-- **Versioning** (`src/versioning/`): VersioningAPI for data versioning
-- **Plugin System**: Registry-based plugin architecture
-- **Patterns** (`src/patterns/`): 7 pattern library JSON files
-
-## Type System
-- **NounType** (42 types, `src/types/graphTypes.ts:850-893`): Person, Organization, Concept, Collection, Document, Task, Project, etc.
-- **VerbType** (127 types, `src/types/graphTypes.ts:900-1087`): Contains, RelatedTo, PartOf, Creates, DependsOn, MemberOf, etc.
-- All types in `src/types/`
-
-## Module Exports (`src/index.ts`)
-38+ named exports including: Brainy class, configuration types, neural APIs (NeuralImport, NeuralEntityExtractor, SmartExtractor, SmartRelationshipExtractor), distance functions, plugin system, migration system, embedding functions, storage adapters, COW infrastructure, pipeline utilities, graph types, MCP components, integration hub, OData utilities, and more.
-
-## File Structure
-```
-src/
-├── index.ts              # 38+ public exports
-├── brainy.ts             # Main Brainy class (6,500+ lines)
-├── setup.ts              # Initialization polyfills
-├── coreTypes.ts          # StorageAdapter interface + core types
-├── storage/
-│   ├── baseStorage.ts    # Base storage (includes type-aware)
-│   ├── adapters/         # All storage backends + cloud adapters
-│   └── cow/              # Copy-on-Write versioning
-├── hnsw/                 # HNSW vector search
-├── graph/                # Graph engine + pathfinding + LSM
-├── triple/               # Triple Intelligence system
-├── neural/               # Smart extractors + NLP
-├── importers/            # File format importers (8 types)
-├── distributed/          # Distributed database (16 files)
-├── transaction/          # ACID transactions (6 files)
-├── integrations/         # Sheets, OData, SSE, Webhooks
-├── vfs/                  # Virtual filesystem + semantic search
-├── mcp/                  # Model Control Protocol
-├── cli/                  # Command-line interface
-├── migration/            # Schema migrations
-├── embeddings/           # Embedding manager + Candle-WASM
-├── streaming/            # Pipeline + backpressure
-├── versioning/           # Versioning API
-├── types/                # TypeScript type definitions
-├── utils/                # Metadata index, logging, etc.
-├── config/               # Configuration system
-├── patterns/             # Pattern library
-├── api/                  # API layer
-├── interfaces/           # Interface definitions
-├── shared/               # Shared utilities
-├── data/                 # Data utilities
-├── errors/               # Error handling
-├── critical/             # Critical error handling
-├── universal/            # Universal utilities
-├── import/               # Import functionality
-└── scripts/              # Build scripts
-```
-
-## Initialization
-`brainy.ts` `init()` method performs initialization cascade:
-1. Load plugins
-2. Initialize storage
-3. Enable COW (Copy-on-Write)
-4. Set up embeddings
-5. Initialize caches
-6. Set up graph indexes
-7. Initialize VFS
-8. Set up transaction manager
-9. Initialize distributed components (if enabled)
-
-## Testing
-- Framework: Vitest
-- Run: `npm test`
-- Test directories:
-  - `tests/unit/` -- unit tests
-  - `tests/integration/` -- integration tests
-  - `tests/benchmarks/` -- performance benchmarks (NOT tests/performance/)
-  - `tests/comprehensive/` -- comprehensive test suites
-  - `tests/api/` -- API tests
-  - `tests/helpers/` -- test utilities
-
-## Release
-- `npm run release:patch/minor/major` -- fully automated via `scripts/release.sh`
-- `npm run release:dry` -- preview without changes
-- Uses conventional commits for changelog generation

.dockerignore

@@ -1,57 +0,0 @@
-# Git
-.git
-.gitignore
-
-# Development
-.vscode
-.idea
-*.swp
-*.swo
-.DS_Store
-
-# Node
-node_modules
-npm-debug.log*
-yarn-debug.log*
-yarn-error.log*
-
-# Testing
-tests
-*.test.ts
-*.test.js
-coverage
-.nyc_output
-
-# Documentation (keep only essentials)
-docs
-*.md
-!README.md
-!LICENSE
-
-# Build artifacts (will be built in Docker)
-dist
-build
-*.tsbuildinfo
-
-# Environment
-.env
-.env.*
-
-# Strategy and private docs
-.strategy
-CLAUDE.md
-
-# Development files
-docker-compose.yml
-Dockerfile
-.dockerignore
-
-# Data (should be mounted, not baked in)
-data
-*.db
-*.sqlite
-
-# Models (should be downloaded at runtime or mounted)
-models
-*.onnx
-*.bin

.forgejo/workflows/ci.yml

@@ -1,40 +0,0 @@
-name: CI
-
-on:
-  push:
-  pull_request:
-
-jobs:
-  node:
-    name: Node ${{ matrix.node-version }}
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        node-version: ['22', '24']
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-        with:
-          node-version: ${{ matrix.node-version }}
-          cache: npm
-      - run: npm ci
-      - run: npm run test:unit
-
-  bun:
-    name: Bun (latest)
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-        with:
-          node-version: '22'
-          cache: npm
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: latest
-      - run: npm ci
-      # test:bun imports the built dist/, so build first.
-      - run: npm run build
-      # Bun as a runtime is the supported Bun story (`bun add` / `bun run`).
-      - run: npm run test:bun

.github/workflows/ci.yml

@@ -1,40 +0,0 @@
-name: CI
-
-on:
-  push:
-  pull_request:
-
-jobs:
-  node:
-    name: Node ${{ matrix.node-version }}
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        node-version: ['22', '24']
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-        with:
-          node-version: ${{ matrix.node-version }}
-          cache: npm
-      - run: npm ci
-      - run: npm run test:unit
-
-  bun:
-    name: Bun (latest)
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-        with:
-          node-version: '22'
-          cache: npm
-      - uses: oven-sh/setup-bun@v2
-        with:
-          bun-version: latest
-      - run: npm ci
-      # test:bun imports the built dist/, so build first.
-      - run: npm run build
-      # Bun as a runtime is the supported Bun story (`bun add` / `bun run`).
-      - run: npm run test:bun

.gitignore

@@ -1,116 +1,37 @@
-# Dependencies
-node_modules/
+/tmp
+/out-tsc
+/dist
+
+/node_modules
 npm-debug.log*
 yarn-debug.log*
 yarn-error.log*
+/.pnp
+.pnp.js
 
-# Build outputs
-dist/
-build/
-*.tsbuildinfo
+# Coverage directory
+/coverage
 
-# Environment variables
+# Environment files
 .env
 .env.local
 .env.development.local
 .env.test.local
 .env.production.local
 
-# Runtime data
-brainy-data/
-.brainy/
-*.log
-*.pid
-*.seed
-*.pid.lock
-
-# Coverage directory used by tools like istanbul
-coverage/
-*.lcov
-
-# Test results
-tests/results/
-
-# Filesystem test artifacts (created by integration tests)
-test-*/
-
 # IDE files
 .vscode/
 .idea/
-*.swp
-*.swo
-*~
+*.iml
+*.iws
+*.ipr
+*.sublime-workspace
+*.sublime-project
 
 # OS files
 .DS_Store
 Thumbs.db
 
-# Temporary files
-tmp/
-temp/
-*.tmp
-
-# Planning and instruction files
-plan.md
-
-# Package files
-*.tgz
-
-# Private/confidential files
-PLAN.md
-INTERNAL_NOTES.md
-TODO_PRIVATE.md
-*.tar.gz
-
-# Strategy and planning documents (private)
-.strategy/
-# Removed: PRODUCTION_*.md (now these should be public documentation)
-DISTRIBUTED_*.md
-*_ASSESSMENT.md
-*_ANALYSIS.md
-*_TRUTH*.md
-
-# Models (downloaded at runtime)
-models/
-models-cache/
-
-# But include bundled WASM model assets
-!assets/models/
-
-# Development planning files (not for commit)
-PLAN.md
-
-# Backup folders
-backup-*
-backup/
-
-# Internal documentation
-docs/internal/
-
-# Cache files
-*.cache
-
-# Rust/Cargo build artifacts
-src/embeddings/candle-wasm/target/
-src/embeddings/candle-wasm/Cargo.lock
-
-# Ignore the wasm-pack output dir's CONTENTS (note the `/*`, not `/`, so the
-# re-includes below can take effect — git cannot re-include a file whose parent
-# DIRECTORY is excluded). Keep the pre-built WASM committed: it ships in the npm
-# package anyway, it lets consumers + CI build without a Rust/wasm-pack toolchain,
-# and versioning it makes the shipped artifact reproducible (not "whatever the
-# maintainer last built").
-src/embeddings/wasm/pkg/*
-!src/embeddings/wasm/pkg/*.wasm
-!src/embeddings/wasm/pkg/*.js
-!src/embeddings/wasm/pkg/*.d.ts
-
-# Log files (redundant but explicit)
-*.log
-
-# Temporary files (redundant but explicit)
-*.tmp
-/.junie/guidelines.md
-
-# Claude Code harness state
-.claude/scheduled_tasks.lock
+# Data directories created by FileSystemStorage
+/brainy-data
+/custom-data

.npmignore

@@ -1,84 +0,0 @@
-# Source files (not needed in package)
-src/
-tests/
-scripts/
-coverage/
-
-# Model files (downloaded on first use, not bundled)
-models/
-models-cache/
-
-# Development and backup files
-backup-*
-backup-*/
-docs/backup*/
-
-# Documentation (except essentials)
-*.md
-!README.md
-!LICENSE
-!CHANGELOG.md
-!MIGRATION.md
-
-# Configuration files
-.gitignore
-.npmignore
-tsconfig.json
-vitest.config.ts
-vitest.config.mts
-*.config.js
-*.config.ts
-.eslintrc*
-.prettierrc*
-
-# Test files
-test-*.js
-test-*.ts
-*.test.ts
-*.test.js
-*.spec.ts
-*.spec.js
-
-# Temporary and log files
-*.log
-*.tmp
-tmp/
-temp/
-brainy-data/
-
-# Git and CI files
-.git/
-.github/
-.gitlab-ci.yml
-.travis.yml
-
-# IDE files
-.vscode/
-.idea/
-*.swp
-*.swo
-
-# OS files
-.DS_Store
-Thumbs.db
-
-# Private files
-PLAN.md
-CLAUDE.md
-INTERNAL_NOTES.md
-TODO_PRIVATE.md
-*-ANALYSIS.md
-*-PLAN.md
-
-# Build artifacts not needed
-*.tsbuildinfo
-*.map
-
-# Development environment
-.env*
-.nvm*
-.node-version
-
-# Keep dist/ for the compiled code
-# Keep bin/ for the CLI
-# Keep package.json, package-lock.json

.nvmrc

@@ -1 +0,0 @@
-22

CLAUDE.md

@@ -1,208 +0,0 @@
-# Brainy - Claude Code Project Guide
-
-This file provides guidance for Claude Code (and human contributors) when working on the Brainy codebase.
-
-## Cross-Project Coordination
-
-Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
-
-**At session START:** Read the handoff. Find rows where Owner = Brainy. Act on those first.
-
-**At session END:** Mark completed actions ✅, delete rows you finished, delete threads with zero remaining actions. File must not grow. **If you shipped anything consumers need to know about, update `RELEASES.md` before closing.**
-
-**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
-
-**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`)
-
----
-
-## Project Overview
-
-Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraft/brainy` on npm under the MIT license.
-
-## Getting Started
-
-```bash
-npm install           # Install dependencies
-npm run build         # Build the project
-npm test              # Run test suite (Vitest)
-```
-
-## Architecture
-
-Full architecture reference: `.claude/skills/architecture.md`
-
-### Core Systems
-- **Storage** (`src/storage/`): Pluggable storage backends via StorageAdapter interface (`src/coreTypes.ts`)
-- **Vector Search** (`src/hnsw/`): HNSW approximate nearest neighbor search
-- **Graph Engine** (`src/graph/`): Relationship traversal with adjacency index and pathfinding
-- **Metadata Index** (`src/utils/metadataIndex.ts`): O(1) exact match, O(log n) range queries
-- **Triple Intelligence** (`src/triple/`): Unified query combining all three intelligence types
-- **Aggregation Engine** (`src/aggregation/`): Write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows
-- **Virtual Filesystem** (`src/vfs/`): Full VFS with semantic search
-
-### Type System
-- **NounType** (42 types): Entity classification -- Person, Concept, Collection, Document, Task, etc.
-- **VerbType** (127 types): Relationship types -- Contains, RelatedTo, PartOf, Creates, DependsOn, etc.
-- Defined in `src/types/graphTypes.ts`
-
-## Code Standards
-
-### TypeScript
-- Strict mode enabled
-- Target: ES2020, NodeNext module resolution
-- All new code must be TypeScript
-- Follow existing patterns -- read related code before writing
-
-### Quality
-- All code must compile without errors
-- All code must have working tests that exercise real behavior
-- No stub returns (`return {} as any`)
-- No incomplete implementations with TODO comments
-- If something can't be fully implemented, throw an explicit error rather than faking it
-
-### Verification Before Code Changes
-1. Check that interfaces and methods actually exist before using them
-2. Check that type properties are in the type definitions
-3. Run `npm test` -- tests must pass
-4. Run `npm run build` -- build must succeed
-
-### Testing
-- Framework: Vitest
-- Tests in `tests/` (unit, integration, benchmarks, comprehensive)
-- Use in-memory storage for speed where possible
-- Tests must exercise real behavior, not mock it
-- Benchmarks are in `tests/benchmarks/` (not tests/performance/)
-
-## Commit Conventions
-
-Use [Conventional Commits](https://www.conventionalcommits.org/):
-
-```
-feat: add new feature        (minor version bump)
-fix: resolve bug             (patch version bump)
-docs: update documentation   (patch version bump)
-perf: improve performance    (patch version bump)
-refactor: restructure code   (patch version bump)
-test: add/update tests       (patch version bump)
-```
-
-**Important:** Never use `BREAKING CHANGE` in commit messages. Major version bumps are manual decisions only (`npm run release:major`).
-
-## Docs Pipeline — soulcraft.com/docs
-
-Docs in `docs/**/*.md` are published with the npm package (included in `files`) and synced to soulcraft.com/docs on every portal deploy. Frontmatter controls what appears publicly.
-
-### Docs check triggers
-
-Run the docs check whenever the user says ANY of:
-- "commit, publish, release" / "release" / "publish"
-- "update the docs" / "make sure docs are accurate" / "check the docs"
-- "review docs" / "clean up docs"
-
-### Pre-release docs check (MANDATORY before every release)
-
-When the user says "commit, publish, release" or any variation, **before committing**:
-
-1. **Scan all files changed in this session** (and any recently added `docs/*.md` files)
-2. For each changed/new doc, decide: is this useful to external users?
-   - **Yes** → ensure it has complete frontmatter (add or update it)
-   - **No** (internal, migration, dev-only) → ensure it has no frontmatter or `public: false`
-3. For docs that already have frontmatter, verify:
-   - `description` still matches the actual content
-   - `next` links still exist and are still the right follow-up pages
-   - `title` matches the doc's h1
-4. Include frontmatter changes in the commit
-
-### Frontmatter format
-
-```yaml
----
-title: Human-readable title
-slug: category/page-name          # URL: soulcraft.com/docs/category/page-name
-public: true                      # false or absent = not published
-category: getting-started | concepts | guides | api
-template: guide | concept | api   # controls layout on soulcraft.com
-order: 1                          # sidebar position within category (lower = first)
-description: One sentence. What this doc covers and why it matters.
-next:                             # "Next steps" links shown at bottom of page
-  - category/other-slug
----
-```
-
-### Category guide
-
-| category | use for |
-|----------|---------|
-| `getting-started` | installation, quick start, first steps |
-| `concepts` | how the system works, mental models |
-| `guides` | how to do specific things, recipes |
-| `api` | method reference, signatures, parameters |
-
-### What stays internal (no frontmatter / `public: false`)
-
-- Release guides, developer learning paths
-- Migration guides for old versions (v3→v4, v5.11)
-- Architecture analysis docs (clustering algorithms, etc.)
-- Anything in `docs/internal/`
-- Deployment/ops/cost docs (cloud-run, kubernetes, cost-optimization)
-
-## Release Process
-
-Fully automated via `scripts/release.sh`:
-
-```bash
-npm run release:dry          # Preview (no changes)
-npm run release:patch        # Bug fixes
-npm run release:minor        # New features
-npm run release:major        # Breaking changes (rare, manual decision)
-```
-
-The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release.
-
-After a successful release, remind the user:
-> "Published. Deploy portal to pick up the new docs → go to the portal project and deploy."
-
-Do NOT deploy portal from here. Portal is always deployed separately from within the portal project.
-
-## Closed-Source Product Names — HARD RULE
-
-Brainy is the only Soulcraft open-source project. Nothing in this repo — code, JSDoc, tests,
-docs, RELEASES.md, CHANGELOG.md, commit messages — may reference closed-source Soulcraft
-products by name (Workshop, Venue, Memory, Muse, Hall, Forge, Academy, Pulse, Heart,
-Collective, SDK) or by their specific class/method names (`BookingDraftService`,
-`getDemandHeatmap`, `systemKind`, etc.).
-
-When recording a consumer-reported bug, regression scenario, or release note:
-- Refer to "a consumer", "a downstream application", "a production deployment", or "an
-  internal report" — never name the product.
-- All doc examples must use generic domain values (`'employee'`, `'customer'`, `'invoice'`,
-  `'milestone'`, `OrderService`, `/orders/...`), not product-specific schemas.
-- Internal session artifacts (`.strategy/`, `~/.claude/plans/`, handoff files outside the
-  repo) MAY name products — those are not public.
-
-If you catch yourself typing a product name into a tracked file, stop and rephrase.
-
-## Performance Claims
-
-When documenting performance characteristics:
-- **MEASURED**: Cite the test file and line number
-- **PROJECTED**: Clearly label as extrapolated from tested scale
-- Never claim a performance figure without context or evidence
-
-## Debugging
-
-When a bug persists through 2+ fix attempts, switch to systematic debugging:
-1. Add comprehensive logging at every step
-2. Test with production-like data
-3. Trace the complete execution path
-4. Check both library code and consumer code
-5. Verify with actual test execution before declaring fixed
-
-## Key Paths
-
-- Main class: `src/brainy.ts`
-- Public API: `src/index.ts` (38+ exports)
-- Storage interface: `src/coreTypes.ts`
-- Type definitions: `src/types/`
-- Strategy/planning docs: `.strategy/` (gitignored, not public)

CONTRIBUTING.md

@@ -1,298 +1,51 @@
+<div align="center">
+<img src="./brainy.png" alt="Brainy Logo" width="200"/>
+
 # Contributing to Brainy
 
-Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for contributing to the project.
+</div>
 
-## Code of Conduct
+Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for
+contributing to the project.
 
-By participating in this project, you agree to abide by our Code of Conduct:
-- Be respectful and inclusive
-- Welcome newcomers and help them get started
-- Focus on constructive criticism
-- Respect differing viewpoints and experiences
+## Commit Message Guidelines
 
-## How to Contribute
+When contributing to this project, please write clear and descriptive commit messages that explain the purpose of your
+changes. Good commit messages help maintainers understand your contributions and make the review process smoother.
 
-### Reporting Issues
+### Best Practices
 
-Before creating an issue, please check existing issues to avoid duplicates.
-
-When creating an issue, include:
-- Clear, descriptive title
-- Detailed description of the problem
-- Steps to reproduce
-- Expected vs actual behavior
-- System information (OS, Node version, Brainy version)
-- Code examples if applicable
-
-### Suggesting Features
-
-Feature requests are welcome! Please provide:
-- Clear use case
-- Proposed API/interface
-- Examples of how it would work
-- Any potential challenges or considerations
-
-### Pull Requests
-
-#### Before Starting
-
-1. Check existing issues and PRs
-2. Open an issue to discuss significant changes
-3. Fork the repository
-4. Create a feature branch from `main`
-
-#### Development Setup
-
-**Quick Setup (Recommended):**
-```bash
-# Clone your fork
-git clone https://github.com/your-username/brainy.git
-cd brainy
-
-# Run setup script (installs all dependencies including Rust)
-./scripts/setup-dev.sh
-```
-
-**Manual Setup:**
-```bash
-# Clone your fork
-git clone https://github.com/your-username/brainy.git
-cd brainy
-
-# Install system dependencies (Ubuntu/Debian)
-sudo apt-get install -y build-essential pkg-config libssl-dev
-
-# Install Rust (for WASM embedding engine)
-curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
-source ~/.cargo/env
-rustup target add wasm32-unknown-unknown
-cargo install wasm-pack
-
-# Install Node.js dependencies
-npm install
-
-# Build Candle WASM embedding engine
-npm run build:candle
-
-# Build TypeScript
-npm run build
-
-# Run tests
-npm test
-```
-
-#### Making Changes
-
-1. **Follow the code style**
-   - TypeScript for all source code
-   - Clear variable and function names
-   - Comments for complex logic
-   - JSDoc for public APIs
-
-2. **Write tests**
-   - Add tests for new features
-   - Update tests for changes
-   - Ensure all tests pass
-
-3. **Update documentation**
-   - Update README if needed
-   - Add/update API documentation
-   - Include examples
-
-#### Commit Guidelines
-
-Follow conventional commits format:
-
-```
-type(scope): description
-
-[optional body]
-
-[optional footer]
-```
-
-Types:
-- `feat`: New feature
-- `fix`: Bug fix
-- `docs`: Documentation changes
-- `style`: Code style changes
-- `refactor`: Code refactoring
-- `perf`: Performance improvements
-- `test`: Test changes
-- `chore`: Build/tooling changes
-
-Examples:
-```bash
-feat(triple): add graph traversal depth limit
-fix(storage): handle concurrent write conflicts
-docs(api): update search method documentation
-```
-
-#### Submitting PR
-
-1. Push to your fork
-2. Create PR against `main` branch
-3. Fill out PR template
-4. Ensure CI checks pass
-5. Wait for review
-
-### Testing
-
-#### Running Tests
-
-```bash
-# Run all tests
-npm test
-
-# Run specific test file
-npm test tests/core.test.ts
-
-# Run with coverage
-npm run test:coverage
-
-# Watch mode
-npm run test:watch
-```
-
-#### Writing Tests
-
-```typescript
-import { describe, it, expect } from 'vitest'
-import { Brainy } from '../src'
-
-describe('Feature Name', () => {
-  it('should do something specific', async () => {
-    const brain = new Brainy()
-    await brain.init()
-    
-    // Test implementation
-    const result = await brain.search("test")
-    
-    expect(result).toBeDefined()
-    expect(result.length).toBeGreaterThan(0)
-  })
-})
-```
-
-## Architecture Guidelines
-
-### Adding New Features
-
-1. **Check existing functionality**
-   - Review `ARCHITECTURE.md`
-   - Check if similar features exist
-   - Consider if it should be an augmentation
-
-2. **Design considerations**
-   - Maintain backward compatibility
-   - Consider performance impact
-   - Think about all storage adapters
-   - Plan for extensibility
-
-3. **Implementation checklist**
-   - [ ] Core functionality
-   - [ ] Tests (unit and integration)
-   - [ ] Documentation
-   - [ ] TypeScript types
-   - [ ] Examples
-   - [ ] Performance benchmarks (if applicable)
-
-### Creating Augmentations
-
-Augmentations extend Brainy's functionality:
-
-```typescript
-import { BrainyAugmentation } from '../types'
-
-export class MyAugmentation extends BrainyAugmentation {
-  name = 'MyAugmentation'
-  
-  async onInit(brain: Brainy): Promise<void> {
-    // Initialize augmentation
-  }
-  
-  async onAdd(item: any, brain: Brainy): Promise<any> {
-    // Process before adding
-    return item
-  }
-  
-  async onSearch(query: any, results: any[], brain: Brainy): Promise<any[]> {
-    // Process search results
-    return results
-  }
-}
-```
-
-### Performance Considerations
-
-- Use batch operations where possible
-- Implement caching strategically
-- Consider memory usage
-- Profile performance impacts
-- Add benchmarks for critical paths
-
-## Documentation
-
-### API Documentation
-
-Use JSDoc for all public APIs:
-
-```typescript
-/**
- * Searches for similar items using vector similarity
- * @param query - Search query (text or vector)
- * @param options - Search options
- * @returns Array of search results with scores
- * @example
- * ```typescript
- * const results = await brain.search("machine learning", { limit: 10 })
- * ```
- */
-async search(query: string | Vector, options?: SearchOptions): Promise<SearchResult[]> {
-  // Implementation
-}
-```
+- Keep the first line concise (ideally under 50 characters)
+- Use the imperative mood ("Add feature" not "Added feature")
+- Reference issues and pull requests where appropriate
+- When necessary, provide more detailed explanations in the commit body
 
 ### Examples
 
-Add examples for new features:
-
-```typescript
-// examples/feature-name.ts
-import { Brainy } from 'brainy'
-
-async function exampleUsage() {
-  const brain = new Brainy()
-  await brain.init()
-  
-  // Show feature usage
-  // Include comments explaining what's happening
-  // Handle errors appropriately
-}
-
-exampleUsage().catch(console.error)
+```
+Add vector normalization option
+Fix distance calculation in HNSW search
+Update API documentation
+Add support for IndexedDB storage
+Change API parameter order
+Simplify vector comparison logic
+Add tests for metadata filtering
+Update build dependencies
 ```
 
-## Release Process
+## Pull Request Process
 
-1. **Version bump**: Follow semantic versioning
-2. **Update CHANGELOG**: Document all changes
-3. **Run tests**: Ensure all tests pass
-4. **Build**: Generate distribution files
-5. **Tag**: Create git tag for version
-6. **Publish**: Release to npm
+1. Ensure your code follows the project's coding standards
+2. Update the documentation if necessary
+3. Make sure all tests pass
+4. Use conventional commit messages in your PR
+5. Your PR will be reviewed by maintainers and merged if approved
 
-## Getting Help
+## Development Setup
 
-- **Discord**: Join our community
-- **Issues**: Ask questions on GitHub
-- **Discussions**: Share ideas and get feedback
+1. Fork and clone the repository
+2. Install dependencies: `npm install`
+3. Build the project: `npm run build`
+4. Run tests: `npm test`
 
-## Recognition
-
-Contributors will be recognized in:
-- CHANGELOG.md for their contributions
-- README.md contributors section
-- GitHub contributors page
-
-Thank you for contributing to Brainy! 🧠
+Thank you for contributing to Brainy!

Dockerfile

@@ -1,72 +0,0 @@
-# Multi-stage Dockerfile for Brainy
-# Optimized for production deployment with minimal image size
-
-# Stage 1: Build stage
-FROM node:22-alpine AS builder
-
-# Install build dependencies
-RUN apk add --no-cache python3 make g++
-
-# Set working directory
-WORKDIR /app
-
-# Copy package files
-COPY package*.json ./
-
-# Install all dependencies (including dev dependencies for building)
-RUN npm ci
-
-# Copy source code
-COPY . .
-
-# Build the TypeScript code
-RUN npm run build
-
-# Remove dev dependencies and only keep production ones
-RUN npm prune --production
-
-# Stage 2: Production stage
-FROM node:22-alpine
-
-# Install production dependencies only
-RUN apk add --no-cache tini
-
-# Create non-root user for security
-RUN addgroup -g 1001 -S nodejs && \
-    adduser -S nodejs -u 1001
-
-# Set working directory
-WORKDIR /app
-
-# Copy package files
-COPY package*.json ./
-
-# Copy built application from builder stage
-COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules
-COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
-
-# Copy necessary static files
-COPY --chown=nodejs:nodejs README.md LICENSE ./
-
-# Create data directory for file-based storage
-RUN mkdir -p /app/data && chown -R nodejs:nodejs /app/data
-
-# Switch to non-root user
-USER nodejs
-
-# Expose default port (can be overridden)
-EXPOSE 3000
-
-# Set environment variables for production
-ENV NODE_ENV=production
-ENV BRAINY_STORAGE_PATH=/app/data
-
-# Health check endpoint
-HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
-  CMD node -e "require('http').get('http://localhost:3000/health', (r) => r.statusCode === 200 ? process.exit(0) : process.exit(1))"
-
-# Use tini to handle signals properly
-ENTRYPOINT ["/sbin/tini", "--"]
-
-# Default command (can be overridden)
-CMD ["node", "dist/index.js"]

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2024 Brainy Data Contributors
+Copyright (c) 2025 Soulcraft Research
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+SOFTWARE.

README.demo.md

@@ -0,0 +1,56 @@
+# Running the Brainy Demo
+
+The Brainy interactive demo showcases the library's features in a web browser. Follow these steps to run it:
+
+## Prerequisites
+
+- Make sure you have Node.js installed (version 23.11.0 or higher)
+- Ensure the project is built (`npm run build:all`)
+
+## Running the Demo
+
+### Option 1: Using the npm script (recommended)
+
+Run the following command from the project root:
+
+```bash
+npm run demo
+```
+
+This will start an HTTP server and automatically open the demo in your default browser.
+
+### Option 2: Manual setup
+
+1. Start an HTTP server in the project root:
+
+```bash
+npx http-server
+```
+
+2. Open your browser and navigate to:
+   http://localhost:8080/examples/demo.html
+
+## Troubleshooting
+
+If you see the error "Could not load Brainy library. Please ensure the project is built and served over HTTP", check the following:
+
+1. Make sure you've built the project with `npm run build:all`
+2. Ensure you're accessing the demo through HTTP (not by opening the file directly)
+3. Check your browser's console for additional error messages
+
+If issues persist, try clearing your browser cache or using a private/incognito window.
+
+## Build Process
+
+The Brainy library uses a two-step build process:
+
+1. `npm run build` - Compiles TypeScript files to JavaScript (used for Node.js environments)
+2. `npm run build:browser` - Creates a browser-compatible bundle using Rollup
+
+You can run both steps together with:
+
+```bash
+npm run build:all
+```
+
+The browser bundle is created from `examples/browser_compatible_exports.ts`, which exports only browser-compatible parts of the library. This ensures that the demo works correctly in browser environments while the full library still works in Node.js environments.

assets/models/all-MiniLM-L6-v2/config.json

@@ -1,24 +0,0 @@
-{
-  "_name_or_path": "nreimers/MiniLM-L6-H384-uncased",
-  "architectures": [
-    "BertModel"
-  ],
-  "attention_probs_dropout_prob": 0.1,
-  "gradient_checkpointing": false,
-  "hidden_act": "gelu",
-  "hidden_dropout_prob": 0.1,
-  "hidden_size": 384,
-  "initializer_range": 0.02,
-  "intermediate_size": 1536,
-  "layer_norm_eps": 1e-12,
-  "max_position_embeddings": 512,
-  "model_type": "bert",
-  "num_attention_heads": 12,
-  "num_hidden_layers": 6,
-  "pad_token_id": 0,
-  "position_embedding_type": "absolute",
-  "transformers_version": "4.8.2",
-  "type_vocab_size": 2,
-  "use_cache": true,
-  "vocab_size": 30522
-}

bin/brainy-interactive.js

@@ -1,564 +0,0 @@
-#!/usr/bin/env node
-
-/**
- * Brainy Interactive Mode
- * 
- * Professional, guided CLI experience for beginners
- */
-
-import { program } from 'commander'
-import { Brainy } from '../dist/index.js'
-import chalk from 'chalk'
-import inquirer from 'inquirer'
-import ora from 'ora'
-import Table from 'cli-table3'
-import boxen from 'boxen'
-
-// Professional color scheme
-const colors = {
-  primary: chalk.hex('#3A5F4A'),    // Teal (from logo)
-  success: chalk.hex('#2D4A3A'),    // Deep teal
-  info: chalk.hex('#4A6B5A'),       // Medium teal  
-  warning: chalk.hex('#D67441'),    // Orange (from logo)
-  error: chalk.hex('#B85C35'),      // Deep orange
-  brain: chalk.hex('#D67441'),      // Brain orange
-  cream: chalk.hex('#F5E6A3'),      // Cream background
-  dim: chalk.dim,
-  bold: chalk.bold,
-  cyan: chalk.cyan,
-  green: chalk.green,
-  yellow: chalk.yellow,
-  red: chalk.red
-}
-
-// Icons for consistent visual language
-const icons = {
-  brain: '🧠',
-  search: '🔍',
-  add: '➕',
-  delete: '🗑️',
-  update: '🔄',
-  import: '📥',
-  export: '📤',
-  connect: '🔗',
-  question: '❓',
-  success: '✅',
-  error: '❌',
-  warning: '⚠️',
-  info: 'ℹ️',
-  sparkle: '✨',
-  rocket: '🚀',
-  thinking: '🤔',
-  chat: '💬',
-  stats: '📊',
-  config: '⚙️',
-  cloud: '☁️'
-}
-
-let brainyInstance = null
-
-async function getBrainy() {
-  if (!brainyInstance) {
-    const spinner = ora('Initializing Brainy...').start()
-    try {
-      brainyInstance = new Brainy()
-      await brainyInstance.init()
-      spinner.succeed('Brainy initialized')
-    } catch (error) {
-      spinner.fail('Failed to initialize Brainy')
-      console.error(colors.error(error.message))
-      process.exit(1)
-    }
-  }
-  return brainyInstance
-}
-
-/**
- * Professional welcome screen
- */
-function showWelcome() {
-  console.clear()
-  
-  const welcomeBox = boxen(
-    colors.primary(`${icons.brain}  BRAINY - Neural Intelligence System\n`) +
-    colors.dim('\nYour AI-Powered Second Brain\n') +
-    colors.info('Version 1.6.0'),
-    {
-      padding: 1,
-      margin: 1,
-      borderStyle: 'round',
-      borderColor: 'cyan',
-      textAlignment: 'center'
-    }
-  )
-  
-  console.log(welcomeBox)
-  console.log()
-}
-
-/**
- * Main interactive menu
- */
-async function mainMenu() {
-  const { action } = await inquirer.prompt([{
-    type: 'list',
-    name: 'action',
-    message: colors.cyan('What would you like to do?'),
-    choices: [
-      new inquirer.Separator(colors.dim('── Core Operations ──')),
-      { name: `${icons.add}  Add data to your brain`, value: 'add' },
-      { name: `${icons.search}  Search your knowledge`, value: 'search' },
-      { name: `${icons.chat}  Chat with your data`, value: 'chat' },
-      { name: `${icons.update}  Update existing data`, value: 'update' },
-      { name: `${icons.delete}  Delete data`, value: 'delete' },
-      
-      new inquirer.Separator(colors.dim('── Advanced Features ──')),
-      { name: `${icons.connect}  Create relationships`, value: 'relate' },
-      { name: `${icons.import}  Import from file/URL`, value: 'import' },
-      { name: `${icons.export}  Export your brain`, value: 'export' },
-      { name: `${icons.brain}  Neural operations`, value: 'neural' },
-      
-      new inquirer.Separator(colors.dim('── System ──')),
-      { name: `${icons.stats}  View statistics`, value: 'stats' },
-      { name: `${icons.config}  Configuration`, value: 'config' },
-      { name: `${icons.cloud}  Brain Cloud`, value: 'cloud' },
-      { name: `${icons.info}  Help & Documentation`, value: 'help' },
-      
-      new inquirer.Separator(),
-      { name: 'Exit', value: 'exit' }
-    ],
-    pageSize: 20
-  }])
-  
-  return action
-}
-
-/**
- * Neural operations submenu
- */
-async function neuralMenu() {
-  const { operation } = await inquirer.prompt([{
-    type: 'list',
-    name: 'operation',
-    message: colors.cyan('Select neural operation:'),
-    choices: [
-      { name: `${icons.brain}  Calculate similarity`, value: 'similar' },
-      { name: `${icons.search}  Find clusters`, value: 'cluster' },
-      { name: `${icons.connect}  Find related items`, value: 'related' },
-      { name: `${icons.thinking}  Build hierarchy`, value: 'hierarchy' },
-      { name: `${icons.rocket}  Find semantic path`, value: 'path' },
-      { name: `${icons.warning}  Detect outliers`, value: 'outliers' },
-      { name: `${icons.sparkle}  Generate visualization`, value: 'visualize' },
-      new inquirer.Separator(),
-      { name: '← Back to main menu', value: 'back' }
-    ]
-  }])
-  
-  return operation
-}
-
-/**
- * Execute commands with beautiful feedback
- */
-async function executeCommand(command) {
-  const brain = await getBrainy()
-  
-  switch (command) {
-    case 'add':
-      await interactiveAdd(brain)
-      break
-      
-    case 'search':
-      await interactiveSearch(brain)
-      break
-      
-    case 'chat':
-      await interactiveChat(brain)
-      break
-      
-    case 'update':
-      await interactiveUpdate(brain)
-      break
-      
-    case 'delete':
-      await interactiveDelete(brain)
-      break
-      
-    case 'relate':
-      await interactiveRelate(brain)
-      break
-      
-    case 'import':
-      await interactiveImport(brain)
-      break
-      
-    case 'export':
-      await interactiveExport(brain)
-      break
-      
-    case 'neural':
-      const neuralOp = await neuralMenu()
-      if (neuralOp !== 'back') {
-        await executeNeuralOperation(neuralOp, brain)
-      }
-      break
-      
-    case 'stats':
-      await showStatistics(brain)
-      break
-      
-    case 'config':
-      await interactiveConfig(brain)
-      break
-      
-    case 'cloud':
-      await showCloudInfo()
-      break
-      
-    case 'help':
-      await showHelp()
-      break
-  }
-}
-
-/**
- * Interactive add with rich prompts
- */
-async function interactiveAdd(brain) {
-  console.log(colors.primary(`\n${icons.add} Add Data\n`))
-  
-  const { inputType } = await inquirer.prompt([{
-    type: 'list',
-    name: 'inputType',
-    message: 'How would you like to add data?',
-    choices: [
-      { name: 'Type or paste text', value: 'text' },
-      { name: 'Multi-line editor', value: 'editor' },
-      { name: 'JSON object', value: 'json' },
-      { name: 'Import from clipboard', value: 'clipboard' }
-    ]
-  }])
-  
-  let data = ''
-  
-  switch (inputType) {
-    case 'text':
-      const { text } = await inquirer.prompt([{
-        type: 'input',
-        name: 'text',
-        message: 'Enter your data:',
-        validate: input => input.trim() ? true : 'Please enter some data'
-      }])
-      data = text
-      break
-      
-    case 'editor':
-      const { editorText } = await inquirer.prompt([{
-        type: 'editor',
-        name: 'editorText',
-        message: 'Enter your data (opens editor):',
-        postfix: '.md'
-      }])
-      data = editorText
-      break
-      
-    case 'json':
-      const { jsonText } = await inquirer.prompt([{
-        type: 'editor',
-        name: 'jsonText',
-        message: 'Enter JSON data:',
-        postfix: '.json',
-        default: '{\n  \n}',
-        validate: input => {
-          try {
-            JSON.parse(input)
-            return true
-          } catch (e) {
-            return `Invalid JSON: ${e.message}`
-          }
-        }
-      }])
-      data = jsonText
-      break
-  }
-  
-  // Optional metadata
-  const { addMetadata } = await inquirer.prompt([{
-    type: 'confirm',
-    name: 'addMetadata',
-    message: 'Would you like to add metadata?',
-    default: false
-  }])
-  
-  let metadata = {}
-  if (addMetadata) {
-    const { metadataJson } = await inquirer.prompt([{
-      type: 'editor',
-      name: 'metadataJson',
-      message: 'Enter metadata (JSON):',
-      postfix: '.json',
-      default: '{\n  "type": "",\n  "tags": [],\n  "category": ""\n}',
-      validate: input => {
-        try {
-          JSON.parse(input)
-          return true
-        } catch (e) {
-          return `Invalid JSON: ${e.message}`
-        }
-      }
-    }])
-    metadata = JSON.parse(metadataJson)
-  }
-  
-  const spinner = ora('Adding data...').start()
-  try {
-    const id = await brain.add(data, metadata)
-    spinner.succeed(`Added successfully with ID: ${id}`)
-    
-    // Show summary
-    console.log(boxen(
-      colors.success(`${icons.success} Data added successfully!\n\n`) +
-      colors.info(`ID: ${id}\n`) +
-      colors.dim(`Size: ${data.length} characters\n`) +
-      (Object.keys(metadata).length > 0 ? colors.dim(`Metadata: ${Object.keys(metadata).join(', ')}`) : ''),
-      { padding: 1, borderColor: 'green', borderStyle: 'round' }
-    ))
-  } catch (error) {
-    spinner.fail('Failed to add data')
-    console.error(colors.error(error.message))
-  }
-}
-
-/**
- * Interactive search with filters
- */
-async function interactiveSearch(brain) {
-  console.log(colors.primary(`\n${icons.search} Search\n`))
-  
-  const { query } = await inquirer.prompt([{
-    type: 'input',
-    name: 'query',
-    message: 'Enter search query:',
-    validate: input => input.trim() ? true : 'Please enter a search query'
-  }])
-  
-  // Advanced options
-  const { useFilters } = await inquirer.prompt([{
-    type: 'confirm',
-    name: 'useFilters',
-    message: 'Apply filters?',
-    default: false
-  }])
-  
-  let searchOptions = { limit: 10 }
-  
-  if (useFilters) {
-    const { limit, threshold } = await inquirer.prompt([
-      {
-        type: 'number',
-        name: 'limit',
-        message: 'Maximum results:',
-        default: 10
-      },
-      {
-        type: 'number',
-        name: 'threshold',
-        message: 'Similarity threshold (0-1):',
-        default: 0.5,
-        validate: input => input >= 0 && input <= 1 ? true : 'Must be between 0 and 1'
-      }
-    ])
-    
-    searchOptions.limit = limit
-    searchOptions.threshold = threshold
-  }
-  
-  const spinner = ora('Searching...').start()
-  try {
-    const results = await brain.search(query, searchOptions.limit, searchOptions)
-    spinner.succeed(`Found ${results.length} results`)
-    
-    if (results.length === 0) {
-      console.log(colors.warning('No results found'))
-    } else {
-      // Display results in a table
-      const table = new Table({
-        head: [colors.cyan('ID'), colors.cyan('Content'), colors.cyan('Score')],
-        style: { head: [], border: [] },
-        colWidths: [20, 50, 10]
-      })
-      
-      results.forEach(result => {
-        const content = result.content || result.id
-        const truncated = content.length > 47 ? content.substring(0, 47) + '...' : content
-        const score = result.score ? `${(result.score * 100).toFixed(1)}%` : 'N/A'
-        
-        table.push([
-          result.id.substring(0, 18),
-          truncated,
-          colors.green(score)
-        ])
-      })
-      
-      console.log(table.toString())
-      
-      // Ask if user wants to see full details
-      const { viewDetails } = await inquirer.prompt([{
-        type: 'confirm',
-        name: 'viewDetails',
-        message: 'View full details of a result?',
-        default: false
-      }])
-      
-      if (viewDetails) {
-        const { selectedId } = await inquirer.prompt([{
-          type: 'list',
-          name: 'selectedId',
-          message: 'Select result:',
-          choices: results.map(r => ({
-            name: `${r.id} - ${r.content?.substring(0, 50)}...`,
-            value: r.id
-          }))
-        }])
-        
-        const selected = results.find(r => r.id === selectedId)
-        console.log(boxen(
-          colors.cyan('Full Details\n\n') +
-          colors.info(`ID: ${selected.id}\n\n`) +
-          `Content:\n${selected.content}\n\n` +
-          (selected.metadata ? `Metadata:\n${JSON.stringify(selected.metadata, null, 2)}` : ''),
-          { padding: 1, borderColor: 'cyan', borderStyle: 'round' }
-        ))
-      }
-    }
-  } catch (error) {
-    spinner.fail('Search failed')
-    console.error(colors.error(error.message))
-  }
-}
-
-/**
- * Show statistics with beautiful formatting
- */
-async function showStatistics(brain) {
-  const spinner = ora('Gathering statistics...').start()
-  
-  try {
-    const stats = brain.getStats()
-    spinner.succeed('Statistics loaded')
-    
-    console.log(boxen(
-      colors.primary(`${icons.stats} Database Statistics\n\n`) +
-      colors.info(`Total Items: ${colors.bold(stats.total || 0)}\n`) +
-      colors.info(`Nouns: ${stats.nounCount || 0}\n`) +
-      colors.info(`Relationships: ${stats.verbCount || 0}\n`) +
-      colors.info(`Metadata Records: ${stats.metadataCount || 0}\n\n`) +
-      colors.dim(`Memory Usage: ${(process.memoryUsage().heapUsed / 1024 / 1024).toFixed(1)} MB`),
-      { 
-        padding: 1, 
-        borderColor: 'blue', 
-        borderStyle: 'round',
-        textAlignment: 'left'
-      }
-    ))
-  } catch (error) {
-    spinner.fail('Failed to get statistics')
-    console.error(colors.error(error.message))
-  }
-}
-
-/**
- * Show help with examples
- */
-async function showHelp() {
-  console.log(boxen(
-    colors.primary(`${icons.info} Brainy Help\n\n`) +
-    colors.cyan('Common Commands:\n') +
-    colors.dim(`
-  brainy add "text"        Add data
-  brainy search "query"    Search your brain
-  brainy chat             Interactive AI chat
-  brainy status           View statistics
-  brainy help             This help menu
-  
-`) +
-    colors.cyan('Interactive Mode:\n') +
-    colors.dim(`
-  brainy                  Start interactive mode
-  brainy -i               Alternative interactive mode
-  
-`) +
-    colors.cyan('Advanced Features:\n') +
-    colors.dim(`
-  brainy similar a b      Calculate similarity
-  brainy cluster          Find semantic clusters
-  brainy export           Export your data
-  brainy cloud            Brain Cloud features
-`),
-    { padding: 1, borderColor: 'yellow', borderStyle: 'round' }
-  ))
-  
-  const { learnMore } = await inquirer.prompt([{
-    type: 'confirm',
-    name: 'learnMore',
-    message: 'View detailed documentation?',
-    default: false
-  }])
-  
-  if (learnMore) {
-    console.log(colors.info('\nDocumentation: https://github.com/TimeSoul/brainy'))
-    console.log(colors.info('Enterprise features: Coming in future releases'))
-  }
-}
-
-/**
- * Main interactive loop
- */
-async function main() {
-  showWelcome()
-  
-  let running = true
-  while (running) {
-    const action = await mainMenu()
-    
-    if (action === 'exit') {
-      console.log(colors.success(`\n${icons.success} Thank you for using Brainy!\n`))
-      running = false
-    } else {
-      await executeCommand(action)
-      
-      // Pause before returning to menu
-      await inquirer.prompt([{
-        type: 'input',
-        name: 'continue',
-        message: colors.dim('\nPress Enter to continue...'),
-        prefix: ''
-      }])
-    }
-  }
-  
-  process.exit(0)
-}
-
-// Handle errors gracefully
-process.on('unhandledRejection', (error) => {
-  console.error(colors.error(`\n${icons.error} Unexpected error:`))
-  console.error(colors.red(error.message))
-  process.exit(1)
-})
-
-// Handle Ctrl+C gracefully
-process.on('SIGINT', () => {
-  console.log(colors.info(`\n\n${icons.info} Exiting Brainy...`))
-  process.exit(0)
-})
-
-// Run if called directly
-if (import.meta.url === `file://${process.argv[1]}`) {
-  main().catch(error => {
-    console.error(colors.error('Fatal error:'), error)
-    process.exit(1)
-  })
-}
-
-export { main as startInteractiveMode }

bin/brainy-minimal.js

@@ -1,82 +0,0 @@
-#!/usr/bin/env node
-
-/**
- * Brainy CLI - Minimal Version (Conversation Commands Only)
- *
- * This is a temporary minimal CLI that only includes working conversation commands
- * Full CLI will be restored in version 3.20.0
- */
-
-import { Command } from 'commander'
-import { readFileSync } from 'fs'
-import { dirname, join } from 'path'
-import { fileURLToPath } from 'url'
-
-const __dirname = dirname(fileURLToPath(import.meta.url))
-const packageJson = JSON.parse(readFileSync(join(__dirname, '..', 'package.json'), 'utf8'))
-
-const program = new Command()
-
-program
-  .name('brainy')
-  .description('🧠 Brainy - Infinite Agent Memory')
-  .version(packageJson.version)
-
-// Dynamically load conversation command
-const conversationCommand = await import('../dist/cli/commands/conversation.js').then(m => m.default)
-
-program
-  .command('conversation')
-  .alias('conv')
-  .description('💬 Infinite agent memory and context management')
-  .addCommand(
-    new Command('setup')
-      .description('Set up MCP server for Claude Code integration')
-      .action(async () => {
-        await conversationCommand.handler({ action: 'setup', _: [] })
-      })
-  )
-  .addCommand(
-    new Command('remove')
-      .description('Remove MCP server and clean up')
-      .action(async () => {
-        await conversationCommand.handler({ action: 'remove', _: [] })
-      })
-  )
-  .addCommand(
-    new Command('search')
-      .description('Search messages across conversations')
-      .requiredOption('-q, --query <query>', 'Search query')
-      .option('-c, --conversation-id <id>', 'Filter by conversation')
-      .option('-r, --role <role>', 'Filter by role')
-      .option('-l, --limit <number>', 'Maximum results', '10')
-      .action(async (options) => {
-        await conversationCommand.handler({ action: 'search', ...options, _: [] })
-      })
-  )
-  .addCommand(
-    new Command('context')
-      .description('Get relevant context for a query')
-      .requiredOption('-q, --query <query>', 'Context query')
-      .option('-l, --limit <number>', 'Maximum messages', '10')
-      .action(async (options) => {
-        await conversationCommand.handler({ action: 'context', ...options, _: [] })
-      })
-  )
-  .addCommand(
-    new Command('thread')
-      .description('Get full conversation thread')
-      .requiredOption('-c, --conversation-id <id>', 'Conversation ID')
-      .action(async (options) => {
-        await conversationCommand.handler({ action: 'thread', ...options, _: [] })
-      })
-  )
-  .addCommand(
-    new Command('stats')
-      .description('Show conversation statistics')
-      .action(async () => {
-        await conversationCommand.handler({ action: 'stats', _: [] })
-      })
-  )
-
-program.parse(process.argv)

bin/brainy-ts.js

@@ -1,18 +0,0 @@
-#!/usr/bin/env node
-
-/**
- * Modern TypeScript CLI Runner
- * 
- * This is the entry point after npm install @soulcraft/brainy
- * It runs the compiled TypeScript CLI code
- */
-
-// Use the compiled TypeScript CLI
-import('../dist/cli/index.js').catch(err => {
-  // Fallback to legacy CLI if new one isn't built yet
-  import('./brainy.js').catch(() => {
-    console.error('Error: CLI not properly built. Please reinstall the package.')
-    console.error(err)
-    process.exit(1)
-  })
-})

bin/brainy.js

@@ -1,14 +0,0 @@
-#!/usr/bin/env node
-
-/**
- * Brainy CLI Wrapper
- *
- * Imports the compiled TypeScript CLI from dist/cli/index.js
- * This ensures TypeScript features work correctly
- */
-
-import('../dist/cli/index.js').catch((error) => {
-  console.error('Failed to load Brainy CLI:', error.message)
-  console.error('Make sure you have built the project: npm run build')
-  process.exit(1)
-})

cli-wrapper.js

@@ -0,0 +1,56 @@
+#!/usr/bin/env node
+
+/**
+ * CLI Wrapper Script
+ *
+ * This script serves as a wrapper for the Brainy CLI, ensuring that command-line arguments
+ * are properly passed to the CLI when invoked through npm scripts.
+ */
+
+import { spawn } from 'child_process'
+import { fileURLToPath } from 'url'
+import { dirname, join } from 'path'
+import fs from 'fs'
+
+// Get the directory of the current module
+const __filename = fileURLToPath(import.meta.url)
+const __dirname = dirname(__filename)
+
+// Path to the actual CLI script
+const cliPath = join(__dirname, 'dist', 'cli.js')
+
+// Check if the CLI script exists
+if (!fs.existsSync(cliPath)) {
+  console.error(`Error: CLI script not found at ${cliPath}`)
+  console.error('Make sure you have built the project with "npm run build"')
+  process.exit(1)
+}
+
+// Special handling for version flags
+if (process.argv.includes('--version') || process.argv.includes('-V')) {
+  // Read version directly from package.json to ensure it's always correct
+  try {
+    const packageJsonPath = join(__dirname, 'package.json')
+    const packageJson = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8'))
+    console.log(packageJson.version)
+    process.exit(0)
+  } catch (error) {
+    console.error('Error loading version information:', error.message)
+    process.exit(1)
+  }
+}
+
+// Forward all arguments to the CLI script
+const args = process.argv.slice(2)
+
+// Check if npm is passing --force flag
+// When npm runs with --force, it sets the npm_config_force environment variable
+if (process.env.npm_config_force === 'true' && args.includes('clear') && !args.includes('--force') && !args.includes('-f')) {
+  args.push('--force')
+}
+
+const cli = spawn('node', [cliPath, ...args], { stdio: 'inherit' })
+
+cli.on('close', (code) => {
+  process.exit(code)
+})

cloud-wrapper/.env.example

@@ -0,0 +1,24 @@
+# Server configuration
+PORT=3000
+
+# Storage configuration
+# Options: 'filesystem', 'memory', 's3'
+STORAGE_TYPE=filesystem
+STORAGE_ROOT_DIR=./data
+
+# S3 Storage configuration (only used if STORAGE_TYPE=s3)
+S3_BUCKET_NAME=your-bucket-name
+S3_ACCESS_KEY_ID=your-access-key
+S3_SECRET_ACCESS_KEY=your-secret-key
+S3_REGION=us-east-1
+# Optional: For custom S3-compatible services like MinIO, DigitalOcean Spaces, etc.
+# S3_ENDPOINT=https://your-custom-endpoint
+
+# Embedding configuration
+# Set to 'true' to use simple embedding (faster but less accurate)
+USE_SIMPLE_EMBEDDING=false
+
+# HNSW index configuration (optional)
+# HNSW_M=16                # Max connections per node
+# HNSW_EF_CONSTRUCTION=200 # Construction candidate list size
+# HNSW_EF_SEARCH=50        # Search candidate list size

cloud-wrapper/README.md

@@ -0,0 +1,296 @@
+<div align="center">
+<img src="../brainy.png" alt="Brainy Logo" width="200"/>
+
+# Brainy Cloud Wrapper
+</div>
+
+A standalone web service wrapper for the [Brainy](https://github.com/soulcraft/brainy) vector graph database. This wrapper allows you to deploy Brainy as a RESTful API service on various cloud platforms including AWS, Google Cloud, and Cloudflare.
+
+## Features
+
+- RESTful API for all Brainy operations
+- Support for multiple storage backends (Memory, FileSystem, S3)
+- Configurable via environment variables
+- Deployment scripts for AWS, Google Cloud, and Cloudflare
+- Secure by default with Helmet middleware
+- Cross-origin resource sharing (CORS) support
+
+## Prerequisites
+
+- Node.js 23.11.0 or higher
+- npm or yarn
+- For cloud deployments:
+  - AWS: AWS CLI installed and configured
+  - Google Cloud: Google Cloud SDK installed and configured
+  - Cloudflare: Wrangler CLI installed and configured
+
+## Installation
+
+1. Clone the repository:
+   ```bash
+   git clone https://github.com/soulcraft/brainy.git
+   cd brainy/cloud-wrapper
+   ```
+
+2. Install dependencies:
+   ```bash
+   npm install --legacy-peer-deps
+   ```
+
+3. Create a `.env` file based on the example:
+   ```bash
+   cp .env.example .env
+   ```
+
+4. Edit the `.env` file to configure your environment.
+
+## Configuration
+
+The cloud wrapper can be configured using environment variables. See the `.env.example` file for available options.
+
+### Storage Options
+
+- `STORAGE_TYPE`: The type of storage to use. Options:
+  - `memory`: In-memory storage (default for Cloudflare)
+  - `filesystem`: File system storage (default for local and AWS/GCP)
+  - `s3`: S3-compatible storage (AWS S3, MinIO, etc.)
+  - `r2`: Cloudflare R2 storage (Cloudflare only)
+
+### S3 Storage Configuration
+
+When using `STORAGE_TYPE=s3`, the following environment variables are required:
+
+- `S3_BUCKET_NAME`: The name of the S3 bucket
+- `S3_ACCESS_KEY_ID`: Your S3 access key ID
+- `S3_SECRET_ACCESS_KEY`: Your S3 secret access key
+- `S3_REGION`: The S3 region (default: `us-east-1`)
+- `S3_ENDPOINT` (optional): Custom endpoint for S3-compatible services
+
+## Local Development
+
+1. Build the project:
+   ```bash
+   npm run build
+   ```
+
+2. Start the development server:
+   ```bash
+   npm run dev
+   ```
+
+3. The API will be available at `http://localhost:3000`.
+
+## API Endpoints
+
+Brainy Cloud Wrapper provides both REST API and WebSocket API for interacting with the database.
+
+### REST API
+
+#### Status
+
+- `GET /api/status`: Get database status
+
+#### Nouns (Entities)
+
+- `POST /api/nouns`: Add a new noun
+  - Body: `{ "text": "Your text", "metadata": { ... } }`
+- `GET /api/nouns/:id`: Get a noun by ID
+- `PUT /api/nouns/:id`: Update noun metadata
+  - Body: `{ "metadata": { ... } }`
+- `DELETE /api/nouns/:id`: Delete a noun
+
+#### Search
+
+- `POST /api/search`: Search for similar nouns
+  - Body: `{ "query": "Your search query", "limit": 10 }`
+
+#### Verbs (Relationships)
+
+- `POST /api/verbs`: Add a relationship between nouns
+  - Body: `{ "sourceId": "...", "targetId": "...", "metadata": { ... } }`
+- `GET /api/verbs`: Get all relationships
+- `GET /api/verbs/source/:id`: Get relationships by source
+- `GET /api/verbs/target/:id`: Get relationships by target
+- `DELETE /api/verbs/:id`: Delete a relationship
+
+#### Database Management
+
+- `DELETE /api/clear`: Clear all data
+
+### WebSocket API
+
+The WebSocket API provides real-time communication with the Brainy database. Connect to the WebSocket server at `ws://your-server:port`.
+
+#### Message Format
+
+All WebSocket messages follow this format:
+
+```json
+{
+  "type": "messageType",
+  "id": "unique-message-id",
+  "payload": {
+    // Message-specific data
+  }
+}
+```
+
+#### Available Message Types
+
+##### Status
+- Request: `{ "type": "status" }`
+- Response: `{ "type": "status", "id": "...", "payload": { /* database status */ } }`
+
+##### Nouns (Entities)
+- Add a noun:
+  - Request: `{ "type": "addNoun", "payload": { "text": "Your text", "metadata": { ... } } }`
+  - Response: `{ "type": "addNoun", "id": "...", "payload": { "id": "new-noun-id" } }`
+
+- Get a noun:
+  - Request: `{ "type": "getNoun", "payload": { "id": "noun-id" } }`
+  - Response: `{ "type": "getNoun", "id": "...", "payload": { /* noun data */ } }`
+
+- Update a noun:
+  - Request: `{ "type": "updateNoun", "payload": { "id": "noun-id", "metadata": { ... } } }`
+  - Response: `{ "type": "updateNoun", "id": "...", "payload": { "success": true } }`
+
+- Delete a noun:
+  - Request: `{ "type": "deleteNoun", "payload": { "id": "noun-id" } }`
+  - Response: `{ "type": "deleteNoun", "id": "...", "payload": { "success": true } }`
+
+##### Search
+- Search for similar nouns:
+  - Request: `{ "type": "search", "payload": { "query": "Your search query", "limit": 10 } }`
+  - Response: `{ "type": "search", "id": "...", "payload": [ /* search results */ ] }`
+
+##### Verbs (Relationships)
+- Add a verb:
+  - Request: `{ "type": "addVerb", "payload": { "sourceId": "...", "targetId": "...", "metadata": { ... } } }`
+  - Response: `{ "type": "addVerb", "id": "...", "payload": { "success": true } }`
+
+- Get all verbs:
+  - Request: `{ "type": "getVerbs" }`
+  - Response: `{ "type": "getVerbs", "id": "...", "payload": [ /* all verbs */ ] }`
+
+- Get verbs by source:
+  - Request: `{ "type": "getVerbsBySource", "payload": { "id": "source-id" } }`
+  - Response: `{ "type": "getVerbsBySource", "id": "...", "payload": [ /* verbs */ ] }`
+
+- Get verbs by target:
+  - Request: `{ "type": "getVerbsByTarget", "payload": { "id": "target-id" } }`
+  - Response: `{ "type": "getVerbsByTarget", "id": "...", "payload": [ /* verbs */ ] }`
+
+- Delete a verb:
+  - Request: `{ "type": "deleteVerb", "payload": { "id": "verb-id" } }`
+  - Response: `{ "type": "deleteVerb", "id": "...", "payload": { "success": true } }`
+
+##### Database Management
+- Clear all data:
+  - Request: `{ "type": "clear" }`
+  - Response: `{ "type": "clear", "id": "...", "payload": { "success": true } }`
+
+#### Real-time Subscriptions
+
+The WebSocket API supports subscribing to real-time updates:
+
+- Subscribe to updates:
+  - Request: `{ "type": "subscribe", "payload": { "type": "nouns" } }`
+  - Response: `{ "type": "subscribe", "id": "...", "payload": { "success": true, "type": "nouns" } }`
+
+- Unsubscribe from updates:
+  - Request: `{ "type": "unsubscribe", "payload": { "type": "nouns" } }`
+  - Response: `{ "type": "unsubscribe", "id": "...", "payload": { "success": true, "type": "nouns" } }`
+
+Available subscription types:
+- `nouns`: Updates about nouns (added, updated, deleted)
+- `verbs`: Updates about verbs (added, deleted)
+- `searchResults`: Updates about search results
+
+When subscribed, you'll receive messages when relevant events occur:
+
+```json
+{
+  "type": "subscribe",
+  "payload": {
+    "type": "nouns",
+    "data": {
+      "type": "added",
+      "id": "noun-id",
+      "data": { /* noun data */ }
+    }
+  }
+}
+```
+
+## Cloud Deployment
+
+### AWS Lambda and API Gateway
+
+1. Configure AWS-specific environment variables:
+   ```
+   AWS_REGION=us-east-1
+   AWS_FUNCTION_NAME=brainy-cloud-service
+   AWS_API_GATEWAY_NAME=brainy-api
+   AWS_STAGE_NAME=prod
+   AWS_ACCOUNT_ID=your-account-id
+   ```
+
+2. Deploy to AWS:
+   ```bash
+   npm run deploy:aws
+   ```
+
+### Google Cloud Run
+
+1. Configure GCP-specific environment variables:
+   ```
+   GCP_PROJECT_ID=your-project-id
+   GCP_REGION=us-central1
+   GCP_SERVICE_NAME=brainy-cloud-service
+   GCP_IMAGE_NAME=brainy-cloud-service
+   GCP_MEMORY=512Mi
+   GCP_CPU=1
+   GCP_MAX_INSTANCES=10
+   GCP_MIN_INSTANCES=0
+   ```
+
+2. Deploy to Google Cloud:
+   ```bash
+   npm run deploy:gcp
+   ```
+
+### Cloudflare Workers
+
+1. Configure Cloudflare-specific environment variables:
+   ```
+   CF_ACCOUNT_ID=your-account-id
+   CF_WORKER_NAME=brainy-cloud-service
+   CF_KV_NAMESPACE=BRAINY_STORAGE
+   CF_R2_BUCKET=brainy-storage
+   ```
+
+2. Deploy to Cloudflare:
+   ```bash
+   npm run deploy:cloudflare
+   ```
+
+## Storage Considerations
+
+### AWS Lambda
+
+When deploying to AWS Lambda, it's recommended to use S3 storage for persistence. The filesystem storage option will work but data will be lost when the Lambda function is recycled.
+
+### Google Cloud Run
+
+For Google Cloud Run, you can use either filesystem storage (for ephemeral storage) or S3-compatible storage (like Google Cloud Storage with an S3 compatibility layer).
+
+### Cloudflare Workers
+
+Cloudflare Workers have limited storage options. The recommended approach is to use:
+- Cloudflare KV for small datasets
+- Cloudflare R2 for larger datasets
+- Memory storage for temporary data
+
+## License
+
+MIT

cloud-wrapper/package.json

@@ -0,0 +1,85 @@
+{
+  "name": "@soulcraft/brainy-cloud",
+  "version": "0.1.0",
+  "description": "Cloud deployment wrapper for Brainy vector graph database",
+  "main": "dist/index.js",
+  "type": "module",
+  "scripts": {
+    "build": "tsc",
+    "start": "node dist/index.js",
+    "dev": "nodemon --exec node --experimental-specifier-resolution=node --loader ts-node/esm src/index.ts",
+    "deploy:aws": "node scripts/deploy-aws.js",
+    "deploy:gcp": "node scripts/deploy-gcp.js",
+    "deploy:cloudflare": "node scripts/deploy-cloudflare.js"
+  },
+  "keywords": [
+    "brainy",
+    "vector-database",
+    "cloud",
+    "aws",
+    "google-cloud",
+    "cloudflare"
+  ],
+  "author": "David Snelling (david@soulcraft.com)",
+  "license": "MIT",
+  "dependencies": {
+    "@soulcraft/brainy": "^0.7.5",
+    "cors": "^2.8.5",
+    "dotenv": "^16.3.1",
+    "express": "^4.18.2",
+    "helmet": "^7.1.0",
+    "morgan": "^1.10.0",
+    "uuid": "^9.0.1",
+    "ws": "^8.16.0"
+  },
+  "devDependencies": {
+    "@types/cors": "^2.8.17",
+    "@types/express": "^4.17.21",
+    "@types/morgan": "^1.9.9",
+    "@types/node": "^20.10.5",
+    "@types/uuid": "^9.0.7",
+    "@types/ws": "^8.5.10",
+    "nodemon": "^3.0.2",
+    "ts-node": "^10.9.2",
+    "typescript": "^5.3.3"
+  },
+  "engines": {
+    "node": ">=23.11.0"
+  },
+  "prettier": {
+    "arrowParens": "always",
+    "bracketSameLine": true,
+    "bracketSpacing": true,
+    "htmlWhitespaceSensitivity": "css",
+    "printWidth": 80,
+    "proseWrap": "preserve",
+    "semi": false,
+    "singleQuote": true,
+    "tabWidth": 2,
+    "trailingComma": "none",
+    "useTabs": false
+  },
+  "eslintConfig": {
+    "root": true,
+    "extends": [
+      "eslint:recommended",
+      "plugin:@typescript-eslint/recommended"
+    ],
+    "parser": "@typescript-eslint/parser",
+    "plugins": [
+      "@typescript-eslint"
+    ],
+    "rules": {
+      "no-unused-vars": "off",
+      "@typescript-eslint/no-unused-vars": [
+        "warn",
+        {
+          "args": "after-used",
+          "argsIgnorePattern": "^_"
+        }
+      ],
+      "semi": ["error", "never"],
+      "@typescript-eslint/semi": ["error", "never"]
+    }
+  }
+}

cloud-wrapper/scripts/deploy-aws.js

@@ -0,0 +1,205 @@
+#!/usr/bin/env node
+
+/**
+ * AWS Deployment Script for Brainy Cloud Wrapper
+ * 
+ * This script helps deploy the Brainy cloud wrapper to AWS Lambda and API Gateway.
+ * It uses the AWS SDK to create and configure the necessary resources.
+ * 
+ * Prerequisites:
+ * - AWS CLI installed and configured with appropriate credentials
+ * - Node.js 23.11.0 or higher
+ * - Brainy cloud wrapper built (npm run build)
+ */
+
+import { execSync } from 'child_process';
+import fs from 'fs';
+import path from 'path';
+import dotenv from 'dotenv';
+
+// Load environment variables
+dotenv.config();
+
+// Configuration
+const config = {
+  region: process.env.AWS_REGION || 'us-east-1',
+  functionName: process.env.AWS_FUNCTION_NAME || 'brainy-cloud-service',
+  s3Bucket: process.env.S3_BUCKET_NAME,
+  s3Key: process.env.S3_ACCESS_KEY_ID,
+  s3Secret: process.env.S3_SECRET_ACCESS_KEY,
+  apiGatewayName: process.env.AWS_API_GATEWAY_NAME || 'brainy-api',
+  stageName: process.env.AWS_STAGE_NAME || 'prod'
+};
+
+// Validate configuration
+if (!config.s3Bucket && process.env.STORAGE_TYPE === 's3') {
+  console.error('Error: S3 bucket name is required when using S3 storage');
+  process.exit(1);
+}
+
+// Create deployment package
+function createDeploymentPackage() {
+  console.log('Creating deployment package...');
+  
+  try {
+    // Create a temporary directory for the deployment package
+    if (!fs.existsSync('deploy')) {
+      fs.mkdirSync('deploy');
+    }
+    
+    // Copy necessary files
+    execSync('cp -r dist deploy/');
+    execSync('cp package.json deploy/');
+    execSync('cp .env deploy/');
+    
+    // Create a zip file
+    execSync('cd deploy && zip -r ../deployment.zip .');
+    
+    console.log('Deployment package created successfully');
+  } catch (error) {
+    console.error('Error creating deployment package:', error);
+    process.exit(1);
+  }
+}
+
+// Deploy to AWS Lambda
+function deployToLambda() {
+  console.log('Deploying to AWS Lambda...');
+  
+  try {
+    // Check if function exists
+    try {
+      execSync(`aws lambda get-function --function-name ${config.functionName} --region ${config.region}`);
+      
+      // Update existing function
+      execSync(`aws lambda update-function-code --function-name ${config.functionName} --zip-file fileb://deployment.zip --region ${config.region}`);
+      console.log(`Lambda function ${config.functionName} updated successfully`);
+    } catch (error) {
+      // Create new function
+      execSync(`aws lambda create-function --function-name ${config.functionName} --runtime nodejs20.x --handler dist/index.handler --zip-file fileb://deployment.zip --role arn:aws:iam::${process.env.AWS_ACCOUNT_ID}:role/lambda-basic-execution --region ${config.region}`);
+      console.log(`Lambda function ${config.functionName} created successfully`);
+    }
+    
+    // Configure environment variables
+    const envVars = {
+      Variables: {
+        NODE_ENV: 'production',
+        STORAGE_TYPE: process.env.STORAGE_TYPE || 'filesystem',
+        S3_BUCKET_NAME: config.s3Bucket,
+        S3_ACCESS_KEY_ID: config.s3Key,
+        S3_SECRET_ACCESS_KEY: config.s3Secret,
+        S3_REGION: config.region
+      }
+    };
+    
+    execSync(`aws lambda update-function-configuration --function-name ${config.functionName} --environment '${JSON.stringify(envVars)}' --region ${config.region}`);
+    console.log('Lambda function environment variables configured');
+    
+  } catch (error) {
+    console.error('Error deploying to AWS Lambda:', error);
+    process.exit(1);
+  }
+}
+
+// Create API Gateway
+function createApiGateway() {
+  console.log('Creating API Gateway...');
+  
+  try {
+    // Check if API Gateway exists
+    let apiId;
+    try {
+      const result = execSync(`aws apigateway get-rest-apis --region ${config.region}`).toString();
+      const apis = JSON.parse(result).items;
+      const api = apis.find(api => api.name === config.apiGatewayName);
+      
+      if (api) {
+        apiId = api.id;
+        console.log(`Using existing API Gateway: ${apiId}`);
+      } else {
+        throw new Error('API Gateway not found');
+      }
+    } catch (error) {
+      // Create new API Gateway
+      const result = execSync(`aws apigateway create-rest-api --name ${config.apiGatewayName} --region ${config.region}`).toString();
+      apiId = JSON.parse(result).id;
+      console.log(`API Gateway created: ${apiId}`);
+    }
+    
+    // Get root resource ID
+    const resourcesResult = execSync(`aws apigateway get-resources --rest-api-id ${apiId} --region ${config.region}`).toString();
+    const rootResourceId = JSON.parse(resourcesResult).items.find(resource => resource.path === '/').id;
+    
+    // Create proxy resource
+    let proxyResourceId;
+    try {
+      const proxyResource = JSON.parse(resourcesResult).items.find(resource => resource.path === '/{proxy+}');
+      if (proxyResource) {
+        proxyResourceId = proxyResource.id;
+      } else {
+        throw new Error('Proxy resource not found');
+      }
+    } catch (error) {
+      const proxyResult = execSync(`aws apigateway create-resource --rest-api-id ${apiId} --parent-id ${rootResourceId} --path-part {proxy+} --region ${config.region}`).toString();
+      proxyResourceId = JSON.parse(proxyResult).id;
+    }
+    
+    // Create ANY method
+    try {
+      execSync(`aws apigateway put-method --rest-api-id ${apiId} --resource-id ${proxyResourceId} --http-method ANY --authorization-type NONE --region ${config.region}`);
+    } catch (error) {
+      console.log('Method already exists, skipping...');
+    }
+    
+    // Create integration
+    try {
+      execSync(`aws apigateway put-integration --rest-api-id ${apiId} --resource-id ${proxyResourceId} --http-method ANY --type AWS_PROXY --integration-http-method POST --uri arn:aws:apigateway:${config.region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${config.region}:${process.env.AWS_ACCOUNT_ID}:function:${config.functionName}/invocations --region ${config.region}`);
+    } catch (error) {
+      console.log('Integration already exists, skipping...');
+    }
+    
+    // Deploy API
+    execSync(`aws apigateway create-deployment --rest-api-id ${apiId} --stage-name ${config.stageName} --region ${config.region}`);
+    
+    console.log(`API Gateway deployed: https://${apiId}.execute-api.${config.region}.amazonaws.com/${config.stageName}`);
+    
+  } catch (error) {
+    console.error('Error creating API Gateway:', error);
+    process.exit(1);
+  }
+}
+
+// Clean up
+function cleanup() {
+  console.log('Cleaning up...');
+  
+  try {
+    execSync('rm -rf deploy');
+    execSync('rm -f deployment.zip');
+    
+    console.log('Cleanup completed');
+  } catch (error) {
+    console.error('Error during cleanup:', error);
+  }
+}
+
+// Main function
+async function main() {
+  try {
+    console.log('Starting AWS deployment...');
+    
+    createDeploymentPackage();
+    deployToLambda();
+    createApiGateway();
+    cleanup();
+    
+    console.log('Deployment completed successfully');
+  } catch (error) {
+    console.error('Deployment failed:', error);
+    cleanup();
+    process.exit(1);
+  }
+}
+
+// Run the script
+main();

cloud-wrapper/scripts/deploy-cloudflare.js

@@ -0,0 +1,245 @@
+#!/usr/bin/env node
+
+/**
+ * Cloudflare Deployment Script for Brainy Cloud Wrapper
+ * 
+ * This script helps deploy the Brainy cloud wrapper to Cloudflare Workers.
+ * It uses the Wrangler CLI to create and configure the necessary resources.
+ * 
+ * Prerequisites:
+ * - Wrangler CLI installed and configured with appropriate credentials
+ * - Node.js 23.11.0 or higher
+ * - Brainy cloud wrapper built (npm run build)
+ */
+
+import { execSync } from 'child_process';
+import fs from 'fs';
+import path from 'path';
+import dotenv from 'dotenv';
+
+// Load environment variables
+dotenv.config();
+
+// Configuration
+const config = {
+  workerName: process.env.CF_WORKER_NAME || 'brainy-cloud-service',
+  accountId: process.env.CF_ACCOUNT_ID,
+  kvNamespace: process.env.CF_KV_NAMESPACE || 'BRAINY_STORAGE',
+  r2Bucket: process.env.CF_R2_BUCKET || 'brainy-storage'
+};
+
+// Validate configuration
+if (!config.accountId) {
+  console.error('Error: CF_ACCOUNT_ID environment variable is required');
+  process.exit(1);
+}
+
+// Create wrangler.toml configuration
+function createWranglerConfig() {
+  console.log('Creating wrangler.toml configuration...');
+  
+  const wranglerContent = `
+name = "${config.workerName}"
+main = "dist/worker.js"
+compatibility_date = "2023-12-01"
+node_compat = true
+
+account_id = "${config.accountId}"
+
+[build]
+command = "npm run build"
+
+[vars]
+NODE_ENV = "production"
+STORAGE_TYPE = "${process.env.STORAGE_TYPE || 'memory'}"
+USE_SIMPLE_EMBEDDING = "${process.env.USE_SIMPLE_EMBEDDING || 'false'}"
+
+# KV Namespace for storage
+[[kv_namespaces]]
+binding = "BRAINY_KV"
+id = "${process.env.CF_KV_NAMESPACE_ID || 'create_kv_namespace_and_add_id_here'}"
+
+# R2 Bucket for storage (if using R2)
+[[r2_buckets]]
+binding = "BRAINY_R2"
+bucket_name = "${config.r2Bucket}"
+`;
+  
+  try {
+    fs.writeFileSync('wrangler.toml', wranglerContent);
+    console.log('wrangler.toml created successfully');
+  } catch (error) {
+    console.error('Error creating wrangler.toml:', error);
+    process.exit(1);
+  }
+}
+
+// Create Cloudflare Worker adapter
+function createWorkerAdapter() {
+  console.log('Creating Cloudflare Worker adapter...');
+  
+  const workerContent = `
+import { createServer } from '@cloudflare/workers-adapter';
+import app from './index.js';
+
+// Create a fetch handler for the worker
+export default {
+  async fetch(request, env, ctx) {
+    // Add environment variables to process.env
+    process.env = {
+      ...process.env,
+      ...env.vars,
+      CF_KV_NAMESPACE: env.BRAINY_KV,
+      CF_R2_BUCKET: env.BRAINY_R2
+    };
+
+    // Create a server adapter
+    const server = createServer(app);
+    
+    // Handle the request
+    return server.fetch(request, env, ctx);
+  }
+};
+`;
+  
+  try {
+    // Create dist directory if it doesn't exist
+    if (!fs.existsSync('dist')) {
+      fs.mkdirSync('dist');
+    }
+    
+    fs.writeFileSync('dist/worker.js', workerContent);
+    console.log('Worker adapter created successfully');
+  } catch (error) {
+    console.error('Error creating worker adapter:', error);
+    process.exit(1);
+  }
+}
+
+// Update package.json to include Cloudflare Workers dependencies
+function updatePackageJson() {
+  console.log('Updating package.json...');
+  
+  try {
+    const packageJson = JSON.parse(fs.readFileSync('package.json', 'utf8'));
+    
+    // Add Cloudflare Workers dependencies
+    packageJson.dependencies = {
+      ...packageJson.dependencies,
+      '@cloudflare/workers-adapter': '^1.1.0',
+      '@cloudflare/kv-asset-handler': '^0.3.0'
+    };
+    
+    // Add Cloudflare Workers scripts
+    packageJson.scripts = {
+      ...packageJson.scripts,
+      'deploy:cloudflare': 'wrangler deploy'
+    };
+    
+    fs.writeFileSync('package.json', JSON.stringify(packageJson, null, 2));
+    console.log('package.json updated successfully');
+    
+    // Install new dependencies
+    execSync('npm install --legacy-peer-deps');
+  } catch (error) {
+    console.error('Error updating package.json:', error);
+    process.exit(1);
+  }
+}
+
+// Create KV namespace if it doesn't exist
+function createKVNamespace() {
+  console.log('Creating KV namespace...');
+  
+  try {
+    // Check if KV namespace ID is provided
+    if (process.env.CF_KV_NAMESPACE_ID) {
+      console.log(`Using existing KV namespace: ${process.env.CF_KV_NAMESPACE_ID}`);
+      return;
+    }
+    
+    // Create KV namespace
+    const result = execSync(`wrangler kv:namespace create "${config.kvNamespace}"`).toString();
+    const match = result.match(/id = "([^"]+)"/);
+    
+    if (match && match[1]) {
+      const namespaceId = match[1];
+      console.log(`KV namespace created with ID: ${namespaceId}`);
+      
+      // Update wrangler.toml with the new namespace ID
+      let wranglerContent = fs.readFileSync('wrangler.toml', 'utf8');
+      wranglerContent = wranglerContent.replace(/id = "create_kv_namespace_and_add_id_here"/, `id = "${namespaceId}"`);
+      fs.writeFileSync('wrangler.toml', wranglerContent);
+    } else {
+      console.error('Failed to extract KV namespace ID from wrangler output');
+    }
+  } catch (error) {
+    console.error('Error creating KV namespace:', error);
+    console.log('You may need to create the KV namespace manually and update wrangler.toml');
+  }
+}
+
+// Create R2 bucket if it doesn't exist
+function createR2Bucket() {
+  console.log('Creating R2 bucket...');
+  
+  try {
+    // Only create R2 bucket if using R2 storage
+    if (process.env.STORAGE_TYPE !== 'r2') {
+      console.log('Skipping R2 bucket creation (not using R2 storage)');
+      return;
+    }
+    
+    // Create R2 bucket
+    execSync(`wrangler r2 bucket create ${config.r2Bucket}`);
+    console.log(`R2 bucket created: ${config.r2Bucket}`);
+  } catch (error) {
+    console.error('Error creating R2 bucket:', error);
+    console.log('You may need to create the R2 bucket manually');
+  }
+}
+
+// Deploy to Cloudflare Workers
+function deployToCloudflare() {
+  console.log('Deploying to Cloudflare Workers...');
+  
+  try {
+    execSync('wrangler deploy', { stdio: 'inherit' });
+    console.log('Deployed to Cloudflare Workers successfully');
+  } catch (error) {
+    console.error('Error deploying to Cloudflare Workers:', error);
+    process.exit(1);
+  }
+}
+
+// Clean up
+function cleanup() {
+  console.log('Cleaning up...');
+  
+  // No cleanup needed for now
+  console.log('Cleanup completed');
+}
+
+// Main function
+async function main() {
+  try {
+    console.log('Starting Cloudflare deployment...');
+    
+    createWranglerConfig();
+    createWorkerAdapter();
+    updatePackageJson();
+    createKVNamespace();
+    createR2Bucket();
+    deployToCloudflare();
+    cleanup();
+    
+    console.log('Deployment completed successfully');
+  } catch (error) {
+    console.error('Deployment failed:', error);
+    cleanup();
+    process.exit(1);
+  }
+}
+
+// Run the script
+main();

cloud-wrapper/scripts/deploy-gcp.js

@@ -0,0 +1,196 @@
+#!/usr/bin/env node
+
+/**
+ * Google Cloud Platform Deployment Script for Brainy Cloud Wrapper
+ * 
+ * This script helps deploy the Brainy cloud wrapper to Google Cloud Run.
+ * It uses the Google Cloud SDK to create and configure the necessary resources.
+ * 
+ * Prerequisites:
+ * - Google Cloud SDK installed and configured with appropriate credentials
+ * - Node.js 23.11.0 or higher
+ * - Brainy cloud wrapper built (npm run build)
+ * - Docker installed (for building container images)
+ */
+
+import { execSync } from 'child_process';
+import fs from 'fs';
+import path from 'path';
+import dotenv from 'dotenv';
+
+// Load environment variables
+dotenv.config();
+
+// Configuration
+const config = {
+  projectId: process.env.GCP_PROJECT_ID,
+  region: process.env.GCP_REGION || 'us-central1',
+  serviceName: process.env.GCP_SERVICE_NAME || 'brainy-cloud-service',
+  imageName: process.env.GCP_IMAGE_NAME || 'brainy-cloud-service',
+  memory: process.env.GCP_MEMORY || '512Mi',
+  cpu: process.env.GCP_CPU || '1',
+  maxInstances: process.env.GCP_MAX_INSTANCES || '10',
+  minInstances: process.env.GCP_MIN_INSTANCES || '0'
+};
+
+// Validate configuration
+if (!config.projectId) {
+  console.error('Error: GCP_PROJECT_ID environment variable is required');
+  process.exit(1);
+}
+
+// Create Dockerfile
+function createDockerfile() {
+  console.log('Creating Dockerfile...');
+  
+  const dockerfileContent = `
+FROM node:23-slim
+
+WORKDIR /app
+
+# Copy package.json and package-lock.json
+COPY package*.json ./
+
+# Install dependencies
+RUN npm install --production --legacy-peer-deps
+
+# Copy built application
+COPY dist/ ./dist/
+
+# Copy environment variables
+COPY .env ./
+
+# Expose port
+EXPOSE 8080
+
+# Start the application
+CMD ["node", "dist/index.js"]
+`;
+  
+  try {
+    fs.writeFileSync('Dockerfile', dockerfileContent);
+    console.log('Dockerfile created successfully');
+  } catch (error) {
+    console.error('Error creating Dockerfile:', error);
+    process.exit(1);
+  }
+}
+
+// Build and push Docker image
+function buildAndPushImage() {
+  console.log('Building and pushing Docker image...');
+  
+  try {
+    // Set Google Cloud project
+    execSync(`gcloud config set project ${config.projectId}`);
+    
+    // Build the Docker image
+    execSync(`docker build -t gcr.io/${config.projectId}/${config.imageName} .`);
+    
+    // Configure Docker to use gcloud as a credential helper
+    execSync('gcloud auth configure-docker');
+    
+    // Push the image to Google Container Registry
+    execSync(`docker push gcr.io/${config.projectId}/${config.imageName}`);
+    
+    console.log('Docker image built and pushed successfully');
+  } catch (error) {
+    console.error('Error building and pushing Docker image:', error);
+    process.exit(1);
+  }
+}
+
+// Deploy to Google Cloud Run
+function deployToCloudRun() {
+  console.log('Deploying to Google Cloud Run...');
+  
+  try {
+    // Create environment variables string
+    const envVars = [
+      `NODE_ENV=production`,
+      `PORT=8080`,
+      `STORAGE_TYPE=${process.env.STORAGE_TYPE || 'filesystem'}`
+    ];
+    
+    // Add S3 environment variables if using S3 storage
+    if (process.env.STORAGE_TYPE === 's3') {
+      envVars.push(`S3_BUCKET_NAME=${process.env.S3_BUCKET_NAME}`);
+      envVars.push(`S3_ACCESS_KEY_ID=${process.env.S3_ACCESS_KEY_ID}`);
+      envVars.push(`S3_SECRET_ACCESS_KEY=${process.env.S3_SECRET_ACCESS_KEY}`);
+      envVars.push(`S3_REGION=${process.env.S3_REGION || 'us-east-1'}`);
+      
+      if (process.env.S3_ENDPOINT) {
+        envVars.push(`S3_ENDPOINT=${process.env.S3_ENDPOINT}`);
+      }
+    }
+    
+    // Add embedding and HNSW configuration if provided
+    if (process.env.USE_SIMPLE_EMBEDDING) {
+      envVars.push(`USE_SIMPLE_EMBEDDING=${process.env.USE_SIMPLE_EMBEDDING}`);
+    }
+    
+    if (process.env.HNSW_M) {
+      envVars.push(`HNSW_M=${process.env.HNSW_M}`);
+      envVars.push(`HNSW_EF_CONSTRUCTION=${process.env.HNSW_EF_CONSTRUCTION || '200'}`);
+      envVars.push(`HNSW_EF_SEARCH=${process.env.HNSW_EF_SEARCH || '50'}`);
+    }
+    
+    // Deploy to Cloud Run
+    const envVarsString = envVars.map(env => `--set-env-vars="${env}"`).join(' ');
+    
+    execSync(`gcloud run deploy ${config.serviceName} \
+      --image=gcr.io/${config.projectId}/${config.imageName} \
+      --platform=managed \
+      --region=${config.region} \
+      --memory=${config.memory} \
+      --cpu=${config.cpu} \
+      --max-instances=${config.maxInstances} \
+      --min-instances=${config.minInstances} \
+      --allow-unauthenticated \
+      ${envVarsString}`);
+    
+    console.log('Deployed to Google Cloud Run successfully');
+    
+    // Get the service URL
+    const serviceUrl = execSync(`gcloud run services describe ${config.serviceName} --region=${config.region} --format="value(status.url)"`).toString().trim();
+    console.log(`Service URL: ${serviceUrl}`);
+  } catch (error) {
+    console.error('Error deploying to Google Cloud Run:', error);
+    process.exit(1);
+  }
+}
+
+// Clean up
+function cleanup() {
+  console.log('Cleaning up...');
+  
+  try {
+    // Remove Dockerfile
+    fs.unlinkSync('Dockerfile');
+    
+    console.log('Cleanup completed');
+  } catch (error) {
+    console.error('Error during cleanup:', error);
+  }
+}
+
+// Main function
+async function main() {
+  try {
+    console.log('Starting Google Cloud Platform deployment...');
+    
+    createDockerfile();
+    buildAndPushImage();
+    deployToCloudRun();
+    cleanup();
+    
+    console.log('Deployment completed successfully');
+  } catch (error) {
+    console.error('Deployment failed:', error);
+    cleanup();
+    process.exit(1);
+  }
+}
+
+// Run the script
+main();

cloud-wrapper/src/index.ts

@@ -0,0 +1,86 @@
+import express from 'express';
+import cors from 'cors';
+import helmet from 'helmet';
+import morgan from 'morgan';
+import dotenv from 'dotenv';
+import http from 'http';
+import { WebSocketServer } from 'ws';
+import { BrainyData } from '@soulcraft/brainy';
+import { setupRoutes } from './routes.js';
+import { initializeBrainy } from './services/brainyService.js';
+import { setupWebSocketHandlers } from './websocket.js';
+
+// Load environment variables
+dotenv.config();
+
+// Initialize Express app
+const app = express();
+const port = process.env.PORT || 3000;
+
+// Middleware
+app.use(helmet()); // Security headers
+app.use(cors()); // Enable CORS
+app.use(morgan('combined')); // Logging
+app.use(express.json()); // Parse JSON bodies
+
+// Initialize Brainy
+let brainyInstance: BrainyData;
+
+// Health check endpoint
+app.get('/health', (req, res) => {
+  res.status(200).json({ status: 'ok' });
+});
+
+// Setup API routes
+const apiRouter = setupRoutes();
+app.use('/api', (req, res, next) => {
+  // Attach brainy instance to request
+  (req as any).brainy = brainyInstance;
+  next();
+}, apiRouter);
+
+// Start the server
+async function startServer() {
+  try {
+    // Initialize Brainy
+    brainyInstance = await initializeBrainy();
+    console.log('Brainy initialized successfully');
+
+    // Create HTTP server
+    const server = http.createServer(app);
+
+    // Create WebSocket server
+    const wss = new WebSocketServer({ server });
+
+    // Setup WebSocket handlers
+    setupWebSocketHandlers(wss, brainyInstance);
+    console.log('WebSocket server initialized');
+
+    // Start HTTP server
+    server.listen(port, () => {
+      console.log(`Server running on port ${port}`);
+      console.log(`WebSocket server running on ws://localhost:${port}`);
+    });
+  } catch (error) {
+    console.error('Failed to start server:', error);
+    process.exit(1);
+  }
+}
+
+// Handle graceful shutdown
+process.on('SIGTERM', async () => {
+  console.log('SIGTERM received, shutting down gracefully');
+  // Cleanup Brainy resources
+  await brainyInstance?.clear();
+  process.exit(0);
+});
+
+process.on('SIGINT', async () => {
+  console.log('SIGINT received, shutting down gracefully');
+  // Cleanup Brainy resources
+  await brainyInstance?.clear();
+  process.exit(0);
+});
+
+// Start the server
+startServer();

cloud-wrapper/src/routes.ts

@@ -0,0 +1,186 @@
+import { Router, Request, Response, NextFunction, RequestHandler } from 'express';
+import { BrainyData, NounType, VerbType } from '@soulcraft/brainy';
+
+// Define a custom request type that includes the Brainy instance
+interface BrainyRequest extends Request {
+  brainy: BrainyData;
+}
+
+// Helper function to handle async route handlers
+const asyncHandler = (fn: (req: BrainyRequest, res: Response, next: NextFunction) => Promise<any>) => 
+  (req: Request, res: Response, next: NextFunction) => {
+    Promise.resolve(fn(req as BrainyRequest, res, next)).catch(next);
+  };
+
+export function setupRoutes() {
+  const router = Router();
+
+  // Get database status
+  router.get('/status', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const status = await req.brainy.status();
+      res.status(200).json(status);
+    } catch (error) {
+      console.error('Error getting status:', error);
+      res.status(500).json({ error: 'Failed to get database status' });
+    }
+  }));
+
+  // Add a noun (entity)
+  router.post('/nouns', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const { text, metadata } = req.body;
+
+      if (!text) {
+        return res.status(400).json({ error: 'Text is required' });
+      }
+
+      const id = await req.brainy.add(text, metadata || {});
+      res.status(201).json({ id });
+    } catch (error) {
+      console.error('Error adding noun:', error);
+      res.status(500).json({ error: 'Failed to add noun' });
+    }
+  }));
+
+  // Get a noun by ID
+  router.get('/nouns/:id', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const { id } = req.params;
+      const noun = await req.brainy.get(id);
+
+      if (!noun) {
+        return res.status(404).json({ error: 'Noun not found' });
+      }
+
+      res.status(200).json(noun);
+    } catch (error) {
+      console.error('Error getting noun:', error);
+      res.status(500).json({ error: 'Failed to get noun' });
+    }
+  }));
+
+  // Update noun metadata
+  router.put('/nouns/:id', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const { id } = req.params;
+      const { metadata } = req.body;
+
+      if (!metadata) {
+        return res.status(400).json({ error: 'Metadata is required' });
+      }
+
+      await req.brainy.updateMetadata(id, metadata);
+      res.status(200).json({ success: true });
+    } catch (error) {
+      console.error('Error updating noun:', error);
+      res.status(500).json({ error: 'Failed to update noun' });
+    }
+  }));
+
+  // Delete a noun
+  router.delete('/nouns/:id', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const { id } = req.params;
+      await req.brainy.delete(id);
+      res.status(200).json({ success: true });
+    } catch (error) {
+      console.error('Error deleting noun:', error);
+      res.status(500).json({ error: 'Failed to delete noun' });
+    }
+  }));
+
+  // Search for similar nouns
+  router.post('/search', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const { query, limit = 10 } = req.body;
+
+      if (!query) {
+        return res.status(400).json({ error: 'Query is required' });
+      }
+
+      const results = await req.brainy.searchText(query, limit);
+      res.status(200).json(results);
+    } catch (error) {
+      console.error('Error searching:', error);
+      res.status(500).json({ error: 'Failed to search' });
+    }
+  }));
+
+  // Add a verb (relationship)
+  router.post('/verbs', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const { sourceId, targetId, metadata } = req.body;
+
+      if (!sourceId || !targetId) {
+        return res.status(400).json({ error: 'Source ID and Target ID are required' });
+      }
+
+      await req.brainy.addVerb(sourceId, targetId, metadata || {});
+      res.status(201).json({ success: true });
+    } catch (error) {
+      console.error('Error adding verb:', error);
+      res.status(500).json({ error: 'Failed to add verb' });
+    }
+  }));
+
+  // Get all verbs
+  router.get('/verbs', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const verbs = await req.brainy.getAllVerbs();
+      res.status(200).json(verbs);
+    } catch (error) {
+      console.error('Error getting verbs:', error);
+      res.status(500).json({ error: 'Failed to get verbs' });
+    }
+  }));
+
+  // Get verbs by source
+  router.get('/verbs/source/:id', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const { id } = req.params;
+      const verbs = await req.brainy.getVerbsBySource(id);
+      res.status(200).json(verbs);
+    } catch (error) {
+      console.error('Error getting verbs by source:', error);
+      res.status(500).json({ error: 'Failed to get verbs by source' });
+    }
+  }));
+
+  // Get verbs by target
+  router.get('/verbs/target/:id', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const { id } = req.params;
+      const verbs = await req.brainy.getVerbsByTarget(id);
+      res.status(200).json(verbs);
+    } catch (error) {
+      console.error('Error getting verbs by target:', error);
+      res.status(500).json({ error: 'Failed to get verbs by target' });
+    }
+  }));
+
+  // Delete a verb
+  router.delete('/verbs/:id', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      const { id } = req.params;
+      await req.brainy.deleteVerb(id);
+      res.status(200).json({ success: true });
+    } catch (error) {
+      console.error('Error deleting verb:', error);
+      res.status(500).json({ error: 'Failed to delete verb' });
+    }
+  }));
+
+  // Clear all data
+  router.delete('/clear', asyncHandler(async (req: BrainyRequest, res: Response, next: NextFunction) => {
+    try {
+      await req.brainy.clear();
+      res.status(200).json({ success: true });
+    } catch (error) {
+      console.error('Error clearing data:', error);
+      res.status(500).json({ error: 'Failed to clear data' });
+    }
+  }));
+
+  return router;
+}

cloud-wrapper/src/services/brainyService.ts

@@ -0,0 +1,97 @@
+import { BrainyData, BrainyDataConfig } from '@soulcraft/brainy'
+import dotenv from 'dotenv'
+
+// Load environment variables
+dotenv.config()
+
+/**
+ * Extended configuration interface that includes rootDirectory
+ */
+interface ExtendedBrainyDataConfig extends BrainyDataConfig {
+  storage: BrainyDataConfig['storage'] & {
+    rootDirectory?: string
+  }
+}
+
+/**
+ * Initialize the Brainy instance with appropriate configuration
+ * based on environment variables
+ */
+export async function initializeBrainy(): Promise<BrainyData> {
+  // Get storage configuration from environment variables
+  const storageType = process.env.STORAGE_TYPE || 'filesystem'
+
+  // Create configuration object
+  const config: ExtendedBrainyDataConfig = {
+    storage: {}
+  }
+
+  // Configure storage based on type
+  switch (storageType) {
+    case 's3':
+      // Configure S3 storage
+      if (process.env.S3_ENDPOINT) {
+        // Use customS3Storage for S3-compatible services with custom endpoints
+        config.storage.customS3Storage = {
+          bucketName: process.env.S3_BUCKET_NAME,
+          accessKeyId: process.env.S3_ACCESS_KEY_ID,
+          secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
+          region: process.env.S3_REGION || 'us-east-1',
+          endpoint: process.env.S3_ENDPOINT
+        }
+      } else {
+        // Use standard S3 storage for AWS S3
+        config.storage.s3Storage = {
+          bucketName: process.env.S3_BUCKET_NAME,
+          accessKeyId: process.env.S3_ACCESS_KEY_ID,
+          secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
+          region: process.env.S3_REGION || 'us-east-1'
+        }
+      }
+      break
+
+    case 'filesystem':
+      // Configure filesystem storage
+      config.storage.rootDirectory = process.env.STORAGE_ROOT_DIR || './data'
+      break
+
+    case 'memory':
+      // No additional configuration needed for memory storage
+      break
+
+    default:
+      // Default to filesystem storage
+      config.storage.rootDirectory = process.env.STORAGE_ROOT_DIR || './data'
+      break
+  }
+
+  // Note: Universal Sentence Encoder is now the only embedding option
+  // TensorFlow.js is required for embedding to work
+
+  // Configure HNSW index parameters if provided
+  if (process.env.HNSW_M) {
+    config.hnsw = {
+      M: parseInt(process.env.HNSW_M, 10),
+      efConstruction: process.env.HNSW_EF_CONSTRUCTION
+        ? parseInt(process.env.HNSW_EF_CONSTRUCTION, 10)
+        : 200,
+      efSearch: process.env.HNSW_EF_SEARCH
+        ? parseInt(process.env.HNSW_EF_SEARCH, 10)
+        : 50
+    }
+  }
+
+  // Create and initialize the Brainy instance
+  const brainy = new BrainyData(config)
+  await brainy.init()
+
+  return brainy
+}
+
+/**
+ * Get the current storage type being used by Brainy
+ */
+export async function getStorageType(brainy: BrainyData): Promise<string> {
+  const status = await brainy.status()
+  return status.type || 'unknown'
+}

cloud-wrapper/src/websocket.ts

@@ -0,0 +1,433 @@
+import { WebSocketServer, WebSocket } from 'ws';
+import { BrainyData } from '@soulcraft/brainy';
+import { v4 as uuidv4 } from 'uuid';
+
+// Define message types
+enum MessageType {
+  STATUS = 'status',
+  ADD_NOUN = 'addNoun',
+  GET_NOUN = 'getNoun',
+  UPDATE_NOUN = 'updateNoun',
+  DELETE_NOUN = 'deleteNoun',
+  SEARCH = 'search',
+  ADD_VERB = 'addVerb',
+  GET_VERBS = 'getVerbs',
+  GET_VERBS_BY_SOURCE = 'getVerbsBySource',
+  GET_VERBS_BY_TARGET = 'getVerbsByTarget',
+  DELETE_VERB = 'deleteVerb',
+  CLEAR = 'clear',
+  SUBSCRIBE = 'subscribe',
+  UNSUBSCRIBE = 'unsubscribe',
+  ERROR = 'error'
+}
+
+// Define subscription types
+enum SubscriptionType {
+  NOUNS = 'nouns',
+  VERBS = 'verbs',
+  SEARCH_RESULTS = 'searchResults'
+}
+
+// Define message interface
+interface WebSocketMessage {
+  type: MessageType;
+  id?: string;
+  payload?: any;
+}
+
+// Define client interface
+interface WebSocketClient {
+  id: string;
+  socket: WebSocket;
+  subscriptions: Set<SubscriptionType>;
+}
+
+// Store connected clients
+const clients: Map<string, WebSocketClient> = new Map();
+
+// Setup WebSocket handlers
+export function setupWebSocketHandlers(wss: WebSocketServer, brainy: BrainyData) {
+  // Handle new connections
+  wss.on('connection', (socket: WebSocket) => {
+    const clientId = uuidv4();
+    
+    // Create client object
+    const client: WebSocketClient = {
+      id: clientId,
+      socket,
+      subscriptions: new Set()
+    };
+    
+    // Store client
+    clients.set(clientId, client);
+    
+    console.log(`WebSocket client connected: ${clientId}`);
+    
+    // Send welcome message
+    sendMessage(socket, {
+      type: MessageType.STATUS,
+      id: uuidv4(),
+      payload: {
+        clientId,
+        message: 'Connected to Brainy WebSocket API',
+        status: 'connected'
+      }
+    });
+    
+    // Handle messages
+    socket.on('message', async (data: WebSocket.Data) => {
+      try {
+        const message: WebSocketMessage = JSON.parse(data.toString());
+        
+        // Ensure message has an ID
+        const messageId = message.id || uuidv4();
+        
+        console.log(`Received message: ${message.type} (${messageId})`);
+        
+        // Process message based on type
+        await processMessage(client, message, messageId, brainy);
+      } catch (error) {
+        console.error('Error processing WebSocket message:', error);
+        sendMessage(socket, {
+          type: MessageType.ERROR,
+          id: uuidv4(),
+          payload: {
+            message: 'Invalid message format',
+            error: (error as Error).message
+          }
+        });
+      }
+    });
+    
+    // Handle disconnection
+    socket.on('close', () => {
+      // Remove client
+      clients.delete(clientId);
+      console.log(`WebSocket client disconnected: ${clientId}`);
+    });
+  });
+}
+
+// Process incoming messages
+async function processMessage(
+  client: WebSocketClient,
+  message: WebSocketMessage,
+  messageId: string,
+  brainy: BrainyData
+) {
+  const { socket } = client;
+  
+  try {
+    switch (message.type) {
+      case MessageType.STATUS:
+        // Return database status
+        const status = await brainy.status();
+        sendMessage(socket, {
+          type: MessageType.STATUS,
+          id: messageId,
+          payload: status
+        });
+        break;
+        
+      case MessageType.ADD_NOUN:
+        // Add a noun
+        if (!message.payload?.text) {
+          throw new Error('Text is required');
+        }
+        
+        const nounId = await brainy.add(
+          message.payload.text,
+          message.payload.metadata || {}
+        );
+        
+        sendMessage(socket, {
+          type: MessageType.ADD_NOUN,
+          id: messageId,
+          payload: { id: nounId }
+        });
+        
+        // Notify subscribers
+        notifySubscribers(SubscriptionType.NOUNS, {
+          type: 'added',
+          id: nounId,
+          data: await brainy.get(nounId)
+        });
+        
+        break;
+        
+      case MessageType.GET_NOUN:
+        // Get a noun by ID
+        if (!message.payload?.id) {
+          throw new Error('Noun ID is required');
+        }
+        
+        const noun = await brainy.get(message.payload.id);
+        
+        if (!noun) {
+          throw new Error('Noun not found');
+        }
+        
+        sendMessage(socket, {
+          type: MessageType.GET_NOUN,
+          id: messageId,
+          payload: noun
+        });
+        break;
+        
+      case MessageType.UPDATE_NOUN:
+        // Update noun metadata
+        if (!message.payload?.id || !message.payload?.metadata) {
+          throw new Error('Noun ID and metadata are required');
+        }
+        
+        await brainy.updateMetadata(message.payload.id, message.payload.metadata);
+        
+        sendMessage(socket, {
+          type: MessageType.UPDATE_NOUN,
+          id: messageId,
+          payload: { success: true }
+        });
+        
+        // Notify subscribers
+        notifySubscribers(SubscriptionType.NOUNS, {
+          type: 'updated',
+          id: message.payload.id,
+          data: await brainy.get(message.payload.id)
+        });
+        
+        break;
+        
+      case MessageType.DELETE_NOUN:
+        // Delete a noun
+        if (!message.payload?.id) {
+          throw new Error('Noun ID is required');
+        }
+        
+        await brainy.delete(message.payload.id);
+        
+        sendMessage(socket, {
+          type: MessageType.DELETE_NOUN,
+          id: messageId,
+          payload: { success: true }
+        });
+        
+        // Notify subscribers
+        notifySubscribers(SubscriptionType.NOUNS, {
+          type: 'deleted',
+          id: message.payload.id
+        });
+        
+        break;
+        
+      case MessageType.SEARCH:
+        // Search for similar nouns
+        if (!message.payload?.query) {
+          throw new Error('Query is required');
+        }
+        
+        const limit = message.payload.limit || 10;
+        const results = await brainy.searchText(message.payload.query, limit);
+        
+        sendMessage(socket, {
+          type: MessageType.SEARCH,
+          id: messageId,
+          payload: results
+        });
+        
+        // Notify subscribers
+        notifySubscribers(SubscriptionType.SEARCH_RESULTS, {
+          type: 'search',
+          query: message.payload.query,
+          results
+        });
+        
+        break;
+        
+      case MessageType.ADD_VERB:
+        // Add a verb (relationship)
+        if (!message.payload?.sourceId || !message.payload?.targetId) {
+          throw new Error('Source ID and Target ID are required');
+        }
+        
+        await brainy.addVerb(
+          message.payload.sourceId,
+          message.payload.targetId,
+          message.payload.metadata || {}
+        );
+        
+        sendMessage(socket, {
+          type: MessageType.ADD_VERB,
+          id: messageId,
+          payload: { success: true }
+        });
+        
+        // Notify subscribers
+        notifySubscribers(SubscriptionType.VERBS, {
+          type: 'added',
+          sourceId: message.payload.sourceId,
+          targetId: message.payload.targetId,
+          metadata: message.payload.metadata
+        });
+        
+        break;
+        
+      case MessageType.GET_VERBS:
+        // Get all verbs
+        const verbs = await brainy.getAllVerbs();
+        
+        sendMessage(socket, {
+          type: MessageType.GET_VERBS,
+          id: messageId,
+          payload: verbs
+        });
+        break;
+        
+      case MessageType.GET_VERBS_BY_SOURCE:
+        // Get verbs by source
+        if (!message.payload?.id) {
+          throw new Error('Source ID is required');
+        }
+        
+        const sourceVerbs = await brainy.getVerbsBySource(message.payload.id);
+        
+        sendMessage(socket, {
+          type: MessageType.GET_VERBS_BY_SOURCE,
+          id: messageId,
+          payload: sourceVerbs
+        });
+        break;
+        
+      case MessageType.GET_VERBS_BY_TARGET:
+        // Get verbs by target
+        if (!message.payload?.id) {
+          throw new Error('Target ID is required');
+        }
+        
+        const targetVerbs = await brainy.getVerbsByTarget(message.payload.id);
+        
+        sendMessage(socket, {
+          type: MessageType.GET_VERBS_BY_TARGET,
+          id: messageId,
+          payload: targetVerbs
+        });
+        break;
+        
+      case MessageType.DELETE_VERB:
+        // Delete a verb
+        if (!message.payload?.id) {
+          throw new Error('Verb ID is required');
+        }
+        
+        await brainy.deleteVerb(message.payload.id);
+        
+        sendMessage(socket, {
+          type: MessageType.DELETE_VERB,
+          id: messageId,
+          payload: { success: true }
+        });
+        
+        // Notify subscribers
+        notifySubscribers(SubscriptionType.VERBS, {
+          type: 'deleted',
+          id: message.payload.id
+        });
+        
+        break;
+        
+      case MessageType.CLEAR:
+        // Clear all data
+        await brainy.clear();
+        
+        sendMessage(socket, {
+          type: MessageType.CLEAR,
+          id: messageId,
+          payload: { success: true }
+        });
+        
+        // Notify all subscribers
+        notifySubscribers(SubscriptionType.NOUNS, { type: 'cleared' });
+        notifySubscribers(SubscriptionType.VERBS, { type: 'cleared' });
+        
+        break;
+        
+      case MessageType.SUBSCRIBE:
+        // Subscribe to events
+        if (!message.payload?.type) {
+          throw new Error('Subscription type is required');
+        }
+        
+        const subscriptionType = message.payload.type as SubscriptionType;
+        
+        // Add subscription
+        client.subscriptions.add(subscriptionType);
+        
+        sendMessage(socket, {
+          type: MessageType.SUBSCRIBE,
+          id: messageId,
+          payload: {
+            success: true,
+            type: subscriptionType
+          }
+        });
+        break;
+        
+      case MessageType.UNSUBSCRIBE:
+        // Unsubscribe from events
+        if (!message.payload?.type) {
+          throw new Error('Subscription type is required');
+        }
+        
+        const unsubscribeType = message.payload.type as SubscriptionType;
+        
+        // Remove subscription
+        client.subscriptions.delete(unsubscribeType);
+        
+        sendMessage(socket, {
+          type: MessageType.UNSUBSCRIBE,
+          id: messageId,
+          payload: {
+            success: true,
+            type: unsubscribeType
+          }
+        });
+        break;
+        
+      default:
+        throw new Error(`Unknown message type: ${message.type}`);
+    }
+  } catch (error) {
+    console.error(`Error processing message ${message.type}:`, error);
+    
+    sendMessage(socket, {
+      type: MessageType.ERROR,
+      id: messageId,
+      payload: {
+        originalType: message.type,
+        message: 'Error processing message',
+        error: (error as Error).message
+      }
+    });
+  }
+}
+
+// Send a message to a client
+function sendMessage(socket: WebSocket, message: WebSocketMessage) {
+  if (socket.readyState === WebSocket.OPEN) {
+    socket.send(JSON.stringify(message));
+  }
+}
+
+// Notify subscribers of events
+function notifySubscribers(type: SubscriptionType, data: any) {
+  for (const client of clients.values()) {
+    if (client.subscriptions.has(type)) {
+      sendMessage(client.socket, {
+        type: MessageType.SUBSCRIBE,
+        payload: {
+          type,
+          data
+        }
+      });
+    }
+  }
+}

cloud-wrapper/tsconfig.json

@@ -0,0 +1,17 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "NodeNext",
+    "moduleResolution": "NodeNext",
+    "esModuleInterop": true,
+    "outDir": "./dist",
+    "rootDir": "./src",
+    "strict": true,
+    "declaration": true,
+    "skipLibCheck": true,
+    "forceConsistentCasingInFileNames": true,
+    "resolveJsonModule": true
+  },
+  "include": ["src/**/*"],
+  "exclude": ["node_modules", "dist"]
+}

docker-compose.yml

@@ -1,62 +0,0 @@
-version: '3.8'
-
-services:
-  brainy:
-    build: .
-    container_name: brainy-app
-    ports:
-      - "3000:3000"
-    environment:
-      - NODE_ENV=production
-      - BRAINY_STORAGE_TYPE=filesystem
-      - BRAINY_STORAGE_PATH=/app/data
-      - BRAINY_LOG_LEVEL=info
-      - BRAINY_RATE_LIMIT_MAX=100
-      - BRAINY_RATE_LIMIT_WINDOW_MS=900000
-    volumes:
-      # Persistent storage for data
-      - brainy-data:/app/data
-      # Optional: Mount local models directory
-      # - ./models:/app/models:ro
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
-      interval: 30s
-      timeout: 3s
-      retries: 3
-      start_period: 10s
-    restart: unless-stopped
-    networks:
-      - brainy-network
-
-  # Optional: MinIO for S3-compatible storage (development)
-  minio:
-    image: minio/minio:latest
-    container_name: brainy-minio
-    ports:
-      - "9000:9000"
-      - "9001:9001"
-    environment:
-      - MINIO_ROOT_USER=brainy
-      - MINIO_ROOT_PASSWORD=brainy123456
-    volumes:
-      - minio-data:/data
-    command: server /data --console-address ":9001"
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
-      interval: 30s
-      timeout: 20s
-      retries: 3
-    networks:
-      - brainy-network
-    profiles:
-      - with-s3
-
-volumes:
-  brainy-data:
-    driver: local
-  minio-data:
-    driver: local
-
-networks:
-  brainy-network:
-    driver: bridge

docs/ADR-001-generational-mvcc.md

@@ -1,295 +0,0 @@
-# ADR-001: Generational MVCC storage and the immutable Db API
-
-**Status:** Accepted (ships in 8.0)
-**Date:** 2026-06-10
-
-## Context
-
-Before 8.0, Brainy carried two overlapping version-control subsystems: a
-copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a
-separate versioning subsystem (`versions.save/list/compare/restore/prune`),
-plus a read-only historical adapter for commit-based time travel. Together
-they were ~5,100 LOC of mechanism for one product need: *read a consistent
-past state while the store keeps moving, and snapshot/restore cheaply.*
-
-Neither subsystem gave a precise isolation guarantee. Reads raced in-place
-JSON overwrites, so a "snapshot" was only as immutable as the bytes it
-happened to share with the live store.
-
-8.0 replaces both with **one mechanism**: generational MVCC over immutable,
-generation-stamped records, exposed through a Datomic-style immutable
-database value (`Db`). The same model is implemented natively by versioned
-index providers (LSM snapshots), so semantics are identical on the pure-JS
-path and the native path.
-
-## Decision
-
-### The model
-
-- A **monotonic u64 generation counter** is the store's logical clock. It
-  advances once per committed `transact()` batch and once per
-  single-operation write (`add`/`update`/`remove`/`relate`/…), so
-  `brain.generation()` is always a meaningful watermark. It is persisted in
-  `_system/generation.json` and never reissued for anything durable.
-- `brain.now()` **pins** the current generation in O(1) and returns a `Db` —
-  an immutable view. Pins are refcounted; `db.release()` (with a
-  `FinalizationRegistry` backstop for leaked values) ends the pin.
-- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative
-  batch atomically as **exactly one generation**, with whole-store
-  compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and
-  reified transaction metadata appended to `_system/tx-log.jsonl`.
-- `brain.asOf(generation | Date | snapshotPath)` opens past state;
-  `db.with(ops)` layers a speculative in-memory overlay (never touching
-  disk, the counter, or index providers); `db.persist(path)` cuts an
-  instant snapshot; `brain.restore(path, { confirm: true })` replaces state
-  from one; `Brainy.load(path)` opens a snapshot read-only with the full
-  query surface.
-
-### Persisted layout
-
-All paths are storage-root-relative:
-
-```
-_system/generation.json          { generation, updatedAt }       atomic tmp+rename
-_system/manifest.json            { version, generation,          atomic tmp+rename
-                                   committedAt, horizon }        (the commit point)
-_system/tx-log.jsonl             one line per committed          append-only
-                                 transact: { generation,
-                                 timestamp, meta? }
-_generations/<N>/tx.json         the generation-N delta:         immutable
-                                 touched noun/verb ids + meta
-_generations/<N>/prev/<id>.json  before-image of <id> as of      immutable
-                                 commit N (raw stored bytes;
-                                 null parts = file was absent)
-```
-
-**Why per-generation deltas instead of a global `id → latest generation`
-map in the manifest:** a global map makes every commit O(all ids) — the
-whole map must be rewritten to swap it atomically. The delta layout makes a
-commit O(ids touched) and keeps the manifest a fixed-size watermark, while
-point-in-time resolution stays correct (see "Read resolution" below). The
-trade is that resolution at a pinned generation scans the deltas of later
-commits — bounded by the number of commits since the pin, which is exactly
-the window compaction keeps short.
-
-Before-images are deliberately the *only* per-id records. They serve both
-roles the layer needs — the crash-recovery undo log and the point-in-time
-read source. After-images would duplicate state that is already readable
-(the canonical entity files hold the latest bytes; earlier states resolve
-from later before-images) and would double record I/O per commit.
-
-### Commit protocol (durability)
-
-`transact()` commits under a store-wide mutex:
-
-1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError`
-   before anything is staged.
-2. **Reserve** generation `N` (counter increment).
-3. **Stage the undo log:** write the before-image of every touched id plus
-   `tx.json` under `_generations/N/`, then **fsync** the files and their
-   directories. From this point, any crash is recoverable to the exact
-   pre-transaction bytes.
-4. **Execute** the planned batch through the TransactionManager (which has
-   its own operation-level rollback for in-flight failures).
-5. **Commit point:** persist the counter, then write `_system/manifest.json`
-   via atomic tmp+rename and fsync it. The rename *is* the commit: a
-   generation directory is committed if and only if `N ≤
-   manifest.generation`.
-6. Append the tx-log line (advisory metadata — a crash between 5 and 6
-   keeps the transaction).
-
-**Crash recovery (on open):** any `_generations/<N>` directory with
-`N > manifest.generation` is an uncommitted transaction. Its before-images
-are restored to the canonical paths (idempotently — recovery itself can
-crash and rerun) and the directory is removed. Because recovery runs before
-any index is built, and a recovery that rolled something back forces a full
-index rebuild, derived indexes never observe rolled-back state. Reader-mode
-instances skip recovery (readers never write; the next writer repairs).
-
-A failed (non-crash) transaction takes the same staging directory down the
-abort path: the TransactionManager rolls back applied operations, the
-staging directory is removed, and the generation reservation is returned —
-a failed batch leaves the generation counter unchanged.
-
-### Read resolution at a pinned generation
-
-The state of id X at pinned generation G is:
-
-- the before-image stored by the **first committed generation after G that
-  touched X**, or
-- the live canonical bytes, when nothing after G touched X.
-
-While nothing has committed past G, *every* read on the `Db` delegates to
-the live fast paths untouched — `now()` adds no read overhead until history
-actually moves.
-
-**Two read paths, one result set.** `get()`, metadata-level `find()`, and
-filter-based `related()` resolve directly through the record layer at any
-reachable pinned generation — no extra cost beyond scanning the deltas of
-later commits. Index-accelerated dimensions (semantic/vector search, graph
-traversal, cursors, aggregation) are served by **at-generation index
-materialization**: the first such query on a historical `Db` copies the
-exact at-G record set (live bytes for ids untouched since the pin,
-before-images for the rest; a final reconciliation pass runs under the
-commit mutex so transactions racing the copy cannot skew it) into an
-ephemeral in-memory store and opens a read-only engine over it — the same
-vector/metadata/graph index classes the live brain uses, sharing the host's
-embedder and aggregate definitions. The handle is cached on the `Db` and
-freed by `release()`.
-
-**Cost, stated plainly:** materialization is O(n at G) time and memory,
-once per `Db`. That is the open-core price of historical index queries. A
-native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
-retained LSM segments) serves the same reads with no rebuild at all — the
-materializer is the correctness baseline, the provider is the accelerator.
-
-**The one remaining boundary.** Speculative `with()` overlays throw
-`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
-overlay entities carry no embeddings (`with()` never invokes the embedder),
-so a "full" index query over an overlay would silently exclude the
-overlay's own entities. Commit with `transact()` to get the full surface.
-
-**History granularity (Model-B).** EVERY write is its own immutable generation
-— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
-Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/
-`diff()`/`history()` exactly like transacts; a pin always freezes against later
-writes. `transact()` groups several operations into ONE atomic generation.
-
-Single-op history durability is **async group-commit**: the live write hits
-canonical storage immediately (acknowledged), while its before-image is buffered
-and persisted to disk in one batched fsync on a size/timer trigger (or forced by
-`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in
-point-in-time resolution exactly like on-disk generations, so the synchronous
-`now()` freezes with no forced flush. A hard crash before the flush loses only
-the buffered *history* of the last window — never live data — and a crash
-*mid-flush* is recovered by **drop-without-restore** (the partial generation's
-before-images are discarded, never replayed, because the live write was already
-acknowledged; restoring them would silently revert it).
-
-### Pinning, retention, compaction
-
-- Each live `Db` holds one refcounted pin on its generation (plus a
-  `pin(generation)` on every registered `VersionedIndexProvider`, whose
-  explicit pin lifetime overrides any time-based snapshot retention the
-  provider has).
-- The constructor **`retention`** knob governs auto-compaction (on `flush()`/
-  `close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config;
-  driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) ·
-  `'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit
-  CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
-  manually on the same caps — the oldest unpinned record-sets are reclaimed
-  while ANY supplied cap is exceeded.
-- A record-set `N` is reclaimed only when `N` is at or below **every** live
-  pin — deleting `N` can only break readers pinned *below* `N`, because
-  resolution reads before-images from generations strictly greater than the
-  pin. Live pins are ALWAYS exempt, in every retention mode.
-- The manifest records the **horizon** (highest reclaimed generation).
-  Generations below the horizon are unreachable; `asOf()` on them throws
-  `GenerationCompactedError`. The horizon itself stays reachable, resolved
-  from the record-sets above it. To keep a generation readable forever,
-  `persist()` it first — snapshots are self-contained.
-
-### Snapshots and restore
-
-`db.persist(path)` flushes indexes, then cuts the snapshot under the
-store's commit mutex (no commit, compaction, or counter write can
-interleave). On filesystem storage it is a **hard-link farm**: every data
-file is immutable-by-rename, so linking is safe — later rewrites swap
-inodes and the snapshot keeps the old bytes. The two exceptions are handled
-explicitly: the append-in-place tx-log is byte-copied, and process-local
-lock state is excluded. Cross-device targets (and filesystems that refuse
-links) fall back to per-file byte copies. In-memory stores serialize to the
-same directory layout, so persisting a memory brain produces a real,
-durable, loadable store.
-
-`persist()` requires the view to still be the store's latest generation
-(a snapshot captures current bytes); a view that history has moved past
-throws rather than persisting the wrong state.
-
-`restore(path, { confirm: true })` replaces the store's contents from a
-snapshot via byte copy (never links — the snapshot stays independent),
-reloads all adapter-internal derived state, rebuilds all indexes, and
-floors the generation counter at its pre-restore value so observed
-generation numbers are never reissued. Live pins do not survive a restore;
-a warning is logged when any exist.
-
-### Versioned index providers
-
-Native index providers may implement the optional 4-method
-`VersionedIndexProvider` capability (`generation()`,
-`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the
-boundary). The locked consistency model: providers are **post-commit
-appliers**. The storage-record commit is the source of truth; provider
-index state is derived. On open, a provider behind the committed watermark
-replays the gap from storage (or requests a rebuild) — there are no
-provider rollback hooks, because uncommitted transactions are repaired at
-the storage layer before any index opens. Speculative `with()` overlays
-never reach providers.
-
-## Guarantees (and their proofs)
-
-Each stated guarantee has a test that proves it, not merely exercises it
-(`tests/integration/db-mvcc.test.ts`, plus
-`tests/unit/db/generationStore.test.ts` for the record layer in isolation):
-
-| Guarantee | Proof |
-|---|---|
-| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) |
-| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) |
-| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) |
-| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c |
-| Compaction never breaks a pinned read; release enables reclaim | proof 5 |
-| `with()` overlays touch nothing durable | proof 6 |
-| Generation monotonicity across close/reopen | proof 7 |
-| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) |
-| Balanced provider pin/release lockstep | proof 9 |
-
-One deliberate softness: single-operation generation bumps persist the
-counter coalesced (per write burst), not per write. Durable artifacts —
-records, manifests, snapshots — always persist the counter synchronously at
-their own commit points, so a crash inside the coalescing window can lose
-only counter values that nothing durable ever referenced.
-
-## Failure modes
-
-| Failure | Outcome |
-|---|---|
-| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. |
-| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. |
-| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. |
-| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
-| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
-| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
-| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
-| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
-| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
-
-## Lineage
-
-The design is an assembly of well-understood prior art, chosen for being
-boring where it counts:
-
-- **Datomic** — the database-as-a-value: an immutable `Db` you query, with
-  `with()` for speculation and reified transaction metadata instead of
-  commit messages.
-- **LMDB** — reader pins: readers never block writers; a reader's view
-  stays valid because nothing overwrites the pages (here: records) it
-  references; reclamation waits for the last reader.
-- **LSM trees / Cassandra** — immutable segments make snapshots hard links
-  and make compaction a retention policy instead of a locking problem.
-
-## Consequences
-
-- One mechanism replaces the COW and versioning subsystems (their removal
-  is the companion change to this ADR).
-- In-place branch switching (`checkout`) is gone by design; the replacement
-  is opening a persisted snapshot as a separate instance — a name→path
-  mapping where a product needs named branches.
-- Every commit pays O(ids touched) extra writes (before-images + delta +
-  manifest). Single-operation writes pay only an in-memory counter bump
-  with coalesced persistence.
-- The full query surface works at every reachable pinned generation.
-  Record-path reads (`get`, metadata `find`, filter `related`) are
-  effectively free; index-accelerated historical queries pay a one-time
-  O(n at G) materialization per `Db` on the open-core path (freed on
-  `release()`), and run rebuild-free on a native `VersionedIndexProvider`.

docs/BATCHING.md

@@ -1,468 +0,0 @@
----
-title: Batch Operations
-slug: guides/batching
-public: true
-category: guides
-template: guide
-order: 5
-description: Eliminate N+1 query patterns with batchGet() and storage-level batch APIs for fast multi-entity reads against filesystem and memory storage.
-next:
-  - api/reference
-  - guides/find-system
----
-
-# Batch Operations API
-> **Production-Ready** | Zero N+1 Query Patterns
-
-## Overview
-
-Brainy provides batch operations at the storage layer to eliminate N+1 query patterns for VFS operations, relationship queries, and entity retrieval.
-
-### Problem Solved
-
-The naive pattern of looping and calling `brain.get(id)` once per item issues sequential reads through the storage layer. Batched APIs collapse that into a single read pass.
-
-**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`** at the VFS layer and the explicit `batchGet()` / `getNounMetadataBatch()` calls — not to `readFile()` or individual `get()` operations.
-
----
-
-## New Public APIs
-
-### 1. `brain.batchGet(ids, options?)`
-
-Batch retrieval of multiple entities (metadata-only by default).
-
-```typescript
-// Fetch multiple entities in a single batched operation
-const ids = ['id1', 'id2', 'id3']
-const results: Map<string, Entity> = await brain.batchGet(ids)
-
-// With vectors (falls back to individual gets)
-const resultsWithVectors = await brain.batchGet(ids, { includeVectors: true })
-
-// Results map
-results.get('id1') // → Entity or undefined
-results.size // → 3 (number of found entities)
-```
-
-**Performance:**
-- Memory storage: Instant (parallel reads)
-- Filesystem storage: Parallel reads, scales with available IOPS
-
-**Use Cases:**
-- Loading multiple entities for display
-- Bulk data export operations
-- Relationship traversal (fetch all connected entities)
-
----
-
-## Storage-Level APIs
-
-### 2. `storage.getNounMetadataBatch(ids)`
-
-Batch metadata retrieval with direct O(1) path construction.
-
-```typescript
-const storage = brain.storage as BaseStorage
-const ids = ['id1', 'id2', 'id3']
-
-const metadataMap: Map<string, NounMetadata> = await storage.getNounMetadataBatch(ids)
-
-for (const [id, metadata] of metadataMap) {
- console.log(metadata.noun) // Type: 'document', 'person', etc.
- console.log(metadata.data) // Entity data
-}
-```
-
-**Features:**
-- ✅ Direct O(1) path construction from ID (no type lookup needed!)
-- ✅ Sharding preservation (all paths include `{shard}/{id}`)
-- ✅ Write-cache coherent (read-after-write consistency)
-- ✅ O(1) path construction — eliminates the per-entity type search of the old type-first layout
-
-**Performance:**
-- Constant-time path construction per ID — no type-cache misses
-- Filesystem: parallel reads bounded by IOPS
-- No type search delays — every ID maps directly to storage path
-
----
-
-### 3. `storage.getVerbsBySourceBatch(sourceIds, verbType?)`
-
-Batch relationship queries by source entity IDs.
-
-```typescript
-const storage = brain.storage as BaseStorage
-
-// Get all relationships from multiple sources
-const results: Map<string, GraphVerb[]> = await storage.getVerbsBySourceBatch([
- 'person1',
- 'person2'
-])
-
-// Filter by verb type
-const createsResults = await storage.getVerbsBySourceBatch(
- ['person1', 'person2'],
- 'creates'
-)
-
-// Process results
-for (const [sourceId, verbs] of results) {
- console.log(`${sourceId} has ${verbs.length} relationships`)
- verbs.forEach(verb => {
- console.log(` → ${verb.verb} → ${verb.targetId}`)
- })
-}
-```
-
-**Use Cases:**
-- Social graph traversal (fetch all connections for multiple users)
-- Knowledge graph queries (find all relationships of specific type)
-- Bulk export of relationship data
-
-**Performance:**
-- Memory storage: single in-memory pass over the metadata index
-- Filesystem storage: parallel reads through the metadata index
-
----
-
-## VFS Integration
-
-VFS operations automatically use batch APIs for maximum performance.
-
-### Directory Traversal
-
-```typescript
-// Tree traversal uses batched reads under the hood
-const tree = await brain.vfs.getTreeStructure('/my-dir')
-// ✅ PathResolver.getChildren() uses brain.batchGet() internally
-// ✅ Parallel traversal of directories at the same tree level
-// ✅ 2-3 batched calls instead of 22 sequential calls
-```
-
-**Architecture:**
-
-```
-VFS.getTreeStructure()
- ↓ PARALLEL (breadth-first traversal)
- → PathResolver.getChildren() [all dirs at level processed in parallel]
- ↓ BATCHED
- → brain.batchGet(childIds) [1 call instead of N]
- ↓ BATCHED
- → storage.getNounMetadataBatch(ids) [1 call instead of N]
- ↓ ADAPTER
- → Filesystem: Promise.all() parallel reads
- → Memory: Promise.all() parallel reads
-```
-
----
-
-## Advanced Features Compatibility
-
-### ✅ ID-First Storage Architecture
-
-All batch operations use direct ID-first paths - no type lookup needed!
-
-**ID-First Path Structure:**
-```
-entities/nouns/{SHARD}/{ID}/metadata.json
-entities/verbs/{SHARD}/{ID}/metadata.json
-```
-
-**Direct O(1) Path Construction:**
-```typescript
-// Every ID maps directly to exactly ONE path - O(1), no type search
-const id = 'abc-123'
-const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
-const path = `entities/nouns/${shard}/${id}/metadata.json`
-
-// No type cache needed!
-// No type search needed!
-// No multi-type fallback needed!
-// Just pure O(1) lookup!
-```
-
-**Benefits:**
-- **O(1)** path lookups (eliminates the 42-type sequential search the old type-first layout required)
-- **Simpler code** - removed 500+ lines of type cache complexity
-- **Scalable** - works at large scale without type tracking overhead
-
----
-
-### ✅ Sharding
-
-All batch paths include shard IDs calculated via `getShardIdFromUuid(id)`:
-
-```typescript
-const id = 'a3c4e5f7-...'
-const shard = getShardIdFromUuid(id) // → 'a3' (first 2 hex chars)
-const path = `entities/nouns/${shard}/${id}/metadata.json`
-```
-
-**Distribution:** 256 shards (00-ff) for optimal load distribution.
-
----
-
-### ✅ Generational MVCC (8.0)
-
-Batch reads always serve the **live** generation through the fast paths
-shown above. Point-in-time reads go through the Db API instead: a pinned
-`Db` (`brain.now()`, `brain.asOf()`) resolves changed ids from immutable
-generation records and unchanged ids from the same live paths batch reads
-use — see the [consistency model](concepts/consistency-model.md).
-
-```typescript
-const db = brain.now()                       // pinned view
-const entity = await db.get(id)              // correct at the pinned generation
-const results = await brain.batchGet(ids)    // live state, batched
-await db.release()
-```
-
----
-
-## Why Batching Is Faster
-
-Batching's advantage is structural, not a fixed multiplier (the actual speedup
-depends on storage backend, IOPS, and batch size):
-
-- **N+1 elimination** — N sequential reads collapse into a single parallel pass
-  (`Promise.all` over the batch).
-- **O(1) path construction** — every ID maps directly to one storage path, with
-  no per-type cache lookup.
-- **One metadata round-trip** — relationship batches fetch all sources' metadata
-  in a single pass instead of one query per source.
-
-The integration test `tests/integration/storage-batch-operations.test.ts`
-exercises batch vs. individual reads and asserts that batch retrieval is not
-slower than the per-entity loop for large batches; it does not pin a specific
-multiplier, since that is hardware- and IOPS-dependent.
-
----
-
-## Error Handling
-
-### Partial Batch Failures
-
-Batch operations gracefully handle missing or invalid entities:
-
-```typescript
-const validId = 'abc-123-...'
-const invalidIds = [
- '11111111-1111-1111-1111-111111111111',
- '22222222-2222-2222-2222-222222222222'
-]
-
-const results = await brain.batchGet([validId, ...invalidIds])
-
-results.size // → 1 (only valid entity)
-results.has(validId) // → true
-results.has(invalidIds[0]) // → false (silently skipped)
-```
-
-**Behavior:**
-- Invalid UUIDs: Silently skipped (not included in results)
-- Missing entities: Silently skipped (not included in results)
-- Storage errors: Logged, entity excluded from results
-- No exceptions thrown for partial failures
-
-### Empty Batches
-
-```typescript
-const results = await brain.batchGet([])
-results.size // → 0 (empty map)
-```
-
-### Duplicate IDs
-
-```typescript
-const results = await brain.batchGet(['id1', 'id1', 'id1'])
-results.size // → 1 (deduplicated automatically)
-```
-
----
-
-## Migration Guide
-
-### From Individual Gets
-
-**Before:**
-```typescript
-const entities = []
-for (const id of ids) {
- const entity = await brain.get(id)
- if (entity) entities.push(entity)
-}
-```
-
-**After:**
-```typescript
-const results = await brain.batchGet(ids)
-const entities = Array.from(results.values())
-```
-
-**Performance Gain:** Replaces N sequential reads with a single batched pass — no fixed multiplier, it scales with storage IOPS.
-
----
-
-### From Individual Relationship Queries
-
-**Before:**
-```typescript
-const allVerbs = []
-for (const sourceId of sourceIds) {
- const verbs = await brain.related({ from: sourceId })
- allVerbs.push(...verbs)
-}
-```
-
-**After:**
-```typescript
-const storage = brain.storage as BaseStorage
-const results = await storage.getVerbsBySourceBatch(sourceIds)
-
-const allVerbs = []
-for (const verbs of results.values()) {
- allVerbs.push(...verbs)
-}
-```
-
-**Performance Gain:** One batched metadata fetch instead of one query per source entity.
-
----
-
-## Best Practices
-
-### 1. **Use Batching for Multiple Entity Operations**
-
-```typescript
-// ✅ GOOD: Batch fetch
-const results = await brain.batchGet(ids)
-
-// ❌ BAD: Individual gets in loop
-for (const id of ids) {
- await brain.get(id)
-}
-```
-
-### 2. **Batch Size Recommendations**
-
-| Storage | Optimal Batch Size | Max Batch Size |
-|---------|--------------------|----------------|
-| **Memory** | Unlimited | Unlimited |
-| **Filesystem** | 100-500 | 1000 |
-
-**Guideline:** For batches >1000, split into chunks of 500-1000.
-
-### 3. **Metadata-Only by Default**
-
-```typescript
-// Default: Metadata-only (fast)
-const results = await brain.batchGet(ids) // No vectors
-
-// Only load vectors if needed
-const withVectors = await brain.batchGet(ids, { includeVectors: true })
-```
-
-### 4. **Error Handling**
-
-```typescript
-// Batch operations never throw for missing entities
-const results = await brain.batchGet(ids)
-
-// Check results
-for (const id of ids) {
- if (results.has(id)) {
- // Entity exists
- const entity = results.get(id)
- } else {
- // Entity missing (not an error)
- console.log(`Entity ${id} not found`)
- }
-}
-```
-
----
-
-## Testing
-
-Comprehensive test coverage in `tests/integration/storage-batch-operations.test.ts`:
-
-```bash
-npx vitest run tests/integration/storage-batch-operations.test.ts
-```
-
-**Test Coverage:**
-- ✅ brain.batchGet() high-level API
-- ✅ storage.getNounMetadataBatch() with ID-first paths
-- ✅ COW integration (branch isolation, inheritance)
-- ✅ storage.getVerbsBySourceBatch() relationship queries
-- ✅ VFS integration (PathResolver.getChildren())
-- ✅ Performance benchmarks (N+1 elimination)
-- ✅ Error handling (partial failures, empty batches, duplicates)
-- ✅ ID-first storage verification
-- ✅ Sharding preservation
-
-**Results:** 23 tests passing ✅
-
----
-
-## Implementation Details
-
-### Architecture Layers
-
-```
-User Code (brain.batchGet)
- ↓
-High-Level API (src/brainy.ts)
- ↓
-Storage Layer (src/storage/baseStorage.ts)
- ↓
-Adapter Layer (readBatchFromAdapter)
- ↓
-Storage Adapter (FileSystemStorage / MemoryStorage)
-```
-
-### Parallel Reads
-
-Both shipped adapters fall back to `Promise.all` over individual reads:
-
-```typescript
-// BaseStorage.readBatchFromAdapter()
-return await Promise.all(resolvedPaths.map(path => this.read(path)))
-```
-
-**Shipped Adapters:**
-- MemoryStorage
-- FileSystemStorage
-
----
-
-## API Summary
-
-- `brain.batchGet(ids, options?)` - High-level batch entity retrieval
-- `storage.getNounMetadataBatch(ids)` - Storage-level metadata batch
-- `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries
-
-**Performance Improvements:**
-- VFS operations: single batched pass instead of N sequential reads
-- Entity retrieval: N+1 reads collapsed into one batched pass
-- Zero N+1 query patterns
-
-**Compatibility:**
-- ✅ ID-first storage
-- ✅ Sharding (256 shards)
-- ✅ Generational MVCC — batch reads serve the live generation; pinned `Db` views serve the past
-- ✅ All indexes respected (vector, metadata, graph adjacency)
-
----
-
-## Support
-
-- **Documentation:** `/docs/BATCHING.md`, `/docs/PERFORMANCE.md`
-- **Tests:** `/tests/integration/storage-batch-operations.test.ts`
-- **Issues:** https://github.com/soulcraft/brainy/issues
-- **Discussions:** https://github.com/soulcraft/brainy/discussions
-
----
-
-**Built with ❤️ for enterprise-scale knowledge graphs**

docs/DATA_MODEL.md

@@ -1,271 +0,0 @@
-# Data Model
-
-> How Brainy stores entities and relationships, and the critical distinction between `data` and `metadata`.
-
----
-
-## Entity (Noun)
-
-An entity is the fundamental data unit in Brainy. Every entity has:
-
-| Field | Type | Indexed | Description |
-|-------|------|---------|-------------|
-| `id` | `string` | Primary key | UUID v4 (auto-generated or custom) |
-| `data` | `any` | **HNSW vector index** | Content used for semantic/hybrid search. Strings auto-embed. |
-| `metadata` | `object` | **MetadataIndex** | Structured queryable fields (tags, dates, flags, etc.) |
-| `type` | `NounType` | MetadataIndex (as `noun`) | Entity type classification |
-| `vector` | `number[]` | HNSW | 384-dim embedding (auto-computed from `data` or user-provided) |
-| `confidence` | `number` | MetadataIndex | Type classification confidence (0-1) |
-| `weight` | `number` | MetadataIndex | Entity importance/salience (0-1) |
-| `service` | `string` | MetadataIndex | Multi-tenancy identifier |
-| `createdAt` | `number` | MetadataIndex | Creation timestamp (ms since epoch) |
-| `updatedAt` | `number` | MetadataIndex | Last update timestamp (ms since epoch) |
-| `createdBy` | `object` | MetadataIndex | Source augmentation info |
-
-### Example
-
-```typescript
-const id = await brain.add({
-  data: 'John Smith is a software engineer at Acme Corp',  // → embedded into vector
-  type: NounType.Person,
-  metadata: {                   // → indexed, queryable via where filters
-    role: 'engineer',
-    department: 'backend',
-    yearsExperience: 8
-  },
-  confidence: 0.95,
-  weight: 0.7
-})
-```
-
----
-
-## Relationship (Verb)
-
-A relationship is a typed, directed edge connecting two entities.
-
-| Field | Type | Indexed | Description |
-|-------|------|---------|-------------|
-| `id` | `string` | Primary key | UUID v4 (auto-generated) |
-| `from` | `string` | **GraphAdjacencyIndex** | Source entity ID |
-| `to` | `string` | **GraphAdjacencyIndex** | Target entity ID |
-| `type` | `VerbType` | GraphAdjacencyIndex (as `verb`) | Relationship type classification |
-| `data` | `any` | — | Opaque content (overrides auto-computed vector if provided) |
-| `metadata` | `object` | — | Structured fields on the edge |
-| `weight` | `number` | — | Connection strength (0-1, default: 1.0) |
-| `confidence` | `number` | — | Relationship certainty (0-1) |
-| `evidence` | `RelationEvidence` | — | Why this relationship was detected |
-| `createdAt` | `number` | — | Creation timestamp (ms since epoch) |
-| `updatedAt` | `number` | — | Last update timestamp (ms since epoch) |
-| `service` | `string` | — | Multi-tenancy identifier |
-
-### Example
-
-```typescript
-const relId = await brain.relate({
-  from: personId,
-  to: projectId,
-  type: VerbType.WorksOn,
-  data: 'Lead engineer on the AI module',   // Optional: content for this edge
-  metadata: {                                // Optional: queryable edge fields
-    role: 'lead',
-    startDate: '2024-01-15'
-  },
-  weight: 0.9
-})
-```
-
----
-
-## Data vs Metadata
-
-This is the most important concept in Brainy's storage model:
-
-### `data` — Content for Semantic Search
-
-- Embedded into a 384-dimensional vector via the WASM embedding engine
-- Searchable via **semantic similarity** (HNSW vector index) and **hybrid text+semantic** search
-- Queried by passing `query` to `find()`:
-  ```typescript
-  brain.find({ query: 'machine learning algorithms' })
-  ```
-- **NOT** indexed by MetadataIndex — you cannot use `where` filters on `data`
-- Stored opaquely: strings, objects, numbers — anything goes
-
-### `metadata` — Structured Queryable Fields
-
-- Indexed by MetadataIndex with O(1) lookups per field
-- Queryable via `where` filters using [BFO operators](./QUERY_OPERATORS.md):
-  ```typescript
-  brain.find({
-    where: {
-      department: 'engineering',
-      yearsExperience: { greaterThan: 5 },
-      tags: { contains: 'senior' }
-    }
-  })
-  ```
-- **NOT** used for vector/semantic search
-- Must be a flat or lightly nested object
-
-### Quick Reference
-
-| | `data` | `metadata` |
-|---|---|---|
-| **Purpose** | Content for embedding / semantic search | Structured fields for filtering |
-| **Searched by** | `find({ query })` — vector similarity, hybrid text+semantic | `find({ where })` — exact, range, set operators |
-| **Indexed by** | HNSW vector index | MetadataIndex |
-| **Queryable with operators?** | No | Yes (`equals`, `greaterThan`, `oneOf`, etc.) |
-| **Auto-embedded?** | Yes (strings → 384-dim vectors) | No |
-| **Typical content** | Text descriptions, document content | Tags, dates, status flags, categories, numeric fields |
-
-### Common Pattern
-
-```typescript
-// Add an article
-await brain.add({
-  data: 'A deep dive into transformer architectures and attention mechanisms',
-  type: NounType.Document,
-  metadata: {
-    title: 'Transformer Deep Dive',
-    author: 'Dr. Chen',
-    publishedYear: 2024,
-    tags: ['AI', 'transformers', 'NLP'],
-    status: 'published'
-  }
-})
-
-// Search by content (semantic — searches data)
-const results = await brain.find({ query: 'neural network attention' })
-
-// Filter by fields (exact — queries metadata)
-const recent = await brain.find({
-  where: {
-    publishedYear: { greaterThan: 2023 },
-    status: 'published'
-  }
-})
-
-// Combine both (Triple Intelligence)
-const precise = await brain.find({
-  query: 'attention mechanisms',             // Semantic search on data
-  where: { author: 'Dr. Chen' },            // Metadata filter
-  connected: { from: authorId, depth: 1 }   // Graph traversal
-})
-```
-
----
-
-## Storage Field Naming
-
-Internally, Brainy uses different field names in storage vs the public API:
-
-| Public API (Entity/Relation) | Storage (metadata object) | Notes |
-|------------------------------|--------------------------|-------|
-| `type` | `noun` | Entity type stored as `noun` |
-| `from` | `sourceId` | Relationship source |
-| `to` | `targetId` | Relationship target |
-| `type` (on Relation) | `verb` | Relationship type stored as `verb` |
-
-When querying with `find()`, you can use:
-- `type` parameter (convenience alias, equivalent to `where.noun`)
-- `where.noun` directly
-
-```typescript
-// These are equivalent:
-brain.find({ type: NounType.Person })
-brain.find({ where: { noun: NounType.Person } })
-```
-
----
-
-## Standard Metadata Fields
-
-When you add an entity, Brainy stores these standard fields in the metadata object alongside your custom fields:
-
-| Field | Set By | Description |
-|-------|--------|-------------|
-| `noun` | System | Entity type (NounType enum value) |
-| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. |
-| `data` | System | The raw `data` value (stored opaquely) |
-| `createdAt` | System | Creation timestamp |
-| `updatedAt` | System | Last update timestamp |
-| `confidence` | User | Type classification confidence |
-| `weight` | User | Entity importance |
-| `service` | User | Multi-tenancy identifier |
-| `createdBy` | User/System | Source augmentation |
-
-On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**.
-
-### Subtype — sub-classification within a NounType
-
-`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type:
-
-```typescript
-// A Person who is an employee:
-await brain.add({
-  data: 'Avery Brooks — runs the AI lab',
-  type: NounType.Person,
-  subtype: 'employee',
-  metadata: { department: 'ai-lab' }
-})
-
-// A Document that is an invoice:
-await brain.add({
-  data: 'INV-2026-001',
-  type: NounType.Document,
-  subtype: 'invoice',
-  metadata: { amount: 1500 }
-})
-```
-
-`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`.
-
-### Subtype — sub-classification within a VerbType (7.30+)
-
-Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive — a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field on `HNSWVerbWithMetadata` and on the public `Relation<T>`:
-
-```typescript
-await brain.relate({
-  from: ceoId,
-  to: vpId,
-  type: VerbType.ReportsTo,
-  subtype: 'direct',                  // top-level standard field
-  metadata: { since: '2025-Q1' }      // user-custom fields stay in metadata
-})
-```
-
-Fast-path filter on the verb side:
-
-```typescript
-const direct = await brain.related({
-  from: ceoId,
-  type: VerbType.ReportsTo,
-  subtype: 'direct'
-})
-```
-
-The verb-side rollup at `_system/verb-subtype-statistics.json` mirrors the noun-side `_system/subtype-statistics.json` — same shape, same self-heal machinery. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`.
-
-Verbs and nouns now have full capability parity — every API on the noun side has a verb-side mirror, including the new `brain.updateRelation()` (which closed a pre-7.30 gap where relationships had no update path).
-
-### Standard verb fields
-
-The verb-side equivalent of `STANDARD_ENTITY_FIELDS` is `STANDARD_VERB_FIELDS`, exported from `src/coreTypes.ts`. Verb-specific standard fields:
-
-| Field | Description |
-|---|---|
-| `verb` | The VerbType enum value |
-| `sourceId` / `targetId` | The two endpoints of the relationship |
-| `subtype` | Sub-classification within the VerbType (7.30+) |
-| `confidence`, `weight`, `createdAt`, `updatedAt`, `service`, `createdBy`, `data` | Same semantics as the noun-side standard fields |
-
-The companion `resolveVerbField(verb, field)` helper resolves field paths the same way `resolveEntityField` does for nouns: standard fields first, metadata fallback for everything else.
-
----
-
-## See Also
-
-- [API Reference](./api/README.md) — Complete API documentation
-- [Query Operators](./QUERY_OPERATORS.md) — All BFO operators with examples
-- [Find System](./FIND_SYSTEM.md) — Natural language find() details

docs/MIGRATION-V3-TO-V4.md

@@ -1,569 +0,0 @@
-# Brainy v3 → v4.0.0 Migration Guide
-
-> **Migration Complexity**: Low
-> **Breaking Changes**: None (fully backward compatible)
-> **New Features**: Lifecycle management, batch operations, compression, quota monitoring
-
-## Overview
-
-Brainy v4.0.0 is a **backward-compatible** release focused on production-ready cost optimization features. Your existing v3 code will continue to work without modifications, but you'll want to enable the new v4.0.0 features for significant cost savings.
-
-**Key Benefits of Upgrading:**
-- 💰 **96% cost savings** with lifecycle policies
-- 🚀 **1000x faster** bulk deletions with batch operations
-- 📦 **60-80% space savings** with gzip compression
-- 📊 **Real-time quota monitoring** for OPFS
-- 🎯 **Zero downtime** migration
-
-## What's New in v4.0.0
-
-### 1. Lifecycle Management (Cloud Storage)
-
-**Automatic tier transitions for massive cost savings:**
-
-```typescript
-// NEW in v4.0.0
-await storage.setLifecyclePolicy({
-  rules: [{
-    id: 'archive-old-data',
-    prefix: 'entities/',
-    status: 'Enabled',
-    transitions: [
-      { days: 30, storageClass: 'STANDARD_IA' },
-      { days: 90, storageClass: 'GLACIER' }
-    ]
-  }]
-})
-```
-
-**Supported on:**
-- ✅ AWS S3 (Lifecycle + Intelligent-Tiering)
-- ✅ Google Cloud Storage (Lifecycle + Autoclass)
-- ✅ Azure Blob Storage (Lifecycle policies)
-
-### 2. Batch Operations
-
-**1000x faster bulk deletions:**
-
-```typescript
-// v3: Delete one at a time (slow, expensive)
-for (const id of idsToDelete) {
-  await brain.remove(id)  // 1000 API calls for 1000 entities
-}
-
-// v4.0.0: Batch delete (fast, cheap)
-const paths = idsToDelete.flatMap(id => [
-  `entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
-  `entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
-])
-await storage.batchDelete(paths)  // 1 API call for 1000 objects (S3)
-```
-
-**Efficiency gains:**
-- S3: 1000 objects per batch
-- GCS: 100 objects per batch
-- Azure: 256 objects per batch
-
-### 3. Compression (FileSystem)
-
-**60-80% space savings for local storage:**
-
-```typescript
-// NEW in v4.0.0
-const brain = new Brainy({
-  storage: {
-    type: 'filesystem',
-    path: './data',
-    compression: true  // Enable gzip compression
-  }
-})
-
-// Automatic compression/decompression on all reads/writes
-```
-
-### 4. Quota Monitoring (OPFS)
-
-**Prevent quota exceeded errors in browsers:**
-
-```typescript
-// NEW in v4.0.0
-const status = await storage.getStorageStatus()
-
-if (status.details.usagePercent > 80) {
-  console.warn('Approaching quota limit:', status.details)
-  // Take action: cleanup old data, notify user, etc.
-}
-```
-
-### 5. Tier Management (Azure)
-
-**Manual or automatic tier transitions:**
-
-```typescript
-// NEW in v4.0.0
-await storage.changeBlobTier(blobPath, 'Cool')  // Hot → Cool (50% savings)
-await storage.batchChangeTier([blob1, blob2], 'Archive')  // 99% savings
-
-// Rehydrate from Archive when needed
-await storage.rehydrateBlob(blobPath, 'High')  // 1-hour rehydration
-```
-
-## Storage Architecture Changes
-
-### v3.x Storage Structure
-
-```
-brainy-data/
-├── nouns/
-│   └── {uuid}.json          # Single file per entity
-├── verbs/
-│   └── {uuid}.json          # Single file per relationship
-├── metadata/
-│   └── __metadata_*.json    # Indexes
-└── _system/
-    └── statistics.json
-```
-
-### v4.0.0 Storage Structure (Automatic Migration)
-
-```
-brainy-data/
-├── entities/
-│   ├── nouns/
-│   │   ├── vectors/         # Vector + HNSW graph (NEW)
-│   │   │   ├── 00/ ... ff/  # 256 UUID shards (NEW)
-│   │   └── metadata/        # Business data (NEW)
-│   │       ├── 00/ ... ff/  # 256 UUID shards (NEW)
-│   └── verbs/
-│       ├── vectors/         # Relationship vectors (NEW)
-│       │   ├── 00/ ... ff/
-│       └── metadata/        # Relationship data (NEW)
-│           ├── 00/ ... ff/
-└── _system/                 # Unchanged
-    └── __metadata_*.json
-```
-
-**Key Changes:**
-1. **Metadata/Vector Separation**: Entities split into 2 files for optimal I/O
-2. **UUID-Based Sharding**: 256 shards for cloud storage optimization
-3. **Automatic Migration**: Brainy handles migration transparently on first run
-
-## Migration Steps
-
-### Step 1: Update Brainy Package
-
-```bash
-npm install @soulcraft/brainy@latest
-```
-
-**Check your version:**
-```bash
-npm list @soulcraft/brainy
-# Should show: @soulcraft/brainy@4.0.0
-```
-
-### Step 2: No Code Changes Required! ✅
-
-Your existing v3 code will work without modifications:
-
-```typescript
-// This v3 code works perfectly in v4.0.0
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: './data' }
-})
-
-await brain.init()
-await brain.add("content", { type: "entity" })
-const results = await brain.search("query")
-```
-
-### Step 3: First Run (Automatic Migration)
-
-On first initialization with v4.0.0:
-
-1. **Brainy detects v3 storage structure**
-2. **Transparently migrates to v4.0.0 structure**:
-   - Creates `entities/` directory
-   - Migrates `nouns/` → `entities/nouns/vectors/` + `entities/nouns/metadata/`
-   - Migrates `verbs/` → `entities/verbs/vectors/` + `entities/verbs/metadata/`
-   - Applies UUID-based sharding
-3. **Old structure preserved** (optional cleanup later)
-
-**Migration time:**
-- 10K entities: ~1 minute
-- 100K entities: ~10 minutes
-- 1M entities: ~2 hours
-
-**Zero downtime:**
-- Migration happens during init()
-- No data loss
-- Automatic rollback on error
-
-### Step 4: Enable v4.0.0 Features (Optional but Recommended)
-
-#### Enable Lifecycle Policies (Cloud Storage)
-
-**AWS S3:**
-```typescript
-// After init()
-await storage.setLifecyclePolicy({
-  rules: [{
-    id: 'optimize-storage',
-    prefix: 'entities/',
-    status: 'Enabled',
-    transitions: [
-      { days: 30, storageClass: 'STANDARD_IA' },
-      { days: 90, storageClass: 'GLACIER' }
-    ]
-  }]
-})
-
-// Or use Intelligent-Tiering (recommended)
-await storage.enableIntelligentTiering('entities/', 'auto-optimize')
-```
-
-**Google Cloud Storage:**
-```typescript
-await storage.enableAutoclass({
-  terminalStorageClass: 'ARCHIVE'
-})
-```
-
-**Azure Blob Storage:**
-```typescript
-await storage.setLifecyclePolicy({
-  rules: [{
-    name: 'optimize-blobs',
-    enabled: true,
-    type: 'Lifecycle',
-    definition: {
-      filters: { blobTypes: ['blockBlob'] },
-      actions: {
-        baseBlob: {
-          tierToCool: { daysAfterModificationGreaterThan: 30 },
-          tierToArchive: { daysAfterModificationGreaterThan: 90 }
-        }
-      }
-    }
-  }]
-})
-```
-
-#### Enable Compression (FileSystem)
-
-```typescript
-const brain = new Brainy({
-  storage: {
-    type: 'filesystem',
-    path: './data',
-    compression: true  // NEW: 60-80% space savings
-  }
-})
-```
-
-#### Use Batch Operations
-
-```typescript
-// Replace individual deletes with batch delete
-const idsToDelete = [/* ... */]
-const paths = idsToDelete.flatMap(id => {
-  const shard = id.substring(0, 2)
-  return [
-    `entities/nouns/vectors/${shard}/${id}.json`,
-    `entities/nouns/metadata/${shard}/${id}.json`
-  ]
-})
-
-await storage.batchDelete(paths)  // Much faster!
-```
-
-#### Monitor Quota (OPFS)
-
-```typescript
-// Periodically check quota in browser apps
-setInterval(async () => {
-  const status = await storage.getStorageStatus()
-  if (status.details.usagePercent > 80) {
-    notifyUser('Storage approaching limit')
-  }
-}, 60000)  // Check every minute
-```
-
-## Backward Compatibility
-
-### Guaranteed to Work (No Changes Needed)
-
-✅ All v3 APIs remain unchanged
-✅ Storage adapters backward compatible
-✅ Metadata structure unchanged
-✅ Query APIs unchanged
-✅ Configuration options unchanged
-
-### New Optional APIs (Add When Ready)
-
-- `storage.setLifecyclePolicy()` - NEW in v4.0.0
-- `storage.getLifecyclePolicy()` - NEW in v4.0.0
-- `storage.removeLifecyclePolicy()` - NEW in v4.0.0
-- `storage.enableIntelligentTiering()` - NEW in v4.0.0 (S3)
-- `storage.enableAutoclass()` - NEW in v4.0.0 (GCS)
-- `storage.batchDelete()` - NEW in v4.0.0
-- `storage.changeBlobTier()` - NEW in v4.0.0 (Azure)
-- `storage.getStorageStatus()` - Enhanced in v4.0.0
-
-## Testing Your Migration
-
-### 1. Test in Development First
-
-```typescript
-// Create test brain with v4.0.0
-const testBrain = new Brainy({
-  storage: { type: 'filesystem', path: './test-data' }
-})
-
-await testBrain.init()
-
-// Verify migration
-console.log('Initialization complete')
-
-// Test basic operations
-const id = await testBrain.add("test content", { type: "test" })
-const results = await testBrain.search("test")
-console.log('Basic operations working:', results.length > 0)
-```
-
-### 2. Verify Storage Structure
-
-```bash
-# Check new directory structure
-ls -la ./test-data/entities/nouns/vectors/
-# Should see: 00/ 01/ 02/ ... ff/ (256 shards)
-
-ls -la ./test-data/entities/nouns/metadata/
-# Should see: 00/ 01/ 02/ ... ff/ (256 shards)
-```
-
-### 3. Verify Data Integrity
-
-```typescript
-// Query all entities
-const allEntities = await testBrain.find({})
-console.log('Total entities:', allEntities.length)
-
-// Verify specific entities
-const entity = await testBrain.get(knownEntityId)
-console.log('Entity retrieved:', entity !== null)
-```
-
-### 4. Test Performance
-
-```typescript
-// Benchmark search
-const start = Date.now()
-const results = await testBrain.search("query")
-const duration = Date.now() - start
-console.log('Search time:', duration, 'ms')
-
-// Should be similar or faster than v3
-```
-
-## Rollback Procedure (If Needed)
-
-If you encounter issues, you can rollback:
-
-### Option 1: Rollback Package
-
-```bash
-# Reinstall v3
-npm install @soulcraft/brainy@^3.50.0
-
-# Restart application
-```
-
-**Important:** v3 can still read v3-structured data (preserved during migration)
-
-### Option 2: Restore from Backup
-
-```bash
-# If you backed up data before migration
-rm -rf ./data
-cp -r ./data-backup ./data
-
-# Reinstall v3
-npm install @soulcraft/brainy@^3.50.0
-```
-
-## Common Migration Scenarios
-
-### Scenario 1: Small Application (<10K Entities)
-
-**Migration time:** 1 minute
-**Recommended approach:**
-1. Update npm package
-2. Restart application (automatic migration)
-3. Enable lifecycle policies immediately
-
-### Scenario 2: Medium Application (10K-1M Entities)
-
-**Migration time:** 10 minutes - 2 hours
-**Recommended approach:**
-1. Backup data
-2. Update npm package
-3. Schedule maintenance window
-4. Restart application (automatic migration)
-5. Verify data integrity
-6. Enable lifecycle policies
-
-### Scenario 3: Large Application (1M+ Entities)
-
-**Migration time:** 2-24 hours
-**Recommended approach:**
-1. **Backup data** (critical!)
-2. Test migration on staging environment
-3. Schedule extended maintenance window
-4. Update npm package on production
-5. Restart application (automatic migration)
-6. Monitor migration progress
-7. Verify data integrity thoroughly
-8. Enable lifecycle policies gradually
-
-## Cost Savings After Migration
-
-### Enable All v4.0.0 Features
-
-**500TB Dataset Example:**
-
-**Before v4.0.0 (v3 with AWS S3 Standard):**
-```
-Storage: $138,000/year
-Operations: $5,000/year
-Total: $143,000/year
-```
-
-**After v4.0.0 (with Intelligent-Tiering):**
-```
-Storage: $51,000/year (64% savings)
-Operations: $5,000/year
-Total: $56,000/year
-```
-
-**After v4.0.0 (with Lifecycle Policies):**
-```
-Storage: $5,940/year (96% savings!)
-Operations: $5,000/year
-Total: $10,940/year
-```
-
-**Annual Savings: $132,060 (96% reduction)**
-
-## Troubleshooting
-
-### Issue: Migration takes too long
-
-**Solution:**
-- Migration is I/O bound
-- For 1M+ entities, consider:
-  - Running during off-peak hours
-  - Using faster storage (SSD vs HDD)
-  - Increasing available memory
-  - Running on more powerful instance
-
-### Issue: "Storage structure not recognized"
-
-**Solution:**
-```typescript
-// Manually trigger migration
-await brain.storage.migrateToV4()  // If automatic migration fails
-
-// Or start fresh (data loss warning!)
-await brain.storage.clear()
-await brain.init()
-```
-
-### Issue: Lifecycle policy not working
-
-**Solution:**
-```typescript
-// Verify policy is set
-const policy = await storage.getLifecyclePolicy()
-console.log('Active rules:', policy.rules)
-
-// Cloud providers may take 24-48 hours to start transitions
-// Check again after 2 days
-
-// Verify in cloud console:
-// - AWS: S3 → Bucket → Management → Lifecycle
-// - GCS: Storage → Bucket → Lifecycle
-// - Azure: Storage Account → Lifecycle management
-```
-
-### Issue: Batch delete not working
-
-**Solution:**
-```typescript
-// Ensure storage adapter supports batch delete
-const status = await storage.getStorageStatus()
-console.log('Storage type:', status.type)
-
-// Batch delete requires:
-// - S3CompatibleStorage ✅
-// - GcsStorage ✅
-// - AzureBlobStorage ✅
-// - FileSystemStorage ✅
-// - OPFSStorage ✅
-// - MemoryStorage ✅
-```
-
-## Best Practices
-
-1. ✅ **Backup before upgrading** (especially for large datasets)
-2. ✅ **Test on staging first** (verify migration works)
-3. ✅ **Monitor during migration** (watch logs for errors)
-4. ✅ **Enable lifecycle policies immediately** (start saving costs)
-5. ✅ **Use batch operations** (for any bulk cleanup)
-6. ✅ **Monitor quota** (OPFS browser apps)
-7. ✅ **Enable compression** (FileSystem storage)
-
-## Getting Help
-
-**Documentation:**
-- [AWS S3 Cost Optimization Guide](./operations/cost-optimization-aws-s3.md)
-- [GCS Cost Optimization Guide](./operations/cost-optimization-gcs.md)
-- [Azure Cost Optimization Guide](./operations/cost-optimization-azure.md)
-- [Cloudflare R2 Cost Optimization Guide](./operations/cost-optimization-cloudflare-r2.md)
-
-**Support:**
-- GitHub Issues: [https://github.com/soulcraft/brainy/issues](https://github.com/soulcraft/brainy/issues)
-- GitHub Discussions: [https://github.com/soulcraft/brainy/discussions](https://github.com/soulcraft/brainy/discussions)
-
-## Summary
-
-**Migration Checklist:**
-- ✅ Backup data
-- ✅ Update npm package (`npm install @soulcraft/brainy@latest`)
-- ✅ Restart application (automatic migration)
-- ✅ Verify data integrity
-- ✅ Enable lifecycle policies
-- ✅ Enable compression (FileSystem)
-- ✅ Use batch operations
-- ✅ Monitor cost savings
-
-**Expected Results:**
-- ✅ Zero downtime migration
-- ✅ Full backward compatibility
-- ✅ 60-96% cost savings
-- ✅ 1000x faster bulk operations
-- ✅ 60-80% space savings (with compression)
-
-**Timeline:**
-- Small app (<10K): 1 minute migration
-- Medium app (10K-1M): 10 minutes - 2 hours
-- Large app (1M+): 2-24 hours
-
-**Welcome to Brainy v4.0.0! 🎉**
-
----
-
-**Version**: v4.0.0
-**Migration Difficulty**: Low
-**Breaking Changes**: None
-**Recommended Upgrade**: Yes (significant cost savings)

docs/PERFORMANCE.md

@@ -1,492 +0,0 @@
-# Brainy Performance & Architecture
-
-## Performance Characteristics
-
-Brainy achieves high performance through carefully optimized data structures and algorithms. The tables below describe each component by its **algorithmic complexity** — the durable, defensible guarantee. The example latencies are figures from a single 100-item run on one machine (see [Benchmarks](#benchmarks)); they are illustrative, not a committed benchmark, and vary with hardware, embedding model, and storage backend. The one component with a committed scale assertion is the graph adjacency index (`tests/performance/graph-scale-performance.test.ts:238`).
-
-### Core Performance Summary
-
-| Component | Operation | Time Complexity | Example latency (100-item run)\* | Data Structure |
-|-----------|-----------|-----------------|---------------------|----------------|
-| **Metadata Index** | Exact match | **O(1)** | 0.8ms | `Map<string, Set<string>>` |
-| **Metadata Index** | Range query | **O(log n) + O(k)** | 0.6ms | Sorted array + binary search |
-| **Graph Index** | Get neighbors | **O(1)** | 0.09ms | `Map<string, Set<string>>` |
-| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | Hierarchical graph |
-| **NLP Parser** | Query parsing | **O(m)** | 8.9ms | 220 pre-computed patterns |
-| **Type-Field Affinity** | Field matching | **O(f)** | 0.1ms | Type-specific field cache |
-| **Type Detection** | Noun/Verb matching | **O(t)** | 0.3ms | Pre-embedded type vectors |
-| **Triple Intelligence** | Combined query | **O(1) to O(log n)** | 1.8ms | Parallel execution |
-
-\* Illustrative single-run figures at 100 items on one machine — not a committed benchmark. Only the graph index carries an asserted scale bound (measured <1 ms per neighbor lookup up to 1M relationships, `tests/performance/graph-scale-performance.test.ts:238`).
-
-Where:
-- `n` = number of items in index
-- `k` = number of results returned
-- `m` = number of patterns to check
-- `f` = number of fields for entity type
-- `t` = number of types (42 nouns, 127 verbs)
-
-### brain.get() Metadata-Only Optimization
-
-`brain.get()` returns **metadata only by default**, skipping the 384-dimensional
-embedding — the bulk of an entity's payload. Callers that need the vector opt in
-with `{ includeVectors: true }`.
-
-| Operation | Default (metadata-only) | With `includeVectors: true` | Use Case |
-|-----------|-------------------------|-----------------------------|----------|
-| **brain.get()** | Skips vector load | Loads full vector | VFS, existence checks, metadata |
-| **VFS readFile() / readdir()** | Inherits metadata-only path | n/a | File operations, directory listings |
-
-**Key Innovation**: Lazy vector loading — only load the 384-dimensional embedding when explicitly needed.
-
-The integration test `tests/integration/metadata-only-comprehensive.test.ts:306`
-asserts metadata-only `get()` is faster than the full-entity `get()`
-(`metadataTime < fullTime`). The *magnitude* of the speedup is
-environment-dependent (the percentage assertion in
-`tests/integration/vfs-performance-v5.11.1.test.ts` is intentionally skipped on
-CI for that reason), so no fixed percentage is quoted here.
-
-**Why this matters**:
-- Most `brain.get()` calls don't need vectors (VFS, admin tools, import utilities, data APIs)
-- The embedding dominates an entity's serialized size, so skipping it is the largest win
-- **Zero code changes** for most applications — automatic by default
-
-**When to use what**:
-```typescript
-// DEFAULT: Metadata-only (skips the vector load) - use for:
-const entity = await brain.get(id)
-// - VFS operations (readFile, stat, readdir)
-// - Existence checks: if (await brain.get(id)) ...
-// - Metadata access: entity.data, entity.type, entity.metadata
-// - Relationship traversal
-
-// EXPLICIT: Full entity (same as before) - use ONLY for:
-const entity = await brain.get(id, { includeVectors: true })
-// - Computing similarity on THIS entity
-// - Manual vector operations
-// - Vector index graph traversal
-```
-
-## Architecture Deep Dive
-
-### 1. Metadata Index - O(1) Lookups
-
-The `MetadataIndexManager` uses inverted indexes for lightning-fast metadata filtering.
-
-**UPDATED**: Sorted indices for range queries are now built **incrementally during CRUD operations**. No lazy loading delays - range queries are consistently fast. Binary search insertions maintain O(log n) performance during updates.
-
-```typescript
-class MetadataIndexManager {
-  // O(1) exact match via HashMap
-  private indexCache = new Map<string, MetadataIndexEntry>()
-  
-  // O(log n) range queries via sorted arrays (incremental updates)
-  private sortedIndices = new Map<string, SortedFieldIndex>()
-  
-  // Type-field affinity for intelligent NLP
-  private typeFieldAffinity = new Map<string, Map<string, number>>()
-  
-  interface MetadataIndexEntry {
-    field: string
-    value: string | number | boolean
-    ids: Set<string>  // O(1) add/remove/has
-  }
-  
-  interface SortedFieldIndex {
-    values: Array<[value: any, ids: Set<string>]>  // Sorted for O(log n) ranges
-    fieldType: 'number' | 'string' | 'date'
-  }
-}
-```
-
-**How it works:**
-1. Each field+value combination gets a unique key: `"category:tech"`
-2. Map lookup is O(1) average case
-3. Returns a Set of matching IDs instantly
-
-**Example Query:**
-```javascript
-// Query: { where: { category: 'tech' } }
-// Internally: indexCache.get('category:tech') → O(1)
-```
-
-### 2. Range Queries - O(log n)
-
-For numeric/date fields, Brainy maintains sorted indices:
-
-```typescript
-interface SortedFieldIndex {
-  values: Array<[value: any, ids: Set<string>]>  // Sorted by value
-  fieldType: 'number' | 'string' | 'date'
-}
-```
-
-**How it works:**
-1. Binary search to find range start: O(log n)
-2. Binary search to find range end: O(log n)
-3. Collect all IDs in range: O(k) where k = items in range
-
-**Example Query:**
-```javascript
-// Query: { where: { age: { greaterThan: 25, lessThan: 40 } } }
-// Internally: binarySearch(25) + binarySearch(40) + collect
-```
-
-### 3. Graph Adjacency Index - O(1) Traversal
-
-The `GraphAdjacencyIndex` provides instant graph traversal:
-
-```typescript
-class GraphAdjacencyIndex {
-  // Bidirectional adjacency lists
-  private sourceIndex = new Map<string, Set<string>>()  // id → outgoing
-  private targetIndex = new Map<string, Set<string>>()  // id → incoming
-  
-  // O(1) neighbor lookup
-  async getNeighbors(id: string, direction: 'in' | 'out' | 'both') {
-    const outgoing = this.sourceIndex.get(id)  // O(1)
-    const incoming = this.targetIndex.get(id)  // O(1)
-  }
-}
-```
-
-**Key Innovation:** Pure Map/Set operations - no database queries, no loops, just direct memory access.
-
-### 4. Vector Index - O(log n)
-
-The default vector index (`JsHnswVectorIndex`) provides logarithmic approximate nearest neighbor search through a hierarchical graph:
-
-```typescript
-class JsHnswVectorIndex {
-  private nouns: Map<string, HNSWNoun> = new Map()
-
-  interface HNSWNoun {
-    id: string
-    vector: number[]
-    connections: Map<number, Set<string>>  // layer → neighbors
-    level: number
-  }
-}
-```
-
-**How it works:**
-1. Start at entry point (top layer)
-2. Greedy search to find nearest neighbor at each layer
-3. Move down layers for progressively finer search
-4. Each layer has M connections (typically 16)
-
-**Performance:** O(log n) due to hierarchical structure
-
-### 5. Type-Aware NLP with Dynamic Field Discovery
-
-The NLP processor uses **zero hardcoded fields** - everything is discovered dynamically from actual data:
-
-```typescript
-class NaturalLanguageProcessor {
-  // Pre-embedded NounTypes (42) and VerbTypes (127) - ONLY hardcoded vocabularies
-  private nounTypeEmbeddings = new Map<string, Vector>()
-  private verbTypeEmbeddings = new Map<string, Vector>()
-  
-  // Dynamic field embeddings from actual indexed data
-  private fieldEmbeddings = new Map<string, Vector>()
-  
-  // Type-field affinity for intelligent prioritization
-  async getFieldsForType(nounType: NounType) {
-    return this.brain.getFieldsForType(nounType)  // Real data patterns
-  }
-}
-```
-
-**Type-Aware Intelligence Flow:**
-1. **Type Detection**: "documents" → `NounType.Document` (semantic similarity)
-2. **Field Prioritization**: Get fields common to Document type from real data
-3. **Semantic Field Matching**: "by" → "author" (with type affinity boost)
-4. **Validation**: Ensure "author" field actually appears with Document entities
-5. **Query Optimization**: Process low-cardinality type-specific fields first
-
-**Performance Characteristics:**
-- Type detection: O(t) where t = 169 total types (42 noun + 127 verb)
-- Field matching: O(f) where f = fields for detected type (typically 5-15)
-- Validation: O(1) lookup in type-field affinity map
-- No hardcoded assumptions - learns from actual data patterns
-
-### 6. NLP with 220 Pre-computed Patterns
-
-Pattern matching with embedded templates for instant semantic understanding:
-
-```typescript
-// 394KB of embedded patterns compiled into the source
-export const EMBEDDED_PATTERNS: Pattern[] = [/* 220 patterns */]
-export const PATTERN_EMBEDDINGS: Float32Array = /* 220 × 384 dimensions */
-```
-
-**How it works:**
-1. Query embedding computed once: O(1) with cached model
-2. Cosine similarity with 220 patterns: O(m) where m = 220
-3. Pattern templates enhanced with type context
-4. No network calls, no external dependencies, no hardcoded fields
-
-## Parallel Execution
-
-Triple Intelligence queries execute searches in parallel:
-
-```javascript
-// Vector and proximity searches run simultaneously
-const searchPromises = [
-  this.executeVectorSearch(params),    // Runs in parallel
-  this.executeProximitySearch(params)  // Runs in parallel
-]
-const results = await Promise.all(searchPromises)
-```
-
-## Memory Efficiency
-
-### Space Complexity
-
-| Component | Memory Usage | Formula |
-|-----------|--------------|---------|
-| Metadata Index | ~40 bytes/entry | `(key_size + 8) × unique_values + 8 × total_items` |
-| Graph Index | ~24 bytes/edge | `16 × edges + 8 × nodes` |
-| Vector Index | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
-| Pattern Library | 394KB fixed | Pre-computed, shared across instances |
-| Type Embeddings | ~60KB fixed | 70 types × 384 dimensions × 4 bytes, cached |
-| Field Embeddings | ~5KB dynamic | Actual fields × 384 dimensions × 4 bytes |
-| Type-Field Affinity | ~2KB dynamic | Type-field occurrence counts |
-
-### Caching Strategy
-
-- **Metadata Cache**: LRU with 5-minute TTL, 500 entries max
-- **Embedding Cache**: Permanent for session, prevents recomputation
-- **Unified Cache**: Coordinates memory across all components
-
-## Benchmarks
-
-### Illustrative Single Run (100 items, one machine)
-
-Example output from a single 100-item run — illustrative only, not a committed
-benchmark; absolute numbers vary by hardware. The values feed the
-[Core Performance Summary](#core-performance-summary) example-latency column.
-
-```
-Metadata exact match: 0.818ms (50 items matched)
-Metadata range query: 0.631ms (40 items in range)
-Graph neighbor lookup: 0.092ms (2 connections)
-Vector k-NN search:    1.773ms (10 nearest neighbors)
-NLP query parsing:     8.906ms (full natural language)
-Triple Intelligence:   1.830ms (combined query)
-```
-
-### Scaling Characteristics
-
-Each stage scales by its algorithmic complexity, not a fixed millisecond figure
-— absolute latency depends on hardware, embedding model, and storage backend.
-Only the graph adjacency index carries a committed scale assertion:
-
-| Query stage | Complexity | Scaling behavior |
-|-------------|------------|------------------|
-| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
-| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
-| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
-| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
-| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
-
-## Comparison with Other Systems
-
-| System | Metadata Filter | Graph Traversal | Vector Search | Natural Language |
-|--------|-----------------|-----------------|---------------|------------------|
-| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) vector index | 220 patterns |
-| Neo4j | O(log n) B-tree | O(k) traversal | Not native | Not native |
-| Elasticsearch | O(log n) inverted | Not native | O(n) brute force* | Basic tokenization |
-| PostgreSQL | O(log n) B-tree | O(k) recursive | O(n) brute force* | Full-text only |
-| Pinecone | Not native | Not native | O(log n) | Not native |
-
-*Without additional plugins/extensions
-
-## Key Innovations
-
-1. **True O(1) Metadata Filtering**: Most databases use B-trees (O(log n)). Brainy uses HashMaps for constant-time lookups.
-
-2. **O(1) Graph Traversal**: Unlike traditional graph databases that traverse edges, Brainy maintains bidirectional adjacency maps for instant neighbor access.
-
-3. **Unified Triple Intelligence**: First system to natively combine O(1) metadata, O(1) graph, and O(log n) vector search in a single query.
-
-4. **Embedded NLP**: 220 research-based patterns with pre-computed embeddings compiled directly into the codebase - no external dependencies.
-
-5. **Parallel Search Execution**: Vector, metadata, and graph searches execute simultaneously, not sequentially.
-
-## Production Readiness
-
-- ✅ **No External Dependencies**: All algorithms implemented in pure TypeScript
-- ✅ **No Network Calls**: Everything runs locally, including embeddings
-- ✅ **Thread-Safe**: Immutable data structures where possible
-- ✅ **Memory Bounded**: Configurable cache sizes and automatic cleanup
-- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
-- ✅ **Zero Stubs**: Every line of code is production-ready
-
-## Lazy Loading Performance
-
-Brainy supports two initialization modes for optimal performance across different use cases:
-
-### Mode 1: Auto-Rebuild (Default)
-
-```javascript
-const brain = new Brainy()
-await brain.init()  // Rebuilds indexes during init (~500ms-3s for 10K entities)
-```
-
-**Performance:**
-- Init time: 500ms-3s (depends on dataset size)
-- First query: Instant (indexes already loaded)
-- Use case: Traditional applications, long-running servers
-
-### Mode 2: Lazy Loading
-
-```javascript
-const brain = new Brainy({ disableAutoRebuild: true })
-await brain.init()  // Returns instantly (0-10ms)
-
-const results = await brain.find({ limit: 10 })  // First query triggers rebuild (~50-200ms)
-const more = await brain.find({ limit: 100 })    // Subsequent queries instant (0ms check)
-```
-
-**Performance:**
-- Init time: 0-10ms (instant)
-- First query: 50-200ms (includes index rebuild for 1K-10K entities)
-- Subsequent queries: 0ms check (instant)
-- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
-
-**Concurrency Safety:**
-```javascript
-// 100 concurrent queries immediately after init
-await brain.init()
-
-const promises = Array.from({ length: 100 }, () =>
-  brain.find({ limit: 10 })
-)
-
-const results = await Promise.all(promises)
-// ✅ Only 1 rebuild triggered (mutex)
-// ✅ All 100 queries return correct results
-// ✅ Total time: ~60ms (not 6000ms!)
-```
-
-**Use Cases for Lazy Loading:**
-- **Serverless/Edge**: Minimize cold start time (0-10ms init)
-- **Development**: Faster restarts during development
-- **Large datasets**: Defer index loading until needed
-- **Read-heavy workloads**: Writes don't wait for index rebuild
-
-## Zero Configuration Required
-
-Brainy is designed to be **smart enough to tune itself dynamically**. No configuration needed:
-
-```javascript
-// That's it. Brainy handles everything.
-const brain = new Brainy()
-await brain.init()
-
-// Or with lazy loading for serverless
-const brain = new Brainy({ disableAutoRebuild: true })
-await brain.init()  // Instant (0-10ms)
-```
-
-### Automatic Self-Tuning
-
-- **Metadata Index**: Auto-builds sorted indices for range queries on first use
-- **Graph Index**: Auto-flushes every 30 seconds
-- **Default Tuning**: Research-based vector index defaults
-- **Lazy Loading**: Indices built only when needed
-- **Cache Management**: LRU caches with TTL
-
-### Intelligent Defaults
-
-- **Vector recall** = `'balanced'` (M=16, ef=200): right for most datasets
-- **Cache TTL** = 5 min: balances freshness and performance
-- **Flush interval** = 30 s: non-blocking background persistence
-
-### Vector Index Tuning Knobs
-
-Brainy 8.0 exposes two knobs on `config.vector`:
-
-```javascript
-const brain = new Brainy({
-  vector: {
-    recall: 'fast',                // 'fast' | 'balanced' | 'accurate'
-    persistMode: 'deferred'        // 'immediate' | 'deferred'
-  }
-})
-```
-
-The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cor`) can replace it with a higher-performing implementation; the public knobs stay the same.
-
-### Scale Scenarios
-
-| Scale | Items | Storage Strategy | Performance |
-|-------|-------|------------------|-------------|
-| **Small** | <10K | Memory | Sub-millisecond |
-| **Medium** | 10K-1M | Filesystem | 1-5ms |
-| **Large** | 1M-10M | Filesystem + tuned cache | 2-10ms |
-| **Massive** | 10M+ | Filesystem + native vector provider + service-layer sharding | 5-20ms |
-
-For >10M entities, run multiple Brainy processes behind your own routing layer — Brainy 8.0 doesn't ship cluster coordination.
-
-### Architecture
-
-```
-┌─────────────────────────────────────────┐
-│         Application Layer               │
-│         (Your Code)                     │
-└─────────────┬───────────────────────────┘
-              │
-┌─────────────▼───────────────────────────┐
-│         Brainy Core                     │
-│   (Triple Intelligence Engine)          │
-├─────────────────────────────────────────┤
-│  Memory     │  Vector     │  Metadata   │
-│  Cache      │  Index      │  Index      │
-└─────────────┬───────────────────────────┘
-              │
-┌─────────────▼───────────────────────────┐
-│         Storage Layer                   │
-├──────────┬──────────┬──────────────────┤
-│ Vectors  │  Graph   │   Files          │
-│ (sharded)│  Edges   │  (filesystem)    │
-└──────────┴──────────┴──────────────────┘
-```
-
-For off-site replication, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`).
-
-### Performance at Scale
-
-- **Metadata queries**: O(1) HashMap
-- **Graph traversal**: O(1) adjacency lookup
-- **Vector search**: O(log n)
-- **Write throughput**: 50K+ writes/second per process (filesystem, batched)
-- **Read throughput**: 1M+ reads/second with caching
-
-### Zero-Config with Autoscaling
-
-- **AutoConfiguration System**: Detects environment and adjusts settings
-- **Learning from Performance**: `learnFromPerformance()` adapts based on metrics
-- **Auto-flush**: Graph index (30s), Metadata index (configurable)
-- **Auto-optimize**: Enabled by default in graph and vector indices
-- **Zero-config presets**: Production, development, minimal modes
-- **Adaptive memory**: Scales caches based on available memory
-
-## Implementation Status
-
-### Fully Implemented and Production-Ready
-- **O(1) metadata lookups** via HashMaps (exact match)
-- **O(log n) range queries** via sorted arrays with lazy building
-- **O(1) graph traversal** via adjacency maps
-- **O(log n) vector search** via the default JS index, swappable for a native provider
-- **220 NLP patterns** with pre-computed embeddings
-- **Filesystem and memory storage** adapters
-- **Auto-configuration system** with environment detection
-- **Zero-config operation** with intelligent defaults
-- **Auto-flush and auto-optimize** in indices
-- **Low-latency Triple Intelligence queries** (O(log n) vector + O(1) metadata/graph)
-
-## Conclusion
-
-Brainy delivers on its promise of **production-ready Triple Intelligence** with documented algorithmic-complexity guarantees and a committed graph-scale benchmark (`tests/performance/graph-scale-performance.test.ts`). All listed features are fully implemented and tested. No stubs, no mocks — just real, working code with characterized performance.

docs/PLUGINS.md

@@ -1,486 +0,0 @@
----
-title: Plugin System
-slug: guides/plugins
-public: true
-category: guides
-template: guide
-order: 4
-description: Replace any Brainy subsystem — distance functions, embeddings, vector index, metadata index, aggregation — with a custom implementation or optional native acceleration.
-next:
-  - guides/storage-adapters
----
-
-# Plugin Development Guide
-
-Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
-
-## Architecture Overview
-
-Brainy's plugin system uses **named providers** — string keys mapped to implementations. During `init()`, brainy:
-
-1. Imports each package listed in the `plugins` config array
-2. Activates each plugin, passing a `BrainyPluginContext`
-3. The plugin calls `context.registerProvider(key, implementation)` for each subsystem it provides
-4. Brainy checks each provider key and wires the implementation into its internal pipeline
-
-Installing the first-party accelerator is the opt-in: with the default config, brainy probes for `@soulcraft/cor` and loads it when present. Everything except "not installed" fails **loud** — a present-but-broken accelerator makes `init()` throw rather than silently degrading to the JS engines.
-
-```typescript
-const brain = new Brainy()                                  // @soulcraft/cor auto-detected when installed
-const pinned = new Brainy({ plugins: ['@soulcraft/cor'] })  // or pin exactly what loads
-const plain = new Brainy({ plugins: [] })                   // or opt out of detection entirely
-```
-
-| `plugins` value | Behavior |
-|---|---|
-| `undefined` (default) | Guarded auto-detection of `@soulcraft/cor`: not installed → no plugins, silently; installed → loads + announces; installed-but-broken → `init()` throws |
-| `false` / `[]` | No plugins, no detection (explicit opt-out) |
-| `['@soulcraft/cor']` | Load only the listed packages; a listed plugin that fails to load throws |
-
-Plugins registered programmatically via `brain.use(plugin)` are always activated regardless of the `plugins` config.
-
-If no plugin provides a given key, brainy uses its built-in JavaScript implementation. This means brainy works perfectly standalone — plugins only enhance performance or add capabilities.
-
-## Creating a Plugin
-
-### 1. Implement the `BrainyPlugin` interface
-
-```typescript
-import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
-
-const myPlugin: BrainyPlugin = {
-  name: 'my-brainy-plugin',  // Must be unique (typically your npm package name)
-
-  async activate(context: BrainyPluginContext): Promise<boolean> {
-    // Register your providers here
-    context.registerProvider('distance', myFastDistanceFunction)
-
-    // Return true if activation succeeded, false to skip
-    return true
-  },
-
-  async deactivate(): Promise<void> {
-    // Optional cleanup when brainy.close() is called
-  }
-}
-
-export default myPlugin
-```
-
-### 2. Package exports
-
-Your package must export the plugin as the default export so brainy's plugin loader can resolve it:
-
-```typescript
-// index.ts
-export { default } from './plugin.js'
-```
-
-### 3. Registration
-
-**Config-based:** List your package name in the brainy config:
-
-```typescript
-const brain = new Brainy({
-  plugins: ['my-brainy-plugin']
-})
-await brain.init()
-```
-
-**Programmatic registration:** For plugins not installed as npm packages, use `brain.use()`:
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-import myPlugin from './my-plugin.js'
-
-const brain = new Brainy()
-brain.use(myPlugin)
-await brain.init()
-```
-
-## Provider Keys Reference
-
-Each key has a specific expected signature. Brainy checks for these during `init()` and wires them into the appropriate code paths.
-
-### Core Providers
-
-#### `distance`
-**Type:** `(a: number[], b: number[]) => number`
-
-Replaces the default cosine distance function used in vector search and neural APIs. This is the highest-impact single provider — it's called for every vector comparison.
-
-```typescript
-context.registerProvider('distance', (a: number[], b: number[]): number => {
-  // Your SIMD-accelerated or GPU distance calculation
-  return myFastCosineDistance(a, b)
-})
-```
-
-#### `embeddings`
-**Type:** `(text: string | string[]) => Promise<number[] | number[][]>`
-
-Replaces the built-in WASM embedding engine. Called for every `brain.add()`, `brain.update()`, and `brain.find()` operation that involves text.
-
-```typescript
-context.registerProvider('embeddings', async (text: string | string[]) => {
-  if (Array.isArray(text)) {
-    return myEngine.embedBatch(text)
-  }
-  return myEngine.embed(text)
-})
-```
-
-#### `embedBatch`
-**Type:** `(texts: string[]) => Promise<number[][]>`
-
-Dedicated batch embedding provider. When registered, brainy uses this for bulk operations (import, reindex, batch add) instead of calling the `embeddings` provider N times. This enables true single-forward-pass batch processing.
-
-Priority order for batch operations:
-1. `embedBatch` provider (single forward pass — fastest)
-2. `embeddings` provider with `Promise.all()` (N individual calls)
-3. Built-in WASM batch API (fallback)
-
-```typescript
-context.registerProvider('embedBatch', async (texts: string[]) => {
-  // Process all texts in a single forward pass
-  return myEngine.batchEmbed(texts)
-})
-```
-
-### Index Providers
-
-> **Write-path invariant (the change-feed contract).** Every canonical
-> mutation flows through Brainy's generation-store commit points — index
-> providers are invoked *inside* that commit and never originate canonical
-> writes of their own. The `brain.onChange` change feed is emitted from those
-> commit points and relies on this: **a plugin must never introduce a write
-> path that bypasses the generation-store commit.** If a future provider ever
-> needs a direct native ingest path, it must either route through the commit
-> or emit equivalent change events — otherwise every `onChange` consumer
-> (live UIs, cache invalidation, realtime sync) silently develops a blind
-> spot.
-
-#### `vector`
-**Type:** `(config: object, distanceFunction: Function, options: object) => VectorIndexProvider-compatible`
-
-Factory function that creates a vector index instance. The returned object must implement the `VectorIndexProvider` public API:
-
-- `addItem(item: { id: string, vector: number[] }): Promise<string>`
-- `search(queryVector: number[], k: number, filter?, options?): Promise<Array<[string, number]>>`
-- `removeItem(id: string): Promise<boolean>`
-- `size(): number`
-- `clear(): void`
-- `flush(): Promise<number>`
-- `rebuild(options?): Promise<void>`
-- `getDirtyNodeCount(): number`
-- `getPersistMode(): 'immediate' | 'deferred'`
-- `getEntryPointId(): string | null`
-- `getMaxLevel(): number`
-- `getDimension(): number | null`
-- `getConfig(): object`
-- `getDistanceFunction(): Function`
-- `enableCOW(parent): void`
-- `setUseParallelization(boolean): void`
-
-For type-aware indexes (separate graph per noun type), also implement:
-- `getIndexForType(type: string): VectorIndexProvider` (duck-typed detection)
-- `search(queryVector, k, type?, filter?, options?): Promise<Array<[string, number]>>`
-
-```typescript
-context.registerProvider('vector', (config, distanceFn, options) => {
-  return new MyNativeVectorIndex(config, distanceFn, options)
-})
-```
-
-#### The readiness contract (all three index providers)
-
-A provider that **persists its derived index** should implement the optional readiness
-members so a warm reopen never pays a redundant rebuild-from-canonical:
-
-- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
-  `brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
-  and **before the rebuild gate**.
-- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
-  loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
-  exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
-  `totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
-  while fully durable. Never return `true` if the durable state failed to load: the
-  signal is honest in both directions, and a not-ready provider gets its rebuild even
-  when `size() > 0`.
-- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
-  migration); brainy skips its rebuild entirely.
-
-Providers that implement none of these keep the size/count heuristics — correct for
-engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
-
-#### `metadataIndex`
-**Type:** `(storage: StorageAdapter) => MetadataIndexManager-compatible`
-
-Factory function that creates a metadata index. The returned object must implement the `MetadataIndexManager` interface including `init()`, `addEntity()`, `removeEntity()`, `query()`, `flush()`, `clear()`, etc.
-
-```typescript
-context.registerProvider('metadataIndex', (storage) => {
-  return new MyNativeMetadataIndex(storage)
-})
-```
-
-#### `graphIndex`
-**Type:** `(storage: StorageAdapter) => GraphAdjacencyIndex-compatible`
-
-Factory function that creates a graph adjacency index for relationship tracking (verbs/triples). Must implement the `GraphAdjacencyIndex` interface including `addVerb()`, `getVerbsBySource()`, `getVerbsByTarget()`, `flush()`, etc.
-
-```typescript
-context.registerProvider('graphIndex', (storage) => {
-  return new MyNativeGraphIndex(storage)
-})
-```
-
-#### `aggregation`
-**Type:** `(storage: StorageAdapter) => AggregationProvider-compatible`
-
-Factory function that creates an aggregation engine for write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows. The returned object must implement the `AggregationProvider` interface.
-
-```typescript
-context.registerProvider('aggregation', (storage) => {
-  return new MyNativeAggregationEngine(storage)
-})
-```
-
-When provided by an optional native acceleration plugin (such as `@soulcraft/cor`), this enables:
-- Compiled source filters (vs per-entity JS object traversal)
-- Precise MIN/MAX via sorted data structures (vs lazy recompute)
-- Parallel aggregate rebuild across CPU cores
-- SIMD-accelerated timestamp bucketing
-
-### Utility Providers
-
-#### `cache`
-**Type:** `UnifiedCache`
-
-Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`).
-
-```typescript
-import type { UnifiedCache } from '@soulcraft/brainy/internals'
-
-context.registerProvider('cache', myNativeCache)
-```
-
-#### `entityIdMapper`
-**Type:** `(storage: StorageAdapter) => EntityIdMapper-compatible`
-
-Factory for bidirectional UUID ↔ integer mapping used by roaring bitmaps. Must implement `getOrAssign()`, `getUuid()`, `getInt()`, `has()`, `remove()`, `flush()`, `clear()`.
-
-#### `roaring`
-**Type:** `RoaringBitmap32 class`
-
-Replacement for the roaring bitmap implementation. Used internally by the metadata index for set operations. Must be API-compatible with `roaring-wasm`.
-
-#### `msgpack`
-**Type:** `{ encode: (data: any) => Buffer, decode: (buffer: Buffer) => any }`
-
-Native msgpack encode/decode for SSTable serialization.
-
-### Analytics Providers (Native-Only)
-
-These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cor`) is installed.
-
-Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it.
-
-#### `analytics:hyperloglog`
-Approximate distinct counts. Count unique values (e.g., unique merchants) across millions of records using ~16KB of memory with ~1% error. Each update is O(1).
-
-#### `analytics:tdigest`
-Streaming percentiles. Compute P50/P90/P95/P99 from streaming data without storing all values. Uses ~4KB per digest with ~1% accuracy at the tails.
-
-#### `analytics:countmin`
-Frequency estimation. Find the most common values (e.g., top-K merchants) using ~40KB with 0.1% error. O(1) per update.
-
-#### `analytics:anomaly`
-Real-time anomaly detection. Flag statistically unusual values at write-time using exponentially weighted moving averages. 64 bytes per group, sub-microsecond decisions.
-
-#### `aggregation:mmap`
-Persistent aggregate storage via memory-mapped files. Aggregate state survives process crashes without explicit flush. Zero serialization overhead.
-
----
-
-## Storage Adapter Plugins
-
-Plugins can register custom storage backends that users reference by name.
-
-### Implementing a Storage Adapter
-
-```typescript
-import type { StorageAdapterFactory } from '@soulcraft/brainy/plugin'
-import type { StorageAdapter } from '@soulcraft/brainy'
-
-class MyStorageAdapter implements StorageAdapter {
-  async init(): Promise<void> { /* ... */ }
-  async saveNoun(noun: HNSWNoun): Promise<void> { /* ... */ }
-  async getNoun(id: string): Promise<HNSWNounWithMetadata | null> { /* ... */ }
-  async deleteNoun(id: string): Promise<void> { /* ... */ }
-  // ... implement all StorageAdapter methods
-}
-```
-
-### Registering a Storage Adapter
-
-```typescript
-context.registerProvider('storage:my-backend', {
-  name: 'my-backend',
-  create: (config: Record<string, unknown>) => {
-    return new MyStorageAdapter(config)
-  }
-} satisfies StorageAdapterFactory)
-```
-
-Users can then use your storage:
-
-```typescript
-const brain = new Brainy({ storage: 'my-backend', myBackendOption: 'value' })
-```
-
-## Import Paths
-
-Brainy provides three entry points for plugin developers:
-
-| Import Path | Contents | Stability |
-|-------------|----------|-----------|
-| `@soulcraft/brainy` | Public API, types, StorageAdapter | Stable (semver) |
-| `@soulcraft/brainy/plugin` | BrainyPlugin, BrainyPluginContext, StorageAdapterFactory | Stable (semver) |
-| `@soulcraft/brainy/internals` | UnifiedCache, EntityIdMapper, logger utilities | Internal (may change between minor versions) |
-
-## Diagnostics
-
-Brainy provides a `diagnostics()` method to verify plugin wiring:
-
-```typescript
-const brain = new Brainy()
-await brain.init()
-
-const diag = brain.diagnostics()
-console.log(diag)
-// {
-//   version: '7.14.0',
-//   plugins: { active: ['my-plugin'], count: 1 },
-//   providers: {
-//     metadataIndex: { source: 'default' },
-//     graphIndex: { source: 'default' },
-//     embeddings: { source: 'plugin' },
-//     embedBatch: { source: 'plugin' },
-//     distance: { source: 'plugin' },
-//     vector: { source: 'default' },
-//     ...
-//   },
-//   indexes: {
-//     vector: { size: 0, type: 'JsHnswVectorIndex' },
-//     metadata: { type: 'MetadataIndexManager', initialized: true },
-//     graph: { type: 'GraphAdjacencyIndex', initialized: true, wiredToStorage: true }
-//   }
-// }
-```
-
-The CLI also supports diagnostics:
-
-```bash
-brainy diagnostics
-```
-
-### Init-Time Summary
-
-When a plugin is active, brainy automatically logs a provider summary after `init()`:
-
-```
-[brainy] Plugin activated: @soulcraft/cor
-[brainy] Providers: 8/10 native (@soulcraft/cor) | default: vector, cache
-```
-
-This tells you at a glance how many subsystems are accelerated and which ones are falling back to JavaScript. The log respects `config.silent`.
-
-### Fail-Fast for Production
-
-Use `requireProviders()` after `init()` to guarantee specific providers are plugin-supplied. This prevents silent fallback to JavaScript in deployments where you expect native acceleration:
-
-```typescript
-const brain = new Brainy()
-await brain.init()
-
-// Throws immediately if any of these are using JS fallback
-brain.requireProviders(['distance', 'embeddings', 'metadataIndex', 'graphIndex'])
-```
-
-If a required provider is missing, the error message tells you exactly what's wrong:
-
-```
-[brainy] Required providers using JS fallback: graphIndex.
-Active plugins: @soulcraft/cor.
-These providers must be supplied by a plugin for this deployment.
-Check plugin installation, license, and native module availability.
-```
-
-This is the recommended pattern for production deployments with paid plugins — fail at startup rather than silently degrading performance.
-
-## Complete Example: Distance Acceleration Plugin
-
-A minimal but useful plugin that provides SIMD-accelerated distance calculations:
-
-```typescript
-// simd-distance-plugin/src/plugin.ts
-import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
-
-// Hypothetical native module
-import { simdCosineDistance } from './native.js'
-
-const simdDistancePlugin: BrainyPlugin = {
-  name: 'brainy-simd-distance',
-
-  async activate(context: BrainyPluginContext): Promise<boolean> {
-    // Check if SIMD is available on this platform
-    if (!checkSimdSupport()) {
-      console.log('[simd-distance] SIMD not available, skipping')
-      return false  // Don't activate — brainy uses JS fallback
-    }
-
-    context.registerProvider('distance', simdCosineDistance)
-    return true
-  }
-}
-
-export default simdDistancePlugin
-```
-
-```json
-// simd-distance-plugin/package.json
-{
-  "name": "brainy-simd-distance",
-  "main": "./dist/plugin.js",
-  "types": "./dist/plugin.d.ts",
-  "peerDependencies": {
-    "@soulcraft/brainy": ">=7.0.0"
-  }
-}
-```
-
-Usage:
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy({ plugins: ['brainy-simd-distance'] })
-await brain.init()
-
-// Verify it's active
-const diag = brain.diagnostics()
-console.log(diag.providers.distance) // { source: 'plugin' }
-```
-
-## Design Principles
-
-1. **Brainy works perfectly without plugins.** Every provider has a JavaScript fallback. Plugins only improve performance or add capabilities.
-
-2. **Provider keys are string-based.** The plugin system is not coupled to any specific plugin. Any package can register any provider.
-
-3. **Clean separation.** Plugins access brainy through the documented `BrainyPluginContext` interface. No direct access to internal classes is needed.
-
-4. **Fail-safe activation.** If a plugin throws during `activate()`, brainy logs a warning and continues with defaults. A broken plugin never prevents brainy from working.
-
-5. **Lifecycle management.** `deactivate()` is called during `brainy.close()` for resource cleanup. Native resources, connections, and file handles should be released here.

docs/PRODUCTION_SERVICE_ARCHITECTURE.md

@@ -1,562 +0,0 @@
-# Production Service Architecture Guide
-
-**How to use Brainy optimally in production services (Bun, Node.js, Deno)**
-
-> **Recommended Runtime:** [Bun](https://bun.sh) provides best performance with Brainy's Candle WASM engine. All examples work with both Bun and Node.js.
-
----
-
-## The Problem: Instance-per-Request Anti-Pattern
-
-### ❌ What NOT to Do
-
-```typescript
-// WRONG - Creates new instance EVERY request
-app.get('/api/entities', async (req, res) => {
-  const brain = new Brainy({ storage: { path: './brainy-data' } })
-  await brain.init()  // FULL INITIALIZATION EVERY TIME!
-  const entities = await brain.find(...)
-  res.json(entities)
-})
-```
-
-### Why This is Terrible
-
-After 40 API calls:
-- **40 Brainy instances** running simultaneously
-- **20GB memory** (40 × 500MB per instance)
-- **2 seconds wasted** (40 × 50ms initialization)
-- **Zero cache benefit** (each instance has its own empty cache)
-- **Index rebuilding** on every request (TypeAware HNSW, LSM-trees, etc.)
-- **Memory leaks** (old instances may not GC properly)
-
----
-
-## ✅ The Solution: Singleton Pattern
-
-**ONE Brainy instance per service, shared across ALL requests.**
-
-### Performance Comparison
-
-| Metric | Instance-per-Request | Singleton (Optimal) |
-|--------|---------------------|---------------------|
-| Memory (40 requests) | 20GB | 500MB |
-| Request 1 latency | 60ms | 60ms (one-time init) |
-| Request 2+ latency | 60ms (no cache!) | 2ms (80% cache hit!) |
-| Cache hit rate | 0% | 80%+ |
-| Speedup | - | **30x faster** |
-
----
-
-## Implementation Patterns
-
-### Pattern 1: Simple Singleton (Recommended)
-
-```typescript
-// server.ts
-import { Brainy } from '@soulcraft/brainy'
-
-// SINGLETON INSTANCE
-let brainInstance: Brainy | null = null
-
-async function getBrain(): Promise<Brainy> {
-  if (brainInstance) {
-    return brainInstance
-  }
-
-  console.log('🧠 Initializing Brainy singleton...')
-
-  brainInstance = new Brainy({
-    storage: {
-      path: './brainy-data',
-      autoOptimize: true
-    },
-    cache: {
-      maxSize: 1000,      // Shared across ALL requests
-      ttl: 3600000,       // 1 hour
-      enableMetrics: true
-    },
-    augmentations: {
-      include: ['cache', 'metrics', 'display', 'vfs']
-    }
-  })
-
-  await brainInstance.init()
-  console.log('✅ Brainy ready')
-
-  return brainInstance
-}
-
-// Initialize BEFORE starting server
-async function startServer() {
-  await getBrain()  // One-time initialization
-
-  app.get('/api/entities', async (req, res) => {
-    const brain = await getBrain()  // Reuses same instance!
-    const entities = await brain.find(req.query)
-    res.json(entities)
-  })
-
-  app.listen(3000)
-}
-
-startServer()
-```
-
-**Benefits:**
-- ✅ Simple to implement
-- ✅ Thread-safe (async initialization)
-- ✅ Shared cache and indexes
-- ✅ 40x memory reduction
-
----
-
-### Pattern 2: Service Class (Production-Grade)
-
-```typescript
-// services/BrainService.ts
-export class BrainService {
-  private brain: Brainy | null = null
-  private initPromise: Promise<Brainy> | null = null
-
-  async getInstance(): Promise<Brainy> {
-    if (this.brain) return this.brain
-    if (this.initPromise) return this.initPromise
-
-    this.initPromise = this.initialize()
-    return this.initPromise
-  }
-
-  private async initialize(): Promise<Brainy> {
-    this.brain = new Brainy({
-      storage: {
-        path: process.env.BRAINY_DATA_PATH || './brainy-data'
-      },
-      cache: { maxSize: 1000, ttl: 3600000 }
-    })
-    await this.brain.init()
-    return this.brain
-  }
-
-  async shutdown(): Promise<void> {
-    if (this.brain) {
-      // Cleanup if needed
-      this.brain = null
-    }
-  }
-}
-
-// server.ts
-const brainService = new BrainService()
-
-app.get('/api/entities', async (req, res) => {
-  const brain = await brainService.getInstance()
-  const entities = await brain.find(req.query)
-  res.json(entities)
-})
-
-// Graceful shutdown
-process.on('SIGTERM', async () => {
-  await brainService.shutdown()
-  process.exit(0)
-})
-```
-
-**Benefits:**
-- ✅ Prevents race conditions (multiple simultaneous inits)
-- ✅ Testable (can inject mock)
-- ✅ Clean shutdown handling
-- ✅ Environment-configurable
-
----
-
-### Pattern 3: Bun Server (Recommended)
-
-```typescript
-// server.ts - Clean Bun implementation
-import { Brainy } from '@soulcraft/brainy'
-
-let brain: Brainy | null = null
-
-async function getBrain(): Promise<Brainy> {
-  if (!brain) {
-    brain = new Brainy({ storage: { path: './brainy-data' } })
-    await brain.init()
-  }
-  return brain
-}
-
-// Initialize before server starts
-await getBrain()
-
-Bun.serve({
-  port: 3000,
-  async fetch(req) {
-    const url = new URL(req.url)
-
-    if (url.pathname === '/api/entities') {
-      const b = await getBrain()
-      const entities = await b.find({})
-      return Response.json(entities)
-    }
-
-    if (url.pathname === '/api/entity' && req.method === 'POST') {
-      const b = await getBrain()
-      const body = await req.json()
-      const id = await b.add(body)
-      return Response.json({ id })
-    }
-
-    return new Response('Not Found', { status: 404 })
-  }
-})
-
-console.log('Server running on http://localhost:3000')
-```
-
-**Benefits:**
-- ✅ Native Bun runtime performance
-- ✅ No framework dependencies
-- ✅ Pure WASM — no native binaries, bundler-friendly
-- ✅ Built-in TypeScript support
-
-### Pattern 4: Express/Node.js Middleware (Legacy)
-
-```typescript
-// middleware/brainy.ts
-let brainInstance: Brainy | null = null
-
-export async function initBrainy() {
-  if (!brainInstance) {
-    brainInstance = new Brainy({ storage: { path: './brainy-data' } })
-    await brainInstance.init()
-  }
-}
-
-export function brainMiddleware(req, res, next) {
-  if (!brainInstance) {
-    return res.status(500).json({ error: 'Brainy not initialized' })
-  }
-  req.brain = brainInstance  // Attach to request
-  next()
-}
-
-// Type extension
-declare global {
-  namespace Express {
-    interface Request {
-      brain: Brainy
-    }
-  }
-}
-
-// server.ts
-import { initBrainy, brainMiddleware } from './middleware/brainy'
-
-async function startServer() {
-  await initBrainy()  // Initialize first
-
-  app.use('/api', brainMiddleware)  // Apply to API routes
-
-  app.get('/api/entities', async (req, res) => {
-    const entities = await req.brain.find(req.query)  // Type-safe!
-    res.json(entities)
-  })
-
-  app.listen(3000)
-}
-```
-
-**Benefits:**
-- ✅ Clean separation of concerns
-- ✅ Type-safe (`req.brain` is typed)
-- ✅ Easy to add auth/validation
-
----
-
-## Optimization Strategies
-
-### 1. Configure Cache for Your Workload
-
-```typescript
-const brain = new Brainy({
-  cache: {
-    maxSize: 1000,           // Number of entities to cache
-    ttl: 3600000,            // Cache lifetime (1 hour)
-    enableMetrics: true,     // Track hit rate
-    evictionPolicy: 'lru'    // Least recently used
-  }
-})
-```
-
-**Cache sizing:**
-- Small service (< 100 req/min): `maxSize: 500`
-- Medium service (< 1000 req/min): `maxSize: 1000`
-- Large service (> 1000 req/min): `maxSize: 5000`
-
-### 2. Lazy Load Augmentations
-
-```typescript
-const brain = new Brainy({
-  augmentations: {
-    // Only load what you actually use
-    include: ['cache', 'metrics', 'display', 'vfs'],
-    exclude: ['neuralImport', 'intelligentImport']  // Skip heavy features
-  }
-})
-```
-
-**Memory savings:**
-- With all augmentations: ~800MB
-- With minimal set: ~400MB
-
-### 3. Warm Up Indexes
-
-```typescript
-async function startServer() {
-  const brain = await getBrain()
-
-  // Pre-warm frequently-used indexes
-  await brain.find({ type: 'person', limit: 1 })
-  await brain.find({ type: 'organization', limit: 1 })
-
-  console.log('✅ Indexes pre-warmed')
-
-  app.listen(3000)
-}
-```
-
-**Benefit:** First requests are fast (no cold-start index building)
-
-### 4. Memory-Aware Configuration
-
-```typescript
-import os from 'os'
-
-const totalMemory = os.totalmem()
-const availableMemory = os.freemem()
-
-const brain = new Brainy({
-  cache: {
-    // Use 10% of total RAM for cache
-    maxSize: Math.floor(totalMemory * 0.1 / (1024 * 1024))
-  },
-  indexes: {
-    // Lazy load indexes if low memory
-    lazyLoad: availableMemory < totalMemory * 0.5,
-    preload: ['person', 'organization']  // Only preload common types
-  }
-})
-```
-
----
-
-## Concurrency & Thread Safety
-
-Brainy is **designed** for concurrent access. A single instance can handle:
-
-```typescript
-// Multiple concurrent requests - all using same instance
-app.get('/api/read/:id', async (req, res) => {
-  const brain = getBrain()
-  const entity = await brain.get(req.params.id)  // Safe - no state mutation
-  res.json(entity)
-})
-
-app.post('/api/write', async (req, res) => {
-  const brain = getBrain()
-  const id = await brain.add(req.body)  // Safe - internal locking
-  res.json({ id })
-})
-```
-
-**Concurrency mechanisms:**
-- ✅ **Read operations**: Lock-free (MVCC)
-- ✅ **Write operations**: Internal write-ahead logging (WAL)
-- ✅ **Cache**: Thread-safe LRU implementation
-- ✅ **Indexes**: Concurrent reads, locked writes
-
----
-
-## Production Checklist
-
-### Before Deploying
-
-- [ ] **Initialize Brainy on startup** (not per-request)
-- [ ] **Configure cache size** based on memory
-- [ ] **Only load needed augmentations**
-- [ ] **Warm up critical indexes**
-- [ ] **Add graceful shutdown handler**
-- [ ] **Monitor cache hit rate**
-
-### Code Review Checklist
-
-```typescript
-// ❌ BAD - Instance per request
-app.get('/api/route', async (req, res) => {
-  const brain = new Brainy(...)  // RED FLAG!
-  await brain.init()             // RED FLAG!
-})
-
-// ✅ GOOD - Singleton pattern
-app.get('/api/route', async (req, res) => {
-  const brain = await getBrain()  // Reuses instance ✓
-})
-```
-
----
-
-## Monitoring & Metrics
-
-```typescript
-// Add metrics endpoint
-app.get('/api/metrics', (req, res) => {
-  const brain = getBrain()
-
-  res.json({
-    cache: {
-      size: brain.cache?.size || 0,
-      maxSize: brain.cache?.maxSize || 0,
-      hitRate: brain.metrics?.cacheHitRate || 0  // Target: >70%
-    },
-    storage: brain.storage.getStats(),
-    memory: {
-      heapUsed: Math.round(process.memoryUsage().heapUsed / 1024 / 1024),
-      heapTotal: Math.round(process.memoryUsage().heapTotal / 1024 / 1024)
-    }
-  })
-})
-```
-
-**Key metrics to track:**
-- **Cache hit rate**: Should be >70% after warm-up
-- **Memory usage**: Should stay constant (~500MB for singleton)
-- **Request latency**: Should be <10ms for cached entities
-
----
-
-## Common Pitfalls
-
-### 1. Creating instances in routes
-```typescript
-// ❌ NEVER do this
-app.get('/api/entities', async (req, res) => {
-  const brain = new Brainy(...)  // Creates new instance every time!
-})
-```
-
-### 2. Not awaiting initialization
-```typescript
-// ❌ Race condition - server starts before Brainy ready
-app.listen(3000)
-getBrain()  // Async init happens AFTER server starts!
-
-// ✅ Correct - wait for init
-await getBrain()
-app.listen(3000)
-```
-
-### 3. Multiple instances for different purposes
-```typescript
-// ❌ Wasteful - creates 2 instances
-const readBrain = new Brainy(...)
-const writeBrain = new Brainy(...)
-
-// ✅ One instance handles both
-const brain = new Brainy(...)
-await brain.get(id)    // Read
-await brain.add(data)  // Write
-```
-
----
-
-## Migration Guide
-
-### Current (Anti-Pattern)
-```typescript
-// Probably in multiple route files
-async function handler(req, res) {
-  const brain = new Brainy({ storage: { path: './brainy-data' } })
-  await brain.init()
-  // ... use brain
-}
-```
-
-### Step 1: Create Singleton Module
-```typescript
-// lib/brainy.ts
-let instance: Brainy | null = null
-
-export async function getBrain(): Promise<Brainy> {
-  if (!instance) {
-    instance = new Brainy({ storage: { path: './brainy-data' } })
-    await instance.init()
-  }
-  return instance
-}
-```
-
-### Step 2: Update Server Startup
-```typescript
-// server.ts
-import { getBrain } from './lib/brainy'
-
-async function startServer() {
-  // Initialize Brainy FIRST
-  await getBrain()
-  console.log('✅ Brainy initialized')
-
-  // THEN start server
-  app.listen(3000)
-}
-```
-
-### Step 3: Update All Routes
-```typescript
-// Before
-async function handler(req, res) {
-  const brain = new Brainy(...)  // Remove this
-  await brain.init()              // Remove this
-
-  // ... rest of code
-}
-
-// After
-import { getBrain } from './lib/brainy'
-
-async function handler(req, res) {
-  const brain = await getBrain()  // Add this
-
-  // ... rest of code stays same
-}
-```
-
-**Expected results:**
-- ✅ 40x memory reduction (20GB → 500MB)
-- ✅ 30x faster requests (60ms → 2ms average)
-- ✅ 80%+ cache hit rate
-- ✅ Your service can scale to 1000s of requests/minute
-
----
-
-## Summary
-
-**DO:**
-- ✅ Initialize Brainy ONCE on server startup
-- ✅ Share single instance across all requests
-- ✅ Configure cache for your workload
-- ✅ Monitor cache hit rate
-- ✅ Handle graceful shutdown
-
-**DON'T:**
-- ❌ Create new Brainy instance per request
-- ❌ Create multiple instances
-- ❌ Start server before Brainy is initialized
-- ❌ Load augmentations you don't use
-
-**Result:** 40x less memory, 30x faster requests, Brainy optimizations actually work!
-
----
-
-**Questions? Issues?**
-- Report issues: https://github.com/soulcraftlabs/brainy/issues

docs/QUERY_OPERATORS.md

@@ -1,298 +0,0 @@
-# Query Operators (BFO)
-
-> Brainy Field Operators — the complete reference for `where` filters in `find()`.
-
-All operators work with `find({ where: { ... } })` and filter on **metadata fields** (not `data`).
-
----
-
-## Equality
-
-| Operator | Alias | Description | Example |
-|----------|-------|-------------|---------|
-| `eq` | `equals` | Exact match | `{ status: { eq: 'active' } }` |
-| `ne` | `notEquals` | Not equal | `{ status: { ne: 'deleted' } }` |
-
-**Shorthand:** A bare value is treated as `equals`:
-
-```typescript
-// These are equivalent:
-brain.find({ where: { status: 'active' } })
-brain.find({ where: { status: { equals: 'active' } } })
-```
-
----
-
-## Comparison
-
-| Operator | Alias | Description | Example |
-|----------|-------|-------------|---------|
-| `gt` | `greaterThan` | Greater than | `{ age: { gt: 18 } }` |
-| `gte` | `greaterThanOrEqual` | Greater or equal | `{ score: { gte: 90 } }` |
-| `lt` | `lessThan` | Less than | `{ price: { lt: 100 } }` |
-| `lte` | `lessThanOrEqual` | Less or equal | `{ rating: { lte: 3 } }` |
-| `between` | — | Inclusive range `[min, max]` | `{ year: { between: [2020, 2025] } }` |
-
-```typescript
-// Range query
-const recent = await brain.find({
-  where: {
-    createdAt: { between: [Date.now() - 86400000, Date.now()] }
-  }
-})
-```
-
----
-
-## Array / Set
-
-| Operator | Alias | Description | Example |
-|----------|-------|-------------|---------|
-| `oneOf` | `in` | Value is one of the given options | `{ color: { oneOf: ['red', 'blue'] } }` |
-| `noneOf` | — | Value is NOT one of the given options | `{ status: { noneOf: ['deleted', 'archived'] } }` |
-| `contains` | — | Array field contains value | `{ tags: { contains: 'ai' } }` |
-| `excludes` | — | Array field does NOT contain value | `{ tags: { excludes: 'spam' } }` |
-| `hasAll` | — | Array field contains ALL listed values | `{ skills: { hasAll: ['js', 'ts'] } }` |
-
-```typescript
-// Find entities tagged with 'ai'
-const aiEntities = await brain.find({
-  where: { tags: { contains: 'ai' } }
-})
-
-// Find entities of specific types
-const people = await brain.find({
-  where: { noun: { oneOf: ['Person', 'Agent'] } }
-})
-```
-
----
-
-## Existence
-
-| Operator | Description | Example |
-|----------|-------------|---------|
-| `exists: true` | Field exists (has any value) | `{ email: { exists: true } }` |
-| `exists: false` | Field does NOT exist | `{ email: { exists: false } }` |
-| `missing: true` | Field does NOT exist (alias for `exists: false`) | `{ email: { missing: true } }` |
-| `missing: false` | Field exists (alias for `exists: true`) | `{ email: { missing: false } }` |
-
-```typescript
-// Find entities that have an email field
-const withEmail = await brain.find({
-  where: { email: { exists: true } }
-})
-```
-
----
-
-## Pattern (In-Memory Only)
-
-These operators work via the in-memory filter path. They are applied **after** the indexed query, so use them with other indexed operators for best performance.
-
-| Operator | Description | Example |
-|----------|-------------|---------|
-| `matches` | Regex or string pattern match | `{ name: { matches: /^Dr\./ } }` |
-| `startsWith` | String prefix | `{ name: { startsWith: 'John' } }` |
-| `endsWith` | String suffix | `{ email: { endsWith: '@gmail.com' } }` |
-
-```typescript
-const doctors = await brain.find({
-  where: {
-    type: NounType.Person,           // Indexed — fast
-    name: { startsWith: 'Dr.' }     // In-memory — applied after
-  }
-})
-```
-
----
-
-## Logical
-
-Combine multiple conditions:
-
-| Operator | Description | Example |
-|----------|-------------|---------|
-| `allOf` | ALL sub-filters must match (AND) | `{ allOf: [{ status: 'active' }, { role: 'admin' }] }` |
-| `anyOf` | ANY sub-filter must match (OR) | `{ anyOf: [{ role: 'admin' }, { role: 'owner' }] }` |
-| `not` | Invert a filter | `{ not: { status: 'deleted' } }` |
-
-```typescript
-// Complex OR query
-const adminsOrOwners = await brain.find({
-  where: {
-    anyOf: [
-      { role: 'admin' },
-      { role: 'owner' }
-    ]
-  }
-})
-
-// NOT query
-const notDeleted = await brain.find({
-  where: {
-    not: { status: 'deleted' }
-  }
-})
-
-// Combined AND + OR
-const results = await brain.find({
-  where: {
-    allOf: [
-      { department: 'engineering' },
-      { anyOf: [
-        { level: 'senior' },
-        { yearsExperience: { greaterThan: 5 } }
-      ]}
-    ]
-  }
-})
-```
-
----
-
-## Indexed vs In-Memory Operators
-
-Brainy's MetadataIndex supports a subset of operators natively for O(1) field lookups. Other operators fall back to in-memory filtering.
-
-| Operator | MetadataIndex (Indexed) | In-Memory Fallback |
-|----------|:-----------------------:|:------------------:|
-| `equals` / `eq` | Yes | Yes |
-| `notEquals` / `ne` | — | Yes |
-| `greaterThan` / `gt` | Yes | Yes |
-| `greaterThanOrEqual` / `gte` | Yes | Yes |
-| `lessThan` / `lt` | Yes | Yes |
-| `lessThanOrEqual` / `lte` | Yes | Yes |
-| `between` | Yes | Yes |
-| `oneOf` / `in` | Yes | Yes |
-| `noneOf` | — | Yes |
-| `contains` | Yes | Yes |
-| `exists` / `missing` | Yes | Yes |
-| `matches` | — | Yes |
-| `startsWith` | — | Yes |
-| `endsWith` | — | Yes |
-| `allOf` | Partial | Yes |
-| `anyOf` | Partial | Yes |
-| `not` | — | Yes |
-
-**Performance tip:** Combine indexed operators (equals, greaterThan, oneOf, between, contains, exists) with pattern operators for optimal speed — the index narrows results first, then patterns filter in memory.
-
----
-
-## Practical Examples
-
-### Filter by entity type
-
-```typescript
-// Using the type shorthand (recommended)
-brain.find({ type: NounType.Person })
-
-// Using where.noun directly
-brain.find({ where: { noun: NounType.Person } })
-
-// Multiple types
-brain.find({ type: [NounType.Person, NounType.Agent] })
-```
-
-### Filter by subtype
-
-`subtype` is a top-level standard field — takes the column-store fast path, not the metadata fallback. Pair with `type` for the typical "Person who is an employee" query:
-
-```typescript
-// Equality on subtype:
-brain.find({ type: NounType.Person, subtype: 'employee' })
-
-// Set membership:
-brain.find({ type: NounType.Person, subtype: ['employee', 'contractor'] })
-
-// Operator-form predicates use `where`:
-brain.find({
-  type: NounType.Person,
-  where: { subtype: { exists: true } }
-})
-```
-
-See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the full surface.
-
-### Filter relationships by subtype (7.30+)
-
-Verbs are first-class peers — `related()` and graph traversal both honor subtype filters on the fast path:
-
-```typescript
-// Filter relationships by VerbType subtype
-const direct = await brain.related({
-  from: ceoId,
-  type: VerbType.ReportsTo,
-  subtype: 'direct'
-})
-
-// Set membership on verb subtype
-const all = await brain.related({
-  from: ceoId,
-  type: VerbType.ReportsTo,
-  subtype: ['direct', 'dotted-line']
-})
-
-// Graph traversal — subtype filters traversal edges (depth-1 in 7.30 JS path;
-// multi-hop subtype filtering lands on Cor native)
-const reports = await brain.find({
-  connected: {
-    from: ceoId,
-    via: VerbType.ReportsTo,
-    subtype: 'direct',
-    depth: 1
-  }
-})
-```
-
-### Combine semantic search with filters
-
-```typescript
-const results = await brain.find({
-  query: 'machine learning engineer',     // Semantic search (on data)
-  type: NounType.Person,                  // Type filter (indexed)
-  where: {
-    department: 'engineering',            // Exact match (indexed)
-    yearsExperience: { greaterThan: 3 }   // Range filter (indexed)
-  },
-  limit: 10
-})
-```
-
-### Temporal queries
-
-```typescript
-const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000
-const recentEntities = await brain.find({
-  where: {
-    createdAt: { greaterThan: lastWeek }
-  },
-  orderBy: 'createdAt',
-  order: 'desc',
-  limit: 50
-})
-```
-
-### Graph + metadata combination
-
-```typescript
-const results = await brain.find({
-  connected: {
-    from: teamLeadId,
-    via: VerbType.WorksWith,
-    depth: 2
-  },
-  where: {
-    role: { oneOf: ['engineer', 'designer'] },
-    active: true
-  }
-})
-```
-
----
-
-## See Also
-
-- [Data Model](./DATA_MODEL.md) — Entity structure, data vs metadata
-- [API Reference](./api/README.md) — Complete API documentation
-- [Find System](./FIND_SYSTEM.md) — Natural language find() details

docs/README.md

@@ -1,130 +0,0 @@
-# Brainy Documentation
-
-> The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API.
-
-## Quick Start
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// Add entities — data is embedded for semantic search, metadata is indexed for filtering
-const id = await brain.add({
-  data: 'Revolutionary AI Breakthrough',
-  type: NounType.Document,
-  metadata: { category: 'technology', rating: 4.8 }
-})
-
-// Search with Triple Intelligence
-const results = await brain.find({
-  query: 'artificial intelligence',              // Semantic search (on data)
-  where: { rating: { greaterThan: 4.0 } },       // Metadata filter
-  connected: { from: authorId, depth: 2 }         // Graph traversal
-})
-```
-
----
-
-## Core Documentation
-
-| Document | Description |
-|----------|-------------|
-| **[API Reference](./api/README.md)** | Complete API documentation — **start here** |
-| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
-| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
-| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
-| [Consistency Model](./concepts/consistency-model.md) | The Db API guarantees — snapshot isolation, atomic transactions, time travel |
-
----
-
-## Architecture
-
-| Document | Description |
-|----------|-------------|
-| [Architecture Overview](./architecture/overview.md) | High-level system design |
-| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Metadata unified query |
-| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | 42 nouns + 127 verbs type system |
-| [Stage 3 Canonical Taxonomy](./STAGE3-CANONICAL-TAXONOMY.md) | Complete type reference |
-| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
-| [Index Architecture](./architecture/index-architecture.md) | Vector, Graph, and Metadata indexing |
-| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
-
----
-
-## Virtual Filesystem (VFS)
-
-| Document | Description |
-|----------|-------------|
-| [VFS Quick Start](./vfs/QUICK_START.md) | Get started in 30 seconds |
-| [VFS Core](./vfs/VFS_CORE.md) | Core concepts and architecture |
-| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
-| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
-
-See [vfs/](./vfs/) for the complete VFS documentation set.
-
----
-
-## Guides
-
-| Document | Description |
-|----------|-------------|
-| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
-| [Snapshots & Time Travel](./guides/snapshots-and-time-travel.md) | Backups, restore, what-if analysis, audit trails |
-| [Natural Language](./guides/natural-language.md) | Query in plain English |
-| [Neural API](./guides/neural-api.md) | AI-powered features |
-| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
-| [Framework Integration](./guides/framework-integration.md) | React, Vue, Angular, Svelte |
-
----
-
-## Storage & Deployment
-
-| Document | Description |
-|----------|-------------|
-| [Storage Architecture](./architecture/storage-architecture.md) | Filesystem and memory adapters, on-disk artifact layout, operator-layer backup |
-| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities |
-
----
-
-## Plugins
-
-| Document | Description |
-|----------|-------------|
-| [Plugins](./PLUGINS.md) | Plugin system overview — providers, `plugins` config, `brain.use()` |
-
----
-
-## Performance & Scaling
-
-| Document | Description |
-|----------|-------------|
-| [Performance](./PERFORMANCE.md) | Optimization techniques |
-| [Scaling](./SCALING.md) | Scale to billions of entities |
-| [Batching](./BATCHING.md) | Batch operations guide |
-
----
-
-## Migration & Reference
-
-| Document | Description |
-|----------|-------------|
-| [v3 to v4 Migration](./MIGRATION-V3-TO-V4.md) | Upgrade guide |
-| [Release Guide](./RELEASE-GUIDE.md) | How to release new versions |
-| [Production Architecture](./PRODUCTION_SERVICE_ARCHITECTURE.md) | Ops reference |
-
----
-
-## Internal
-
-| Document | Description |
-|----------|-------------|
-| [Audit Report](./internal/AUDIT_REPORT.md) | Feature audit |
-| [Honest Status](./internal/HONEST_STATUS.md) | Actual implementation status |
-
----
-
-## License
-
-Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.

docs/RELEASE-GUIDE.md

@@ -1,131 +0,0 @@
-# Brainy Release Guide
-
-## Standard Semantic Versioning (Industry Guidelines)
-
-### Official SemVer 2.0.0 says:
-- **MAJOR**: Incompatible API changes (breaking changes)
-- **MINOR**: Add functionality in backwards compatible manner
-- **PATCH**: Backwards compatible bug fixes
-
-## Our Approach for Brainy (More Conservative)
-
-### We intentionally diverge from strict SemVer:
-- **PATCH (2.3.0 → 2.3.1)**: Bug fixes, internal improvements, dependency updates
-- **MINOR (2.3.0 → 2.4.0)**: New features, API changes, enhancements
-- **MAJOR (3.0.0)**: Reserved for strategic platform shifts (manual decision)
-
-### Why We Do This:
-1. **User Trust**: Major versions signal huge changes and scare users
-2. **Adoption**: People hesitate to upgrade major versions
-3. **Flexibility**: We can evolve the API without version explosion
-4. **Industry Practice**: Many successful projects (React, Vue) do this
-
-## CRITICAL: Never Use "BREAKING CHANGE" 
-
-**"BREAKING CHANGE" in commits = Automatic major version = BAD!**
-- Even if removing methods, just use `feat:` or `refactor:`
-- Major versions are MANUAL decisions: `npm run release:major`
-- Most API changes can be handled gracefully in minor versions
-
-## Commit Message Guidelines
-
-### ✅ CORRECT Examples:
-```bash
-# New features → MINOR bump
-git commit -m "feat: add new model delivery system"
-
-# Bug fixes → PATCH bump
-git commit -m "fix: resolve model download timeout"
-
-# Internal improvements → PATCH bump
-git commit -m "refactor: simplify model manager logic"
-git commit -m "perf: optimize model caching"
-git commit -m "chore: remove unused dependency"
-```
-
-### ❌ AVOID These Mistakes:
-```bash
-# DON'T use BREAKING CHANGE for internal changes
-git commit -m "feat: improve model delivery
-
-BREAKING CHANGE: removed tar-stream dependency"  # WRONG! This triggers
-```
-
-## Release Workflow Checklist
-
-### Before Committing:
-- [ ] Review commit message - no "BREAKING CHANGE" unless API changes
-- [ ] Consider: Will users need to change their code? If NO → Not breaking
-
-### Release Commands:
-```bash
-# Let standard-version figure it out from commits
-npm run release              # Recommended - auto-detects version
-
-# Or be explicit:
-npm run release:patch        # 2.4.0 → 2.4.1 (fixes)
-npm run release:minor        # 2.4.0 → 2.5.0 (features)
-npm run release:major        # 2.4.0 → 3.0.0 (API changes only!)
-```
-
-### After Release:
-```bash
-git push --follow-tags origin main
-npm publish
-gh release create $(git describe --tags --abbrev=0) --generate-notes
-```
-
-## When to Use Major Version (3.0.0)
-
-ONLY when we make changes like:
-- Removing methods from the public API
-- Changing method signatures (parameters, return types)
-- Renaming public methods
-- Changing default behaviors that break existing code
-
-Examples:
-- ❌ `search(query, limit, options)` → `search(query, options)` (major)
-- ✅ Adding `find()` method (minor - doesn't break existing code)
-- ✅ Internal refactoring (patch - users don't see it)
-
-## Quick Decision Tree
-
-1. **Does this fix a bug?** → PATCH (fix:)
-2. **Does this add new functionality?** → MINOR (feat:)
-3. **Will users' existing code break?** → MAJOR (with BREAKING CHANGE)
-4. **Is it internal/maintenance?** → PATCH (chore:/refactor:/perf:)
-
-## Emergency: If Wrong Version is Released
-
-```bash
-# 1. Deprecate wrong version on npm
-npm deprecate @soulcraft/brainy@X.X.X "Incorrect version - use Y.Y.Y"
-
-# 2. Fix version in package.json
-# 3. Republish correct version
-npm publish
-
-# 4. Delete wrong GitHub tag/release
-git push origin :vX.X.X
-gh release delete vX.X.X --yes
-
-# 5. Create correct tag/release
-git tag vY.Y.Y
-git push --tags
-gh release create vY.Y.Y --generate-notes
-```
-
-## Remember:
-- **Most releases should be MINOR or PATCH**
-- **Major versions should be RARE**
-- **When in doubt, it's probably MINOR**
-- **NEVER use "BREAKING CHANGE" for internal changes**
-## Hard Ordering Constraints (check before EVERY release)
-
-- **Embedding model changes are SEQUENCED, not free.** No release may change the
-  embedding model (or its quantization/dimensions) before **vector model-version
-  stamping + hard-error-on-mismatch** ships. Stored vectors carry no model version
-  today; mixing vectors from two models silently corrupts every similarity
-  comparison. If a model bump is ever proposed, the stamping work moves ahead of
-  it in the schedule — coordinate with the native provider so both engines stamp
-  and enforce identically. (Registered with the native-provider team 2026-07-07.)

docs/SCALING.md

@@ -1,239 +0,0 @@
-# Brainy Scaling Guide
-
-> **One Line Summary**: Single-node by design — Brainy scales by getting the most out of one machine plus operator-layer backup.
-
-## Table of Contents
-- [Quick Start](#quick-start)
-- [How Brainy Scales](#how-brainy-scales)
-- [Storage Configurations](#storage-configurations)
-- [Scaling Patterns](#scaling-patterns)
-- [Real World Examples](#real-world-examples)
-
-## Quick Start
-
-### In-Memory
-```typescript
-import Brainy from '@soulcraft/brainy'
-const brain = new Brainy({ storage: { type: 'memory' } })
-```
-
-### On-Disk (Default for Node)
-```typescript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: './brainy-data' }
-})
-```
-
-## How Brainy Scales
-
-Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery, no S3 coordination. Scaling means:
-
-- **Up**: give the process more RAM, CPU, and IOPS
-- **Out**: stand up multiple independent Brainy instances behind your own service layer
-- **Cold storage**: snapshot the on-disk artifact off-site so you can rehydrate elsewhere
-
-The three knobs that matter most:
-
-1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
-2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
-
-The native vector provider (via the optional `@soulcraft/cor` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
-
-## Measured Performance
-
-Numbers below are **measured** by `tests/benchmarks/find-composition-scale.js` (a single
-Node 22 process, in-memory storage, 384-dim vectors, `balanced` recall). They are the
-open-core (pure-TypeScript) path — what you get from `@soulcraft/brainy` with no native
-provider installed. Run it yourself: `node --max-old-space-size=8192 tests/benchmarks/find-composition-scale.js 100000`.
-
-`find()` query latency, p50 / p95 (200 queries each):
-
-| Query | 5,000 entities | 100,000 entities |
-|---|---|---|
-| Vector similarity (`{ vector }`) | 0.8 / 1.3 ms | 1.4 / 4.7 ms |
-| Graph 1-hop (`{ connected }`) | 0.5 / 0.7 ms | 0.7 / 0.8 ms |
-| Metadata filter (`{ where }`, low-selectivity) | 0.7 / 1.2 ms | 23.5 / 30.1 ms |
-| Vector + metadata | 7.7 / 8.3 ms | 78.8 / 93.8 ms |
-
-What the shape tells you:
-
-- **Vector and graph lookups scale ~logarithmically** — they barely move from 5k to 100k,
-  because HNSW search is ~O(ef·log n) and graph adjacency is O(degree).
-- **Metadata-filtered paths scale with the size of the match set, not the database.** The
-  benchmark's `category` filter matches ~10% of rows (10,000 at 100k); the cost is
-  materializing that candidate set and running the vector search *inside* it (`find()` does
-  metadata-first hard filtering, then ranks within the candidates — see
-  [How find works](./FIND_SYSTEM.md)). A **high-selectivity** filter (few matches) is far
-  cheaper; a 10%-of-everything filter is the worst case. This candidate-restricted search is
-  precisely the path the native provider accelerates (Rust roaring-bitmap candidate
-  intersection).
-- **Composition is correct, not lossy.** Combining vector + metadata + graph returns exactly
-  the entities satisfying all constraints — verified by
-  `tests/integration/find-triple-composition.test.ts`.
-
-Memory: ~62 KB resident per entity at 100k (6.2 GB RSS for 100k × 384-dim including the HNSW
-graph, metadata index, and 100k edges).
-
-**Scale ceiling (open-core).** The pure-JS HNSW *build* cost (~100 inserts/s at 384-dim on
-one core) makes the in-process open-core path most appropriate up to ~10⁵–10⁶ entities.
-*Query* latency stays low well beyond that, but for the 10⁸–10¹⁰ regime install the native
-provider (`@soulcraft/cor`, on-disk DiskANN) — same API, no code change. _Projected from
-the two measured points, vector p50 at 1M is ~2 ms; metadata-heavy composition grows with
-match-set size and is the path to move onto the native provider first._
-
-## Storage Configurations
-
-### Filesystem (Recommended for Production)
-```typescript
-const brain = new Brainy({
-  storage: {
-    type: 'filesystem',
-    path: '/var/lib/brainy'
-  }
-})
-```
-- Stores everything in a sharded JSON tree under `path`
-- Atomic writes via rename
-- Survives process restarts
-- Snapshot it off-site with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` from your scheduler
-
-### Memory
-```typescript
-const brain = new Brainy({ storage: { type: 'memory' } })
-```
-- Zero I/O, fastest possible
-- No persistence — process exit discards everything
-- Use for tests and ephemeral caches
-
-### Auto
-```typescript
-const brain = new Brainy({
-  storage: { type: 'auto', path: './data' }
-})
-```
-- Picks `filesystem` when running on Node with a writable `path`
-- Falls back to `memory` otherwise
-
-## Scaling Patterns
-
-### Stage 1: Prototype (Memory)
-```typescript
-const brain = new Brainy({ storage: { type: 'memory' } })
-// Development, tests, <100K items
-```
-
-### Stage 2: Production (Filesystem)
-```typescript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' }
-})
-// Most production workloads up to ~10M entities on a single host
-```
-
-### Stage 3: Higher Throughput (Tune the Vector Index)
-```typescript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' },
-  vector: {
-    recall: 'fast',                // Trade recall for latency
-    persistMode: 'deferred'        // Batch persistence
-  }
-})
-```
-
-### Stage 4: Multi-Instance (Operator-Layer)
-Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `path`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes.
-
-## Real World Examples
-
-### Example 1: Single-Node App With Backup
-```typescript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' }
-})
-```
-Schedule (cron / systemd timer):
-```bash
-*/15 * * * *  rclone sync /var/lib/brainy remote:brainy-backup
-```
-
-### Example 2: Tests
-```typescript
-const brain = new Brainy({ storage: { type: 'memory' } })
-// Fast, no cleanup needed between runs
-```
-
-### Example 3: Multi-Tenant Service
-Spin up one Brainy instance per tenant, each in its own directory:
-```typescript
-function brainForTenant(tenantId: string) {
-  return new Brainy({
-    storage: {
-      type: 'filesystem',
-      path: `/var/lib/brainy/${tenantId}`
-    }
-  })
-}
-```
-Your service layer handles routing and isolation; Brainy stays simple.
-
-### Example 4: Higher Recall at Scale
-```typescript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' },
-  vector: {
-    recall: 'accurate'
-  }
-})
-```
-
-## Tuning Knobs Summary
-
-| Setting | Values | When to change |
-|---------|--------|----------------|
-| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
-| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
-| `storage.cache.maxSize` | integer | Hot-path read cache size |
-| `storage.cache.ttl` | ms | Cache freshness |
-
-## Monitoring & Observability
-
-```typescript
-const stats = await brain.stats()
-// {
-//   nounCount: 50000,
-//   verbCount: 80000,
-//   vectorIndex: { ... },
-//   storage: { used: '45GB' }
-// }
-```
-
-## Troubleshooting
-
-### Issue: Slow queries
-1. Switch to `vector.recall: 'fast'`
-2. Increase the read cache (`storage.cache.maxSize`)
-3. Consider the optional native vector provider via `@soulcraft/cor`
-
-### Issue: Memory pressure
-1. Reduce `storage.cache.maxSize`
-2. Move to `vector.persistMode: 'deferred'` to batch writes
-3. Consider the optional native vector provider via `@soulcraft/cor` for at-scale index acceleration
-
-### Issue: Slow startup after a crash
-1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
-2. Verify backup integrity periodically
-
-## Best Practices
-
-1. **One process = one `path`** — never share a directory between processes
-2. **Snapshot from your scheduler** — Brainy doesn't ship cloud SDKs; use `rclone` / `aws s3 sync` / `gsutil`
-3. **Profile before tuning** — `recall: 'balanced'` is right for most workloads
-4. **Install the native vector provider only when measured profiling shows it pays off**
-
-## Summary
-
-- Brainy 8.0 is a **library**, not a cluster
-- Storage adapters: `filesystem`, `memory`, `auto`
-- Vector tuning: `recall`, `persistMode`
-- Backup is an operator-layer concern — snapshot `path`

docs/STAGE3-CANONICAL-TAXONOMY.md

@@ -1,373 +0,0 @@
-# Brainy Stage 3: Canonical Taxonomy
-
-**Status:** FINAL - This is the definitive, timeless taxonomy
-**Total Types:** 169 (42 nouns + 127 verbs)
-**Coverage:** 96-97% of all human knowledge
-**Designed to last:** 20+ years without changes
-
----
-
-## Summary
-
-- **Nouns:** 42 types
-- **Verbs:** 127 types
-- **Total:** 169 types
-- **Previous (v5.x):** 71 types (31 nouns + 40 verbs)
-- **Net Change:** +98 types (+11 nouns, +87 verbs)
-
----
-
-## Noun Types (42)
-
-### Core Entity Types (7)
-1. **person** - Individual human entities
-2. **organization** - Collective entities, companies, institutions
-3. **location** - Geographic and named spatial entities
-4. **thing** - Discrete physical objects and artifacts
-5. **concept** - Abstract ideas, principles, and intangibles
-6. **event** - Temporal occurrences and happenings
-7. **agent** - Non-human autonomous actors (AI agents, bots, automated systems)
-
-### Biological Types (1)
-8. **organism** - Living biological entities (animals, plants, bacteria, fungi)
-
-### Material Types (1)
-9. **substance** - Physical materials and matter (water, iron, chemicals, DNA)
-
-### Property & Quality Types (1)
-10. **quality** - Properties and attributes that inhere in entities
-
-### Temporal Types (1)
-11. **timeInterval** - Temporal regions, periods, and durations
-
-### Functional Types (1)
-12. **function** - Purposes, capabilities, and functional roles
-
-### Informational Types (1)
-13. **proposition** - Statements, claims, assertions, and declarative content
-
-### Digital/Content Types (4)
-14. **document** - Text-based files and written content
-15. **media** - Non-text media files (audio, video, images)
-16. **file** - Generic digital files and data blobs
-17. **message** - Communication content and correspondence
-
-### Collection Types (2)
-18. **collection** - Groups and sets of items
-19. **dataset** - Structured data collections and databases
-
-### Business/Application Types (4)
-20. **product** - Commercial products and offerings
-21. **service** - Service offerings and intangible products
-22. **task** - Actions, todos, and work items
-23. **project** - Organized initiatives and programs
-
-### Descriptive Types (6)
-24. **process** - Workflows, procedures, and ongoing activities
-25. **state** - Conditions, status, and situational contexts
-26. **role** - Positions, responsibilities, and functional classifications
-27. **language** - Natural and formal languages
-28. **currency** - Monetary units and exchange mediums
-29. **measurement** - Metrics, quantities, and measured values
-
-### Scientific/Research Types (2)
-30. **hypothesis** - Scientific theories, propositions, and conjectures
-31. **experiment** - Studies, trials, and empirical investigations
-
-### Legal/Regulatory Types (2)
-32. **contract** - Legal agreements, terms, and binding documents
-33. **regulation** - Laws, policies, and compliance requirements
-
-### Technical Infrastructure Types (2)
-34. **interface** - APIs, protocols, and connection points
-35. **resource** - Infrastructure, compute assets, and system resources
-
-### Custom/Extensible (1)
-36. **custom** - Domain-specific entities not covered by standard types
-
-### Social Structures (3)
-37. **socialGroup** - Informal social groups and collectives
-38. **institution** - Formal social structures and practices
-39. **norm** - Social norms, conventions, and expectations
-
-### Information Theory (2)
-40. **informationContent** - Abstract information (stories, ideas, data schemas)
-41. **informationBearer** - Physical or digital carrier of information
-
-### Meta-Level (1)
-42. **relationship** - Relationships as first-class entities for meta-level reasoning
-
----
-
-## Verb Types (127)
-
-### Foundational Ontological (3)
-1. **instanceOf** - Individual to class relationship
-2. **subclassOf** - Taxonomic hierarchy
-3. **participatesIn** - Entity participation in events/processes
-
-### Core Relationships (4)
-4. **relatedTo** - Generic relationship (fallback)
-5. **contains** - Containment relationship
-6. **partOf** - Part-whole mereological relationship
-7. **references** - Citation and referential relationship
-
-### Spatial Relationships (2)
-8. **locatedAt** - Spatial location relationship
-9. **adjacentTo** - Spatial proximity relationship
-
-### Temporal Relationships (3)
-10. **precedes** - Temporal sequence (before)
-11. **during** - Temporal containment
-12. **occursAt** - Temporal location
-
-### Causal & Dependency (5)
-13. **causes** - Direct causal relationship
-14. **enables** - Enablement without direct causation
-15. **prevents** - Prevention relationship
-16. **dependsOn** - Dependency relationship
-17. **requires** - Necessity relationship
-
-### Creation & Transformation (5)
-18. **creates** - Creation relationship
-19. **transforms** - Transformation relationship
-20. **becomes** - State change relationship
-21. **modifies** - Modification relationship
-22. **consumes** - Consumption relationship
-
-### Lifecycle Operations (1)
-23. **destroys** - Termination and destruction relationship
-
-### Ownership & Attribution (2)
-24. **owns** - Ownership relationship
-25. **attributedTo** - Attribution relationship
-
-### Property & Quality (2)
-26. **hasQuality** - Entity to quality attribution
-27. **realizes** - Function realization relationship
-
-### Effects & Experience (1)
-28. **affects** - Patient/experiencer relationship
-
-### Composition (2)
-29. **composedOf** - Material composition
-30. **inherits** - Inheritance relationship
-
-### Social & Organizational (7)
-31. **memberOf** - Membership relationship
-32. **worksWith** - Professional collaboration
-33. **friendOf** - Friendship relationship
-34. **follows** - Following/subscription relationship
-35. **likes** - Liking/favoriting relationship
-36. **reportsTo** - Hierarchical reporting relationship
-37. **mentors** - Mentorship relationship
-38. **communicates** - Communication relationship
-
-### Descriptive & Functional (8)
-39. **describes** - Descriptive relationship
-40. **defines** - Definition relationship
-41. **categorizes** - Categorization relationship
-42. **measures** - Measurement relationship
-43. **evaluates** - Evaluation relationship
-44. **uses** - Utilization relationship
-45. **implements** - Implementation relationship
-46. **extends** - Extension relationship
-
-### Advanced Relationships (4)
-47. **equivalentTo** - Equivalence/identity relationship
-48. **believes** - Epistemic relationship
-49. **conflicts** - Conflict relationship
-50. **synchronizes** - Synchronization relationship
-51. **competes** - Competition relationship
-
-### Modal Relationships (6)
-52. **canCause** - Potential causation (possibility)
-53. **mustCause** - Necessary causation (necessity)
-54. **wouldCauseIf** - Counterfactual causation
-55. **couldBe** - Possible states
-56. **mustBe** - Necessary identity
-57. **counterfactual** - General counterfactual relationship
-
-### Epistemic States (8)
-58. **knows** - Knowledge (justified true belief)
-59. **doubts** - Uncertainty/skepticism
-60. **desires** - Want/preference
-61. **intends** - Intentionality
-62. **fears** - Fear/anxiety
-63. **loves** - Strong positive emotional attitude
-64. **hates** - Strong negative emotional attitude
-65. **hopes** - Hopeful expectation
-66. **perceives** - Sensory perception
-
-### Learning & Cognition (1)
-67. **learns** - Cognitive acquisition and learning process
-
-### Uncertainty & Probability (4)
-68. **probablyCauses** - Probabilistic causation
-69. **uncertainRelation** - Unknown relationship with confidence bounds
-70. **correlatesWith** - Statistical correlation
-71. **approximatelyEquals** - Fuzzy equivalence
-
-### Scalar Properties (5)
-72. **greaterThan** - Scalar comparison
-73. **similarityDegree** - Graded similarity
-74. **moreXThan** - Comparative property
-75. **hasDegree** - Scalar property assignment
-76. **partiallyHas** - Graded possession
-
-### Information Theory (2)
-77. **carries** - Bearer carries content
-78. **encodes** - Encoding relationship
-
-### Deontic Relationships (5)
-79. **obligatedTo** - Moral/legal obligation
-80. **permittedTo** - Permission/authorization
-81. **prohibitedFrom** - Prohibition/forbidden
-82. **shouldDo** - Normative expectation
-83. **mustNotDo** - Strong prohibition
-
-### Context & Perspective (5)
-84. **trueInContext** - Context-dependent truth
-85. **perceivedAs** - Subjective perception
-86. **interpretedAs** - Interpretation relationship
-87. **validInFrame** - Frame-dependent validity
-88. **trueFrom** - Perspective-dependent truth
-
-### Advanced Temporal (6)
-89. **overlaps** - Partial temporal overlap
-90. **immediatelyAfter** - Direct temporal succession
-91. **eventuallyLeadsTo** - Long-term consequence
-92. **simultaneousWith** - Exact temporal alignment
-93. **hasDuration** - Temporal extent
-94. **recurringWith** - Cyclic temporal relationship
-
-### Advanced Spatial (7)
-95. **containsSpatially** - Spatial containment
-96. **overlapsSpatially** - Spatial overlap
-97. **surrounds** - Encirclement
-98. **connectedTo** - Topological connection
-99. **above** - Vertical spatial relationship (superior)
-100. **below** - Vertical spatial relationship (inferior)
-101. **inside** - Within containment boundaries
-102. **outside** - Beyond containment boundaries
-103. **facing** - Directional orientation
-
-### Social Structures (5)
-104. **represents** - Representative relationship
-105. **embodies** - Exemplification or personification
-106. **opposes** - Opposition relationship
-107. **alliesWith** - Alliance relationship
-108. **conformsTo** - Norm conformity
-
-### Measurement (4)
-109. **measuredIn** - Unit relationship
-110. **convertsTo** - Unit conversion
-111. **hasMagnitude** - Quantitative value
-112. **dimensionallyEquals** - Dimensional analysis
-
-### Change & Persistence (4)
-113. **persistsThrough** - Persistence through change
-114. **gainsProperty** - Property acquisition
-115. **losesProperty** - Property loss
-116. **remainsSame** - Identity through time
-
-### Parthood Variations (4)
-117. **functionalPartOf** - Functional component
-118. **topologicalPartOf** - Spatial part
-119. **temporalPartOf** - Temporal slice
-120. **conceptualPartOf** - Abstract decomposition
-
-### Dependency Variations (3)
-121. **rigidlyDependsOn** - Necessary dependency
-122. **functionallyDependsOn** - Operational dependency
-123. **historicallyDependsOn** - Causal history dependency
-
-### Meta-Level (4)
-124. **endorses** - Second-order validation
-125. **contradicts** - Logical contradiction
-126. **supports** - Evidential support
-127. **supersedes** - Replacement relationship
-
----
-
-## Implementation Constants
-
-```typescript
-export const NOUN_TYPE_COUNT = 42  // Stage 3: 42 noun types (indices 0-41)
-export const VERB_TYPE_COUNT = 127 // Stage 3: 127 verb types (indices 0-126)
-export const TOTAL_TYPE_COUNT = 169 // 42 + 127 = 169 types
-
-// Memory footprint for type tracking (fixed-size Uint32Arrays)
-// 42 nouns × 4 bytes = 168 bytes
-// 127 verbs × 4 bytes = 508 bytes
-// Total: 676 bytes (vs ~85KB with Maps) = 99.2% memory reduction
-```
-
----
-
-## Changes from v5.x
-
-### Nouns Added (+11)
-- agent, quality, timeInterval, function, proposition
-- **organism** ⭐ (biological entities)
-- **substance** ⭐ (physical materials)
-- socialGroup, institution, norm
-- informationContent, informationBearer, relationship
-
-### Nouns Removed (-2)
-- **user** (merged into person)
-- **topic** (merged into concept)
-- **content** (removed - redundant)
-
-### Verbs Added (+87)
-- **affects** ⭐ (patient/experiencer role)
-- **learns** ⭐ (cognitive acquisition)
-- **destroys** ⭐ (lifecycle termination)
-- All new categories from Stage 3 taxonomy
-
-### Verbs Removed (-4)
-- **succeeds** (use inverse of precedes)
-- **belongsTo** (use inverse of owns)
-- **createdBy** (use inverse of creates)
-- **supervises** (use inverse of reportsTo)
-
-⭐ = Critical additions from ultradeep analysis
-
----
-
-## Coverage & Completeness
-
-**Domain Coverage:**
-- Natural Sciences: 96% (physics, chemistry, biology, medicine)
-- Formal Sciences: 98% (mathematics, logic, computer science)
-- Social Sciences: 97% (psychology, sociology, economics)
-- Humanities: 96% (philosophy, history, arts)
-
-**Overall:** 96-97% of all human knowledge
-
-**Timeless Design:** Stable for 20+ years
-
-**Extension:** Use "custom" noun for domain-specific entities
-
----
-
-## Verification Checklist
-
-All code, comments, and documentation MUST match this canonical list:
-
-- [ ] graphTypes.ts: NounType has exactly 42 entries
-- [ ] graphTypes.ts: VerbType has exactly 127 entries
-- [ ] graphTypes.ts: NOUN_TYPE_COUNT = 42
-- [ ] graphTypes.ts: VERB_TYPE_COUNT = 127
-- [ ] graphTypes.ts: NounTypeEnum has indices 0-41
-- [ ] graphTypes.ts: VerbTypeEnum has indices 0-126
-- [ ] metadataIndex.ts: Arrays sized for 42 & 127
-- [ ] buildTypeEmbeddings.ts: Descriptions for all 169 types
-- [ ] brainyTypes.ts: Descriptions for all 169 types
-- [ ] index.ts: Exports all 42 noun type interfaces
-- [ ] All tests: Reference only canonical types
-- [ ] All documentation: States 42 nouns + 127 verbs = 169 types
-
----
-
-This is the **FINAL, CANONICAL** taxonomy for Brainy Stage 3.

docs/architecture/PERFORMANCE_ANALYSIS.md

@@ -1,114 +0,0 @@
-# Brainy Performance Analysis & Optimization
-
-## Current Issues Found
-
-### 1. ❌ CRITICAL: notEquals Operator is O(n)
-```javascript
-// PROBLEM: Gets ALL items to filter
-case 'notEquals':
-  const allItemIds = await this.getAllIds() // O(n) - TERRIBLE!
-```
-
-### 2. ❌ Soft Delete Performance
-- Every query adds `deleted: { notEquals: true }` 
-- This makes EVERY query O(n) instead of O(log n)
-
-### 3. ❌ exists Operator is Inefficient
-```javascript
-case 'exists':
-  // Scans all cache entries - O(n)
-  for (const [key, entry] of this.indexCache.entries()) {
-    if (entry.field === field) {
-      entry.ids.forEach(id => allIds.add(id))
-    }
-  }
-```
-
-### 4. ⚠️ Query Optimizer Not Smart Enough
-- `isSelectiveFilter()` needs to understand which filters are fast
-- Should prioritize O(1) and O(log n) operations
-
-## Performance Characteristics
-
-### ✅ Fast Operations (Keep These)
-| Operation | Complexity | Example |
-|-----------|-----------|---------|
-| Vector Search (HNSW) | O(log n) | `like: "query"` |
-| Exact Match | O(1) | `where: { status: "active" }` |
-| Deleted Filter (NEW) | O(1) | `where: { deleted: false }` |
-| Range Query (sorted) | O(log n) | `where: { year: { gt: 2000 } }` |
-| Graph Traversal | O(k) | `connected: { from: id }` |
-
-### ❌ Slow Operations (Need Fixing)
-| Operation | Current | Should Be | Fix |
-|-----------|---------|-----------|-----|
-| notEquals | O(n) | O(1) or O(log n) | Use complement index |
-| exists | O(n) | O(1) | Maintain field existence bitmap |
-| noneOf | O(n) | O(k) | Use set operations |
-
-## Optimized Architecture
-
-### Solution 1: Positive Indexing for Soft Delete ✅
-```javascript
-// Instead of: deleted !== true (O(n))
-// Use: deleted === false (O(1))
-where: { deleted: false }
-
-// Ensure all items have deleted field
-if (!metadata.deleted) metadata.deleted = false
-```
-
-### Solution 2: Complement Indices for notEquals
-```javascript
-class MetadataIndexManager {
-  // For common notEquals queries, maintain complement sets
-  private complementIndices: Map<string, Set<string>> = new Map()
-  
-  // Example: Track non-deleted items separately
-  private activeItems: Set<string> = new Set()
-  private deletedItems: Set<string> = new Set()
-}
-```
-
-### Solution 3: Field Existence Bitmap
-```javascript
-class FieldExistenceIndex {
-  private fieldBitmaps: Map<string, BitSet> = new Map()
-  
-  hasField(id: string, field: string): boolean {
-    return this.fieldBitmaps.get(field)?.has(id) ?? false
-  }
-}
-```
-
-## Query Execution Strategy
-
-### Progressive Search (When Metadata is Selective)
-```
-1. Field Filter (O(1) or O(log n)) → Small candidate set
-2. Vector Search within candidates (O(k log k))
-3. Fusion if needed
-```
-
-### Parallel Search (When Nothing is Selective)
-```
-1. Vector Search (O(log n)) → Top K results
-2. Graph Traversal (O(m)) → Connected items  
-3. Field Filter (O(1)) → Metadata matches
-4. Fusion: Intersection or Union
-```
-
-## Implementation Priority
-
-1. **DONE** ✅ Fix soft delete to use `deleted: false`
-2. **TODO** 🔧 Optimize notEquals for common fields
-3. **TODO** 🔧 Add field existence index
-4. **TODO** 🔧 Improve query optimizer intelligence
-5. **TODO** 🔧 Add query explain mode for debugging
-
-## Performance Targets
-
-- Vector search: < 10ms for 1M items
-- Metadata filter: < 1ms for exact match
-- Combined query: < 20ms for complex queries
-- Soft delete overhead: < 0.1ms (O(1))

docs/architecture/aggregation.md

@@ -1,242 +0,0 @@
-# Aggregation Architecture
-
-> Write-time incremental aggregation with O(1) reads
-
-## Design Principles
-
-1. **Write-time computation** — aggregates update on every `add()`, `update()`, and `delete()`, not as batch jobs
-2. **Incremental state** — running totals maintained per group, never rescanning the dataset
-3. **Provider interface** — TypeScript engine is the default; plugins can replace it with native implementations
-4. **Zero-allocation reads** — query results are computed from pre-aggregated state
-
-## Component Overview
-
-```
-┌──────────────────────────────────────────────────────────┐
-│                         Brainy                           │
-│                                                          │
-│  add() / update() / delete()                             │
-│          │                                               │
-│          ▼                                               │
-│  ┌──────────────────────┐    ┌────────────────────────┐  │
-│  │  AggregationIndex    │    │  AggregateMaterializer  │  │
-│  │                      │───▶│  (debounced writes)     │  │
-│  │  ├─ definitions Map  │    └────────────────────────┘  │
-│  │  ├─ states Map       │                                │
-│  │  └─ staleMinMax Set  │    ┌────────────────────────┐  │
-│  │                      │    │  timeWindows.ts         │  │
-│  │  Source filter ──────│───▶│  bucketTimestamp()      │  │
-│  │  Group key ──────────│───▶│  parseBucketRange()     │  │
-│  └──────────┬───────────┘    └────────────────────────┘  │
-│             │                                            │
-│             │ provider interface                          │
-│             ▼                                            │
-│  ┌──────────────────────┐                                │
-│  │  AggregationProvider │  (optional, registered by      │
-│  │  ├─ incrementalUpdate│   plugin like @soulcraft/cor)   │
-│  │  ├─ rebuildAggregate │                                │
-│  │  ├─ queryAggregate   │                                │
-│  │  └─ serialize/restore│                                │
-│  └──────────────────────┘                                │
-└──────────────────────────────────────────────────────────┘
-```
-
-## State Management
-
-### Definitions
-
-Registered via `brain.defineAggregate(def)`. Stored in a `Map<string, AggregateDefinition>` keyed by aggregate name. Persisted to storage under `__aggregation_definitions__` on flush.
-
-### Group State
-
-Each aggregate maintains a `Map<string, AggregateGroupState>` where keys are serialized group key values (e.g., `category=food|date=2024-01`). Each group holds per-metric `MetricState`:
-
-```typescript
-interface MetricState {
-  sum: number    // Running total
-  count: number  // Entity count
-  min: number    // Minimum (Infinity if empty)
-  max: number    // Maximum (-Infinity if empty)
-  m2?: number    // Welford's M2 for stddev/variance
-}
-```
-
-### Change Detection
-
-On restart, definition hashes (FNV-1a 32-bit) are compared with the persisted hash. If a definition changed (different groupBy, metrics, or source), the aggregate state is reset and must be rebuilt.
-
-## Write-Time Update Flow
-
-When `brain.add(entity)` is called:
-
-```
-1. For each registered aggregate:
-   ├─ Source filter check (type, service, where)
-   │  └─ Skip if entity doesn't match
-   ├─ Aggregate entity check
-   │  └─ Skip if entity.service === 'brainy:aggregation'
-   │     or entity.metadata.__aggregate is set
-   ├─ Group key computation
-   │  └─ Extract groupBy fields from metadata
-   │     Apply time bucketing for windowed dimensions
-   └─ Metric update
-      └─ For each metric in the definition:
-         ├─ count: increment count
-         ├─ sum/avg: add value to sum, increment count
-         ├─ min/max: compare and update
-         └─ stddev/variance: Welford's online update
-```
-
-### Update Handling
-
-On `brain.update(entity)`, the engine reverses the old entity's contribution and applies the new entity's contribution. This correctly handles:
-
-- **Value changes**: old amount=10, new amount=20 — sum adjusts by +10
-- **Group key changes**: entity moves from category "food" to "drink" — both groups update
-- **Source filter changes**: entity type changes from Event to Document — removed from matching aggregates
-
-### Delete Handling
-
-On `brain.remove(id)`, the engine reverses the entity's contribution:
-
-- `count` and `sum` are decremented
-- `min`/`max` may become stale (marked in `staleMinMax` for lazy recompute)
-- Welford's M2 is updated with the inverse formula
-- Empty groups (all metric counts at zero) are removed
-
-## Algorithms
-
-### Welford's Online Algorithm
-
-Standard deviation and variance use Welford's numerically stable online algorithm with M2 tracking. This computes incrementally without storing individual values:
-
-```
-On add(x):
-  count += 1
-  oldMean = (sum - x) / (count - 1)    // mean before this value
-  sum += x
-  mean = sum / count                     // mean after this value
-  M2 += (x - oldMean) * (x - mean)
-
-On remove(x):
-  oldMean = sum / count
-  sum -= x
-  count -= 1
-  newMean = sum / count
-  M2 = max(0, M2 - (x - oldMean) * (x - newMean))
-
-Sample variance = M2 / (count - 1)
-Sample stddev = sqrt(variance)
-```
-
-M2 is clamped to zero on remove to prevent floating-point drift from producing negative values.
-
-### MIN/MAX Handling
-
-The TypeScript engine uses simple comparison for add operations and marks MIN/MAX as potentially stale on delete (since removing the current min/max value requires a rescan). Stale values are lazily recomputed on the next query.
-
-The Cor native engine uses a `BTreeMap<OrderedFloat<f64>, u64>` that tracks the exact frequency of every value, providing precise MIN/MAX after any sequence of operations without rescanning.
-
-### Time Window Bucketing
-
-Timestamps (Unix milliseconds) are bucketed using UTC-based formatting:
-
-| Granularity | Bucket Key | Algorithm |
-|------------|-----------|-----------|
-| `hour` | `2024-01-15T14` | UTC year-month-day-hour |
-| `day` | `2024-01-15` | UTC year-month-day |
-| `week` | `2024-W03` | ISO 8601 week (Monday start, week 1 contains first Thursday) |
-| `month` | `2024-01` | UTC year-month |
-| `quarter` | `2024-Q1` | `ceil((month) / 3)` |
-| `year` | `2024` | UTC year |
-| `{ seconds: N }` | ISO timestamp | `floor(timestamp / interval) * interval` |
-
-Bucket keys can be parsed back into `{ start, end }` timestamp ranges via `parseBucketRange()`.
-
-## Provider Interface
-
-The `AggregationProvider` interface defines the contract between Brainy's `AggregationIndex` and plugin-provided native implementations:
-
-```typescript
-interface AggregationProvider {
-  defineAggregate?(def: AggregateDefinition): void
-  removeAggregate?(name: string): void
-
-  incrementalUpdate(
-    name: string,
-    def: AggregateDefinition,
-    entity: Record<string, unknown>,
-    op: 'add' | 'update' | 'delete',
-    prev?: Record<string, unknown>
-  ): AggregateGroupState[]
-
-  computeGroupKey(
-    entity: Record<string, unknown>,
-    groupBy: GroupByDimension[]
-  ): Record<string, string | number>
-
-  rebuildAggregate(
-    def: AggregateDefinition,
-    entities: Array<Record<string, unknown>>
-  ): Map<string, AggregateGroupState>
-
-  queryAggregate(
-    state: Map<string, AggregateGroupState>,
-    params: AggregateQueryParams
-  ): AggregateResult[]
-
-  restoreState?(data: string): void
-  serializeState?(): string
-}
-```
-
-When a native provider is registered:
-
-1. `AggregationIndex` delegates `incrementalUpdate()` to the provider instead of running TypeScript logic
-2. Provider returns updated `AggregateGroupState[]` which are applied back into the state maps
-3. Query execution is delegated via `queryAggregate()`
-4. State serialization is delegated via `serializeState()`/`restoreState()`
-
-Brainy retains ownership of the state maps and persistence. The provider handles computation.
-
-## Materialization
-
-The `AggregateMaterializer` converts aggregate group states into `NounType.Measurement` entities:
-
-1. When an aggregate group is updated and `materialize` is enabled, `scheduleMaterialize()` is called
-2. Materialization is debounced (default: 1000ms) to batch rapid updates during ingestion
-3. On trigger, the materializer either creates or updates a `NounType.Measurement` entity
-4. Materialized entities include `service: 'brainy:aggregation'` and `metadata.__aggregate` to prevent infinite loops
-
-Materialized entities are automatically visible through:
-- OData endpoints
-- Google Sheets integration
-- Server-Sent Events (SSE)
-- Webhook notifications
-
-## Persistence
-
-### Storage Keys
-
-| Key | Content |
-|-----|---------|
-| `__aggregation_definitions__` | Array of all definitions with FNV-1a hashes |
-| `__aggregation_state_{name}__` | Per-aggregate group states (array of `AggregateGroupState`) |
-| `__aggregation_native_state__` | Serialized native provider state (JSON string) |
-
-### Lifecycle
-
-1. **`init()`** — Load definitions, compare hashes, load matching state, restore native provider state
-2. **Write operations** — Mark modified aggregates as dirty
-3. **`flush()`** — Persist all dirty aggregate states and native provider state
-4. **`close()`** — Flush and release resources
-
-## Source Files
-
-| File | Purpose |
-|------|---------|
-| `src/aggregation/AggregationIndex.ts` | Core engine: definitions, state, write hooks, query |
-| `src/aggregation/materializer.ts` | Debounced materialization of results as entities |
-| `src/aggregation/timeWindows.ts` | Time bucketing and bucket range parsing |
-| `src/aggregation/index.ts` | Module exports |
-| `src/types/brainy.types.ts` | Type definitions for all aggregation interfaces |

docs/architecture/augmentations-actual.md

@@ -1,302 +0,0 @@
-# Augmentations System - What Actually Exists
-
-> **Important Update**: Investigation reveals Brainy has MORE augmentations than documented!
-
-## ✅ Actually Implemented Augmentations (12+)
-
-Full implementation with crash recovery, checkpointing, and replay.
-```typescript
-// Fully working with all features documented
-```
-
-### 2. Entity Registry Augmentation ✅
-High-performance deduplication using bloom filters.
-```typescript
-import { EntityRegistryAugmentation } from 'brainy'
-// Complete with all features
-```
-
-### 3. Auto-Register Entities Augmentation ✅
-Automatic entity extraction from text.
-```typescript
-import { AutoRegisterEntitiesAugmentation } from 'brainy'
-// Extracts and registers entities automatically
-```
-
-### 4. Intelligent Verb Scoring Augmentation ✅
-Multi-factor relationship strength calculation.
-```typescript
-import { IntelligentVerbScoringAugmentation } from 'brainy'
-// Semantic, temporal, frequency scoring
-```
-
-### 5. Batch Processing Augmentation ✅
-Dynamic batching with adaptive backpressure.
-```typescript
-import { BatchProcessingAugmentation } from 'brainy'
-// Smart batching with flow control
-```
-
-### 6. Connection Pool Augmentation ✅
-Intelligent connection management.
-```typescript
-import { ConnectionPoolAugmentation } from 'brainy'
-// Auto-scaling connection pools
-```
-
-### 7. Request Deduplicator Augmentation ✅
-Prevents duplicate operations.
-```typescript
-import { RequestDeduplicatorAugmentation } from 'brainy'
-// In-flight request deduplication
-```
-
-### 8. WebSocket Conduit Augmentation ✅
-Real-time bidirectional streaming.
-```typescript
-import { WebSocketConduitAugmentation } from 'brainy'
-// Full WebSocket support
-```
-
-### 9. WebRTC Conduit Augmentation ✅
-Peer-to-peer communication.
-```typescript
-import { WebRTCConduitAugmentation } from 'brainy'
-// P2P data channels
-```
-
-### 10. Memory Storage Augmentation ✅
-Optimized in-memory operations.
-```typescript
-import { MemoryStorageAugmentation } from 'brainy'
-// Memory-specific optimizations
-```
-
-### 11. Server Search Augmentation ✅
-Server-side search delegation over a conduit.
-```typescript
-import { ServerSearchConduitAugmentation } from 'brainy'
-// Forwards queries to a remote Brainy server
-```
-
-### 12. Neural Import Augmentation ✅
-AI-powered data understanding and import.
-```typescript
-import { NeuralImportAugmentation } from 'brainy'
-// Full entity detection and classification
-```
-
-## 🎯 Hidden Features in Augmentations
-
-### Neural Import Capabilities (Fully Implemented!)
-```typescript
-const neuralImport = new NeuralImport(brain)
-
-// These ALL work:
-await neuralImport.neuralImport('data.csv')
-await neuralImport.detectEntitiesWithNeuralAnalysis(data)
-await neuralImport.detectNounType(entity)
-await neuralImport.detectRelationships(entities)
-await neuralImport.generateInsights(data)
-```
-
-### Operation Modes (Fully Implemented!)
-```typescript
-// Read-only mode with optimized caching
-const readerMode = new ReaderMode()
-// 80% cache, aggressive prefetch, 1hr TTL
-
-// Write-only mode with batching
-const writerMode = new WriterMode()
-// Large write buffer, batch writes, minimal cache
-
-// Hybrid mode
-const hybridMode = new HybridMode()
-// Balanced for mixed workloads
-```
-
-### Advanced Caching (3-Level System!)
-```typescript
-const cacheManager = new CacheManager({
-  hotCache: { size: 1000, ttl: 60000 },    // L1 - RAM
-  warmCache: { size: 10000, ttl: 300000 }, // L2 - Fast storage
-  coldCache: { size: 100000, ttl: null }   // L3 - Persistent
-})
-```
-
-### Performance Monitoring (Complete!)
-```typescript
-const monitor = new PerformanceMonitor(brain)
-
-// All these metrics work:
-monitor.getMetrics() // Returns comprehensive stats
-monitor.getQueryPatterns() // Query analysis
-monitor.getCacheStats() // Cache performance
-monitor.getThrottlingMetrics() // Rate limiting info
-```
-
-## 📊 Statistics System (Fully Working!)
-
-```typescript
-const stats = await brain.getStats()
-// Returns comprehensive metrics:
-{
-  nouns: {
-    count: number,
-    created: number,
-    updated: number,
-    deleted: number,
-    size: number,
-    avgSize: number
-  },
-  verbs: {
-    count: number,
-    created: number,
-    types: Record<string, number>,
-    weights: { min, max, avg }
-  },
-  vectors: {
-    dimensions: 384,
-    indexSize: number,
-    partitions: number,
-    avgSearchTime: number
-  },
-  cache: {
-    hits: number,
-    misses: number,
-    evictions: number,
-    hitRate: number,
-    hotCacheSize: number,
-    warmCacheSize: number
-  },
-  performance: {
-    operations: number,
-    avgAddTime: number,
-    avgSearchTime: number,
-    avgUpdateTime: number,
-    p95Latency: number,
-    p99Latency: number
-  },
-  storage: {
-    used: number,
-    available: number,
-    compression: number,
-    files: number
-  },
-  throttling: {
-    delays: number,
-    rateLimited: number,
-    backoffMs: number,
-    retries: number
-  }
-}
-```
-
-## 🚀 GPU Support (Partial but Real!)
-
-```typescript
-// GPU detection WORKS:
-const device = await detectBestDevice()
-// Returns: 'cpu' | 'webgpu' | 'cuda'
-
-// WebGPU support in browser:
-if (device === 'webgpu') {
-  // Transformer models can use WebGPU
-}
-
-// CUDA detection in Node:
-if (device === 'cuda') {
-  // Future: GPU acceleration support
-}
-```
-
-## 🔄 Adaptive Systems (All Working!)
-
-### Adaptive Backpressure
-```typescript
-const backpressure = new AdaptiveBackpressure()
-// Automatically adjusts flow based on system load
-```
-
-### Adaptive Socket Manager
-```typescript
-const socketManager = new AdaptiveSocketManager()
-// Dynamic connection pooling based on traffic
-```
-
-### Cache Auto-Configuration
-```typescript
-const cacheConfig = await getCacheAutoConfig()
-// Sizes cache based on available memory
-```
-
-### S3 Throttling Protection
-```typescript
-// Built into S3 storage adapter
-// Automatic exponential backoff
-// Rate limit detection and adaptation
-```
-
-## 🎨 How to Use Hidden Features
-
-### Enable Reader / Writer Modes
-```typescript
-const brain = new Brainy({
-  mode: 'reader' // or 'writer' or 'hybrid'
-})
-```
-
-### Use Neural Import
-```typescript
-const brain = new Brainy({
-  augmentations: [
-    new NeuralImportAugmentation({
-      confidenceThreshold: 0.7,
-      autoDetect: true
-    })
-  ]
-})
-
-// Import with AI understanding
-await brain.neuralImport('data.csv')
-```
-
-### Access Statistics
-```typescript
-// Get comprehensive stats
-const stats = await brain.getStats()
-
-// Get specific service stats
-const nounStats = await brain.getStatistics({ 
-  service: 'nouns' 
-})
-
-// Force refresh
-const freshStats = await brain.getStatistics({ 
-  forceRefresh: true 
-})
-```
-
-## 📝 What Needs Documentation
-
-These features EXIST but need better docs:
-1. Reader / writer operation modes
-2. Neural import full API
-3. 3-level cache configuration
-4. Performance monitoring API
-5. GPU acceleration setup
-6. Advanced statistics queries
-7. Throttling configuration
-8. Backpressure tuning
-
-## 💡 The Truth
-
-Brainy is MORE powerful than its own documentation suggests! Most "missing" features are actually implemented but hidden or not properly exposed. The codebase contains sophisticated systems for:
-- Reader / writer operation modes
-- AI-powered import
-- Advanced caching
-- Performance monitoring
-- GPU support
-- Adaptive optimization
-
-The main work needed is integration and documentation, not implementation!

docs/architecture/augmentations.md

@@ -1,494 +0,0 @@
-# Augmentations System
-
-## Overview
-
-Brainy's Augmentation System provides a powerful plugin architecture that extends core functionality without modifying the base code. Augmentations can intercept, modify, and enhance any operation in the database.
-
-## Built-in Augmentations
-
-> **Note**: This document shows both available and planned augmentations. Each section is marked with its current status.
-
-### 1. Entity Registry Augmentation ✅ Available
-
-High-performance deduplication for streaming data ingestion.
-
-```typescript
-import { EntityRegistryAugmentation } from 'brainy'
-
-const brain = new Brainy({
-  augmentations: [
-    new EntityRegistryAugmentation({
-      maxCacheSize: 100000,      // Track up to 100k unique entities
-      ttl: 3600000,              // 1-hour TTL for cache entries
-      hashFields: ['id', 'url']  // Fields to use for deduplication
-    })
-  ]
-})
-
-// Automatically prevents duplicate entities
-await brain.add("Same content", { id: "123" }) // Added
-await brain.add("Same content", { id: "123" }) // Skipped (duplicate)
-```
-
-**Benefits:**
-- O(1) duplicate detection using bloom filters
-- Configurable cache size and TTL
-- Custom hash field selection
-- Perfect for real-time data streams
-
-
-Enterprise-grade durability and crash recovery.
-
-```typescript
-
-const brain = new Brainy({
-  augmentations: [
-      checkpointInterval: 1000,  // Checkpoint every 1000 operations
-      compression: true,          // Enable log compression
-      maxLogSize: 100 * 1024 * 1024  // 100MB max log size
-    })
-  ]
-})
-
-// All operations are now durably logged
-
-// Recover from crash
-const recovered = new Brainy({ 
-})
-```
-
-**Features:**
-- ACID compliance
-- Automatic crash recovery
-- Point-in-time recovery
-- Log compression and rotation
-- Minimal performance impact
-
-### 3. Intelligent Verb Scoring Augmentation ✅ Available
-
-AI-powered relationship strength calculation.
-
-```typescript
-import { IntelligentVerbScoringAugmentation } from 'brainy'
-
-const brain = new Brainy({
-  augmentations: [
-    new IntelligentVerbScoringAugmentation({
-      factors: {
-        semantic: 0.4,    // Weight for semantic similarity
-        temporal: 0.3,    // Weight for time proximity
-        frequency: 0.2,   // Weight for interaction frequency
-        explicit: 0.1     // Weight for explicit ratings
-      }
-    })
-  ]
-})
-
-// Relationships automatically get intelligent scores
-await brain.relate(user1, product1, "viewed", { timestamp: Date.now() })
-await brain.relate(user1, product1, "purchased", { timestamp: Date.now() })
-// Automatically calculates relationship strength based on multiple factors
-
-// Query using intelligent scores
-const strongRelationships = await brain.find({
-  connected: {
-    from: user1,
-    minScore: 0.8  // Only highly relevant relationships
-  }
-})
-```
-
-**Capabilities:**
-- Multi-factor relationship scoring
-- Temporal decay functions
-- Semantic similarity integration
-- Customizable weight factors
-
-### 4. Auto-Register Entities Augmentation ⚠️ Basic Implementation
-
-Automatically extracts and registers entities from text.
-
-```typescript
-import { AutoRegisterEntitiesAugmentation } from 'brainy'
-
-const brain = new Brainy({
-  augmentations: [
-    new AutoRegisterEntitiesAugmentation({
-      types: ['person', 'organization', 'location', 'product'],
-      confidence: 0.8,
-      createRelationships: true
-    })
-  ]
-})
-
-// Automatically extracts and registers entities
-await brain.add(
-  "Apple CEO Tim Cook announced the new iPhone 15 in Cupertino",
-  { type: "news" }
-)
-// Automatically creates:
-// - Noun: "Tim Cook" (person)
-// - Noun: "Apple" (organization)
-// - Noun: "iPhone 15" (product)
-// - Noun: "Cupertino" (location)
-// - Verbs: relationships between entities
-```
-
-**Features:**
-- NER (Named Entity Recognition)
-- Automatic relationship inference
-- Configurable entity types
-- Confidence thresholds
-
-### 5. Batch Processing Augmentation ✅ Available
-
-Optimizes bulk operations for maximum throughput.
-
-```typescript
-import { BatchProcessingAugmentation } from 'brainy'
-
-const brain = new Brainy({
-  augmentations: [
-    new BatchProcessingAugmentation({
-      batchSize: 100,
-      flushInterval: 1000,  // Flush every second
-      parallel: true,        // Parallel processing
-      maxQueueSize: 10000
-    })
-  ]
-})
-
-// Operations are automatically batched
-for (let i = 0; i < 10000; i++) {
-  await brain.add(`Item ${i}`)  // Internally batched
-}
-// Processes in optimized batches of 100
-```
-
-**Benefits:**
-- 10-100x throughput improvement
-- Automatic batching
-- Configurable batch sizes
-- Memory-efficient queue management
-
-### 6. Caching Augmentation 🚧 Coming Soon
-
-Intelligent multi-level caching system.
-
-```typescript
-import { CachingAugmentation } from 'brainy'
-
-const brain = new Brainy({
-  augmentations: [
-    new CachingAugmentation({
-      levels: {
-        l1: { size: 100, ttl: 60000 },      // Hot cache: 100 items, 1 min
-        l2: { size: 1000, ttl: 300000 },    // Warm cache: 1000 items, 5 min
-        l3: { size: 10000, ttl: 3600000 }   // Cold cache: 10k items, 1 hour
-      },
-      strategies: ['lru', 'lfu'],  // Least Recently/Frequently Used
-      preload: true                 // Preload popular items
-    })
-  ]
-})
-
-// Queries automatically use cache
-const results = await brain.find("popular query")  // Cached
-const again = await brain.find("popular query")    // From cache (instant)
-```
-
-**Features:**
-- Multi-level cache hierarchy
-- Multiple eviction strategies
-- Query result caching
-- Embedding cache
-- Automatic cache invalidation
-
-### 7. Compression Augmentation 🚧 Coming Soon
-
-Reduces storage size while maintaining query performance.
-
-```typescript
-import { CompressionAugmentation } from 'brainy'
-
-const brain = new Brainy({
-  augmentations: [
-    new CompressionAugmentation({
-      algorithm: 'brotli',
-      level: 6,  // Compression level (1-11)
-      threshold: 1024,  // Only compress items > 1KB
-      excludeFields: ['id', 'type']  // Don't compress these
-    })
-  ]
-})
-
-// Data automatically compressed/decompressed
-await brain.add(largeDocument)  // Compressed before storage
-const doc = await brain.getNoun(id)  // Decompressed on retrieval
-```
-
-**Benefits:**
-- 60-80% storage reduction
-- Transparent compression
-- Selective field compression
-- Multiple algorithm support
-
-### 8. Monitoring Augmentation 🚧 Coming Soon
-
-Real-time performance monitoring and metrics.
-
-```typescript
-import { MonitoringAugmentation } from 'brainy'
-
-const brain = new Brainy({
-  augmentations: [
-    new MonitoringAugmentation({
-      metrics: ['operations', 'latency', 'cache', 'memory'],
-      interval: 5000,  // Report every 5 seconds
-      webhook: 'https://metrics.example.com/brainy',
-      console: true  // Also log to console
-    })
-  ]
-})
-
-// Automatic metric collection
-brain.on('metrics', (metrics) => {
-  console.log(`
-    Operations/sec: ${metrics.opsPerSecond}
-    Avg latency: ${metrics.avgLatency}ms
-    Cache hit rate: ${metrics.cacheHitRate}%
-    Memory usage: ${metrics.memoryMB}MB
-  `)
-})
-```
-
-**Metrics:**
-- Operation throughput
-- Query latency percentiles
-- Cache hit rates
-- Memory usage
-- Storage growth
-- Error rates
-
-## Neural Import Capabilities 🚧 Coming Soon
-
-> **Note**: Import/Export features are currently in development. Expected Q1 2025.
-
-### 1. Document Import with Auto-Structuring
-
-```typescript
-import { NeuralImportAugmentation } from 'brainy'
-
-const brain = new Brainy({
-  augmentations: [
-    new NeuralImportAugmentation({
-      autoStructure: true,
-      extractEntities: true,
-      generateSummaries: true,
-      detectLanguage: true
-    })
-  ]
-})
-
-// Import unstructured documents
-await brain.importDocument('./research-paper.pdf')
-// Automatically:
-// - Extracts text and metadata
-// - Identifies sections and structure
-// - Extracts entities and concepts
-// - Generates embeddings per section
-// - Creates relationship graph
-```
-
-### 2. Database Migration Import
-
-```typescript
-// Import from existing databases
-await brain.importFromSQL({
-  connection: 'postgres://localhost/mydb',
-  tables: {
-    users: { type: 'person', idField: 'user_id' },
-    products: { type: 'product', idField: 'sku' },
-    orders: { 
-      type: 'relationship',
-      from: 'user_id',
-      to: 'product_id',
-      verb: 'purchased'
-    }
-  }
-})
-
-// Import from MongoDB
-await brain.importFromMongo({
-  uri: 'mongodb://localhost:27017',
-  database: 'myapp',
-  collections: {
-    users: { type: 'person' },
-    posts: { type: 'content' }
-  }
-})
-```
-
-### 3. Stream Import
-
-```typescript
-// Import from real-time streams
-await brain.importStream({
-  source: 'kafka://localhost:9092/events',
-  format: 'json',
-  transform: (event) => ({
-    noun: event.data,
-    metadata: {
-      type: event.type,
-      timestamp: event.timestamp
-    }
-  }),
-  deduplication: true
-})
-```
-
-### 4. Bulk CSV/JSON Import
-
-```typescript
-// Import CSV with automatic type detection
-await brain.importCSV('./data.csv', {
-  headers: true,
-  typeColumn: 'entity_type',
-  detectRelationships: true,
-  batchSize: 1000
-})
-
-// Import JSON with nested structure handling
-await brain.importJSON('./data.json', {
-  rootPath: '$.entities',
-  nounPath: '$.content',
-  metadataPath: '$.properties',
-  relationshipPath: '$.connections'
-})
-```
-
-## Creating Custom Augmentations
-
-```typescript
-import { Augmentation } from 'brainy'
-
-class CustomAugmentation extends Augmentation {
-  name = 'CustomAugmentation'
-  
-  async onInit(brain: Brainy): Promise<void> {
-    // Initialize augmentation
-    console.log('Custom augmentation initialized')
-  }
-  
-  async onBeforeAddNoun(content: any, metadata: any): Promise<[any, any]> {
-    // Modify before adding noun
-    metadata.processed = true
-    metadata.timestamp = Date.now()
-    return [content, metadata]
-  }
-  
-  async onAfterAddNoun(id: string, noun: any): Promise<void> {
-    // React to noun addition
-    console.log(`Noun ${id} added`)
-  }
-  
-  async onBeforeSearch(query: any): Promise<any> {
-    // Modify search query
-    query.boost = 'recent'
-    return query
-  }
-  
-  async onAfterSearch(results: any[]): Promise<any[]> {
-    // Process search results
-    return results.map(r => ({
-      ...r,
-      customScore: r.score * 1.5
-    }))
-  }
-}
-
-// Use custom augmentation
-const brain = new Brainy({
-  augmentations: [new CustomAugmentation()]
-})
-```
-
-## Augmentation Lifecycle Hooks
-
-### Available Hooks
-
-```typescript
-interface AugmentationHooks {
-  // Initialization
-  onInit(brain: Brainy): Promise<void>
-  onShutdown(): Promise<void>
-  
-  // Noun operations
-  onBeforeAddNoun(content, metadata): Promise<[content, metadata]>
-  onAfterAddNoun(id, noun): Promise<void>
-  onBeforeGetNoun(id): Promise<string>
-  onAfterGetNoun(noun): Promise<any>
-  onBeforeUpdateNoun(id, updates): Promise<[string, any]>
-  onAfterUpdateNoun(id, noun): Promise<void>
-  onBeforeDeleteNoun(id): Promise<string>
-  onAfterDeleteNoun(id): Promise<void>
-  
-  // Verb operations
-  onBeforeAddVerb(source, target, type, metadata): Promise<[any, any, string, any]>
-  onAfterAddVerb(id, verb): Promise<void>
-  onBeforeGetVerb(id): Promise<string>
-  onAfterGetVerb(verb): Promise<any>
-  
-  // Search operations
-  onBeforeSearch(query): Promise<any>
-  onAfterSearch(results): Promise<any[]>
-  onBeforeFind(query): Promise<any>
-  onAfterFind(results): Promise<any[]>
-  
-  // Storage operations
-  onBeforeSave(data): Promise<any>
-  onAfterLoad(data): Promise<any>
-  
-  // Events
-  onError(error): Promise<void>
-  onMetric(metric): Promise<void>
-}
-```
-
-## Augmentation Composition
-
-```typescript
-// Combine multiple augmentations
-const brain = new Brainy({
-  augmentations: [
-    // Order matters - executed in sequence
-    new EntityRegistryAugmentation(),    // Deduplication first
-    new AutoRegisterEntitiesAugmentation(), // Entity extraction
-    new IntelligentVerbScoringAugmentation(), // Scoring
-    new CompressionAugmentation(),       // Compression
-    new CachingAugmentation(),          // Caching
-    new MonitoringAugmentation()        // Monitoring last
-  ]
-})
-```
-
-## Performance Considerations
-
-1. **Order Matters**: Place filtering augmentations early
-2. **Resource Usage**: Monitor memory with many augmentations
-3. **Async Operations**: Use parallel processing where possible
-4. **Caching**: Enable caching augmentation for read-heavy workloads
-
-## Best Practices
-
-1. **Single Responsibility**: Each augmentation should do one thing well
-2. **Non-Blocking**: Avoid blocking operations in hooks
-3. **Error Handling**: Always handle errors gracefully
-4. **Configuration**: Make augmentations configurable
-5. **Documentation**: Document augmentation behavior and options
-
-## See Also
-
-- [Architecture Overview](./overview.md)
-- [API Reference](../api/README.md)
-- [Performance Guide](../guides/performance.md)

docs/architecture/data-storage-architecture.md

@@ -1,388 +0,0 @@
-# Brainy Data Storage Architecture (8.0)
-
-**Complete on-disk reference for the 8.0 layout.**
-
-This document describes what a Brainy 8.0 data directory actually contains: the
-canonical entity records, the system area, the generational-MVCC bookkeeping,
-the column store, the blob area, and the lock files — plus how the in-memory
-indexes rebuild from them. The authoritative design records are
-[ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and
-[index-architecture.md](./index-architecture.md); this document is the on-disk
-map that ties them together.
-
-8.0 removed the 7.x copy-on-write subsystem (`_cow/`, `branches/{branch}/…`
-paths) and the cloud/OPFS storage adapters. The two storage backends are
-**filesystem** and **memory**; both speak the same path vocabulary (memory
-storage keys its internal map by the identical path strings).
-
----
-
-## 1. Directory Tree
-
-A real 8.0 filesystem store (`path`, default `./brainy-data`):
-
-```
-brainy-data/
-│
-├── entities/                            # Canonical records (current state)
-│   ├── nouns/
-│   │   └── {shard}/                     # 256 shards: first 2 hex chars of the UUID
-│   │       └── {id}/                    # One directory per entity UUID
-│   │           ├── vectors.json.gz      # Embedding + HNSW node state
-│   │           └── metadata.json.gz     # Everything else (type, subtype, data, fields, _rev)
-│   └── verbs/
-│       └── {shard}/
-│           └── {id}/
-│               ├── vectors.json.gz      # Relationship embedding (when present)
-│               └── metadata.json.gz     # sourceId, targetId, verb, subtype, weight, data…
-│
-├── _system/                             # System singletons + bucketed system keys
-│   ├── generation.json(.gz)             # { generation, updatedAt } — the write watermark
-│   ├── manifest.json                    # { version, generation, … } — MVCC commit point
-│   ├── tx-log.jsonl                     # One line per committed transact() batch (append-only)
-│   ├── counts.json                      # Entity/verb totals
-│   ├── type-statistics.json.gz          # Per-NounType counts
-│   ├── subtype-statistics.json.gz       # Per-(NounType, subtype) counts
-│   ├── verb-subtype-statistics.json.gz  # Per-(VerbType, subtype) counts
-│   ├── statistics.json                  # Aggregate statistics blob (counts, index sizes)
-│   ├── hnsw-system.json                 # Vector-index entry point + max level
-│   ├── __metadata_field_registry__.json.gz   # Which metadata fields are indexed
-│   ├── brainy:entityIdMapper.json.gz    # UUID ↔ u64 mapping for native index providers
-│   └── idx/
-│       └── {bucket}/                    # 256 buckets: FNV-1a hash of the key
-│           ├── __metadata_field_index__field_{name}.json.gz   # Sparse field indexes
-│           ├── __chunk__*.json.gz       # Metadata-index roaring-bitmap chunks
-│           ├── __sparse_index__*.json.gz# Zone maps + bloom filters
-│           └── graph-lsm-verbs-{source|target}-*.json.gz      # Graph LSM SSTables + manifest
-│
-├── _generations/                        # MVCC history (written ONLY by transact())
-│   └── {N}/                             # One directory per committed generation N
-│       ├── tx.json                      # The generation-N delta (immutable)
-│       └── prev/
-│           └── {id}.json                # Before-image of each touched record (immutable)
-│
-├── _column_index/                       # Column store manifests (one dir per field)
-│   └── {field}/
-│       └── MANIFEST.json.gz             # Run list + zone metadata for that column
-│
-├── _blobs/                              # Binary blob area (`<key>.bin` convention)
-│   ├── _column_index/
-│   │   └── {field}/
-│   │       └── L0-000001.bin            # Column-store runs (level-0 segments)
-│   └── …                                # VFS file content and other binary blobs
-│
-└── locks/                               # Process coordination (NEVER snapshotted)
-    ├── _writer.lock                     # Single-writer lock: pid, hostname, heartbeat
-    ├── _flush_requests/                 # Reader→writer flush RPC (.req files)
-    └── _flush_responses/                # Writer acks (.ack files)
-```
-
-Most JSON objects are gzip-compressed (`.json.gz`) — compression is on by
-default for filesystem storage (`storage.options.compression`, zlib level 6).
-A few hot singletons (`manifest.json`, `counts.json`, `hnsw-system.json`,
-`tx-log.jsonl`) are written uncompressed for cheap partial reads and appends.
-
----
-
-## 2. Canonical Entity Records
-
-Each entity (noun) and relationship (verb) is **two files** under one
-ID-first directory. The split keeps vector I/O (large, append-mostly) separate
-from metadata I/O (small, read-heavy).
-
-### Noun vector file — `entities/nouns/{shard}/{id}/vectors.json.gz`
-
-```json
-{
-  "id": "421d92e7-4241-470a-80f4-4b39414e7a83",
-  "vector": [-0.139, -0.056, 0.028, "…384 dims…"],
-  "connections": { "0": ["neighbor-uuid…"] },
-  "level": 0
-}
-```
-
-The HNSW node state (`connections`, `level`) is persisted with the vector so
-the vector index can rebuild without recomputing the graph.
-
-### Noun metadata file — `entities/nouns/{shard}/{id}/metadata.json.gz`
-
-```json
-{
-  "data": "React is a JavaScript library for building user interfaces",
-  "noun": "concept",
-  "subtype": "cli-add",
-  "createdAt": 1781198053726,
-  "updatedAt": 1781198053726,
-  "_rev": 1
-}
-```
-
-- `noun` is the NounType. **Type lives in metadata, not in the path** — lookup
-  by ID is a single path construction, no type needed.
-- `subtype` is the per-product sub-classification (required on write by
-  default in 8.0).
-- `_rev` increments on every write and backs `update({ ifRev })` CAS.
-- Consumer metadata fields sit alongside the standard ones.
-
-### Verb files — `entities/verbs/{shard}/{id}/…`
-
-Same two-file split. The verb metadata record carries the graph edge:
-`sourceId`, `targetId`, `verb` (VerbType), `subtype`, `weight`, `data`,
-`metadata`, timestamps, `_rev`. Verb IDs are Brainy-generated UUIDs by
-contract (8.0 rejects caller-supplied verb ids) because native graph providers
-intern the raw UUID bytes as u64 handles.
-
-### Path construction
-
-```typescript
-const shard = id.substring(0, 2)                                  // '42'
-const metadataPath = `entities/nouns/${shard}/${id}/metadata.json`
-const vectorsPath  = `entities/nouns/${shard}/${id}/vectors.json`
-// Verbs: same shape under entities/verbs/
-```
-
-Implemented in `src/storage/baseStorage.ts` (path generators) and
-`src/storage/sharding.ts` (`getShardIdFromUuid`).
-
----
-
-## 3. The `_system/` Area
-
-Two kinds of keys live here, resolved by `BaseStorage.parsePath()`:
-
-1. **Singletons** — well-known keys written at `_system/<key>.json`:
-   `counts`, `statistics`, `type-statistics`, `hnsw-system`,
-   `__metadata_field_registry__`, `brainy:entityIdMapper`, plus the MVCC
-   trio (`generation.json`, `manifest.json`, `tx-log.jsonl`).
-2. **Bucketed system keys** — everything else (field indexes, bitmap chunks,
-   sparse-index segments, graph LSM SSTables) hashes into one of 256
-   `_system/idx/{bucket}/` directories via FNV-1a, so no single directory
-   accumulates unbounded entries.
-
-Notable singletons:
-
-| File | Contents |
-|------|----------|
-| `generation.json` | `{ generation, updatedAt }` — monotonic watermark, bumped by **every** write batch |
-| `manifest.json` | MVCC commit point: highest *committed* generation (see §4) |
-| `tx-log.jsonl` | One JSON line per committed `transact()` batch: generation, timestamp, `meta` |
-| `type-statistics.json.gz` | Per-NounType counts (backs `brain.counts.byType`) |
-| `subtype-statistics.json.gz` | `{ counts: { [type]: { [subtype]: n } }, updatedAt }` (contract-bound shape) |
-| `verb-subtype-statistics.json.gz` | Same shape for VerbTypes |
-| `hnsw-system.json` | `{ entryPointId, maxLevel }` — per-node state lives in each entity's `vectors.json` |
-| `brainy:entityIdMapper.json.gz` | UUID ↔ u64 interning table for the BigInt provider contract |
-| `__metadata_field_registry__.json.gz` | Registry of indexed metadata field names |
-
----
-
-## 4. Generational MVCC (`_generations/` + the `_system` trio)
-
-Full design in [ADR-001](../ADR-001-generational-mvcc.md). The on-disk shape:
-
-```
-_system/generation.json      { generation, updatedAt }          atomic tmp+rename
-_system/manifest.json        { version, generation, … }         atomic tmp+rename — THE commit point
-_system/tx-log.jsonl         one line per committed transact()  append-only
-_generations/{N}/tx.json     the generation-N delta             immutable once written
-_generations/{N}/prev/{id}.json   before-image of {id}          immutable once written
-```
-
-Commit protocol (writer side): stage before-images and the delta under
-`_generations/N/`, fsync, apply the delta to the canonical `entities/…`
-records, then atomically rename `manifest.json` to publish generation N. The
-tx-log line is appended last (advisory). Crash recovery on open discards any
-`_generations/{N}` newer than the manifest.
-
-Two write classes share the generation clock:
-
-- **Single-operation writes** (`add`/`update`/`remove`/`relate` outside
-  `transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound,
-  but write **no** history — they are not visible to `db.since()` and remain
-  visible through earlier pins.
-- **`transact()` batches** write the full `_generations/{N}` record and a
-  tx-log line, and are the unit of time travel (`brain.asOf()`).
-
-Snapshots (`db.persist(path)`) hard-link the entire store **except `locks/`**
-into a self-contained directory openable via `Brainy.load(path)`.
-
----
-
-## 5. Column Store (`_column_index/` + `_blobs/_column_index/`)
-
-The metadata index persists per-field columnar runs for O(log n) range and
-membership queries at scale:
-
-- `_column_index/{field}/MANIFEST.json.gz` — the run list and zone metadata
-  for one field (`createdAt`, `subtype`, `noun`, `_rev`, consumer fields,
-  `__words__` for tokenized text…).
-- `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run
-  segments, stored through the shared `_blobs/<key>.bin` binary convention.
-
-Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments
-additionally live as bucketed keys under `_system/idx/` (see §3). Which path
-serves a given `where` clause is the query planner's decision — inspect it
-with `brainy inspect explain <dir> --where '…'`.
-
----
-
-## 6. Blob Area (`_blobs/`)
-
-`_blobs/<key>.bin` is the flat binary-blob convention shared by every storage
-adapter (`saveBinaryBlob`/`getBinaryBlob` in the storage contract):
-
-- **VFS file content** — VFS entities are regular nouns (path, ownership, and
-  timestamps in entity metadata); the file *bytes* are blobs.
-- **Column-store runs** (under the `_column_index/` key prefix, §5).
-- Any other binary payload an index provider persists.
-
-Writes use unique temp names + rename, so concurrent writers of the same key
-cannot tear each other's blobs.
-
----
-
-## 7. Locks (`locks/`)
-
-```
-locks/_writer.lock          # single-writer lock: { pid, hostname, startedAt, heartbeat, version }
-locks/_flush_requests/      # readers drop <uuid>.req to ask the writer to flush
-locks/_flush_responses/     # writer answers with <uuid>.ack
-```
-
-- One **writer** per data directory, enforced at `init()`; stale locks (dead
-  PID / stale heartbeat) are reclaimed automatically.
-- Read-only processes (`Brainy.openReadOnly()`, the `brainy inspect` CLI
-  family) can ask the live writer to flush via the request/response files, so
-  out-of-process diagnostics see fresh state.
-- `locks/` is excluded from snapshots (`SNAPSHOT_EXCLUDED_TOP_DIRS` in
-  `src/storage/adapters/fileSystemStorage.ts`).
-
----
-
-## 8. In-Memory Indexes and What Rebuilds From What
-
-| Index | In memory | Persisted state | Rebuild source |
-|-------|-----------|-----------------|----------------|
-| **Vector (HNSW)** | Graph of vector connections | `_system/hnsw-system.json` + per-entity `vectors.json` | Walk entity vector files; lazy mode loads structure only and pages vectors on demand |
-| **Metadata index** | Field → value bitmaps + column-store readers | `_system/idx/` chunks + `_column_index/` manifests + `_blobs/_column_index/` runs | Loaded directly; full rebuild re-scans entity metadata |
-| **Graph adjacency** | sourceId/targetId → verb-id LSM trees | `graph-lsm-verbs-{source,target}-*` SSTables under `_system/idx/` | Loaded from SSTables; full rebuild re-scans verb metadata |
-| **Counts/statistics** | Per-type and per-subtype maps | `_system/{type,subtype,verb-subtype}-statistics.json.gz`, `counts.json` | Recomputable by scanning entities (`brainy inspect repair`) |
-
-A pluggable index provider (the 8.0 plugin contract in
-`@soulcraft/brainy/plugin`) may replace any of the JS implementations; the
-persisted formats above are contract-bound so JS and native implementations
-can interleave on the same directory.
-
----
-
-## 9. Sharding Strategy
-
-**Entities:** first 2 hex characters of the UUID → 256 uniform shards.
-Deterministic, configuration-free, and keeps per-directory entry counts low
-(at 1M entities: ~3,900 directories per shard). Paginated whole-store walks
-(`getNouns`/`getVerbs`) iterate shards `00`–`ff` in order.
-
-**System keys:** FNV-1a hash of the key → 256 `_system/idx/` buckets. Same
-motivation, different keyspace (system keys are not UUIDs).
-
-**What is never sharded:** the `_system/` singletons, `_generations/{N}`
-directories (keyed by generation number), `_column_index/{field}` manifests
-(keyed by field name), and `locks/`.
-
----
-
-## 10. Durability and Atomicity
-
-- **Per-object atomicity:** every JSON object and blob is written to a unique
-  temp file then `rename()`d — readers never observe torn objects.
-- **Transaction atomicity:** the `manifest.json` rename is the single commit
-  point for `transact()` batches (§4); everything staged before it is
-  discarded by crash recovery if the rename never lands.
-- **Compression:** gzip per object (`.json.gz`), transparent to all readers.
-  Native index providers that mmap binary formats use the uncompressed
-  `_blobs/` area instead.
-
----
-
-## 11. `clear()` Semantics
-
-`brain.clear()` removes all entities, relationships, indexes, statistics, and
-MVCC history, then re-resolves every index exactly as `init()` does —
-including plugin-provided vector/metadata/id-mapper factories and VFS root
-re-creation. The data directory afterwards contains a fresh, empty store (the
-writer lock remains held by the running process).
-
----
-
-## 12. Common Scenarios
-
-### Adding an entity
-
-```
-brain.add({ data, type, subtype })
-1. Generate UUID → shard = first 2 hex chars
-2. Embed data → 384-dim vector
-3. Write entities/nouns/{shard}/{id}/vectors.json.gz   (vector + HNSW node state)
-4. Write entities/nouns/{shard}/{id}/metadata.json.gz  (type/subtype/data/fields, _rev: 1)
-5. Update in-memory indexes (HNSW insert, metadata index, statistics)
-6. Bump _system/generation.json (no _generations/ entry — single-op write)
-```
-
-### Committing a transaction
-
-```
-await brain.transact(tx => { tx.add(…); tx.update(…) })
-1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync
-2. Apply the delta to canonical entities/… records
-3. Atomic-rename _system/manifest.json → generation N is committed
-4. Append one line to _system/tx-log.jsonl
-```
-
-### Cold start
-
-```
-await brain.init()
-1. Acquire locks/_writer.lock (or open read-only)
-2. Crash recovery: drop _generations/{N} newer than manifest.json
-3. Load _system singletons (counts, statistics, field registry, id mapper)
-4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large)
-5. Graph adjacency: load LSM SSTables from _system/idx/
-6. Metadata index: column-store manifests + bitmap chunks on demand
-```
-
-### Snapshot and restore
-
-```
-const db = brain.now(); await db.persist('/backups/today'); await db.release()
-→ hard-links everything except locks/ into a self-contained directory
-
-await Brainy.load('/backups/today')        // open snapshot read-only as a Db
-await brain.restore('/backups/today', { confirm: true })   // replace store state
-```
-
----
-
-## 13. Summary
-
-- **Two backends** (filesystem, memory), one path vocabulary.
-- **Two files per entity** under ID-first `entities/{kind}/{shard}/{id}/`.
-- **Type and subtype are metadata**, not directory structure; type queries go
-  through the metadata index, not the filesystem.
-- **`_system/`** holds singletons plus 256 hash buckets of index state.
-- **`_generations/` + `manifest.json` + `tx-log.jsonl`** implement
-  generational MVCC. History is per-write: every `add()`/`update()`/`remove()`/
-  `relate()` gets its own generation, and `transact()` groups several ops into
-  one atomic generation. (Single-op retention has been the model since 8.0;
-  you never need to route a write through `transact()` just to keep its history.)
-- **`_column_index/` + `_blobs/`** hold the columnar metadata runs and binary
-  blobs (VFS content included).
-- **`locks/`** coordinates the single writer and reader flush requests, and
-  never travels with snapshots.
-
----
-
-## Next Steps
-
-- [ADR-001 — Generational MVCC](../ADR-001-generational-mvcc.md)
-- [Index Architecture](./index-architecture.md)
-- [Consistency Model](../concepts/consistency-model.md)
-- [VFS Guide](../vfs/README.md)

docs/architecture/finite-type-system.md

@@ -1,538 +0,0 @@
-# 🎯 Brainy's Finite Noun/Verb Type System
-
-> **Why Brainy's Finite Type System is Revolutionary for Knowledge Graphs at Billion Scale**
-
-## Overview
-
-Brainy introduces a **finite type system** that sits between traditional schemaless NoSQL and rigid relational databases. This approach unlocks unprecedented optimization opportunities while maintaining semantic flexibility.
-
----
-
-## The Three-Way Comparison
-
-### 1. Traditional NoSQL (Schemaless)
-
-```typescript
-// Complete freedom, zero optimization
-{
-  id: '123',
-  randomField1: 'value',
-  anotherWeirdKey: 42,
-  whoKnowsWhatElse: { nested: 'chaos' }
-}
-```
-
-**Problems:**
-- ❌ No index optimization possible
-- ❌ Tools can't understand data structure
-- ❌ Incompatible augmentations/extensions
-- ❌ Memory explosion with billions of unique keys
-- ❌ No semantic understanding
-- ❌ Query planning impossible
-
-### 2. Traditional Relational (Rigid Schema)
-
-```sql
-CREATE TABLE entities (
-  id UUID PRIMARY KEY,
-  field1 VARCHAR(255),
-  field2 INTEGER,
-  ...
-  field50 TEXT
-);
-```
-
-**Problems:**
-- ❌ Must define schema upfront
-- ❌ Schema migrations are painful
-- ❌ Can't handle heterogeneous data
-- ❌ Requires restart for schema changes
-- ❌ Fixed columns waste space
-
-### 3. Brainy's Finite Type System (Semantic Structure)
-
-```typescript
-// Finite noun types (extensible but constrained)
-type NounType =
-  | 'person' | 'place' | 'organization' | 'document'
-  | 'event' | 'concept' | 'thing' | ...
-
-// Finite verb types (semantic relationships)
-type VerbType =
-  | 'relatedTo' | 'contains' | 'isA' | 'causedBy'
-  | 'precedes' | 'influences' | ...
-
-// Example usage
-const entity = {
-  id: '123',
-  nounType: 'person',        // Finite! Known type
-  vector: [...],             // Semantic embedding
-  metadata: {
-    noun: 'person',          // Required type field
-    name: 'Alice',           // Custom fields allowed
-    occupation: 'Engineer'   // Flexible metadata
-  }
-}
-```
-
-**Benefits:**
-- ✅ **Index Optimization**: Fixed-size Uint32Arrays for type tracking (99.76% memory reduction)
-- ✅ **Semantic Understanding**: Types have meaning, not just structure
-- ✅ **Tool Compatibility**: All augmentations understand core types
-- ✅ **Concept Extraction**: NLP can map text to known types
-- ✅ **Explicit Types**: Clear type specification in API
-- ✅ **Query Optimization**: Type-aware query planning
-- ✅ **Flexible Metadata**: Any fields within typed structure
-- ✅ **Billion-Scale Ready**: Type tracking scales linearly
-
----
-
-## Revolutionary Benefits in Detail
-
-### 1. Index Optimization at Billion Scale
-
-**The Problem**: Traditional NoSQL stores arbitrary field names in indexes:
-
-```typescript
-// Memory explosion with unique keys
-Map<string, Set<string>> {
-  "user_preference_notification_email_enabled": Set(['id1', 'id2', ...]),
-  "customer_shipping_address_line_1": Set(['id3', 'id4', ...]),
-  // Billions of unique, unpredictable keys!
-}
-```
-
-**Brainy's Solution**: Fixed noun/verb types enable fixed-size tracking:
-
-```typescript
-// 99.76% memory reduction with Uint32Arrays
-class TypeAwareMetadataIndex {
-  // Fixed size: nounTypes × verbTypes × fieldCount
-  private nounTypeBitmaps: RoaringBitmap32[]        // One per noun type
-  private verbTypeBitmaps: RoaringBitmap32[]        // One per verb type
-
-  // Example: 100 noun types × 50 verb types = 5KB overhead
-  // vs 500MB+ for arbitrary keys!
-}
-```
-
-**Real-World Impact (PROJECTED - not yet benchmarked)**:
-- **Before**: 500MB memory for 1M entities with diverse keys
-- **After**: PROJECTED 1.2MB memory for same dataset (385x reduction - calculated from Uint32Array size, not measured)
-- **Scales to billions**: Memory grows with entity count, not key diversity
-
-### 2. Explicit Type System
-
-**The Design**: Specify types clearly in your API calls:
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-// Add entity with explicit type
-await brain.add({
-  data: { name: 'Alice', role: 'CEO of Acme Corp' },
-  type: NounType.Person  // Explicit type specification
-})
-
-// Query with type filtering
-await brain.find({
-  query: 'Alice',
-  type: NounType.Person  // Type-optimized search
-})
-```
-
-**Why Explicit Types?**:
-1. **Deterministic**: You control exactly how entities are classified
-2. **Predictable**: No inference surprises or edge cases
-3. **Fast**: No neural processing overhead on every add/query
-4. **Smaller**: No embedded keyword models needed
-
-**Real-World Use Case**:
-```typescript
-// Import data with known types
-await brain.add({
-  data: { name: 'Apple Inc.', industry: 'Technology' },
-  type: NounType.Organization
-})
-
-await brain.add({
-  data: { name: 'Cupertino', country: 'USA' },
-  type: NounType.Location
-})
-
-// Create relationship
-await brain.relate({
-  from: appleId,
-  to: cupertinoId,
-  type: VerbType.LocatedIn
-})
-```
-
-### 3. Tool & Augmentation Compatibility
-
-**The Problem with Schemaless**: Every tool must handle infinite variations:
-
-```typescript
-// Incompatible tools
-const tool1Data = { type: 'person', name: 'Alice' }
-const tool2Data = { kind: 'human', fullName: 'Alice' }
-const tool3Data = { entity_type: 'individual', person_name: 'Alice' }
-
-// Tools can't understand each other!
-```
-
-**Brainy's Solution**: Finite types create a common language:
-
-```typescript
-// All tools/augmentations understand core types
-interface NounMetadata {
-  noun: NounType  // Agreed-upon type system
-  // ... custom fields
-}
-
-// Augmentation 1: Adds caching for 'person' entities
-class PersonCacheAugmentation {
-  execute(op, params) {
-    if (params.noun?.metadata?.noun === 'person') {
-      // All person entities are understood!
-    }
-  }
-}
-
-// Augmentation 2: Enriches 'organization' entities
-class OrgEnrichmentAugmentation {
-  execute(op, params) {
-    if (params.noun?.metadata?.noun === 'organization') {
-      // Fetch industry data, employees, etc.
-    }
-  }
-}
-
-// Augmentations compose seamlessly!
-```
-
-**Ecosystem Benefits**:
-- Third-party augmentations are **interoperable**
-- Type-specific optimizations are **portable**
-- Query builders understand **semantic structure**
-- Visualization tools render **type-appropriate** displays
-- Import/export tools map to **universal types**
-
-### 4. Concept Extraction & NLP Integration
-
-**Traditional Approach**: Extract entities, ignore types:
-
-```typescript
-// Generic NER (Named Entity Recognition)
-"Alice works at Google"
-// → ['Alice', 'Google']  // What are these?
-```
-
-**Brainy's Approach**: Extract **typed** concepts:
-
-```typescript
-import { NaturalLanguageProcessor } from '@soulcraft/brainy'
-
-const nlp = new NaturalLanguageProcessor()
-const concepts = await nlp.extractConcepts("Alice works at Google in San Francisco")
-
-// Returns typed entities:
-[
-  { text: 'Alice', nounType: 'person', confidence: 0.95 },
-  { text: 'Google', nounType: 'organization', confidence: 0.98 },
-  { text: 'San Francisco', nounType: 'place', confidence: 0.92 }
-]
-
-// And typed relationships:
-[
-  {
-    from: 'Alice',
-    to: 'Google',
-    verbType: 'worksAt',
-    confidence: 0.88
-  },
-  {
-    from: 'Google',
-    to: 'San Francisco',
-    verbType: 'locatedIn',
-    confidence: 0.85
-  }
-]
-```
-
-**Downstream Benefits**:
-- **Smart Clustering**: Group by semantic type, not arbitrary keys
-- **Type-Aware Queries**: "Find all organizations in California"
-- **Relationship Reasoning**: "Who works at companies in SF?"
-- **Automatic Ontology**: Types form natural hierarchy
-
-### 5. Query Optimization & Planning
-
-**The Problem**: Schemaless queries are guesswork:
-
-```sql
--- MongoDB: No idea what fields exist
-db.collection.find({ someField: 'value' })
-// Full collection scan!
-```
-
-**Brainy's Solution**: Type-aware query planning:
-
-```typescript
-// Query planner knows types exist!
-brain.find({
-  where: { noun: 'person' }  // Type index lookup: O(1)!
-})
-
-// Multi-type queries are optimized
-brain.find({
-  where: {
-    noun: ['person', 'organization'],  // Bitmap union
-    location: 'California'              // Then filter
-  }
-})
-
-// Relationship traversal is type-aware
-brain.find({
-  verb: 'worksAt',          // Verb type index
-  sourceType: 'person',     // Source noun type index
-  targetType: 'organization' // Target noun type index
-})
-```
-
-**Query Performance**:
-- **Type Filtering**: O(1) bitmap intersection
-- **Join Planning**: Type-aware join order optimization
-- **Index Selection**: Automatic best index for type
-- **Cardinality Estimation**: Type statistics guide planning
-
-### 6. Architecture & Development Benefits
-
-#### Memory-Efficient Type Tracking
-
-```typescript
-// Traditional approach: Map per field
-class TraditionalIndex {
-  private fieldIndexes: Map<string, Map<any, Set<string>>>
-  // Memory: O(unique_fields × unique_values × entities)
-}
-
-// Brainy approach: Fixed Uint32Array per type
-class TypeAwareIndex {
-  private nounTypeTracking: Uint32Array  // Fixed size!
-  private typeIndexes: RoaringBitmap32[] // One per type
-  // Memory: O(noun_types) + O(entities_per_type)
-  // PROJECTED: 385x smaller at billion scale (calculated from architecture, not benchmarked)
-}
-```
-
-#### Type-Driven Code Organization
-
-```typescript
-// Natural code structure follows types
-/src
-  /nouns
-    /person
-      personStorage.ts       // Type-specific storage
-      personQueries.ts       // Type-specific queries
-      personAugmentation.ts  // Type-specific logic
-    /organization
-      orgStorage.ts
-      orgQueries.ts
-      orgAugmentation.ts
-  /verbs
-    /worksAt
-      worksAtValidation.ts   // Relationship rules
-      worksAtInference.ts    // Type inference
-```
-
-#### Type Safety in TypeScript
-
-```typescript
-// Compiler-enforced type correctness
-function processPerson(noun: Noun) {
-  if (noun.metadata.noun === 'person') {
-    // TypeScript narrows type!
-    const name: string = noun.metadata.name  // Safe access
-  }
-}
-
-// Exhaustive type checking
-function processNoun(noun: Noun) {
-  switch (noun.metadata.noun) {
-    case 'person': return handlePerson(noun)
-    case 'place': return handlePlace(noun)
-    case 'organization': return handleOrg(noun)
-    // Compiler error if missing cases!
-  }
-}
-```
-
----
-
-## Public API: Type System
-
-The type system is **fully public** for developers and augmentation authors:
-
-```typescript
-import {
-  NounType,
-  VerbType,
-  getNounTypes,
-  getVerbTypes,
-  BrainyTypes,
-  suggestType
-} from '@soulcraft/brainy'
-
-// Get all available noun types
-const nounTypes = getNounTypes()
-// → ['Person', 'Organization', 'Location', 'Thing', 'Concept', ...]
-
-// Get all available verb types
-const verbTypes = getVerbTypes()
-// → ['RelatedTo', 'Contains', 'CreatedBy', 'LocatedIn', ...]
-
-// Use types directly
-await brain.add({
-  data: { name: 'Alice' },
-  type: NounType.Person
-})
-
-// Query by type
-await brain.find({
-  type: NounType.Person,
-  where: { name: 'Alice' }
-})
-```
-
-**Use Cases**:
-- **Type-Safe Code**: Use TypeScript enums for compile-time checking
-- **Import Tools**: Specify entity types during data import
-- **Query Builders**: Filter by known types
-- **Augmentations**: Type-specific processing pipelines
-- **Visualization**: Type-appropriate rendering
-
----
-
-## Real-World Performance Comparison
-
-### Scenario: 1 Billion Entities with Rich Metadata
-
-| Aspect | NoSQL (Schemaless) | Relational (Fixed) | Brainy (Finite Types) |
-|--------|-------------------|-------------------|----------------------|
-| **Memory (Indexes)** | 500GB+ | 250GB | 1.3GB |
-| **Type Lookup** | Full scan | O(log n) | O(1) bitmap |
-| **Add New Type** | Zero cost | Schema migration! | Register type |
-| **Query Planning** | Impossible | Table statistics | Type statistics |
-| **Tool Compatibility** | None | SQL only | Full ecosystem |
-| **Semantic Understanding** | None | None | Built-in |
-| **Concept Extraction** | Manual | Manual | Via SmartExtractor |
-| **Flexibility** | Infinite | Zero | Optimal balance |
-
----
-
-## Design Principles
-
-### 1. Finite but Extensible
-
-```typescript
-// Core types are finite
-const coreNounTypes = [
-  'person', 'place', 'organization', 'thing', ...
-]
-
-// But easily extended
-brain.registerNounType('chemical_compound', {
-  keywords: ['molecule', 'compound', 'element'],
-  synonyms: ['substance', 'material'],
-  parentType: 'thing'
-})
-```
-
-### 1a. Subtypes — sub-classification without hierarchy
-
-The 42-type taxonomy is intentionally coarse. Per-product vocabulary fits on the **`subtype`** axis — a top-level standard string field on every entity. Flat by design — no hierarchy, no parent chain, no recursive resolution. That preserves the Uint32Array-backed O(1) type stats while giving consumers a place to put `'employee'` / `'customer'` / `'invoice'` / `'milestone'` without burning a slot in the global enum.
-
-```typescript
-// Same NounType, different subtypes:
-await brain.add({ type: NounType.Person,   subtype: 'employee' })
-await brain.add({ type: NounType.Person,   subtype: 'customer' })
-await brain.add({ type: NounType.Document, subtype: 'invoice'  })
-
-// Fast path — column-store hit, not metadata fallback:
-await brain.find({ type: NounType.Person, subtype: 'employee' })
-
-// Per-NounType-per-subtype counts maintained incrementally:
-brain.counts.bySubtype(NounType.Person)
-// → { employee: 12, customer: 847 }
-```
-
-Subtype has its own statistics rollup (`_system/subtype-statistics.json`) maintained alongside `nounCountsByType`, so per-subtype counts stay O(1) at billion scale.
-
-The same principle applies to **VerbTypes** (7.30+): the 127-verb taxonomy is intentionally coarse, and `subtype` is the per-product axis for relationships too. A `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'colleague'`. Verb-side rollup lives at `_system/verb-subtype-statistics.json` with identical shape to the noun-side rollup. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`. Brainy's design is fully symmetric — nouns and verbs are first-class peers with identical capability surfaces.
-
-Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
-
-### 2. Semantic not Structural
-
-```typescript
-// NOT structural types
-type Person = {
-  name: string
-  age: number
-  // Fixed structure
-}
-
-// Semantic types
-type Noun = {
-  nounType: 'person',  // Semantic meaning!
-  metadata: {
-    noun: 'person',    // Required type
-    // Any custom fields!
-  }
-}
-```
-
-### 3. Optimizable yet Flexible
-
-```typescript
-// Optimized type tracking
-const typeIndex = new RoaringBitmap32()  // 99.76% smaller!
-
-// Flexible metadata
-const metadata = {
-  noun: 'person',          // Required type
-  customField1: 'value',   // Your fields
-  customField2: 123,       // Any structure
-  nested: { ... }          // Full flexibility
-}
-```
-
----
-
-## Conclusion
-
-Brainy's **Finite Noun/Verb Type System** is revolutionary because it achieves the impossible:
-
-1. ✅ **Billion-scale performance** (99.76% memory reduction)
-2. ✅ **Semantic understanding** (NLP integration)
-3. ✅ **Tool compatibility** (ecosystem interoperability)
-4. ✅ **Query optimization** (type-aware planning)
-5. ✅ **Concept extraction** (via SmartExtractor for imports)
-6. ✅ **Developer experience** (clean architecture)
-7. ✅ **Flexibility** (metadata freedom within types)
-
-It's not schemaless chaos. It's not rigid relational constraints. It's **semantic structure** - the perfect balance for knowledge graphs at scale.
-
----
-
-## Further Reading
-
-- [Storage Architecture](./storage-architecture.md) - How types enable billion-scale storage
-- [Augmentation System](./augmentations.md) - Building type-aware augmentations
-- [Query Optimization](../api/query-optimization.md) - Type-aware query planning
-- [Import Flow](../guides/import-flow.md) - How types work in the import pipeline
-
----
-
-*Brainy's finite type system: The foundation of billion-scale, semantically-aware knowledge graphs.*

docs/architecture/index-architecture.md

@@ -1,941 +0,0 @@
-# Index Architecture
-
-Brainy uses a sophisticated **3-tier index architecture** that enables "Triple Intelligence" - the unified combination of vector similarity, graph relationships, and metadata filtering. This document provides a comprehensive architectural overview of how these indexes work internally and coordinate with each other.
-
-## Overview: The Three Main Indexes + Sub-Indexes
-
-Brainy has **3 main indexes** at the top level, each with multiple sub-indexes managed automatically:
-
-### Main Indexes (Level 1)
-
-| Index | Purpose | Data Structure | Complexity | File Location | rebuild() Method |
-|-------|---------|----------------|------------|---------------|------------------|
-| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
-| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 |
-| **GraphAdjacencyIndex** | Relationship traversal | 2 verb-id LSM-trees + tombstone-filtered adjacency derivation | O(degree) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
-
-### Sub-Indexes (Level 2)
-
-**TypeAwareVectorIndex contains:**
-- **42 type-specific vector indexes** - One per NounType (automatically rebuilt via parent)
-
-**MetadataIndexManager contains:**
-- **ChunkManager** - Adaptive chunked sparse indexing
-- **EntityIdMapper** - UUID ↔ integer mapping for roaring bitmaps
-- **FieldTypeInference** - DuckDB-inspired value-based field type detection
-- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count)
-- **Sorted Indexes** - Support orderBy queries (automatically maintained)
-- **Word Index (`__words__`)** - Text search via FNV-1a word hashes
-
-**GraphAdjacencyIndex contains:**
-- **lsmTreeSource** - Source → Targets (outgoing edges)
-- **lsmTreeTarget** - Target → Sources (incoming edges)
-- **lsmTreeVerbsBySource** - Source → Verb IDs
-- **lsmTreeVerbsByTarget** - Target → Verb IDs
-
-All indexes share a **UnifiedCache** for coordinated memory management, ensuring fair resource allocation and preventing any single index from monopolizing memory.
-
-## 1. MetadataIndex - Fast Field Filtering
-
-**Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing.
-
-### Internal Architecture
-
-```typescript
-class MetadataIndexManager {
-  // Chunked sparse indices: field → SparseIndex (replaces flat files)
-  private sparseIndices = new Map<string, SparseIndex>()
-
-  // Chunk management
-  private chunkManager: ChunkManager
-  private chunkingStrategy: AdaptiveChunkingStrategy
-
-  // Lightweight field statistics
-  private fieldIndexes = new Map<string, FieldIndexData>()  // value → count
-  private fieldStats = new Map<string, FieldStats>()        // cardinality tracking
-
-  // Type-field affinity for NLP understanding
-  private typeFieldAffinity = new Map<string, Map<string, number>>()
-
-  // Shared memory management
-  private unifiedCache: UnifiedCache
-}
-```
-
-### Key Data Structures
-
-#### Chunked Sparse Index
-```typescript
-// SparseIndex: Directory of chunks for a field
-// Example: field="status"
-class SparseIndex {
-  field: string
-  chunks: ChunkDescriptor[]  // Metadata about each chunk
-  bloomFilters: BloomFilter[] // Fast membership testing
-}
-
-// ChunkDescriptor: Metadata about a chunk
-interface ChunkDescriptor {
-  chunkId: number
-  valueCount: number         // How many unique values in this chunk
-  idCount: number            // Total entity IDs
-  zoneMap: ZoneMap          // Min/max for range queries
-  lastUpdated: number
-}
-
-// Actual chunk data stored separately
-class ChunkData {
-  chunkId: number
-  field: string
-  entries: Map<value, RoaringBitmap32>  // ~50 values per chunk (roaring bitmaps!)
-}
-```
-
-**Performance**:
-- O(1) exact lookup with bloom filters (1% false positive rate)
-- O(log n) range queries with zone maps
-- 630x file reduction (560k flat files → 89 chunk files)
-
-#### Roaring Bitmap Optimization
-
-**Problem Solved**: JavaScript `Set<string>` for storing entity IDs was inefficient:
-- Memory overhead: ~40 bytes per UUID string (36 chars + overhead)
-- Slow intersection: JavaScript array filtering for multi-field queries
-- No hardware acceleration
-
-**Solution**: Replace `Set<string>` with `RoaringBitmap32` (WebAssembly implementation) for 90% memory savings and hardware-accelerated operations. Uses `roaring-wasm` package for universal compatibility (Node.js, browsers, serverless) without requiring native compilation.
-
-```typescript
-// EntityIdMapper: UUID ↔ Integer mapping
-class EntityIdMapper {
-  private uuidToInt = new Map<string, number>()
-  private intToUuid = new Map<number, string>()
-  private nextId = 1
-
-  getOrAssign(uuid: string): number {
-    // O(1) mapping: UUIDs → integers for bitmap storage
-    let intId = this.uuidToInt.get(uuid)
-    if (!intId) {
-      intId = this.nextId++
-      this.uuidToInt.set(uuid, intId)
-      this.intToUuid.set(intId, uuid)
-    }
-    return intId
-  }
-
-  intsIterableToUuids(ints: Iterable<number>): string[] {
-    // Convert bitmap results back to UUIDs
-    const result: string[] = []
-    for (const intId of ints) {
-      const uuid = this.intToUuid.get(intId)
-      if (uuid) result.push(uuid)
-    }
-    return result
-  }
-}
-
-// ChunkData now uses RoaringBitmap32 instead of Set<string>
-class ChunkData {
-  chunkId: number
-  field: string
-  entries: Map<string, RoaringBitmap32>  // value → bitmap of integer IDs
-}
-```
-
-**Key Benefits**:
-- **90% memory savings**: Roaring bitmaps compress much better than UUID strings
-- **Hardware-accelerated operations**: SIMD instructions (AVX2/SSE4.2) for ultra-fast bitmap AND/OR
-- **Portable serialization**: Cross-platform compatible format (Java/Go/Node.js)
-- **Lazy conversion**: UUIDs converted to integers only once, not per query
-
-**Multi-Field Intersection (THE BIG WIN!)**:
-```typescript
-// Before: JavaScript array filtering
-async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string[]> {
-  // 1. Fetch UUID arrays for each field
-  const statusIds = await this.getIds('status', 'active')  // ["uuid1", "uuid2", ...]
-  const roleIds = await this.getIds('role', 'admin')       // ["uuid2", "uuid3", ...]
-
-  // 2. JavaScript intersection (SLOW!)
-  return statusIds.filter(id => roleIds.includes(id))      // O(n*m) array filtering
-}
-
-// After: Roaring bitmap intersection
-async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
-  // 1. Fetch roaring bitmaps (integers, not UUIDs)
-  const bitmaps: RoaringBitmap32[] = []
-  for (const {field, value} of pairs) {
-    const bitmap = await this.getBitmapFromChunks(field, value)
-    if (!bitmap) return []  // Short-circuit if any field has no matches
-    bitmaps.push(bitmap)
-  }
-
-  // 2. Hardware-accelerated intersection (FAST! AVX2/SSE4.2 SIMD)
-  const result = RoaringBitmap32.and(...bitmaps)           // O(1) hardware operation!
-
-  // 3. Convert final bitmap to UUIDs (once, not per-field)
-  return this.idMapper.intsIterableToUuids(result)
-}
-```
-
-**Performance Impact**:
-- Multi-field intersection: **1.4x average speedup**, up to 3.3x on 10K entities
-- Memory usage: **90% reduction** (17.17 MB → 2.01 MB for 100K entities)
-- Hardware acceleration: SIMD instructions make bitmap operations nearly free
-
-**Benchmark Results** — example output from a single run of `tests/performance/roaring-bitmap-benchmark.ts` (1,000 queries per size, one machine; absolute times vary by hardware, the relative speedup and memory savings are the durable signal):
-| Dataset Size | Operation | Set Time | Roaring Time | Speedup | Memory Savings |
-|--------------|-----------|----------|--------------|---------|----------------|
-| 10,000 entities | 3-field intersection | 3.74ms | 1.14ms | **3.3x faster** | 90% |
-| 100,000 entities | 3-field intersection | 2.60ms | 1.78ms | **1.5x faster** | 88% |
-
-**Implementation**: See `src/utils/entityIdMapper.ts` and benchmark at `tests/performance/roaring-bitmap-benchmark.ts`
-
-#### Bloom Filter (Probabilistic Membership Testing)
-```typescript
-class BloomFilter {
-  bits: Uint8Array        // Bit array
-  size: number           // Total bits
-  hashCount: number      // Number of hash functions (FNV-1a, DJB2)
-
-  mightContain(value): boolean  // ~1% false positive, 0% false negative
-}
-```
-
-**Use case**: Quickly skip chunks that definitely don't contain a value
-
-#### Zone Map (Range Query Optimization)
-```typescript
-interface ZoneMap {
-  min: any | null       // Minimum value in chunk
-  max: any | null       // Maximum value in chunk
-  count: number         // Number of entries
-  hasNulls: boolean     // Whether chunk contains null values
-}
-```
-
-**Use case**: Skip entire chunks during range queries (ClickHouse-inspired)
-
-#### Type-Field Affinity
-```typescript
-// Tracks which fields are commonly used with which types
-// Example:
-// typeFieldAffinity.get('character') → {
-//   'name': 127,      // 127 characters have a 'name' field
-//   'age': 89,        // 89 characters have an 'age' field
-//   'alignment': 45   // 45 characters have an 'alignment' field
-// }
-```
-
-**Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field
-
-#### Word Index (`__words__`) -
-```typescript
-// Special field for text/keyword search
-// Entity text content is tokenized and indexed as word hashes
-
-// Tokenization:
-// "David Smith is a software engineer" → ["david", "smith", "is", "software", "engineer"]
-
-// Word Hashing (FNV-1a):
-// "david" → hashWord("david") → 1234567  (int32)
-// "smith" → hashWord("smith") → 9876543  (int32)
-
-// Index structure (same as other fields):
-// __words__ → 1234567 → RoaringBitmap{entity1, entity5, ...}
-// __words__ → 9876543 → RoaringBitmap{entity1, entity3, ...}
-```
-
-**Design Decisions**:
-- **Max 50 words per entity**: Prevents index bloat for large documents
-- **FNV-1a hashing**: Fast, low collision rate, int32 output
-- **Min word length 2 chars**: Filters out noise words
-- **Lowercase normalization**: Case-insensitive matching
-- **Automatic integration**: Words extracted via `extractIndexableFields()`
-
-**Hybrid Search**: Text results combined with vector results using Reciprocal Rank Fusion (RRF):
-```typescript
-// RRF formula: score(d) = sum(1 / (k + rank(d)))
-// where k = 60 (standard constant)
-// alpha = weight for semantic (0 = text only, 1 = semantic only)
-```
-
-### Query Algorithm
-
-**Exact Match Query**:
-```typescript
-async getIds(field: string, value: any): Promise<string[]> {
-  // 1. Load sparse index for field
-  const sparseIndex = await this.loadSparseIndex(field)
-
-  // 2. Find candidate chunks using bloom filters
-  const candidateChunks = sparseIndex.findChunksForValue(value)
-  //    → Bloom filter checks all chunks (~1ms)
-  //    → Returns only chunks that *might* contain value
-
-  // 3. Load candidate chunks and collect IDs
-  const results = []
-  for (const chunkId of candidateChunks) {
-    const chunk = await this.chunkManager.loadChunk(field, chunkId)
-    const ids = chunk.entries.get(value)
-    if (ids) results.push(...ids)
-  }
-
-  return results
-}
-```
-
-**Range Query**:
-```typescript
-async getIdsForRange(field: string, min: any, max: any): Promise<string[]> {
-  // 1. Load sparse index for field
-  const sparseIndex = await this.loadSparseIndex(field)
-
-  // 2. Find candidate chunks using zone maps
-  const candidateChunks = sparseIndex.findChunksForRange(min, max)
-  //    → Check zoneMap.min and zoneMap.max for each chunk
-  //    → Skip chunks where max < min or min > max
-
-  // 3. Load chunks and filter values
-  const results = []
-  for (const chunkId of candidateChunks) {
-    const chunk = await this.chunkManager.loadChunk(field, chunkId)
-    for (const [value, ids] of chunk.entries) {
-      if (value >= min && value <= max) {
-        results.push(...ids)
-      }
-    }
-  }
-
-  return results
-}
-```
-
-**Benefits**:
-- Bloom filters: Skip 99% of irrelevant chunks (exact match)
-- Zone maps: Skip entire chunks that fall outside range
-- Adaptive chunking: ~50 values per chunk optimizes I/O
-- Immediate flushing: No need for dirty tracking or batch writes
-
-### Temporal Bucketing
-
-**Problem Solved**: High-cardinality timestamp fields created massive file pollution.
-- Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!)
-
-**Solution**: Automatic bucketing of temporal fields to 1-minute intervals.
-
-```typescript
-// In normalizeValue(value, field):
-if (field && typeof value === 'number') {
-  const fieldLower = field.toLowerCase()
-  const isTemporal = fieldLower.includes('time') ||
-                     fieldLower.includes('date') ||
-                     fieldLower.includes('accessed') ||
-                     fieldLower.includes('modified') ||
-                     fieldLower.includes('created') ||
-                     fieldLower.includes('updated')
-
-  if (isTemporal) {
-    // Bucket to 1-minute intervals
-    const bucketSize = 60000 // milliseconds
-    const bucketed = Math.floor(value / bucketSize) * bucketSize
-    return bucketed.toString()
-  }
-}
-```
-
-**Benefits**:
-- ✅ Reduces 575 unique timestamps → ~10 buckets
-- ✅ File count: 358,407 → ~4,600 (98.7% reduction)
-- ✅ Zero configuration - automatic field name detection
-- ✅ Still enables range queries (not excluded like before)
-- ✅ 1-minute precision sufficient for most use cases
-
-**Field Name Detection**: Automatically buckets fields with these keywords:
-- `time`, `date`, `accessed`, `modified`, `created`, `updated`
-- Examples: `timestamp`, `createdAt`, `lastModified`, `birthdate`, `eventTime`
-
-### Operations
-
-```typescript
-// Add to index (src/brainy.ts:387)
-await this.metadataIndex.addToIndex(id, metadata)
-
-// Query exact match
-const ids = await this.metadataIndex.getIds('status', 'active')
-
-// Query range
-const ids = await this.metadataIndex.getIdsForFilter({
-  publishDate: { greaterThan: 1640995200000 }
-})
-
-// Filter discovery (what values exist for a field)
-const values = await this.metadataIndex.getFilterValues('status')
-// → ['active', 'archived', 'draft']
-
-// Statistics (O(1))
-const totalEntities = this.metadataIndex.getTotalEntityCount()
-const typeBreakdown = this.metadataIndex.getAllEntityCounts()
-// → Map { 'character': 127, 'item': 89, 'location': 45 }
-```
-
-### Excluded Fields
-
-Some fields are excluded from indexing to prevent pollution:
-
-```typescript
-const DEFAULT_EXCLUDE_FIELDS = [
-  'id',               // Primary key (redundant to index)
-  'uuid',             // Alternative primary key
-  'vector',           // High-dimensional data
-  'embedding',        // Same as vector
-  'content',          // Large text content
-  'description',      // Large text content
-  'metadata',         // Nested object (too large)
-  'data'              // Generic nested object
-]
-```
-
-**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
-
-## 2. Vector Index - Vector Similarity Search
-
-**Purpose**: O(log n) semantic similarity search using vector embeddings.
-
-The default JS implementation is `JsHnswVectorIndex`; an optional native acceleration package (`@soulcraft/cor`) can register a higher-performing `VectorIndexProvider` through the plugin system. The public API stays the same either way.
-
-### Internal Architecture
-
-```typescript
-class JsHnswVectorIndex {
-  // Per-noun indexes for efficiency
-  private nouns: Map<string, HNSWNoun> = new Map()
-
-  // Global entry point for search
-  private entryPointId: string | null = null
-  private maxLevel = 0
-
-  // Shared memory management
-  private unifiedCache: UnifiedCache
-  private storage: BaseStorage | null = null
-}
-
-// Each noun has its own HNSW graph
-class HNSWNoun {
-  noun: string
-  nodes: Map<string, HNSWNode>
-  entryPointId: string | null
-  maxLevel: number
-}
-
-// Each node in the graph
-class HNSWNode {
-  id: string
-  vector: Vector | null  // Lazy-loaded from storage
-  level: number
-  connections: Map<number, string[]>  // level → neighbor IDs
-}
-```
-
-### Hierarchical Graph Structure
-
-The default vector index builds a multi-layered graph:
-
-```
-Layer 2: [entry] ←→ [node1]                    (sparse, long-range connections)
-           ↓         ↓
-Layer 1: [entry] ←→ [node1] ←→ [node2] ←→ [node3]  (medium density)
-           ↓         ↓         ↓         ↓
-Layer 0: [entry] ←→ [node1] ←→ [node2] ←→ [node3] ←→ [node4] ←→ [node5]  (dense, all nodes)
-```
-
-**Search Algorithm**:
-1. Start at entry point in top layer
-2. Greedy search for nearest neighbor in current layer
-3. Move down to next layer with found neighbor
-4. Repeat until reaching layer 0
-5. Return k nearest neighbors
-
-**Complexity**: O(log n) due to hierarchical structure
-
-### Adaptive Vector Loading
-
-Vectors are lazy-loaded on demand based on memory availability:
-
-```typescript
-private async getVectorSafe(noun: HNSWNoun): Promise<Vector> {
-  // Check UnifiedCache first
-  const cached = this.unifiedCache.get(noun.id)
-  if (cached) return cached
-
-  // Load from storage if memory available
-  if (this.unifiedCache.canCache()) {
-    const vector = await this.storage.loadVector(noun.id)
-    this.unifiedCache.set(noun.id, vector)
-    return vector
-  }
-
-  // Load transiently if memory pressure
-  return await this.storage.loadVector(noun.id)
-}
-```
-
-### Operations
-
-```typescript
-// Add entity (src/brainy.ts:add)
-await this.index.addEntity(id, vector, noun)
-
-// Search for similar vectors
-const results = await this.index.search(queryVector, k, threshold)
-// Returns: Array<{id: string, similarity: number}>
-
-// Rebuild from storage
-await this.index.rebuild()
-```
-
-## 3. GraphAdjacencyIndex - O(1) Relationship Traversal
-
-**Purpose**: Constant-time neighbor lookups regardless of graph size.
-
-### Internal Architecture
-
-```typescript
-class GraphAdjacencyIndex {
-  // O(1) bidirectional lookups
-  private sourceIndex = new Map<string, Set<string>>()  // sourceId → targetIds
-  private targetIndex = new Map<string, Set<string>>()  // targetId → sourceIds
-
-  // Full relationship data
-  private verbIndex = new Map<string, GraphVerb>()      // verbId → metadata
-
-  // Statistics
-  private relationshipCountsByType = new Map<string, number>()
-
-  // Shared memory
-  private unifiedCache: UnifiedCache
-  private storage: BaseStorage
-}
-```
-
-### Key Innovation: Bidirectional Adjacency
-
-**Core Insight**: Store BOTH directions of each relationship for O(1) lookups.
-
-```typescript
-// Example: Alice KNOWS Bob
-// verbId = "verb-123"
-
-// Source index: Alice → Bob
-sourceIndex.set('alice', Set(['bob']))
-
-// Target index: Bob ← Alice
-targetIndex.set('bob', Set(['alice']))
-
-// Full metadata
-verbIndex.set('verb-123', {
-  id: 'verb-123',
-  verb: 'knows',
-  source: 'alice',
-  target: 'bob',
-  metadata: { since: 2020 }
-})
-```
-
-**Result**: Finding Alice's friends OR Bob's friends is O(1) - just one Map lookup!
-
-### Operations
-
-```typescript
-// Add relationship (src/brainy.ts:relate)
-await this.graphIndex.addRelationship(verbId, sourceId, targetId, verb)
-
-// Get neighbors (O(1) per hop)
-const outgoing = await this.graphIndex.getNeighbors(id, 'out')  // Who does id point to?
-const incoming = await this.graphIndex.getNeighbors(id, 'in')   // Who points to id?
-const both = await this.graphIndex.getNeighbors(id, 'both')     // All neighbors
-
-// Get relationships
-const verbs = await this.graphIndex.getRelationships(sourceId, targetId)
-
-// Statistics (O(1))
-const totalRelationships = this.graphIndex.getTotalRelationshipCount()
-const byType = this.graphIndex.getRelationshipCountsByType()
-// → Map { 'knows': 45, 'created': 23, 'located_at': 12 }
-```
-
-### Graph Traversal
-
-The index supports multi-hop traversal:
-
-```typescript
-// Find all entities within 2 hops
-const reachable = await this.graphIndex.traverse({
-  startId: 'alice',
-  depth: 2,
-  direction: 'out'
-})
-// Complexity: O(V + E) breadth-first search, but each neighbor lookup is O(1)
-```
-
-## Shared Memory Management: UnifiedCache
-
-All three main indexes share a single **UnifiedCache** instance for coordinated memory management.
-
-### Architecture
-
-```typescript
-class UnifiedCache {
-  private cache: Map<string, CachedItem> = new Map()
-  private maxSize: number
-  private currentSize: number = 0
-  private evictionPolicy: 'LRU' | 'LFU' = 'LRU'
-}
-
-// Each index gets the same cache instance
-const unifiedCache = new UnifiedCache({ maxSize: 1000 })
-this.metadataIndex = new MetadataIndexManager(storage, { unifiedCache })
-this.vectorIndex = new JsHnswVectorIndex(storage, { unifiedCache })
-this.graphIndex = new GraphAdjacencyIndex(storage, { unifiedCache })
-```
-
-### Benefits
-
-1. **Fair Resource Allocation**: All indexes compete for the same memory pool
-2. **Prevents Monopolization**: No single index can starve others of memory
-3. **Coordinated Eviction**: LRU eviction across all cached items system-wide
-4. **Memory Pressure Handling**: Automatic cache shrinking when memory is tight
-5. **Adaptive Loading**: Indexes load data transiently under memory pressure
-
-### Cache Key Patterns
-
-Each index uses different key prefixes:
-
-```typescript
-// Metadata index
-cache.set(`meta:${field}:${value}`, indexEntry)
-
-// Vector index
-cache.set(`vector:${id}`, vectorData)
-
-// Graph index
-cache.set(`graph:${sourceId}`, neighbors)
-
-// Deleted items (no caching needed - uses Set)
-```
-
-## How Indexes Work Together
-
-### 1. Entity Creation (`brainy.add()`)
-
-```typescript
-// src/brainy.ts:add()
-async add(params: AddParams): Promise<string> {
-  const id = generateId()
-  const vector = await this.embedder(params.content)
-
-  // Add to metadata index (field filtering)
-  await this.metadataIndex.addToIndex(id, params.metadata)
-
-  // Add to vector index (vector search)
-  await this.index.addEntity(id, vector, params.noun)
-
-  // Relationships added via separate relate() calls
-
-  return id
-}
-```
-
-### 2. Entity Search (`brainy.find()`)
-
-```typescript
-// src/brainy.ts:find()
-async find(query: FindQuery): Promise<Result[]> {
-  let results: Result[] = []
-
-  // Step 1: Metadata filtering (fast pre-filter)
-  if (query.where) {
-    const filteredIds = await this.metadataIndex.getIdsForFilter(query.where)
-    results = await this.getEntitiesByIds(filteredIds)
-  }
-
-  // Step 2: Vector similarity search (semantic ranking)
-  if (query.like) {
-    const queryVector = await this.embedder(query.like)
-    const vectorResults = await this.index.search(queryVector, query.limit)
-
-    // Intersect or union with metadata results
-    results = this.combineResults(results, vectorResults)
-  }
-
-  // Step 3: Graph traversal (relationship filtering)
-  if (query.connected) {
-    const connectedIds = await this.graphIndex.traverse(query.connected)
-    results = results.filter(r => connectedIds.includes(r.id))
-  }
-
-  return results
-}
-```
-
-### 3. Entity Update (`brainy.update()`)
-
-```typescript
-// src/brainy.ts:update()
-async update(params: UpdateParams): Promise<void> {
-  const existing = await this.get(params.id)
-
-  // Update metadata index (remove old, add new)
-  await this.metadataIndex.removeFromIndex(params.id, existing.metadata)
-  await this.metadataIndex.addToIndex(params.id, params.metadata)
-
-  // Update vector index (re-embed if content changed)
-  if (params.content) {
-    const newVector = await this.embedder(params.content)
-    await this.index.updateEntity(params.id, newVector)
-  }
-
-  // Graph relationships unchanged (managed separately)
-}
-```
-
-### 4. Statistics (`brainy.stats()`)
-
-All indexes provide O(1) statistics:
-
-```typescript
-// src/brainy.ts:stats()
-async stats(): Promise<Statistics> {
-  return {
-    // From metadata index
-    entities: this.metadataIndex.getTotalEntityCount(),
-    entityTypes: this.metadataIndex.getAllEntityCounts(),
-
-    // From graph index
-    relationships: this.graphIndex.getTotalRelationshipCount(),
-    relationshipTypes: this.graphIndex.getRelationshipCountsByType(),
-
-    // From vector index
-    vectorIndexSize: this.index.getSize()
-  }
-}
-```
-
-### 5. Index Rebuilding (Lazy Loading Support)
-
-**Two modes of index loading:**
-
-#### Mode 1: Auto-Rebuild on init() (default)
-
-```typescript
-// src/brainy.ts:init()
-async init(): Promise<void> {
-  // When disableAutoRebuild: false (default)
-  const metadataStats = await this.metadataIndex.getStats()
-  const vectorIndexSize = this.index.size()
-  const graphIndexSize = await this.graphIndex.size()
-
-  if (metadataStats.totalEntries === 0 ||
-      vectorIndexSize === 0 ||
-      graphIndexSize === 0) {
-    // Rebuild all indexes in parallel
-    await Promise.all([
-      metadataStats.totalEntries === 0 ? this.metadataIndex.rebuild() : Promise.resolve(),
-      vectorIndexSize === 0 ? this.index.rebuild() : Promise.resolve(),
-      graphIndexSize === 0 ? this.graphIndex.rebuild() : Promise.resolve()
-    ])
-  }
-}
-```
-
-#### Mode 2: Lazy Loading on First Query
-
-```typescript
-// When disableAutoRebuild: true
-const brain = new Brainy({
-  storage: { type: 'filesystem' },
-  disableAutoRebuild: true  // Enable lazy loading
-})
-
-await brain.init()  // Returns instantly, indexes empty
-
-// First query triggers lazy rebuild
-const results = await brain.find({ limit: 10 })
-// → Calls ensureIndexesLoaded() (line 4617)
-// → Rebuilds all 3 main indexes with concurrency control
-// → Subsequent queries are instant (0ms check)
-```
-
-**Performance:**
-- First query with lazy loading: ~50-200ms rebuild (1K-10K entities)
-- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
-- Subsequent queries: 0ms check (instant)
-
-See [initialization-and-rebuild.md](./initialization-and-rebuild.md) for detailed lazy loading implementation.
-
-## Triple Intelligence Integration
-
-The **TripleIntelligenceSystem** (`src/triple/TripleIntelligenceSystem.ts`) combines all three core indexes:
-
-```typescript
-class TripleIntelligenceSystem {
-  constructor(
-    private metadataIndex: MetadataIndexManager,
-    private vectorIndex: VectorIndexProvider,
-    private graphIndex: GraphAdjacencyIndex,
-    private embedder: EmbedderFunction,
-    private storage: BaseStorage
-  ) {}
-
-  async query(nlpQuery: string): Promise<Result[]> {
-    // Parse natural language
-    const parsed = await this.parseQuery(nlpQuery)
-
-    // Execute across all three indexes
-    const [metadataResults, vectorResults, graphResults] = await Promise.all([
-      this.metadataIndex.getIdsForFilter(parsed.filters),
-      this.vectorIndex.search(parsed.vector, parsed.limit),
-      this.graphIndex.traverse(parsed.graphConstraints)
-    ])
-
-    // Fuse results with weighted scoring
-    return this.fuseResults(metadataResults, vectorResults, graphResults)
-  }
-}
-```
-
-## Performance Characteristics
-
-### Operation Complexity by Index
-
-| Operation | MetadataIndexManager | TypeAwareVectorIndex | GraphAdjacencyIndex |
-|-----------|---------------------|-------------------|---------------------|
-| **Add** | O(1) per field | O(log n) | O(1) |
-| **Remove** | O(1) per field | O(log n) | O(1) |
-| **Exact lookup** | O(1) | N/A | O(1) |
-| **Range query** | O(log n) + O(k) | N/A | N/A |
-| **Similarity search** | N/A | O(log n) | N/A |
-| **Neighbor lookup** | N/A | N/A | O(1) |
-| **Statistics** | O(1) | O(1) | O(1) |
-| **Rebuild** | O(n) | O(n) | O(n) |
-
-Where:
-- n = total number of entities
-- k = number of matching results
-
-**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for the vector index).
-
-### Memory Footprint
-
-| Index | Per-Entity Memory | Notes |
-|-------|-------------------|-------|
-| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) |
-| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
-| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional verb-id references in 2 LSM-trees |
-
-**Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship
-
-**Sub-index memory:**
-- ChunkManager: ~20 bytes per chunk descriptor
-- EntityIdMapper: ~32 bytes per UUID mapping (50-90% savings vs Set\<string\>)
-- LSM-trees: ~200 bytes per relationship (SSTable storage)
-
-### Scalability
-
-All indexes scale gracefully. The cost of each stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion:
-
-| Query stage | Complexity | Scaling behavior |
-|-------------|------------|------------------|
-| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
-| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
-| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
-| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
-| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
-
-**Key observations**:
-- Graph queries stay O(1) regardless of scale
-- Metadata filtering scales sub-linearly
-- Vector search degrades gracefully due to the hierarchical index
-- Combined queries remain fast even at scale
-
-## Best Practices
-
-### When to Use Each Index
-
-**MetadataIndex**:
-- Filtering by exact field values (status, type, category)
-- Range queries on numeric/temporal fields (dates, prices, counts)
-- Field discovery (what filters are available)
-- Type-based querying (find all characters, all items)
-
-**Vector Index**:
-- Semantic similarity search ("find similar documents")
-- Content-based retrieval ("find posts about AI")
-- Fuzzy matching (when exact matches aren't required)
-- Recommendation systems (find related items)
-
-**GraphAdjacencyIndex**:
-- Relationship queries ("who knows whom")
-- Path finding ("how are these entities connected")
-- Network analysis ("find communities")
-- Multi-hop traversal ("friends of friends")
-
-**Note**: Soft-delete functionality is not currently integrated. Brainy uses hard deletes via storage layer.
-
-### Query Optimization
-
-1. **Start with metadata filters** - They're fastest and most selective
-2. **Use graph constraints** - O(1) lookups significantly reduce search space
-3. **Vector search last** - Most expensive, best used on pre-filtered set
-4. **Leverage temporal bucketing** - Timestamp range queries work efficiently
-5. **Monitor statistics** - Use O(1) stats methods for cardinality estimation
-
-### Memory Management
-
-1. **Configure UnifiedCache appropriately** - Balance between speed and memory
-2. **Use lazy loading** - Vector index loads vectors on-demand
-3. **Monitor cache hit rates** - Adjust cache size if hit rate is low
-4. **Consider storage adapter** - Memory = fastest, filesystem = persistent
-
-## Related Documentation
-
-- [Find System](../FIND_SYSTEM.md) - Query-centric view of index usage
-- [Triple Intelligence](./triple-intelligence.md) - Advanced query system
-- [Storage Architecture](./storage-architecture.md) - Storage layer details
-- [Performance Guide](../PERFORMANCE.md) - Performance tuning
-- [Overview](./overview.md) - High-level architecture
-
-## Summary: Index Hierarchy
-
-### Level 1: Main Indexes (3)
-All have rebuild() methods and are covered by lazy loading:
-1. **TypeAwareVectorIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403`
-2. **MetadataIndexManager** - `src/utils/metadataIndex.ts:2318`
-3. **GraphAdjacencyIndex** - `src/graph/graphAdjacencyIndex.ts:389`
-
-### Level 2: Sub-Indexes (~50+)
-Automatically managed by parent rebuild():
-- **42 type-specific vector indexes** (one per NounType)
-- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes)
-- **2 LSM-trees** (lsmTreeVerbsBySource, lsmTreeVerbsByTarget — the verb set is the single adjacency source of truth; neighbor reads derive from live verbs so removals are honored)
-- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
-
-### Lazy Loading
-- **Mode 1**: Auto-rebuild on init() (default)
-- **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`)
-- **Concurrency-safe**: Mutex prevents duplicate rebuilds
-- **Performance**: First query ~50-200ms, subsequent queries instant
-
-### Total Functional Index Count
-- **3 main indexes** with independent rebuild() methods
-- **~50+ sub-components** managed automatically
-- **All covered** by rebuildIndexesIfNeeded() or built-in lazy initialization
-
-## Version History
-
-- **v5.7.7** (November 2025): Added production-scale lazy loading with concurrency control. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added `ensureIndexesLoaded()` helper and `getIndexStatus()` diagnostic.
-- **v3.43.0** (October 2025): Migrated from `roaring` (native C++) to `roaring-wasm` (WebAssembly) for universal compatibility. No API changes - maintains identical RoaringBitmap32 interface. Benefits: works in all environments (Node.js, browsers, serverless) without build tools, zero compilation errors, simpler developer experience. 90% memory savings and hardware-accelerated operations unchanged.
-- **v3.42.0** (October 2025): Replaced flat file indexing with adaptive chunked sparse indexing. Bloom filters + zone maps for O(1) exact match and O(log n) range queries. 630x file reduction (560k → 89 files). Removed dual code paths.
-- **v3.41.0** (October 2025): Added automatic temporal bucketing to MetadataIndex
-- **v3.40.0** (October 2025): Enhanced batch processing for imports
-- **v3.0.0** (September 2025): Introduced 3-tier index architecture with UnifiedCache

docs/architecture/initialization-and-rebuild.md

@@ -1,713 +0,0 @@
-# Initialization and Rebuild Processes
-
-This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
-
-## Core Principle: All Indexes Are Disk-Based
-
-**KEY INSIGHT**: All indexes in Brainy are already disk-based. There is no need for snapshots or separate backup mechanisms. Initialization simply loads the right amount of data from storage into memory based on available resources.
-
-### What Gets Persisted
-
-| Index | Persisted Data | Storage Method | Since Version |
-|-------|---------------|----------------|---------------|
-| **MetadataIndex** | Field registry + chunked sparse indices with bloom filters + zone maps | `storage.saveMetadata()` | v3.42.0 (chunks), v4.2.1 (registry) |
-| **Vector Index** | Vector embeddings + graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
-| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
-| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
-
-#### MetadataIndex Persistence Details
-
-The MetadataIndex now persists two components:
-
-1. **Field Registry** (`__metadata_field_registry__`): Directory of indexed fields for O(1) discovery
-   - Size: ~4-8KB (50-200 fields typical)
-   - Enables instant cold starts by discovering persisted indices
-   - Auto-saved during every flush operation
-
-2. **Sparse Indices** (`__sparse_index__<field>`): Per-field index directories
-   - Contains chunk metadata, zone maps, and bloom filters
-   - Lazy-loaded via UnifiedCache on first query
-
-3. **Chunks** (`__metadata_chunk__<field>_<chunkId>`): Actual inverted index data
-   - Roaring bitmaps for compressed entity ID storage
-   - Loaded on-demand based on query patterns
-
-All storage operations use the **StorageAdapter** interface, which works with FileSystem and Memory backends.
-
-## Initialization Process
-
-### 1. Lazy Initialization Pattern
-
-All indexes use lazy initialization - they don't load data until first use:
-
-```typescript
-// Example: GraphAdjacencyIndex
-class GraphAdjacencyIndex {
-  private initialized = false
-
-  private async ensureInitialized(): Promise<void> {
-    if (this.initialized) return
-
-    // Initialize LSM-trees from storage
-    await this.lsmTreeSource.init()
-    await this.lsmTreeTarget.init()
-
-    this.initialized = true
-  }
-
-  // Every public method calls ensureInitialized() first
-  async getNeighbors(id: string): Promise<string[]> {
-    await this.ensureInitialized()  // Lazy init!
-    // ... actual logic
-  }
-}
-```
-
-**Benefits**:
-- Zero-cost abstraction: No initialization overhead if index not used
-- Faster startup: Indexes initialize in parallel on first use
-- Lower memory: Only used indexes consume memory
-
-### 2. Brain Initialization Flow
-
-When you create a `Brain` instance and call `init()`, behavior depends on the `disableAutoRebuild` configuration:
-
-#### Mode 1: Auto-Rebuild on init() (Default)
-
-```typescript
-// src/brainy.ts (lines 192-237)
-async init(): Promise<void> {
-  const initStartTime = Date.now()
-
-  // STEP 1: Initialize storage and unified cache
-  await this.storage.init()
-
-  // STEP 2: Check index sizes (lazy initialization triggers here)
-  const metadataStats = await this.metadataIndex.getStats()
-  const vectorIndexSize = this.index.size()
-  const graphIndexSize = await this.graphIndex.size()
-
-  // STEP 3: Rebuild empty indexes from storage in parallel
-  if (metadataStats.totalEntries === 0 ||
-      vectorIndexSize === 0 ||
-      graphIndexSize === 0) {
-
-    const rebuildStartTime = Date.now()
-    await Promise.all([
-      metadataStats.totalEntries === 0
-        ? this.metadataIndex.rebuild()
-        : Promise.resolve(),
-      vectorIndexSize === 0
-        ? this.index.rebuild()
-        : Promise.resolve(),
-      graphIndexSize === 0
-        ? this.graphIndex.rebuild()
-        : Promise.resolve()
-    ])
-
-    const rebuildDuration = Date.now() - rebuildStartTime
-    console.log(`✅ All indexes rebuilt in ${rebuildDuration}ms`)
-  }
-
-  // STEP 4: Log statistics
-  const stats = await this.stats()
-  console.log(`📊 Brain initialized with ${stats.entities} entities`)
-}
-```
-
-**Timeline** (typical cold start with 10K entities):
-- 0-50ms: Storage adapter initialization
-- 50-100ms: Field registry loading (O(1) discovery of persisted indices)
-- 100-200ms: Index lazy initialization (LSM-tree loading)
-- 200-500ms: Cache warming (preload common fields)
-- **No rebuild needed!** Registry discovers existing indices
-- Total: ~0.5-1 second (instant cold starts)
-
-**Timeline** (cold start WITHOUT field registry - first run only):
-- 0-50ms: Storage adapter initialization
-- 50-100ms: Index lazy initialization
-- 100-2000ms: One-time rebuild to create indices
-- Total: ~1-3 seconds (one time only)
-
-#### Mode 2: Lazy Loading on First Query
-
-When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query:
-
-```typescript
-// User code
-const brain = new Brainy({
-  storage: { type: 'filesystem' },
-  disableAutoRebuild: true  // Enable lazy loading
-})
-
-await brain.init()  // Returns instantly (0-10ms)
-
-// First query triggers lazy rebuild
-const results = await brain.find({ limit: 10 })
-// → Calls ensureIndexesLoaded() internally (brainy.ts:4617)
-// → Rebuilds all 3 main indexes with concurrency control
-// → Returns results (~50-200ms total for 1K-10K entities)
-
-// Subsequent queries are instant
-const more = await brain.find({ limit: 100 })  // 0ms check, instant
-```
-
-**ensureIndexesLoaded() Implementation** (brainy.ts:4617-4664):
-```typescript
-private async ensureIndexesLoaded(): Promise<void> {
-  // Fast path: Already loaded
-  if (this.lazyRebuildCompleted) {
-    return  // 0ms
-  }
-
-  // Concurrency control: Wait for in-progress rebuild
-  if (this.lazyRebuildInProgress && this.lazyRebuildPromise) {
-    await this.lazyRebuildPromise  // Wait for same rebuild
-    return
-  }
-
-  // Check if storage has data
-  const entities = await this.storage.getNouns({ pagination: { limit: 1 } })
-  const hasData = (entities.totalCount && entities.totalCount > 0) || entities.items.length > 0
-
-  if (!hasData) {
-    this.lazyRebuildCompleted = true
-    return
-  }
-
-  // Start lazy rebuild with mutex
-  this.lazyRebuildInProgress = true
-  this.lazyRebuildPromise = this.rebuildIndexesIfNeeded(true)
-    .then(() => {
-      this.lazyRebuildCompleted = true
-    })
-    .finally(() => {
-      this.lazyRebuildInProgress = false
-      this.lazyRebuildPromise = null
-    })
-
-  await this.lazyRebuildPromise
-}
-```
-
-**Lazy Loading Performance:**
-- First query: ~50-200ms (1K-10K entities) - triggers rebuild
-- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
-- Subsequent queries: 0ms check (instant)
-- Zero-config: Works automatically, no code changes needed
-
-**Use Cases for Lazy Loading:**
-- **Serverless/Edge**: Minimize cold start time, indexes load on demand
-- **Development**: Faster restarts during development
-- **Large datasets**: Defer index loading until actually needed
-- **Read-heavy workloads**: Write operations don't wait for index rebuild
-
-## Rebuild Process
-
-### What "Rebuild" Actually Means
-
-**IMPORTANT**: "Rebuild" does NOT mean recomputing data. It means:
-1. **Load persisted data** from storage (vector index connections, metadata chunks, LSM-tree SSTables)
-2. **Populate in-memory structures** (Maps, Sets, graphs)
-3. **Apply adaptive caching** (preload vectors if small dataset, lazy load if large)
-
-**Complexity**: O(N) - linear scan through storage, NOT O(N log N) recomputation!
-
-### 1. Vector Index Rebuild (Correct Pattern)
-
-```typescript
-// src/hnsw/hnswIndex.ts (lines 809-947)
-public async rebuild(options: {
-  lazy?: boolean
-  batchSize?: number
-  onProgress?: (loaded: number, total: number) => void
-} = {}): Promise<void> {
-  // STEP 1: Clear in-memory structures
-  this.clear()
-
-  // STEP 2: Load system data (entry point, max level)
-  const systemData = await this.storage.getHNSWSystem()
-  this.entryPointId = systemData.entryPointId
-  this.maxLevel = systemData.maxLevel
-
-  // STEP 3: Determine preloading strategy (adaptive caching)
-  const totalNouns = await this.storage.getNounCount()
-  const vectorMemory = totalNouns * 384 * 4  // 384 dims × 4 bytes
-  const availableCache = this.unifiedCache.getRemainingCapacity()
-  const shouldPreload = vectorMemory < availableCache * 0.3
-
-  // STEP 4: Load entities with persisted vector index connections
-  let hasMore = true
-  let cursor: string | undefined = undefined
-
-  while (hasMore) {
-    const result = await this.storage.getNouns({
-      pagination: { limit: 1000, cursor }
-    })
-
-    for (const nounData of result.items) {
-      // Load vector graph data from storage (NOT recomputed!)
-      const hnswData = await this.storage.getHNSWData(nounData.id)
-
-      // Create noun with restored connections
-      const noun: HNSWNoun = {
-        id: nounData.id,
-        vector: shouldPreload ? nounData.vector : [],  // Adaptive!
-        connections: new Map(),
-        level: hnswData.level
-      }
-
-      // Restore connections from persisted data
-      for (const [levelStr, nounIds] of Object.entries(hnswData.connections)) {
-        const level = parseInt(levelStr, 10)
-        noun.connections.set(level, new Set<string>(nounIds))
-      }
-
-      // Just add to memory (no recomputation!)
-      this.nouns.set(nounData.id, noun)
-    }
-
-    hasMore = result.hasMore
-    cursor = result.nextCursor
-  }
-}
-```
-
-**Key Points**:
-- ✅ Loads vector index connections from storage via `getHNSWData()`
-- ✅ Uses adaptive caching (preload vectors if < 30% of available cache)
-- ✅ O(N) complexity - just loads existing data
-- ❌ Does NOT call `addItem()` which would recompute connections (O(N log N))
-
-### 2. TypeAwareVectorIndex Rebuild (Fixed in v3.45.0)
-
-**Critical Architectural Fix**: The type-aware vector index previously had TWO major bugs:
-
-1. **Bug #1**: Called `addItem()` during rebuild → O(N log N) recomputation instead of O(N) loading
-2. **Bug #2**: Loaded ALL nouns 31 times in parallel (once per type) → O(31*N) complexity causing timeouts
-
-Both were fixed in v3.45.0 by loading ALL nouns ONCE and routing to correct type indexes:
-
-```typescript
-// src/hnsw/typeAwareHNSWIndex.ts (lines 379-571)
-public async rebuild(options?: {
-  lazy?: boolean
-  batchSize?: number
-  onProgress?: (loaded: number, total: number) => void
-}): Promise<void> {
-  // STEP 1: Clear all type-specific indexes
-  for (const index of this.typeIndexes.values()) {
-    index.clear()
-  }
-
-  // STEP 2: Determine preloading strategy (same as vector index)
-  const totalNouns = await this.storage.getNounCount()
-  const vectorMemory = totalNouns * 384 * 4
-  const availableCache = this.unifiedCache.getRemainingCapacity()
-  const shouldPreload = vectorMemory < availableCache * 0.3
-
-  // STEP 3: Load entities grouped by type
-  for (const nounType of ALL_NOUN_TYPES) {
-    const index = this.getOrCreateIndex(nounType)
-    let hasMore = true
-    let cursor: string | undefined = undefined
-
-    while (hasMore) {
-      const result = await this.storage.getNouns({
-        type: nounType,
-        pagination: { limit: 1000, cursor }
-      })
-
-      for (const nounData of result.items) {
-        // CORRECT: Load persisted vector index data (not recomputed!)
-        const hnswData = await this.storage.getHNSWData(nounData.id)
-
-        const noun = {
-          id: nounData.id,
-          vector: shouldPreload ? nounData.vector : [],
-          connections: new Map(),
-          level: hnswData.level
-        }
-
-        // Restore connections from storage
-        for (const [levelStr, nounIds] of Object.entries(hnswData.connections)) {
-          const level = parseInt(levelStr, 10)
-          noun.connections.set(level, new Set<string>(nounIds))
-        }
-
-        // Add to in-memory index (no recomputation!)
-        index.nouns.set(nounData.id, noun)
-      }
-
-      hasMore = result.hasMore
-      cursor = result.nextCursor
-    }
-  }
-}
-```
-
-**Bug Fix**: Changed from `index.addItem()` (recomputation) to direct `nouns.set()` (restoration).
-
-**Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities)
-
-**Correct Pattern**:
-```typescript
-// Load ALL nouns ONCE (not 31 times!)
-while (hasMore) {
-  const result = await storage.getNounsWithPagination({ limit: 1000, cursor })
-
-  for (const noun of result.items) {
-    const type = noun.nounType || noun.metadata?.noun
-    const index = this.getIndexForType(type)
-
-    // Load persisted HNSW data
-    const hnswData = await storage.getHNSWData(noun.id)
-
-    // Restore connections (not recompute!)
-    const restoredNoun = {
-      id: noun.id,
-      vector: shouldPreload ? noun.vector : [],
-      connections: restoreConnections(hnswData),
-      level: hnswData.level
-    }
-
-    // Add to correct type index
-    index.nouns.set(noun.id, restoredNoun)
-  }
-
-  cursor = result.nextCursor
-  hasMore = result.hasMore
-}
-```
-
-**Performance Improvements**:
-- 31x speedup: Load nouns ONCE instead of 31 times (O(N) vs O(31*N))
-- 200-600x speedup: Load from storage instead of recomputing (O(N) vs O(N log N))
-- **Combined**: ~6000x speedup! (150 minutes → 1.5 seconds for 10K entities)
-
-### 3. MetadataIndex Rebuild (v4.2.1+ with Field Registry)
-
-**v4.2.1 Critical Fix**: Field registry persistence eliminates unnecessary rebuilds!
-
-```typescript
-// src/utils/metadataIndex.ts (lines 202-216)
-async init(): Promise<void> {
-  // STEP 1: Load field registry to discover persisted indices
-  // This is THE KEY FIX - O(1) discovery of existing indices
-  await this.loadFieldRegistry()
-
-  // If registry found, fieldIndexes Map is now populated
-  // getStats() will return totalEntries > 0 → skips rebuild!
-
-  // STEP 2: Initialize EntityIdMapper
-  await this.idMapper.init()
-
-  // STEP 3: Warm cache with discovered fields
-  await this.warmCache()
-}
-
-async loadFieldRegistry(): Promise<void> {
-  const registry = await this.storage.getMetadata('__metadata_field_registry__')
-
-  if (registry?.fields) {
-    // Populate fieldIndexes Map from discovered fields
-    // Sparse indices are lazy-loaded when first accessed
-    for (const field of registry.fields) {
-      this.fieldIndexes.set(field, {
-        values: {},
-        lastUpdated: registry.lastUpdated
-      })
-    }
-    // Result: getStats() now returns totalEntries > 0
-    // → Brain skips rebuild, cold start in 2-3 seconds!
-  }
-}
-```
-
-**Rebuild Only Happens If**:
-1. **First run** (no field registry exists yet)
-2. **Registry corruption** (rare)
-3. **Explicit rebuild request** (manual operation)
-
-```typescript
-// Only runs if field registry not found
-async rebuild(): Promise<void> {
-  // STEP 1: Clear in-memory structures
-  this.fieldIndexes.clear()
-
-  // STEP 2: Load all entity metadata and rebuild indices
-  // Sequential batching (25/batch) to prevent socket exhaustion
-  // After rebuild: Field registry saved during next flush()
-
-  // One-time cost: ~2-3 seconds for 1K entities
-}
-```
-
-**Performance Comparison**:
-
-| Version | Cold Start | Discovery Method | Rebuild Needed? |
-|---------|------------|------------------|-----------------|
-| v4.2.0 | 8-9 min | None (always rebuild) | Always |
-| v4.2.1 | 2-3 sec | Field registry O(1) | First run only |
-
-**Key Points**:
-- ✅ Field registry enables O(1) discovery (4-8KB file)
-- ✅ Sparse indices lazy-loaded on first query
-- ✅ Bloom filters + zone maps loaded for fast filtering
-- ✅ One-time rebuild on first run, then instant restarts forever
-- ✅ Automatic: No configuration needed
-
-### 4. GraphAdjacencyIndex Rebuild
-
-```typescript
-// src/graph/graphAdjacencyIndex.ts (lines 279-336)
-async rebuild(): Promise<void> {
-  // STEP 1: Clear in-memory caches
-  this.verbIndex.clear()
-  this.relationshipCountsByType.clear()
-
-  // STEP 2: Load all verbs from storage
-  let hasMore = true
-  let cursor: string | undefined = undefined
-
-  while (hasMore) {
-    const result = await this.storage.getVerbs({
-      pagination: { limit: 1000, cursor }
-    })
-
-    for (const verb of result.items) {
-      // Add to index (which updates LSM-trees)
-      await this.addVerb(verb)
-    }
-
-    hasMore = result.hasMore
-    cursor = result.nextCursor
-  }
-
-  // Note: LSM-trees (lsmTreeSource, lsmTreeTarget) are already
-  // initialized from persisted SSTables during ensureInitialized()
-}
-```
-
-**Key Points**:
-- ✅ LSM-tree SSTables already loaded during `init()`
-- ✅ Rebuild just repopulates verb cache
-- ✅ O(E) complexity where E = number of edges
-
-## Adaptive Memory Management
-
-### Strategy: Preload vs Lazy Load
-
-All indexes use the **UnifiedCache** to determine memory allocation:
-
-```typescript
-// Decision logic (in all indexes)
-const totalDataSize = estimateDataSize()
-const availableCache = unifiedCache.getRemainingCapacity()
-
-if (totalDataSize < availableCache * 0.3) {
-  // PRELOAD: Dataset is small relative to available memory
-  // Load everything into memory for maximum performance
-  shouldPreload = true
-} else {
-  // LAZY LOAD: Dataset is large
-  // Load on-demand with LRU eviction
-  shouldPreload = false
-}
-```
-
-**Thresholds**:
-- **< 30% of available cache**: Preload all vectors
-- **> 30% of available cache**: Lazy load on demand
-
-**Example** (default 100MB cache):
-- 10K entities × 1.5KB = 15MB → **Preload** (15MB < 30MB)
-- 100K entities × 1.5KB = 150MB → **Lazy load** (150MB > 30MB)
-
-### UnifiedCache Integration
-
-```typescript
-// All indexes share the same cache
-const unifiedCache = getGlobalCache()  // Singleton, 100MB default
-
-// MetadataIndex
-this.unifiedCache = unifiedCache
-
-// Vector index
-this.unifiedCache = unifiedCache
-
-// GraphAdjacencyIndex
-this.unifiedCache = unifiedCache
-```
-
-**Benefits**:
-- Fair resource allocation across indexes
-- Prevents any single index from monopolizing memory
-- Coordinated LRU eviction system-wide
-
-## Performance Characteristics
-
-### Rebuild Times (Typical Hardware)
-
-| Dataset Size | Metadata | Vector | Graph | Total (Parallel) |
-|--------------|----------|------|-------|------------------|
-| 1K entities | 50ms | 100ms | 30ms | **150ms** |
-| 10K entities | 200ms | 500ms | 150ms | **600ms** |
-| 100K entities | 1s | 3s | 1s | **3.5s** |
-| 1M entities | 8s | 25s | 10s | **28s** |
-
-**Note**: Parallel rebuild means total time ≈ max(individual times), not sum.
-
-### Memory Overhead
-
-| Index | In-Memory Overhead | Disk Storage |
-|-------|-------------------|--------------|
-| **MetadataIndex** | ~100 bytes/entity | ~500 bytes/entity (chunks) |
-| **Vector Index** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
-| **GraphAdjacencyIndex** | ~128 bytes/relationship | ~200 bytes/relationship (LSM-tree) |
-| **DeletedItemsIndex** | ~40 bytes/deleted ID | ~50 bytes/deleted ID |
-
-**Total overhead** (lazy loading):
-- **In-memory**: ~300 bytes per entity + ~128 bytes per relationship
-- **On-disk**: ~2 KB per entity + ~200 bytes per relationship
-
-### O(N) vs O(N log N) Comparison
-
-**Before fix** (TypeAwareVectorIndex bug):
-```typescript
-// BAD: Recomputes vector index connections during rebuild
-for (const noun of nouns) {
-  await index.addItem(noun)  // O(log N) per item → O(N log N) total
-}
-// 10K entities: ~5 minutes
-```
-
-**After fix** (correct pattern):
-```typescript
-// GOOD: Loads connections from storage
-for (const noun of nouns) {
-  const hnswData = await storage.getHNSWData(noun.id)  // O(1) per item
-  noun.connections = restoreConnections(hnswData)      // O(1) per item
-  index.nouns.set(noun.id, noun)                       // O(1) per item
-}
-// 10K entities: ~500ms (600x faster!)
-```
-
-## Common Patterns
-
-### Cold Start (Empty Storage)
-
-```typescript
-const brain = new Brain({ storage })
-
-// First init: All indexes are empty
-await brain.init()
-// → No rebuild needed, indexes start empty
-
-// Add data
-await brain.add({ content: 'Hello', noun: 'message' })
-
-// Second init: Indexes populated
-const brain2 = new Brain({ storage })
-await brain2.init()
-// → Rebuilds all indexes from storage (~1-3s for 10K entities)
-```
-
-### Warm Start (Storage Already Populated)
-
-```typescript
-const brain = new Brain({ storage })
-
-// Init with existing data
-await brain.init()
-// → Detects non-empty storage
-// → Rebuilds indexes in parallel
-// → Uses adaptive caching (preload if small, lazy if large)
-```
-
-### Manual Rebuild
-
-```typescript
-const brain = new Brain({ storage })
-await brain.init()
-
-// Force rebuild (e.g., after data corruption)
-await brain.metadataIndex.rebuild()
-await brain.index.rebuild()
-await brain.graphIndex.rebuild()
-```
-
-## Troubleshooting
-
-### Slow Rebuild Times
-
-**Symptom**: Rebuild takes minutes instead of seconds
-
-**Diagnosis**:
-```typescript
-// Check if rebuild is recomputing instead of loading
-console.time('rebuild')
-await brain.index.rebuild()
-console.timeEnd('rebuild')
-
-// For 10K entities:
-// - Expected: 500-800ms (loading from storage)
-// - Bug: 5-10 minutes (recomputing vector index connections)
-```
-
-**Solution**: Ensure index is loading from storage, not calling `addItem()` during rebuild.
-
-### High Memory Usage
-
-**Symptom**: Memory usage exceeds expectations
-
-**Diagnosis**:
-```typescript
-// Check if vectors are being preloaded
-const stats = brain.index.getStats()
-console.log('Preloaded vectors:', stats.preloadedVectors)
-
-// Expected:
-// - Small dataset (< 30% cache): Most vectors preloaded
-// - Large dataset (> 30% cache): Few vectors preloaded
-```
-
-**Solution**: Adjust `UnifiedCache` size or force lazy loading:
-```typescript
-const brain = new Brain({
-  storage,
-  cache: { maxSize: 50 * 1024 * 1024 }  // 50MB cache
-})
-```
-
-### Missing Data After Rebuild
-
-**Symptom**: Entities disappear after restart
-
-**Diagnosis**:
-```typescript
-// Check storage persistence
-const nouns = await storage.getNouns({ pagination: { limit: 10 } })
-console.log('Nouns in storage:', nouns.items.length)
-
-// If empty: Storage not persisting
-// If populated: Rebuild not loading correctly
-```
-
-**Solution**: Verify storage adapter is configured correctly (e.g., FileSystem path exists).
-
-## Related Documentation
-
-- [Index Architecture](./index-architecture.md) - Data structures and operations
-- [Storage Architecture](./storage-architecture.md) - Storage layer details
-- [Performance Guide](../PERFORMANCE.md) - Performance tuning
-- [Scaling Guide](../SCALING.md) - Large dataset optimization
-
-## Version History
-
-- **v5.7.7** (November 2025): Added production-scale lazy loading with `ensureIndexesLoaded()` helper. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added concurrency control (mutex) to prevent duplicate rebuilds from concurrent queries. Added `getIndexStatus()` diagnostic method. Zero-config operation - works automatically.
-- **v3.45.0** (October 2025): Fixed type-aware vector index `rebuild()` to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
-- **v3.44.0** (October 2025): GraphAdjacencyIndex migrated to LSM-tree storage for billion-scale relationships
-- **v3.42.0** (October 2025): MetadataIndex migrated to chunked sparse indexing
-- **v3.35.0** (August 2025): Vector index connections first persisted to storage
-- **v3.0.0** (September 2025): Initial 3-tier index architecture

docs/architecture/multiprocess-storage-mixin.md

@@ -1,134 +0,0 @@
-# Design note: multi-process storage mixin
-
-**Status:** Proposed (future minor)
-**Owner:** Brainy core
-**Filed:** 2026-05-15
-**Companion:** [`concepts/storage-adapters`](../concepts/storage-adapters.md)
-
-## Context
-
-Brainy 7.21 added seven storage-adapter methods to support multi-process
-safety:
-
-```
-supportsMultiProcessLocking()
-acquireWriterLock(opts)
-releaseWriterLock()
-readWriterLock()
-startFlushRequestWatcher(cb)
-stopFlushRequestWatcher()
-requestFlushOverFilesystem(timeoutMs)
-```
-
-They live on `BaseStorage` as no-op defaults and are overridden on
-`FileSystemStorage` with real implementations. Any adapter extending
-`FileSystemStorage` (e.g. Cor's `MmapFileSystemStorage`) inherits the
-real ones for free.
-
-This works correctly today. The question is whether the methods *belong*
-on `BaseStorage`.
-
-## The case for moving them out
-
-`BaseStorage` already mixes several concerns:
-- entity / verb CRUD primitives
-- generational record hooks (8.0 MVCC)
-- type-statistics tracking
-- count persistence
-- multi-process safety (new)
-
-Adapters that have no notion of multi-process semantics — `MemoryStorage`,
-cloud adapters (S3, GCS, R2, Azure, OPFS) — still carry seven inherited
-no-ops on their prototype chain. A reader can't tell from the class
-declaration whether a given adapter participates in the locking protocol;
-it has to call `supportsMultiProcessLocking()` and trust the answer.
-
-A cleaner separation:
-
-```typescript
-interface MultiProcessSafeStorage {
-  supportsMultiProcessLocking(): boolean
-  acquireWriterLock(opts?: { force?: boolean }): Promise<WriterLockInfo | null>
-  releaseWriterLock(): Promise<void>
-  readWriterLock(): Promise<WriterLockInfo | null>
-  startFlushRequestWatcher(cb: () => Promise<void>): void
-  stopFlushRequestWatcher(): void
-  requestFlushOverFilesystem(timeoutMs: number): Promise<boolean>
-}
-
-function isMultiProcessSafe(s: BaseStorage): s is BaseStorage & MultiProcessSafeStorage {
-  return typeof (s as any).supportsMultiProcessLocking === 'function'
-    && (s as any).supportsMultiProcessLocking()
-}
-```
-
-Brainy's call sites become:
-
-```typescript
-if (this.config.mode !== 'reader' && isMultiProcessSafe(this.storage)) {
-  await this.storage.acquireWriterLock({ force: this.config.force })
-  // ... TypeScript narrows the rest correctly ...
-}
-```
-
-Benefits:
-- Type system enforces the capability — no more `(this.storage as any).X()`.
-- Adapters that opt out (memory, cloud) are visibly clean.
-- `hasStorageMethod()` defensive helper can stay (still guards
-  build/install artifacts) but doesn't carry the conceptual weight of
-  "did the plugin implement the interface."
-- ADR-style trail for future capability additions: each new capability
-  gets its own interface, opted into explicitly.
-
-## The case against doing it now
-
-- Breaking change for any adapter that already overrides these methods.
-  `FileSystemStorage` is the only one in-tree, but Cor's
-  `MmapFileSystemStorage` inherits from it — interface relocation would
-  ripple through the plugin ecosystem.
-- The current state works. The real failure modes seen in the field
-  were build/install artifacts, not type-system failures.
-- 7.22.0 just shipped a clean fix. Stacking another refactor before
-  consumers absorb it adds churn without urgency.
-- The `hasStorageMethod()` guard accomplishes the same runtime safety the
-  interface narrowing would in TypeScript-aware code.
-
-## Recommendation
-
-**Defer.** Keep the current architecture through the 7.x line. Revisit
-when:
-- A second multi-process capability lands (e.g. distributed-readers
-  coordination) and the natural surface area is more than seven
-  methods. Five+ becomes the moment a separate interface earns its
-  keep.
-- A v8 major is on the table for unrelated reasons. Bundle the
-  interface extraction with that release so consumers absorb both
-  changes in one upgrade.
-
-Until then:
-- Document the inheritance contract (done — see
-  [`concepts/storage-adapters`](../concepts/storage-adapters.md)).
-- Keep `hasStorageMethod()` as the runtime guard.
-- Don't add new methods to `BaseStorage` defaults without re-evaluating
-  the surface-area boundary.
-
-## Migration sketch (when we do it)
-
-For reference, a clean migration path:
-
-1. Add `MultiProcessSafeStorage` interface to `src/storage/coreTypes.ts`.
-2. Move the seven method signatures from `BaseStorage` to the new
-   interface. Default implementations stay on `BaseStorage` but only as
-   private helpers consumed by `FileSystemStorage`'s explicit
-   implementations.
-3. `FileSystemStorage implements MultiProcessSafeStorage` becomes
-   explicit; methods get the `public` modifier with full JSDoc.
-4. Brainy call sites switch from `hasStorageMethod` to
-   `isMultiProcessSafe` type-guard. Keep `hasStorageMethod` for
-   build/install artifact protection.
-5. Document the new contract in `concepts/storage-adapters.md`.
-6. Major-version-bump the `@soulcraft/brainy` peerDep range expected by
-   plugins.
-
-Estimated work: ~half a day of code, ~2 hours of doc/example updates,
-ecosystem coordination via the platform handoff.

docs/architecture/overview.md

@@ -1,150 +0,0 @@
-# Architecture Overview
-
-Brainy is a multi-dimensional AI database that combines vector similarity, graph relationships, and metadata filtering into a unified query system. This document provides a comprehensive overview of the system architecture.
-
-## Core Components
-
-### Brainy (Main Entry Point)
-The central orchestrator that manages all subsystems:
-- **4-Index Architecture**: MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md))
-- **Storage System**: FileSystem and Memory adapters
-- **Augmentation System**: Extensible plugin architecture
-- **Triple Intelligence**: Unified query engine
-
-### Triple Intelligence Engine
-Brainy's revolutionary feature that unifies three types of search:
-- **Vector Search**: Semantic similarity via the pluggable vector index
-- **Graph Traversal**: Relationship-based queries
-- **Field Filtering**: Precise metadata filtering with O(1) performance
-
-```typescript
-// Single query combining all three intelligence types
-const results = await brain.find({
-  like: "machine learning papers",              // Vector similarity
-  connected: { to: "research-team", depth: 2 }, // Graph traversal  
-  where: { published: { $gte: "2024-01-01" } }  // Metadata filtering
-})
-```
-
-### Storage Architecture
-
-```
-brainy-data/
-├── _system/           # System management
-│   └── statistics.json
-├── nouns/            # Entity data storage
-│   └── {uuid}.json
-├── metadata/         # Metadata and indexing
-│   ├── {uuid}.json
-│   ├── __entity_registry__.json
-│   └── __metadata_index__*.json
-├── verbs/            # Relationship storage
-└── locks/            # Concurrent access control
-```
-
-### Vector Index
-Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph:
-- **Performance**: O(log n) search complexity
-- **Configurable recall**: `fast` / `balanced` / `accurate` presets trade recall for latency
-- **Scalable**: Handles millions of vectors per process
-- **Persistent**: Serializable to storage
-- **Swappable**: Replace with a native implementation (such as `@soulcraft/cor`) via the plugin system without changing application code
-
-### Metadata Index Manager
-High-performance field indexing system:
-- **O(1) Lookups**: Inverted index for field→value→IDs mapping
-- **Query Support**: equals, anyOf, allOf, range queries
-- **Chunked Storage**: Supports massive datasets
-- **Auto-indexing**: Automatically maintains indexes on updates
-
-## Performance Characteristics
-
-### Operation Complexity
-- **Vector Search**: O(log n) via the vector index
-- **Field Filtering**: O(1) via inverted indexes
-- **Graph Traversal**: O(V + E) for breadth-first search
-- **Add Operation**: O(log n) for index insertion
-- **Update Operation**: O(1) for metadata updates
-
-### Memory Usage
-- **Base Memory**: ~50MB for core system
-- **Per Vector**: ~1KB (384 dimensions × 4 bytes)
-- **Index Overhead**: ~20% of vector data
-- **Cache Size**: Configurable (default 1000 entries)
-
-### Throughput
-- **Writes**: 1000+ ops/second (with batching)
-- **Reads**: 10,000+ ops/second
-- **Search**: 100+ queries/second (varies by complexity)
-
-## Augmentation System
-
-Brainy's extensible plugin architecture allows for powerful enhancements:
-
-### Core Augmentations
-- **Entity Registry**: High-speed deduplication for streaming data
-- **Batch Processing**: Optimized bulk operations
-- **Request Deduplicator**: Prevents duplicate processing
-
-### Creating Custom Augmentations
-```typescript
-class CustomAugmentation extends BrainyAugmentation {
-  async onInit(brain: Brainy): Promise<void> {
-    // Initialize augmentation
-  }
-  
-  async onAdd(item: any, brain: Brainy): Promise<any> {
-    // Process item before adding
-    return item
-  }
-}
-```
-
-## Caching Strategy
-
-Multi-layered caching for optimal performance:
-- **Search Cache**: LRU cache for query results
-- **Metadata Cache**: Field index caching
-- **Pattern Cache**: NLP pattern matching cache
-- **Entity Cache**: In-memory entity registry
-
-## Integration Points
-
-### Key Objects for Extensions
-- `brain.index`: Access the vector index
-- `brain.metadataIndex`: Access field indexing
-- `brain.graphIndex`: Access graph adjacency index
-- `brain.storage`: Access storage layer
-- `brain.augmentations`: Access augmentation manager
-
-For detailed information about each index, see [Index Architecture](./index-architecture.md).
-
-### Event System
-```typescript
-brain.on('add', (item) => console.log('Item added:', item))
-brain.on('search', (query) => console.log('Search performed:', query))
-brain.on('error', (error) => console.error('Error:', error))
-```
-
-## Best Practices
-
-### When Adding Features
-1. Check if similar functionality exists
-2. Consider if it should be an augmentation
-3. Use existing indexes and caches
-4. Avoid duplicating functionality
-5. Follow the established patterns
-
-### Performance Optimization
-1. Use batch operations for bulk data
-2. Enable appropriate caching
-3. Choose the right storage adapter
-4. Configure index parameters for your use case
-5. Monitor statistics for bottlenecks
-
-## Next Steps
-
-- [Index Architecture](./index-architecture.md) - Deep dive into the 4-index system
-- [Storage Architecture](./storage-architecture.md) - Deep dive into storage system
-- [Triple Intelligence](./triple-intelligence.md) - Advanced query system
-- [API Reference](../api/README.md) - Complete API documentation

docs/architecture/storage-architecture.md

@@ -1,318 +0,0 @@
-# Storage Architecture
-
-> **Updated**: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup
-
-## Storage Structure
-
-### Architecture: Metadata/Vector Separation
-
-Entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
-
-```
-brainy-data/
-├── _system/                              # System metadata (not sharded)
-│   ├── statistics.json                   # Performance metrics
-│   ├── __metadata_field_index__*.json    # Field indexes
-│   └── __metadata_sorted_index__*.json   # Sorted indexes
-│
-├── entities/
-│   ├── nouns/
-│   │   ├── vectors/                      # Vector graph data (sharded by UUID)
-│   │   │   ├── 00/                       # Shard 00 (first 2 hex digits)
-│   │   │   │   ├── 00123456-....json     # Vector + graph connections
-│   │   │   │   └── 00abcdef-....json
-│   │   │   ├── 01/ ... ff/               # 256 shards total
-│   │   │
-│   │   └── metadata/                     # Business data (sharded by UUID)
-│   │       ├── 00/
-│   │       │   ├── 00123456-....json     # Entity metadata only
-│   │       │   └── 00abcdef-....json
-│   │       ├── 01/ ... ff/
-│   │
-│   └── verbs/
-│       ├── vectors/                      # Relationship vectors (sharded)
-│       │   ├── 00/ ... ff/
-│       │
-│       └── metadata/                     # Relationship data (sharded)
-│           ├── 00/ ... ff/
-```
-
-### Why Split Metadata and Vectors?
-
-**Performance at scale:**
-- **Vector search operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
-- **Filtering**: Only load metadata during filtering, not vectors
-- **Pagination**: Load metadata IDs first, fetch vectors/metadata on-demand
-- **Result**: 60-70% reduction in I/O for typical queries at million-entity scale
-
-### UUID-Based Sharding (256 Shards)
-
-**How it works:**
-```typescript
-const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
-const shard = uuid.substring(0, 2) // "3f"
-
-// Vector path:   entities/nouns/vectors/3f/3fa85f64-....json
-// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
-```
-
-**Benefits:**
-- **Uniform distribution**: ~3,900 entities per shard (at 1M scale)
-- **Filesystem optimization**: avoids huge flat directories that bog down `readdir`
-- **Parallel operations**: walk 256 shards in parallel
-- **Predictable**: Deterministic shard assignment
-
-## Storage Adapters
-
-Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` interface:
-
-### FileSystem Storage (Node.js, default)
-```typescript
-const brain = new Brainy({
-  storage: {
-    type: 'filesystem',
-    path: './data'
-  }
-})
-```
-- **Use case**: Server applications, CLI tools, single-node deployments
-- **Performance**: Direct file I/O
-- **Persistence**: Permanent on disk
-- **Features**:
-  - **Batch Delete**: Efficient bulk deletion with retries
-  - **UUID Sharding**: Automatic 256-shard distribution
-
-### Memory Storage
-```typescript
-const brain = new Brainy({
-  storage: {
-    type: 'memory'
-  }
-})
-```
-- **Use case**: Tests, ephemeral workloads, single-process caches
-- **Performance**: No I/O — all data lives in process memory
-- **Persistence**: None — data is lost when the process exits
-
-### Auto
-```typescript
-const brain = new Brainy({
-  storage: {
-    type: 'auto',
-    path: './data'
-  }
-})
-```
-`'auto'` picks `'filesystem'` when running on Node.js with a writable `path`, and falls back to `'memory'` otherwise.
-
-## Backup and Off-Site Replication
-
-Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `path` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:
-
-- `gsutil rsync -r ./data gs://my-bucket/brainy-data`
-- `aws s3 sync ./data s3://my-bucket/brainy-data`
-- `rclone sync ./data remote:brainy-data`
-- Periodic `tar` snapshots to any object store
-
-Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory.
-
-## Metadata Indexing System
-
-### Field Discovery Index
-Tracks all unique values for each field:
-
-```json
-// __metadata_field_index__field_category.json
-{
-  "values": {
-    "technology": 45,
-    "science": 32,
-    "business": 28
-  },
-  "lastUpdated": 1699564234567
-}
-```
-
-### Value-Based Indexes
-Maps field+value combinations to entity IDs:
-
-```json
-// __metadata_index__category_technology_chunk0.json
-{
-  "field": "category",
-  "value": "technology",
-  "ids": ["uuid1", "uuid2", "uuid3", ...],
-  "chunk": 0,
-  "total": 45
-}
-```
-
-### Index Chunking
-Large indexes automatically chunk for performance:
-- **Chunk size**: 10,000 IDs per chunk
-- **Auto-splitting**: Transparent to queries
-- **Parallel loading**: Chunks load on demand
-
-## Entity Registry
-
-High-performance deduplication system for streaming data:
-
-### Registry Structure
-```json
-// __entity_registry__.json
-{
-  "mappings": {
-    "did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
-    "handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
-  },
-  "stats": {
-    "totalMappings": 10000,
-    "lastSync": 1699564234567
-  }
-}
-```
-
-### Performance Characteristics
-- **Lookup**: O(1) in-memory hash map
-- **Persistence**: Configurable (memory/storage/hybrid)
-- **Cache**: LRU with configurable TTL
-- **Sync**: Periodic or on-demand
-
-## Durability
-
-Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `path` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).
-
-## Storage Optimization
-
-### 1. Batch Operations
-
-```typescript
-// Efficient batch delete
-await storage.batchDelete([
-  'entities/nouns/vectors/00/00123456-....json',
-  'entities/nouns/metadata/00/00123456-....json'
-  // ...
-])
-
-// Batch writes for performance
-await brain.addBatch([
-  { content: "item1", metadata: {} },
-  { content: "item2", metadata: {} },
-  { content: "item3", metadata: {} }
-])
-// Single transaction, optimized I/O
-```
-
-### 2. Caching Strategy
-
-```typescript
-// Configure caching
-const brain = new Brainy({
-  storage: {
-    type: 'filesystem',
-    path: './data',
-    cache: {
-      enabled: true,
-      maxSize: 1000,   // Maximum cached items
-      ttl: 300000,     // 5 minutes
-      strategy: 'lru'  // Least recently used
-    }
-  }
-})
-```
-
-## Concurrent Access
-
-### Locking Mechanism
-```typescript
-// Automatic locking for write operations
-await brain.storage.withLock('resource-id', async () => {
-  // Exclusive access to resource
-  await brain.storage.saveNoun(id, data)
-})
-```
-
-### Read-Write Separation
-- **Reads**: Non-blocking, parallel
-- **Writes**: Serialized with locks
-- **Hybrid**: Read-heavy optimization
-
-## Migration and Backup
-
-Backup and restore go through the Db API — see
-[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) for the full
-recipe book.
-
-### Snapshot (backup)
-```typescript
-// Instant, self-contained snapshot (hard links on filesystem storage)
-const db = brain.now()
-await db.persist('/backups/2026-06-11')
-await db.release()
-```
-
-### Restore
-```typescript
-// Replace the store's entire state from a snapshot (destructive — confirm required)
-await brain.restore('/backups/2026-06-11', { confirm: true })
-```
-
-### Move to a new directory
-```typescript
-// A snapshot directory is a complete store: restore it into a fresh brain
-const brain = new Brainy({ storage: { type: 'filesystem', path: './new' } })
-await brain.init()
-await brain.restore('/backups/2026-06-11', { confirm: true })
-```
-
-## Performance Tuning
-
-### FileSystem Optimizations
-- **Directory sharding**: 256 shards spread files across subdirectories
-- **Async I/O**: Non-blocking file operations
-- **Buffer pooling**: Reuse buffers for efficiency
-
-### Monitoring
-
-```typescript
-// Get storage statistics
-const stats = await brain.storage.getStatistics()
-console.log(stats)
-// {
-//   totalSize: 1048576,
-//   entityCount: 1000,
-//   indexSize: 204800,
-//   walSize: 10240,
-//   cacheHitRate: 0.85
-// }
-```
-
-## Best Practices
-
-### Choose the Right Adapter
-1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence
-2. **Single-process production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
-3. **Horizontal scaling**: Brainy runs in one process — there is no built-in cluster. Run independent instances behind a service layer and replicate the on-disk artifact with your operator tooling; or run many reader processes against one shared store with a single writer
-
-### Optimize for Your Use Case
-1. **Read-heavy**: Enable caching and let the OS page cache do its job
-2. **Write-heavy**: Batch operations and tune the cache `maxSize`
-3. **Real-time**: FileSystem with periodic snapshots
-4. **Archival**: Snapshot `path` to cold object storage on a schedule
-5. **Large-scale**: Rely on metadata/vector separation + UUID sharding
-
-### Monitor and Maintain
-1. Regular statistics collection
-2. Watch disk usage and shard balance
-3. Index optimization
-4. Cache tuning based on hit rates
-5. Verify backup runs (test restore quarterly)
-
-## API Reference
-
-See the [Storage API](../api/storage.md) for complete method documentation.
-
----
-
-**Last Updated**: 2026
-**Key Features**: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup

docs/architecture/triple-intelligence.md

@@ -1,372 +0,0 @@
----
-title: Triple Intelligence
-slug: concepts/triple-intelligence
-public: true
-category: concepts
-template: concept
-order: 1
-description: Unified vector similarity, graph traversal, and metadata filtering in one query. Auto-optimizes between parallel execution and progressive filtering.
-next:
-  - concepts/noun-types
-  - api/reference
----
-
-# Triple Intelligence System
-
-The Triple Intelligence System is Brainy's revolutionary query engine that unifies vector similarity, graph relationships, and metadata filtering into a single, optimized query interface.
-
-## Overview
-
-Traditional databases force you to choose between vector search, graph traversal, OR metadata filtering. Brainy combines all three intelligences into one magical API that automatically optimizes execution for maximum performance.
-
-## Query Interface
-
-### Unified Query Structure
-
-`find()` accepts a single `FindParams` object (or a natural-language string). One
-object combines all three intelligences:
-
-```typescript
-interface FindParams {
-  // Vector intelligence — semantic similarity
-  query?: string                 // Natural-language / semantic query (embedded, matched via HNSW + text index)
-  vector?: number[]              // Pre-computed embedding for direct vector search
-
-  // Metadata intelligence — structured field filters
-  type?: NounType | NounType[]   // Filter by entity type
-  subtype?: string | string[]    // Filter by per-product subtype
-  where?: Record<string, any>    // Field predicates with bare operators (gte, lt, in, contains, exists…)
-
-  // Graph intelligence — relationship traversal
-  connected?: {
-    to?: string                  // Reachable to this entity
-    from?: string                // Reachable from this entity
-    via?: VerbType | VerbType[]  // Relationship type(s) to traverse (alias: type)
-    depth?: number               // Max traversal depth (default: 1)
-    direction?: 'in' | 'out' | 'both'
-  }
-
-  // Proximity — nearest neighbours of a known entity
-  near?: { id: string; threshold?: number }
-
-  // Control
-  limit?: number                 // Max results (default: 10)
-  offset?: number                // Skip N results
-  orderBy?: string               // Field to sort by (e.g. 'createdAt')
-  order?: 'asc' | 'desc'         // Sort direction
-}
-```
-
-### Example Queries
-
-#### Natural Language Queries with find()
-```typescript
-// Brainy understands natural language and extracts intent
-const results = await brain.find("research papers about neural networks from 2023")
-// Automatically interprets: document type, topic, time range
-
-// Complex temporal and numeric queries
-const reports = await brain.find("quarterly reports from Q3 2024 with revenue over 10M")
-// Automatically extracts: report type, date range, numeric filters
-
-// Multi-condition natural language
-const articles = await brain.find("verified articles by John Smith about machine learning published this year")
-// Automatically identifies: author, topic, verification status, time range
-```
-
-#### Simple Vector Search
-```typescript
-const results = await brain.find("machine learning concepts")
-```
-
-#### Combined Intelligence Query
-```typescript
-const results = await brain.find({
-  query: "neural networks",
-  where: {
-    category: "research",
-    year: { gte: 2023 }
-  },
-  connected: {
-    to: "deep-learning-team",
-    depth: 2
-  },
-  limit: 20
-})
-```
-
-## Query Optimization
-
-### Automatic Plan Generation
-
-The Triple Intelligence engine analyzes each query to create an optimal execution plan:
-
-1. **Selectivity Analysis**: Identifies the most selective filters
-2. **Cost Estimation**: Estimates computational cost for each operation
-3. **Strategy Selection**: Chooses between parallel or progressive execution
-4. **Plan Caching**: Caches successful plans for similar queries
-
-### Execution Strategies
-
-#### Parallel Execution
-All three search types execute simultaneously:
-- **Best for**: Balanced queries with multiple signals
-- **Performance**: Maximum speed through parallelization
-- **Use case**: Complex queries needing all intelligence types
-
-```typescript
-// Parallel execution for balanced query
-const results = await brain.find({
-  query: "AI research",          // ~1000 potential matches
-  where: { kind: "paper" },       // ~500 potential matches
-  connected: { to: "stanford" }   // ~200 potential matches
-})
-// All three execute in parallel, results fused
-```
-
-#### Progressive Filtering
-Operations chain for maximum efficiency:
-- **Best for**: Queries with highly selective filters
-- **Performance**: Reduces search space at each step
-- **Use case**: Large datasets with specific criteria
-
-```typescript
-// Progressive execution for selective query
-const results = await brain.find({
-  where: { userId: "user123" },   // Very selective (1-10 matches)
-  query: "recent posts",           // Applied to filtered set
-  limit: 5
-})
-// Metadata filter first, then vector search on results
-```
-
-## Fusion Ranking
-
-### Score Combination
-
-When multiple intelligence types return results, scores are intelligently combined:
-
-```typescript
-fusionScore = (
-  vectorScore * vectorWeight +    // Semantic relevance (0.4)
-  graphScore * graphWeight +      // Relationship strength (0.3)
-  fieldScore * fieldWeight        // Exact match confidence (0.3)
-) / totalWeight
-```
-
-### Adaptive Weights
-
-Weights adjust based on query characteristics:
-- **Text-heavy query**: Higher vector weight
-- **Relationship query**: Higher graph weight  
-- **Specific filters**: Higher field weight
-
-## Natural Language Processing
-
-### Pattern Recognition
-
-Brainy includes 220+ embedded patterns for natural language understanding:
-
-```typescript
-// Natural language automatically parsed
-const results = await brain.find(
-  "show me recent AI papers from Stanford published this year"
-)
-// Automatically converts to:
-// {
-//   query: "AI papers",
-//   where: {
-//     institution: "Stanford",
-//     published: { gte: "2024-01-01" }
-//   }
-// }
-```
-
-### Intent Detection
-
-The NLP processor identifies query intent:
-- **Informational**: "what is", "how does"
-- **Navigational**: "find", "show me"
-- **Transactional**: "create", "update"
-- **Analytical**: "compare", "analyze"
-
-## Performance Optimization
-
-### Query Plan Caching
-
-Successful execution plans are cached:
-```typescript
-// First call parses the natural-language query and builds an execution plan
-await brain.find("machine learning papers")
-
-// A structurally similar query reuses that plan, skipping plan generation
-await brain.find("deep learning papers")
-```
-
-### Self-Optimization
-
-Brainy uses itself to optimize queries:
-- Query patterns stored in separate brain instance
-- Execution times tracked and analyzed
-- Plans automatically improved based on performance
-
-### Index Utilization
-
-Triple Intelligence leverages all available indexes:
-- **HNSW Index**: For vector similarity
-- **Metadata Index**: For metadata filtering
-- **Graph Index**: For relationship traversal
-
-## Advanced Features
-
-### Explain Mode
-
-Diagnose how a query's `where` fields map to the index. Run `brain.explain()`
-first whenever `find()` returns surprising or empty results:
-
-```typescript
-const plan = await brain.explain({
-  query: "quantum computing",
-  where: { category: "research" }
-})
-
-console.log(plan.fieldPlan)
-// [
-//   { field: 'category', path: 'column-store', notes: '...' }
-// ]
-
-console.log(plan.warnings)
-// e.g. ['Field "category" has no index entries. find() will return [] silently...']
-```
-
-### Result Ordering
-
-Sort results by any stored field with `orderBy` / `order`:
-
-```typescript
-const results = await brain.find({
-  query: "news articles",
-  where: { verified: true },
-  orderBy: 'createdAt',  // Newest first
-  order: 'desc'
-})
-```
-
-### Similarity Threshold
-
-Find the nearest neighbours of a known entity and keep only close matches with
-`near`:
-
-```typescript
-const results = await brain.find({
-  near: { id: anchorId, threshold: 0.9 },  // Only results >= 0.9 similarity
-  limit: 10
-})
-```
-
-## Best Practices
-
-### Query Design
-
-1. **Start specific**: Use selective filters when possible
-2. **Combine intelligently**: Don't force all three types if not needed
-3. **Use limits**: Always specify reasonable result limits
-4. **Cache results**: For repeated queries, cache at application level
-
-### Performance Tips
-
-1. **Index first**: Ensure fields used in `where` clauses are indexed
-2. **Batch operations**: Use batch methods for bulk queries
-3. **Monitor plans**: Use explain mode to understand performance
-4. **Optimize patterns**: Train custom patterns for your domain
-
-### Common Patterns
-
-#### Semantic Search with Filtering
-```typescript
-// Find similar content with constraints
-const results = await brain.find({
-  query: searchText,
-  where: {
-    status: 'published',
-    language: 'en'
-  }
-})
-```
-
-#### Related Items Discovery
-```typescript
-// Find items related to a specific item
-const results = await brain.find({
-  connected: {
-    to: itemId,
-    depth: 2,
-    via: VerbType.RelatedTo
-  },
-  limit: 20
-})
-```
-
-#### Time-based Queries
-```typescript
-// Recent items matching criteria
-const results = await brain.find({
-  where: {
-    timestamp: { gte: Date.now() - 86400000 }
-  },
-  query: "trending topics",
-  orderBy: 'timestamp',
-  order: 'desc'
-})
-```
-
-## Natural Language Processing
-
-The `find()` method includes advanced NLP capabilities powered by 220+ embedded patterns that understand natural language queries.
-
-### Supported Query Types
-
-```typescript
-// Temporal queries
-await brain.find("documents from last week")
-await brain.find("reports created yesterday")
-await brain.find("articles published in Q3 2024")
-await brain.find("data from January to March")
-
-// Numeric filters
-await brain.find("products with price under $100")
-await brain.find("articles with more than 1000 views")
-await brain.find("reports showing revenue over 10M")
-
-// Combined conditions
-await brain.find("verified research papers about AI from 2024 with high citations")
-await brain.find("recent customer reviews with rating above 4 stars")
-await brain.find("blog posts by John Smith about machine learning published this month")
-
-// Relationship queries
-await brain.find("documents related to project X")
-await brain.find("people who work at TechCorp")
-await brain.find("products similar to iPhone")
-```
-
-### How It Works
-
-1. **Intent Detection**: Identifies what the user is looking for
-2. **Entity Extraction**: Extracts names, dates, numbers, categories
-3. **Temporal Parsing**: Converts "last week", "Q3 2024" to date ranges
-4. **Filter Generation**: Creates appropriate where clauses
-5. **Query Fusion**: Combines NLP understanding with vector search
-
-### Pattern Coverage
-
-Brainy includes 220+ pre-computed patterns covering:
-- **Temporal**: 40+ patterns for dates and time ranges
-- **Numeric**: 30+ patterns for comparisons and ranges
-- **Relationships**: 25+ patterns for connections
-- **Actions**: 35+ patterns for verbs and intents
-- **Entities**: 40+ patterns for people, places, things
-- **Domain-specific**: 50+ patterns for tech, business, social
-
-## API Reference
-
-See the [Triple Intelligence API](../api/triple-intelligence.md) for complete method documentation.

docs/architecture/zero-config.md

@@ -1,157 +0,0 @@
----
-title: Zero Configuration
-slug: concepts/zero-config
-public: false
-category: concepts
-template: concept
-order: 3
-description: Brainy auto-detects storage, initializes embeddings, and builds indexes — no configuration required. Works in Node.js and Bun (server-only since 8.0).
-next:
-  - getting-started/installation
-  - guides/storage-adapters
----
-
-# Zero Configuration & Auto-Adaptation
-
-> **"Zero config by default, fully tunable when you need it."** Construct a
-> `Brainy()` with no options and it picks sensible, environment-aware defaults.
-> Every default below is overridable through the constructor — see the
-> [API Reference](../api/README.md#configuration).
-
-## Overview
-
-Brainy 8.0 is server-only (Node.js 22+ / Bun). With no configuration it:
-
-- selects a storage adapter from the runtime,
-- initializes the embedding model (all-MiniLM-L6-v2, 384 dimensions),
-- builds and maintains the metadata, graph, and vector indexes,
-- sizes its caches and write buffers to the detected memory budget,
-- chooses a persistence mode that matches the storage backend, and
-- quiets its own logging when it detects a production environment.
-
-There is no public config-generation function — adaptation happens inside the
-constructor and `init()`.
-
-## Instant Start
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-// That's it. No config needed.
-const brain = new Brainy()
-await brain.init()
-
-await brain.add({ data: 'First entity', type: 'concept' })
-const results = await brain.find('first')
-```
-
-## What Auto-Adaptation Covers
-
-### 1. Storage auto-detection
-
-With no `storage` option, Brainy uses `type: 'auto'`:
-
-- **Filesystem** when running on a runtime with a writable Node filesystem and a
-  resolvable root directory. This is the default for typical Node/Bun servers and
-  persists across restarts.
-- **In-memory** otherwise (no filesystem access, or an explicit memory request).
-  Fast, zero I/O, discarded on process exit — ideal for tests and ephemeral
-  caches.
-
-8.0 ships exactly two storage adapters — `memory` and `filesystem` — plus the
-`auto` selector that resolves to one of them. See
-[Storage Adapters](../concepts/storage-adapters.md) for the full contract.
-
-```typescript
-// Explicit override when you want a specific root
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: './brainy-data' }
-})
-```
-
-### 2. HNSW quality from the `recall` preset
-
-Vector-index quality comes from a single preset rather than hand-tuned graph
-parameters. `config.vector.recall` accepts `'fast'`, `'balanced'`, or
-`'accurate'` and defaults to `'balanced'`. The preset maps internally to the
-HNSW construction and search parameters (`M` / `efConstruction` / `efSearch`),
-so you trade recall against latency with one knob instead of three.
-
-```typescript
-const brain = new Brainy({
-  vector: { recall: 'fast' } // favor latency over recall
-})
-```
-
-The default JS index is `JsHnswVectorIndex`. An optional native acceleration
-provider (the `@soulcraft/cor` package) can replace it with a
-higher-performing implementation; the public knobs stay the same. Quantization
-and other index-internal acceleration are the native provider's concern, not a
-Brainy configuration option.
-
-### 3. Persistence mode follows the backend
-
-`config.vector.persistMode` accepts `'immediate'` or `'deferred'`. Left unset,
-Brainy chooses for you:
-
-- **Immediate** on filesystem storage, so the index file stays in lock-step with
-  the data and survives a crash.
-- **Deferred** on in-memory storage, where there is nothing durable to sync to,
-  so writes are batched for throughput.
-
-```typescript
-const brain = new Brainy({
-  vector: { persistMode: 'deferred' } // batch persistence for write-heavy loads
-})
-```
-
-### 4. Memory-aware cache and buffer sizing
-
-Brainy reads the container's memory budget — `CLOUD_RUN_MEMORY`, `MEMORY_LIMIT`,
-or the cgroup memory limit when running in a container — and sizes its read
-caches and write buffers to fit. On a small instance it stays conservative; on a
-large one it uses more of the available headroom. Query-result limits are capped
-against the same budget (roughly 25 KB per result) to keep a single oversized
-query from exhausting memory.
-
-You can pin the cache explicitly:
-
-```typescript
-const brain = new Brainy({
-  cache: { maxSize: 10000, ttl: 3_600_000 }
-})
-```
-
-### 5. Logging quiets in production
-
-Brainy detects production-style environments (for example `NODE_ENV` set to a
-non-development value) and reduces its own log verbosity automatically. This is
-logging-only behavior — it does not change indexing, storage, or query results.
-
-## Configuration Override
-
-Zero-config is the default, not a ceiling. Every adaptive decision above has an
-explicit constructor option:
-
-```typescript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: '/var/lib/brainy' },
-  vector: {
-    recall: 'accurate',
-    persistMode: 'immediate'
-  },
-  cache: { maxSize: 50000, ttl: 600_000 }
-})
-
-await brain.init()
-```
-
-See the [API Reference](../api/README.md#configuration) for the complete option
-list.
-
-## See Also
-
-- [Architecture Overview](./overview.md)
-- [Storage Adapters](../concepts/storage-adapters.md)
-- [Scaling Guide](../SCALING.md)
-- [API Reference](../api/README.md)

docs/concepts/consistency-model.md

@@ -1,386 +0,0 @@
----
-title: Consistency Model
-slug: concepts/consistency-model
-public: true
-category: concepts
-template: concept
-order: 4
-description: The exact guarantees behind Brainy's Db API — snapshot isolation, atomic transactions, two levels of compare-and-swap, time travel, retention, snapshots, crash recovery, and the reserved-field contract.
-next:
-  - guides/snapshots-and-time-travel
-  - guides/optimistic-concurrency
----
-
-# Consistency Model
-
-Brainy 8.0's consistency story rests on one mechanism: **generational MVCC**
-— multi-version concurrency control over immutable, generation-stamped
-records. It is exposed through a single value type, the **`Db`**: an
-immutable, point-in-time view of the whole store that you query like the
-live brain.
-
-```typescript
-const db = brain.now()                 // pin the current state — O(1), no I/O
-
-await brain.transact([
-  { op: 'update', id: invoiceId, metadata: { status: 'paid' } }
-])
-
-await db.get(invoiceId)                // still 'pending' — pinned, forever
-await brain.get(invoiceId)             // 'paid' — live
-await db.release()                     // unpin when done
-```
-
-This page states the guarantees precisely — what is promised, what it costs,
-and where the honest limits are. The design record is
-[ADR-001](../ADR-001-generational-mvcc.md); every guarantee below is proven
-by a dedicated test in `tests/integration/db-mvcc.test.ts`.
-
-## The generation clock
-
-A **monotonic generation counter** is the store's logical clock:
-
-- It advances **once per committed `transact()` batch** and once per
-  single-operation write (`add`/`update`/`remove`/`relate`/…).
-- `brain.generation()` reads it; it is persisted in the data directory and
-  **never reissued** — not across restarts, and not across `restore()`
-  (the counter is floored at its pre-restore value).
-
-Every `Db` is pinned at one generation. `db.generation` and `db.timestamp`
-identify the view; `newerDb.since(olderDb)` returns exactly the entity and
-relationship ids that committed transactions touched between two views.
-
-## Snapshot isolation for reads
-
-**Guarantee:** a `Db` reads exactly the state at its pinned generation, no
-matter what commits afterwards — including deletes. There are no torn reads,
-no partially applied batches, and no drift over time.
-
-- `brain.now()` pins the current generation in O(1).
-- `brain.transact()` returns a `Db` pinned at the freshly committed
-  generation.
-- `brain.asOf(generation | Date | snapshotPath)` pins past state.
-
-While nothing has committed past the pin, reads delegate to the live fast
-paths — pinning is free until history actually moves. Once later
-transactions commit, the view keeps serving the **full query surface** at
-its generation (see "Reading the past" below).
-
-Writers are never blocked by readers and readers never block writers: a
-pinned view stays valid because nothing overwrites the immutable records it
-resolves from (the LMDB reader-pin model).
-
-## Transaction atomicity
-
-`brain.transact(ops)` executes a declarative batch — `add`, `update`,
-`remove`, `relate`, `unrelate` — **atomically as exactly one generation**:
-
-```typescript
-const db = await brain.transact([
-  { op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' },
-  { op: 'add', id: itemId, type: NounType.Thing, subtype: 'line-item', data: 'Widget x3' },
-  { op: 'relate', from: orderId, to: itemId, type: VerbType.Contains, subtype: 'order-line' }
-], { meta: { author: 'order-service', requestId: 'req-9f2' } })
-
-db.receipt.ids   // resolved id per operation, in input order
-```
-
-Either every operation applies, or none do and the store is byte-identical
-to its pre-transaction state. Operation semantics mirror the corresponding
-single-operation methods — validation, subtype enforcement, relationship
-deduplication, delete cascades — and later operations may reference ids
-created earlier in the same batch.
-
-**The commit point is one atomic rename.** The durability protocol:
-
-1. Before-images of every touched id are staged into an immutable
-   generation directory and **fsynced**.
-2. The batch executes through the transaction manager (which has its own
-   operation-level rollback for non-crash failures).
-3. The store manifest is replaced via atomic temp-file rename and fsynced.
-   **The rename is the commit** — a generation is committed if and only if
-   the manifest says so.
-
-**Crash recovery:** on the next open, any staged generation above the
-manifest watermark is an uncommitted transaction; its before-images are
-restored (idempotently — recovery can itself crash and rerun) and derived
-indexes never observe the rolled-back state. A crash anywhere before the
-rename rolls back to the exact pre-transaction bytes; a crash after it keeps
-the transaction.
-
-Transaction metadata (`meta`) is reified Datomic-style: recorded in an
-append-only transaction log readable via `brain.transactionLog()` — audit
-fields live in the database, not in commit messages.
-
-## Two levels of compare-and-swap
-
-Concurrent `transact()` calls commit serially (snapshot-isolated batches).
-For lost-update protection across a read–modify–write cycle, Brainy offers
-CAS at two granularities:
-
-| Granularity | Mechanism | Conflict error | Use when |
-|---|---|---|---|
-| **Per entity** | `_rev` + `{ op: 'update', ifRev }` (also on `brain.update()`) | `RevisionConflictError` | "This entity must not have changed since I read it." |
-| **Whole store** | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` | "*Nothing* may have committed since I read." |
-
-```typescript
-const view = brain.now()
-const order = await view.get(orderId)
-
-try {
-  await brain.transact(
-    [{ op: 'update', id: orderId, metadata: { total: recompute(order) }, ifRev: order._rev }],
-    { ifAtGeneration: view.generation }
-  )
-} catch (err) {
-  if (err instanceof GenerationConflictError) {
-    // Something committed since the pin — re-read and retry.
-  }
-} finally {
-  await view.release()
-}
-```
-
-An `ifRev` conflict on any operation rejects the **whole batch**; an
-`ifAtGeneration` conflict is detected before anything is staged. Both leave
-the store untouched and the generation counter unchanged. See
-[Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md)
-for the per-entity pattern in depth.
-
-## Reading the past
-
-`brain.asOf()` accepts a generation number, a `Date` (resolved through the
-transaction log to the newest generation committed at or before it), or a
-snapshot directory path. Historical views serve the **full query surface**
-— `get()`, `find()` in every mode, semantic search, graph traversal,
-cursors, aggregation — through two complementary paths:
-
-- **Record path** (free): `get()`, metadata-level `find()`, and
-  filter-based `related()` resolve directly through the immutable record
-  layer. Ids untouched since the pin still ride the live fast paths.
-- **Index path** (paid once): index-accelerated queries — semantic/vector
-  search, graph traversal, cursors, aggregation — are served by an
-  **at-generation index materialization** built lazily on first use:
-  Brainy reconstructs in-memory indexes over the exact record set at that
-  generation. This costs O(n at the pinned generation) time and memory,
-  **once per `Db`**, cached until `release()`. That is the open-core price
-  of historical index queries, stated plainly.
-
-A native index provider implementing the optional
-`VersionedIndexProvider` plugin capability serves the same historical reads
-from its retained index segments **without any rebuild** — the materializer
-is the correctness baseline, the provider is the accelerator. Semantics are
-identical on both paths.
-
-### History granularity — the honest limit
-
-Generation *records* are written per `transact()` batch only.
-Single-operation writes (`add`/`update`/`remove`/`relate`/… outside
-`transact()`) advance the generation counter — so watermarks and CAS stay
-sound — but do **not** stage before-images: they remain visible through
-earlier pins and are not reported by `db.since()`. Code that needs pinned
-isolation across its own writes uses `transact()`. This is the documented
-8.0 contract, not an accident.
-
-## Speculative writes: `db.with()`
-
-`db.with(ops)` returns a new `Db` whose reads see the operations applied
-**in memory, on top of the view** — Datomic's `with`. Nothing touches disk,
-the generation counter, or index providers:
-
-```typescript
-const current = brain.now()
-const whatIf = await current.with([
-  { op: 'update', id: employeeId, metadata: { team: 'platform' } }
-])
-
-await whatIf.find({ where: { team: 'platform' } })   // sees the change
-await brain.get(employeeId)                          // unchanged — nothing committed
-```
-
-**The one boundary:** overlay entities carry no embeddings (`with()` never
-invokes the embedder), so index-accelerated queries and `persist()` on a
-speculative view throw `SpeculativeOverlayError` rather than returning
-silently incomplete results. `get()`, metadata-filter `find()`, and
-filter-based `related()` work fully on overlays. To get the full surface,
-commit the same operations with `brain.transact()`.
-
-## Retention and compaction
-
-Historical records cost disk space, so retention is explicit:
-
-- Every live `Db` holds a refcounted **pin**; a record-set is never
-  reclaimed while any pin could need it — pinned reads stay correct across
-  compaction, always.
-- The **`retention`** knob governs auto-compaction (on every `flush()`/
-  `close()`): unset → ADAPTIVE (disk/RAM-pressure, zero-config) · `'all'` →
-  unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit caps.
-  `brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
-  manually on the same caps, and records the **horizon** — `asOf()` below it
-  throws `GenerationCompactedError`, explicitly, never partial data.
-- To keep a state readable forever, `persist()` it first: snapshots are
-  self-contained and unaffected by compaction of the source store.
-
-Release `Db` values you do not keep (including the ones `transact()`
-returns). A `FinalizationRegistry` backstop releases leaked pins at garbage
-collection, but explicit `release()` is what makes compaction
-deterministic.
-
-## Durability: snapshots and restore
-
-`db.persist(path)` cuts a **self-contained snapshot** under the store's
-commit mutex, so no commit or compaction can interleave. On filesystem
-storage it is built from **hard links**: because every data file is
-immutable-by-rename, linking is safe — the snapshot is created without
-copying entity data, shares disk space with the source, and later writes to
-the source can never alter it (rewrites swap inodes; the snapshot keeps the
-old bytes). Cross-device targets fall back to byte copies; in-memory stores
-serialize to the same directory layout, producing a real, durable store.
-
-Two rules keep snapshots honest:
-
-- `persist()` requires the view to still be the store's **latest**
-  generation (a snapshot captures current bytes); a view that history has
-  moved past throws `GenerationConflictError` instead of persisting the
-  wrong state.
-- `brain.restore(path, { confirm: true })` replaces the store's entire
-  state from a snapshot via byte copy (never links — the snapshot stays
-  independent), rebuilds all indexes, and floors the generation counter so
-  observed generation numbers are never reissued. Live pins do not survive
-  a restore — release them first (a warning is logged when any exist).
-
-`Brainy.load(path)` (or `brain.asOf(path)`) opens a snapshot as a
-self-contained **read-only** store with the full query surface, including
-vector search.
-
-## Reserved fields
-
-Some field names belong to Brainy, not to your metadata. They live at **top
-level** on every entity and relationship, have dedicated write paths, and may
-never appear inside a `metadata` bag:
-
-| Entities (nouns) | Relationships (verbs) | Canonical write path |
-|---|---|---|
-| `noun` | `verb` | the `type` param of `add()` / `relate()` |
-| `subtype` | `subtype` | the `subtype` param |
-| `visibility` | `visibility` | the `visibility` param (`'public'` \| `'internal'`) |
-| `confidence` | `confidence` | the `confidence` param |
-| `weight` | `weight` | the `weight` param |
-| `service` | `service` | the `service` param (fixed at create time) |
-| `data` | `data` | the `data` param |
-| `createdBy` | `createdBy` | the `createdBy` param of `add()` (system-managed on verbs) |
-| `createdAt`, `updatedAt`, `_rev` | `createdAt`, `updatedAt`, `_rev` | system-managed (`ifRev` for CAS) |
-
-The canonical machine-readable lists are exported as
-`RESERVED_ENTITY_FIELDS` and `RESERVED_RELATION_FIELDS` (defined in
-`src/types/reservedFields.ts`, the single source of truth). Three layers
-enforce the contract:
-
-1. **Compile time** — every `metadata` param (`add`, `update`, `relate`,
-   `updateRelation`, and the matching `transact()` operations) rejects a
-   literal reserved key as a TypeScript error.
-2. **Write time** — untyped (JavaScript) callers that pass one anyway are
-   normalized: user-settable fields (`confidence`, `weight`, `subtype`, and
-   `service`/`createdBy` at create time) are remapped to their dedicated
-   param — **top-level wins** when both are supplied — and system-managed
-   fields are dropped with a one-shot warning naming the correct write path.
-   `update({ metadata: { confidence: 0.9 } })` therefore behaves exactly
-   like `update({ confidence: 0.9 })`.
-3. **Read time** — every read path (`get`, `find`, `search`,
-   `related`, batch reads, and historical `asOf()` materialization)
-   surfaces reserved fields **only at top level**: `entity.metadata` and
-   `relation.metadata` contain only your custom fields, always.
-
-```typescript
-const id = await brain.add({
-  type: 'document', subtype: 'invoice',
-  data: 'Invoice #42', confidence: 0.95,        // reserved → top-level params
-  metadata: { customer: 'acme', total: 129.5 }  // custom fields only
-})
-
-const entity = await brain.get(id)
-entity.confidence                                // 0.95 — top level
-entity.metadata                                  // { customer: 'acme', total: 129.5 }
-```
-
-### Visibility — `public` / `internal` / `system`
-
-`visibility` is a reserved tier that controls whether an entity or relationship
-surfaces on Brainy's **default** user-facing reads. The absence of the field is
-exactly equivalent to `'public'`.
-
-| Tier | Counted in `getNounCount()` / `stats()`? | Returned by default `find()` / `related()`? | Opt-in |
-|---|---|---|---|
-| `'public'` (default, or field absent) | yes | yes | — |
-| `'internal'` | no | no | `find({ includeInternal: true })` / `related({ includeInternal: true })` |
-| `'system'` | no | no | `find({ includeSystem: true })` / `related({ includeSystem: true })` |
-
-- **`'public'`** — normal data. Counted and returned everywhere. Stored lean:
-  the field is omitted on disk for public records, so existing data needs no
-  migration.
-- **`'internal'`** — your app's own bookkeeping (audit trails, derived caches,
-  scratch entities) that should not pollute default queries, counts, or
-  `stats()`, yet must stay retrievable on demand. Set it via the `visibility`
-  param; read it back with the `includeInternal` opt-in.
-- **`'system'`** — Brainy's own plumbing (for example the Virtual File System
-  root entity). Hidden everywhere by default — even when `includeInternal` is
-  set — and surfaced only with the explicit `includeSystem` opt-in. The
-  `'system'` tier is **not** part of the public `add()` / `relate()` param type
-  (`'public' | 'internal'`); only internal Brainy code assigns it.
-
-The opt-ins are applied as a **hard candidate filter** — hidden entities are
-removed before `limit` / `offset` are applied, so a default `find({ limit: 10 })`
-always returns ten *visible* results when that many exist, never a short page.
-
-```typescript
-// App-internal scratch entity: present, retrievable, but out of the way.
-await brain.add({ type: 'task', data: 'reindex job', visibility: 'internal' })
-
-await brain.getNounCount()                       // unchanged — internal not counted
-await brain.find({ type: 'task' })               // [] — hidden by default
-await brain.find({ type: 'task', includeInternal: true })   // includes it
-
-// A brand-new brain reports zero user entities even though the VFS root exists:
-const fresh = new Brainy()
-await fresh.init()
-await fresh.getNounCount()                        // 0 — the root is visibility:'system'
-```
-
-> **Note (8.0):** the structural `Contains` edges the VFS creates between
-> directories and files are left at the default (`public`) visibility for now —
-> only the VFS *root entity* is `'system'`. Marking those edges system requires
-> companion changes to VFS traversal and is out of scope for this change.
-
-## What is not guaranteed
-
-Stated plainly, so nothing surprises you in production:
-
-- **Single-writer.** Brainy is a single-writer, many-reader database
-  ([multi-process model](./multi-process.md)). Transactions are atomic
-  within one writer process — there is no distributed or cross-process
-  transaction coordination.
-- **History granularity.** Every write is its own immutable generation —
-  `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
-  A pin always freezes against later writes, and every write is addressable
-  via `asOf()`. (`transact()` groups several operations into ONE atomic
-  generation; durability of single-op history is batched via async
-  group-commit — a hard crash can lose only the last un-flushed window's
-  *history*, never live data.)
-- **Compacted history is gone.** `asOf()` below the compaction horizon
-  fails explicitly; persist what you must keep.
-- **Counter persistence is coalesced for single-operation writes.** Durable
-  artifacts (records, manifests, snapshots) always persist the counter at
-  their own commit points, so a crash inside the coalescing window can lose
-  only counter values nothing durable ever referenced.
-- **Speculative overlays are metadata-only readers.** Index-accelerated
-  queries on `with()` views throw rather than guess.
-
-## Where to go next
-
-- [Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) — the
-  recipes: backup, restore, time-travel debugging, what-if analysis, audit
-  trails.
-- [Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md)
-  — the per-entity CAS pattern.
-- [ADR-001: Generational MVCC](../ADR-001-generational-mvcc.md) — the full
-  design record, including the persisted layout and the proof table.

docs/concepts/generation-fact-log.md

@@ -1,117 +0,0 @@
----
-title: The Generation Fact Log
-slug: concepts/generation-fact-log
-public: true
-category: concepts
-template: concept
-order: 6
-description: Every committed write also appends a self-verifying "fact" — an after-image commit record — to an append-only log. What facts are, the crash-safety model, the scanFacts() streaming surface, family stamps, and how index providers consume the log for sequential heals.
-next:
-  - concepts/consistency-model
-  - guides/snapshots-and-time-travel
----
-
-# The Generation Fact Log
-
-Since 8.4.0, every committed generation also appends a **fact** — a compact record of what each
-touched entity or relationship *became* — to an append-only, checksummed log under
-`_generations/facts/`. Where the generational history answers *"what did things look like
-before?"* (before-images, powering `asOf()` and rollback), the fact log answers *"what happened,
-in order?"* — one sequential, self-verifying stream of the store's present being written.
-
-Nothing about querying changes. The fact log exists for three consumers:
-
-1. **Index heals and rebuilds** — one sequential read in commit order replaces a per-entity
-   directory walk over millions of files.
-2. **Incremental catch-up** — a derived index that knows which generation it reflects reads *just
-   the gap*, instead of rebuilding from scratch.
-3. **Replay and audit tooling** — anything that wants the store's committed timeline as a stream.
-
-## What a fact is
-
-One fact per committed generation:
-
-- **`generation`** and **`timestamp`** — which commit, when.
-- **`ops`** — every write in that commit: `{ kind: 'noun' | 'verb', id, record }` where `record`
-  holds the entity's full after-image (both stored legs), or **`null` for a tombstone** — a
-  removal carries no body, by design.
-- **`meta`** — the transaction metadata `transact()` was submitted with, when present.
-- **`blobHashes`** — content-blob references, for exact reclamation accounting.
-
-Facts accumulate **from the first write after upgrading** — pre-existing history is not
-retroactively converted, and consumers fall back to the enumeration walk when no log exists.
-
-## Crash safety, in one paragraph
-
-Facts are appended and fsynced **inside the same durability window as the commit itself**, before
-the commit point — so after a crash, the log can only ever be *ahead* of committed truth, never
-behind it with a hole. On open, the store reconciles the log back to the committed watermark:
-torn tails are detected by per-record checksums and cut; whole records beyond the watermark are
-truncated. The invariant every reader can rely on: **an absent generation was never committed; a
-present fact was.** `transact()` facts are durable the moment `transact()` returns; single-op
-facts share the same group-commit flush as the rest of their generation, so a hard kill loses the
-fact and the generation *together* — never a torn state.
-
-## Reading the log
-
-```typescript
-const scan = brain.scanFacts({ fromGeneration: 1 })
-if (scan) {
-  // Telemetry up front — progress bars get a denominator from second zero.
-  console.log(scan.headGeneration, scan.segmentCount, scan.approxFactCount)
-
-  for await (const batch of scan.batches()) {
-    // Each batch: { facts, firstGeneration, lastGeneration, factCount, byteSize, segmentId }
-    for (const fact of batch.facts) {
-      for (const op of fact.ops) {
-        if (op.record === null) {
-          // a tombstone: op.id was removed in this generation
-        }
-      }
-    }
-  }
-
-  console.log(scan.summary()) // { factsYielded, segmentsRead } — the cross-check
-}
-```
-
-- `scanFacts()` returns `null` when the store hosts no fact log (older store, or a storage adapter
-  without binary append support) — fall back to enumerating entities.
-- Scans run against a **snapshot**: facts appended after the scan opens never bleed in, each fact
-  is yielded exactly once, and a detected gap aborts loudly — never a silent skip.
-- `brain.factSegmentPaths()` returns the immutable, *sealed* segment files for zero-copy consumers
-  (the append-mutable tail is excluded — read it through `scanFacts()`).
-
-## Family stamps: how a projection proves it's current
-
-Anything derived from the store — an index, the entity file tree itself — carries a **family
-stamp**: a small JSON record of *which committed generation the projection reflects*
-(`sourceGeneration`) plus the invariants that verify it whole (exact per-file byte sizes for
-bounded families; rollup invariants like entity counts for unbounded ones). At open, coherence is
-a **comparison**, not a walk:
-
-- stamp equals the committed watermark and invariants hold → serve;
-- stamp is behind → the projection reads just the gap from the fact log;
-- invariants fail → loud, named divergence — `brain.repairIndex()` rebuilds from canonical and
-  re-stamps.
-
-The verifier is exported (`verifyFamilyStamp`) so every projection — TypeScript or native — runs
-literally the same check.
-
-## For plugin authors: the storage capability
-
-Index providers receive the storage adapter, not the brain — so the host wires the log onto it.
-Feature-detect and prefer the stream; fall back to enumeration:
-
-```typescript
-const scan = storage.scanFacts?.({ fromGeneration: stamp.sourceGeneration + 1 })
-if (scan) {
-  // sequential catch-up from the log
-} else {
-  // enumeration walk (older store or adapter)
-}
-const committed = storage.committedGeneration?.() // the watermark stamps compare against
-```
-
-Providers must never construct their own reader over the log's files — the open path belongs to
-the single writer (it reconciles the log at open); the capability is the sanctioned seam.

docs/concepts/multi-process.md

@@ -1,153 +0,0 @@
----
-title: Multi-Process Model
-slug: concepts/multi-process
-public: true
-category: concepts
-template: concept
-order: 5
-description: How Brainy coordinates a single writer with any number of readers on a filesystem data directory — and how to safely inspect a live store.
-next:
-  - guides/inspection
----
-
-# Multi-Process Model
-
-Brainy is a **single-writer, many-reader** database when backed by filesystem
-storage. This page explains the model, the guarantees, and the safe ways to
-inspect a live store from a second process.
-
-## The rule
-
-For one data directory:
-
-- **One writer** at a time. The writer acquires an exclusive lock on the
-  directory at `init()` and releases it on `close()`.
-- **Any number of readers**, concurrent with each other and with the writer.
-  Readers open via `Brainy.openReadOnly()` — they never touch the writer
-  lock.
-
-Any attempt to open a second writer on the same directory throws:
-
-```
-BrainyError: Another writer holds this Brainy directory.
-  PID:        1774431 on host app-host-1
-  Started:    2026-05-15T14:22:11Z
-  Heartbeat:  2026-05-15T14:22:34Z
-  Version:    7.21.0
-  Directory:  /data/brain
-```
-
-This is intentional. Two writers sharing a directory would silently corrupt
-in-memory indexes and produce wrong query results — the worst possible default
-for an operations tool.
-
-## Why a lock?
-
-Brainy keeps its primary indexes (HNSW, metadata, graph adjacency) in memory.
-On disk, those indexes are persisted incrementally as writes flush. A second
-process opening the same directory:
-
-- Loads the *persisted* state into a fresh in-memory copy.
-- Has no awareness of writes the first process buffered but hasn't flushed.
-- Will overwrite the persisted state on its own next flush, racing the first
-  process and corrupting whichever wins.
-
-The fix is the lock: refuse to open a second writer. SQLite has done the same
-since the late 1990s (`SQLITE_BUSY`).
-
-## What about Cor?
-
-Brainy + Cor compose cleanly under this model:
-
-- Cor stores its column-index segments inside the same `rootDir` (under
-  `indexes/_column_index/{field}/`).
-- Segments (`*.cidx` files) are **immutable** once written. Cor mmaps them
-  read-only.
-- The `MANIFEST.json` per field is updated via atomic rename — readers see
-  either the old or new manifest, never a torn file.
-
-A reader process can safely mmap Cor segments alongside a live writer
-without coordination. The single Brainy writer lock at
-`<rootDir>/locks/_writer.lock` covers Cor too, because Cor segment
-writes happen on the writer's side.
-
-## Stale-lock detection
-
-If a writer crashes or is forcibly killed, its lock file is left behind. To
-avoid a permanently-jammed directory, Brainy treats a lock as stale when:
-
-1. The recorded `hostname` equals the current host (cross-host PID checks
-   are unsafe), AND
-2. The recorded `pid` is no longer alive (`process.kill(pid, 0)` returns
-   `ESRCH`), OR the `lastHeartbeat` field is older than 60 seconds.
-
-A live writer rewrites `lastHeartbeat` every 10 seconds, so a hung writer
-that's missed several heartbeats is treated as dead. Stale locks are
-overwritten with a warning.
-
-If stale detection cannot prove the existing lock is dead — for example, a
-crashed writer on a different host writing to a shared filesystem — pass
-`{ force: true }` to override. A warning is logged either way.
-
-## Heartbeat and shutdown
-
-The heartbeat interval rewrites the lock file every 10 seconds. The timer
-is unref'd, so it does not keep the event loop alive on its own.
-
-On normal shutdown the writer releases the lock in `close()`. The shutdown
-hooks Brainy registers for `SIGTERM`, `SIGINT`, and `beforeExit` also
-release the lock so a container restart doesn't strand the directory.
-
-## How to inspect a live writer
-
-Use `Brainy.openReadOnly()`. It does not acquire the writer lock, so it
-coexists with whatever the writer is doing:
-
-```typescript
-const reader = await Brainy.openReadOnly({
-  storage: { type: 'filesystem', path: '/data/brain' }
-})
-
-const stats = await reader.stats()
-const bookings = await reader.find({ where: { entityType: 'booking' } })
-
-await reader.close()
-```
-
-What the reader sees reflects the writer's most recent **flush** to disk. If
-you need fresher state, ask the writer to flush before opening:
-
-```typescript
-const reader = await Brainy.openReadOnly({
-  storage: { type: 'filesystem', path: '/data/brain' }
-})
-
-const acked = await reader.requestFlush({ timeoutMs: 5000 })
-if (!acked) {
-  console.warn('Writer did not respond; results reflect last natural flush.')
-}
-
-const fresh = await reader.find({ where: { entityType: 'booking' } })
-```
-
-The CLI `brainy inspect` subcommands all do this for you by default
-(`--no-fresh` to opt out).
-
-## What's not enforced (yet)
-
-- **Non-filesystem backends** are out of scope in 8.0, which ships only the
-  filesystem and memory adapters. A custom `BaseStorage` subclass that is not
-  filesystem-backed does not enforce multi-process locking by default: two
-  processes can both succeed at `init()` in writer mode and clobber each
-  other's writes. A best-effort warning is logged in writer mode against a
-  non-filesystem backend.
-- **Long-running readers** do not automatically pick up new Cor segments
-  the writer publishes. One-shot inspector calls re-open the store and see
-  fresh segments; a reader that stays open for hours sees its column store
-  as-of the time it opened.
-
-## Reading material
-
-- `Brainy.openReadOnly()` — [API reference](../api/brainy.md)
-- `brainy inspect` — [inspection guide](../guides/inspection.md)
-- Cor columnar storage — see `node_modules/@soulcraft/cor/README.md`

docs/concepts/storage-adapters.md

@@ -1,189 +0,0 @@
----
-title: Storage Adapter Inheritance Contract
-slug: concepts/storage-adapters
-public: true
-category: concepts
-template: concept
-order: 6
-description: How storage adapters extend Brainy's BaseStorage and FileSystemStorage to inherit multi-process safety, what plugin authors need to override, and the contract Brainy promises to keep stable.
-next:
-  - concepts/multi-process
-  - guides/inspection
----
-
-# Storage Adapter Inheritance Contract
-
-Brainy's storage layer is designed for plugins to extend cleanly. A plugin
-that subclasses `BaseStorage` or `FileSystemStorage` inherits new behavior
-Brainy adds over time without code changes — provided the import resolution
-brings in the right Brainy version at runtime.
-
-This page documents what plugin authors can rely on, what they need to
-override, and the install-time failure modes that the defensive
-`hasStorageMethod()` guard exists to handle.
-
-## The class hierarchy
-
-```
-BaseStorageAdapter         (counts, batch ops, multi-tenancy hooks)
-        ↑
-BaseStorage                (type-statistics, lifecycle helpers, generation
-                            hooks, default no-op multi-process methods)
-        ↑
-FileSystemStorage          (real filesystem I/O, writer-lock implementation,
-                            flush-request watcher, atomic writes)
-        ↑
-<your plugin's storage>    (Cor's MmapFileSystemStorage, etc.)
-```
-
-When you `extend FileSystemStorage`, your adapter inherits every method on
-the chain — including the ones Brainy adds in a later release — for free.
-JavaScript's prototype chain resolves method lookups dynamically; nothing
-about the inheritance is baked in at class-definition time.
-
-## What you inherit for free
-
-A plugin whose storage class extends `FileSystemStorage` automatically gets
-the full multi-process safety surface:
-
-| Method | What it does | Override? |
-|---|---|---|
-| `acquireWriterLock(opts)` | Write `_writer.lock`, start heartbeat, throw on conflict | No |
-| `releaseWriterLock()` | Clean up lock file + heartbeat timer | No |
-| `readWriterLock()` | Read lock file as `WriterLockInfo` | No |
-| `startFlushRequestWatcher(cb)` | Poll `_flush_requests/` and invoke `cb` | No |
-| `stopFlushRequestWatcher()` | Stop the polling timer | No |
-| `requestFlushOverFilesystem(timeoutMs)` | Drop a `.req`, await `.ack` | No |
-| `supportsMultiProcessLocking()` | Return `false` (default) | **Yes — override to `true`** |
-
-The only required override is the capability flag. Returning `true` from
-`supportsMultiProcessLocking()` is the signal Brainy uses to decide whether
-to call `acquireWriterLock()` at init.
-
-```typescript
-import { FileSystemStorage } from '@soulcraft/brainy'
-
-export class MmapFileSystemStorage extends FileSystemStorage {
-  public supportsMultiProcessLocking(): boolean {
-    return true
-  }
-  // ... your mmap-specific overrides ...
-}
-```
-
-That's the full ceremony for inheriting multi-process safety.
-
-## When NOT to extend FileSystemStorage
-
-If your storage is **not filesystem-backed** (a custom
-network backend), extend `BaseStorage` directly:
-
-```typescript
-import { BaseStorage } from '@soulcraft/brainy'
-
-export class MyCloudStorage extends BaseStorage {
-  // BaseStorage's default no-op implementations of the multi-process
-  // methods stay in effect. `supportsMultiProcessLocking()` returns false
-  // by default — keep it that way unless you've implemented an object-
-  // versioned lease or similar cross-process synchronization for your
-  // backend.
-}
-```
-
-Brainy treats cloud backends as not-multi-process-safe by default and logs a
-one-line warning at init. That's the correct behavior until cloud locking
-ships (currently out of scope — see
-[`concepts/multi-process`](./multi-process.md)).
-
-## What `hasStorageMethod()` actually guards against
-
-The defensive check at every new-storage-method call site (`brainy.ts`,
-`hasStorageMethod(name)`) does **not** exist to handle "plugin bundles a
-stale BaseStorage." Plugins ship a dist that preserves the dynamic ESM
-import (verify in your plugin's `dist/`: `import { FileSystemStorage } from
-'@soulcraft/brainy'` is not rewritten to a vendored copy). The prototype
-chain at runtime resolves to whatever Brainy version your consumer has
-installed.
-
-`hasStorageMethod()` protects against **build/install artifacts** that break
-the prototype chain at the consumer-app level:
-
-- **Stale `node_modules`** — a lingering install from before the consumer
-  upgraded Brainy. The package.json says `@soulcraft/brainy@7.22.0` but
-  `node_modules/@soulcraft/brainy` is still 7.20.x.
-- **Lockfile drift** — `bun.lockb` / `package-lock.json` pins a brainy
-  version older than the package.json range, and `bun install` honors the
-  lockfile.
-- **Docker layer cache** — the image reuses a `node_modules` from an
-  earlier build that predates the brainy bump.
-- **Bundler quirks** — some bundlers (esbuild, webpack) flatten the
-  prototype chain at build time and lose later prototype mutations. Brainy
-  doesn't mutate prototypes at runtime, but bundler behavior can still
-  cause method lookups to fail in non-Node environments.
-
-In any of those, calling `storage.acquireWriterLock(...)` unconditionally
-throws `TypeError: storage.acquireWriterLock is not a function`. The guard
-turns that into a logged warning + graceful no-op so the app still boots,
-and the warning names the adapter class plus a remediation hint:
-
-```
-[brainy] Storage adapter `MmapFileSystemStorage` is missing the multi-process
-methods on its prototype chain. Writer locking and the flush-request RPC are
-disabled for this directory. Likely fix: clean install (`rm -rf node_modules
-bun.lockb && bun install`) or rebuild your container image to refresh
-`@soulcraft/brainy` to ≥7.21. See docs/concepts/storage-adapters.md.
-```
-
-## Authoring a new storage adapter — minimum checklist
-
-1. **Extend the right base class.**
-   - Filesystem-backed → `FileSystemStorage`.
-   - Cloud / network / custom → `BaseStorage`.
-
-2. **Override the capability flag.** If filesystem-backed:
-   ```typescript
-   public supportsMultiProcessLocking(): boolean { return true }
-   ```
-
-3. **Don't `super.X()`-wrap the multi-process methods**. They're inherited;
-   leaving them inherited means `hasStorageMethod()` finds them on the
-   prototype chain. Re-declaring them as `super.X()` wrappers makes the
-   helper resolve to your wrapper, which can fool the guard if your
-   constructor runs before the super class initializes.
-
-4. **Do override `init()` / `flush()` / `close()`** as needed. Always call
-   `super.init()` / `super.flush()` / `super.close()` first so the
-   filesystem prep, writer-lock acquisition, and lock release happen in the
-   expected order.
-
-5. **Verify the inheritance.** A one-line smoke test in your plugin's
-   `__tests__/`:
-   ```typescript
-   const s = new MyStorage(rootDir)
-   await s.init()
-   assert(typeof s.acquireWriterLock === 'function')
-   assert(s.supportsMultiProcessLocking() === true)
-   ```
-   If `acquireWriterLock` is undefined the prototype chain is broken at
-   install time — fix install, not your plugin.
-
-6. **Pin your peer dep generously.** `"peerDependencies": {
-   "@soulcraft/brainy": "^7.21.0" }` accepts any compatible 7.x. Don't pin
-   to an exact patch unless you're tracking a known regression.
-
-## Future direction
-
-The 7 multi-process methods are currently defaults on `BaseStorage`. A
-future refactor may extract them into a `MultiProcessSafeStorage`
-interface/mixin for cleaner separation — only adapters that opt in would
-expose them. This would require a minor bump and is tracked as an internal
-follow-up; consumers don't need to anticipate the change.
-
-## Reading material
-
-- [`concepts/multi-process`](./multi-process.md) — the writer-lock model,
-  heartbeat semantics, what the lock protects.
-- [`guides/inspection`](../guides/inspection.md) — `brainy inspect` and the
-  read-only mode.
-- `node_modules/@soulcraft/brainy/dist/storage/baseStorage.d.ts` — the
-  authoritative type signatures for every method this page references.

docs/eli5.md

@@ -1,155 +0,0 @@
----
-title: What is Brainy?
-slug: getting-started/what-is-brainy
-public: true
-category: getting-started
-template: guide
-order: 0
-description: Plain-language guide covering what Brainy does, how it compares to other tools, and what you can build with it. No jargon, no code — just clear analogies.
-next:
-  - getting-started/installation
-  - getting-started/quick-start
----
-
-# Brainy and Cor — Explained Simply
-
-*A plain-language guide for anyone who wants to understand what this thing actually does.*
-
----
-
-## What is Brainy?
-
-Imagine you have the world's smartest librarian.
-
-You walk up and say *"I'm looking for something about climate change — but only books published after 2020, and only ones written by authors I've already read."* A normal library would make you dig through a card catalogue, then cross-reference a list of authors, then scan the shelves yourself. That takes a while.
-
-Your smart librarian does all three at the same time — in less than the time it takes to blink.
-
-That's Brainy. It's a knowledge database that can search by **meaning**, follow **connections**, and filter by **labels** — all at once, in a single question.
-
----
-
-## The Three Superpowers
-
-### 1. Meaning Search (the "fuzzy" superpower)
-
-When you search for "automobile," Brainy also finds results about "car," "vehicle," and "sedan" — because it understands what words *mean*, not just how they're spelled. It reads your data the way a person would, not the way a search box does.
-
-Think of it like the librarian who finds books on "heartbreak" when you ask for something about "loneliness."
-
-### 2. Relationship Walking (the "follow the thread" superpower)
-
-Every piece of information can be connected to other pieces. A Person *works at* a Company. A Project *depends on* a Tool. A Recipe *contains* Ingredients.
-
-Brainy can follow these connections across many hops in one step. Ask for "everything connected to this author, two steps out" and Brainy returns the author's books, the books' publishers, the publishers' other authors — without you needing to chain four separate lookups yourself.
-
-Think of it like the librarian who not only hands you the book you asked for, but also knows which shelf it came from, who donated it, and what other books arrived in the same donation.
-
-### 3. Label Filtering (the "narrow it down" superpower)
-
-Sometimes meaning and connections aren't enough — you need precision. "Only recipes with fewer than 500 calories." "Only events from last week." "Only documents tagged as urgent."
-
-Brainy can narrow any result set down by exact labels or ranges in the same breath as the other two searches. No extra steps.
-
----
-
-## What Else Can It Do?
-
-- **Virtual file cabinet.** Brainy includes a full filesystem you can use to store, organize, and semantically search files — PDFs, documents, anything — the same way you search everything else.
-
-- **Live dashboards.** You can define running totals that Brainy keeps updated automatically — things like "total sales this month by region" or "average response time per service." Every time new data comes in, the numbers stay current with no manual recalculation.
-
-- **Time travel.** Every committed change becomes part of the database's history. You can pin the current state as a frozen view, see the whole knowledge base exactly as it was last week, try out changes in a scratch copy that never touches the real data, and take instant backups.
-
-- **Universal vocabulary.** Brainy ships with a shared language of 42 kinds of things (Person, Document, Task, Concept, Event…) and 127 kinds of connections (Contains, DependsOn, Creates, RelatedTo…). This means data from different sources speaks the same language without you having to translate.
-
----
-
-## What is Cor?
-
-Cor is a turbocharger for Brainy.
-
-Same car. Same controls. Same fuel. You just swap in a faster engine under the hood, and everything that used to take a moment now happens instantly.
-
-Technically, Cor is an optional plugin written in Rust — a lower-level language that runs much closer to the raw metal of your processor. It plugs into Brainy and takes over the most compute-intensive work: the distance calculations that power meaning search, the number-crunching behind live aggregates, and the set operations that drive label filtering.
-
-You install it with one line, register it with one call, and Brainy automatically uses it everywhere it can help.
-
----
-
-## How Much Faster?
-
-Plain language:
-
-- **Searches** go from "the blink of an eye" to "faster than a blink." The overall speedup is **5.2× on average** across all operations.
-- **Live aggregates** are rebuilt using all CPU cores in parallel, so re-indexing large datasets takes a fraction of the time.
-- **Analytics** that aren't even possible in pure JavaScript — real-time anomaly detection, streaming percentile estimates, approximate unique counts — become available because Cor brings the native capabilities required to run them efficiently.
-
-If Brainy is what makes knowledge fast, Cor is what makes Brainy feel instant.
-
----
-
-## What Does Brainy Replace?
-
-Most applications that need to store and search knowledge end up stitching together several specialized tools. Brainy replaces all of them with one — a single free, open-source library in place of multiple paid services.
-
-### Before and After
-
-**Before Brainy** — a pile of services:
-- Pinecone (vectors) + Neo4j (graph) + MongoDB (docs)
-- Algolia (search) + Redis (cache) + PostgreSQL + pgvector
-- Plus glue code, sync jobs, ETL pipelines, and 3am incidents
-
-**After Brainy** — one thing:
-Search, graph, filter, files, time travel, and imports — unified in a single library.
-
-### What Each Tool Is Missing
-
-| Tool | Search | Graph | Filter | VFS | Time travel | Import |
-|---|:---:|:---:|:---:|:---:|:---:|:---:|
-| **Brainy** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
-| *— Vector databases —* | | | | | | |
-| Pinecone | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Weaviate | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Qdrant | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Chroma | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| *— Graph databases —* | | | | | | |
-| Neo4j | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
-| *— Document stores —* | | | | | | |
-| MongoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Firestore | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| DynamoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| *— Relational + vector —* | | | | | | |
-| PostgreSQL + pgvector | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| MySQL | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| SQLite | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| *— Search engines —* | | | | | | |
-| Elasticsearch | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| Algolia | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
-| *— Cache —* | | | | | | |
-| Redis | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
-
-Brainy is the only row with every box checked. And it runs all of them in a single query — no stitching services together.
-
-### One library, any scale
-
-Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way.
-
-Add Cor and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.
-
----
-
-## What Can You Build?
-
-### Common applications
-
-- **AI agents with persistent memory** — Give any AI an always-on, self-organizing knowledge graph that persists between sessions and across agents.
-- **Searchable knowledge bases** — Build institutional memory that links documents automatically and surfaces answers across the full web of related information.
-- **Semantic document search** — Index PDFs, code, or media and find them by meaning, not just keywords.
-- **Relationship-aware recommendations** — Power product catalogs or content platforms where every recommendation understands what connects to what.
-- **Safe experiments** — Test risky changes against a scratch copy of the knowledge base, audit exactly what changed and when, and roll back to any snapshot instantly.
-- **Unified business platforms** — Combine booking, CRM, inventory, and analytics in one queryable knowledge graph with no sync pipeline.
-
-### What Brainy is good at
-
-Brainy is the engine underneath production systems that need to combine semantic search, structured filtering, and graph traversal in a single query — agent memory, knowledge-base platforms, business operations consoles, multi-agent coordination, and more. The combination of vector + graph + metadata search in one indexed call is what differentiates it from running three engines side by side.

docs/guides/MIGRATING_TO_V5.11.md

@@ -1,230 +0,0 @@
-# Migrating to v5.11.1
-
-## Overview
-
-v5.11.1 introduces a **breaking change** with **massive performance benefits**:
-
-- `brain.get()` now loads **metadata-only by default** (76-81% faster!)
-- Vector embeddings require **explicit opt-in**: `{ includeVectors: true }`
-
-**Impact**: Only ~6% of codebases need changes (code that computes similarity on retrieved entities).
-
-## What Changed
-
-### Before (v5.11.0 and earlier)
-
-```typescript
-const entity = await brain.get(id)
-// entity.vector was ALWAYS loaded (384 dimensions, 6KB)
-console.log(entity.vector.length)  // 384
-```
-
-### After (v5.11.1)
-
-```typescript
-// DEFAULT: Metadata-only (76-81% faster)
-const entity = await brain.get(id)
-console.log(entity.vector)  // [] (empty array - not loaded)
-
-// EXPLICIT: Full entity with vectors
-const entity = await brain.get(id, { includeVectors: true })
-console.log(entity.vector.length)  // 384
-```
-
-## Who Needs to Update?
-
-### ✅ NO CHANGES NEEDED (94% of code)
-
-If you use `brain.get()` for:
-- **VFS operations** (readFile, stat, readdir)
-- **Existence checks**: `if (await brain.get(id))`
-- **Metadata access**: `entity.data`, `entity.type`, `entity.metadata`
-- **Relationship traversal**
-- **Admin tools**, import utilities, data APIs
-
-→ **Zero changes needed, automatic 76-81% speedup!**
-
-### ⚠️ REQUIRES UPDATE (~6% of code)
-
-If you use `brain.get()` AND then compute similarity on the returned entity:
-
-```typescript
-// ❌ BEFORE (v5.11.0) - will break in v5.11.1
-const entity = await brain.get(id)
-const similar = await brain.similar({ to: entity.vector })  // entity.vector is [] !
-
-// ✅ AFTER (v5.11.1) - add includeVectors
-const entity = await brain.get(id, { includeVectors: true })
-const similar = await brain.similar({ to: entity.vector })  // Works!
-```
-
-**Note**: `brain.similar({ to: entityId })` (using ID) still works - no changes needed!
-
-## Migration Steps
-
-### Step 1: Find Affected Code
-
-Search your codebase for patterns that use vectors from `brain.get()`:
-
-```bash
-# Find brain.get() calls that access .vector
-grep -r "await brain.get(" --include="*.ts" --include="*.js" | \
-  grep -E "(\.vector|entity\.vector)"
-```
-
-### Step 2: Update Pattern-by-Pattern
-
-#### Pattern 1: Similarity Using Retrieved Entity Vector
-
-```typescript
-// ❌ BEFORE
-const entity = await brain.get(id)
-const similar = await brain.similar({ to: entity.vector })
-
-// ✅ AFTER - Option A: Add includeVectors
-const entity = await brain.get(id, { includeVectors: true })
-const similar = await brain.similar({ to: entity.vector })
-
-// ✅ AFTER - Option B: Use ID directly (recommended)
-const similar = await brain.similar({ to: id })
-```
-
-#### Pattern 2: Manual Vector Operations
-
-```typescript
-// ❌ BEFORE
-const entity = await brain.get(id)
-const magnitude = Math.sqrt(entity.vector.reduce((sum, v) => sum + v*v, 0))
-
-// ✅ AFTER
-const entity = await brain.get(id, { includeVectors: true })
-const magnitude = Math.sqrt(entity.vector.reduce((sum, v) => sum + v*v, 0))
-```
-
-#### Pattern 3: Vector Assertions in Tests
-
-```typescript
-// ❌ BEFORE
-const entity = await brain.get(id)
-expect(entity.vector).toBeDefined()
-expect(entity.vector.length).toBe(384)
-
-// ✅ AFTER
-const entity = await brain.get(id, { includeVectors: true })
-expect(entity.vector).toBeDefined()
-expect(entity.vector.length).toBe(384)
-```
-
-### Step 3: Verify Migration
-
-Run your test suite to catch any remaining issues:
-
-```bash
-npm test
-```
-
-Look for errors like:
-- `entity.vector is empty` or `entity.vector.length is 0`
-- `Cannot compute similarity on empty vector`
-
-Add `{ includeVectors: true }` wherever these errors occur.
-
-## Performance Impact
-
-### Before Migration
-```
-brain.get(): 43ms, 6KB per call
-VFS readFile(): 53ms per file
-VFS readdir(100 files): 5.3s
-```
-
-### After Migration
-```
-brain.get(): 10ms, 300 bytes per call (76-81% faster) ✨
-brain.get({ includeVectors: true }): 43ms, 6KB (unchanged)
-VFS readFile(): ~13ms per file (75% faster) ✨
-VFS readdir(100 files): ~1.3s (75% faster) ✨
-```
-
-**Result**:
-- VFS operations: **75% faster**
-- Metadata access: **76-81% faster**
-- Vector similarity: **Unchanged** (still fast when needed)
-
-## TypeScript Support
-
-The new `GetOptions` interface is fully typed:
-
-```typescript
-interface GetOptions {
-  /**
-   * Include 384-dimensional vector embeddings in the response
-   *
-   * Default: false (metadata-only for 76-81% speedup)
-   */
-  includeVectors?: boolean
-}
-
-// TypeScript will autocomplete and validate
-const entity = await brain.get(id, { includeVectors: true })
-```
-
-## Rollback Plan
-
-If you encounter issues, you can temporarily force full entity loading everywhere:
-
-```typescript
-// Temporary wrapper (NOT RECOMMENDED - defeats optimization)
-async function getLegacy(id: string) {
-  return brain.get(id, { includeVectors: true })
-}
-
-// Use throughout codebase while migrating
-const entity = await getLegacy(id)
-```
-
-**Important**: This defeats the 76-81% performance improvement. Only use temporarily while fixing affected code.
-
-## FAQ
-
-### Q: Why did you make this a breaking change?
-
-**A**: The performance gains are massive (76-81% speedup, 95% less bandwidth) and affect 94% of code positively. Only ~6% of code needs updates. The net benefit is enormous.
-
-### Q: Do I need to update my VFS code?
-
-**A**: No! VFS automatically benefits from the optimization with zero code changes. Your VFS operations are now 75% faster automatically.
-
-### Q: Will brain.similar() still work?
-
-**A**: Yes! `brain.similar({ to: entityId })` works exactly as before. Only `brain.similar({ to: entity.vector })` requires the entity to be loaded with `{ includeVectors: true }`.
-
-### Q: What about backward compatibility?
-
-**A**: Entities returned without vectors have `vector: []` (empty array), which is type-safe. Code that doesn't use vectors continues to work. Only code that explicitly uses `entity.vector` needs updating.
-
-### Q: Can I check if vectors are loaded?
-
-**A**: Yes! Check `entity.vector.length > 0` to detect if vectors were loaded.
-
-```typescript
-const entity = await brain.get(id)
-if (entity.vector.length > 0) {
-  // Vectors are loaded
-} else {
-  // Metadata-only
-}
-```
-
-## Support
-
-If you encounter migration issues:
-
-1. Check the [VFS Performance Guide](../vfs/VFS_PERFORMANCE.md)
-2. Review [API Reference](../api/README.md)
-3. See [Performance Documentation](../PERFORMANCE.md)
-4. File an issue: https://github.com/soulcraft/brainy/issues
-
-## Changelog
-
-See [CHANGELOG.md](../../CHANGELOG.md) for complete v5.11.1 release notes.

docs/guides/aggregation.md

@@ -1,593 +0,0 @@
-# Aggregation Guide
-
-> Real-time analytics on your entity data with incremental running totals
-
-## Overview
-
-Brainy's aggregation engine computes running totals at write time, so reading aggregate results is always O(1) regardless of dataset size. Define an aggregate once, and every `add()`, `update()`, and `delete()` automatically updates the running metrics.
-
-No batch jobs. No scheduled recalculations. Aggregates stay current with every write.
-
-**Defining over existing data:** if you define an aggregate on a store that already holds
-matching entities, Brainy backfills it from those entities on the first query (a one-time scan,
-then purely incremental). So `defineAggregate()` behaves the same whether you define it before
-or after the data exists.
-
-**Reopening a persisted brain:** aggregate state persists across restarts. Re-defining the
-same aggregate at boot (the normal declarative pattern) adopts the persisted state directly —
-no rescan. A backfill scan runs only when the definition actually changed, when no persisted
-state exists, or when the state failed to load; and however many aggregates need backfilling,
-they share a single scan.
-
-## Quick Start
-
-```typescript
-import { Brainy, NounType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// 1. Define an aggregate
-brain.defineAggregate({
-  name: 'sales_by_category',
-  source: { type: NounType.Event },
-  groupBy: ['category'],
-  metrics: {
-    revenue: { op: 'sum', field: 'amount' },
-    count:   { op: 'count' },
-    average: { op: 'avg', field: 'amount' }
-  }
-})
-
-// 2. Add entities — aggregates update automatically
-await brain.add({
-  data: 'Coffee purchase',
-  type: NounType.Event,
-  metadata: { category: 'food', amount: 5.50 }
-})
-
-await brain.add({
-  data: 'Laptop purchase',
-  type: NounType.Event,
-  metadata: { category: 'electronics', amount: 1200 }
-})
-
-await brain.add({
-  data: 'Lunch purchase',
-  type: NounType.Event,
-  metadata: { category: 'food', amount: 12.00 }
-})
-
-// 3. Query results
-const results = await brain.find({ aggregate: 'sales_by_category' })
-
-// Results:
-// [
-//   { groupKey: { category: 'food' }, metrics: { revenue: 17.50, count: 2, average: 8.75 } },
-//   { groupKey: { category: 'electronics' }, metrics: { revenue: 1200, count: 1, average: 1200 } }
-// ]
-```
-
-## Aggregation Operations
-
-Brainy supports 7 aggregation operations:
-
-### `sum` — Running Total
-
-Adds up all values of a numeric field.
-
-```typescript
-metrics: {
-  total_revenue: { op: 'sum', field: 'amount' }
-}
-```
-
-### `count` — Entity Count
-
-Counts the number of matching entities. No `field` required.
-
-```typescript
-metrics: {
-  order_count: { op: 'count' }
-}
-```
-
-### `avg` — Running Average
-
-Computes `sum / count` incrementally.
-
-```typescript
-metrics: {
-  average_price: { op: 'avg', field: 'price' }
-}
-```
-
-### `min` — Minimum Value
-
-Tracks the minimum value across all entities in each group.
-
-```typescript
-metrics: {
-  lowest_price: { op: 'min', field: 'price' }
-}
-```
-
-### `max` — Maximum Value
-
-Tracks the maximum value across all entities in each group.
-
-```typescript
-metrics: {
-  highest_price: { op: 'max', field: 'price' }
-}
-```
-
-### `stddev` — Sample Standard Deviation
-
-Computes the sample standard deviation using Welford's numerically stable online algorithm. Updates incrementally without storing individual values.
-
-```typescript
-metrics: {
-  price_spread: { op: 'stddev', field: 'price' }
-}
-```
-
-### `variance` — Sample Variance
-
-Computes the sample variance (square of standard deviation) using Welford's online algorithm.
-
-```typescript
-metrics: {
-  price_variance: { op: 'variance', field: 'price' }
-}
-```
-
-## GROUP BY Dimensions
-
-Every aggregate requires at least one `groupBy` dimension. Results are grouped by the unique combinations of dimension values.
-
-### Plain Fields
-
-Group by a metadata field value:
-
-```typescript
-groupBy: ['category']
-// Produces groups: { category: 'food' }, { category: 'electronics' }, ...
-```
-
-### Multiple Fields
-
-Group by multiple fields for composite keys:
-
-```typescript
-groupBy: ['category', 'region']
-// Produces groups: { category: 'food', region: 'US' }, { category: 'food', region: 'EU' }, ...
-```
-
-### Time Windows
-
-Group by a timestamp field bucketed into time periods:
-
-```typescript
-groupBy: [{ field: 'date', window: 'month' }]
-// Produces groups: { date: '2024-01' }, { date: '2024-02' }, ...
-```
-
-Available time window granularities:
-
-| Window | Format | Example |
-|--------|--------|---------|
-| `hour` | `YYYY-MM-DDThh` | `2024-01-15T14` |
-| `day` | `YYYY-MM-DD` | `2024-01-15` |
-| `week` | `YYYY-Wnn` | `2024-W03` |
-| `month` | `YYYY-MM` | `2024-01` |
-| `quarter` | `YYYY-Qn` | `2024-Q1` |
-| `year` | `YYYY` | `2024` |
-| `{ seconds: N }` | ISO 8601 | Custom interval |
-
-### Combined Dimensions
-
-Mix plain fields and time windows:
-
-```typescript
-brain.defineAggregate({
-  name: 'monthly_sales',
-  source: { type: NounType.Event },
-  groupBy: ['region', { field: 'date', window: 'month' }],
-  metrics: {
-    revenue: { op: 'sum', field: 'amount' },
-    count: { op: 'count' }
-  }
-})
-
-// Produces groups like:
-// { region: 'US', date: '2024-01' }
-// { region: 'US', date: '2024-02' }
-// { region: 'EU', date: '2024-01' }
-```
-
-### Array Fields (Unnest)
-
-Group by **each element** of an array-valued field — for tag frequencies, label counts, and
-faceted breakdowns. Mark the dimension `{ field, unnest: true }`:
-
-```typescript
-brain.defineAggregate({
-  name: 'tag_frequency',
-  source: { type: NounType.Document },
-  groupBy: [{ field: 'tags', unnest: true }],
-  metrics: { count: { op: 'count' } }
-})
-
-// A document tagged ['ml', 'ai'] contributes once to the 'ml' group and once to 'ai'.
-// Duplicate tags on one entity count once; an entity with no tags joins no group.
-const top = await brain.queryAggregate('tag_frequency', { orderBy: 'count', order: 'desc' })
-// [ { groupKey: { tags: 'ai' }, metrics: { count: 3 }, count: 3 }, ... ]
-```
-
-## Querying Aggregates
-
-Aggregate results are queried through the standard `find()` method.
-
-### Basic Query
-
-```typescript
-const results = await brain.find({ aggregate: 'sales_by_category' })
-```
-
-### Filter by Group Key
-
-Use `where` to filter on group key values:
-
-```typescript
-const foodOnly = await brain.find({
-  aggregate: 'sales_by_category',
-  where: { category: 'food' }
-})
-```
-
-### Filter by Metric Value (HAVING)
-
-Use `having` to filter groups by their **computed metric values** — the analytics equivalent of
-SQL `HAVING`. (`where` filters group *keys*; `having` filters *metrics*.)
-
-```typescript
-const bigCategories = await brain.find({
-  aggregate: 'sales_by_category',
-  having: { revenue: { greaterThan: 1000 } }
-})
-```
-
-`having` accepts the same operators as `where`, applied to each group's metric results plus
-`count`. It is evaluated per group — **O(groups), independent of entity count** — before sorting
-and pagination, so it stays cheap even over billions of entities.
-
-### Sort and Paginate
-
-Sort by any metric or group key field:
-
-```typescript
-const topCategories = await brain.find({
-  aggregate: {
-    name: 'sales_by_category',
-    orderBy: 'revenue',
-    order: 'desc',
-    limit: 10
-  }
-})
-```
-
-### Combined Parameters
-
-`where`, `orderBy`, `limit`, and `offset` from the outer `find()` call merge automatically with the aggregate query:
-
-```typescript
-const recentTopSpenders = await brain.find({
-  aggregate: 'monthly_sales',
-  where: { region: 'US' },
-  orderBy: 'revenue',
-  order: 'desc',
-  limit: 12,
-  offset: 0
-})
-```
-
-### Result Format
-
-`find({ aggregate })` returns `Result<T>` rows (for uniformity with the rest of `find()`),
-with the aggregate fields surfaced **both** at the top level and, for backward compatibility,
-flattened into `metadata`:
-
-```typescript
-{
-  id: string,
-  score: 1.0,
-  type: NounType.Measurement,
-  groupKey: { category: 'food' },             // top-level — the group key values
-  metrics:  { revenue: 17.50, count: 2, average: 8.75 }, // top-level — computed metrics
-  count: 2,                                   // top-level — entities in the group
-  metadata: {                                 // legacy mirror of the same data
-    __aggregate: 'sales_by_category',
-    category: 'food',
-    revenue: 17.50, count: 2, average: 8.75
-  },
-  entity: Entity
-}
-```
-
-### `queryAggregate()` — the report-friendly view
-
-For dashboards and reports, prefer `brain.queryAggregate(name, params)`. It returns the clean
-`AggregateResult[]` shape directly — no search-result wrapper:
-
-```typescript
-const rows = await brain.queryAggregate('sales_by_category', {
-  orderBy: 'revenue',
-  order: 'desc',
-  limit: 10
-})
-// [
-//   { groupKey: { category: 'electronics' }, metrics: { revenue: 1200, count: 1, average: 1200 }, count: 1 },
-//   { groupKey: { category: 'food' }, metrics: { revenue: 17.50, count: 2, average: 8.75 }, count: 2 }
-// ]
-```
-
-It accepts the same `where` / `having` / `orderBy` / `order` / `limit` / `offset` params as the
-`find({ aggregate })` form.
-
-## Source Filtering
-
-Control which entities feed into an aggregate with the `source` property.
-
-### Filter by Entity Type
-
-```typescript
-brain.defineAggregate({
-  name: 'event_stats',
-  source: { type: NounType.Event },
-  groupBy: ['category'],
-  metrics: { count: { op: 'count' } }
-})
-```
-
-### Filter by Multiple Types
-
-```typescript
-source: { type: [NounType.Event, NounType.Document] }
-```
-
-### Filter by Metadata
-
-Use the same `where` syntax as `find()`:
-
-```typescript
-source: {
-  type: NounType.Event,
-  where: { domain: 'financial', subtype: 'transaction' }
-}
-```
-
-### Filter by Service
-
-For multi-tenant deployments:
-
-```typescript
-source: { service: 'tenant-123' }
-```
-
-Entities that don't match the source filter are silently skipped during incremental updates.
-
-## Incremental Updates
-
-The aggregation engine hooks into every write operation:
-
-### On `add()`
-
-When a new entity matches an aggregate's source filter:
-1. The group key is computed from the entity's metadata
-2. Each metric in the matching group is incremented
-3. New groups are created automatically
-
-### On `update()`
-
-When an existing entity is updated:
-1. The old entity's contribution is reversed from its group
-2. The new entity's contribution is applied to its (potentially different) group
-3. Handles group key changes — an entity moving from category "food" to "drink" updates both groups
-
-### On `delete()`
-
-When an entity is deleted:
-1. The entity's contribution is reversed from its group
-2. If a group becomes empty (all metric counts reach zero), it's removed
-
-### Aggregate Entity Exclusion
-
-Materialized `NounType.Measurement` entities are automatically excluded from all source matching, preventing infinite feedback loops. Entities with `service: 'brainy:aggregation'` or `metadata.__aggregate` are always skipped.
-
-## Materialization
-
-Materialization writes aggregate results as `NounType.Measurement` entities, making them automatically available through OData, Google Sheets, SSE, and webhook integrations.
-
-```typescript
-brain.defineAggregate({
-  name: 'daily_metrics',
-  source: { type: NounType.Event },
-  groupBy: [{ field: 'date', window: 'day' }],
-  metrics: {
-    total: { op: 'sum', field: 'amount' },
-    count: { op: 'count' }
-  },
-  materialize: true
-})
-```
-
-### Debounce Configuration
-
-During high-throughput ingestion, materialization is debounced to avoid excessive writes:
-
-```typescript
-materialize: {
-  debounceMs: 2000,      // Wait 2 seconds after last update before writing
-  trackSources: true     // Track which entities contributed
-}
-```
-
-The default debounce interval is 1000ms.
-
-## Multiple Aggregates
-
-Define multiple aggregates that process the same entities:
-
-```typescript
-// Revenue by category
-brain.defineAggregate({
-  name: 'category_revenue',
-  source: { type: NounType.Event },
-  groupBy: ['category'],
-  metrics: { total: { op: 'sum', field: 'amount' } }
-})
-
-// Monthly trends
-brain.defineAggregate({
-  name: 'monthly_trends',
-  source: { type: NounType.Event },
-  groupBy: [{ field: 'date', window: 'month' }],
-  metrics: {
-    revenue: { op: 'sum', field: 'amount' },
-    count: { op: 'count' },
-    avg_order: { op: 'avg', field: 'amount' }
-  }
-})
-
-// Regional breakdown with statistical analysis
-brain.defineAggregate({
-  name: 'regional_analysis',
-  source: { type: NounType.Event },
-  groupBy: ['region'],
-  metrics: {
-    revenue: { op: 'sum', field: 'amount' },
-    spread: { op: 'stddev', field: 'amount' },
-    variance: { op: 'variance', field: 'amount' }
-  }
-})
-```
-
-Each `add()` call updates all matching aggregates automatically.
-
-## Removing Aggregates
-
-Remove an aggregate and clean up its state:
-
-```typescript
-brain.removeAggregate('category_revenue')
-```
-
-## Persistence
-
-Aggregate definitions and running state are automatically persisted:
-
-- **On `flush()`/`close()`**: All dirty aggregate state is written to storage
-- **On `init()`**: Definitions and state are restored from storage
-- **Change detection**: Definition changes are detected via FNV-1a hashing — only changed aggregates reset their state on restart
-
-## Native Acceleration
-
-When [Cor](https://www.npmjs.com/package/@soulcraft/cor) is installed as a plugin, the aggregation engine automatically uses Rust-accelerated computation:
-
-- Incremental updates run in Rust with BTreeMap-backed precise MIN/MAX
-- Welford's online stddev/variance computed natively
-- Rebuild uses Rayon parallel iterators across CPU cores (above 1,000 entities)
-- Time window bucketing uses integer arithmetic without `Date` object allocation
-
-```typescript
-const brain = new Brainy({
-  plugins: ['@soulcraft/cor']
-})
-await brain.init()
-
-// Aggregation automatically uses native engine
-brain.defineAggregate({ ... })
-```
-
-Verify native acceleration is active:
-
-```typescript
-const diag = brain.diagnostics()
-console.log(diag.providers.aggregation)
-// { source: 'plugin' }
-```
-
-## Common Patterns
-
-### Financial Analytics
-
-```typescript
-brain.defineAggregate({
-  name: 'monthly_spending',
-  source: {
-    type: NounType.Event,
-    where: { domain: 'financial', subtype: 'transaction' }
-  },
-  groupBy: [
-    'category',
-    { field: 'date', window: 'month' }
-  ],
-  metrics: {
-    total:   { op: 'sum', field: 'amount' },
-    count:   { op: 'count' },
-    average: { op: 'avg', field: 'amount' },
-    highest: { op: 'max', field: 'amount' },
-    lowest:  { op: 'min', field: 'amount' }
-  },
-  materialize: true
-})
-```
-
-### Time-Series Monitoring
-
-```typescript
-brain.defineAggregate({
-  name: 'hourly_metrics',
-  source: { type: NounType.Event, where: { domain: 'monitoring' } },
-  groupBy: [
-    'service',
-    { field: 'timestamp', window: 'hour' }
-  ],
-  metrics: {
-    request_count:  { op: 'count' },
-    avg_latency:    { op: 'avg', field: 'latency_ms' },
-    max_latency:    { op: 'max', field: 'latency_ms' },
-    error_count:    { op: 'sum', field: 'is_error' },
-    latency_spread: { op: 'stddev', field: 'latency_ms' }
-  }
-})
-```
-
-### Content Analytics
-
-```typescript
-brain.defineAggregate({
-  name: 'content_stats',
-  source: { type: NounType.Document },
-  groupBy: ['author', { field: 'publishedAt', window: 'month' }],
-  metrics: {
-    articles:   { op: 'count' },
-    total_words: { op: 'sum', field: 'wordCount' },
-    avg_words:   { op: 'avg', field: 'wordCount' }
-  }
-})
-```
-
-## Performance
-
-Aggregation complexity per write is O(A x G x M) where A = matching aggregates, G = groupBy dimensions, M = metrics. For typical configurations (2-5 aggregates, 1-3 dimensions, 3-5 metrics), this is effectively O(1).
-
-With Cor native acceleration:
-
-| Operation | Throughput | Latency |
-|-----------|-----------|---------|
-| Incremental update (1K entities) | 809 ops/s | 1.2 ms |
-| Rebuild (10K entities) | 475 ops/s | 2.1 ms |
-| Rebuild (100K entities, Rayon) | 66 ops/s | 15.2 ms |
-| Query (1K groups, sort + paginate) | 986 ops/s | 1.0 ms |

docs/guides/enterprise-for-everyone.md

@@ -1,441 +0,0 @@
-# Enterprise for Everyone
-
-> **Philosophy**: We believe enterprise features should be available to everyone. This document shows what's available now and what's coming soon.
-
-## Our Philosophy: No Premium Tiers, No Limitations
-
-Brainy believes that **enterprise-grade features should be available to everyone**—from indie developers to Fortune 500 companies. Every Brainy installation includes the complete feature set with no artificial limitations, no premium tiers, and no feature gates.
-
-> "Why should a student project have worse data durability than a billion-dollar company? They shouldn't." - Brainy Philosophy
-
-## What You Get
-
-### ✅ Available Now
-Core enterprise features that work today.
-
-### 🚧 Coming Soon
-Enterprise features on our roadmap.
-
-### 🔒 Enterprise Security 🚧 Coming Soon
-
-**Everyone gets bank-level security features:**
-
-```typescript
-const brain = new Brainy({
-  security: {
-    encryption: 'aes-256-gcm',        // Military-grade encryption
-    keyRotation: true,                // Automatic key rotation
-    auditLog: true,                   // Complete audit trail
-    zeroKnowledge: true,              // Client-side encryption available
-    compliance: ['SOC2', 'HIPAA', 'GDPR']  // Compliance-ready
-  }
-})
-```
-
-**Features included:**
-- **At-rest encryption**: All data encrypted with AES-256
-- **In-transit encryption**: TLS 1.3 for all communications
-- **Key management**: Automatic rotation and secure storage
-- **Access control**: Role-based permissions
-- **Audit logging**: Every operation tracked
-- **Data residency**: Control where your data lives
-- **Zero-knowledge option**: Even Brainy can't read your data
-
-### 💾 Enterprise Durability ✅ Available Now
-
-**Everyone gets mission-critical reliability:**
-
-```typescript
-
-const brain = new Brainy({
-  augmentations: [
-      enabled: true,                    // Write-ahead logging
-      redundancy: 3,                    // Triple redundancy
-      checkpointInterval: 1000,         // Frequent checkpoints
-      crashRecovery: true,              // Automatic recovery
-      pointInTimeRecovery: true         // Time travel capability
-    })
-  ]
-})
-
-// Your data is as safe as any Fortune 500 company's
-```
-
-**Features included:**
-- **Write-ahead logging**: Never lose a write
-- **ACID compliance**: Full transactional guarantees
-- **Automatic backups**: Continuous protection
-- **Point-in-time recovery**: Restore to any moment
-- **Crash recovery**: Automatic healing
-- **Zero data loss**: RPO = 0
-- **High availability**: 99.99% uptime capable
-
-### 🚀 Enterprise Performance ✅ Available Now
-
-**Everyone gets blazing-fast performance:**
-
-```typescript
-// These optimizations are automatic and free for everyone
-const performance = {
-  vectorSearch: 'HNSW',              // O(log n) similarity search
-  fieldLookup: 'O(1)',               // Constant-time metadata access
-  caching: 'Multi-level',            // L1/L2/L3 intelligent caching
-  indexing: 'Automatic',             // Self-optimizing indexes
-  batching: 'Dynamic',               // Adaptive batch processing
-  parallelism: 'Auto-scaled',       // Uses all available cores
-  gpu: 'Auto-detected'               // GPU acceleration when available
-}
-```
-
-**Performance features:**
-- **Sub-millisecond queries**: With proper indexing
-- **Million+ entities**: Handles massive scale
-- **Streaming ingestion**: 100k+ operations/second
-- **Auto-optimization**: Learns and improves
-- **Resource adaptation**: Uses available hardware optimally
-- **No artificial limits**: No throttling or quotas
-
-### 📊 Enterprise Observability 🚧 Coming Soon
-
-**Everyone gets complete visibility:**
-
-```typescript
-import { MonitoringAugmentation } from 'brainy'
-
-const brain = new Brainy({
-  augmentations: [
-    new MonitoringAugmentation({
-      metrics: 'all',                  // Complete metrics
-      tracing: true,                   // Distributed tracing
-      profiling: true,                 // Performance profiling
-      alerting: true,                  // Anomaly detection
-      dashboard: true                  // Real-time dashboard
-    })
-  ]
-})
-
-brain.on('metrics', (metrics) => {
-  // Same metrics Facebook uses, but free for you
-  console.log({
-    qps: metrics.queriesPerSecond,
-    p99: metrics.latencyP99,
-    errorRate: metrics.errorRate,
-    cacheHit: metrics.cacheHitRate
-  })
-})
-```
-
-**Observability features:**
-- **Real-time metrics**: Operations, latency, throughput
-- **Distributed tracing**: Track requests across systems
-- **Performance profiling**: Find bottlenecks
-- **Anomaly detection**: Automatic alerts
-- **Custom dashboards**: Visualize your data
-- **Export to any system**: Prometheus, Grafana, DataDog
-
-### 🔄 Enterprise Integration 🚧 Coming Soon
-
-**Everyone gets seamless connectivity:**
-
-```typescript
-// Import from any data source
-await brain.importFromSQL('postgres://production-db')
-await brain.importFromMongo('mongodb://analytics')
-await brain.importFromAPI('https://api.company.com/data')
-await brain.importFromStream('kafka://events')
-
-// Export to any format
-await brain.exportToParquet('./data.parquet')
-await brain.exportToJSON('./backup.json')
-await brain.exportToSQL('mysql://backup')
-
-// Sync with any system
-await brain.syncWith({
-  elasticsearch: 'https://search.company.com',
-  redis: 'redis://cache.company.com',
-  webhooks: 'https://api.company.com/hooks'
-})
-```
-
-**Integration features:**
-- **Universal import**: SQL, NoSQL, CSV, JSON, XML, APIs
-- **Universal export**: Any format you need
-- **Real-time sync**: Keep systems in sync
-- **Streaming connectors**: Kafka, Redis, WebSockets
-- **Webhook support**: React to changes
-- **API generation**: Auto-generate REST/GraphQL APIs
-
-### 🌍 Scale
-
-**Everyone gets the same scale model:**
-
-```typescript
-// Pure JS by default; install the optional native provider for billions of vectors
-const brain = new Brainy()
-
-// 1 → ~1M vectors: pure-JS HNSW, zero extra setup
-// 1M → 10B+ vectors: install @soulcraft/cor for the native DiskANN provider
-```
-
-**Scaling model:**
-- **Single process, no cluster**: Brainy runs in one process — no coordinator,
-  no peer discovery, no consensus to operate
-- **Optional native provider**: install `@soulcraft/cor` to back the index with
-  on-disk DiskANN that scales to 10B+ vectors on one machine
-- **Per-tenant pools**: isolate tenants by giving each its own Brainy instance and
-  storage directory
-- **Horizontal read scaling**: run many reader processes against one shared on-disk
-  store (single writer, many readers); replicate the artifact with your operator
-  tooling
-
-### 🛡️ Enterprise Compliance 🚧 Coming Soon
-
-**Everyone gets compliance tools:**
-
-```typescript
-const brain = new Brainy({
-  compliance: {
-    gdpr: {
-      rightToDelete: true,           // Automatic PII deletion
-      rightToExport: true,           // Data portability
-      consentTracking: true,         // Consent management
-      dataMinimization: true         // Automatic data pruning
-    },
-    hipaa: {
-      encryption: true,               // PHI encryption
-      accessLogging: true,            // Access audit trail
-      minimumNecessary: true         // Access restrictions
-    },
-    sox: {
-      auditTrail: true,              // Complete audit log
-      changeControl: true,           // Version control
-      segregationOfDuties: true      // Role separation
-    }
-  }
-})
-```
-
-**Compliance features:**
-- **GDPR ready**: Full data privacy toolkit
-- **HIPAA compliant**: Healthcare data protection
-- **SOX compliant**: Financial controls
-- **CCPA support**: California privacy rights
-- **ISO 27001**: Information security
-- **PCI DSS**: Payment card security
-
-### 🤖 Enterprise AI/ML ⚠️ Partially Available
-
-> **Current**: Basic embeddings and vector search work. Advanced features coming soon.
-
-**Everyone gets advanced AI features:**
-
-```typescript
-// Advanced AI capabilities for everyone
-const brain = new Brainy({
-  ai: {
-    embeddings: 'state-of-the-art',  // Best models available
-    dimensions: 1536,                 // High-precision vectors
-    multimodal: true,                 // Text, image, audio
-    fineTuning: true,                 // Custom model training
-    activeLearning: true,             // Improves with usage
-    explainability: true              // Understand decisions
-  }
-})
-
-// Use enterprise AI features
-const results = await brain.find("complex natural language query")
-const explanation = await brain.explain(results)
-const recommendations = await brain.recommend(userId)
-const anomalies = await brain.detectAnomalies()
-```
-
-**AI features:**
-- **State-of-the-art models**: Latest embeddings
-- **Multi-modal support**: Text, images, code, audio
-- **Fine-tuning**: Adapt to your domain
-- **Active learning**: Improves with feedback
-- **Explainable AI**: Understand decisions
-- **Anomaly detection**: Find outliers automatically
-
-### 🔧 Enterprise Operations
-
-**Everyone gets DevOps excellence:**
-
-```typescript
-// CI/CD and DevOps features
-const brain = new Brainy({
-  operations: {
-    blueGreen: true,                 // Zero-downtime deployments
-    canary: true,                    // Gradual rollouts
-    featureFlags: true,              // Feature toggling
-    migrations: true,                // Automatic migrations
-    versioning: true,                // API versioning
-    rollback: true                   // Instant rollback
-  }
-})
-
-// Same deployment strategies as Google
-```
-
-**Operations features:**
-- **Blue-green deployments**: Zero downtime
-- **Canary releases**: Gradual rollout
-- **Feature flags**: Toggle features instantly
-- **Automatic migrations**: Schema evolution
-- **Version control**: Full history
-- **Instant rollback**: Undo mistakes quickly
-
-## Why Enterprise for Everyone?
-
-### 1. **Democratizing Technology**
-Small teams and individual developers deserve the same powerful tools as large corporations. Innovation shouldn't be limited by budget.
-
-### 2. **No Artificial Limitations**
-We don't cripple our software to create premium tiers. Every limitation in Brainy is technical, not commercial.
-
-### 3. **Community-Driven**
-When everyone has access to enterprise features, the entire community benefits from improvements, bug fixes, and innovations.
-
-### 4. **True Open Source**
-MIT licensed means you can:
-- Use commercially without fees
-- Modify for your needs
-- Contribute improvements
-- Build a business on it
-- Never worry about licensing
-
-### 5. **Future-Proof**
-Your hobby project today might be tomorrow's unicorn startup. With Brainy, you won't need to migrate to "enterprise" software as you grow.
-
-## Real-World Impact
-
-### Startups
-```typescript
-// A 2-person startup gets the same features as Amazon
-const startup = new Brainy()
-// ✓ Full durability
-// ✓ Complete security
-// ✓ Unlimited scale
-// ✓ Zero licensing fees
-```
-
-### Education
-```typescript
-// Students learn with production-grade tools
-const classroom = new Brainy()
-// ✓ No feature restrictions
-// ✓ Real enterprise experience
-// ✓ Free forever
-```
-
-### Non-Profits
-```typescript
-// NGOs get enterprise features without enterprise costs
-const nonprofit = new Brainy()
-// ✓ Compliance tools
-// ✓ Security features
-// ✓ Scale for impact
-// ✓ $0 licensing
-```
-
-### Enterprises
-```typescript
-// Enterprises get everything plus peace of mind
-const enterprise = new Brainy()
-// ✓ Proven at scale
-// ✓ Community tested
-// ✓ No vendor lock-in
-// ✓ Optional support available
-```
-
-## No Compromises
-
-### What you DON'T get with Brainy:
-- ❌ Artificial rate limits
-- ❌ Feature gates
-- ❌ Premium tiers
-- ❌ Usage quotas
-- ❌ Seat licenses
-- ❌ Renewal fees
-- ❌ Vendor lock-in
-- ❌ Proprietary formats
-
-### What you DO get:
-- ✅ Everything
-- ✅ Forever
-- ✅ For free
-- ✅ MIT licensed
-
-## Support Options
-
-While the software is free and complete, we offer optional support:
-
-### Community Support (Free)
-- GitHub Discussions
-- Stack Overflow
-- Discord community
-- Extensive documentation
-
-### Professional Support (Optional)
-- Priority response
-- Architecture review
-- Performance tuning
-- Custom training
-- SLA guarantees
-
-## Getting Started
-
-```bash
-# Install Brainy - get everything immediately
-npm install brainy
-
-# That's it. You now have enterprise-grade AI database
-```
-
-```typescript
-import { Brainy } from 'brainy'
-
-// Create your enterprise-grade database
-const brain = new Brainy()
-await brain.init()
-
-// You're now running the same tech as Fortune 500 companies
-await brain.add("Your data is enterprise-grade", {
-  secure: true,
-  durable: true,
-  scalable: true,
-  free: true
-})
-```
-
-## Comparison
-
-| Feature | Traditional Enterprise DB | Brainy |
-|---------|--------------------------|--------|
-| License Cost | $100k-1M/year | $0 |
-| User Limits | Per seat licensing | Unlimited |
-| Feature Access | Tiered | Everything |
-| Durability | ✅ | ✅ |
-| Security | ✅ | ✅ |
-| Scale | ✅ | ✅ |
-| AI/ML | Additional cost | ✅ Included |
-| Support | Required | Optional |
-| Lock-in | Significant | None |
-| Source Code | Proprietary | MIT Open Source |
-
-## Our Promise
-
-> "Every feature we build goes to everyone. Every optimization benefits all users. Every security enhancement protects the entire community. This is Enterprise for Everyone."
-
-## Join the Revolution
-
-Brainy is more than software—it's a movement to democratize enterprise technology. When everyone has access to the best tools, we all build better things.
-
-**Welcome to enterprise-grade. Welcome to Brainy.**
-
-## See Also
-
-- [Zero Configuration](../architecture/zero-config.md)
-- [Augmentations System](../architecture/augmentations.md)
-- [Architecture Overview](../architecture/overview.md)
-- [API Reference](../api/README.md)

docs/guides/export-and-import.md

@@ -1,181 +0,0 @@
----
-title: Export & Import (portable graph)
-slug: guides/export-and-import
-public: true
-category: guides
-template: guide
-order: 9
-description: Export part or all of a brain to a portable, versioned PortableGraph document and import it back — by id, collection, connected neighbourhood, VFS subtree, predicate, or the whole brain. export() lives on the immutable Db, so asOf()/with() give time-travel and what-if exports.
-next:
-  - guides/subtypes-and-facets
-  - api/README
----
-
-# Export & Import (portable graph)
-
-Brainy serializes part or all of a brain — an item, a collection, a connected
-neighbourhood, a VFS subtree, a predicate match, or the whole brain — into a single
-versioned JSON document (`PortableGraph`), and restores it.
-
-```typescript
-const graph = await brain.export()   // whole brain → PortableGraph
-await brain.import(graph)             // restore (merge by id, re-embed if no vectors)
-```
-
-It is **portable** (human-readable JSON), **versioned** (`formatVersion`, so a document
-written by 7.x imports cleanly into 8.0), and **current-state** (the entities and edges as
-they are now — no generation history). Use it for portable artifacts, partial exports,
-cross-environment moves, and version upgrades.
-
-`export()` is a method on the **immutable `Db` value**, so it composes with every way of
-obtaining one:
-
-```typescript
-brain.export(sel)                                   // = brain.now().export(sel)
-;(await brain.asOf(gen)).export(sel)                // time-travel export (a past generation)
-brain.now().with(ops).export(sel)                   // what-if export (a speculative state)
-```
-
-## When to use which
-
-| You want… | Use |
-|-----------|-----|
-| A portable, partial-or-whole, cross-version graph document | **`brain.export()` / `brain.import()`** (this guide) |
-| A whole-brain snapshot **with generation history** | `brain.now().persist(path)` / `Brainy.load(path)` (native) |
-| To ingest a CSV / PDF / Excel / JSON **file** as new entities | `brain.import(file)` — see [Import Anything](./import-anything.md) |
-
-`import()` is **polymorphic**: hand it a `PortableGraph` and it does the graph round-trip;
-hand it a file/buffer and it does foreign-file ingestion (dispatched on the document's
-`format: 'brainy-portable-graph'` tag).
-
-## Exporting
-
-```typescript
-brain.export(selector?, options?): Promise<PortableGraph>
-// (also on any Db: brain.now().export(...), (await brain.asOf(g)).export(...))
-```
-
-### Selectors — *what* to export
-
-Omit the selector to export the whole brain. Otherwise pick a node set:
-
-| Scenario | Selector |
-|----------|----------|
-| Just an item (or items) | `{ ids: ['a', 'b'] }` |
-| A collection + its children | `{ collection: collectionId }` (alias `memberOf`) |
-| A connected neighbourhood | `{ connected: { from: id, depth: 2, verbs?, direction? } }` |
-| A VFS directory / file (+ subtree) | `{ vfsPath: '/docs', recursive?: true }` |
-| Everything matching a predicate | `{ type, subtype, where, service, visibility }` |
-| The whole brain | *(omit)* |
-
-The selector reuses `find()`'s grammar — *"export what `find()` would match, minus ranking
-and limit."* Structural and predicate selectors **compose**:
-
-```typescript
-// Members of a collection whose status is "open"
-await brain.export({ collection: collectionId, where: { status: 'open' } })
-
-// Already have find() results? Export exactly those with the ids selector
-const hits = await brain.find({ type: NounType.Document })
-await brain.export({ ids: hits.map(r => r.id) })
-```
-
-### Options — *how* to serialize
-
-| Option | Default | Effect |
-|--------|---------|--------|
-| `includeVectors` | `false` | Carry embedding vectors verbatim. Off ⇒ `import()` re-embeds from `data`. |
-| `includeContent` | `false` | Include VFS file bytes in `blobs` so files round-trip byte-identically. |
-| `includeSystem` | `false` | Include `visibility:'system'` entities such as the VFS root. |
-| `edges` | `'induced'` | `'induced'` (both endpoints in the set), `'incident'` (also dangling edges, recorded in `danglingIds`), or `'none'` (nodes only). |
-
-## Importing
-
-```typescript
-brain.import(graph, options?): Promise<ImportResult>
-```
-
-The whole graph is applied as **one atomic transaction** — it advances the brain exactly
-one generation, or none on failure.
-
-```typescript
-const result = await brain.import(graph, { onConflict: 'merge' })
-// → { imported, merged, skipped, reembedded, blobsWritten, errors }
-```
-
-| Option | Default | Effect |
-|--------|---------|--------|
-| `onConflict` | `'merge'` | `'merge'` (update existing id in place — assemble many exported graphs), `'replace'` (delete + recreate), or `'skip'`. |
-| `reembed` | `'auto'` | `'auto'` (use the carried vector, else re-embed from `data`) or `'never'` (require a carried vector; record an error if absent). |
-| `remapIds` | — | Rewrite every id on the way in, e.g. to clone a template subgraph under fresh ids. |
-| `meta` | — | Transaction metadata recorded in the tx-log alongside the new generation. |
-
-The default `onConflict: 'merge'` lets you assemble one working graph from many exported
-documents that share entity ids — re-importing an id merges rather than duplicates.
-
-## The `PortableGraph` format
-
-```jsonc
-{
-  "format": "brainy-portable-graph",  // identifies the document type
-  "formatVersion": 1,                 // import gates on this (cross-version migration)
-  "brainyVersion": "8.0.0",
-  "createdAt": "2026-06-16T…Z",
-  "embedding": { "model": "all-MiniLM-L6-v2", "dimensions": 384 },
-  "selector": { … },                  // echoes what was exported (provenance)
-  "entities": [
-    {
-      "id": "…", "type": "Document", "subtype": "invoice", "visibility": "public",
-      "data": "…",                    // the embedding source
-      "confidence": 1, "weight": 1, "service": "…",
-      "vector": [ … ],                // only with includeVectors
-      "metadata": { … }               // custom fields only (reserved fields are top-level)
-    }
-  ],
-  "relations": [
-    { "id":"…", "from":"…", "to":"…", "type":"Contains", "subtype":"…",
-      "weight":1, "confidence":1, "metadata": { … } }
-  ],
-  "blobs": { "<sha256>": "<base64>" }, // only with includeContent
-  "danglingIds": [ "…" ],             // only with edges:'incident'
-  "stats": { "entityCount": 0, "relationCount": 0, "blobCount": 0, "vectorDimensions": 384 }
-}
-```
-
-Standard fields (`subtype`, `visibility`, `data`, `confidence`, `weight`, `service`) sit at
-the top level of each entity; `metadata` holds **only** custom user fields — mirroring the
-in-memory `Entity` shape, so `import()` maps each field to its dedicated parameter. The
-TypeScript types (`PortableGraph`, `PortableGraphEntity`, `PortableGraphRelation`, `ExportSelector`,
-`ExportOptions`, `ImportOptions`, `ImportResult`) are exported from the package root.
-
-## Generations & time-travel
-
-The portable document is **current-state** — it never embeds generation history (that keeps
-it cross-version-portable). History lives where it's queryable:
-
-- **During a session:** `brain.asOf(g)` / `brain.now().with(ops)` on the live brain. Because
-  `export()` is on the `Db`, `(await brain.asOf(g)).export()` serializes a *past* generation
-  and `brain.now().with(ops).export()` serializes a *speculative* one.
-- **A whole-brain snapshot with history:** `brain.now().persist(path)` / `Brainy.load(path)`
-  (native, generation-preserving) — a separate facility from this portable format.
-
-Note: only `transact()` (and the write shortcuts that commit through it) advances a
-generation, so time-travel export differs across transaction boundaries.
-
-## Cross-version (7.x → 8.0)
-
-Because the document is shared and versioned, a PortableGraph written by 7.x imports into 8.0:
-`formatVersion` is read forward, `subtype` is carried so 8.0 re-types correctly, and the
-same 384-dimension model on both lines means `includeVectors:false` re-embeds identically
-(or `true` carries vectors verbatim).
-
-## VFS
-
-VFS directories are `Collection` entities and files are entities linked by `Contains`, so
-the whole filesystem (or any subtree) exports through the `vfsPath` selector:
-
-```typescript
-await brain.export({ vfsPath: '/' },        { includeContent: true }) // all VFS + bytes
-await brain.export({ vfsPath: '/docs' },    { includeContent: true }) // one directory
-await brain.export({ vfsPath: '/a/b.txt' }, { includeContent: true }) // one file
-```

docs/guides/external-backups-and-sparse-storage.md

@@ -1,99 +0,0 @@
----
-title: External Backups & Sparse Storage
-slug: guides/external-backups
-public: true
-category: guides
-template: guide
-order: 10
-description: How to back up a brain directory with external tools (tar, rsync, cp) without exploding sparse files — why a store can show 100+ GB "apparent" size on a small disk, which files are sparse, and how persist()/restore() handle it for you.
-next:
-  - guides/snapshots-and-time-travel
-  - concepts/storage-adapters
----
-
-# External Backups & Sparse Storage
-
-The built-in snapshot path — [`db.persist()` and `brain.restore()`](/docs/guides/snapshots-and-time-travel) —
-already handles everything on this page for you. Read this when you back up a brain directory with
-**external tools**: `tar`, `rsync`, `cp`, `scp`, or a filesystem-level backup agent.
-
-## The one-sentence rule
-
-> **Always use the sparse-aware flag**: `tar czSf` (capital `S`), `rsync --sparse`,
-> `cp --sparse=always`. A naive copy can turn a 2 GB store into a 100+ GB one — or fail
-> the disk entirely.
-
-## Why: some files are sparse
-
-When a native accelerator plugin is active, parts of the index live in **memory-mapped files**
-created at a large fixed virtual size — the file's *apparent* size — while the filesystem only
-allocates blocks that were actually written. A brand-new id-mapper file can report tens of
-gigabytes in `ls -l` while occupying a few megabytes on disk.
-
-Check the difference yourself:
-
-```bash
-ls -lh  brain-data/_id_mapper/    # APPARENT size (can be huge)
-du -sh  brain-data/               # ALLOCATED size (the real footprint)
-```
-
-The sparse candidates in a brain directory:
-
-| Path | What it is |
-|---|---|
-| `_id_mapper/` | The native id-mapper's mmap files (large fixed virtual size) |
-| `_blobs/` | Native index files (vector base, segments) — may be mmap-backed |
-
-Everything else (entities, `_system`, `_generations`, `_cas` content blobs) is ordinary dense data.
-
-## Doing it right
-
-**tar** — the `S` flag detects holes and stores only real data:
-
-```bash
-tar czSf brain-backup.tgz /data/brain
-# restore preserves the holes:
-tar xzSf brain-backup.tgz -C /data/
-```
-
-**rsync**:
-
-```bash
-rsync -a --sparse /data/brain/ backup-host:/backups/brain/
-```
-
-**cp**:
-
-```bash
-cp -a --sparse=always /data/brain /backups/brain
-```
-
-**What goes wrong without the flag:** the copy *materializes* every hole as real zero bytes.
-A store whose apparent size exceeds the target disk fails with `ENOSPC` partway through — and a
-copy that *does* fit silently costs the full apparent size in storage and transfer time.
-
-## What the built-in paths do (so you don't have to)
-
-- **`db.persist(path)`** snapshots via **hard links** — instant and space-shared, since every data
-  file is immutable-by-rename. The handful of append-in-place files (the transaction log, the
-  commit fact log's tail segment) and mmap-mutated directories (`_id_mapper/`) are **byte-copied**
-  instead, so a post-snapshot write can never reach through a shared inode into your backup.
-- **`brain.restore(path, { confirm: true })`** is **non-destructive and sparse-aware**: the snapshot
-  is copied into a staging area *before* any live data is touched (all-zero blocks stay holes), and
-  only after the copy fully succeeds does an atomic swap move it into place. A failed copy —
-  including `ENOSPC` — leaves the live store exactly as it was. A crash mid-swap completes forward
-  on the next open.
-
-## Live-store caveats for external tools
-
-1. **Prefer snapshotting a `persist()` output, not the live directory.** `persist()` produces a
-   crash-consistent, immutable snapshot; running `tar` against a live, actively-written directory
-   can capture a torn mid-write state. If you must archive live, stop writes first (or accept that
-   the archive is only as consistent as the moment's flush state).
-2. **Never prune or "clean up" files inside a brain directory.** Index files that look stale or
-   redundant are load-bearing; the store protects its declared index families from in-process
-   deletion, but an external `rm` bypasses that fence. If space is the concern, `du -sh` first —
-   the allocated size is usually far smaller than it looks.
-3. **Verify restores by opening them.** `Brainy.load(path)` opens any snapshot or restored
-   directory read-only — the store verifies its own coherence at open and reports loudly if
-   anything is missing or torn.

docs/guides/find-limits.md

@@ -1,153 +0,0 @@
----
-title: Query Limits & Pagination
-slug: guides/find-limits
-public: true
-category: guides
-template: guide
-order: 8
-description: How Brainy caps `find({ limit })` to prevent OOM, the three escape valves when the cap is too tight, and why pagination is the future-proof pattern.
-next:
-  - guides/aggregation
-  - api/reference
----
-
-# Query Limits & Pagination
-
-Brainy's `find()` returns entities into a JavaScript array. The size of that array is bounded by an auto-configured cap so a single query can never run the host out of memory. This guide explains the cap, the three ways to raise it when your use case justifies it, and the one pattern that scales no matter what cap is in effect: pagination.
-
-## Why the cap exists
-
-Every entity Brainy returns carries:
-
-- A 384-dim float32 embedding vector (1.5 KB)
-- Standard fields: `id`, `type`, `subtype`, timestamps, confidence, weight (~200 bytes)
-- User metadata (variable — typical 5-10 KB, can spike to 20+ KB)
-
-Conservative budget: **25 KB per result**. A `find({ limit: 100_000 })` against a brain with rich metadata can claim ~2.5 GB before Brainy's iteration starts. JavaScript's GC + V8's heap targets can't absorb that swing without paging or OOM in production.
-
-The cap is a safety net. It's not the only reason your query might be slow — graph traversal and HNSW search have their own perf characteristics — but it's the one that turns a slow query into a sudden runtime error.
-
-## The auto-configured cap (7.30.2+)
-
-Brainy picks `maxLimit` from the first of these that's available:
-
-| Priority | Source | Formula |
-|---|---|---|
-| 1 | Constructor option `maxQueryLimit` | Hard cap at supplied value, max 100 000 |
-| 2 | Constructor option `reservedQueryMemory` | `floor(reservedQueryMemory / 25 KB)` capped at 100 000 |
-| 3 | Detected container memory limit (Cloud Run, Kubernetes, cgroups v1/v2) | `floor(containerLimit × 0.25 / 25 KB)` capped at 100 000 |
-| 4 | Free system memory | `floor(availableMemory / 25 KB)` capped at 100 000 |
-
-Worked example: a 4 GB Cloud Run container picks priority 3 → `floor(4 GB × 0.25 / 25 KB) = floor(40 960) = 40 000` results. A 900 MB free-memory box on priority 4 gets `floor(900 MB / 25 KB) = ~36 000`.
-
-The cap is fixed at construction and never changes at runtime. Query timing is recorded
-for diagnostics only — a burst of slow queries cannot silently shrink the cap, and the
-auto-detected tiers (3 and 4) never go below a floor of 10 000.
-
-> **Calibration note.** Pre-7.30.2 used 100 KB per result instead of 25 KB, which produced caps that were 4× too tight for typical workloads (an 8 KB / result reality). 7.30.2 recalibrated to match observed entity sizes; existing `limit: 10_000` safety patterns now pass silently on any reasonably-sized box.
-
-## What happens when you exceed the cap
-
-`find({ limit })` enforces in **two tiers**:
-
-### Soft tier: `maxLimit < limit ≤ 2 × maxLimit`
-
-You get a one-time warning per call site:
-
-```
-[Brainy] find({ limit: 50000 }) exceeds the auto-configured query limit of
-40000 (basis: detected container memory limit). Choose one:
-  • Increase the cap: new Brainy({ maxQueryLimit: 50000 })
-  • Reserve more memory: new Brainy({ reservedQueryMemory: 1310720000 })
-  • Paginate: split the query with { limit, offset } pages
-  at YourService.loadDashboard (/app/src/dashboard.ts:142:18)
-Docs: https://soulcraft.com/docs/guides/find-limits
-```
-
-**The query proceeds.** Brainy returns the result set you asked for; the warning is a teaching signal, not a block. Existing code that relied on the cap silently allowing safety-cap limits (`limit: 10_000` against a 9 K-cap box) keeps working — the warning shows you the recipe so you can fix it intentionally.
-
-### Hard tier: `limit > 2 × maxLimit`
-
-Same message, but thrown as an error. This is real OOM territory; the cap stops being a recommendation and becomes a guardrail.
-
-## The three escape valves
-
-### 1. Raise the cap at construction — `maxQueryLimit`
-
-When the auto-config is wrong for your workload (e.g. you know your entities are smaller than 25 KB average and you need bigger result sets), set an explicit cap:
-
-```typescript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: './data' },
-  maxQueryLimit: 50_000   // raises the cap; still hard-clamped at 100 000
-})
-```
-
-This is the right answer when:
-- Your entity metadata is genuinely small (e.g. 1-2 KB) and 25 KB per result is over-conservative
-- You're running on a box with lots of headroom and 25% of memory underestimates what you can spare for queries
-- You need a known-good limit that doesn't change when the box's free-memory wiggles at startup
-
-### 2. Reserve more memory for queries — `reservedQueryMemory`
-
-When you want the cap to be memory-derived but more generous than the default 25% slice:
-
-```typescript
-const brain = new Brainy({
-  reservedQueryMemory: 1024 * 1024 * 1024  // 1 GB → ~40 000 result cap
-})
-```
-
-This is the right answer when:
-- Your host's memory budget for queries is known and stable, regardless of free-memory at startup
-- You want the formula to scale with the documented per-result size (25 KB) instead of a hard number
-
-### 3. Paginate — the future-proof pattern
-
-If your query genuinely needs to walk all matches in a category, don't fight the cap — walk in pages:
-
-```typescript
-async function findAll<T>(params: FindParams<T>, pageSize = 1000): Promise<Result<T>[]> {
-  const all: Result<T>[] = []
-  let offset = 0
-  while (true) {
-    const page = await brain.find({ ...params, limit: pageSize, offset })
-    all.push(...page)
-    if (page.length < pageSize) break
-    offset += page.length
-  }
-  return all
-}
-
-// Use it just like find():
-const allEvents = await findAll({ type: NounType.Event, where: { status: 'open' } })
-```
-
-For very large brains, prefer the streaming API which avoids holding the full result set in memory at all:
-
-```typescript
-for await (const entity of brain.streaming.entities({ type: NounType.Event })) {
-  // process one entity at a time
-}
-```
-
-## When to use which
-
-| Situation | Recommended valve |
-|---|---|
-| The cap is unreasonably low for your known entity size | `maxQueryLimit` |
-| You want a memory-derived cap but more generous than 25% | `reservedQueryMemory` |
-| Your query needs ALL matches in a category | Pagination or `brain.streaming.entities()` |
-| You hit the cap once during a one-off migration | `maxQueryLimit` or `migrateField` (which already paginates internally) |
-| You're hitting the cap on a recurring user-facing query | Pagination — the cap will get tighter in 8.0, not looser |
-
-## A note on Brainy 8.0
-
-8.0's Datomic-style `Db` API may make per-call limits stricter to keep snapshot semantics cheap. **Pagination is the only pattern that's guaranteed to keep working unchanged.** Code that paginates today doesn't need to revisit when 8.0 ships.
-
-## Reference
-
-- `BrainyConfig.maxQueryLimit?: number` — explicit cap override (max 100 000)
-- `BrainyConfig.reservedQueryMemory?: number` — memory budget for queries (bytes)
-- `find({ limit, offset })` — paginated find
-- `brain.streaming.entities(filter)` — streaming alternative for very large traversals

docs/guides/framework-integration.md

@@ -1,545 +0,0 @@
-# Framework Integration Guide
-
-Brainy is **framework-friendly** - designed to drop into the server side of any modern JavaScript framework. This guide shows you how to integrate Brainy into framework-based apps.
-
-> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun (server-side only). It is not a browser library. In framework apps, run Brainy in API routes, server components, server actions, loaders, or a dedicated backend service - never in client-side bundles.
-
-## 🎯 Why Server-Side?
-
-Brainy embeds an HNSW vector index, a graph engine, and a filesystem-backed persistence layer. These belong on the server:
-
-- **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'`
-- **Auto storage detection**: `new Brainy()` auto-selects filesystem persistence on Node
-- **Cleaner code**: No browser polyfills, no conditional client/server imports
-- **Better DX**: One instance shared across your server routes
-
-## 🚀 Quick Start
-
-### Install Brainy
-
-```bash
-npm install @soulcraft/brainy
-```
-
-### Basic Integration
-
-```javascript
-import { Brainy } from '@soulcraft/brainy'
-
-// Run on the server (API route, server component, backend service)
-// new Brainy() auto-detects filesystem persistence on Node
-const brain = new Brainy()
-await brain.init()
-
-// Add data
-await brain.add({
-  data: "Framework integration is awesome!",
-  type: "concept",
-  metadata: { framework: "any" }
-})
-
-// Search
-const results = await brain.find("framework integration")
-```
-
-## ⚛️ React Integration
-
-Brainy runs on the server, so a React client component talks to it through an API endpoint (see the Next.js API route below). The hook fetches results; it never instantiates Brainy in the browser.
-
-### Basic Hook Pattern
-
-```jsx
-import { useState, useCallback } from 'react'
-
-function useBrainySearch(endpoint = '/api/search') {
-  const [results, setResults] = useState([])
-  const [loading, setLoading] = useState(false)
-
-  const search = useCallback(async (query) => {
-    if (!query) return
-    setLoading(true)
-    try {
-      const res = await fetch(endpoint, {
-        method: 'POST',
-        headers: { 'Content-Type': 'application/json' },
-        body: JSON.stringify({ query })
-      })
-      const { results } = await res.json()
-      setResults(results)
-    } finally {
-      setLoading(false)
-    }
-  }, [endpoint])
-
-  return { results, loading, search }
-}
-
-// Usage in component
-function SearchComponent() {
-  const { results, loading, search } = useBrainySearch()
-
-  return (
-    <div>
-      <input
-        type="text"
-        placeholder="Search..."
-        onChange={(e) => search(e.target.value)}
-      />
-      {loading && <div>Searching...</div>}
-      <div>
-        {results.map(result => (
-          <div key={result.id}>
-            <h3>{result.data}</h3>
-            <p>Score: {(result.score * 100).toFixed(1)}%</p>
-          </div>
-        ))}
-      </div>
-    </div>
-  )
-}
-```
-
-### Shared Server Instance
-
-On the server, create one Brainy instance and reuse it across requests. This module is imported only by server code (API routes, server components), never by client components:
-
-```javascript
-// lib/brain.server.js
-import { Brainy } from '@soulcraft/brainy'
-
-let brainPromise
-
-export function getBrain() {
-  if (!brainPromise) {
-    brainPromise = (async () => {
-      // new Brainy() auto-detects filesystem persistence on Node
-      const brain = new Brainy()
-      await brain.init()
-      return brain
-    })()
-  }
-  return brainPromise
-}
-```
-
-## 🟢 Vue.js Integration
-
-Vue components call a server endpoint (see the Nuxt server route in the [Vue.js Integration Guide](vue-integration.md)); Brainy itself runs on the server.
-
-### Composition API (client component)
-
-```vue
-<template>
-  <div>
-    <input v-model="query" @input="search" placeholder="Search..." />
-    <div v-for="result in results" :key="result.id">
-      <h3>{{ result.data }}</h3>
-      <p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
-    </div>
-  </div>
-</template>
-
-<script setup>
-import { ref } from 'vue'
-
-const query = ref('')
-const results = ref([])
-
-const search = async () => {
-  if (!query.value) return
-  const res = await fetch('/api/search', {
-    method: 'POST',
-    headers: { 'Content-Type': 'application/json' },
-    body: JSON.stringify({ query: query.value })
-  })
-  results.value = (await res.json()).results
-}
-</script>
-```
-
-### Shared Server Instance
-
-On the server, create one Brainy instance and reuse it across requests:
-
-```javascript
-// server/brain.js (server-only module)
-import { Brainy } from '@soulcraft/brainy'
-
-let brainPromise
-
-export function getBrain() {
-  if (!brainPromise) {
-    brainPromise = (async () => {
-      // new Brainy() auto-detects filesystem persistence on Node
-      const brain = new Brainy()
-      await brain.init()
-      return brain
-    })()
-  }
-  return brainPromise
-}
-```
-
-## 🅰️ Angular Integration
-
-The Angular service calls your backend over HTTP; Brainy lives in that backend, not in the browser.
-
-### Service Pattern (calls the backend)
-
-```typescript
-// brainy.service.ts
-import { Injectable } from '@angular/core'
-import { HttpClient } from '@angular/common/http'
-import { Observable } from 'rxjs'
-
-@Injectable({
-  providedIn: 'root'
-})
-export class BrainyService {
-  constructor(private http: HttpClient) {}
-
-  search(query: string): Observable<{ results: any[] }> {
-    return this.http.post<{ results: any[] }>('/api/search', { query })
-  }
-
-  add(data: any, type: string, metadata?: any): Observable<{ id: string }> {
-    return this.http.post<{ id: string }>('/api/add', { data, type, metadata })
-  }
-}
-```
-
-```typescript
-// search.component.ts
-import { Component } from '@angular/core'
-import { BrainyService } from './brainy.service'
-
-@Component({
-  selector: 'app-search',
-  template: `
-    <div>
-      <input
-        [(ngModel)]="query"
-        (input)="search()"
-        placeholder="Search..."
-      />
-      <div *ngFor="let result of results">
-        <h3>{{ result.data }}</h3>
-        <p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
-      </div>
-    </div>
-  `
-})
-export class SearchComponent {
-  query = ''
-  results: any[] = []
-
-  constructor(private brainyService: BrainyService) {}
-
-  search() {
-    if (!this.query) return
-    this.brainyService.search(this.query).subscribe(({ results }) => {
-      this.results = results
-    })
-  }
-}
-```
-
-The matching backend endpoint uses Brainy directly (Node/Bun):
-
-```typescript
-// server: api/search
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy() // auto-detects filesystem persistence on Node
-await brain.init()
-
-export async function handleSearch(query: string) {
-  return await brain.find(query)
-}
-```
-
-## 🚀 Next.js Integration
-
-In Next.js, Brainy lives in server code only: API routes, server components, or server actions. Create one shared instance in a server-only module.
-
-### Shared Server Instance
-
-```javascript
-// lib/brain.server.js  (imported only by server code)
-import { Brainy } from '@soulcraft/brainy'
-
-let brainPromise
-
-export function getBrain() {
-  if (!brainPromise) {
-    brainPromise = (async () => {
-      const brain = new Brainy({
-        storage: { type: 'filesystem', path: './data' }
-      })
-      await brain.init()
-      return brain
-    })()
-  }
-  return brainPromise
-}
-```
-
-### API Routes
-
-```javascript
-// app/api/search/route.js
-import { getBrain } from '@/lib/brain.server'
-
-export async function POST(request) {
-  const { query } = await request.json()
-  const brain = await getBrain()
-  const results = await brain.find(query)
-
-  return Response.json({ results })
-}
-```
-
-### Server Action
-
-```javascript
-// app/actions.js
-'use server'
-import { getBrain } from '@/lib/brain.server'
-
-export async function search(query) {
-  const brain = await getBrain()
-  return await brain.find(query)
-}
-```
-
-## 🔷 SvelteKit Integration
-
-Brainy runs in a server-only module (`*.server.js`); the component fetches results from an endpoint.
-
-```javascript
-// src/lib/server/brain.js  (server-only — note the .server suffix)
-import { Brainy } from '@soulcraft/brainy'
-
-let brainPromise
-
-export function getBrain() {
-  if (!brainPromise) {
-    brainPromise = (async () => {
-      const brain = new Brainy() // auto-detects filesystem persistence
-      await brain.init()
-      return brain
-    })()
-  }
-  return brainPromise
-}
-```
-
-```javascript
-// src/routes/api/search/+server.js
-import { json } from '@sveltejs/kit'
-import { getBrain } from '$lib/server/brain'
-
-export async function POST({ request }) {
-  const { query } = await request.json()
-  const brain = await getBrain()
-  return json({ results: await brain.find(query) })
-}
-```
-
-```svelte
-<!-- SearchComponent.svelte -->
-<script>
-  let query = ''
-  let results = []
-
-  async function search() {
-    if (!query) return
-    const res = await fetch('/api/search', {
-      method: 'POST',
-      headers: { 'Content-Type': 'application/json' },
-      body: JSON.stringify({ query })
-    })
-    results = (await res.json()).results
-  }
-</script>
-
-<div>
-  <input bind:value={query} on:input={search} placeholder="Search..." />
-
-  {#each results as result}
-    <div>
-      <h3>{result.data}</h3>
-      <p>Score: {(result.score * 100).toFixed(1)}%</p>
-    </div>
-  {/each}
-</div>
-```
-
-## 🌟 Solid.js Integration
-
-The component calls a server endpoint (use SolidStart server routes, or any backend, to host Brainy):
-
-```jsx
-import { createSignal } from 'solid-js'
-
-function SearchComponent() {
-  const [query, setQuery] = createSignal('')
-  const [results, setResults] = createSignal([])
-
-  const search = async () => {
-    if (!query()) return
-    const res = await fetch('/api/search', {
-      method: 'POST',
-      headers: { 'Content-Type': 'application/json' },
-      body: JSON.stringify({ query: query() })
-    })
-    setResults((await res.json()).results)
-  }
-
-  return (
-    <div>
-      <input
-        value={query()}
-        onInput={(e) => {
-          setQuery(e.target.value)
-          search()
-        }}
-        placeholder="Search..."
-      />
-
-      <For each={results()}>
-        {(result) => (
-          <div>
-            <h3>{result.data}</h3>
-            <p>Score: {(result.score * 100).toFixed(1)}%</p>
-          </div>
-        )}
-      </For>
-    </div>
-  )
-}
-```
-
-## 📦 Bundler Configuration
-
-Brainy is a server-side dependency, so keep it out of client bundles. Import it only from server-only modules (`*.server.js`, API routes, server components, server actions). If your bundler ever tries to pull Brainy into a client bundle, that's a sign it's being imported from a client component — move the import to a server module.
-
-For server builds, mark Brainy as external so the bundler doesn't inline it:
-
-```javascript
-// vite.config.js  (SSR build)
-import { defineConfig } from 'vite'
-
-export default defineConfig({
-  ssr: {
-    external: ['@soulcraft/brainy']
-  }
-})
-```
-
-```javascript
-// rollup.config.js  (server bundle)
-export default {
-  external: ['@soulcraft/brainy', 'node:fs', 'node:path', 'node:crypto']
-}
-```
-
-## 🌐 SSR/SSG Considerations
-
-### Server-Side Rendering
-
-Instantiate Brainy on the server and feed its results into the rendered page. Never construct it in client code.
-
-```javascript
-// Server-side data loading (framework loader / getServerSideProps / load fn)
-import { getBrain } from './brain.server'
-
-export async function load({ url }) {
-  const brain = await getBrain()
-  const query = url.searchParams.get('q') ?? ''
-  const results = query ? await brain.find(query) : []
-  return { results }
-}
-```
-
-### Static Site Generation
-
-```javascript
-// For build-time usage (runs in Node during the build)
-import { Brainy } from '@soulcraft/brainy'
-
-export async function generateStaticProps() {
-  const brain = new Brainy({
-    storage: { type: 'filesystem', path: './content' }
-  })
-  await brain.init()
-
-  // Build search index (paginate with { limit, offset } for larger stores)
-  const allContent = await brain.find({ limit: 1000 })
-
-  return {
-    props: { searchIndex: allContent }
-  }
-}
-```
-
-## 🔧 Framework-Specific Tips
-
-### React
-- Keep components client-side and call a Brainy-backed API route
-- Use `useCallback` for fetch handlers to prevent re-renders
-- Debounce keystroke-driven searches before hitting the endpoint
-
-### Vue
-- Components call an endpoint; the shared instance lives in a server module
-- Consider Pinia for caching results client-side
-- Debounce reactive search queries
-
-### Angular
-- Use `HttpClient` and RxJS to call the backend
-- Hold the shared Brainy instance in your Node backend, not the app
-- Consider lazy loading search features in feature modules
-
-### Next.js
-- Put Brainy in server-only modules (`*.server.js`), API routes, or server actions
-- Reuse one shared instance across requests
-- Implement proper error boundaries for failed fetches
-
-## 🚨 Common Issues & Solutions
-
-### Issue: "fs module not found" / "crypto is not defined" in the browser
-**Cause**: Brainy was imported into a client bundle. Brainy 8.0 is a server-side library (Node 22+/Bun) and uses Node built-ins like `fs` and `crypto`.
-**Solution**: Import Brainy only from server code — server-only modules (`*.server.js`), API routes, server components, or server actions. From client components, call those endpoints instead.
-
-### Issue: Large client bundle size
-**Cause**: A client module is pulling in Brainy.
-**Solution**: Move the `import { Brainy } from '@soulcraft/brainy'` into a server-only module so it never reaches the browser bundle.
-
-### Issue: SSR hydration mismatch
-**Solution**: Run the search on the server (loader / server action / API route) and pass the results down as props, so server and client render the same markup.
-
-## 🎯 Best Practices
-
-1. **Initialize Once**: Create one shared Brainy instance per server process, not per request
-2. **Server-Only**: Import Brainy only from server modules — never from client components
-3. **Endpoint Boundary**: Expose search/add through API routes or server actions
-4. **Handle Loading**: Show loading states in the client while the fetch is in flight
-5. **Error Handling**: Catch and surface failed endpoint calls gracefully
-6. **Storage**: Use `filesystem` for persistence (the default on Node) or `memory` for ephemeral/tests
-
-## 📚 Next Steps
-
-- [Next.js Integration Guide](nextjs-integration.md) - Detailed Next.js examples
-- [Vue.js Integration Guide](vue-integration.md) - Complete Vue.js patterns
-- [API Reference](../api/README.md) - Complete API documentation
-- [Production Deployment](../deployment/CLOUD_DEPLOYMENT_GUIDE.md) - Deploy to production
-
-## 🤝 Community Examples
-
-Check out community examples in the [examples repository](https://github.com/soulcraftlabs/brainy-examples):
-
-- React + TypeScript starter
-- Vue 3 + Composition API
-- Next.js full-stack app
-- Svelte SPA with search
-- Angular enterprise app

docs/guides/import-anything.md

@@ -1,400 +0,0 @@
-# Import Anything - ONE Method, Infinite Intelligence 🚀
-
-Brainy's import is **ONE magical method** that understands EVERYTHING:
-- 📊 Data (objects, arrays, strings)
-- 📁 Files (auto-detects by path)
-- 🌐 URLs (auto-fetches with authentication support)
-- 📄 Formats (JSON, CSV, Excel, PDF, YAML, DOCX, Markdown - all auto-detected)
-
-## The Ultimate Simplicity
-
-```javascript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// ONE method for EVERYTHING:
-await brain.import(anything)
-```
-
-## Import Examples - It Just Works™
-
-### 📊 Import JSON Data
-```javascript
-// Array of objects? No problem.
-const people = [
- { name: 'Alice', role: 'Engineer', company: 'TechCorp' },
- { name: 'Bob', role: 'Designer', company: 'TechCorp' }
-]
-
-await brain.import(people)
-// ✨ Automatically detected as Person entities with Organization relationships!
-```
-
-### 📄 Import CSV - File or String
-```javascript
-// From file? Just pass the path!
-await brain.import('customers.csv')
-// ✨ Auto-detects encoding, delimiter, types - creates entities!
-
-// Or pass CSV content directly
-const csv = `name,age,city
-John,30,NYC
-Jane,25,SF`
-
-await brain.import(csv, { format: 'csv' })
-// ✨ Smart CSV parsing handles quotes, escapes, everything!
-```
-
-### 📊 Import Excel - Multi-Sheet Support
-```javascript
-// Import entire Excel workbook — every sheet is processed automatically
-await brain.import('sales-report.xlsx')
-// ✨ Processes all sheets, preserves structure, infers types!
-
-// Mirror the workbook into the VFS, grouped by sheet
-await brain.import('data.xlsx', {
- vfsPath: '/imports/data',
- groupBy: 'sheet'
-})
-// ✨ Multi-sheet data becomes interconnected entities!
-```
-
-### 📑 Import PDF - Text & Tables
-```javascript
-// Import PDF documents — text and tables are extracted automatically
-await brain.import('research-paper.pdf')
-// ✨ Extracts text, detects tables, preserves metadata!
-```
-
-### 📝 Import YAML - File or String
-```javascript
-// From file? Auto-detected!
-await brain.import('config.yaml')
-// ✨ Knows it's a file, reads it, parses YAML!
-
-// Or directly:
-const yaml = `
-project: AI Assistant
-team:
- - name: Alice
- role: Lead
- - name: Bob
- role: Dev
-`
-await brain.import(yaml, { format: 'yaml' })
-// ✨ Hierarchical data becomes a connected graph!
-```
-
-### 📄 Import Word Documents (DOCX) -
-```javascript
-// From file path
-await brain.import('research-paper.docx')
-// ✨ Extracts text, headings, tables, and metadata!
-
-// Or from buffer
-const buffer = fs.readFileSync('document.docx')
-await brain.import(buffer, { format: 'docx' })
-// ✨ Uses heading hierarchy for entity organization!
-
-// With neural extraction
-await brain.import('report.docx', {
- enableNeuralExtraction: true,
- enableHierarchicalRelationships: true
-})
-// ✨ Extracts entities from paragraphs and creates relationships within sections!
-```
-
-### 🌐 Import from URLs - Auto-Detected!
-```javascript
-// Just pass the URL - it knows!
-await brain.import('https://api.example.com/data.json')
-// ✨ Auto-detects URL, fetches, parses, processes!
-
-// Works with any URL
-await brain.import('https://data.gov/census.csv')
-// ✨ Fetches CSV from web, parses, imports!
-
-// With authentication
-await brain.import({
- type: 'url',
- data: 'https://api.example.com/private/data.xlsx',
- auth: {
- username: 'user',
- password: 'pass'
- }
-})
-// ✨ Supports basic authentication for protected resources!
-
-// With custom headers
-await brain.import({
- type: 'url',
- data: 'https://api.example.com/data.json',
- headers: {
- 'Authorization': 'Bearer TOKEN',
- 'X-API-Key': 'your-key'
- }
-})
-// ✨ Full HTTP header customization support!
-```
-
-### 📖 Import Plain Text
-```javascript
-// Even unstructured text works
-const article = `Artificial Intelligence is transforming industries.
-Machine learning enables predictive analytics.
-Natural language processing powers chatbots.`
-
-await brain.import(article, { format: 'text' })
-// ✨ Extracts concepts, creates semantic connections!
-```
-
-## The Magic Behind the Scenes
-
-When you import data, Brainy:
-
-1. **Auto-detects format** - CSV, Excel, PDF, JSON, YAML, DOCX, Markdown, or by file extension
-2. **Intelligent parsing** - CSV (encoding/delimiter), Excel (multi-sheet), PDF (text/tables), DOCX (headings/paragraphs)
-3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
-4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
-5. **Scores confidence & weight** - Every entity and relationship gets quality metrics
-6. **Creates embeddings** - Makes everything semantically searchable
-7. **Indexes metadata** - Enables lightning-fast filtering with range queries
-
-## Intelligent Type Detection
-
-Brainy automatically detects what TYPE of data you're importing:
-
-```javascript
-// This becomes a Person entity
-{ name: 'John', email: 'john@example.com' }
-
-// This becomes an Organization
-{ companyName: 'Acme', employees: 500 }
-
-// This becomes a Document
-{ title: 'Report', content: '...', author: 'Jane' }
-
-// This becomes a Location
-{ latitude: 37.7, longitude: -122.4, city: 'SF' }
-```
-
-**42 noun types and 127 verb types** cover EVERYTHING!
-
-## Relationship Detection
-
-Brainy finds connections in your data:
-
-```javascript
-const data = [
- { id: 'u1', name: 'Alice', managerId: 'u2' },
- { id: 'u2', name: 'Bob', departmentId: 'd1' },
- { id: 'd1', name: 'Engineering' }
-]
-
-await brain.import(data)
-// ✨ Automatically creates:
-// - Alice "reportsTo" Bob
-// - Bob "memberOf" Engineering
-```
-
-## Confidence & Weight Scoring -
-Every entity and relationship gets confidence and weight scores:
-
-```javascript
-// Import with confidence threshold
-await brain.import(data, {
- confidenceThreshold: 0.8 // Only extract entities with >80% confidence
-})
-
-// Query high-confidence entities using range queries
-const highConfidence = await brain.find({
- where: {
- confidence: { gte: 0.8 } // Get entities with confidence >= 0.8
- }
-})
-
-// Range query operators: gt, gte, lt, lte, between
-const mediumConfidence = await brain.find({
- where: {
- confidence: { between: [0.6, 0.8] }
- }
-})
-```
-
-**What do confidence scores mean?**
-- **High (>0.8)**: Very confident entity classification
-- **Medium (0.6-0.8)**: Reasonable confidence
-- **Low (<0.6)**: Uncertain classification (filtered by default)
-
-**Weights** indicate importance/relevance within the document context.
-
-## Per-Sheet Excel Extraction -
-Excel files with multiple sheets can be organized by sheet:
-
-```javascript
-// Group entities by sheet in VFS
-await brain.import('multi-sheet-data.xlsx', {
- groupBy: 'sheet' // Creates separate directories for each sheet
-})
-
-// Result VFS structure:
-// /imports/data/
-// ├── Sheet1/
-// │ ├── entity1.json
-// │ └── entity2.json
-// └── Sheet2/
-// ├── entity3.json
-// └── entity4.json
-
-// Other groupBy options:
-// - 'type': Group by entity type (Person, Place, etc.)
-// - 'flat': All entities in one directory
-// - 'custom': Use custom grouping function
-```
-
-## Query Your Imported Data
-
-Once imported, use Triple Intelligence to query:
-
-```javascript
-// Vector search
-const similar = await brain.find('engineers')
-
-// Natural language
-const results = await brain.find('people in engineering who joined this year')
-
-// Graph traversal + filters
-const connected = await brain.find({
- like: 'Alice',
- connected: { depth: 2 },
- where: { department: 'Engineering' }
-})
-```
-
-## Import Options (Optional!)
-
-Everything works with zero config, but you can customize:
-
-```javascript
-await brain.import(data, {
- // Format detection
- format: 'excel', // Force specific format (auto-detected if not specified)
-
- // VFS & Organization
- vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
- groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
- preserveSource: true, // Keep original source file in VFS (default: true)
-
- // Entity & Relationship Creation
- createEntities: true, // Create entities in knowledge graph (default: true)
- createRelationships: true, // Create relationships in knowledge graph (default: true)
-
- // Neural Intelligence
- enableNeuralExtraction: true, // Use AI to extract entities (default: true)
- enableRelationshipInference: true, // Use AI to infer relationships (default: true)
- enableConceptExtraction: true, // Extract concepts from text (default: true)
- confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
-
- // Deduplication
- enableDeduplication: true, // Check for duplicate entities (default: true)
- deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
- // Notes: false disables BOTH the inline merge and the background pass that
- // runs ~5 min after the last import (merged duplicates are deleted).
- // The inline pass auto-disables for imports >100 entities (O(n²) cost);
- // the background pass still covers those unless the flag is false.
-
- // Performance
- chunkSize: 100, // Batch size for processing (default: varies by operation)
-
- // History & Progress
- enableHistory: true, // Track import history (default: true)
- onProgress: (progress) => { // Progress callback
- console.log(progress.stage, progress.message)
- }
-})
-```
-
-## Error Handling
-
-Import continues even if some items fail:
-
-```javascript
-const results = await brain.import(problematicData)
-// Returns IDs of successful imports
-// Logs warnings for failures
-// Never crashes your app!
-```
-
-## Performance
-
-- **Parallel processing** - Fast imports with concurrent operations
-- **Batch operations** - Memory efficient chunk processing
-- **Lazy loading** - Import system loads only when needed
-- **Smart caching** - Type detection and format parsing results cached
-
-## Use Cases
-
-### 🏢 Business Data
-```javascript
-// Import ANY source - ONE method!
-await brain.import('customers.csv') // File
-await brain.import('https://api.co/orders') // URL
-await brain.import(productsArray) // Data
-
-// Now query across all of it!
-await brain.find('customers who bought products in Q4')
-```
-
-### 🔬 Research Data
-```javascript
-// Import research papers
-await brain.import(papers)
-
-// Import citations
-await brain.import(citations)
-
-// Find connections
-await brain.find('papers citing machine learning from 2024')
-```
-
-### 📱 Application Data
-```javascript
-// Import users
-await brain.import(users)
-
-// Import posts
-await brain.import(posts)
-
-// Import comments
-await brain.import(comments)
-
-// Query the social graph
-await brain.find('posts by users following Alice with >10 comments')
-```
-
-## The Philosophy
-
-**Zero Configuration**: Works perfectly out of the box
-**Maximum Intelligence**: AI understands your data's meaning
-**Universal Protocol**: 42 nouns × 127 verbs = ANY data model
-**Delightful DX**: Simple, clean, modern API
-
-## The ONE Method Philosophy
-
-```javascript
-// ONE method that understands EVERYTHING:
-await brain.import(data) // Objects, arrays, strings
-await brain.import('file.csv') // Files (auto-detected)
-await brain.import('http://..') // URLs (auto-fetched)
-
-// It ALWAYS knows what to do! ✨
-```
-
-**Why ONE method?**
-- 🎯 **Simpler** - No need to remember different methods
-- 🧠 **Smarter** - Auto-detects what you're importing
-- ✨ **Magical** - It just works, every time
-
-That's the power of the Universal Knowledge Protocol™ - infinite intelligence, zero complexity!

docs/guides/import-progress-examples.md

@@ -1,370 +0,0 @@
-# Import Progress - Usage Examples
-
-**How to Use Progress Tracking in Your Applications**
-
-Brainy provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
-
-> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation.
-
----
-
-## 🚀 Quick Start
-
-### Basic Progress Tracking
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-import * as fs from 'fs'
-
-const brain = await Brainy.create()
-
-// Import with progress tracking
-const result = await brain.import(fs.readFileSync('large-file.xlsx'), {
- onProgress: (progress) => {
- console.log(`Progress: ${progress.stage}`)
- console.log(` Message: ${progress.message}`)
- console.log(` Entities: ${progress.entities || 0}`)
- console.log(` Relationships: ${progress.relationships || 0}`)
- }
-})
-
-console.log(`Import complete: ${result.entities.length} entities created`)
-```
-
-**Expected Output:**
-```
-Progress: detecting
- Message: Detecting format...
- Entities: 0
- Relationships: 0
-Progress: extracting
- Message: Loading Excel workbook...
- Entities: 0
- Relationships: 0
-Progress: extracting
- Message: Reading sheet: Sales (1/3)
- Entities: 0
- Relationships: 0
-Progress: extracting
- Message: Parsing Excel (33%)
- Entities: 0
- Relationships: 0
-Progress: extracting
- Message: Reading sheet: Products (2/3)
- Entities: 0
- Relationships: 0
-... (more progress updates)
-Progress: complete
- Message: Import complete
- Entities: 1523
- Relationships: 892
-Import complete: 1523 entities created
-```
-
----
-
-## 🎯 Universal Progress Handler (Works for ALL Formats)
-
-The examples below show format-specific messages, but **you don't need format-specific code**! The `ImportProgress` interface is the same for all formats:
-
-```typescript
-// ONE HANDLER FOR ALL FORMATS!
-function universalProgressHandler(progress) {
- console.log(`[${progress.stage}] ${progress.message}`)
-
- if (progress.processed && progress.total) {
- console.log(` Progress: ${progress.processed}/${progress.total}`)
- }
-
- if (progress.entities || progress.relationships) {
- console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
- }
-
- if (progress.throughput && progress.eta) {
- console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
- }
-}
-
-// Use it for ANY format!
-await brain.import(csvBuffer, { onProgress: universalProgressHandler })
-await brain.import(pdfBuffer, { onProgress: universalProgressHandler })
-await brain.import(excelBuffer, { onProgress: universalProgressHandler })
-await brain.import(jsonBuffer, { onProgress: universalProgressHandler })
-await brain.import(markdownString, { onProgress: universalProgressHandler })
-await brain.import(yamlBuffer, { onProgress: universalProgressHandler })
-await brain.import(docxBuffer, { onProgress: universalProgressHandler })
-```
-
----
-
-## 📊 What Different Formats Look Like (Same Handler!)
-
-The examples below show **what messages look like** for different formats using the **same universal handler** above.
-
-### CSV Import (Row-by-Row Progress)
-
-```typescript
-await brain.import(csvBuffer, {
- format: 'csv',
- onProgress: (progress) => {
- if (progress.stage === 'extracting') {
- // CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
- console.log(progress.message)
- }
- }
-})
-```
-
-**CSV Progress Messages:**
-- ✅ "Detecting CSV encoding and delimiter..."
-- ✅ "Parsing CSV rows (delimiter: ",")"
-- ✅ "Parsed 75%" (via bytes processed)
-- ✅ "Extracted 1000 rows"
-- ✅ "Converting types: 5000/10000 rows..."
-- ✅ "CSV processing complete: 10000 rows"
-
----
-
-### PDF Import (Page-by-Page Progress)
-
-```typescript
-await brain.import(pdfBuffer, {
- format: 'pdf',
- onProgress: (progress) => {
- // PDF reports exact page numbers
- console.log(progress.message)
- // Example: "Processing page 5 of 23"
- }
-})
-```
-
-**PDF Progress Messages:**
-- ✅ "Loading PDF document..."
-- ✅ "Processing 23 pages..."
-- ✅ "Processing page 5 of 23"
-- ✅ "Parsed 22%" (via bytes processed)
-- ✅ "Extracted 156 items from PDF"
-- ✅ "PDF complete: 23 pages, 156 items extracted"
-
----
-
-### Excel Import (Sheet-by-Sheet Progress)
-
-```typescript
-await brain.import(excelBuffer, {
- format: 'excel',
- onProgress: (progress) => {
- // Excel reports sheet names
- console.log(progress.message)
- // Example: "Reading sheet: Q2 Sales (2/5)"
- }
-})
-```
-
-**Excel Progress Messages:**
-- ✅ "Loading Excel workbook..."
-- ✅ "Processing 3 sheets..."
-- ✅ "Reading sheet: Sales (1/3)"
-- ✅ "Parsing Excel (33%)" (via bytes processed)
-- ✅ "Extracted 5234 rows from Excel"
-- ✅ "Excel complete: 3 sheets, 5234 rows"
-
----
-
-### JSON Import (Node Traversal)
-
-```typescript
-await brain.import(jsonBuffer, {
- format: 'json',
- onProgress: (progress) => {
- // JSON reports every 10 nodes
- console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
- }
-})
-```
-
----
-
-### Markdown Import (Section-by-Section)
-
-```typescript
-await brain.import(markdownString, {
- format: 'markdown',
- onProgress: (progress) => {
- console.log(`Section ${progress.processed}/${progress.total}`)
- }
-})
-```
-
----
-
-## 🎯 Building Progress UI Components
-
-### React Progress Bar
-
-```typescript
-function ImportProgress({ file }: { file: File }) {
- const [progress, setProgress] = useState({
- stage: 'idle',
- message: '',
- percent: 0,
- entities: 0,
- relationships: 0
- })
-
- const handleImport = async () => {
- const buffer = await file.arrayBuffer()
-
- await brain.import(Buffer.from(buffer), {
- onProgress: (p) => {
- setProgress({
- stage: p.stage,
- message: p.message,
- // Estimate percentage from stage
- percent: {
- detecting: 10,
- extracting: 50,
- 'storing-vfs': 80,
- 'storing-graph': 90,
- complete: 100
- }[p.stage] || 0,
- entities: p.entities || 0,
- relationships: p.relationships || 0
- })
- }
- })
- }
-
- return (
- <div>
- <ProgressBar value={progress.percent} />
- <p>{progress.message}</p>
- <p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
- </div>
- )
-}
-```
-
----
-
-### CLI Progress Spinner
-
-```typescript
-import ora from 'ora'
-
-const spinner = ora('Starting import...').start()
-
-await brain.import(buffer, {
- onProgress: (progress) => {
- spinner.text = progress.message
-
- if (progress.stage === 'complete') {
- spinner.succeed(`Import complete: ${progress.entities} entities`)
- }
- }
-})
-```
-
-**CLI Output:**
-```
-⠋ Detecting format...
-⠙ Loading Excel workbook...
-⠹ Reading sheet: Sales (1/3)
-⠸ Parsing Excel (33%)
-⠼ Reading sheet: Products (2/3)
-...
-✔ Import complete: 1523 entities
-```
-
----
-
-### Progress Dashboard with ETA
-
-```typescript
-let startTime = Date.now()
-let lastUpdate = startTime
-
-await brain.import(buffer, {
- onProgress: (progress) => {
- const elapsed = Date.now() - startTime
- const rate = progress.entities / (elapsed / 1000) // entities/sec
-
- console.clear()
- console.log('Import Progress Dashboard')
- console.log('========================')
- console.log(`Stage: ${progress.stage}`)
- console.log(`Status: ${progress.message}`)
- console.log(`Entities: ${progress.entities}`)
- console.log(`Relationships: ${progress.relationships}`)
- console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
- console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
- }
-})
-```
-
----
-
-## 🔧 Advanced: Format-Specific Optimization
-
-### Detecting Format to Show Appropriate Progress
-
-```typescript
-const formatMessages = {
- csv: (p) => `CSV: ${p.message}`,
- pdf: (p) => `PDF: ${p.message}`,
- excel: (p) => `Excel: ${p.message}`,
- json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
- markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
- yaml: (p) => `YAML: ${p.processed} nodes`,
- docx: (p) => `DOCX: ${p.processed} paragraphs`
-}
-
-await brain.import(buffer, {
- onProgress: (progress) => {
- // Format is available in progress.stage metadata
- const message = formatMessages[detectedFormat]?.(progress) || progress.message
- console.log(message)
- }
-})
-```
-
----
-
-## ⚡ Performance Tips
-
-### Throttle UI Updates
-
-```typescript
-let lastUIUpdate = 0
-const THROTTLE_MS = 100 // Update UI max once per 100ms
-
-await brain.import(buffer, {
- onProgress: (progress) => {
- const now = Date.now()
- if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
- return // Skip this update
- }
-
- lastUIUpdate = now
- updateUI(progress) // Only update every 100ms
- }
-})
-```
-
-**Note:** Brainy already throttles progress callbacks internally, but additional UI throttling can help with heavy rendering.
-
----
-
-## 📝 Summary
-
-✅ **All 7 formats** have consistent progress reporting
-✅ **Real-time updates** during long imports (no more "0%" hangs)
-✅ **Contextual messages** show exactly what's happening
-✅ **Build reliable tools** with standardized progress callbacks
-✅ **Problem SOLVED** - users see progress throughout import
-
-**Files Modified:**
-- 3 handlers: `csvHandler.ts`, `pdfHandler.ts`, `excelHandler.ts`
-- 7 importers: `SmartCSVImporter.ts`, `SmartPDFImporter.ts`, `SmartExcelImporter.ts`, `SmartJSONImporter.ts`, `SmartMarkdownImporter.ts`, `SmartYAMLImporter.ts`, `SmartDOCXImporter.ts`
-
-**Result:** Comprehensive, consistent progress tracking across ALL import formats!

docs/guides/import-progress-implementation.md

@@ -1,734 +0,0 @@
-# Import Progress Implementation Guide
-**For Developers: How to Add Progress Tracking to ANY File Handler**
-
-> This guide shows the **standard pattern** for implementing rich progress tracking in Brainy import handlers. Follow this template for **all 7 supported formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX) or any future file format.
-
----
-
-## 📊 Supported Formats & Consistent Progress Reporting
-
-> **⚠️ IMPORTANT FOR DEVELOPERS:** The public API (`ImportProgress`) is 100% standardized across all formats. You can build ONE progress handler that works for CSV, PDF, Excel, JSON, Markdown, YAML, and DOCX with **zero format-specific code**. See [Standard Import Progress API](./standard-import-progress.md) for details.
-
-**ALL 7 formats now have consistent, standardized progress reporting** for building reliable import tools:
-
-| Format | Category | Progress Points | File Location | Status |
-|--------|----------|-----------------|---------------|--------|
-| **CSV** | Tabular | Parsing → Row extraction → Type conversion → Complete | `handlers/csvHandler.ts` + `SmartCSVImporter.ts` | ✅ Complete |
-| **PDF** | Document | Loading → Page-by-page → Item extraction → Complete | `handlers/pdfHandler.ts` + `SmartPDFImporter.ts` | ✅ Complete |
-| **Excel** | Tabular | Loading → Sheet-by-sheet → Row extraction → Type conversion → Complete | `handlers/excelHandler.ts` + `SmartExcelImporter.ts` | ✅ Complete |
-| **JSON** | Structured | Parsing → Node traversal (every 10 nodes) → Complete | `SmartJSONImporter.ts` | ✅ Complete |
-| **Markdown** | Document | Parsing → Section-by-section → Complete | `SmartMarkdownImporter.ts` | ✅ Complete |
-| **YAML** | Structured | Parsing → Node traversal (every 10 nodes) → Complete | `SmartYAMLImporter.ts` | ✅ Complete |
-| **DOCX** | Document | Parsing → Paragraph-by-paragraph (every 10) → Complete | `SmartDOCXImporter.ts` | ✅ Complete |
-
-### The Standard Public API
-
-**Developers calling `brain.import()` see ONE standardized interface** regardless of format:
-
-```typescript
-// THE PUBLIC API - Same for ALL 7 formats!
-brain.import(buffer, {
- onProgress: (progress: ImportProgress) => {
- // These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
- progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
- progress.message // Human-readable status (varies by format, always readable)
- progress.processed // Items processed (optional)
- progress.total // Total items (optional)
- progress.entities // Entities extracted (optional)
- progress.relationships // Relationships inferred (optional)
- progress.throughput // Items/sec (optional, during extraction)
- progress.eta // Time remaining in ms (optional)
- }
-})
-```
-
-**Internal Implementation** (for developers adding new format handlers):
-
-The table below shows how formats implement progress *internally*. Normal developers don't need to know this - they just use the standard `ImportProgress` interface above!
-
-```typescript
-// Internal: Binary formats use handler hooks (you added these!)
-interface FormatHandlerProgressHooks {
- onBytesProcessed?: (bytes: number) => void
- onCurrentItem?: (message: string) => void
- onDataExtracted?: (count: number, total?: number) => void
-}
-
-// Internal: Text formats use importer callbacks
-interface ImporterProgressCallback {
- onProgress?: (stats: { processed, total, entities, relationships }) => void
-}
-
-// Both are converted to ImportProgress by ImportCoordinator!
-```
-
-### Developer Benefits
-
-✅ **Consistent API** - Same pattern across all 7 formats
-✅ **Throttled Updates** - Progress reported every 10-1000 items (no spam)
-✅ **Contextual Messages** - "Processing page 5 of 23", "Reading sheet: Sales (2/5)"
-✅ **Real-time Estimates** - Users see progress during long imports
-✅ **Build Monitoring Tools** - Reliable progress data for UIs, dashboards, CLI tools
-
----
-
-## 🎯 Overview
-
-Brainy supports comprehensive, multi-dimensional progress tracking for imports:
-- **Bytes processed** (always available, most deterministic)
-- **Entities extracted** (AI extraction phase)
-- **Stage-specific metrics** (parsing: MB/s, extraction: entities/s)
-- **Time estimates** (remaining time, total time)
-- **Context information** ("Processing page 5 of 23")
-
-All handlers follow a simple, consistent pattern using **progress hooks**.
-
----
-
-## 📋 The Progress Hooks Pattern
-
-### 1. Progress Hooks Interface
-
-```typescript
-export interface FormatHandlerProgressHooks {
- /**
- * Report bytes processed
- * Call this as you read/parse the file
- */
- onBytesProcessed?: (bytes: number) => void
-
- /**
- * Set current processing context
- * Examples: "Processing page 5", "Reading sheet: Q2 Sales"
- */
- onCurrentItem?: (item: string) => void
-
- /**
- * Report structured data extraction progress
- * Examples: "Extracted 100 rows", "Parsed 50 paragraphs"
- */
- onDataExtracted?: (count: number, total?: number) => void
-}
-```
-
-### 2. Handler Options (Automatic)
-
-Progress hooks are automatically passed to your handler via `FormatHandlerOptions`:
-
-```typescript
-export interface FormatHandlerOptions {
- // ... existing options ...
-
- /**
- * Progress hooks
- * Handlers call these to report progress during processing
- */
- progressHooks?: FormatHandlerProgressHooks
-
- /**
- * Total file size in bytes
- * Used for progress percentage calculation
- */
- totalBytes?: number
-}
-```
-
-**You don't need to modify FormatHandlerOptions** - it's already done!
-
-### 3. Standard Implementation Pattern
-
-Every handler follows these 5 steps:
-
-```typescript
-async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
- const progressHooks = options.progressHooks // Step 1: Get hooks
-
- // Step 2: Report initial progress
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem('Starting import...')
- }
-
- // Step 3: Report bytes as you process
- const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(0) // Start
- }
-
- // ... do parsing ...
-
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(buffer.length) // Complete
- }
-
- // Step 4: Report data extraction
- if (progressHooks?.onDataExtracted) {
- progressHooks.onDataExtracted(data.length, data.length)
- }
-
- // Step 5: Report completion
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Complete: ${data.length} items processed`)
- }
-
- return { format, data, metadata }
-}
-```
-
----
-
-## 📚 Complete Example: CSV Handler
-
-Here's the **ACTUAL implementation** from CSV handler showing all the key progress points:
-
-```typescript
-async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
- const startTime = Date.now()
- const progressHooks = options.progressHooks // ✅ Step 1
-
- // Convert to buffer if string
- const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
- const totalBytes = buffer.length
-
- // ✅ Step 2: Report start
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(0)
- }
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...')
- }
-
- // Detect encoding
- const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
- const text = buffer.toString(detectedEncoding as BufferEncoding)
-
- // Detect delimiter
- const delimiter = options.csvDelimiter || this.detectDelimiter(text)
-
- // ✅ Progress update: Parsing phase
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
- }
-
- // Parse CSV
- const records = parse(text, { /* options */ })
-
- // ✅ Step 3: Report bytes processed (entire file parsed)
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(totalBytes)
- }
-
- const data = Array.isArray(records) ? records : [records]
-
- // ✅ Step 4: Report data extraction
- if (progressHooks?.onDataExtracted) {
- progressHooks.onDataExtracted(data.length, data.length)
- }
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`)
- }
-
- // Type inference and conversion
- const fields = data.length > 0 ? Object.keys(data[0]) : []
- const types = this.inferFieldTypes(data)
-
- const convertedData = data.map((row, index) => {
- const converted = this.convertRow(row, types)
-
- // ✅ Progress update every 1000 rows (avoid spam)
- if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
- progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
- }
-
- return converted
- })
-
- // ✅ Step 5: Report completion
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
- }
-
- return {
- format: this.format,
- data: convertedData,
- metadata: { /* ... */ }
- }
-}
-```
-
-### Key Progress Points in CSV Handler
-
-| Progress Point | Hook Used | Message Example |
-|----------------|-----------|-----------------|
-| **Start** | `onCurrentItem` | "Detecting CSV encoding and delimiter..." |
-| **Start bytes** | `onBytesProcessed(0)` | 0 bytes |
-| **Parsing** | `onCurrentItem` | "Parsing CSV rows (delimiter: \",\")..." |
-| **Bytes complete** | `onBytesProcessed(totalBytes)` | All bytes read |
-| **Data extracted** | `onDataExtracted(count, total)` | Number of rows extracted |
-| **Type conversion** | `onCurrentItem` (every 1000 rows) | "Converting types: 5000/10000 rows..." |
-| **Complete** | `onCurrentItem` | "CSV processing complete: 10000 rows" |
-
----
-
-## 📖 Implementation Guide by File Type
-
-### Supported Formats
-
-Brainy supports **7 file formats** with full progress tracking:
-
-**Binary Formats** (use handlers):
-1. **CSV** - Row-by-row parsing with type inference
-2. **PDF** - Page-by-page extraction with table detection
-3. **Excel** - Sheet-by-sheet processing with formula evaluation
-
-**Text/Structured Formats** (parse inline):
-4. **JSON** - Recursive traversal of nested structures
-5. **Markdown** - Section-by-section with heading extraction
-6. **YAML** - Hierarchical traversal with relationship inference
-7. **DOCX** - Paragraph-by-paragraph with structure analysis
-
----
-
-### PDF Handler (Multi-Page)
-
-```typescript
-async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
- const progressHooks = options.progressHooks
- const totalBytes = data.length
-
- // Report start
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem('Loading PDF document...')
- }
-
- const pdfDoc = await loadPDF(data)
- const totalPages = pdfDoc.numPages
-
- const extractedData: any[] = []
- let bytesProcessed = 0
-
- for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
- // ✅ Report current page
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`)
- }
-
- const page = await pdfDoc.getPage(pageNum)
- const text = await page.getTextContent()
- extractedData.push(this.processPageText(text))
-
- // ✅ Estimate bytes processed (pages are sequential)
- bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes)
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(bytesProcessed)
- }
-
- // ✅ Report extraction progress
- if (progressHooks?.onDataExtracted) {
- progressHooks.onDataExtracted(pageNum, totalPages)
- }
- }
-
- // Final progress
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(totalBytes)
- }
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`)
- }
-
- return { format: 'pdf', data: extractedData, metadata: { /* ... */ } }
-}
-```
-
-### Excel Handler (Multi-Sheet)
-
-```typescript
-async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
- const progressHooks = options.progressHooks
- const totalBytes = data.length
-
- // Load workbook
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem('Loading Excel workbook...')
- }
-
- const workbook = XLSX.read(data)
- const sheetNames = options.excelSheets === 'all'
- ? workbook.SheetNames
- : (options.excelSheets || [workbook.SheetNames[0]])
-
- const allData: any[] = []
- let bytesProcessed = 0
-
- for (let i = 0; i < sheetNames.length; i++) {
- const sheetName = sheetNames[i]
-
- // ✅ Report current sheet
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`)
- }
-
- const sheet = workbook.Sheets[sheetName]
- const sheetData = XLSX.utils.sheet_to_json(sheet)
- allData.push(...sheetData)
-
- // ✅ Estimate bytes processed (sheets processed sequentially)
- bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes)
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(bytesProcessed)
- }
-
- // ✅ Report data extraction
- if (progressHooks?.onDataExtracted) {
- progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done
- }
- }
-
- // Final progress
- if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(totalBytes)
- }
- if (progressHooks?.onCurrentItem) {
- progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`)
- }
-
- return { format: 'xlsx', data: allData, metadata: { /* ... */ } }
-}
-```
-
-### JSON Importer (Recursive Traversal)
-
-```typescript
-async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResult> {
- // ✅ Report parsing start
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Parse JSON if string
- let jsonData = typeof data === 'string' ? JSON.parse(data) : data
-
- // ✅ Report parsing complete
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Traverse and extract (reports progress every 10 nodes)
- const entities: ExtractedJSONEntity[] = []
- const relationships: ExtractedJSONRelationship[] = []
- let nodesProcessed = 0
-
- await this.traverseJSON(
- jsonData,
- entities,
- relationships,
- () => {
- nodesProcessed++
- if (nodesProcessed % 10 === 0) {
- options.onProgress?.({
- processed: nodesProcessed,
- entities: entities.length,
- relationships: relationships.length
- })
- }
- }
- )
-
- // ✅ Report completion
- options.onProgress?.({
- processed: nodesProcessed,
- entities: entities.length,
- relationships: relationships.length
- })
-
- return { nodesProcessed, entitiesExtracted: entities.length, ... }
-}
-```
-
-### Markdown Importer (Section-Based)
-
-```typescript
-async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<SmartMarkdownResult> {
- // ✅ Report parsing start
- options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 })
-
- // Parse markdown into sections
- const parsedSections = this.parseMarkdown(markdown, options)
-
- // ✅ Report parsing complete
- options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 })
-
- // Process each section (reports progress after each section)
- const sections: MarkdownSection[] = []
- for (let i = 0; i < parsedSections.length; i++) {
- const section = await this.processSection(parsedSections[i], options)
- sections.push(section)
-
- options.onProgress?.({
- processed: i + 1,
- total: parsedSections.length,
- entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
- relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
- })
- }
-
- // ✅ Report completion
- options.onProgress?.({
- processed: sections.length,
- total: sections.length,
- entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
- relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
- })
-
- return { sectionsProcessed: sections.length, ... }
-}
-```
-
-### YAML Importer (Hierarchical)
-
-```typescript
-async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Promise<SmartYAMLResult> {
- // ✅ Report parsing start
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Parse YAML
- const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8')
- const data = yaml.load(yamlString)
-
- // ✅ Report parsing complete
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Traverse YAML structure (reports progress every 10 nodes)
- // ... similar to JSON traversal ...
-
- // ✅ Report completion (already implemented)
- options.onProgress?.({
- processed: nodesProcessed,
- entities: entities.length,
- relationships: relationships.length
- })
-
- return { nodesProcessed, entitiesExtracted: entities.length, ... }
-}
-```
-
-### DOCX Importer (Paragraph-Based)
-
-```typescript
-async extract(buffer: Buffer, options: SmartDOCXOptions = {}): Promise<SmartDOCXResult> {
- // ✅ Report parsing start
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Extract text and HTML using Mammoth
- const textResult = await mammoth.extractRawText({ buffer })
- const htmlResult = await mammoth.convertToHtml({ buffer })
-
- // ✅ Report parsing complete
- options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
-
- // Process paragraphs (reports progress every 10 paragraphs)
- const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength)
-
- for (let i = 0; i < paragraphs.length; i++) {
- await this.processParagraph(paragraphs[i])
-
- if (i % 10 === 0) {
- options.onProgress?.({
- processed: i + 1,
- entities: entities.length,
- relationships: relationships.length
- })
- }
- }
-
- // ✅ Report completion (already implemented)
- options.onProgress?.({
- processed: paragraphs.length,
- entities: entities.length,
- relationships: relationships.length
- })
-
- return { paragraphsProcessed: paragraphs.length, ... }
-}
-```
-
----
-
-## 🎯 Best Practices
-
-### 1. Always Check if Hooks Exist
-
-Progress hooks are **optional**. Always check before calling:
-
-```typescript
-// ✅ Good - safe
-if (progressHooks?.onBytesProcessed) {
- progressHooks.onBytesProcessed(bytes)
-}
-
-// ❌ Bad - will crash if hooks undefined
-progressHooks.onBytesProcessed(bytes) // TypeError!
-```
-
-### 2. Report Bytes at Start and End
-
-```typescript
-// ✅ Good - clear start and end
-progressHooks?.onBytesProcessed(0) // Start
-// ... processing ...
-progressHooks?.onBytesProcessed(totalBytes) // End
-
-// ❌ Bad - no clear boundaries
-// ... just start processing without reporting start
-```
-
-### 3. Throttle Frequent Updates
-
-```typescript
-// ✅ Good - report every 1000 items
-for (let i = 0; i < items.length; i++) {
- processItem(items[i])
-
- if (i > 0 && i % 1000 === 0) {
- progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`)
- }
-}
-
-// ❌ Bad - report EVERY item (spam!)
-for (let i = 0; i < items.length; i++) {
- processItem(items[i])
- progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks!
-}
-```
-
-### 4. Provide Contextual Messages
-
-```typescript
-// ✅ Good - specific and helpful
-progressHooks?.onCurrentItem('Parsing CSV rows (delimiter: ",")')
-progressHooks?.onCurrentItem('Processing page 5 of 23')
-progressHooks?.onCurrentItem('Reading sheet: Q2 Sales Data')
-
-// ❌ Bad - vague
-progressHooks?.onCurrentItem('Processing...')
-progressHooks?.onCurrentItem('Working...')
-```
-
-### 5. Report Data Extraction with Totals (if known)
-
-```typescript
-// ✅ Good - total known
-progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows
-
-// ✅ Also good - total unknown (streaming)
-progressHooks?.onDataExtracted(100, undefined) // 100 rows so far
-
-// ✅ Also good - complete
-progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows
-```
-
----
-
-## 🔧 Testing Your Handler
-
-### Manual Test
-
-```typescript
-import { CSVHandler } from './csvHandler.js'
-import * as fs from 'fs'
-
-const handler = new CSVHandler()
-const data = fs.readFileSync('./test.csv')
-
-const result = await handler.process(data, {
- filename: 'test.csv',
- progressHooks: {
- onBytesProcessed: (bytes) => {
- console.log(`Bytes: ${bytes}`)
- },
- onCurrentItem: (item) => {
- console.log(`Status: ${item}`)
- },
- onDataExtracted: (count, total) => {
- console.log(`Extracted: ${count}${total ? `/${total}` : ''}`)
- }
- }
-})
-
-console.log(`Complete: ${result.data.length} rows`)
-```
-
-### Expected Output
-
-```
-Status: Detecting CSV encoding and delimiter...
-Bytes: 0
-Status: Parsing CSV rows (delimiter: ",")...
-Bytes: 52438
-Extracted: 1000/1000
-Status: Extracted 1000 rows, inferring types...
-Status: CSV processing complete: 1000 rows
-Complete: 1000 rows
-```
-
----
-
-## 📊 Progress Flow Diagram
-
-```
-User Imports File
- ↓
-ImportManager
- ↓
-Creates ProgressTracker
- ↓
-Calls Handler.process() with progressHooks
- ↓
-Handler Reports Progress:
- ├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated
- ├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated
- ├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated
- ├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated
- └─ onCurrentItem("Complete") → ProgressTracker → final progress
- ↓
-ProgressTracker emits to callback (throttled 100ms)
- ↓
-User sees:
- "Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..."
-```
-
----
-
-## ✅ Checklist for New Handlers
-
-When implementing a new file format handler:
-
-- [ ] Get `progressHooks` from `options`
-- [ ] Get `totalBytes` (if available)
-- [ ] Report `onBytesProcessed(0)` at start
-- [ ] Report `onCurrentItem()` for key stages
-- [ ] Report `onBytesProcessed()` as you process
-- [ ] Report `onDataExtracted()` when you extract data
-- [ ] Throttle frequent updates (every 1000 items max)
-- [ ] Report `onBytesProcessed(totalBytes)` at end
-- [ ] Report final `onCurrentItem()` with summary
-- [ ] Test with progress callback to verify output
-
----
-
-## 🎓 Summary
-
-**The Pattern (5 Steps)**:
-1. Get `progressHooks` from options
-2. Report start (`onBytesProcessed(0)`, `onCurrentItem("Starting...")`)
-3. Report progress as you process (`onBytesProcessed(bytes)`, `onCurrentItem("Page 5...")`)
-4. Report data extraction (`onDataExtracted(count, total)`)
-5. Report completion (`onBytesProcessed(totalBytes)`, `onCurrentItem("Complete")`)
-
-**Always Check**: `progressHooks?.method()`
-
-**Throttle**: Report every N items, not every single item
-
-**Context**: Provide specific, helpful messages
-
-**Testing**: Use manual test with console.log callbacks
-
----
-
-**This pattern makes it trivial to add progress tracking to ANY file format. Copy this template and adapt for your handler!**

docs/guides/import-quick-reference.md

@@ -1,461 +0,0 @@
-# 📥 Import Quick Reference
-
-> **Quick guide to importing data into Brainy**
-
----
-
-## Basic Import
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// Import from file path
-await brain.import('/path/to/data.xlsx')
-
-// Import from buffer
-const buffer = fs.readFileSync('data.csv')
-await brain.import(buffer)
-
-// Import from object
-const jsonData = { items: [...] }
-await brain.import(jsonData)
-```
-
----
-
-## Supported Formats
-
-| Format | Extensions | Auto-Detect |
-|--------|------------|-------------|
-| **Excel** | `.xlsx`, `.xls` | ✅ Yes |
-| **CSV** | `.csv` | ✅ Yes |
-| **JSON** | `.json` | ✅ Yes |
-| **Markdown** | `.md` | ✅ Yes |
-| **PDF** | `.pdf` | ✅ Yes |
-| **YAML** | `.yaml`, `.yml` | ✅ Yes |
-| **DOCX** | `.docx` | ✅ Yes |
-
----
-
-## Common Options
-
-### Basic Options
-
-```typescript
-await brain.import(file, {
-  // Specify format (optional - auto-detects by default)
-  format: 'excel',
-
-  // VFS destination path
-  vfsPath: '/imports/products',
-
-  // Enable/disable features
-  createEntities: true,           // Create graph entities (default: true)
-  createRelationships: true,      // Create relationships (default: true)
-  preserveSource: true,           // Keep original file (default: true)
-
-  // Progress tracking
-  onProgress: (progress) => {
-    console.log(`${progress.processed}/${progress.total}`)
-  }
-})
-```
-
-### Neural Intelligence
-
-```typescript
-await brain.import(file, {
-  // Entity type classification
-  enableNeuralExtraction: true,      // Auto-classify entity types (default: true)
-
-  // Relationship type inference
-  enableRelationshipInference: true, // Auto-infer relationship types (default: true)
-
-  // Concept extraction
-  enableConceptExtraction: true,     // Extract key concepts (default: true)
-
-  // Confidence threshold
-  confidenceThreshold: 0.6           // Min confidence for extraction (default: 0.6)
-})
-```
-
-### Deduplication
-
-```typescript
-await brain.import(file, {
-  enableDeduplication: true,         // Check for duplicates (default: true)
-  deduplicationThreshold: 0.85       // Similarity threshold (default: 0.85)
-})
-```
-
-Deduplication merges entities judged duplicates — the non-primary records are
-**deleted**. Set `enableDeduplication: false` to disable it entirely: the flag
-gates both the inline merge during import and the background pass that runs
-about 5 minutes after the last import.
-
-```typescript
-await brain.import(file, {
-  enableDeduplication: false         // No merging, inline or background
-})
-```
-
-### Import Tracking
-
-Track and organize imports by project:
-
-```typescript
-await brain.import(file, {
-  projectId: 'worldbuilding',        // Group related imports
-  importId: 'import-001',            // Custom ID (auto-generated if not provided)
-  customMetadata: {                  // Additional metadata
-    campaign: 'fall-2024',
-    author: 'gamemaster'
-  }
-})
-
-// Query all entities in a project
-const entities = await brain.find({
-  where: { projectId: 'worldbuilding' }
-})
-
-// Query entities from specific import
-const importedEntities = await brain.find({
-  where: { importIds: { $includes: 'import-001' } }
-})
-
-// Exclude a project from search
-const results = await brain.find({
-  query: 'dragon',
-  where: { projectId: { $ne: 'archived-project' } }
-})
-```
-
-**All created items (entities, relationships, VFS files) are automatically tagged with:**
-- `importIds: string[]` - Import operation IDs
-- `projectId: string` - Project identifier
-- `importedAt: number` - Timestamp
-- `importFormat: string` - Format type ('excel', 'csv', etc.)
-- `importSource: string` - Source filename/URL
-
-### VFS Organization
-
-```typescript
-await brain.import(file, {
-  vfsPath: '/imports/catalog',
-
-  // Grouping strategy
-  groupBy: 'type',  // Group by entity type (default)
-  // OR
-  groupBy: 'sheet', // Group by Excel sheet name
-  // OR
-  groupBy: 'flat',  // All entities in root directory
-  // OR
-  groupBy: 'custom',
-  customGrouping: (entity) => {
-    return `/by-category/${entity.category}`
-  }
-})
-```
-
-### Always-On Streaming
-
-All imports use streaming with adaptive flush intervals. Query data as it's imported:
-
-```typescript
-await brain.import(file, {
-  onProgress: async (progress) => {
-    // Query data during import
-    if (progress.queryable) {
-      const products = await brain.find({ type: 'product', limit: 10000 })
-      console.log(`${products.length} products imported so far`)
-    }
-  }
-})
-```
-
-**Progressive intervals** (automatic):
-- 0-999 entities: Flush every 100 (frequent early updates)
-- 1K-9.9K: Flush every 1000 (balanced)
-- 10K+: Flush every 5000 (minimal overhead)
-- Adjusts dynamically as import grows
-
----
-
-## Complete Example
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-import * as fs from 'fs'
-
-async function importCatalog() {
-  const brain = new Brainy({
-    storage: {
-      type: 'gcs',
-      bucket: 'my-bucket',
-      prefix: 'brainy/'
-    }
-  })
-  await brain.init()
-
-  const buffer = fs.readFileSync('catalog.xlsx')
-
-  const result = await brain.import(buffer, {
-    format: 'excel',
-    vfsPath: '/imports/product-catalog',
-    groupBy: 'type',
-
-    // Neural intelligence
-    enableNeuralExtraction: true,
-    enableRelationshipInference: true,
-    confidenceThreshold: 0.7,
-
-    // Deduplication
-    enableDeduplication: true,
-    deduplicationThreshold: 0.85,
-
-    // Progress tracking (streaming always enabled)
-    onProgress: async (progress) => {
-      console.log(`Stage: ${progress.stage}`)
-      console.log(`Progress: ${progress.processed}/${progress.total}`)
-
-      // Query live data (available after each flush)
-      if (progress.queryable) {
-        const products = await brain.find({ type: 'product', limit: 100000 })
-        const people = await brain.find({ type: 'person', limit: 100000 })
-        const all = await brain.find({ limit: 100000 })
-
-        const stats = {
-          products: products.length,
-          people: people.length,
-          total: all.length
-        }
-        console.log('Current counts:', stats)
-      }
-    }
-  })
-
-  console.log('Import complete!')
-  console.log(`Entities: ${result.entities.length}`)
-  console.log(`Relationships: ${result.relationships.length}`)
-  console.log(`VFS path: ${result.vfs.rootPath}`)
-  console.log(`Processing time: ${result.stats.processingTime}ms`)
-
-  return result
-}
-
-importCatalog()
-```
-
----
-
-## Import Result
-
-```typescript
-interface ImportResult {
-  importId: string
-  format: string
-  formatConfidence: number
-
-  vfs: {
-    rootPath: string
-    directories: string[]
-    files: Array<{
-      path: string
-      entityId?: string
-      type: 'entity' | 'metadata' | 'source' | 'relationships'
-    }>
-  }
-
-  entities: Array<{
-    id: string
-    name: string
-    type: NounType
-    vfsPath?: string
-  }>
-
-  relationships: Array<{
-    id: string
-    from: string
-    to: string
-    type: VerbType
-  }>
-
-  stats: {
-    entitiesExtracted: number
-    relationshipsInferred: number
-    vfsFilesCreated: number
-    graphNodesCreated: number
-    graphEdgesCreated: number
-    entitiesMerged: number        // From deduplication
-    entitiesNew: number           // Newly created
-    processingTime: number        // In milliseconds
-  }
-}
-```
-
----
-
-## Progress Callback
-
-```typescript
-interface ImportProgress {
-  stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
-  message: string
-  processed?: number              // Current item number
-  total?: number                  // Total items
-  entities?: number               // Entities extracted
-  relationships?: number          // Relationships inferred
-  throughput?: number             // Rows per second
-  eta?: number                    // Estimated time remaining (ms)
-  queryable?: boolean             // Data queryable now (streaming mode)
-}
-```
-
----
-
-## Tips & Best Practices
-
-### Performance
-
-```typescript
-// Streaming is always on with adaptive intervals (zero config)
-// - Small imports (<1K): Flush every 100 entities
-// - Medium (1K-10K): Flush every 1000 entities
-// - Large (>10K): Flush every 5000 entities
-
-// Disable features you don't need for faster imports
-await brain.import(file, {
-  enableNeuralExtraction: false,      // 10x faster
-  enableRelationshipInference: false, // 5x faster
-  enableConceptExtraction: false      // 2x faster
-})
-```
-
-### Error Handling
-
-```typescript
-try {
-  const result = await brain.import(file, {
-    vfsPath: '/imports/data',
-    onProgress: (p) => console.log(p.message)
-  })
-  console.log('Success:', result.stats)
-} catch (error) {
-  console.error('Import failed:', error.message)
-
-  // Check partial results in VFS
-  const files = await brain.vfs().readdir('/imports')
-  console.log('Partial files:', files)
-}
-```
-
-### Querying Imported Data
-
-```typescript
-// After import completes
-const result = await brain.import(file)
-
-// Find entities by type
-const products = await brain.find({ type: 'Product' })
-
-// Get entity relationships
-const relations = await brain.related(products[0].id)
-
-// Search VFS
-const vfsFiles = await brain.vfs().find(result.vfs.rootPath + '/**/*.json')
-
-// Read entity from VFS
-const entity = await brain.vfs().readJSON(vfsFiles[0].path)
-```
-
----
-
-## Excel-Specific Tips
-
-### Column Detection
-
-Brainy auto-detects columns with flexible matching:
-
-| Your Column | Matches Pattern |
-|-------------|-----------------|
-| `Name` | term\|name\|title\|concept |
-| `Description` | definition\|description\|desc\|details |
-| `Type` | type\|category\|kind\|class |
-| `Related` | related\|see also\|links\|references |
-
-### Multiple Sheets
-
-All sheets are processed automatically:
-
-```typescript
-// catalog.xlsx with 3 sheets: Products, People, Places
-const result = await brain.import('catalog.xlsx', {
-  groupBy: 'sheet'  // Creates /Products/, /People/, /Places/
-})
-```
-
----
-
-## CSV-Specific Tips
-
-### Headers
-
-First row is treated as headers. Ensure headers exist:
-
-```csv
-Term,Definition,Type
-Product A,Description A,Product
-Product B,Description B,Product
-```
-
-### Large CSVs
-
-For large CSV files (>100K rows), streaming is automatic:
-
-```typescript
-await brain.import(largeCsv, {
-  // Automatically flushes every 5000 entities (adaptive)
-  enableNeuralExtraction: false  // Faster for large imports
-})
-```
-
----
-
-## JSON-Specific Tips
-
-### Supported Structures
-
-```javascript
-// Array of objects
-[
-  { name: "Item 1", type: "Product" },
-  { name: "Item 2", type: "Product" }
-]
-
-// Nested objects (creates hierarchical relationships)
-{
-  "company": {
-    "name": "Acme Corp",
-    "products": [
-      { "name": "Widget", "price": 9.99 }
-    ]
-  }
-}
-```
-
----
-
-## Further Reading
-
-- [Import Flow Guide](./import-flow.md) - Deep dive into how imports work
-- [Streaming Imports](./streaming-imports.md) - Progressive imports for large files
-- [VFS Guide](./vfs-guide.md) - Working with the virtual file system
-- [Type Classification](./type-classification.md) - How entity types are inferred
-- [Relationship Inference](./relationship-inference.md) - How relationships are classified
-
----
-
-**Questions?** Check the [FAQ](../faq.md) or [open an issue](https://github.com/soulcraft/brainy/issues)!

docs/guides/inspection.md

@@ -1,213 +0,0 @@
----
-title: Inspecting a Live Brainy
-slug: guides/inspection
-public: true
-category: guides
-template: guide
-order: 30
-description: Operator recipes for diagnosing a running Brainy data directory — counts, queries, query plans, health checks, and snapshots — without stopping the live writer.
-next:
-  - concepts/multi-process
----
-
-# Inspecting a Live Brainy
-
-When something is wrong in production, you need to see what's actually in the
-store. This guide covers the safe ways to query a running Brainy directory.
-
-## The cardinal rule
-
-**Never open a second writer on the same directory.** Filesystem storage will throw, and any other write path will corrupt the live writer's state. Use `Brainy.openReadOnly()` or the `brainy inspect` CLI instead.
-
-## The CLI is the fastest path
-
-```bash
-# What's in this brain?
-brainy inspect stats /data/brain
-
-# Find specific entities
-brainy inspect find /data/brain --type Event --where '{"status":"paid"}' --limit 20
-
-# Single entity by ID
-brainy inspect get /data/brain 0b7a9...
-
-# Why is this query returning empty?
-brainy inspect explain /data/brain --where '{"entityType":"booking"}'
-
-# Quick invariants
-brainy inspect health /data/brain
-
-# Random sample (no query needed)
-brainy inspect sample /data/brain --type Event --n 20
-
-# Tail new writes as they happen
-brainy inspect watch /data/brain --type Event
-
-# Save a snapshot
-brainy inspect backup /data/brain /backups/brain-$(date +%Y%m%d).tar
-```
-
-Every subcommand internally:
-
-1. Asks the live writer to flush via the cross-process RPC (skip with `--no-fresh`).
-2. Opens the data directory via `Brainy.openReadOnly()`.
-3. Runs the query.
-4. Closes cleanly.
-
-Results are JSON by default. Add `--pretty` for indented output.
-
-## When a query returns surprising results
-
-If `find()` returns `0` for a query you expect to match: run `inspect
-explain` first. It shows which index path will serve each `where` clause:
-
-```bash
-$ brainy inspect explain /data/brain --where '{"entityType":"booking","status":"paid"}'
-{
-  "query": { "where": { "entityType": "booking", "status": "paid" } },
-  "fieldPlan": [
-    { "field": "entityType", "path": "none", "notes": "No index entries for field..." },
-    { "field": "status", "path": "column-store", "notes": "O(log n) binary search..." }
-  ],
-  "warnings": [
-    "Field \"entityType\" has no index entries. find() will return [] silently."
-  ]
-}
-```
-
-The `"path": "none"` is the smoking gun. It means the field has no column
-store manifest and no sparse chunked index — so `find()` will return `[]`
-regardless of what's actually on disk. Likely causes:
-
-- The writer registered the field in memory but hasn't flushed. Run
-  `brain.requestFlush()` from the writer side, or use `brainy inspect
-  --fresh` (default).
-- The field name has a typo or wrong casing.
-- The field is genuinely absent from every entity.
-
-## Health checks
-
-`inspect health` runs a fixed battery of cheap invariant checks:
-
-```bash
-$ brainy inspect health /data/brain
-{
-  "overall": "warn",
-  "checks": [
-    { "name": "index-parity", "status": "pass", "message": "Vector (1851) and metadata (1851) agree." },
-    { "name": "field-registry", "status": "pass", "message": "23 fields registered for 1851 entities." },
-    { "name": "seeded-records", "status": "warn", "message": "15 entities tagged _seeded:true." },
-    { "name": "writer-heartbeat", "status": "pass", "message": "Writer healthy (PID 1774431...)." }
-  ]
-}
-```
-
-Each check returns `pass`, `warn`, or `fail`. The exit code is `2` when any
-check fails — useful for piping into monitoring or CI.
-
-## Programmatic inspection
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const reader = await Brainy.openReadOnly({
-  storage: { type: 'filesystem', path: '/data/brain' }
-})
-
-// Force the writer to flush before reading
-await reader.requestFlush({ timeoutMs: 5000 })
-
-// What's in there?
-const stats = await reader.stats()
-console.log(`${stats.entityCount} entities`, stats.entitiesByType)
-
-// Why is this query empty?
-const plan = await reader.explain({ where: { entityType: 'booking' } })
-for (const f of plan.fieldPlan) {
-  console.log(`${f.field} -> ${f.path}`)
-}
-
-// Run invariants
-const health = await reader.health()
-console.log(health.overall)
-
-await reader.close()
-```
-
-Every mutation method (`add`, `update`, `remove`, `relate`, `transact`,
-`restore`, ...) throws on a read-only instance with a clear message.
-
-## Backups
-
-`brainy inspect backup` asks the writer to flush first, then tars the
-directory. The snapshot reflects the writer's state at the moment of the
-flush:
-
-```bash
-brainy inspect backup /data/brain /backups/brain-2026-05-15.tar
-```
-
-For periodic backups (hourly, daily), schedule this via cron or your
-container scheduler. For point-in-time recovery, use the Db API's
-`db.persist(path)` — a self-contained hard-link snapshot that later writes
-can never alter, restorable with `brain.restore(path, { confirm: true })`.
-See [Snapshots & Time Travel](./snapshots-and-time-travel.md).
-
-## Comparing two stores
-
-`brainy inspect diff` returns a JSON summary of counts and a sample of
-entity IDs present in one but not the other. Useful when debugging
-replication or migrations:
-
-```bash
-brainy inspect diff /data/brain-prod /data/brain-staging
-```
-
-Sample-based — for a full diff, dump both with `inspect dump` and compare
-the JSONL.
-
-## Auditing graph-read truth
-
-`brain.auditGraph()` (8.6.0+) proves — or disproves — that relationship reads
-return canonical truth on a given brain, without mutating anything. It walks
-every stored relationship record, asks the same read path your application
-uses (`related()`, VFS `readdir`) with every visibility tier included, and
-classifies every discrepancy:
-
-```typescript
-const report = await brain.auditGraph()
-
-report.coherent              // true = related()/readdir can be trusted on this brain
-report.missingFromReadsCount // records the read path omits — stale index
-report.danglingEndpointsCount // relationships whose endpoint entity is gone
-report.readOnlyCount         // read-path edges with NO stored record — ghosts
-report.visibilityHiddenCount // internal/system edges hidden by design (not a fault)
-```
-
-Counts are always exact; the example lists (`missingFromReads`,
-`danglingEndpoints`, `readOnlyVerbIds`) are capped at `maxExamples`
-(default 100) and `truncatedExamples` says so when they are.
-
-Run it after any engine upgrade, restore, or migration. If it reports
-discrepancies, run `brain.repairIndex()` and audit again — a `coherent`
-report after the repair is the verified statement that the heal worked.
-Cost: one relationship-record walk plus one indexed read per distinct
-source entity — safe on a live brain.
-
-## Repairing a corrupted store
-
-If invariants fail and you suspect index corruption, `inspect repair`
-opens the store in writer mode and rebuilds all indexes from raw storage.
-**Stop the live writer first** — `repair` will throw if another writer
-holds the lock. Add `--force` only if you have personally verified the
-existing lock is stale.
-
-```bash
-brainy inspect repair /data/brain
-```
-
-## Multi-process safety summary
-
-See [concepts/multi-process](../concepts/multi-process.md) for the lock
-semantics, heartbeat behavior, and what's not yet enforced on cloud
-backends.

docs/guides/installation.md

@@ -1,89 +0,0 @@
----
-title: Installation
-slug: getting-started/installation
-public: true
-category: getting-started
-template: guide
-order: 1
-description: Install Brainy with npm, bun, yarn, or pnpm. Works in Node.js 22+ and Bun 1.0+ (server-only since 8.0). TypeScript included.
-next:
-  - getting-started/quick-start
-  - guides/storage-adapters
----
-
-# Installation
-
-## Requirements
-
-- **Node.js 22+** or **Bun 1.0+**
-- TypeScript is optional — Brainy ships with full type definitions
-
-## Install
-
-```bash
-npm install @soulcraft/brainy
-```
-
-Or with your preferred package manager:
-
-```bash
-bun add @soulcraft/brainy
-yarn add @soulcraft/brainy
-pnpm add @soulcraft/brainy
-```
-
-## Verify
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-console.log('Brainy ready.')
-```
-
-## Native Acceleration (Optional)
-
-For production workloads, add Cor for Rust-accelerated SIMD distance calculations and native embeddings:
-
-```bash
-npm install @soulcraft/cor
-```
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy({ plugins: ['@soulcraft/cor'] })
-await brain.init()  // native providers registered during init
-```
-
-Cor registers native (Rust/SIMD) vector, metadata, and graph engines behind the same Brainy `find()` API — no code changes, an optional dependency for production-scale workloads.
-
-## Server-only since 8.0
-
-Brainy 8.0 runs on Node.js 22+ and Bun 1.0+. Browser support (OPFS storage,
-Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line
-remains available on npm if you need it.
-
-## TypeScript
-
-Brainy ships with full TypeScript types. No `@types/` package needed:
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-const id = await brain.add({
-  data: 'Hello, Brainy',
-  type: NounType.Concept,
-  metadata: { created: Date.now() }
-})
-```
-
-## Next Steps
-
-- [Quick Start](/docs/getting-started/quick-start) — build your first knowledge graph in 60 seconds
-- [Storage Adapters](/docs/guides/storage-adapters) — choose the right storage for your deployment

docs/guides/migrating-to-v4.md

@@ -1,491 +0,0 @@
-# Migrating from Brainy v3.x to v4.x
-
-**Brainy v4.0.0** introduces breaking changes to the import API for improved clarity, better defaults, and more powerful features.
-
-This guide will help you migrate your code quickly and painlessly.
-
----
-
-## 🎯 Quick Migration Checklist
-
-If you just want to fix your code fast, here's what to do:
-
-- [ ] Replace `extractRelationships` with `enableRelationshipInference`
-- [ ] Remove `autoDetect` (auto-detection is now always enabled)
-- [ ] Replace `createFileStructure: true` with `vfsPath: '/your/path'`
-- [ ] Remove `excelSheets` (all sheets are now processed automatically)
-- [ ] Remove `pdfExtractTables` (table extraction is now automatic)
-- [ ] Add `enableNeuralExtraction: true` to enable AI entity extraction
-- [ ] Add `preserveSource: true` if you want to keep the original file
-
----
-
-## 📋 Option Name Changes
-
-### Complete Mapping Table
-
-| v3.x Option | v4.x Option | Action Required |
-|-------------|-------------|-----------------|
-| `extractRelationships` | `enableRelationshipInference` | **Rename option** |
-| `autoDetect` | *(removed)* | **Delete option** (always enabled) |
-| `createFileStructure` | `vfsPath` | **Replace** with VFS directory path |
-| `excelSheets` | *(removed)* | **Delete option** (all sheets processed) |
-| `pdfExtractTables` | *(removed)* | **Delete option** (always enabled) |
-| - | `enableNeuralExtraction` | **Add option** (new in v4.x) |
-| - | `enableConceptExtraction` | **Add option** (new in v4.x) |
-| - | `preserveSource` | **Add option** (new in v4.x) |
-
----
-
-## 🔄 Migration Examples
-
-### Example 1: Basic Excel Import
-
-**Before (v3.x):**
-```typescript
-const result = await brain.import('./glossary.xlsx', {
-  extractRelationships: true,
-  createFileStructure: true,
-  groupBy: 'type'
-})
-```
-
-**After (v4.x):**
-```typescript
-const result = await brain.import('./glossary.xlsx', {
-  enableRelationshipInference: true,  // ✅ Renamed
-  vfsPath: '/imports/glossary',       // ✅ Replaced createFileStructure
-  groupBy: 'type'                     // ✅ No change
-})
-```
-
----
-
-### Example 2: Full-Featured Import
-
-**Before (v3.x):**
-```typescript
-const result = await brain.import('./data.xlsx', {
-  extractRelationships: true,
-  autoDetect: true,
-  createFileStructure: true,
-  groupBy: 'type',
-  enableDeduplication: true
-})
-```
-
-**After (v4.x):**
-```typescript
-const result = await brain.import('./data.xlsx', {
-  // AI features
-  enableNeuralExtraction: true,      // ✅ NEW - Extract entity names
-  enableRelationshipInference: true, // ✅ Renamed from extractRelationships
-  enableConceptExtraction: true,     // ✅ NEW - Extract entity types
-
-  // VFS features
-  vfsPath: '/imports/data',          // ✅ Replaced createFileStructure
-  groupBy: 'type',                   // ✅ No change
-  preserveSource: true,              // ✅ NEW - Save original file
-
-  // Performance
-  enableDeduplication: true          // ✅ No change
-})
-```
-
----
-
-### Example 3: Simple Import (Defaults)
-
-**Before (v3.x):**
-```typescript
-const result = await brain.import('./data.csv', {
-  autoDetect: true,
-  extractRelationships: true
-})
-```
-
-**After (v4.x):**
-```typescript
-// Auto-detection is always enabled now
-// Just enable the features you want
-const result = await brain.import('./data.csv', {
-  enableRelationshipInference: true
-})
-
-// Or use all defaults (AI features enabled)
-const result = await brain.import('./data.csv')
-```
-
----
-
-### Example 4: PDF Import
-
-**Before (v3.x):**
-```typescript
-const result = await brain.import('./document.pdf', {
-  pdfExtractTables: true,
-  extractRelationships: true,
-  createFileStructure: true
-})
-```
-
-**After (v4.x):**
-```typescript
-const result = await brain.import('./document.pdf', {
-  // pdfExtractTables removed - always enabled
-  enableRelationshipInference: true,
-  vfsPath: '/imports/documents'
-})
-```
-
----
-
-## 💡 Why These Changes?
-
-### Clearer Option Names
-
-**v3.x naming was ambiguous:**
-- `extractRelationships` → Could mean "create relationships" or "infer relationships"
-- `createFileStructure` → Doesn't explain what structure or where
-
-**v4.x naming is explicit:**
-- `enableRelationshipInference` → Clearly means "use AI to infer semantic relationships"
-- `vfsPath` → Explicitly sets the virtual filesystem directory path
-- `enableNeuralExtraction` → Clearly indicates AI-powered entity extraction
-
-### Separation of Concerns
-
-**v4.x separates import features into clear categories:**
-
-1. **Neural/AI Features:**
-   - `enableNeuralExtraction` - Extract entity names and metadata
-   - `enableRelationshipInference` - Infer semantic relationships
-   - `enableConceptExtraction` - Extract entity types and concepts
-
-2. **VFS Features:**
-   - `vfsPath` - Virtual filesystem directory
-   - `groupBy` - Grouping strategy
-   - `preserveSource` - Keep original file
-
-3. **Performance Features:**
-   - `enableDeduplication` - Merge similar entities
-   - `confidenceThreshold` - AI confidence threshold
-   - `onProgress` - Progress callbacks
-
-### Better Defaults
-
-**v3.x required explicit enabling:**
-```typescript
-// Had to enable everything manually
-await brain.import(file, {
-  autoDetect: true,
-  extractRelationships: true,
-  createFileStructure: true
-})
-```
-
-**v4.x has smart defaults:**
-```typescript
-// Auto-detection and AI features enabled by default
-await brain.import(file)
-
-// Or customize specific features
-await brain.import(file, {
-  vfsPath: '/my/data',
-  confidenceThreshold: 0.8
-})
-```
-
----
-
-## 🆕 New Features in v4.x
-
-### Neural Entity Extraction
-Extract entity names, types, and metadata using AI:
-
-```typescript
-const result = await brain.import('./glossary.xlsx', {
-  enableNeuralExtraction: true,      // Extract entity names from "Term" column
-  enableConceptExtraction: true,     // Detect entity types (Place, Person, etc.)
-  confidenceThreshold: 0.7           // Minimum AI confidence (0-1)
-})
-
-// Result includes rich entity metadata
-result.entities.forEach(entity => {
-  console.log(`${entity.name} (${entity.type})`)
-  console.log(`Confidence: ${entity.confidence}`)
-})
-```
-
-### VFS Integration
-Imported data is organized in a virtual filesystem:
-
-```typescript
-const result = await brain.import('./data.xlsx', {
-  vfsPath: '/projects/myproject/data',
-  groupBy: 'type',        // Group by entity type
-  preserveSource: true    // Save original .xlsx file
-})
-
-// Access via VFS
-const vfs = brain.vfs()
-const files = await vfs.readdir('/projects/myproject/data')
-// ['Places/', 'Characters/', 'Concepts/', '_source.xlsx', '_metadata.json']
-
-// Read entity file
-const content = await vfs.readFile('/projects/myproject/data/Places/Talifar.json')
-```
-
-### Semantic Relationship Inference
-AI infers relationship types from context:
-
-```typescript
-const result = await brain.import('./glossary.xlsx', {
-  enableRelationshipInference: true
-})
-
-// Instead of generic "contains" relationships,
-// you get semantic verbs like:
-// - "capital_of"
-// - "located_in"
-// - "guards"
-// - "part_of"
-// - "related_to"
-
-const relations = await brain.related({ limit: 100 })
-const types = new Set(relations.map(r => r.label))
-console.log(types)
-// Set { 'capital_of', 'guards', 'located_in', 'related_to' }
-```
-
----
-
-## 🔍 What Breaks & How to Fix It
-
-### Error: "Invalid import options: 'extractRelationships'"
-
-**Cause:** Using v3.x option name
-
-**Fix:**
-```typescript
-// Before
-await brain.import(file, { extractRelationships: true })
-
-// After
-await brain.import(file, { enableRelationshipInference: true })
-```
-
----
-
-### Error: "Invalid import options: 'autoDetect'"
-
-**Cause:** Using v3.x option that's been removed
-
-**Fix:**
-```typescript
-// Before
-await brain.import(file, { autoDetect: true })
-
-// After - just remove it (auto-detection always enabled)
-await brain.import(file)
-```
-
----
-
-### Error: "Invalid import options: 'createFileStructure'"
-
-**Cause:** Using v3.x option name
-
-**Fix:**
-```typescript
-// Before
-await brain.import(file, { createFileStructure: true })
-
-// After - specify VFS path explicitly
-await brain.import(file, { vfsPath: '/imports/mydata' })
-```
-
----
-
-### Issue: Import succeeds but entities have generic names like "Entity_144"
-
-**Cause:** Neural extraction is disabled
-
-**Fix:**
-```typescript
-// Ensure AI features are enabled
-await brain.import(file, {
-  enableNeuralExtraction: true,      // ✅ Extract entity names
-  enableRelationshipInference: true, // ✅ Infer relationships
-  enableConceptExtraction: true      // ✅ Extract types
-})
-```
-
----
-
-### Issue: All relationships are type "contains"
-
-**Cause:** Relationship inference is disabled
-
-**Fix:**
-```typescript
-// Enable relationship inference
-await brain.import(file, {
-  enableRelationshipInference: true  // ✅ Use AI to detect semantic relationships
-})
-```
-
----
-
-### Issue: VFS directory doesn't exist in filesystem
-
-**This is NORMAL!** VFS is virtual - it uses Brainy entities, not physical files.
-
-**How to access VFS:**
-```typescript
-// DON'T do this:
-// ls brainy-data/vfs/  ❌ Won't work
-
-// DO this instead:
-const vfs = brain.vfs()
-await vfs.init()
-const files = await vfs.readdir('/imports')  // ✅ Correct
-```
-
----
-
-## 📦 TypeScript Users
-
-### Compile-Time Errors
-
-If you're using TypeScript, you'll get compile-time errors when using deprecated options:
-
-```typescript
-// TypeScript will show error:
-// "Type 'true' is not assignable to type 'never'"
-await brain.import(file, {
-  extractRelationships: true  // ❌ Type error
-})
-
-// Fix: Use correct option name
-await brain.import(file, {
-  enableRelationshipInference: true  // ✅ Type correct
-})
-```
-
-### IDE Autocomplete
-
-Your IDE will show deprecation warnings and suggest the correct option names:
-
-```typescript
-await brain.import(file, {
-  extract...  // IDE suggests: enableNeuralExtraction, enableRelationshipInference
-})
-```
-
----
-
-## 🎓 Best Practices for v4.x
-
-### 1. Enable All AI Features by Default
-
-```typescript
-// Good: Enable all intelligent features
-await brain.import('./data.xlsx', {
-  enableNeuralExtraction: true,
-  enableRelationshipInference: true,
-  enableConceptExtraction: true,
-  vfsPath: '/imports/data'
-})
-```
-
-### 2. Use VFS for Organization
-
-```typescript
-// Good: Organize by project
-await brain.import('./project-A.xlsx', {
-  vfsPath: '/projects/project-a/data'
-})
-
-await brain.import('./project-B.csv', {
-  vfsPath: '/projects/project-b/data'
-})
-```
-
-### 3. Preserve Source Files
-
-```typescript
-// Good: Keep original files for reference
-await brain.import('./important-data.xlsx', {
-  preserveSource: true,  // Saves original .xlsx in VFS
-  vfsPath: '/archives/2025'
-})
-```
-
-### 4. Tune Confidence Threshold
-
-```typescript
-// For high-quality data: Lower threshold
-await brain.import('./curated-glossary.xlsx', {
-  confidenceThreshold: 0.5  // Extract more entities
-})
-
-// For noisy data: Higher threshold
-await brain.import('./scraped-data.csv', {
-  confidenceThreshold: 0.8  // Only high-confidence entities
-})
-```
-
-### 5. Disable Deduplication for Large Imports
-
-```typescript
-// For small imports: Keep deduplication
-await brain.import('./small-data.xlsx', {
-  enableDeduplication: true
-})
-
-// For large imports (>1000 rows): Disable for performance
-await brain.import('./huge-database.csv', {
-  enableDeduplication: false  // Much faster
-})
-```
-
----
-
-## 🚀 Migration Automation (Future)
-
-We're working on an automated migration tool:
-
-```bash
-# Coming soon
-npx @soulcraft/brainy-migrate
-
-# Will scan your code and automatically update:
-# - Option names
-# - TypeScript types
-# - Import patterns
-```
-
----
-
-## 📚 Additional Resources
-
-- **API Documentation:** [https://brainy.dev/docs/api/import](https://brainy.dev/docs/api/import)
-- **Examples:** [examples/import-excel/](../../examples/import-excel/)
-- **Changelog:** [CHANGELOG.md](../../CHANGELOG.md)
-- **Support:** [GitHub Issues](https://github.com/soulcraft/brainy/issues)
-
----
-
-## 💬 Need Help?
-
-If you're stuck migrating:
-
-1. Check the error message - it includes migration hints
-2. Review the examples in this guide
-3. Open an issue on GitHub with your use case
-4. Join our Discord community for real-time help
-
----
-
-**Happy migrating! 🎉**

docs/guides/migration-3.36.0.md

@@ -1,386 +0,0 @@
-# Migration Guide: v3.36.0
-
-## Overview
-
-Brainy v3.36.0 introduces **enterprise-grade adaptive memory sizing** and **sync fast path optimizations** for production-scale deployments. These are **internal optimizations** that improve performance and resource efficiency with **zero breaking changes** to your existing code.
-
-**TL;DR**: Your code continues to work exactly as before. These improvements are automatic and require no migration.
-
----
-
-## What's New in v3.36.0
-
-### 1. Adaptive Memory Sizing
-
-**Automatic resource-aware cache allocation from 2GB to 128GB+ systems.**
-
-**Before v3.36.0:**
-```typescript
-// Fixed cache sizes, manual tuning required
-const brain = new Brainy()
-// Cache size: ~512MB (hardcoded default)
-```
-
-**After v3.36.0:**
-```typescript
-// Automatic adaptive sizing - no code changes needed!
-const brain = new Brainy()
-// Cache adapts:
-// - 2GB system → 400MB cache (after 150MB model reservation)
-// - 16GB system → 4GB cache
-// - 128GB system → 32GB+ cache (logarithmic scaling)
-```
-
-**Features:**
-- ✅ Container-aware (Docker/K8s cgroups v1/v2 detection)
-- ✅ Environment-smart (dev 25%, container 40%, production 50%)
-- ✅ Model memory accounting (150MB Q8, 250MB FP32)
-- ✅ Memory pressure monitoring with actionable warnings
-
-### 2. Sync Fast Path Optimization
-
-**Zero async overhead when vectors are in memory.**
-
-**Before v3.36.0:**
-```typescript
-// Every distance calculation was async (overhead even when cached)
-const results = await brain.search("query")  // Always async
-```
-
-**After v3.36.0:**
-```typescript
-// Same API, but internally optimized
-const results = await brain.search("query")
-// - Sync path: Vector in UnifiedCache → zero overhead
-// - Async path: Vector needs loading → minimal overhead
-// Your code: Unchanged! ✅
-```
-
-**Performance Impact:**
-- 🚀 Hot paths (cached vectors): **30-50% faster** (no async overhead)
-- 🔥 Cold paths (storage loading): Same as before (async when needed)
-- 📊 Production workloads: **15-25% overall speedup** (assuming 70%+ cache hit rate)
-
-### 3. Production Monitoring
-
-**New diagnostics for capacity planning and performance tuning.**
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// NEW: Comprehensive cache performance statistics
-const stats = brain.hnsw.getCacheStats()
-
-console.log(`
-Caching Strategy: ${stats.cachingStrategy}
-Cache Hit Rate: ${stats.unifiedCache.hitRatePercent}%
-Memory: ${stats.hnswCache.estimatedMemoryMB}MB HNSW cache
-Recommendations: ${stats.recommendations.join(', ')}
-`)
-
-// Example output:
-// Caching Strategy: on-demand
-// Cache Hit Rate: 89.2%
-// Memory: 245.3MB HNSW cache
-// Recommendations: All metrics healthy - no action needed
-```
-
----
-
-## Breaking Changes
-
-### ✅ Zero Breaking Changes
-
-**All changes are internal optimizations.** Your existing code continues to work without modification.
-
-**Public API:**
-- ✅ `brain.add()` - Unchanged
-- ✅ `brain.search()` - Unchanged
-- ✅ `brain.find()` - Unchanged
-- ✅ `brain.relate()` - Unchanged
-- ✅ All storage adapters - Unchanged
-
-**The only visible change:** Better performance and automatic memory sizing.
-
----
-
-## Upgrading
-
-### Step 1: Update Package
-
-```bash
-npm install @soulcraft/brainy@latest
-```
-
-### Step 2: Restart Your Application
-
-```bash
-# Development
-npm run dev
-
-# Production
-npm run start
-```
-
-**That's it!** No code changes required.
-
----
-
-## Verification
-
-### Check Adaptive Sizing is Working
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// Check UnifiedCache allocation
-const cacheStats = brain.hnsw.unifiedCache.getStats()
-console.log(`Cache Size: ${cacheStats.maxSize / 1024 / 1024} MB`)
-console.log(`Environment: ${cacheStats.memory.environment}`)
-console.log(`Allocation Ratio: ${(cacheStats.memory.allocationRatio * 100).toFixed(0)}%`)
-
-// Example output (2GB system):
-// Cache Size: 400 MB
-// Environment: development
-// Allocation Ratio: 25%
-
-// Example output (16GB production):
-// Cache Size: 4000 MB
-// Environment: production
-// Allocation Ratio: 50%
-```
-
-### Monitor Performance Improvements
-
-```typescript
-// Before: Track baseline performance
-console.time('search')
-const results = await brain.search("query", { limit: 10 })
-console.timeEnd('search')
-// Before v3.36.0: ~15ms (with async overhead)
-// After v3.36.0:  ~10ms (sync fast path when cached)
-```
-
-### Check Cache Performance Stats
-
-```typescript
-const stats = brain.hnsw.getCacheStats()
-
-console.log('Cache Performance Stats:')
-console.log(`  Strategy: ${stats.cachingStrategy}`)
-console.log(`  Entity Count: ${stats.autoDetection.entityCount.toLocaleString()}`)
-console.log(`  Cache Hit Rate: ${stats.unifiedCache.hitRatePercent}%`)
-console.log(`  HNSW Memory: ${stats.hnswCache.estimatedMemoryMB}MB`)
-console.log(`  Fairness: ${stats.fairness.fairnessViolation ? 'VIOLATION' : 'OK'}`)
-console.log(`  Recommendations:`)
-stats.recommendations.forEach(r => console.log(`    - ${r}`))
-```
-
----
-
-## Configuration (Optional)
-
-### Manual Cache Sizing
-
-If you need to override adaptive sizing:
-
-```typescript
-const brain = new Brainy({
-  cache: {
-    maxSize: 1024 * 1024 * 1024  // Force 1GB cache
-  }
-})
-```
-
-**Note:** Adaptive sizing is recommended. Manual sizing should only be used for specific deployment constraints.
-
-### Disable Sync Fast Path (Not Recommended)
-
-For debugging or compatibility testing:
-
-```typescript
-// Internal feature flag (not exposed in public API)
-// Contact support if you need to disable sync fast path
-```
-
-**Why not recommended:** Sync fast path has zero breaking changes and significant performance benefits.
-
----
-
-## Rollback
-
-If you need to rollback to v3.35.0:
-
-```bash
-npm install @soulcraft/brainy@3.35.0
-```
-
-**Note:** We don't anticipate any issues, but rollback is straightforward if needed.
-
----
-
-## Performance Tuning
-
-### Scenario 1: Low Memory Environment (2GB-4GB)
-
-```typescript
-// Adaptive sizing automatically allocates 25% in development
-const brain = new Brainy()
-await brain.init()
-
-// Monitor memory pressure
-const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
-console.log(memoryInfo.currentPressure)
-// { pressure: 'moderate', warnings: [...] }
-```
-
-**Recommendation:**
-- Let adaptive sizing handle allocation
-- Monitor `getCacheStats()` for cache hit rate
-- If hit rate < 50%, consider increasing available RAM
-
-### Scenario 2: High Memory Environment (32GB-128GB+)
-
-```typescript
-// Adaptive sizing uses logarithmic scaling to prevent over-allocation
-const brain = new Brainy()
-await brain.init()
-
-// Check allocation
-const stats = brain.hnsw.unifiedCache.getStats()
-console.log(`Allocated: ${stats.maxSize / 1024 / 1024 / 1024} GB`)
-// 64GB system → ~32GB cache (50% production allocation)
-// 128GB system → ~40GB cache (logarithmic scaling prevents waste)
-```
-
-**Recommendation:**
-- Adaptive sizing prevents over-allocation on large systems
-- Monitor fairness metrics to ensure HNSW doesn't dominate cache
-- Use `getCacheStats()` to verify cache efficiency
-
-### Scenario 3: Container Deployments (Docker/K8s)
-
-```typescript
-// Adaptive sizing detects cgroup limits automatically
-const brain = new Brainy()
-await brain.init()
-
-// Verify container detection
-const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
-console.log(`Container: ${memoryInfo.memoryInfo.isContainer}`)
-console.log(`Source: ${memoryInfo.memoryInfo.source}`)  // 'cgroup-v2' or 'cgroup-v1'
-console.log(`Available: ${memoryInfo.memoryInfo.available / 1024 / 1024} MB`)
-```
-
-**Recommendation:**
-- Set explicit memory limits in Docker/K8s (don't use unlimited)
-- Adaptive sizing allocates 40% in container environments (vs 50% bare metal)
-- Monitor warnings for container memory limit detection
-
----
-
-## Troubleshooting
-
-### Cache Size Too Small
-
-**Symptom:** On-demand caching active but cache hit rate < 50%
-
-**Solution:**
-```typescript
-const stats = brain.hnsw.getCacheStats()
-console.log(stats.recommendations)
-// Recommendation: "Low cache hit rate (42.3%). Consider increasing UnifiedCache size for better performance"
-```
-
-**Action:** Increase available system memory or reduce entity count.
-
-### Memory Pressure Warnings
-
-**Symptom:** Log warnings about memory utilization > 85%
-
-**Solution:**
-```typescript
-const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
-console.log(memoryInfo.currentPressure)
-// { pressure: 'high', warnings: ['HIGH: Memory utilization at 87.2%...'] }
-```
-
-**Action:** Either:
-1. Increase available system memory
-2. Reduce cache size manually
-3. Reduce dataset size (system automatically uses on-demand caching for large datasets)
-
-### Fairness Violations
-
-**Symptom:** HNSW using >90% of cache with <10% access
-
-**Solution:**
-```typescript
-const stats = brain.hnsw.getCacheStats()
-if (stats.fairness.fairnessViolation) {
-  console.log(`HNSW cache: ${stats.fairness.hnswAccessPercent}% access`)
-  console.log(`HNSW size: ${stats.hnswCache.estimatedMemoryMB}MB`)
-}
-```
-
-**Action:** This indicates cache eviction policies need tuning. Contact support or file an issue.
-
----
-
-## FAQ
-
-### Q: Do I need to change my code?
-
-**A:** No. All changes are internal optimizations. Your existing code works unchanged.
-
-### Q: Will my application use more memory?
-
-**A:** No. Adaptive sizing respects available system resources. On small systems (2GB), it allocates *less* than before (400MB vs 512MB) because it now accounts for model memory (150MB Q8).
-
-### Q: What if I'm in a container with memory limits?
-
-**A:** Adaptive sizing automatically detects Docker/K8s cgroup limits (v1 and v2) and allocates appropriately (40% vs 50% on bare metal).
-
-### Q: Can I disable adaptive sizing?
-
-**A:** Yes, set manual cache size in config. But adaptive sizing is recommended for production - it handles edge cases and automatically scales.
-
-### Q: Will sync fast path break anything?
-
-**A:** No. Public API remains async. Internally, it's sync when possible, async when needed. Your `await` statements work identically.
-
-### Q: How do I know what caching strategy is being used?
-
-**A:** Check `brain.hnsw.getCacheStats().cachingStrategy` (returns 'preloaded' or 'on-demand') or watch initialization logs.
-
-### Q: What's the performance impact?
-
-**A:** **15-25% overall speedup** in production workloads (assuming 70%+ cache hit rate). Hot paths (cached vectors) see **30-50% improvement**.
-
----
-
-## Next Steps
-
-1. ✅ **Upgrade:** `npm install @soulcraft/brainy@latest`
-2. 📊 **Monitor:** Use `getCacheStats()` to verify performance improvements
-3. 🎯 **Tune:** Adjust based on recommendations (if needed)
-4. 📖 **Read:** [Operations Guide](../operations/capacity-planning.md) for capacity planning
-
----
-
-## Support
-
-**Issues or questions?**
-- 📖 [Operations Guide](../operations/capacity-planning.md)
-- 🐛 [GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)
-- 💬 [Discord Community](https://discord.gg/brainy)
-
----
-
-**Built with ❤️ for production scale** | v3.36.0 | [Full Changelog](../../CHANGELOG.md)

docs/guides/model-loading.md

@@ -1,238 +0,0 @@
-# Model Loading Guide
-
-Brainy uses AI embedding models to understand and process your data. With the Candle WASM engine, the model is **embedded at compile time** - no downloads, no configuration, no external dependencies.
-
-## Zero Configuration (Default)
-
-**For all developers, no configuration is needed:**
-
-```typescript
-const brain = new Brainy()
-await brain.init() // Model is already embedded - nothing to download!
-```
-
-**What happens automatically:**
-1. Candle WASM module loads (~90MB, includes model weights)
-2. Model initializes in ~200ms
-3. Ready to use immediately
-
-**No downloads. No CDN. No configuration. Just works.**
-
-## How It Works
-
-The all-MiniLM-L6-v2 model is embedded in the WASM binary using Rust's `include_bytes!` macro:
-
-```
-candle_embeddings_bg.wasm (~90MB)
-├── Candle ML Runtime (~3MB)
-├── Model Weights (safetensors format, ~87MB)
-└── Tokenizer (HuggingFace tokenizers, ~450KB)
-```
-
-This single WASM file contains everything needed for sentence embeddings.
-
-## Environments
-
-### Bun (Recommended)
-
-```bash
-# Bun as a runtime — supported and recommended
-bun add @soulcraft/brainy
-bun run server.ts
-```
-
-Brainy is pure WebAssembly with no native binaries, so the module graph stays
-bundler-friendly. Single-binary `bun build --compile` is **not a supported
-target** at present: Bun 1.3.10 has a `--compile` codegen regression
-(`__promiseAll is not defined`) triggered by top-level `await` in the bundled
-graph. Run Brainy under the Bun runtime (above) instead.
-
-### Node.js
-
-```typescript
-// Standard Node.js
-node dist/server.js
-
-// Runs identically to Bun
-```
-
-### Browser
-
-```typescript
-// Model loads via WASM (single file, no additional assets)
-const brain = new Brainy()
-await brain.init()
-```
-
-### Docker/Kubernetes
-
-```dockerfile
-FROM oven/bun:1.1
-WORKDIR /app
-COPY package*.json ./
-RUN bun install
-COPY . .
-EXPOSE 3000
-CMD ["bun", "run", "server.ts"]
-
-# That's it! No model download step needed.
-# Model is embedded in the npm package.
-```
-
-## Model Information
-
-### all-MiniLM-L6-v2 (Embedded)
-- **Dimensions**: 384 (fixed)
-- **Format**: Safetensors (FP32)
-- **Size**: ~87MB (embedded in WASM)
-- **Total WASM Size**: ~90MB
-- **Language**: English-optimized, works with all languages
-- **Inference**: ~2-10ms per embedding
-- **Initialization**: ~200ms
-
-### Memory Usage
-- **Loaded WASM**: ~90MB
-- **Inference peak**: ~140MB total
-- **Steady state**: ~100MB
-
-## Comparing to Previous Architecture
-
-| Feature | Before (ONNX) | Now (Candle WASM) |
-|---------|--------------|-------------------|
-| Model downloads | Required on first use | None - embedded |
-| External dependencies | onnxruntime-web | None |
-| Model files | model.onnx, tokenizer.json | Embedded in WASM |
-| Offline support | Required setup | Works by default |
-| Bun compile | Broken | Works |
-| Configuration | Environment variables | None needed |
-
-## Troubleshooting
-
-### "Failed to initialize Candle Embedding Engine"
-
-**Cause**: WASM loading issue.
-
-**Solutions**:
-```bash
-# Rebuild the WASM
-npm run build:candle
-
-# Verify WASM exists
-ls dist/embeddings/wasm/pkg/candle_embeddings_bg.wasm
-# Should be ~90MB
-```
-
-### Out of Memory
-
-**Cause**: Container/environment has less than 256MB RAM.
-
-**Solutions**:
-```dockerfile
-# Increase memory limit (recommended: 512MB+)
-docker run -m 512m my-app
-```
-
-### Slow Initialization (>500ms)
-
-**Cause**: Cold start, large WASM parsing.
-
-**Solutions**:
-```typescript
-// Initialize once at startup, not per-request
-await brain.init()  // Do this once
-
-// Then reuse for all requests
-app.get('/api', async (req, res) => {
-  const results = await brain.find(req.query)
-  res.json(results)
-})
-```
-
-## Migration from Previous Versions
-
-### From v6.x (ONNX)
-
-No changes needed for most users:
-
-```typescript
-// Same API - just upgrade
-const brain = new Brainy()
-await brain.init()
-```
-
-**What's removed:**
-- `BRAINY_ALLOW_REMOTE_MODELS` - no downloads
-- `BRAINY_MODELS_PATH` - no external model files
-- `npm run download-models` - no longer needed
-
-**What's new:**
-- Faster initialization
-- Bundler-friendly (pure WASM, no native binaries)
-- No network requirements
-
-### From Custom Embedding Functions
-
-If you provided a custom embedding function, it still works:
-
-```typescript
-const brain = new Brainy({
-  embeddingFunction: myCustomEmbedder  // Still supported
-})
-```
-
-## Advanced: Building Custom WASM
-
-For contributors who want to modify the embedding engine:
-
-```bash
-# Navigate to Candle WASM source
-cd src/embeddings/candle-wasm
-
-# Build with wasm-pack
-wasm-pack build --target web --release
-
-# Copy to pkg folder
-cp pkg/* ../wasm/pkg/
-
-# Build TypeScript
-npm run build
-```
-
-## Best Practices
-
-### Development
-```typescript
-// Just works - no setup
-const brain = new Brainy()
-await brain.init()
-```
-
-### Production
-```typescript
-// Initialize once at startup
-const brain = new Brainy()
-await brain.init()
-
-// Singleton pattern recommended
-export { brain }
-```
-
-### Deployment
-```bash
-# Option 1: Bun runtime
-bun run server.ts
-
-# Option 2: Docker
-docker build -t my-app .
-docker run -p 3000:3000 my-app
-```
-
----
-
-## Additional Resources
-
-- [Production Service Architecture](../PRODUCTION_SERVICE_ARCHITECTURE.md)
-- [Zero Configuration Guide](../architecture/zero-config.md)
-- [Troubleshooting Guide](../troubleshooting.md)
-
-**Need help?** [Open an issue](https://github.com/soulcraftlabs/brainy/issues)

docs/guides/natural-language.md

@@ -1,283 +0,0 @@
-# Natural Language Queries with Brainy
-
-> **Current Status**: Basic natural language support with 220+ patterns. Advanced NLP features coming in Q1 2025.
-
-Brainy's `find()` method understands natural language, allowing you to query your data using plain English instead of complex query syntax.
-
-## Overview
-
-The natural language processing (NLP) system is powered by 220+ pre-computed patterns that understand common query intents, temporal expressions, numeric comparisons, and domain-specific terminology.
-
-## What Works Today
-
-The current NLP implementation supports:
-- ✅ Basic pattern matching (220+ patterns)
-- ✅ Simple temporal expressions ("recent", "today", "last week")
-- ✅ Common query intents ("find", "show", "search")
-- ✅ Domain keywords recognition
-- ⚠️ Limited entity extraction
-- ⚠️ Basic numeric comparisons
-
-## Basic Usage
-
-```typescript
-import { Brainy } from 'brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-// Simply ask in natural language
-const results = await brain.find("show me recent articles about AI")
-```
-
-## Supported Query Types
-
-### ✅ Currently Working
-These query patterns are supported today.
-
-### ⚠️ Basic Support
-These work with limitations.
-
-### 🚧 Coming Soon
-Planned for future releases.
-
-### Temporal Queries ⚠️ Basic Support
-
-```typescript
-// Relative time expressions
-await brain.find("documents from last week")
-await brain.find("posts created yesterday")
-await brain.find("data from the past 30 days")
-
-// Specific dates and ranges
-await brain.find("articles published in Q3 2024")
-await brain.find("reports from January to March")
-await brain.find("meetings scheduled for tomorrow")
-
-// Named periods
-await brain.find("quarterly reports from this year")
-await brain.find("summer vacation photos")
-await brain.find("holiday sales data")
-```
-
-### Numeric Filters ⚠️ Basic Support
-
-```typescript
-// Comparisons
-await brain.find("products with price under $100")
-await brain.find("articles with more than 1000 views")
-await brain.find("employees with salary above 75000")
-
-// Ranges
-await brain.find("items priced between $50 and $200")
-await brain.find("posts with 10 to 50 likes")
-await brain.find("companies with 100-500 employees")
-
-// Percentages and metrics
-await brain.find("stocks with growth over 20%")
-await brain.find("projects with completion above 80%")
-await brain.find("products with 5 star ratings")
-```
-
-### Entity and Relationship Queries 🚧 Coming Soon
-
-```typescript
-// People and organizations
-await brain.find("articles by John Smith")
-await brain.find("employees at TechCorp")
-await brain.find("papers from Stanford University")
-
-// Relationships
-await brain.find("documents related to Project X")
-await brain.find("products similar to iPhone")
-await brain.find("people who work with Sarah")
-
-// Ownership and attribution
-await brain.find("repos owned by user123")
-await brain.find("designs created by the marketing team")
-await brain.find("patents filed by Apple")
-```
-
-### Combined Complex Queries 🚧 Coming Soon
-
-```typescript
-// Multiple conditions
-await brain.find("verified research papers about machine learning from 2024 with high citations")
-
-// Business queries
-await brain.find("quarterly financial reports from Q2 2024 with revenue over 10M")
-
-// Content queries  
-await brain.find("popular blog posts by tech influencers published this month about AI")
-
-// E-commerce queries
-await brain.find("electronics under $500 with 4+ star reviews and free shipping")
-
-// Academic queries
-await brain.find("peer-reviewed papers on quantum computing published in Nature after 2020")
-```
-
-## How It Works
-
-### 1. Pattern Matching
-The NLP system first matches your query against 220+ pre-built patterns to identify:
-- Query intent (search, filter, aggregate)
-- Temporal expressions
-- Numeric comparisons
-- Entity mentions
-- Relationship indicators
-
-### 2. Entity Extraction
-Named entities are extracted and classified:
-- People names
-- Organization names
-- Product names
-- Locations
-- Dates and times
-
-### 3. Intent Classification
-The query intent is determined:
-- **Search**: Finding similar items
-- **Filter**: Applying specific criteria
-- **Aggregate**: Grouping or summarizing
-- **Navigate**: Following relationships
-
-### 4. Query Construction
-The natural language is converted to a structured Triple Intelligence query:
-
-```typescript
-// Input: "recent AI papers with high citations"
-// Output:
-{
-  like: "AI papers",
-  where: {
-    type: "paper",
-    citations: { $gte: 100 },
-    published: { $gte: "2024-01-01" }
-  },
-  boost: "recent"
-}
-```
-
-### 5. Execution
-The structured query is executed using Triple Intelligence, combining:
-- Vector similarity search
-- Metadata filtering
-- Graph traversal
-
-## Advanced Features
-
-### Contextual Understanding 🚧 Coming Soon
-
-```typescript
-// Brainy understands context and synonyms
-await brain.find("latest ML research") // Understands ML = Machine Learning
-await brain.find("top rated items")    // Understands top = high score/rating
-await brain.find("trending topics")    // Understands trending = recent + popular
-```
-
-### Fuzzy Matching 🚧 Coming Soon
-
-```typescript
-// Handles typos and variations
-await brain.find("articals about blockchian") // Still finds blockchain articles
-await brain.find("Jon Smith papers")          // Matches "John Smith"
-```
-
-### Domain-Specific Understanding ⚠️ Basic Support
-
-```typescript
-// Tech domain
-await brain.find("repos with MIT license")
-await brain.find("APIs with OAuth support")
-await brain.find("npm packages with zero dependencies")
-
-// Business domain
-await brain.find("SaaS companies with ARR over 1M")
-await brain.find("startups in Series A")
-await brain.find("B2B products with enterprise pricing")
-
-// Academic domain
-await brain.find("papers with h-index above 50")
-await brain.find("journals with impact factor over 10")
-await brain.find("conferences with double-blind review")
-```
-
-## Fallback to Structured Queries
-
-When natural language isn't sufficient, you can always use structured queries:
-
-```typescript
-// Structured query for precise control
-const results = await brain.find({
-  $and: [
-    { $vector: { $similar: "neural networks", threshold: 0.8 } },
-    { category: "research" },
-    { year: { $gte: 2023 } },
-    { $or: [
-      { author: "LeCun" },
-      { author: "Hinton" },
-      { author: "Bengio" }
-    ]}
-  ],
-  limit: 50
-})
-```
-
-## Performance Tips
-
-1. **Be specific**: More specific queries execute faster
-2. **Use proper nouns**: Names and specific terms improve accuracy
-3. **Include time frames**: Temporal filters reduce search space
-4. **Specify limits**: Always include reasonable result limits
-
-## Examples by Use Case
-
-### Customer Support
-```typescript
-await brain.find("urgent tickets from VIP customers today")
-await brain.find("unresolved issues older than 3 days")
-await brain.find("positive feedback about product X this month")
-```
-
-### Content Management
-```typescript
-await brain.find("draft posts scheduled for next week")
-await brain.find("published articles needing review")
-await brain.find("videos with over 10k views")
-```
-
-### E-commerce
-```typescript
-await brain.find("best selling products in electronics")
-await brain.find("items with low stock under 10 units")
-await brain.find("orders from California pending shipping")
-```
-
-### Analytics
-```typescript
-await brain.find("user sessions longer than 5 minutes yesterday")
-await brain.find("conversion events from mobile users")
-await brain.find("page views for /pricing in the last hour")
-```
-
-## Limitations
-
-While powerful, the NLP system has some limitations:
-
-1. **Complex logic**: Very complex boolean logic may require structured queries
-2. **Ambiguity**: Ambiguous queries may not parse as expected
-3. **Domain terms**: Highly specialized terminology may need training
-4. **Languages**: Currently optimized for English queries
-
-## Best Practices
-
-1. **Start simple**: Begin with simple queries and add complexity
-2. **Test understanding**: Use explain mode to see how queries are interpreted
-3. **Provide feedback**: Help improve the system by reporting misunderstood queries
-4. **Combine approaches**: Use NLP for exploration, structured for precision
-
-## Next Steps
-
-- [Triple Intelligence Architecture](../architecture/triple-intelligence.md)
-- [API Reference](../api/README.md)

docs/guides/nextjs-integration.md

@@ -1,930 +0,0 @@
-# Next.js Integration Guide
-
-Complete guide to integrating Brainy with Next.js applications, covering App Router, Pages Router, API routes, and deployment strategies.
-
-## 🚀 Quick Start
-
-### Installation
-
-```bash
-npx create-next-app@latest my-brainy-app
-cd my-brainy-app
-npm install @soulcraft/brainy
-```
-
-### Basic Setup
-
-```jsx
-// app/components/BrainyProvider.jsx
-'use client'
-import { createContext, useContext, useEffect, useState } from 'react'
-import { Brainy } from '@soulcraft/brainy'
-
-const BrainyContext = createContext()
-
-export function BrainyProvider({ children }) {
-  const [brain, setBrain] = useState(null)
-  const [isReady, setIsReady] = useState(false)
-
-  useEffect(() => {
-    const initBrain = async () => {
-      const newBrain = new Brainy({
-        storage: { type: 'opfs' } // Browser storage for client-side
-      })
-      await newBrain.init()
-      setBrain(newBrain)
-      setIsReady(true)
-    }
-
-    initBrain()
-  }, [])
-
-  return (
-    <BrainyContext.Provider value={{ brain, isReady }}>
-      {children}
-    </BrainyContext.Provider>
-  )
-}
-
-export const useBrainy = () => {
-  const context = useContext(BrainyContext)
-  if (!context) {
-    throw new Error('useBrainy must be used within BrainyProvider')
-  }
-  return context
-}
-```
-
-## 📱 App Router (Next.js 13+)
-
-### Root Layout Setup
-
-```jsx
-// app/layout.jsx
-import { BrainyProvider } from './components/BrainyProvider'
-import './globals.css'
-
-export const metadata = {
-  title: 'My Brainy App',
-  description: 'AI-powered search with Brainy'
-}
-
-export default function RootLayout({ children }) {
-  return (
-    <html lang="en">
-      <body>
-        <BrainyProvider>
-          {children}
-        </BrainyProvider>
-      </body>
-    </html>
-  )
-}
-```
-
-### Search Component
-
-```jsx
-// app/components/Search.jsx
-'use client'
-import { useState, useCallback } from 'react'
-import { useBrainy } from './BrainyProvider'
-
-export function Search() {
-  const { brain, isReady } = useBrainy()
-  const [query, setQuery] = useState('')
-  const [results, setResults] = useState([])
-  const [loading, setLoading] = useState(false)
-
-  const handleSearch = useCallback(async (searchQuery) => {
-    if (!isReady || !searchQuery.trim()) {
-      setResults([])
-      return
-    }
-
-    setLoading(true)
-    try {
-      const searchResults = await brain.find(searchQuery)
-      setResults(searchResults)
-    } catch (error) {
-      console.error('Search error:', error)
-      setResults([])
-    } finally {
-      setLoading(false)
-    }
-  }, [brain, isReady])
-
-  if (!isReady) {
-    return (
-      <div className="flex items-center justify-center p-4">
-        <div className="animate-spin rounded-full h-8 w-8 border-b-2 border-blue-600"></div>
-        <span className="ml-2">Initializing AI...</span>
-      </div>
-    )
-  }
-
-  return (
-    <div className="max-w-4xl mx-auto p-6">
-      <div className="mb-6">
-        <input
-          type="text"
-          value={query}
-          onChange={(e) => {
-            setQuery(e.target.value)
-            handleSearch(e.target.value)
-          }}
-          placeholder="Search with AI..."
-          className="w-full px-4 py-2 border border-gray-300 rounded-lg focus:ring-2 focus:ring-blue-500 focus:border-transparent"
-        />
-      </div>
-
-      {loading && (
-        <div className="text-center py-4">
-          <span className="text-gray-600">Searching...</span>
-        </div>
-      )}
-
-      <div className="space-y-4">
-        {results.map((result, index) => (
-          <div key={result.id || index} className="bg-white p-4 rounded-lg shadow border">
-            <h3 className="font-semibold text-lg mb-2">{result.data}</h3>
-            <div className="flex justify-between items-center text-sm text-gray-600">
-              <span>Score: {(result.score * 100).toFixed(1)}%</span>
-              {result.metadata && (
-                <span>Type: {result.metadata.type || 'Unknown'}</span>
-              )}
-            </div>
-          </div>
-        ))}
-      </div>
-
-      {query && !loading && results.length === 0 && (
-        <div className="text-center py-8 text-gray-500">
-          No results found for "{query}"
-        </div>
-      )}
-    </div>
-  )
-}
-```
-
-### Main Page
-
-```jsx
-// app/page.jsx
-import { Search } from './components/Search'
-
-export default function HomePage() {
-  return (
-    <main className="min-h-screen bg-gray-50">
-      <div className="container mx-auto py-8">
-        <h1 className="text-3xl font-bold text-center mb-8">
-          AI-Powered Search with Brainy
-        </h1>
-        <Search />
-      </div>
-    </main>
-  )
-}
-```
-
-## 🗂️ Pages Router
-
-### _app.jsx Setup
-
-```jsx
-// pages/_app.jsx
-import { BrainyProvider } from '../components/BrainyProvider'
-import '../styles/globals.css'
-
-export default function App({ Component, pageProps }) {
-  return (
-    <BrainyProvider>
-      <Component {...pageProps} />
-    </BrainyProvider>
-  )
-}
-```
-
-### Search Page
-
-```jsx
-// pages/search.jsx
-import { useState } from 'react'
-import { useBrainy } from '../components/BrainyProvider'
-
-export default function SearchPage() {
-  const { brain, isReady } = useBrainy()
-  const [query, setQuery] = useState('')
-  const [results, setResults] = useState([])
-
-  const handleSearch = async (e) => {
-    e.preventDefault()
-    if (!isReady || !query.trim()) return
-
-    const searchResults = await brain.find(query)
-    setResults(searchResults)
-  }
-
-  return (
-    <div className="container mx-auto p-6">
-      <h1 className="text-3xl font-bold mb-6">Search</h1>
-
-      <form onSubmit={handleSearch} className="mb-6">
-        <div className="flex gap-2">
-          <input
-            type="text"
-            value={query}
-            onChange={(e) => setQuery(e.target.value)}
-            placeholder="Search..."
-            className="flex-1 px-4 py-2 border rounded"
-            disabled={!isReady}
-          />
-          <button
-            type="submit"
-            disabled={!isReady || !query.trim()}
-            className="px-6 py-2 bg-blue-600 text-white rounded disabled:opacity-50"
-          >
-            Search
-          </button>
-        </div>
-      </form>
-
-      <div className="space-y-4">
-        {results.map((result, index) => (
-          <div key={index} className="p-4 border rounded">
-            <h3 className="font-semibold">{result.data}</h3>
-            <p className="text-sm text-gray-600">
-              Score: {(result.score * 100).toFixed(1)}%
-            </p>
-          </div>
-        ))}
-      </div>
-    </div>
-  )
-}
-```
-
-## 🔌 API Routes
-
-### Search API Endpoint
-
-```javascript
-// app/api/search/route.js (App Router)
-import { Brainy } from '@soulcraft/brainy'
-
-let brain = null
-
-async function initBrain() {
-  if (!brain) {
-    brain = new Brainy({
-      storage: {
-        type: 'filesystem',
-        path: process.env.BRAINY_DATA_PATH || './brainy-data'
-      }
-    })
-    await brain.init()
-  }
-  return brain
-}
-
-export async function POST(request) {
-  try {
-    const { query, options = {} } = await request.json()
-
-    if (!query) {
-      return Response.json({ error: 'Query is required' }, { status: 400 })
-    }
-
-    const brainInstance = await initBrain()
-    const results = await brainInstance.find(query, options)
-
-    return Response.json({ results, count: results.length })
-  } catch (error) {
-    console.error('Search API error:', error)
-    return Response.json(
-      { error: 'Search failed', details: error.message },
-      { status: 500 }
-    )
-  }
-}
-
-export async function GET() {
-  try {
-    const brainInstance = await initBrain()
-    const stats = await brainInstance.stats()
-
-    return Response.json({
-      status: 'ready',
-      stats: {
-        totalItems: stats.totalItems,
-        storageType: stats.storageType
-      }
-    })
-  } catch (error) {
-    return Response.json(
-      { status: 'error', error: error.message },
-      { status: 500 }
-    )
-  }
-}
-```
-
-```javascript
-// pages/api/search.js (Pages Router)
-import { Brainy } from '@soulcraft/brainy'
-
-let brain = null
-
-async function initBrain() {
-  if (!brain) {
-    brain = new Brainy({
-      storage: { type: 'filesystem', path: './brainy-data' }
-    })
-    await brain.init()
-  }
-  return brain
-}
-
-export default async function handler(req, res) {
-  if (req.method === 'POST') {
-    try {
-      const { query, options = {} } = req.body
-
-      if (!query) {
-        return res.status(400).json({ error: 'Query is required' })
-      }
-
-      const brainInstance = await initBrain()
-      const results = await brainInstance.find(query, options)
-
-      res.status(200).json({ results, count: results.length })
-    } catch (error) {
-      console.error('Search API error:', error)
-      res.status(500).json({ error: 'Search failed', details: error.message })
-    }
-  } else {
-    res.setHeader('Allow', ['POST'])
-    res.status(405).end(`Method ${req.method} Not Allowed`)
-  }
-}
-```
-
-### Add Data API
-
-```javascript
-// app/api/data/route.js
-import { Brainy } from '@soulcraft/brainy'
-
-let brain = null
-
-async function initBrain() {
-  if (!brain) {
-    brain = new Brainy({
-      storage: { type: 'filesystem', path: './brainy-data' }
-    })
-    await brain.init()
-  }
-  return brain
-}
-
-export async function POST(request) {
-  try {
-    const { data, type, metadata } = await request.json()
-
-    if (!data || !type) {
-      return Response.json(
-        { error: 'Data and type are required' },
-        { status: 400 }
-      )
-    }
-
-    const brainInstance = await initBrain()
-    const id = await brainInstance.add({ data, type, metadata })
-
-    return Response.json({ id, success: true })
-  } catch (error) {
-    console.error('Add data API error:', error)
-    return Response.json(
-      { error: 'Failed to add data', details: error.message },
-      { status: 500 }
-    )
-  }
-}
-```
-
-## 🔗 Server Actions (App Router)
-
-```jsx
-// app/actions/brainy.js
-'use server'
-import { Brainy } from '@soulcraft/brainy'
-
-let brain = null
-
-async function initBrain() {
-  if (!brain) {
-    brain = new Brainy({
-      storage: { type: 'filesystem', path: './brainy-data' }
-    })
-    await brain.init()
-  }
-  return brain
-}
-
-export async function searchAction(query, options = {}) {
-  try {
-    const brainInstance = await initBrain()
-    const results = await brainInstance.find(query, options)
-    return { results, error: null }
-  } catch (error) {
-    console.error('Search action error:', error)
-    return { results: [], error: error.message }
-  }
-}
-
-export async function addDataAction(data, type, metadata) {
-  try {
-    const brainInstance = await initBrain()
-    const id = await brainInstance.add({ data, type, metadata })
-    return { id, error: null }
-  } catch (error) {
-    console.error('Add data action error:', error)
-    return { id: null, error: error.message }
-  }
-}
-```
-
-## 📊 Data Management Features
-
-### Admin Dashboard
-
-```jsx
-// app/admin/page.jsx
-'use client'
-import { useState, useEffect } from 'react'
-import { useBrainy } from '../components/BrainyProvider'
-
-export default function AdminPage() {
-  const { brain, isReady } = useBrainy()
-  const [stats, setStats] = useState(null)
-  const [newData, setNewData] = useState('')
-  const [newType, setNewType] = useState('concept')
-
-  useEffect(() => {
-    if (isReady) {
-      loadStats()
-    }
-  }, [isReady])
-
-  const loadStats = async () => {
-    try {
-      const brainStats = await brain.stats()
-      setStats(brainStats)
-    } catch (error) {
-      console.error('Failed to load stats:', error)
-    }
-  }
-
-  const handleAddData = async (e) => {
-    e.preventDefault()
-    if (!newData.trim()) return
-
-    try {
-      await brain.add({
-        data: newData,
-        type: newType,
-        metadata: { addedAt: new Date().toISOString() }
-      })
-      setNewData('')
-      loadStats() // Refresh stats
-    } catch (error) {
-      console.error('Failed to add data:', error)
-    }
-  }
-
-  if (!isReady) {
-    return <div>Loading admin panel...</div>
-  }
-
-  return (
-    <div className="container mx-auto p-6">
-      <h1 className="text-3xl font-bold mb-6">Admin Dashboard</h1>
-
-      {/* Stats */}
-      {stats && (
-        <div className="grid grid-cols-1 md:grid-cols-3 gap-4 mb-8">
-          <div className="bg-blue-100 p-4 rounded">
-            <h3 className="font-semibold">Total Items</h3>
-            <p className="text-2xl">{stats.totalItems}</p>
-          </div>
-          <div className="bg-green-100 p-4 rounded">
-            <h3 className="font-semibold">Storage Type</h3>
-            <p className="text-lg">{stats.storageType}</p>
-          </div>
-          <div className="bg-purple-100 p-4 rounded">
-            <h3 className="font-semibold">Memory Usage</h3>
-            <p className="text-lg">{stats.memoryUsage || 'N/A'}</p>
-          </div>
-        </div>
-      )}
-
-      {/* Add Data Form */}
-      <div className="bg-white p-6 rounded-lg shadow">
-        <h2 className="text-xl font-semibold mb-4">Add New Data</h2>
-        <form onSubmit={handleAddData} className="space-y-4">
-          <div>
-            <label className="block text-sm font-medium mb-1">Data</label>
-            <textarea
-              value={newData}
-              onChange={(e) => setNewData(e.target.value)}
-              placeholder="Enter data to add..."
-              className="w-full px-3 py-2 border rounded focus:ring-2 focus:ring-blue-500"
-              rows={3}
-              required
-            />
-          </div>
-          <div>
-            <label className="block text-sm font-medium mb-1">Type</label>
-            <select
-              value={newType}
-              onChange={(e) => setNewType(e.target.value)}
-              className="w-full px-3 py-2 border rounded focus:ring-2 focus:ring-blue-500"
-            >
-              <option value="concept">Concept</option>
-              <option value="document">Document</option>
-              <option value="person">Person</option>
-              <option value="project">Project</option>
-              <option value="task">Task</option>
-            </select>
-          </div>
-          <button
-            type="submit"
-            className="px-6 py-2 bg-blue-600 text-white rounded hover:bg-blue-700"
-          >
-            Add Data
-          </button>
-        </form>
-      </div>
-    </div>
-  )
-}
-```
-
-## 🚀 Deployment
-
-### Environment Variables
-
-```bash
-# .env.local
-BRAINY_DATA_PATH=/app/brainy-data
-NODE_ENV=production
-```
-
-### Vercel Deployment
-
-```json
-// vercel.json
-{
-  "functions": {
-    "app/api/**/*.js": {
-      "maxDuration": 30
-    }
-  },
-  "env": {
-    "BRAINY_DATA_PATH": "/tmp/brainy-data"
-  }
-}
-```
-
-### Docker Setup
-
-```dockerfile
-# Dockerfile
-FROM node:22-alpine
-
-WORKDIR /app
-
-# Copy package files
-COPY package*.json ./
-RUN npm ci --only=production
-
-# Copy app files
-COPY . .
-
-# Build the app
-RUN npm run build
-
-# Create data directory
-RUN mkdir -p /app/brainy-data
-
-EXPOSE 3000
-
-CMD ["npm", "start"]
-```
-
-### next.config.js
-
-```javascript
-// next.config.js
-/** @type {import('next').NextConfig} */
-const nextConfig = {
-  experimental: {
-    serverComponentsExternalPackages: ['@soulcraft/brainy']
-  },
-  webpack: (config, { isServer }) => {
-    if (!isServer) {
-      config.resolve.fallback = {
-        ...config.resolve.fallback,
-        fs: false,
-        path: false,
-        crypto: false
-      }
-    }
-    return config
-  }
-}
-
-module.exports = nextConfig
-```
-
-## ⚡ Performance Optimization
-
-### Client-Side Optimization
-
-```jsx
-// app/hooks/useBrainCache.js
-import { useState, useCallback, useMemo } from 'react'
-
-export function useBrainCache() {
-  const [cache, setCache] = useState(new Map())
-
-  const getCachedResult = useCallback((query) => {
-    return cache.get(query)
-  }, [cache])
-
-  const setCachedResult = useCallback((query, result) => {
-    setCache(prev => {
-      const newCache = new Map(prev)
-      newCache.set(query, result)
-      // Keep only last 100 results
-      if (newCache.size > 100) {
-        const firstKey = newCache.keys().next().value
-        newCache.delete(firstKey)
-      }
-      return newCache
-    })
-  }, [])
-
-  return { getCachedResult, setCachedResult }
-}
-```
-
-### Debounced Search
-
-```jsx
-// app/hooks/useDebounceSearch.js
-import { useState, useEffect, useCallback } from 'react'
-import { useBrainy } from '../components/BrainyProvider'
-
-export function useDebounceSearch(delay = 300) {
-  const { brain, isReady } = useBrainy()
-  const [query, setQuery] = useState('')
-  const [results, setResults] = useState([])
-  const [loading, setLoading] = useState(false)
-
-  const search = useCallback(async (searchQuery) => {
-    if (!isReady || !searchQuery.trim()) {
-      setResults([])
-      return
-    }
-
-    setLoading(true)
-    try {
-      const searchResults = await brain.find(searchQuery)
-      setResults(searchResults)
-    } catch (error) {
-      console.error('Search error:', error)
-      setResults([])
-    } finally {
-      setLoading(false)
-    }
-  }, [brain, isReady])
-
-  useEffect(() => {
-    const timer = setTimeout(() => {
-      search(query)
-    }, delay)
-
-    return () => clearTimeout(timer)
-  }, [query, delay, search])
-
-  return { query, setQuery, results, loading }
-}
-```
-
-## 🔒 Security Best Practices
-
-### Input Validation
-
-```javascript
-// app/utils/validation.js
-export function validateSearchQuery(query) {
-  if (typeof query !== 'string') {
-    throw new Error('Query must be a string')
-  }
-
-  if (query.length > 1000) {
-    throw new Error('Query too long')
-  }
-
-  // Sanitize query
-  return query.trim()
-}
-
-export function validateDataInput(data, type, metadata) {
-  if (!data || !type) {
-    throw new Error('Data and type are required')
-  }
-
-  if (typeof data !== 'string') {
-    throw new Error('Data must be a string')
-  }
-
-  if (data.length > 10000) {
-    throw new Error('Data too long')
-  }
-
-  return { data: data.trim(), type, metadata }
-}
-```
-
-### Rate Limiting
-
-```javascript
-// app/middleware/rateLimit.js
-const requests = new Map()
-
-export function rateLimit(req, limit = 100, window = 60000) {
-  const ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress
-  const now = Date.now()
-
-  if (!requests.has(ip)) {
-    requests.set(ip, [])
-  }
-
-  const userRequests = requests.get(ip)
-
-  // Remove old requests
-  const validRequests = userRequests.filter(time => now - time < window)
-
-  if (validRequests.length >= limit) {
-    throw new Error('Rate limit exceeded')
-  }
-
-  validRequests.push(now)
-  requests.set(ip, validRequests)
-
-  return true
-}
-```
-
-## 📚 Advanced Patterns
-
-### Context + Reducer Pattern
-
-```jsx
-// app/contexts/BrainyContext.jsx
-'use client'
-import { createContext, useContext, useReducer, useEffect } from 'react'
-import { Brainy } from '@soulcraft/brainy'
-
-const BrainyContext = createContext()
-
-const initialState = {
-  brain: null,
-  isReady: false,
-  error: null,
-  stats: null
-}
-
-function brainyReducer(state, action) {
-  switch (action.type) {
-    case 'INIT_START':
-      return { ...state, error: null }
-    case 'INIT_SUCCESS':
-      return { ...state, brain: action.brain, isReady: true, error: null }
-    case 'INIT_ERROR':
-      return { ...state, error: action.error, isReady: false }
-    case 'UPDATE_STATS':
-      return { ...state, stats: action.stats }
-    default:
-      return state
-  }
-}
-
-export function BrainyProvider({ children }) {
-  const [state, dispatch] = useReducer(brainyReducer, initialState)
-
-  useEffect(() => {
-    const initBrain = async () => {
-      dispatch({ type: 'INIT_START' })
-      try {
-        const brain = new Brainy()
-        await brain.init()
-        dispatch({ type: 'INIT_SUCCESS', brain })
-
-        // Load initial stats
-        const stats = await brain.stats()
-        dispatch({ type: 'UPDATE_STATS', stats })
-      } catch (error) {
-        console.error('Brain initialization failed:', error)
-        dispatch({ type: 'INIT_ERROR', error: error.message })
-      }
-    }
-
-    initBrain()
-  }, [])
-
-  return (
-    <BrainyContext.Provider value={{ state, dispatch }}>
-      {children}
-    </BrainyContext.Provider>
-  )
-}
-
-export const useBrainyContext = () => {
-  const context = useContext(BrainyContext)
-  if (!context) {
-    throw new Error('useBrainyContext must be used within BrainyProvider')
-  }
-  return context
-}
-```
-
-## 🔍 Testing
-
-### Unit Tests
-
-```javascript
-// __tests__/brainy.test.js
-import { render, screen, waitFor } from '@testing-library/react'
-import { BrainyProvider } from '../app/components/BrainyProvider'
-import { Search } from '../app/components/Search'
-
-// Mock Brainy
-jest.mock('@soulcraft/brainy', () => ({
-  Brainy: jest.fn().mockImplementation(() => ({
-    init: jest.fn().mockResolvedValue(undefined),
-    find: jest.fn().mockResolvedValue([
-      { id: '1', data: 'Test result', score: 0.9 }
-    ])
-  }))
-}))
-
-describe('Search Component', () => {
-  it('renders search input', async () => {
-    render(
-      <BrainyProvider>
-        <Search />
-      </BrainyProvider>
-    )
-
-    await waitFor(() => {
-      expect(screen.getByPlaceholderText('Search with AI...')).toBeInTheDocument()
-    })
-  })
-})
-```
-
-## 📖 Complete Example Project
-
-Here's a complete mini-project structure:
-
-```
-my-brainy-app/
-├── app/
-│   ├── components/
-│   │   ├── BrainyProvider.jsx
-│   │   ├── Search.jsx
-│   │   └── AdminPanel.jsx
-│   ├── api/
-│   │   ├── search/route.js
-│   │   └── data/route.js
-│   ├── admin/
-│   │   └── page.jsx
-│   ├── layout.jsx
-│   └── page.jsx
-├── next.config.js
-├── package.json
-└── README.md
-```
-
-This structure provides a complete, production-ready Next.js application with Brainy integration.
-
-## 🎯 Next Steps
-
-- [Vue.js Integration Guide](vue-integration.md) - Vue.js patterns
-- [Production Deployment](../deployment/CLOUD_DEPLOYMENT_GUIDE.md) - Deploy to production
-- [API Reference](../api/README.md) - Complete API documentation
-- [Examples Repository](https://github.com/soulcraftlabs/brainy-examples) - More examples

docs/guides/optimistic-concurrency.md

@@ -1,214 +0,0 @@
----
-title: Optimistic concurrency with _rev
-slug: guides/optimistic-concurrency
-public: true
-category: guides
-template: guide
-order: 8
-description: Use the per-entity `_rev` counter and `update({ ifRev })` to coordinate concurrent writes safely. Covers the lock pattern, idempotent inserts with `ifAbsent`, and recovery on conflict.
-next:
-  - concepts/consistency-model
-  - guides/find-limits
----
-
-# Optimistic concurrency with `_rev`
-
-Brainy 7.31.0 adds a per-entity revision counter so multiple writers can coordinate without a global lock or external coordinator. The pattern is the same one CouchDB, PouchDB, and ETag-based HTTP caches use: read the current revision, do your work, write back with `ifRev: <whatRevYouSaw>`. If the revision moved, your write is rejected and you retry against the latest state.
-
-## What gets added
-
-| Surface | Behavior |
-|---|---|
-| `entity._rev: number` | Returned on every `get()`, `find()`, `search()`. Initialized to `1` on `add()`. Bumped by `1` on every successful `update()`. Pre-7.31.0 entities without `_rev` are surfaced as `1`. |
-| `update({ id, ..., ifRev: number })` | If the persisted `_rev` does not equal `ifRev`, throws `RevisionConflictError`. Omitting `ifRev` keeps the prior (unconditional) update behavior. |
-| `RevisionConflictError` | Carries `{ id, expected, actual }` for principled recovery. |
-| `add({ id, ifAbsent: true })` | By-ID idempotent insert. Returns the existing `id` if one is already present; no throw, no overwrite. |
-| `addMany({ items, ifAbsent: true })` | Applies `ifAbsent` to every item. Per-item `ifAbsent` overrides the batch flag. |
-
-`_rev` is the **per-entity** counter. Its store-wide counterpart is the generation counter behind the [Db API](../concepts/consistency-model.md): `brain.transact(ops, { ifAtGeneration })` is CAS over the whole store, `update({ ifRev })` (and `ifRev` on `transact()` update operations) is CAS over one entity.
-
-## The lock pattern
-
-Every distributed-job scheduler eventually wants this exact loop:
-
-```ts
-import { Brainy, RevisionConflictError } from '@soulcraft/brainy'
-
-const LOCK_ID = '...uuid for this job slot...'
-
-// Bootstrap the lock document once (idempotent).
-await brain.add({
-  id: LOCK_ID,
-  type: NounType.Document,
-  data: { owner: null, expiresAt: 0 },
-  ifAbsent: true
-})
-
-async function tryAcquireLock(workerId: string, ttlMs: number) {
-  const lock = await brain.get(LOCK_ID)
-  if (!lock) throw new Error('lock document missing')
-
-  const state = lock.data as { owner: string | null; expiresAt: number }
-  const now = Date.now()
-
-  // Only take the lock if it's free or expired.
-  if (state.owner && state.expiresAt > now) return false
-
-  try {
-    await brain.update({
-      id: LOCK_ID,
-      data: { owner: workerId, expiresAt: now + ttlMs },
-      ifRev: lock._rev
-    })
-    return true
-  } catch (err) {
-    if (err instanceof RevisionConflictError) {
-      // Another worker grabbed it between our read and write.
-      return false
-    }
-    throw err
-  }
-}
-```
-
-No external lock service, no Redis SETNX, no Cloud Tasks. The CAS check is the lock.
-
-## Read-modify-write with retry
-
-The other common shape is "update a counter / config object" with bounded retries on conflict:
-
-```ts
-async function incrementCounter(id: string, by: number) {
-  for (let attempt = 0; attempt < 5; attempt++) {
-    const entity = await brain.get(id)
-    if (!entity) throw new Error('counter does not exist')
-    const current = (entity.data as { value: number }).value
-    try {
-      await brain.update({
-        id,
-        data: { value: current + by },
-        ifRev: entity._rev
-      })
-      return
-    } catch (err) {
-      if (err instanceof RevisionConflictError) continue   // refetch + retry
-      throw err
-    }
-  }
-  throw new Error('counter update conflict after 5 attempts')
-}
-```
-
-The retry bound matters — without one, two unlucky writers can ping-pong forever.
-
-## Idempotent bootstrap with `ifAbsent`
-
-For singletons (config rows, well-known seed entities, job-state documents) where the natural ID is deterministic:
-
-```ts
-await brain.add({
-  id: 'config:singleton',
-  type: NounType.Document,
-  data: { tenantQuota: 1000 },
-  ifAbsent: true
-})
-```
-
-- First caller writes; gets back `'config:singleton'`.
-- Every subsequent caller short-circuits at the pre-read; gets back the same `'config:singleton'` without touching the existing entity.
-- `_rev` is **not** bumped on the no-op path (no write happened).
-
-`ifAbsent` is only meaningful when you supply an `id`. With no `id`, Brainy generates a fresh UUID that can never collide, so the flag is silently ignored.
-
-`addMany({ items, ifAbsent: true })` applies the flag to every item. Mixing per-item overrides with the batch flag works as you'd expect: per-item `ifAbsent: false` opts an individual row out, per-item `ifAbsent: true` opts it in.
-
-### Why no `addIfMissing({ match, add })` for attribute-based dedup?
-
-You may want "create if no entity with this email exists" (lookup by attribute, not by ID). That's a different operation:
-
-```ts
-// What we DID NOT ship in 7.31.0 — the attribute-based variant.
-await brain.addIfMissing({                       // ← not a real API
-  match: { type: 'Person', where: { email: 'x@y.com' } },
-  add:   { data: '...', metadata: { email: 'x@y.com' } }
-})
-```
-
-It's race-prone as a plain read-then-write: two concurrent imports both see "not found," both insert, you get duplicates. Without a unique-index primitive (which Brainy doesn't have today), close the race with whole-store CAS — read at a pinned generation, then commit only if nothing moved:
-
-```ts
-import { GenerationConflictError } from '@soulcraft/brainy'
-
-async function addIfMissingByEmail(email: string, data: string) {
-  for (let attempt = 0; attempt < 5; attempt++) {
-    const db = brain.now()
-    try {
-      const existing = await db.find({
-        type: NounType.Person,
-        where: { email },
-        limit: 1
-      })
-      if (existing.length > 0) return existing[0].id
-
-      const committed = await brain.transact(
-        [{ op: 'add', type: NounType.Person, subtype: 'customer', data, metadata: { email } }],
-        { ifAtGeneration: db.generation }   // rejects if ANYTHING committed since the read
-      )
-      return committed.receipt!.ids[0]
-    } catch (err) {
-      if (err instanceof GenerationConflictError) continue   // world moved — re-read + retry
-      throw err
-    } finally {
-      await db.release()
-    }
-  }
-  throw new Error('addIfMissingByEmail conflict after 5 attempts')
-}
-```
-
-`ifAtGeneration` is deliberately coarse — *any* committed write invalidates it — so keep the retry bound. When you control the ID, `ifAbsent` stays the cheaper tool.
-
-## How `_rev` relates to generations
-
-Brainy 8.0 has exactly two write-coordination counters, at two granularities:
-
-| Counter | Scope | What it tracks | CAS surface | Conflict error |
-|---|---|---|---|---|
-| **`_rev`** | One entity | Per-entity write count, bumped on every successful update | `update({ ifRev })`, `{ op: 'update', ifRev }` in `transact()` | `RevisionConflictError` |
-| **Generation** | Whole store | One tick per committed `transact()` batch or single-operation write | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` |
-
-They compose: a `transact()` batch can carry per-entity `ifRev` checks *and* a whole-store `ifAtGeneration`; any failed check rejects the entire batch before anything is staged. Generations also power snapshots and time travel (`brain.now()`, `brain.asOf()`, `db.persist()`) — see the [consistency model](../concepts/consistency-model.md) and [Snapshots & Time Travel](./snapshots-and-time-travel.md).
-
-A snapshot or historical view captures each entity *including* its `_rev` at that moment, so reading the past and writing back with `ifRev` against the live state works exactly as you'd hope: the write fails if the entity moved since the state you copied from.
-
-## The transact envelope: batch size, budget, and bulk imports
-
-`transact()` applies its batch atomically under one commit — which means the whole batch
-shares one **apply budget**. Since 8.7.0 the budget scales with the batch:
-`max(30 s, opCount × 2 s)`, or exactly what you pass as `timeoutMs`. A tripped budget rolls
-the entire batch back (nothing partial survives) and throws a retryable
-`TransactionTimeoutError` that names the operation it stopped at, the batch size, and the
-elapsed vs budgeted time — a diagnosis, not just a failure:
-
-```
-Transaction timed out at operation 41/120 ('add') — 246012ms elapsed, budget 240000ms.
-The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch.
-```
-
-Practical envelope guidance for bulk work:
-
-1. **Precompute embeddings outside the commit path.** Embedding inside `transact()` spends
-   the budget on model inference. Use `brain.embedBatch(texts)` and pass each vector via
-   the op's `vector` field — the commit then pays only storage costs, and a retried batch
-   never re-pays inference. (The win is *where* the inference happens, not raw embedding
-   throughput: on the default WASM engine, batch and sequential embedding measure
-   comparably, ~160 ms/text; native embedding providers may batch faster.)
-2. **Chunk very large imports** into batches of a few hundred ops with one `transact()`
-   each. You lose whole-import atomicity but keep per-chunk atomicity, bounded memory, and
-   resumability — pair with `ifAbsent` upserts so a retried chunk is idempotent.
-3. **Slow disks change the math, not the contract.** On network-attached storage a single
-   op can cost ~2 s (canonical write + fsync + index maintenance). The scaled default
-   absorbs that; pass an explicit `timeoutMs` only when you know better than the scale.
-4. **`addMany`/`relateMany` are the convenience tier** — they chunk and batch-embed for
-   you, with per-item error reporting instead of batch atomicity. Choose by what you need:
-   atomic-all-or-nothing → `transact()`; resilient bulk load → `addMany`.

docs/guides/quick-start.md

@@ -1,111 +0,0 @@
----
-title: Quick Start
-slug: getting-started/quick-start
-public: true
-category: getting-started
-template: guide
-order: 2
-description: Build your first knowledge graph in 60 seconds. Add entities, create relationships, and query with Triple Intelligence — vector + graph + metadata in one call.
-next:
-  - concepts/triple-intelligence
-  - api/reference
----
-
-# Quick Start
-
-Get Brainy running in under a minute.
-
-## 1. Install
-
-```bash
-npm install @soulcraft/brainy
-```
-
-## 2. Initialize
-
-```typescript
-import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-```
-
-That's it. Brainy auto-configures storage, loads the embedding model, and builds the indexes.
-
-## 3. Add Knowledge
-
-```typescript
-// Text is automatically embedded into 384-dim vectors
-const reactId: string = await brain.add({
-  data: 'React is a JavaScript library for building user interfaces',
-  type: NounType.Concept,
-  subtype: 'library',                                  // Sub-classification within Concept
-  metadata: { category: 'frontend', year: 2013 }
-})
-
-const nextId: string = await brain.add({
-  data: 'Next.js framework for React with server-side rendering',
-  type: NounType.Concept,
-  subtype: 'framework',
-  metadata: { category: 'framework', year: 2016 }
-})
-```
-
-`type` is one of Brainy's 42 stable NounTypes. `subtype` is your free-form sub-classification within that type — flat string, no hierarchy, indexed on the fast path. See **[Subtypes & Facets](./subtypes-and-facets.md)** for the full guide.
-
-## 4. Create Relationships
-
-```typescript
-// Typed graph relationships
-await brain.relate({
-  from: nextId,
-  to: reactId,
-  type: VerbType.DependsOn
-})
-```
-
-## 5. Query with Triple Intelligence
-
-```typescript
-import type { Result } from '@soulcraft/brainy'
-
-// All three search paradigms in one call
-const results: Result[] = await brain.find({
-  query: 'modern frontend frameworks',    // Vector similarity search
-  where: { year: { greaterThan: 2015 } }, // Metadata filtering
-  connected: { to: reactId, depth: 2 }   // Graph traversal
-})
-
-console.log(results[0].data)   // 'Next.js framework for React...'
-console.log(results[0].score)  // 0.94
-```
-
-## What Just Happened
-
-Every entity you `add()` lives in three indexes simultaneously:
-
-| Index | What it stores | Query with |
-|-------|---------------|------------|
-| Vector | 384-dim embedding of `data` | `find({ query: '...' })` |
-| Metadata | All `metadata` fields | `find({ where: { ... } })` |
-| Graph | Typed relationships from `relate()` | `find({ connected: { ... } })` |
-
-`find()` queries all three in parallel and fuses the results.
-
-## Natural Language Queries
-
-Brainy understands 220+ natural language patterns:
-
-```typescript
-// These all work without any configuration
-await brain.find({ query: 'recent documents about machine learning' })
-await brain.find({ query: 'articles created this week' })
-await brain.find({ query: 'people who work at Anthropic' })
-```
-
-## Next Steps
-
-- [Triple Intelligence](/docs/concepts/triple-intelligence) — understand how the query engine works
-- [The Find System](/docs/guides/find-system) — advanced queries, operators, and graph traversal
-- [API Reference](/docs/api/reference) — complete method documentation
-- [Storage Adapters](/docs/guides/storage-adapters) — filesystem, memory

docs/guides/reacting-to-changes.md

@@ -1,117 +0,0 @@
----
-title: Reacting to Changes
-slug: guides/reacting-to-changes
-public: true
-category: guides
-template: guide
-order: 11
-description: Subscribe to every committed mutation with `brain.onChange` — the in-process change feed behind live UIs, cache invalidation, and realtime sync. Covers the event shape, delivery guarantees, and catch-up patterns.
-next:
-  - guides/optimistic-concurrency
-  - guides/snapshots-and-time-travel
----
-
-# Reacting to Changes
-
-`brain.onChange(cb)` is Brainy's in-process change feed: subscribe once and
-receive one event per committed mutation — **every** mutation, regardless of
-how it happened. Direct calls, batch methods, `transact()`, imports, and
-Virtual Filesystem writes all funnel through the same commit point the feed is
-emitted from, so nothing slips past it.
-
-```ts
-const off = brain.onChange((e) => {
-  if (e.kind === 'entity') {
-    console.log(`${e.op} ${e.entity?.type} ${e.id} @ generation ${e.generation}`)
-  }
-})
-
-await brain.add({ data: 'Ada Lovelace', type: 'person' })
-// → "add person 0198... @ generation 42"
-
-off() // unsubscribe when done
-```
-
-## The event
-
-```ts
-interface BrainyChangeEvent {
-  kind: 'entity' | 'relation' | 'store'
-  op: 'add' | 'update' | 'remove' | 'relate' | 'unrelate' | 'updateRelation'
-    | 'clear' | 'restore'
-  id?: string
-  entity?:   { id: string; type: string; subtype?: string;
-               metadata: Record<string, unknown>; service?: string }
-  relation?: { id: string; from: string; to: string; type: string;
-               metadata?: Record<string, unknown> }
-  generation?: number
-  timestamp: number
-}
-```
-
-- **Entity events** (`add` / `update` / `remove`) carry the post-commit indexed
-  view — `type`, `subtype`, and the full custom `metadata`, so you can match
-  your own `where`-style filters against events without a read.
-- **Deletes are fully described.** A `remove` or `unrelate` event carries the
-  record's *last committed state* (sourced from the commit's own history
-  record), not just an id.
-- **Batches emit per item.** `addMany` / `updateMany` / `relateMany` /
-  `removeMany` emit one event per affected record; a `transact()` batch emits
-  one event per item, all sharing the batch's single `generation`.
-- **Cascades are visible.** Removing an entity also emits `unrelate` for each
-  relationship the delete cascaded to.
-- **Store-level events** (`kind: 'store'`) fire for the two wholesale
-  operations — `clear()` and `restore()` — and mean *"everything may have
-  changed; refetch what you care about."*
-
-## Delivery guarantees
-
-- **Post-commit only.** An aborted write — a losing
-  [`ifRev` compare-and-swap](optimistic-concurrency.md), a rejected
-  transaction — never emits. If you received the event, the write is durable.
-- **Commit-ordered.** Events arrive in the order writes committed;
-  `generation` is monotonic.
-- **Asynchronous, never blocking.** Delivery happens in a microtask after the
-  write completes. A slow listener cannot delay a write; a throwing listener
-  is logged and isolated from other listeners.
-- **Zero overhead when unused.** With no subscribers, the write path does no
-  event work at all.
-- **Fire-and-forget.** There is no replay or backpressure. For catch-up after
-  a disconnect, use the `generation` on each event together with
-  [`asOf()` / the transaction log](snapshots-and-time-travel.md): record the
-  last generation you processed, and on reconnect diff from there. For file
-  content specifically, `vfs.readFile(path, { asOf })` and
-  `vfs.history(path)` are the temporal read — see
-  [Snapshots & Time Travel](snapshots-and-time-travel.md).
-
-## Patterns
-
-**Cache invalidation** — drop cached reads for whatever changed:
-
-```ts
-brain.onChange((e) => {
-  if (e.kind === 'store') return cache.clear()
-  if (e.id) cache.delete(e.id)
-})
-```
-
-**Live queries (notify-and-refetch)** — re-run a query when a relevant change
-lands, rather than diffing incrementally:
-
-```ts
-brain.onChange((e) => {
-  if (e.kind === 'entity' && e.entity?.type === 'order') {
-    refreshOpenOrdersView() // debounce as needed
-  }
-})
-```
-
-**Forwarding to other processes** — the feed is in-process by design. To push
-changes to browsers or other services, forward events through your own
-transport (WebSocket, SSE) from the process that owns the brain.
-
-## Lifecycle
-
-`onChange` returns an unsubscribe function — call it when tearing down a
-subscriber (for example, when evicting a pooled instance). `brain.close()`
-drops all listeners; no events are delivered for or after `close()`.

docs/guides/schema-migrations.md

@@ -1,284 +0,0 @@
-# Schema Migrations
-
-Brainy includes a built-in migration system for transforming entity and verb metadata across storage versions. Migrations are pure functions that run once per storage instance, with optional snapshot backup (`backupTo`), resume support, and error tracking.
-
----
-
-## Quick Start
-
-### 1. Define a migration
-
-Add your migration to the `MIGRATIONS` array in `src/migration/migrations.ts`:
-
-```typescript
-import type { Migration } from './types.js'
-
-export const MIGRATIONS: Migration[] = [
-  {
-    id: '7.17.0-rename-status',
-    version: '7.17.0',
-    description: 'Rename "state" field to "status"',
-    applies: 'nouns',
-    transform: (m) => {
-      if ('state' in m) {
-        const { state, ...rest } = m
-        return { ...rest, status: state }
-      }
-      return null // already migrated or not applicable
-    }
-  }
-]
-```
-
-### 2. Ship the new version
-
-That's it. Brainy detects pending migrations on `init()` and either runs them automatically or warns the user to call `brain.migrate()`.
-
----
-
-## How It Works
-
-When `brain.init()` runs:
-
-1. **Detection** — reads migration state from storage (one key lookup). Compares completed migration IDs against the `MIGRATIONS` array. If nothing pending, cost is ~0ms.
-
-2. **Small datasets** (`autoMigrate: true`, <10K entities) — migrates inline during `init()`.
-
-3. **Large datasets or manual mode** — logs a warning. User calls `brain.migrate()` when ready.
-
-When `brain.migrate()` runs:
-
-1. **Backup (optional)** — with `backupTo`, a hard-link snapshot of the current generation is persisted before any transform runs. Rollback is `brain.restore(backupPath, { confirm: true })`.
-
-2. **Transform** — iterates all nouns/verbs in paginated batches. For each entity, calls the `transform` function. If it returns a new object, saves it. If it returns `null`, skips. Vectors are never touched.
-
-3. **Save state** — records each completed migration ID so it never re-runs.
-
-4. **Rebuild indexes** — if any entities were modified, rebuilds the MetadataIndex.
-
----
-
-## Writing Migrations
-
-### The Migration interface
-
-```typescript
-interface Migration {
-  id: string          // Unique ID, e.g. "7.17.0-rename-field"
-  version: string     // Version that introduced this migration
-  description: string // Human-readable description
-  applies: 'nouns' | 'verbs' | 'both'
-  transform: (metadata: Record<string, unknown>) => Record<string, unknown> | null
-}
-```
-
-### Transform rules
-
-- **Return a new object** to modify the entity's metadata.
-- **Return `null`** to skip (no change needed).
-- **Must be idempotent** — running the same transform twice on the same data should produce the same result (or return `null` the second time). This is required because interrupted runs resume and re-encounter already-migrated entities.
-- **Must be pure** — no side effects, no async, no external state.
-- Transforms only see metadata. Vectors, embeddings, and the `data` field stored inside metadata are available as properties on the metadata object.
-
-### Ordering
-
-Migrations run in array order. Add new migrations at the end of the `MIGRATIONS` array. Each migration runs independently per entity — migration 2 sees the output of migration 1.
-
-### Validation
-
-`MigrationRunner.validateMigrations()` checks migration definitions and will throw on:
-
-- Duplicate IDs
-- Invalid `applies` values (must be `'nouns'`, `'verbs'`, or `'both'`)
-- Non-function `transform`
-- Missing or empty `id`, `version`, or `description`
-
----
-
-## API Reference
-
-### `brain.migrate(options?)`
-
-```typescript
-// Dry-run: preview what would change without writing
-const preview = await brain.migrate({ dryRun: true })
-// preview.pendingMigrations — array of { id, description }
-// preview.affectedEntities  — count of entities that would change
-// preview.totalEntities     — count of entities scanned
-// preview.sampleChanges     — up to 5 before/after samples
-// preview.estimatedTime     — rough time estimate string
-
-// Apply migrations (optionally with a pre-migration snapshot)
-const result = await brain.migrate({ backupTo: '/backups/pre-migration' })
-// result.backupPath        — snapshot path, or null when no backupTo was supplied
-// result.migrationsApplied — array of migration IDs that ran
-// result.entitiesProcessed — total entities scanned
-// result.entitiesModified  — entities actually changed
-// result.errors            — array of entity-level errors (non-fatal)
-```
-
-### Options
-
-```typescript
-interface MigrateOptions {
-  dryRun?: boolean       // Preview without writing (default: false)
-  maxErrors?: number     // Bail out after N entity errors (default: 100)
-  onProgress?: (progress: {
-    migrationId: string
-    processed: number
-    modified: number
-    hasMore: boolean
-  }) => void
-}
-```
-
----
-
-## Error Handling
-
-If a transform function throws on a specific entity, the error is recorded and migration continues to the next entity. The failed entity's metadata is left unchanged.
-
-```typescript
-const result = await brain.migrate()
-
-if (result.errors.length > 0) {
-  for (const err of result.errors) {
-    console.warn(`Entity ${err.entityId} failed in ${err.migrationId}: ${err.error}`)
-  }
-}
-```
-
-If errors exceed `maxErrors` (default: 100), the migration stops early and returns partial results. Successfully migrated entities keep their changes; failed entities are unchanged.
-
-```typescript
-// Strict mode: fail fast on any error
-const result = await brain.migrate({ maxErrors: 1 })
-
-// Lenient mode: tolerate many errors
-const result = await brain.migrate({ maxErrors: 10000 })
-```
-
----
-
-## Backup and Rollback
-
-Pass `backupTo` and `brain.migrate()` persists a snapshot of the current generation **before any transform runs**. On filesystem storage the snapshot is a hard-link farm — created without copying entity data, and immune to later writes (see [Snapshots & Time Travel](./snapshots-and-time-travel.md)):
-
-```typescript
-const result = await brain.migrate({ backupTo: '/backups/pre-migration-8.0' })
-console.log(result.backupPath) // '/backups/pre-migration-8.0' (null when no backupTo)
-```
-
-To roll back, restore the snapshot wholesale:
-
-```typescript
-await brain.restore('/backups/pre-migration-8.0', { confirm: true })
-```
-
-Without `backupTo`, no backup is taken — transforms are idempotent (they return `null` when already applied), but a pre-migration snapshot is the cheap insurance for anything destructive.
-
----
-
-## Progress Tracking
-
-For large datasets, use the `onProgress` callback:
-
-```typescript
-await brain.migrate({
-  onProgress: ({ migrationId, processed, modified, hasMore }) => {
-    console.log(`[${migrationId}] ${processed} scanned, ${modified} modified${hasMore ? '...' : ' (done)'}`)
-  }
-})
-```
-
-Progress is reported after each batch (batch size is determined by the storage adapter).
-
----
-
-## Examples
-
-### Rename a field
-
-```typescript
-{
-  id: '7.17.0-rename-state-to-status',
-  version: '7.17.0',
-  description: 'Rename metadata.state to metadata.status',
-  applies: 'nouns',
-  transform: (m) => {
-    if ('state' in m) {
-      const { state, ...rest } = m
-      return { ...rest, status: state }
-    }
-    return null
-  }
-}
-```
-
-### Add a default value
-
-```typescript
-{
-  id: '7.18.0-add-priority-default',
-  version: '7.18.0',
-  description: 'Add priority field with default "normal"',
-  applies: 'both',
-  transform: (m) => {
-    if (!('priority' in m)) {
-      return { ...m, priority: 'normal' }
-    }
-    return null
-  }
-}
-```
-
-### Remove a deprecated field
-
-```typescript
-{
-  id: '7.19.0-remove-legacy-flag',
-  version: '7.19.0',
-  description: 'Remove deprecated "legacy" field',
-  applies: 'nouns',
-  transform: (m) => {
-    if ('legacy' in m) {
-      const { legacy, ...rest } = m
-      return rest
-    }
-    return null
-  }
-}
-```
-
-### Transform verb metadata
-
-```typescript
-{
-  id: '7.20.0-normalize-verb-weights',
-  version: '7.20.0',
-  description: 'Normalize verb weights from 0-100 to 0-1 scale',
-  applies: 'verbs',
-  transform: (m) => {
-    if (typeof m.weight === 'number' && m.weight > 1) {
-      return { ...m, weight: m.weight / 100 }
-    }
-    return null
-  }
-}
-```
-
----
-
-## Storage Backend Compatibility
-
-Migrations work identically across all storage backends (Memory, FileSystem). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter.
-
-Batch size and rate limiting are automatically configured per adapter — no tuning required.
-
----
-
-## What Migrations Don't Do
-
-- **Re-embedding** — migrations transform metadata only. If you change your embedding model or dimensions, that requires re-vectorizing data, which is a separate concern (not part of this system).
-- **Vector modification** — the `vectors.json` files are never touched by migrations.
-- **Schema enforcement** — migrations are opt-in transforms, not schema validators. Brainy's metadata is schemaless by design.

docs/guides/snapshots-and-time-travel.md

@@ -1,449 +0,0 @@
----
-title: Snapshots & Time Travel
-slug: guides/snapshots-and-time-travel
-public: true
-category: guides
-template: guide
-order: 9
-description: Recipes for the Db API — instant backups with persist(), restore, time-travel debugging with asOf(), range queries over history (diff, history, since, log windows), persist-before-migrate, what-if analysis with with(), and audit trails via transaction metadata.
-next:
-  - concepts/consistency-model
-  - guides/optimistic-concurrency
-  - guides/external-backups
----
-
-# Snapshots & Time Travel
-
-Brainy 8.0 treats the database as a **value**: `brain.now()` pins the
-current state as an immutable `Db`, `brain.transact()` commits an atomic
-batch and hands you the resulting value, `brain.asOf()` opens past state,
-and `db.persist()` cuts a self-contained snapshot. This guide is the recipe
-book. The precise guarantees behind every recipe live in the
-[consistency model](../concepts/consistency-model.md).
-
-## Instant backup
-
-Pin the current state, persist it, release:
-
-```typescript
-const db = brain.now()
-try {
-  await db.persist('/backups/2026-06-11')
-} finally {
-  await db.release()
-}
-```
-
-On filesystem storage the snapshot is built from **hard links**: every data
-file in Brainy is immutable-by-rename, so the snapshot is created without
-copying entity data and shares disk space with the live store. Later writes
-can never alter it — a rewrite swaps the inode, the snapshot keeps the old
-bytes. Cross-device targets fall back to per-file byte copies, and
-persisting an in-memory brain serializes it to the same directory layout —
-a real, durable store.
-
-> Archiving a brain directory with **external tools** (`tar`, `rsync`, `cp`)?
-> Some index files are sparse and can explode to their apparent size under a
-> naive copy — see [External Backups & Sparse Storage](/docs/guides/external-backups).
-
-Two things to know:
-
-- `persist()` requires the view to still be the store's **latest**
-  generation. If something committed after your pin, it throws
-  `GenerationConflictError` instead of snapshotting the wrong state — pin
-  and persist before further writes, or retry with a fresh `brain.now()`.
-- The target directory must be empty or absent.
-
-For scheduled backups, this loop is the whole job:
-
-```typescript
-const db = brain.now()
-try {
-  await db.persist(`/backups/${new Date().toISOString().slice(0, 10)}`)
-} finally {
-  await db.release()
-}
-```
-
-## Restore
-
-`restore()` replaces the store's **entire** current state from a snapshot —
-entities, relationships, indexes, history. It is deliberately loud about it:
-
-```typescript
-await brain.restore('/backups/2026-06-11', { confirm: true })
-```
-
-- `{ confirm: true }` is mandatory — current state is destroyed.
-- The snapshot is copied in (never linked), so it stays independent and can
-  be restored again later.
-- All indexes are rebuilt from the restored records.
-- The generation counter is floored at its pre-restore value, so generation
-  numbers you observed before the restore are never reissued.
-- Live `Db` pins do not survive a restore — release them first.
-
-## Open a snapshot read-only
-
-You do not have to restore to look inside a snapshot. `Brainy.load()` opens
-it as a self-contained read-only store with the **full query surface**,
-including vector search:
-
-```typescript
-const db = await Brainy.load('/backups/2026-06-11')
-
-const hits = await db.find({ query: 'unpaid invoices from the spring campaign' })
-const orders = await db.find({ type: NounType.Document, subtype: 'order' })
-
-await db.release()   // closes the underlying read-only instance
-```
-
-`brain.asOf('/backups/2026-06-11')` does the same from an existing brain.
-This is also the 8.0 answer to "named branches": a branch is a name → path
-mapping your application keeps, where each path is a persisted snapshot.
-Need a writable copy? Restore the snapshot into a fresh data directory and
-open a writer on it — instead of switching a shared store between branches
-in place, every line of code always sees exactly the store it opened.
-
-## Time-travel debugging
-
-When production data looks wrong, query the past directly — by wall-clock
-time or by generation:
-
-```typescript
-// What did this order look like yesterday?
-const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000))
-const before = await yesterday.get(orderId)
-
-// Full queries work at any reachable generation — search, graph, filters:
-const thenActive = await yesterday.find({
-  type: NounType.Document,
-  subtype: 'order',
-  where: { status: 'active' }
-})
-
-await yesterday.release()
-```
-
-Pin two points in time and diff them:
-
-```typescript
-const before = await brain.asOf(1041)
-const after = brain.now()
-
-const changed = await after.since(before)
-changed.nouns   // entity ids touched by transactions in between
-changed.verbs   // relationship ids touched in between
-
-await before.release()
-await after.release()
-```
-
-Three things to remember:
-
-- History granularity is per-write: EVERY write — `transact()` AND a
-  single-operation `add`/`update`/`remove`/`relate` — is its own immutable
-  generation, so a pin always freezes against later writes and every write is
-  individually addressable via `asOf()` (see the
-  [consistency model](../concepts/consistency-model.md)). Use `transact()` when
-  you want several operations to share ONE atomic generation.
-- The first index-accelerated query (semantic search, traversal, cursors,
-  aggregation) at a historical generation builds an in-memory index
-  materialization — O(n at that generation), once per `Db`, freed on
-  `release()`. Metadata-level reads are free.
-- Generations reclaimed by `compactHistory()` throw
-  `GenerationCompactedError` — persist anything you need to keep forever.
-
-## Range queries over history
-
-`asOf()` answers "what was the state AT a point". Four range verbs answer
-"what happened BETWEEN two points" and "what is one entity's whole history".
-They all build on the same generation records — no extra bookkeeping.
-
-### `diff(a, b)` — what changed, classified
-
-`since()` gives you the raw set of *touched* ids. `diff()` goes further: it
-resolves each touched id at both endpoints and classifies it as **added**,
-**removed**, or **modified** — split by entities (`nouns`) and relationships
-(`verbs`). An id that was touched but ended up identical (changed then
-reverted, or created and deleted within the interval) lands in **none** of the
-buckets. Endpoints are a generation, a `Date`, or a `Db`, in either order:
-
-```typescript
-const d = await brain.diff(1041, brain.generation())
-
-d.added.nouns      // entity ids created between the two states
-d.removed.nouns    // entity ids deleted
-d.modified.nouns   // entity ids whose stored value actually changed
-d.added.verbs      // …relationships, the same three ways
-```
-
-Orientation is `a → b`: `added` means "exists at `b`, not at `a`". The
-comparison behind `modified` is key-order-insensitive, so a no-op re-write of
-the same fields never shows up as a change.
-
-### `history(id, range?)` — one entity, every version
-
-`asOf()` is per-*generation*; `history()` is per-*entity*. It returns every
-distinct version of one id over a range, oldest first — each `value` is the
-materialized state at that version (and `null` marks a removal):
-
-```typescript
-const h = await brain.history(invoiceId)
-
-for (const v of h.versions) {
-  console.log(v.generation, v.value?.metadata?.status ?? '(deleted)')
-}
-// 1041 'draft'
-// 1043 'approved'
-// 1050 'paid'
-```
-
-Every version ties to the trusted `asOf()` path — `v.value` equals
-`(await brain.asOf(v.generation)).get(id)`. Pass `{ from, to }` (generation or
-`Date`) to bound the range; a `from` below the compaction horizon is quietly
-truncated to it rather than throwing (history is best-effort over surviving
-records).
-
-### `since()` and `transactionLog()` take ranges too
-
-`since()` accepts a `Db`, a generation number, or a `Date` — all equivalent,
-all an **exclusive** lower bound (`db.since(prior)` equals
-`db.since(prior.generation)`):
-
-```typescript
-await brain.now().since(1041)                          // ids changed after generation 1041
-await brain.now().since(new Date(Date.now() - 3_600_000)) // …in the last hour
-```
-
-`transactionLog({ from, to })` windows the commit log **inclusively** on both
-ends (a log window names the commits it spans — the deliberate contrast to
-`since`'s exclusive lower bound); `limit` applies after the window, newest
-first:
-
-```typescript
-const window = await brain.transactionLog({ from: 1041, to: 1050 })  // commits 1041…1050
-const recent = await brain.transactionLog({ from: lastHour, limit: 20 })
-```
-
-### Composing them
-
-"Which orders changed in this window?" is `diff` ids intersected with an
-`asOf` query — the two agree by construction:
-
-```typescript
-const changed = await brain.diff(g1, g2)
-const atG2 = await brain.asOf(g2)
-const changedOrders = (await atG2.find({ type: NounType.Document, subtype: 'order' }))
-  .map(r => r.id)
-  .filter(id => changed.added.nouns.includes(id) || changed.modified.nouns.includes(id))
-await atG2.release()
-```
-
-One contrast to keep straight: `diff` and `since` **throw**
-`GenerationCompactedError` for a bound below the horizon, while `history`
-**truncates** to the horizon — diffs must be exact, history is best-effort.
-
-## Safe schema migration
-
-`brain.migrate()` integrates with snapshots directly: pass `backupTo` and a
-hard-link snapshot of the current generation is persisted **before any
-transform runs**:
-
-```typescript
-const result = await brain.migrate({ backupTo: '/backups/pre-migration-8.0' })
-console.log(result.migrationsApplied, result.backupPath)
-
-// If the migration went wrong, roll the whole store back:
-await brain.restore('/backups/pre-migration-8.0', { confirm: true })
-```
-
-The same persist-before-mutate pattern works for any risky bulk operation,
-not just migrations:
-
-```typescript
-const pin = brain.now()
-try {
-  await pin.persist('/backups/pre-bulk-edit')
-} finally {
-  await pin.release()
-}
-await runRiskyBulkEdit(brain)
-```
-
-## What-if analysis
-
-`db.with(ops)` applies a transaction **speculatively, in memory** — nothing
-touches disk, the generation counter, or the indexes. Ask "what would the
-store look like if…", then commit the same operations for real:
-
-```typescript
-const ops = [
-  { op: 'update', id: employeeId, metadata: { team: 'platform' } },
-  { op: 'relate', from: employeeId, to: milestoneId, type: VerbType.ParticipatesIn, subtype: 'assignment' }
-]
-
-const base = brain.now()
-const whatIf = await base.with(ops)
-
-await whatIf.get(employeeId)                          // sees the change
-await whatIf.find({ where: { team: 'platform' } })    // metadata finds work
-await whatIf.related(employeeId)                      // overlay relations included
-
-await whatIf.release()
-await base.release()
-
-// Looks right — make it real, atomically:
-await brain.transact(ops)
-```
-
-**The boundary:** speculative entities carry no embeddings (`with()` never
-invokes the embedder), so semantic search, traversal, cursors, aggregation,
-and `persist()` throw `SpeculativeOverlayError` on overlay views instead of
-returning silently incomplete results. `get()`, metadata-filter `find()`,
-and filter-based `related()` are fully supported. Overlays chain — calling
-`with()` on an overlay stacks another layer.
-
-## Audit trails
-
-`transact()` reifies transaction metadata: whatever you pass as `meta` is
-recorded durably alongside the committed generation and timestamp, readable
-via `brain.transactionLog()`:
-
-```typescript
-await brain.transact(
-  [{ op: 'update', id: invoiceId, metadata: { status: 'approved' } }],
-  { meta: { author: 'approvals-service', actor: 'jane@example.com', reason: 'PO-7741' } }
-)
-
-const log = await brain.transactionLog({ limit: 20 })   // newest first
-// [{ generation: 1042, timestamp: 1765432100000, meta: { author: 'approvals-service', ... } }]
-```
-
-Combine the log with `asOf()` to reconstruct exactly what any transaction
-did:
-
-```typescript
-const [entry] = await brain.transactionLog({ limit: 1 })
-
-const after = await brain.asOf(entry.generation)
-const before = await brain.asOf(entry.generation - 1)
-
-const touched = await after.since(before)
-for (const id of touched.nouns) {
-  console.log(id, await before.get(id), '→', await after.get(id))
-}
-
-await before.release()
-await after.release()
-```
-
-For per-entity write coordination (rather than whole-store history), the
-`_rev` counter and `ifRev` CAS remain the right tool — see
-[optimistic concurrency](./optimistic-concurrency.md).
-
-## Keeping history bounded
-
-Under Model-B every write is a generation, so history can grow quickly —
-Brainy auto-compacts at `close()` (time-bounded per pass) under the
-**`retention`** knob (configured on the constructor). Since 8.9.0, `flush()`
-never compacts: flushing is durability work and costs only what the current
-window's writes cost, regardless of history backlog. A long-lived writer that
-never closes keeps its history until its next explicit `compactHistory()` —
-schedule one in your maintenance window if you run bounded retention:
-
-```typescript
-// Zero-config: ADAPTIVE — keep as much history as free disk/RAM allows,
-// reclaiming oldest-first under pressure. (This is the default.)
-new Brainy({ /* retention unset */ })
-
-// Unbounded — never reclaim history (opt in explicitly):
-new Brainy({ retention: 'all' })
-
-// Explicit CAPS — reclaim oldest-unpinned generations while ANY cap is exceeded:
-new Brainy({ retention: { maxGenerations: 1000, maxAge: 7 * 86_400_000, maxBytes: 512 * 1024 ** 2 } })
-```
-
-Reclaim manually at any time (the same caps, plus an optional per-pass time
-budget for maintenance windows — an early stop is a consistent prefix and the
-next pass resumes):
-
-```typescript
-await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 24 * 60 * 60 * 1000 })
-await brain.compactHistory({ maxBytes: 512 * 1024 ** 2, timeBudgetMs: 10_000 })
-```
-
-Compaction never breaks a pinned read — record-sets are reclaimed only when
-no live `Db` could need them (live pins are ALWAYS exempt). Release views you
-are done with (including the ones `transact()` returns), and `persist()` any
-generation you want to keep beyond the retention window: snapshots are
-self-contained and unaffected by compaction.
-
-## Time travel for files (the VFS)
-
-Since 8.2.0, time travel covers Virtual Filesystem **content**, not just
-entity records. File bytes are retention-protected: a content blob referenced
-by any generation inside the retention window is never reclaimed, so reading
-the past always returns the exact bytes — never a stale field or a
-dangling hash.
-
-**`vfs.readFile(path, { asOf })`** takes a generation number or a `Date` and
-returns the file's exact bytes as they stood then. It resolves the path's
-current entity, then materializes its state at the target generation — so it
-answers *"what did the file at this path hold at that point?"* It bypasses
-the content cache; the `encoding` option still applies. Asking about a
-generation before the file existed throws the usual not-found error, and
-asking past the retention window's compaction horizon throws a
-compacted-generation error.
-
-**`vfs.history(path)`** returns the file's versions inside the retention
-window, oldest first — one `FileVersion` per generation that wrote the file,
-the newest entry being the current state:
-
-```typescript
-// A CMS page evolves…
-await brain.vfs.writeFile('/pages/home.json', '{"title":"Launch"}')
-await brain.vfs.writeFile('/pages/home.json', '{"title":"Launch v2"}')
-await brain.vfs.writeFile('/pages/home.json', '{"title":""}') // bad deploy!
-
-// Every version is listed and readable:
-const versions = await brain.vfs.history('/pages/home.json')
-// → [{ generation, timestamp, hash, size, mimeType? }, …] ascending
-
-const good = versions[versions.length - 2]
-const bytes = await brain.vfs.readFile('/pages/home.json', {
-  asOf: good.generation
-})
-
-// Restore = write the old bytes back. This is a NEW write (a new
-// generation) — history is never rewritten, so the bad version stays
-// visible in the audit trail.
-await brain.vfs.writeFile('/pages/home.json', bytes)
-```
-
-Two lifecycle consequences worth stating plainly:
-
-- **Deleting or overwriting a file no longer frees its bytes immediately.**
-  Old content lives until history compaction reclaims the generations that
-  reference it — the same `retention` budget that bounds all Model-B history
-  (and pinned views are exempt, exactly as above). Size your `retention` for
-  the file-version depth you want; `retention: 'all'` keeps every version of
-  every file forever.
-- **After `compactHistory()` reclaims a generation, its file versions are
-  gone** and their bytes are physically reclaimed. (This also fixed a
-  pre-8.2.0 defect where overwritten content was never reclaimed at all — an
-  unbounded silent leak.)
-
-## From branches to values
-
-If you used the pre-8.0 `fork`/`checkout`/`commit`/`versions` surface, every
-use case maps to a sharper tool:
-
-| Pre-8.0 habit | 8.0 recipe |
-|---|---|
-| `fork()` to experiment safely | `db.with(ops)` for speculation in memory; a restored snapshot in a fresh directory for a long-lived writable copy |
-| `commit()` checkpoints | `transact(ops, { meta })` — every batch is an atomic, logged, time-travelable commit |
-| `checkout()` to switch branches | Open the snapshot you want — `Brainy.load(path)` read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store. |
-| `getHistory()` | `brain.transactionLog()` + `db.since(priorDb)` |
-| `versions.save()` per-entity snapshots | A pinned `Db` or persisted snapshot captures *every* entity at that moment; `asOf()` reads any entity's past state |
-| `versions.restore()` | `brain.restore(snapshot, { confirm: true })` for the whole store, or read the old entity via `asOf()` and write it back with `transact()` |
-| Backup branches | `db.persist(path)` — instant, hard-link-shared, self-contained |

docs/guides/standard-import-progress.md

@@ -1,453 +0,0 @@
-# Standard Import Progress API
-
-## ✅ Build Once, Works for ALL Formats
-
-**Brainy provides a 100% standardized progress API** - write your UI/tool once, and it works for all 7 supported formats (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX) with **zero format-specific code**.
-
----
-
-## 🎯 The Standard Interface
-
-### One Interface for Everything
-
-```typescript
-import { Brainy } from '@soulcraft/brainy'
-
-const brain = await Brainy.create()
-
-// THIS CODE WORKS FOR ALL 7 FORMATS - NO FORMAT-SPECIFIC LOGIC NEEDED!
-await brain.import(anyBuffer, {
-  onProgress: (progress) => {
-    // Standard fields - ALWAYS available regardless of format
-    console.log(progress.stage)          // Current stage
-    console.log(progress.message)        // Human-readable status
-
-    // Optional fields - available when relevant
-    console.log(progress.processed)      // Items processed so far
-    console.log(progress.total)          // Total items (if known)
-    console.log(progress.entities)       // Entities extracted
-    console.log(progress.relationships)  // Relationships inferred
-    console.log(progress.throughput)     // Items/sec (during extraction)
-    console.log(progress.eta)            // Time remaining in ms
-  }
-})
-```
-
-### The Complete Interface
-
-```typescript
-interface ImportProgress {
-  // === ALWAYS PRESENT ===
-
-  /** High-level stage (5 stages for all formats) */
-  stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
-
-  /** Human-readable status message */
-  message: string
-
-  // === AVAILABLE WHEN RELEVANT ===
-
-  /** Items processed (rows, pages, nodes, etc.) */
-  processed?: number
-
-  /** Total items to process (if known ahead of time) */
-  total?: number
-
-  /** Entities extracted so far */
-  entities?: number
-
-  /** Relationships inferred so far */
-  relationships?: number
-
-  /** Processing rate (items per second) */
-  throughput?: number
-
-  /** Estimated time remaining (milliseconds) */
-  eta?: number
-
-  /** Whether data is queryable at this point */
-  queryable?: boolean
-}
-```
-
----
-
-## 🎨 Generic UI Components
-
-### React Progress Component (Works for ALL Formats)
-
-```typescript
-import { useState } from 'react'
-import { Brainy } from '@soulcraft/brainy'
-
-function UniversalImportProgress({ file }: { file: File }) {
-  const [progress, setProgress] = useState({
-    stage: 'idle',
-    message: 'Ready to import',
-    percent: 0,
-    entities: 0,
-    relationships: 0
-  })
-
-  const handleImport = async () => {
-    const buffer = await file.arrayBuffer()
-    const brain = await Brainy.create()
-
-    await brain.import(Buffer.from(buffer), {
-      // THIS WORKS FOR CSV, PDF, EXCEL, JSON, MARKDOWN, YAML, DOCX!
-      onProgress: (p) => {
-        setProgress({
-          stage: p.stage,
-          message: p.message,
-
-          // Calculate percentage from stage + processed/total
-          percent: calculatePercent(p),
-
-          entities: p.entities || 0,
-          relationships: p.relationships || 0
-        })
-      }
-    })
-  }
-
-  // Helper: Calculate percentage from progress
-  function calculatePercent(p: ImportProgress): number {
-    // Use processed/total if available
-    if (p.processed && p.total) {
-      return Math.round((p.processed / p.total) * 100)
-    }
-
-    // Otherwise estimate from stage
-    const stagePercents = {
-      detecting: 5,
-      extracting: 50,
-      'storing-vfs': 80,
-      'storing-graph': 90,
-      complete: 100
-    }
-    return stagePercents[p.stage] || 0
-  }
-
-  return (
-    <div className="import-progress">
-      {/* Stage Indicator */}
-      <div className="stages">
-        {['detecting', 'extracting', 'storing-vfs', 'storing-graph', 'complete'].map(s => (
-          <span
-            key={s}
-            className={progress.stage === s ? 'active' : ''}
-          >
-            {s}
-          </span>
-        ))}
-      </div>
-
-      {/* Progress Bar */}
-      <div className="progress-bar">
-        <div style={{ width: `${progress.percent}%` }} />
-      </div>
-
-      {/* Status Message (format-specific but always readable) */}
-      <p className="message">{progress.message}</p>
-
-      {/* Counts */}
-      <div className="counts">
-        <span>Entities: {progress.entities}</span>
-        <span>Relationships: {progress.relationships}</span>
-      </div>
-    </div>
-  )
-}
-```
-
-**This component works perfectly for:**
-- ✅ CSV files with 10,000 rows
-- ✅ PDF documents with 200 pages
-- ✅ Excel workbooks with 5 sheets
-- ✅ JSON files with nested structures
-- ✅ Markdown documents with sections
-- ✅ YAML configuration files
-- ✅ DOCX documents with paragraphs
-
-**No format detection needed. No format-specific rendering. Just works.**
-
----
-
-### CLI Progress Indicator (Works for ALL Formats)
-
-```typescript
-import ora from 'ora'
-import { Brainy } from '@soulcraft/brainy'
-
-async function importWithProgress(filePath: string) {
-  const spinner = ora('Starting import...').start()
-  const brain = await Brainy.create()
-
-  try {
-    await brain.import(filePath, {
-      // THIS WORKS FOR ALL 7 FORMATS!
-      onProgress: (p) => {
-        // Update spinner text with current message
-        spinner.text = p.message
-
-        // Add counts if available
-        if (p.entities || p.relationships) {
-          spinner.text += ` (${p.entities || 0} entities, ${p.relationships || 0} relationships)`
-        }
-
-        // Add throughput/ETA if available (during extraction)
-        if (p.throughput && p.eta) {
-          const etaSec = Math.round(p.eta / 1000)
-          spinner.text += ` [${p.throughput.toFixed(1)}/sec, ETA: ${etaSec}s]`
-        }
-
-        // Change spinner when complete
-        if (p.stage === 'complete') {
-          spinner.succeed(p.message)
-        }
-      }
-    })
-  } catch (error) {
-    spinner.fail(`Import failed: ${error.message}`)
-  }
-}
-
-// Works for ANY format!
-await importWithProgress('data.csv')
-await importWithProgress('document.pdf')
-await importWithProgress('workbook.xlsx')
-await importWithProgress('config.yaml')
-```
-
-**CLI Output (same code, different formats):**
-
-```bash
-# CSV Import
-⠋ Detecting format...
-⠙ Parsing CSV rows (delimiter: ",")
-⠹ Extracting entities from csv (45 rows/sec, ETA: 120s) (150 entities, 45 relationships)
-⠸ Extracting entities from csv (45 rows/sec, ETA: 60s) (750 entities, 223 relationships)
-✔ Import complete (1350 entities, 401 relationships)
-
-# PDF Import
-⠋ Detecting format...
-⠙ Loading PDF document...
-⠹ Processing page 5 of 23
-⠸ Extracting entities from pdf (2.5 pages/sec, ETA: 30s) (45 entities, 12 relationships)
-✔ Import complete (156 entities, 89 relationships)
-
-# Excel Import
-⠋ Detecting format...
-⠙ Loading Excel workbook...
-⠹ Reading sheet: Sales (2/5)
-⠸ Extracting entities from excel (120 rows/sec, ETA: 45s) (500 entities, 234 relationships)
-✔ Import complete (2340 entities, 892 relationships)
-```
-
-**Same code. Different formats. Perfect progress for all.**
-
----
-
-### Dashboard with Real-Time Stats (Works for ALL Formats)
-
-```typescript
-function ImportDashboard() {
-  const [stats, setStats] = useState({
-    stage: '',
-    message: '',
-    elapsed: 0,
-    entities: 0,
-    relationships: 0,
-    throughput: 0,
-    eta: 0
-  })
-
-  const startTime = Date.now()
-
-  const handleImport = async (file: File) => {
-    await brain.import(await file.arrayBuffer(), {
-      // UNIVERSAL PROGRESS HANDLER - WORKS FOR ALL FORMATS!
-      onProgress: (p) => {
-        setStats({
-          stage: p.stage,
-          message: p.message,
-          elapsed: Date.now() - startTime,
-          entities: p.entities || 0,
-          relationships: p.relationships || 0,
-          throughput: p.throughput || 0,
-          eta: p.eta || 0
-        })
-      }
-    })
-  }
-
-  return (
-    <div className="dashboard">
-      <h2>Import Progress</h2>
-
-      <div className="metric">
-        <label>Stage</label>
-        <value>{stats.stage}</value>
-      </div>
-
-      <div className="metric">
-        <label>Status</label>
-        <value>{stats.message}</value>
-      </div>
-
-      <div className="metric">
-        <label>Elapsed</label>
-        <value>{(stats.elapsed / 1000).toFixed(1)}s</value>
-      </div>
-
-      <div className="metric">
-        <label>Entities</label>
-        <value>{stats.entities.toLocaleString()}</value>
-      </div>
-
-      <div className="metric">
-        <label>Relationships</label>
-        <value>{stats.relationships.toLocaleString()}</value>
-      </div>
-
-      {stats.throughput > 0 && (
-        <div className="metric">
-          <label>Throughput</label>
-          <value>{stats.throughput.toFixed(1)} items/sec</value>
-        </div>
-      )}
-
-      {stats.eta > 0 && (
-        <div className="metric">
-          <label>ETA</label>
-          <value>{(stats.eta / 1000).toFixed(0)}s</value>
-        </div>
-      )}
-    </div>
-  )
-}
-```
-
-**This dashboard shows live stats for ANY format** - CSV, PDF, Excel, JSON, Markdown, YAML, DOCX.
-
----
-
-## 📊 What Messages Look Like (Format-Specific Text, Standard Fields)
-
-While the **fields are standardized**, the **message text** varies by format to be most helpful:
-
-```typescript
-// CSV Import Messages
-"Detecting format..."
-"Parsing CSV rows (delimiter: ",")"
-"Extracted 1000 rows, inferring types..."
-"Extracting entities from csv (45 rows/sec, ETA: 120s)..."
-"Creating VFS structure..."
-"Import complete"
-
-// PDF Import Messages
-"Detecting format..."
-"Loading PDF document..."
-"Processing page 5 of 23"
-"Extracting entities from pdf (2.5 pages/sec, ETA: 30s)..."
-"Creating VFS structure..."
-"Import complete"
-
-// Excel Import Messages
-"Detecting format..."
-"Loading Excel workbook..."
-"Reading sheet: Sales (2/5)"
-"Extracting entities from excel (120 rows/sec, ETA: 45s)..."
-"Creating VFS structure..."
-"Import complete"
-```
-
-**Key Point:** You can display `progress.message` directly in your UI **without parsing it**. It's always human-readable and contextually appropriate.
-
----
-
-## ✅ The 5 Standard Stages (Same for ALL Formats)
-
-Every import goes through these 5 stages in order:
-
-| Stage | Duration | Description | Fields Available |
-|-------|----------|-------------|------------------|
-| **detecting** | ~1% | Format detection | `stage`, `message` |
-| **extracting** | ~70% | Parse file + AI extraction | `stage`, `message`, `processed`, `total`, `entities`, `relationships`, `throughput`, `eta` |
-| **storing-vfs** | ~5% | Create file structure | `stage`, `message` |
-| **storing-graph** | ~20% | Create graph nodes | `stage`, `message`, `entities`, `relationships` |
-| **complete** | ~1% | Finalize | `stage`, `message`, `entities`, `relationships` |
-
-**These 5 stages are the same whether you're importing:**
-- A 10MB CSV file with 50,000 rows
-- A 200-page PDF document
-- A 5-sheet Excel workbook
-- A nested JSON structure
-- A Markdown document
-- A YAML configuration
-- A DOCX document
-
----
-
-## 🎯 Why This Matters
-
-### Build Tools That Work for Everything
-
-```typescript
-// ONE progress handler for your entire application
-function universalProgressHandler(progress: ImportProgress) {
-  // Update UI (works for all formats)
-  updateProgressBar(progress)
-  updateStatusText(progress.message)
-  updateCounts(progress.entities, progress.relationships)
-
-  // Log to analytics (works for all formats)
-  analytics.track('import_progress', {
-    stage: progress.stage,
-    processed: progress.processed,
-    total: progress.total
-  })
-
-  // Send to monitoring (works for all formats)
-  monitoring.gauge('import.entities', progress.entities)
-  monitoring.gauge('import.throughput', progress.throughput)
-}
-
-// Use it everywhere
-await brain.import(csvFile, { onProgress: universalProgressHandler })
-await brain.import(pdfFile, { onProgress: universalProgressHandler })
-await brain.import(excelFile, { onProgress: universalProgressHandler })
-await brain.import(jsonFile, { onProgress: universalProgressHandler })
-```
-
-### No Format Detection Needed
-
-```typescript
-// ❌ DON'T DO THIS (format-specific handling)
-if (format === 'csv') {
-  // CSV-specific progress code
-} else if (format === 'pdf') {
-  // PDF-specific progress code
-} else if (format === 'excel') {
-  // Excel-specific progress code
-}
-
-// ✅ DO THIS (universal handling)
-onProgress: (p) => {
-  // Works for ALL formats!
-  updateUI(p.stage, p.message, p.entities, p.relationships)
-}
-```
-
----
-
-## 📝 Summary
-
-✅ **100% Standardized** - Same `ImportProgress` interface for all 7 formats
-✅ **Build Once** - Your progress UI works for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
-✅ **No Format Detection** - No need to check file type in your progress handler
-✅ **Human-Readable Messages** - Display `progress.message` directly, no parsing needed
-✅ **Standard Fields** - `stage`, `processed`, `total`, `entities`, `relationships` work everywhere
-✅ **Optional Enhancements** - `throughput`, `eta` available during extraction (all formats)
-
-**Developers can now build monitoring tools, dashboards, CLIs, and UIs that work perfectly for all import formats with zero format-specific code!**

docs/guides/storage-adapters.md

@@ -1,162 +0,0 @@
----
-title: Storage Adapters
-slug: guides/storage-adapters
-public: true
-category: guides
-template: guide
-order: 2
-description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both share one on-disk contract, including generational history and snapshots. Cloud backup is operator tooling, not a built-in adapter."
-next:
-  - guides/plugins
-  - concepts/consistency-model
----
-
-# Storage Adapters
-
-Brainy 8.0 ships **two storage adapters**:
-
-- **`FileSystemStorage`** — persistent on-disk storage. The default for any
-  deployment that needs to survive a restart. Runs on Node.js, Bun, and Deno.
-- **`MemoryStorage`** — in-memory only. The right choice for tests, ephemeral
-  workloads, and short-lived demos.
-
-Both implement the same `StorageAdapter` interface, support the full Db API
-(generational history, snapshots, restore — see the
-[consistency model](../concepts/consistency-model.md)), and use the same
-on-disk layout (memory's "disk" is a JS Map).
-
-## Quick start
-
-```ts
-import { Brainy } from '@soulcraft/brainy'
-
-// Filesystem (recommended for any persistent workload):
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: './brainy-data' }
-})
-
-// Memory (tests, ephemeral):
-const brainMem = new Brainy({ storage: { type: 'memory' } })
-
-// Auto-detect (filesystem on Node-like runtimes, memory in browsers):
-const brainAuto = new Brainy({ storage: { type: 'auto' } })
-```
-
-## When to use which
-
-| Use case | Adapter | Why |
-|---|---|---|
-| Production app | `filesystem` | Durable, snapshot-able, mmap-able |
-| Tests, CI | `memory` | No disk teardown; fast |
-| Short-lived data pipeline | `memory` | No persistence needed |
-| In-browser demo | `memory` | Filesystem unavailable in browsers |
-| Cloud deployment | `filesystem` on local disk + operator backup | See "Cloud backup" below |
-
-## Cloud backup — operator tooling, not a built-in
-
-Brainy 8.0 deliberately ships **no cloud storage adapters**. Cloud backup is
-handled at the operator layer with standard tooling, the same pattern every
-production database uses (Postgres, SQLite, Redis):
-
-```bash
-# After a brainy flush, sync the on-disk artefact to your cloud of choice.
-gsutil rsync -r /var/lib/brainy gs://my-backup-bucket/brainy/
-# or:
-aws s3 sync /var/lib/brainy s3://my-backup-bucket/brainy/
-# or:
-rclone sync /var/lib/brainy remote:brainy-backups/
-# or:
-azcopy sync /var/lib/brainy "https://account.blob.core.windows.net/brainy?sv=..."
-```
-
-Brainy's filesystem layout is sync-friendly:
-- Atomic writes (temp + rename) — readers never see torn files
-- Per-shard files — `rsync`-style incremental sync works well
-- Immutable generation records (`_generations/`) — append-only, cache-friendly
-
-For point-in-time backups, take a filesystem snapshot (ZFS, btrfs, LVM, EBS,
-etc.) or use `brain.now().persist(path)` to write a self-contained snapshot
-you can sync independently of the live brain — see
-[Snapshots & Time Travel](./snapshots-and-time-travel.md).
-
-## Why no cloud adapters in 8.0?
-
-Cloud storage adapters lived in Brainy 4.x-7.x. They were dropped in 8.0
-because:
-
-- Zero production consumers used them at scale — every known production
-  deployment ran on local filesystem.
-- Cloud-storage HNSW / DiskANN doesn't perform — vector indexes need
-  low-latency random reads that S3 / GCS / R2 / Azure can't provide
-  consistently.
-- Bundling cloud SDKs into the library cost ~3000-5000 LOC + 4-7 transitive
-  dependencies for a feature nobody used.
-- Cloud backup via operator tooling is strictly more reliable than in-app
-  upload (better retry semantics, better observability, better cost control).
-
-Brainy 8.0 is smaller, faster to install, and clearer about what it does.
-
-## Configuration
-
-```ts
-// BrainyConfig['storage'] — either a config object or a pre-constructed adapter:
-storage?:
-  | {
-      // The adapter type. Optional — a top-level `path` implies 'filesystem',
-      // so { path: '/data' } works without it. Defaults to 'auto'
-      // (filesystem on Node-like runtimes, memory otherwise).
-      type?: 'auto' | 'memory' | 'filesystem'
-
-      // CANONICAL directory for filesystem storage. The rest of the API already
-      // speaks `path` (persist(path), Brainy.load(path), asOf(path),
-      // restore(path)). Specifying it implies type: 'filesystem'. Passed through
-      // to storage factories, including plugin-provided ones.
-      path?: string
-
-      // REMOVED in 8.0 — passing `rootDirectory` THROWS. Use `path`.
-      rootDirectory?: string
-
-      // REMOVED in 8.0 — a nested `options.{path,rootDirectory}` THROWS. Use `path`.
-      options?: any
-    }
-  | StorageAdapter   // e.g. storage: new MemoryStorage()
-```
-
-The canonical — and only — key is the top-level **`path`**. The pre-8.0 aliases
-`rootDirectory`, `options.{path,rootDirectory}`, and
-`fileSystemStorage.{path,rootDirectory}` were **removed in 8.0**: passing one now
-throws with the exact rename (never a silent default that would misplace data on
-upgrade). Because `path` implies filesystem, `{ path: '/data' }` is a complete
-config; the `type` is optional.
-
-## Direct construction
-
-If you want to skip the factory:
-
-```ts
-import { FileSystemStorage, MemoryStorage } from '@soulcraft/brainy'
-
-const fsStorage = new FileSystemStorage('./brainy-data')
-const memStorage = new MemoryStorage()
-
-const brain = new Brainy({ storage: fsStorage })
-```
-
-## Migration from 7.x cloud adapters
-
-7.x consumers of `OPFSStorage`, `GcsStorage`, `R2Storage`, `S3CompatibleStorage`,
-or `AzureBlobStorage` need to migrate to `FileSystemStorage` plus operator
-backup tooling. The recipe:
-
-1. On the host running Brainy, mount a local disk (NVMe recommended). Cloud
-   providers all expose persistent local disks: GCP Persistent Disk, AWS EBS,
-   Azure Managed Disks.
-2. Set `storage: { type: 'filesystem', path: '/mnt/brainy-data' }`.
-3. Run your existing data import once into the new local store.
-4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` /
-   `azcopy` on a cron — hourly or whatever your RPO requires. Point it at
-   the brainy data dir.
-5. For point-in-time backups, use filesystem snapshots or
-   `brain.now().persist(path)`.
-
-Same data, same APIs, no library-side cloud code.

docs/guides/streaming-imports.md

@@ -1,395 +0,0 @@
-# 🌊 Streaming Imports
-
-> **All imports stream by default - query data as it's imported**
-
-Brainy imports always use streaming architecture with progressive index flushing, enabling you to query data while it's being imported.
-
----
-
-## How It Works
-
-Every import streams with adaptive flush intervals:
-
-```typescript
-await brain.import(file, {
-  onProgress: async (progress) => {
-    // Query data during import
-    if (progress.queryable) {
-      const products = await brain.find({ type: 'product', limit: 1000 })
-      console.log(`${products.length} products imported so far...`)
-    }
-  }
-})
-```
-
-**Benefits**:
-- ✅ **Progressive queries**: Data queryable as import proceeds
-- ✅ **Crash resilient**: Partial imports survive server restarts
-- ✅ **Live monitoring**: Real-time progress with actual data counts
-- ✅ **Zero configuration**: Works optimally out of the box
-
----
-
-## Progressive Flush Intervals
-
-Brainy automatically adjusts flush intervals **as the import progresses**, based on current entity count:
-
-| Current Count | Flush Interval | Reason |
-|---------------|----------------|--------|
-| 0-999 entities | Every 100 | Frequent early updates for better UX |
-| 1K-9.9K entities | Every 1000 | Balanced performance/responsiveness |
-| 10K+ entities | Every 5000 | Performance focused, minimal overhead |
-
-**Example**: Importing 5,000 entities
-- Flushes at: 100, 200, ..., 900 (9 flushes with interval=100)
-- At entity #1000: Interval adjusts to 1000
-- Flushes at: 1000, 2000, 3000, 4000, 5000 (5 more flushes)
-- Total flushes: 14
-- Overhead: ~700ms (~0.14% of import time for 5K entities)
-
-**Why Progressive?**
-- ✅ Works with known totals (file imports)
-- ✅ Works with unknown totals (streaming APIs, database cursors)
-- ✅ Adapts automatically as import grows
-- ✅ No configuration needed
-
-### 🎯 Engineering Insight: Why This Is Advanced
-
-Most import systems use either:
-1. **Fixed intervals** (simple but inefficient for large imports)
-2. **Adaptive intervals** (efficient but requires knowing total count upfront)
-
-Brainy uses **progressive intervals** which combine the best of both:
-
-```typescript
-// Traditional approach (requires total count)
-const interval = total < 1000 ? 100 : (total < 10000 ? 1000 : 5000)
-
-// Brainy's approach (works with unknown totals)
-const interval = getProgressiveInterval(currentCount)
-// Adjusts dynamically: 100 → 1000 → 5000 as import grows
-```
-
-**Real-World Impact**:
-- **Known totals** (files): Optimal performance automatically
-- **Unknown totals** (APIs): Still works perfectly - adjusts on the fly
-- **Growing datasets**: UX-focused early (frequent updates), performance-focused later
-- **Zero overhead** decisions: Algorithm adapts, developer configures nothing
-
-This makes Brainy the **only import system** that:
-- ✅ Optimizes automatically without configuration
-- ✅ Works for both batch and streaming scenarios
-- ✅ Balances UX and performance dynamically
-- ✅ Scales from 10 to 10 million entities seamlessly
-
----
-
-## Architecture
-
-```
-Import Process (Always Streaming)
-├─ For each entity:
-│  ├─ Extract from source
-│  ├─ Classify type (SmartExtractor)
-│  ├─ Write to storage ← IMMEDIATE
-│  ├─ Update in-memory indexes
-│  └─ entitiesSinceFlush++
-│
-├─ When entitiesSinceFlush >= interval:
-│  ├─ brain.flush() ← Write indexes to disk
-│  ├─ onProgress({ queryable: true })
-│  └─ entitiesSinceFlush = 0
-│
-└─ Final flush at end
-```
-
-### Key Insight
-
-Entities write to storage **immediately** on creation. Flushing only writes the search indexes:
-
-- **Metadata Index** → Fast filtering by type, fields
-- **Graph Adjacency Index** → Fast relationship traversal
-- **Storage Counts** → Type statistics
-
-**Without flush**: Entities exist but queries are slow (full table scans)
-**With periodic flush**: Entities exist AND queries are fast (index lookups)
-
----
-
-## Use Cases
-
-### Use Case 1: Live Import Dashboard
-
-Show real-time progress with queryable data:
-
-```typescript
-const stats = {
-  total: 0,
-  byType: {} as Record<string, number>
-}
-
-await brain.import(largeCSV, {
-  onProgress: async (progress) => {
-    stats.total = progress.entities || 0
-
-    // Only query after flush
-    if (progress.queryable) {
-      const products = await brain.find({ type: 'product', limit: 10000 })
-      const people = await brain.find({ type: 'person', limit: 10000 })
-
-      stats.byType = {
-        product: products.length,
-        person: people.length
-      }
-
-      // Update UI
-      websocket.send({ stage: progress.stage, stats })
-    }
-  }
-})
-```
-
-**Output**:
-```
-Importing products.csv...
-━━━━━━━━━━━━━━━━░░░░░░░░░░ 60%
-  Products: 12,453
-  People: 2,871
-  Queryable: ✅
-```
-
----
-
-### Use Case 2: Progress Bar with Live Counts
-
-```typescript
-import { ProgressBar } from 'cli-progress'
-
-const progressBar = new ProgressBar.SingleBar({
-  format: 'Importing |{bar}| {percentage}% | {stats}'
-})
-
-await brain.import(file, {
-  onProgress: async (progress) => {
-    if (progress.stage === 'storing-graph' && progress.total) {
-      if (!progressBar.getProgress()) {
-        progressBar.start(progress.total, 0, { stats: '' })
-      }
-
-      let stats = `${progress.entities || 0} entities`
-
-      // Add queryable count after flush
-      if (progress.queryable) {
-        const all = await brain.find({ limit: 100000 })
-        stats += ` (${all.length} queryable)`
-      }
-
-      progressBar.update(progress.processed || 0, { stats })
-    }
-
-    if (progress.stage === 'complete') {
-      progressBar.stop()
-    }
-  }
-})
-```
-
----
-
-### Use Case 3: Conditional Processing
-
-Make decisions during import based on imported data:
-
-```typescript
-let shouldImportPricing = false
-
-await brain.import(catalogCSV, {
-  onProgress: async (progress) => {
-    if (progress.queryable && progress.processed! > 1000) {
-      // Check if we have enough products
-      const products = await brain.find({ type: 'product', limit: 20000 })
-
-      if (products.length > 10000 && !shouldImportPricing) {
-        console.log(`Found ${products.length} products - will import pricing next`)
-        shouldImportPricing = true
-      }
-    }
-  }
-})
-
-// Conditionally import related data
-if (shouldImportPricing) {
-  await brain.import(pricingCSV)
-}
-```
-
----
-
-## Performance
-
-### Benchmarks
-
-| Import Size | Total Time | Flush Overhead | % Overhead |
-|-------------|------------|----------------|------------|
-| 1K entities | 1.5s | +5ms | 0.3% |
-| 10K entities | 15s | +50ms | 0.3% |
-| 100K entities | 150s | +500ms | 0.3% |
-| 1M entities | 1500s | +5s | 0.3% |
-
-**Conclusion**: Streaming overhead is negligible (~0.3%) for the benefits gained.
-
-### Performance Tips
-
-**1. Limit Query Results**
-
-```typescript
-// ❌ Bad: Fetch all entities (slow for large imports)
-onProgress: async (p) => {
-  if (p.queryable) {
-    const all = await brain.find({})  // Could be 100K+ entities!
-  }
-}
-
-// ✅ Good: Limit results or query specific types
-onProgress: async (p) => {
-  if (p.queryable) {
-    const count = await brain.find({ type: 'product', limit: 10000 }).then(r => r.length)
-  }
-}
-```
-
-**2. Only Query When Needed**
-
-```typescript
-// ❌ Bad: Query on every progress event
-onProgress: async (p) => {
-  const all = await brain.find({ limit: 10000 })  // Runs 100+ times!
-}
-
-// ✅ Good: Only query after flush
-onProgress: async (p) => {
-  if (p.queryable) {
-    const all = await brain.find({ limit: 10000 })  // Runs ~10 times
-  }
-}
-```
-
-**3. Disable Features You Don't Need**
-
-```typescript
-await brain.import(file, {
-  enableNeuralExtraction: false,      // 10x faster
-  enableRelationshipInference: false, // 5x faster
-  enableConceptExtraction: false      // 2x faster
-})
-```
-
----
-
-## API Reference
-
-### ImportProgress
-
-```typescript
-interface ImportProgress {
-  stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
-  message: string
-  processed?: number              // Current item number
-  total?: number                  // Total items
-  entities?: number               // Entities extracted so far
-  relationships?: number          // Relationships inferred so far
-
-  /**
-   * Whether data is queryable
-   *
-   * true  = Indexes flushed, queries will be fast and complete
-   * false/undefined = Data in storage but indexes not flushed yet
-   */
-  queryable?: boolean
-}
-```
-
-### brain.flush()
-
-Manually flush indexes to disk:
-
-```typescript
-// Add many entities
-for (const entity of entities) {
-  await brain.add(entity)
-}
-
-// Flush indexes to make queryable
-await brain.flush()
-
-// Now queries will be fast
-const results = await brain.find({ type: 'product', limit: 1000 })
-```
-
-**Performance**: ~5-50ms per flush (depends on index size)
-
-**What Gets Flushed**:
-- Metadata index (field indexes + EntityIdMapper)
-- Graph adjacency index (relationship cache)
-- Storage adapter counts (type statistics)
-
-**What Doesn't Get Flushed** (already persisted):
-- Entities (written immediately on `add()`)
-- Relationships (written immediately on `relate()`)
-
----
-
-## Troubleshooting
-
-### Q: Queries during import are slow
-
-**A:** Only query when `queryable === true`:
-
-```typescript
-onProgress: async (p) => {
-  // ✅ Good
-  if (p.queryable) {
-    const results = await brain.find({ type: 'product', limit: 1000 })
-  }
-
-  // ❌ Bad - queries before flush are slow
-  const results = await brain.find({ type: 'product', limit: 1000 })
-}
-```
-
-### Q: How often does data flush?
-
-**A:** Progressively adjusts based on current entity count:
-- 0-999 entities: Every 100 entities
-- 1K-9.9K: Every 1000 entities
-- 10K+: Every 5000 entities
-
-The interval increases automatically as more data is imported. Check console output to see when intervals adjust.
-
----
-
-## Migration from v3.x/v4.0/v4.1
-
-No changes required! Streaming is now always enabled with optimal defaults:
-
-```typescript
-// Before (v3.x, v4.0, v4.1): Works the same
-await brain.import(file)
-
-// After: Streaming always on, zero config
-await brain.import(file)
-```
-
-The `flushInterval` option has been removed in favor of automatic progressive intervals that adjust dynamically as the import proceeds.
-
----
-
-## Further Reading
-
-- [Import Flow Guide](./import-flow.md) - Complete import pipeline explanation
-- [Import Quick Reference](./import-quick-reference.md) - API cheat sheet
-- [VFS Guide](./vfs-guide.md) - Virtual file system organization
-
----
-
-**Questions?** Check the [FAQ](../faq.md) or [open an issue](https://github.com/soulcraft/brainy/issues)!

docs/guides/subtypes-and-facets.md

@@ -1,575 +0,0 @@
----
-title: Subtypes & Facets
-slug: guides/subtypes-and-facets
-public: true
-category: guides
-template: guide
-order: 7
-description: Use the top-level `subtype` field to sub-classify entities within a NounType, track other metadata facets like status or role, and migrate field names without downtime.
-next:
-  - guides/aggregation
-  - api/reference
----
-
-# Subtypes & Facets
-
-> Sub-classify entities within a NounType, track arbitrary metadata facets, and migrate field names without downtime.
-
-## Why this exists
-
-Brainy's `NounType` (Person, Document, Event, Concept, Task, …) is a stable, flat 42-type taxonomy. It's deliberately coarse — every product needs a way to further classify entities *within* a type. Is this `Person` an employee or a customer? Is this `Document` an invoice or a contract? Is this `Event` a meeting or a milestone?
-
-Three layers solve this:
-
-| Layer | Use it for | Shape |
-|---|---|---|
-| **`subtype`** | The primary sub-classification of an entity, one value per entity | Top-level standard field |
-| **`trackField()`** | Other facets you want to count or filter on (`status`, `source`, `role`, `paradigm`) | Registered metadata field |
-| **`migrateField()`** | Renaming or restructuring fields across an existing dataset | One-shot stream-and-rewrite |
-
-## Layer 1 — `subtype`
-
-`subtype` is a top-level standard field on every entity, alongside `type` / `confidence` / `weight`. Flat string, no hierarchy. The vocabulary is your choice — Brainy stores and counts, never validates.
-
-### Write
-
-```typescript
-import { Brainy, NounType } from '@soulcraft/brainy'
-
-const brain = new Brainy()
-await brain.init()
-
-await brain.add({
-  data: 'Avery Brooks — runs the AI lab',
-  type: NounType.Person,
-  subtype: 'employee',                  // top-level write param
-  metadata: { department: 'ai-lab' }
-})
-```
-
-### Read
-
-```typescript
-// Top-level filter — standard-field fast path, no `where` wrapper:
-const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
-
-// Set membership:
-const internal = await brain.find({
-  type: NounType.Person,
-  subtype: ['employee', 'contractor']
-})
-
-// Operator-form predicates use `where`:
-const typed = await brain.find({
-  type: NounType.Person,
-  where: { subtype: { exists: true } }
-})
-```
-
-### What it looks like
-
-```json
-{
-  "id": "01HZK3M7TW...",
-  "type": "person",
-  "subtype": "employee",
-  "data": "Avery Brooks — runs the AI lab",
-  "metadata": { "department": "ai-lab" }
-}
-```
-
-`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's what makes the fast path possible: queries on `subtype` hit the column-store index directly, never the metadata fallback.
-
-### Counts (O(1))
-
-```typescript
-// All subtypes for a NounType
-brain.counts.bySubtype(NounType.Person)
-// → { employee: 12, customer: 847, vendor: 34 }
-
-// Point count
-brain.counts.bySubtype(NounType.Person, 'employee')
-// → 12
-
-// Top N
-brain.counts.topSubtypes(NounType.Person, 3)
-// → [['customer', 847], ['employee', 12], ['vendor', 34]]
-
-// Distinct subtypes for a NounType
-brain.subtypesOf(NounType.Person)
-// → ['customer', 'employee', 'vendor']
-```
-
-These are O(1) lookups backed by `_system/subtype-statistics.json` — no scan, no storage round-trip. The rollup is incrementally maintained as entities are added, updated, and deleted.
-
-### Aggregation
-
-`subtype` is a first-class group-by dimension:
-
-```typescript
-const rows = await brain.find({
-  aggregate: {
-    groupBy: ['type', 'subtype'],
-    metrics: { count: { op: 'COUNT' } }
-  }
-})
-// [
-//   { groupKey: { type: 'person',   subtype: 'customer' }, count: 847 },
-//   { groupKey: { type: 'person',   subtype: 'employee' }, count: 12 },
-//   { groupKey: { type: 'document', subtype: 'invoice'  }, count: 2103 },
-//   ...
-// ]
-```
-
-## Layer 2 — `trackField()`
-
-Some metadata fields aren't *the* sub-classification of an entity but are still worth counting: `status`, `source`, `role`, `paradigm`. Promoting each one to top-level would clutter the contract. `trackField()` registers a metadata field for cardinality + per-NounType breakdown stats without the contract growth.
-
-### Register and query
-
-```typescript
-// Track a single facet
-brain.trackField('status')
-
-await brain.add({ data: 'Ship subtype', type: NounType.Task, metadata: { status: 'todo' } })
-await brain.add({ data: 'Write docs',   type: NounType.Task, metadata: { status: 'done' } })
-
-await brain.counts.byField('status')
-// → { todo: 1, done: 1 }
-```
-
-### Per-NounType breakdown
-
-```typescript
-brain.trackField('status', { perType: true })
-
-await brain.counts.byField('status', { type: NounType.Task })
-// → { todo: 1, done: 1 }
-```
-
-### Vocabulary whitelist (opt-in validation)
-
-```typescript
-brain.trackField('priority', { values: ['low', 'medium', 'high'] })
-
-// Throws — 'urgent' isn't in the vocabulary:
-await brain.add({
-  data: 'Fix bug',
-  type: NounType.Task,
-  metadata: { priority: 'urgent' }
-})
-```
-
-`trackField` piggybacks on the existing aggregation engine (see the [Aggregation guide](./aggregation.md) for the underlying mechanism). Backfill-on-define means the first call to `counts.byField()` scans existing entities; subsequent calls are O(groups).
-
-### subtype vs trackField — when to use which
-
-- **`subtype`** when there's one primary sub-classification per entity. Limit yourself to one per NounType. Examples: Person→employee/customer/vendor; Document→invoice/contract/policy.
-- **`trackField`** for anything else you want to count or filter on. No limit on how many you register. Examples: status, source, role, paradigm.
-
-## Layer 3 — `migrateField()`
-
-Use this when you need to rename or restructure a field across an entire dataset — for example, moving a `metadata.kind` convention up to the top-level `subtype` standard field.
-
-### One-shot rewrite
-
-```typescript
-// Starting state: every entity has metadata.kind
-const result = await brain.migrateField({
-  from: 'metadata.kind',
-  to: 'subtype'
-})
-
-console.log(result)
-// {
-//   scanned: 1500,
-//   migrated: 1500,
-//   skipped: 0,
-//   errors: []
-// }
-```
-
-After this returns, every entity has `subtype` populated from the old `metadata.kind` value, and `metadata.kind` is cleared.
-
-### Deprecation window — keep both fields readable
-
-When you can't coordinate all readers and the migration in a single deploy, use `readBoth: true` to preserve the source field alongside the new one:
-
-```typescript
-// Phase 1: dual-populate (existing readers still work against metadata.kind):
-await brain.migrateField({
-  from: 'metadata.kind',
-  to: 'subtype',
-  readBoth: true
-})
-
-// ... readers migrate to query subtype at their own pace ...
-
-// Phase 2: clear the source field when ready:
-await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
-```
-
-### Supported paths
-
-| Path form | Refers to |
-|---|---|
-| `'subtype'`, `'type'`, `'confidence'` | Top-level standard fields |
-| `'metadata.X'` | A key under `entity.metadata` |
-| `'data.X'` | A key under `entity.data` (when `data` is an object) |
-| `'X'` (bare, non-standard) | Shorthand for `metadata.X` |
-
-### Idempotent
-
-`migrateField` is safe to re-run. Entities where the source is absent, or where the destination already holds the same value, are skipped. This makes it safe to use in a deploy-once-then-cleanup workflow.
-
-### Progress reporting
-
-```typescript
-await brain.migrateField({
-  from: 'metadata.kind',
-  to: 'subtype',
-  batchSize: 500,
-  onProgress: ({ scanned, migrated }) => {
-    console.log(`${scanned} scanned, ${migrated} migrated`)
-  }
-})
-```
-
-## Putting it together
-
-A realistic adoption sequence for a brain that started without these primitives:
-
-```typescript
-import { Brainy, NounType } from '@soulcraft/brainy'
-
-const brain = new Brainy({ storage: { type: 'filesystem', path: './brain-data' } })
-await brain.init()
-
-// 1. Migrate any existing metadata.kind convention to the new top-level subtype
-await brain.migrateField({ from: 'metadata.kind', to: 'subtype', readBoth: true })
-
-// 2. Register the other facets you want counted
-brain.trackField('status', { perType: true })
-brain.trackField('source')
-
-// 3. Use subtype on every new write
-await brain.add({
-  data: 'Quarterly review',
-  type: NounType.Event,
-  subtype: 'milestone',
-  metadata: { status: 'todo', source: 'planning-session' }
-})
-
-// 4. Query the breakdowns
-brain.counts.bySubtype(NounType.Event)
-// → { milestone: 14, meeting: 203, deadline: 7 }
-
-await brain.counts.byField('status', { type: NounType.Event })
-// → { todo: 12, done: 212 }
-
-// 5. Once all readers are on the new field, drop the source:
-await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
-```
-
-## Layer V — `subtype` on relationships (verbs)
-
-Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive: a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field, indexed on the fast path.
-
-### Write
-
-```typescript
-const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' })
-const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' })
-const matrixId = await brain.add({ type: NounType.Person, subtype: 'contractor', data: 'Sam' })
-
-await brain.relate({
-  from: ceoId,
-  to: vpId,
-  type: VerbType.ReportsTo,
-  subtype: 'direct'
-})
-
-await brain.relate({
-  from: ceoId,
-  to: matrixId,
-  type: VerbType.ReportsTo,
-  subtype: 'dotted-line'
-})
-```
-
-### Read & filter
-
-```typescript
-// Direct reports — fast path filter (column-store hit, not metadata fallback)
-const direct = await brain.related({
-  from: ceoId,
-  type: VerbType.ReportsTo,
-  subtype: 'direct'
-})
-
-// Set membership
-const allReports = await brain.related({
-  from: ceoId,
-  type: VerbType.ReportsTo,
-  subtype: ['direct', 'dotted-line']
-})
-```
-
-### Update (new — `updateRelation()`)
-
-Verbs previously had no update method — the only way to change a relationship was delete-then-recreate. 7.30 closes that gap:
-
-```typescript
-// Promote a dotted-line report to direct without losing the edge id
-await brain.updateRelation({ id: relationId, subtype: 'direct' })
-
-// Or change weight/confidence
-await brain.updateRelation({ id: relationId, weight: 0.5, confidence: 0.9 })
-```
-
-### Traversal filter
-
-`find({ connected, subtype })` filters traversal edges by their subtype. Composes with `via` (verb-type filter):
-
-```typescript
-// All direct reports two hops deep (will be supported in Cor; for now depth-1)
-const directChain = await brain.find({
-  connected: {
-    from: ceoId,
-    via: VerbType.ReportsTo,
-    subtype: 'direct',
-    depth: 1
-  }
-})
-```
-
-Multi-hop subtype filtering (`depth > 1`) lights up on the Cor native path; the JS path throws today rather than return incorrect partial results.
-
-### Counts
-
-Same shape as the noun-side counts API:
-
-```typescript
-brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
-// → { direct: 12, 'dotted-line': 3 }
-
-brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')   // O(1) point
-// → 12
-
-brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
-// → [['direct', 12], ['dotted-line', 3]]
-
-brain.relationshipSubtypesOf(VerbType.ReportsTo)
-// → ['direct', 'dotted-line']
-```
-
-These are O(1) lookups backed by the persisted `_system/verb-subtype-statistics.json` rollup — same self-heal machinery as the noun-side rollup.
-
-### Migrate verb fields
-
-`migrateField()` now walks verbs too:
-
-```typescript
-// Migrate verb-side metadata.kind → top-level subtype
-await brain.migrateField({
-  from: 'metadata.kind',
-  to: 'subtype',
-  entityKind: 'verb'
-})
-
-// Or migrate both nouns and verbs in one pass
-await brain.migrateField({
-  from: 'metadata.kind',
-  to: 'subtype',
-  entityKind: 'both'
-})
-```
-
-Default is `entityKind: 'noun'` (backward-compatible with 7.29).
-
-## Enforcement — `requireSubtype()` + brain-wide strict mode
-
-By default (7.30) subtype is optional. Two complementary opt-in mechanisms let you enforce the pairing of type + subtype on every write:
-
-### Per-type registration
-
-Mark a specific `NounType` or `VerbType` as requiring a subtype, optionally with a fixed vocabulary:
-
-```typescript
-brain.requireSubtype(NounType.Person, {
-  values: ['employee', 'customer', 'vendor'],
-  required: true
-})
-
-brain.requireSubtype(VerbType.ReportsTo, {
-  values: ['direct', 'dotted-line'],
-  required: true
-})
-
-// Now this throws — Person requires a subtype:
-await brain.add({ type: NounType.Person, data: 'no subtype' })
-
-// And this throws — 'matrix' isn't in the registered vocabulary:
-await brain.relate({
-  from: a,
-  to: b,
-  type: VerbType.ReportsTo,
-  subtype: 'matrix'
-})
-```
-
-### Brain-wide strict mode
-
-Enforce on every write across the whole brain:
-
-```typescript
-const brain = new Brainy({ requireSubtype: true })
-
-// Allow specific types to omit subtype (e.g. catch-all `Thing`)
-const brain2 = new Brainy({
-  requireSubtype: { except: [NounType.Thing, NounType.Custom] }
-})
-```
-
-When strict mode is on:
-
-- Every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()` rejects writes missing a subtype on a non-exempt type.
-- `addMany()` and `relateMany()` validate every item BEFORE any storage write — atomic-fail semantics, no partial writes.
-- Per-type rules registered via `requireSubtype()` compose with the brain-wide flag; specific rules win when both apply.
-- Brainy's own internal writes (VFS root, VFS directories, VFS file entities) bypass enforcement via the `metadata.isVFSEntity: true` infrastructure marker.
-
-### Migrating to strict mode on an existing brain
-
-`brain.migrateField()` is your friend — populate `subtype` from an existing convention before flipping strict mode on:
-
-```typescript
-// Step 1: backfill subtype from your existing metadata.kind convention
-await brain.migrateField({
-  from: 'metadata.kind',
-  to: 'subtype',
-  entityKind: 'both'
-})
-
-// Step 2: register the vocabulary
-brain.requireSubtype(NounType.Person, {
-  values: ['employee', 'customer', 'vendor'],
-  required: true
-})
-
-// Step 3: future writes must have a subtype matching the vocabulary
-await brain.add({ type: NounType.Person, subtype: 'employee', data: '...' })
-```
-
-## Strict mode in practice (for SDK-style vocabulary consumers)
-
-When a platform layer like the Soulcraft SDK registers `requireSubtype()` rules on behalf of every consumer's brain, every downstream product that calls `brain.add()` / `brain.relate()` against those types must pass a matching `subtype`. Skipping the field — or passing one outside the registered vocabulary — throws at the boundary.
-
-This pattern is powerful but surfaces a class of latent bug: any `brain.add()` call site that was written before strict-mode adoption starts rejecting writes. A representative production incident: a booking flow started returning 500 on every request because a service method called `brain.add({ type: NounType.Event, ... })` without subtype, and an SDK layer had just registered `requireSubtype()` for `NounType.Event` on every brain instance.
-
-The fix is a four-step migration recipe — and Brainy 7.30.1+ ships diagnostic tools to make it deterministic.
-
-### Migration recipe
-
-1. **Inventory the gap with `brain.audit()`** — returns the deterministic list of which NounTypes and VerbTypes have entities/relationships missing subtype, grouped by type:
-
-   ```typescript
-   const report = await brain.audit()
-   //   {
-   //     entitiesWithoutSubtype: { event: 24, document: 3, ... },
-   //     relationshipsWithoutSubtype: { relatedTo: 1402 },
-   //     total: 1429,
-   //     scanned: 8400,
-   //     recommendation: 'Found 1429 entries without subtype. ...'
-   //   }
-   ```
-
-   By default, VFS infrastructure entities are excluded (they bypass enforcement anyway via the `metadata.isVFSEntity` marker). Pass `{ includeVFS: true }` to surface them too.
-
-2. **Bulk-migrate any existing convention** with `brain.migrateField()` if a legacy field can be lifted:
-
-   ```typescript
-   // Common pattern: subtype mirrors a discriminator field already in metadata
-   await brain.migrateField({
-     from: 'metadata.entityType',
-     to: 'subtype',
-     readBoth: true  // safety: keep the source field readable during cutover
-   })
-   ```
-
-3. **Hand-fix the remaining call sites.** The exact list is in `report.entitiesWithoutSubtype`. For each call site, add `subtype: '<value>'` to the `brain.add()` / `brain.relate()` params. Choose a stable convention (e.g. mirror `metadata.entityType` if you have one; any rule that's deterministic from the data works).
-
-4. **Verify with `brain.audit()` again.** Re-run; total should be `0`. If you turn on brain-wide strict mode at this point, all future writes are protected.
-
-### Brainy's own infrastructure subtype labels (reference)
-
-Brainy's internal write paths set subtype on every entity and edge they create. Consumers don't need to do anything for these — they're documented here so you understand the data shape:
-
-| Code path | NounType / VerbType | Subtype label |
-|---|---|---|
-| VFS root directory `/` | `NounType.Collection` | `'vfs-root'` |
-| VFS subdirectories | `NounType.Collection` | `'vfs-directory'` |
-| VFS files | (mime-driven, e.g. `Document`/`Code`/`Image`) | `'vfs-file'` |
-| VFS symlinks | `NounType.File` | `'vfs-symlink'` |
-| VFS Contains edges | `VerbType.Contains` | `'vfs-contains'` |
-| Aggregation materialized output | `NounType.Measurement` | `'materialized-aggregate'` |
-| Import-document provenance entity | `NounType.Document` | `'import-source'` |
-| Importer-extracted entities (no caller default) | extractor-driven | `'imported'` |
-| Importer placeholder targets | `NounType.Thing` | `'import-placeholder'` |
-| Neural extraction (no caller default) | extractor-driven | `'extracted'` |
-| GoogleSheets API entity writes | request-driven | `'imported-from-sheets'` |
-| OData API entity writes | request-driven | `'imported-from-odata'` |
-| MCP client message storage | `NounType.Message` | `'mcp-message'` |
-| `brainy add` CLI (no `--subtype` flag) | user-supplied type | `'cli-add'` |
-| `brainy relate` CLI (no `--subtype` flag) | user-supplied verb | `'cli-relate'` |
-
-You can query these directly: `await brain.find({ subtype: 'vfs-file' })` returns every VFS-managed file regardless of NounType. `await brain.counts.bySubtype(NounType.Document)` shows you the import-source / imported / extracted / vfs-file breakdown.
-
-Importer and extraction paths accept a caller-supplied `defaultSubtype` option so you can tag a whole batch with your own provenance label (e.g. `'customer-upload-2026q2'`) instead of the Brainy default `'imported'` / `'extracted'`.
-
-### Looking ahead — Brainy 8.0
-
-Brainy 8.0 ships:
-
-- **`brain.fillSubtypes(rules)`** — the bulk migration helper that pairs with `audit()`. Given caller-supplied rules per NounType / VerbType, it walks the brain and fills in missing subtypes via `update()`. Pre-8.0 brains run this once before upgrading to clear migration debt.
-- **`subtype: string` (non-optional)** on `AddParams<T>` and `RelateParams<T>`. TypeScript catches missing subtype at compile time, not just runtime.
-- **`new Brainy({ requireSubtype: true })` becomes the default.** Consumers explicitly opt out with `{ requireSubtype: false }` during migration.
-
-7.30.1's `audit()` is the diagnostic; 8.0's `fillSubtypes()` is the bulk fixer. Together they close the migration gap deterministically.
-
-## Reference
-
-### Layer 1 — `subtype` (nouns)
-
-- `brain.add({ ..., subtype: 'value' })` — write the field
-- `brain.update({ id, subtype: 'value' })` — change the field
-- `brain.find({ type, subtype })` — filter (fast path)
-- `brain.find({ subtype: ['a', 'b'] })` — set membership
-- `brain.counts.bySubtype(type, subtype?)` — O(1) counts
-- `brain.counts.topSubtypes(type, n?)` — top N by count
-- `brain.subtypesOf(type)` — distinct subtype list
-
-### Layer V — `subtype` (verbs / relationships)
-
-- `brain.relate({ ..., subtype: 'value' })` — write the field
-- `brain.updateRelation({ id, subtype, type?, weight?, ... })` — change a relationship
-- `brain.related({ subtype })` — filter (fast path)
-- `brain.related({ subtype: ['a', 'b'] })` — set membership
-- `brain.find({ connected: { via, subtype, depth } })` — traversal filter
-- `brain.counts.byRelationshipSubtype(verb, subtype?)` — O(1) counts
-- `brain.counts.topRelationshipSubtypes(verb, n?)` — top N by count
-- `brain.relationshipSubtypesOf(verb)` — distinct subtype list
-
-### Layer 2 — generic facets
-
-- `brain.trackField(name, { perType?, values? })` — register a facet
-- `brain.counts.byField(name, { type? })` — facet counts
-
-### Layer 3 — migration
-
-- `brain.migrateField({ from, to, readBoth?, batchSize?, onProgress?, entityKind? })` — rewrite a field (nouns, verbs, or both)
-
-### Enforcement
-
-- `brain.requireSubtype(type, { values?, required })` — per-`NounType` / `VerbType` rule
-- `new Brainy({ requireSubtype: true })` — brain-wide strict mode
-- `new Brainy({ requireSubtype: { except: [type, ...] } })` — strict with exemptions