Compare commits
No commits in common. "main" and "v7.9.1" have entirely different histories.
780 changed files with 174973 additions and 88379 deletions
|
|
@ -1,170 +0,0 @@
|
|||
# Brainy Architecture Reference
|
||||
|
||||
## What Is Brainy
|
||||
|
||||
@soulcraft/brainy (v7.17.0) is a Universal Knowledge Protocol -- a Triple Intelligence database combining vector search, graph traversal, and metadata filtering in a single library. Published to npm as a public MIT-licensed package.
|
||||
|
||||
## Core Architecture
|
||||
|
||||
### Storage Layer (`src/storage/`)
|
||||
- **StorageAdapter interface** (`src/coreTypes.ts:576`): The contract ALL storage backends implement. ALWAYS check this interface before adding storage methods.
|
||||
- **BaseStorage** (`src/storage/baseStorage.ts`): Base implementation with built-in type-aware partitioning (TypeAwareStorageAdapter was removed -- functionality merged into BaseStorage).
|
||||
- **Adapters** (`src/storage/adapters/`):
|
||||
- `fileSystemStorage.ts` -- local filesystem
|
||||
- `memoryStorage.ts` -- in-memory
|
||||
- `baseStorageAdapter.ts` -- shared adapter base (counts, batch ops)
|
||||
- Cloud + OPFS adapters were removed in 8.0 (cloud backup is operator tooling)
|
||||
- **Generational MVCC / Db API** (`src/db/`): immutable `Db` values over generation-stamped records
|
||||
- `db.ts` (the `Db` value), `generationStore.ts` (record layer + commit protocol), `types.ts`, `errors.ts`, `whereMatcher.ts`
|
||||
- Design record: `docs/ADR-001-generational-mvcc.md`; replaced the pre-8.0 COW branching + versioning subsystems
|
||||
|
||||
### Vector Search (`src/hnsw/`)
|
||||
- `hnswIndex.ts` -- HNSW-based approximate nearest neighbor search
|
||||
- `typeAwareHNSWIndex.ts` -- type-partitioned vector search
|
||||
- NOT in `src/intelligence/` (that directory does not exist)
|
||||
|
||||
### Graph Engine (`src/graph/`)
|
||||
- `graphAdjacencyIndex.ts` -- adjacency-based graph representation
|
||||
- `pathfinding.ts` -- relationship traversal and pathfinding
|
||||
- `lsm/` -- LSM tree implementation for graph storage
|
||||
|
||||
### Metadata Index (`src/utils/metadataIndex.ts`)
|
||||
- O(1) exact match via hash indexes
|
||||
- O(log n) range queries via sorted indexes
|
||||
- Roaring bitmap set operations for efficient filtering
|
||||
- Adaptive chunking strategy (`metadataIndexChunking.ts`)
|
||||
- Caching layer (`metadataIndexCache.ts`)
|
||||
|
||||
### Triple Intelligence (`src/triple/`)
|
||||
- `TripleIntelligenceSystem.ts` -- combines vector + graph + metadata into unified queries
|
||||
- Lazy-loaded indexes (loaded on first use, not at startup)
|
||||
|
||||
### Neural/AI Components (`src/neural/`)
|
||||
- Smart Importers (`src/importers/`): CSV, Excel, PDF, DOCX, YAML, JSON, Markdown, Orchestrator
|
||||
- `SmartExtractor.ts` -- entity extraction from unstructured data
|
||||
- `SmartRelationshipExtractor.ts` -- relationship detection
|
||||
- `NeuralEntityExtractor.ts` -- ML-based entity recognition
|
||||
- Natural language processing utilities
|
||||
|
||||
### Distributed Systems (`src/distributed/`)
|
||||
- Distributed Coordinator for multi-node operation
|
||||
- Shard Manager for data partitioning
|
||||
- Cache Synchronization across nodes
|
||||
- Read/Write separation
|
||||
- Network and HTTP transport layers
|
||||
- Storage discovery and shard migration
|
||||
|
||||
### Transaction Management (`src/transaction/`)
|
||||
- TransactionManager for ACID operations
|
||||
- Operations: SaveNoun, AddToHNSW, UpdateMetadata, etc.
|
||||
- Distributed transaction support
|
||||
|
||||
### Integration Hub (`src/integrations/`)
|
||||
- Google Sheets integration
|
||||
- OData (Open Data Protocol)
|
||||
- Server-Sent Events (SSE)
|
||||
- Webhooks
|
||||
- Event bus system
|
||||
|
||||
### Virtual Filesystem (`src/vfs/`)
|
||||
- `VirtualFileSystem.ts` -- full VFS implementation (87 KB)
|
||||
- `PathResolver.ts`, `FSCompat.ts`, `MimeTypeDetector.ts`, `TreeUtils.ts`
|
||||
- Subdirectories: `semantic/` (semantic search), `streams/` (streaming), `importers/`
|
||||
|
||||
### MCP Support (`src/mcp/`)
|
||||
- BrainyMCPAdapter, MCPAugmentationToolset, BrainyMCPService
|
||||
- Model Control Protocol request/response handling
|
||||
|
||||
### Aggregation Engine (`src/aggregation/`)
|
||||
- **AggregationIndex** (`AggregationIndex.ts`): Write-time incremental aggregation — SUM, COUNT, AVG, MIN, MAX with GROUP BY and time windows
|
||||
- **Time Windows** (`timeWindows.ts`): ISO 8601 bucketing — hour, day, week, month, quarter, year, custom intervals
|
||||
- **Materializer** (`materializer.ts`): Debounced writes of aggregate results as `NounType.Measurement` entities
|
||||
- Integrates into `brain.find({ aggregate })` for unified query API
|
||||
- Write hooks in `add()`, `update()`, `delete()` for O(1) incremental updates
|
||||
- `'aggregation'` provider key enables native plugin acceleration
|
||||
|
||||
### Additional Systems
|
||||
- **CLI** (`src/cli/`): Complete command-line tool with interactive mode and catalog system
|
||||
- **Migration** (`src/migration/`): MigrationRunner for database schema migrations
|
||||
- **Embeddings** (`src/embeddings/`): Embedding manager with Candle-WASM Rust source
|
||||
- **Streaming** (`src/streaming/`): Pipeline support with adaptive backpressure
|
||||
- **Versioning** (`src/versioning/`): VersioningAPI for data versioning
|
||||
- **Plugin System**: Registry-based plugin architecture
|
||||
- **Patterns** (`src/patterns/`): 7 pattern library JSON files
|
||||
|
||||
## Type System
|
||||
- **NounType** (42 types, `src/types/graphTypes.ts:850-893`): Person, Organization, Concept, Collection, Document, Task, Project, etc.
|
||||
- **VerbType** (127 types, `src/types/graphTypes.ts:900-1087`): Contains, RelatedTo, PartOf, Creates, DependsOn, MemberOf, etc.
|
||||
- All types in `src/types/`
|
||||
|
||||
## Module Exports (`src/index.ts`)
|
||||
38+ named exports including: Brainy class, configuration types, neural APIs (NeuralImport, NeuralEntityExtractor, SmartExtractor, SmartRelationshipExtractor), distance functions, plugin system, migration system, embedding functions, storage adapters, COW infrastructure, pipeline utilities, graph types, MCP components, integration hub, OData utilities, and more.
|
||||
|
||||
## File Structure
|
||||
```
|
||||
src/
|
||||
├── index.ts # 38+ public exports
|
||||
├── brainy.ts # Main Brainy class (6,500+ lines)
|
||||
├── setup.ts # Initialization polyfills
|
||||
├── coreTypes.ts # StorageAdapter interface + core types
|
||||
├── storage/
|
||||
│ ├── baseStorage.ts # Base storage (includes type-aware)
|
||||
│ ├── adapters/ # All storage backends + cloud adapters
|
||||
│ └── cow/ # Copy-on-Write versioning
|
||||
├── hnsw/ # HNSW vector search
|
||||
├── graph/ # Graph engine + pathfinding + LSM
|
||||
├── triple/ # Triple Intelligence system
|
||||
├── neural/ # Smart extractors + NLP
|
||||
├── importers/ # File format importers (8 types)
|
||||
├── distributed/ # Distributed database (16 files)
|
||||
├── transaction/ # ACID transactions (6 files)
|
||||
├── integrations/ # Sheets, OData, SSE, Webhooks
|
||||
├── vfs/ # Virtual filesystem + semantic search
|
||||
├── mcp/ # Model Control Protocol
|
||||
├── cli/ # Command-line interface
|
||||
├── migration/ # Schema migrations
|
||||
├── embeddings/ # Embedding manager + Candle-WASM
|
||||
├── streaming/ # Pipeline + backpressure
|
||||
├── versioning/ # Versioning API
|
||||
├── types/ # TypeScript type definitions
|
||||
├── utils/ # Metadata index, logging, etc.
|
||||
├── config/ # Configuration system
|
||||
├── patterns/ # Pattern library
|
||||
├── api/ # API layer
|
||||
├── interfaces/ # Interface definitions
|
||||
├── shared/ # Shared utilities
|
||||
├── data/ # Data utilities
|
||||
├── errors/ # Error handling
|
||||
├── critical/ # Critical error handling
|
||||
├── universal/ # Universal utilities
|
||||
├── import/ # Import functionality
|
||||
└── scripts/ # Build scripts
|
||||
```
|
||||
|
||||
## Initialization
|
||||
`brainy.ts` `init()` method performs initialization cascade:
|
||||
1. Load plugins
|
||||
2. Initialize storage
|
||||
3. Enable COW (Copy-on-Write)
|
||||
4. Set up embeddings
|
||||
5. Initialize caches
|
||||
6. Set up graph indexes
|
||||
7. Initialize VFS
|
||||
8. Set up transaction manager
|
||||
9. Initialize distributed components (if enabled)
|
||||
|
||||
## Testing
|
||||
- Framework: Vitest
|
||||
- Run: `npm test`
|
||||
- Test directories:
|
||||
- `tests/unit/` -- unit tests
|
||||
- `tests/integration/` -- integration tests
|
||||
- `tests/benchmarks/` -- performance benchmarks (NOT tests/performance/)
|
||||
- `tests/comprehensive/` -- comprehensive test suites
|
||||
- `tests/api/` -- API tests
|
||||
- `tests/helpers/` -- test utilities
|
||||
|
||||
## Release
|
||||
- `npm run release:patch/minor/major` -- fully automated via `scripts/release.sh`
|
||||
- `npm run release:dry` -- preview without changes
|
||||
- Uses conventional commits for changelog generation
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
node:
|
||||
name: Node ${{ matrix.node-version }}
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
node-version: ['22', '24']
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ matrix.node-version }}
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- run: npm run test:unit
|
||||
|
||||
bun:
|
||||
name: Bun (latest)
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
cache: npm
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: latest
|
||||
- run: npm ci
|
||||
# test:bun imports the built dist/, so build first.
|
||||
- run: npm run build
|
||||
# Bun as a runtime is the supported Bun story (`bun add` / `bun run`).
|
||||
- run: npm run test:bun
|
||||
206
.github/workflows/ci.yml
vendored
Normal file
206
.github/workflows/ci.yml
vendored
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
env:
|
||||
NODE_VERSION: '20'
|
||||
RUST_VERSION: 'stable'
|
||||
|
||||
jobs:
|
||||
# Build and test TypeScript/JavaScript
|
||||
test-node:
|
||||
name: Test (Node.js)
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
cache: 'npm'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Build TypeScript
|
||||
run: npm run build
|
||||
|
||||
- name: Run unit tests
|
||||
run: npm run test:unit
|
||||
|
||||
- name: Run integration tests
|
||||
run: npm run test:integration
|
||||
|
||||
# Build and test with Bun
|
||||
test-bun:
|
||||
name: Test (Bun)
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Bun
|
||||
uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: latest
|
||||
|
||||
- name: Setup Node.js (for npm compatibility)
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
cache: 'npm'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Build TypeScript
|
||||
run: npm run build
|
||||
|
||||
- name: Run Bun tests
|
||||
run: npm run test:bun
|
||||
|
||||
# Build Candle WASM (if Rust source changed)
|
||||
build-candle-wasm:
|
||||
name: Build Candle WASM
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Rust
|
||||
uses: dtolnay/rust-action@stable
|
||||
with:
|
||||
toolchain: ${{ env.RUST_VERSION }}
|
||||
targets: wasm32-unknown-unknown
|
||||
|
||||
- name: Install wasm-pack
|
||||
run: cargo install wasm-pack
|
||||
|
||||
- name: Cache Cargo registry
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
~/.cargo/registry
|
||||
~/.cargo/git
|
||||
src/embeddings/candle-wasm/target
|
||||
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
|
||||
|
||||
- name: Build WASM
|
||||
run: |
|
||||
cd src/embeddings/candle-wasm
|
||||
wasm-pack build --target web --release
|
||||
|
||||
- name: Upload WASM artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: candle-wasm
|
||||
path: src/embeddings/wasm/pkg/
|
||||
retention-days: 7
|
||||
|
||||
# Test Bun compile (standalone binary)
|
||||
test-bun-compile:
|
||||
name: Test Bun Compile
|
||||
runs-on: ubuntu-latest
|
||||
needs: [build-candle-wasm]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Bun
|
||||
uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: latest
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
cache: 'npm'
|
||||
|
||||
- name: Download WASM artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: candle-wasm
|
||||
path: src/embeddings/wasm/pkg/
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Build TypeScript
|
||||
run: npm run build
|
||||
|
||||
- name: Test Bun compile
|
||||
run: |
|
||||
# Create a test script
|
||||
cat > /tmp/test-compile.ts << 'EOF'
|
||||
import { Brainy } from './dist/index.js'
|
||||
|
||||
const brainy = new Brainy()
|
||||
await brainy.init()
|
||||
console.log('Brainy initialized!')
|
||||
|
||||
const embedding = await brainy.embed('Hello world')
|
||||
console.log(`Embedding dimension: ${embedding.length}`)
|
||||
|
||||
if (embedding.length !== 384) {
|
||||
throw new Error('Expected 384-dimensional embedding')
|
||||
}
|
||||
|
||||
console.log('Test passed!')
|
||||
EOF
|
||||
|
||||
# Compile to standalone binary
|
||||
bun build --compile /tmp/test-compile.ts --outfile /tmp/brainy-test
|
||||
|
||||
# Run the compiled binary
|
||||
/tmp/brainy-test
|
||||
|
||||
# Lint and type check
|
||||
lint:
|
||||
name: Lint & Type Check
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
cache: 'npm'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Type check
|
||||
run: npm run typecheck
|
||||
|
||||
- name: Lint
|
||||
run: npm run lint
|
||||
|
||||
# License check
|
||||
license-check:
|
||||
name: License Check
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
cache: 'npm'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Check licenses
|
||||
run: |
|
||||
# Install license checker
|
||||
npm install -g license-checker
|
||||
|
||||
# Check for problematic licenses
|
||||
license-checker --production --excludePrivatePackages \
|
||||
--failOn 'GPL;LGPL;AGPL;SSPL' \
|
||||
--summary
|
||||
18
.gitignore
vendored
18
.gitignore
vendored
|
|
@ -31,9 +31,6 @@ coverage/
|
|||
# Test results
|
||||
tests/results/
|
||||
|
||||
# Filesystem test artifacts (created by integration tests)
|
||||
test-*/
|
||||
|
||||
# IDE files
|
||||
.vscode/
|
||||
.idea/
|
||||
|
|
@ -52,12 +49,14 @@ temp/
|
|||
|
||||
# Planning and instruction files
|
||||
plan.md
|
||||
CLAUDE.md
|
||||
|
||||
# Package files
|
||||
*.tgz
|
||||
|
||||
# Private/confidential files
|
||||
PLAN.md
|
||||
CLAUDE.md
|
||||
INTERNAL_NOTES.md
|
||||
TODO_PRIVATE.md
|
||||
*.tar.gz
|
||||
|
|
@ -79,6 +78,7 @@ models-cache/
|
|||
|
||||
# Development planning files (not for commit)
|
||||
PLAN.md
|
||||
CLAUDE.md
|
||||
|
||||
# Backup folders
|
||||
backup-*
|
||||
|
|
@ -93,14 +93,9 @@ docs/internal/
|
|||
# Rust/Cargo build artifacts
|
||||
src/embeddings/candle-wasm/target/
|
||||
src/embeddings/candle-wasm/Cargo.lock
|
||||
src/embeddings/wasm/pkg/
|
||||
|
||||
# Ignore the wasm-pack output dir's CONTENTS (note the `/*`, not `/`, so the
|
||||
# re-includes below can take effect — git cannot re-include a file whose parent
|
||||
# DIRECTORY is excluded). Keep the pre-built WASM committed: it ships in the npm
|
||||
# package anyway, it lets consumers + CI build without a Rust/wasm-pack toolchain,
|
||||
# and versioning it makes the shipped artifact reproducible (not "whatever the
|
||||
# maintainer last built").
|
||||
src/embeddings/wasm/pkg/*
|
||||
# But keep the pre-built WASM (committed for users without Rust)
|
||||
!src/embeddings/wasm/pkg/*.wasm
|
||||
!src/embeddings/wasm/pkg/*.js
|
||||
!src/embeddings/wasm/pkg/*.d.ts
|
||||
|
|
@ -111,6 +106,3 @@ src/embeddings/wasm/pkg/*
|
|||
# Temporary files (redundant but explicit)
|
||||
*.tmp
|
||||
/.junie/guidelines.md
|
||||
|
||||
# Claude Code harness state
|
||||
.claude/scheduled_tasks.lock
|
||||
|
|
|
|||
24
.versionrc.json
Normal file
24
.versionrc.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"types": [
|
||||
{"type": "feat", "section": "✨ Features"},
|
||||
{"type": "fix", "section": "🐛 Bug Fixes"},
|
||||
{"type": "docs", "section": "📚 Documentation"},
|
||||
{"type": "refactor", "section": "♻️ Code Refactoring"},
|
||||
{"type": "perf", "section": "⚡ Performance Improvements"},
|
||||
{"type": "test", "section": "✅ Tests"},
|
||||
{"type": "build", "section": "🔧 Build System"},
|
||||
{"type": "ci", "section": "🔄 CI/CD"},
|
||||
{"type": "style", "hidden": true},
|
||||
{"type": "chore", "hidden": true}
|
||||
],
|
||||
"compareUrlFormat": "https://github.com/soulcraftlabs/brainy/compare/{{previousTag}}...{{currentTag}}",
|
||||
"commitUrlFormat": "https://github.com/soulcraftlabs/brainy/commit/{{hash}}",
|
||||
"issueUrlFormat": "https://github.com/soulcraftlabs/brainy/issues/{{id}}",
|
||||
"userUrlFormat": "https://github.com/{{user}}",
|
||||
"releaseCommitMessageFormat": "chore(release): {{currentTag}}",
|
||||
"issuePrefixes": ["#"],
|
||||
"header": "# Changelog\n\nAll notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.\n",
|
||||
"scripts": {
|
||||
"postbump": "echo '✅ Version bumped to' $(node -p \"require('./package.json').version\")"
|
||||
}
|
||||
}
|
||||
743
CHANGELOG.md
743
CHANGELOG.md
|
|
@ -2,709 +2,6 @@
|
|||
|
||||
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
||||
|
||||
### [8.10.0](https://github.com/soulcraftlabs/brainy/compare/v8.9.0...v8.10.0) (2026-07-23)
|
||||
|
||||
- docs: adoption storefront — contributing guide, security policy, README support + cor section (9a99a7b)
|
||||
- fix(release): push the public mirror explicitly and verify the tag lands at the right commit before publishing (6ba94c8)
|
||||
- docs: project guide version line points at npm instead of a hardcoded stale number (3a1efc9)
|
||||
- feat: vector provider identity is a required name field (hnsw-js), rendered [vector-index:<name>] (3be4ba9)
|
||||
- feat: warm contract (warm/warmOnOpen/provider warm hook), configurable transact budget floor, backend-neutral vector index op names (55b867c)
|
||||
|
||||
|
||||
### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19)
|
||||
|
||||
- docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78)
|
||||
- fix: release drains in-flight writer-lock heartbeat — no phantom lock after unlink (70e4bc8)
|
||||
- feat: flush() never compacts — history maintenance moves to close() with bounded passes (300d9f2)
|
||||
|
||||
|
||||
### [8.8.2](https://github.com/soulcraftlabs/brainy/compare/v8.8.1...v8.8.2) (2026-07-19)
|
||||
|
||||
- fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings (945d92d)
|
||||
- chore: push public docs to the soulcraft.com ingest door on release (42037d0)
|
||||
|
||||
|
||||
### [8.8.1](https://github.com/soulcraftlabs/brainy/compare/v8.8.0...v8.8.1) (2026-07-18)
|
||||
|
||||
- fix: O(1) adaptive retention accounting + historyStats fleet audit (6207e48)
|
||||
- fix: import dedup off-switch honesty + brain-owned lifecycle for the background pass (4fcef7b)
|
||||
|
||||
|
||||
### [8.8.0](https://github.com/soulcraftlabs/brainy/compare/v8.7.1...v8.8.0) (2026-07-17)
|
||||
|
||||
- feat: OS-limit detection for pool-scale deployments (16a73b8)
|
||||
|
||||
|
||||
### [8.7.1](https://github.com/soulcraftlabs/brainy/compare/v8.7.0...v8.7.1) (2026-07-17)
|
||||
|
||||
- fix: race-proof writer-lock acquisition + machine-readable conflict through init (01a3b46)
|
||||
|
||||
|
||||
### [8.7.0](https://github.com/soulcraftlabs/brainy/compare/v8.6.0...v8.7.0) (2026-07-17)
|
||||
|
||||
- feat: scaled transact budgets + labeled timeout diagnostics + envelope docs (6ef9fcb)
|
||||
|
||||
|
||||
### [8.6.0](https://github.com/soulcraftlabs/brainy/compare/v8.5.2...v8.6.0) (2026-07-17)
|
||||
|
||||
- feat: brain.auditGraph() — read-only graph-truth audit (2a03fae)
|
||||
|
||||
|
||||
### [8.5.2](https://github.com/soulcraftlabs/brainy/compare/v8.5.1...v8.5.2) (2026-07-17)
|
||||
|
||||
- fix: exception-safe aggregation backfill + generation-verified adoption + loud open-path guards (a77b064)
|
||||
|
||||
|
||||
### [8.5.1](https://github.com/soulcraftlabs/brainy/compare/v8.5.0...v8.5.1) (2026-07-17)
|
||||
|
||||
- fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal (da55be7)
|
||||
- docs: external-backups/sparse-storage guide + generation fact log concept (593bb8b)
|
||||
|
||||
|
||||
### [8.5.0](https://github.com/soulcraftlabs/brainy/compare/v8.4.0...v8.5.0) (2026-07-15)
|
||||
|
||||
- test: tolerant timing assertion in the execution-time measure test (4dc0a92)
|
||||
- feat: committedGeneration capability + pinned durability/stability contracts (d1ecee1)
|
||||
- docs: RELEASES.md entry for 8.5.0 (provider fact-log access + shared verifier) (e4f37cd)
|
||||
- feat: provider access to the fact log + shared stamp verifier via internals (352e356)
|
||||
|
||||
|
||||
### [8.4.0](https://github.com/soulcraftlabs/brainy/compare/v8.3.3...v8.4.0) (2026-07-15)
|
||||
|
||||
- docs: RELEASES.md entry for 8.4.0 (generation fact log + family stamp) (4a60b43)
|
||||
- feat: entity-tree family stamp — sourceGeneration + rollup coherence at open (2888ae6)
|
||||
- feat: generation fact log — after-image commit records, dual-written at every commit point (38b0041)
|
||||
|
||||
|
||||
### [8.3.3](https://github.com/soulcraftlabs/brainy/compare/v8.3.2...v8.3.3) (2026-07-15)
|
||||
|
||||
- docs: RELEASES.md entry for 8.3.3 (rename containment fix + repair) (c3feafd)
|
||||
- test: lens-consistency regression — combined vs subtype-only vs canonical ground truth (4fb41f9)
|
||||
- fix: VFS rename moves the containment edge — no ghost in the old directory (af8c179)
|
||||
|
||||
|
||||
### [8.3.2](https://github.com/soulcraftlabs/brainy/compare/v8.3.1...v8.3.2) (2026-07-14)
|
||||
|
||||
- docs: RELEASES.md entry for 8.3.2 (honest counters) (0932ecd)
|
||||
- fix: honest counters — removal never re-reads the removed record + repairIndex recounts and persists all rollups (2e2ba9c)
|
||||
|
||||
|
||||
### [8.3.1](https://github.com/soulcraftlabs/brainy/compare/v8.3.0...v8.3.1) (2026-07-14)
|
||||
|
||||
- docs: RELEASES.md entry for 8.3.1 (full-removal deletes + family-scoped gate) (c0c68ac)
|
||||
- fix: full-removal canonical deletes + family-scoped migration gate (366f9a9)
|
||||
- docs: cite the cross-layer integrity contract generically in comments and notes (1d26988)
|
||||
|
||||
|
||||
### [8.3.0](https://github.com/soulcraftlabs/brainy/compare/v8.2.8...v8.3.0) (2026-07-13)
|
||||
|
||||
- docs: RELEASES.md entry for 8.3.0 (heal-cost + cross-layer integrity contract) (7692c6f)
|
||||
- perf: parallel + id-only canonical enumeration (heal-cost dominant term) (ec5b933)
|
||||
- feat: registered-blob family contract — declared index blobs are undeletable (bfa1762)
|
||||
- feat: validateIndexConsistency delegates to provider invariants (6bcb54f)
|
||||
|
||||
|
||||
### [8.2.8](https://github.com/soulcraftlabs/brainy/compare/v8.2.7...v8.2.8) (2026-07-13)
|
||||
|
||||
- fix: honest index readiness — no silently-empty queries on a cold index (d0f69c7)
|
||||
|
||||
|
||||
### [8.2.7](https://github.com/soulcraftlabs/brainy/compare/v8.2.6...v8.2.7) (2026-07-13)
|
||||
|
||||
- fix: restore loadBinaryBlob fault-propagation (native column-store lockstep) (b6c7039)
|
||||
|
||||
|
||||
### [8.2.6](https://github.com/soulcraftlabs/brainy/compare/v8.2.5...v8.2.6) (2026-07-13)
|
||||
|
||||
- docs: RELEASES.md entry for 8.2.6 (write/index-spine hardening) (a873852)
|
||||
- chore: hold loadBinaryBlob fault-propagation for the cortex column-store lockstep (36c10c1)
|
||||
- fix: aggregation surfaces materialize/state-load failures loudly (02eff64)
|
||||
- fix: surface a degraded derived index on reads instead of serving it silently (ba958d9)
|
||||
- fix: saveBinaryBlob never acks a durable write that stored nothing (7feba49)
|
||||
- fix: refuse writes when single-op history cannot be made durable (54c1836)
|
||||
- fix: clear() wipes the full native/derived footprint, not a subset (d8301f8)
|
||||
- fix: surface segment/entity read faults loudly instead of masking as absent (af5d2f3)
|
||||
- fix: spine hardening pass 1 (part) — count symmetry, honest partial-load, flush durability, read-fault propagation (119087a)
|
||||
- test: pin the read-your-writes contract under the single writer (eb9c4eb)
|
||||
|
||||
|
||||
### [8.2.5](https://github.com/soulcraftlabs/brainy/compare/v8.2.4...v8.2.5) (2026-07-12)
|
||||
|
||||
- docs: RELEASES.md entry for 8.2.5 (honest rollback-failure response) (a7c7aa5)
|
||||
- fix: honest response when a transaction rollback cannot complete (711d2f0)
|
||||
|
||||
|
||||
### [8.2.4](https://github.com/soulcraftlabs/brainy/compare/v8.2.3...v8.2.4) (2026-07-12)
|
||||
|
||||
- docs: RELEASES.md entry for 8.2.4 (non-destructive restore) (4574695)
|
||||
- fix: non-destructive, crash-resumable restore (a2f4f6a)
|
||||
|
||||
|
||||
### [8.2.3](https://github.com/soulcraftlabs/brainy/compare/v8.2.2...v8.2.3) (2026-07-12)
|
||||
|
||||
- docs: RELEASES.md entry for 8.2.3 (transact durability barrier) (be5ce0b)
|
||||
- fix: transact durability barrier — committed transactions are durable on return (3b8fa51)
|
||||
|
||||
|
||||
### [8.2.2](https://github.com/soulcraftlabs/brainy/compare/v8.2.1...v8.2.2) (2026-07-11)
|
||||
|
||||
- docs: RELEASES.md entry for 8.2.2 (transaction timeout rollback) (ed97006)
|
||||
- fix: transaction timeout rolls back applied operations (no torn state) (508a8e3)
|
||||
|
||||
|
||||
### [8.2.1](https://github.com/soulcraftlabs/brainy/compare/v8.2.0...v8.2.1) (2026-07-10)
|
||||
|
||||
- test: update graph-index operation constructors to the VerbEndpointInts signature (62a449d)
|
||||
- docs: RELEASES.md entry for 8.2.1 (transact forward-ref parity fix) (7089782)
|
||||
- fix: transact forward references resolve graph endpoint ints at execute time (a175406)
|
||||
|
||||
|
||||
### [8.2.0](https://github.com/soulcraftlabs/brainy/compare/v8.1.0...v8.2.0) (2026-07-10)
|
||||
|
||||
- docs: RELEASES.md entry for 8.2.0 (temporal VFS) (98ceadc)
|
||||
- feat: temporal VFS — file content joins the Model-B immutability model (a3467e1)
|
||||
- docs: pin the write-path invariant in the plugin contract (the onChange change-feed guarantee) (4af8fb3)
|
||||
|
||||
|
||||
### [8.1.0](https://github.com/soulcraftlabs/brainy/compare/v8.0.17...v8.1.0) (2026-07-10)
|
||||
|
||||
- docs: RELEASES.md entry for 8.1.0 (brain.onChange change feed) (4e9be08)
|
||||
- feat: brain.onChange — the in-process change feed for every committed mutation (fd5edb5)
|
||||
|
||||
|
||||
### [8.0.17](https://github.com/soulcraftlabs/brainy/compare/v8.0.16...v8.0.17) (2026-07-08)
|
||||
|
||||
- docs: RELEASES.md entry for 8.0.17 (canonical count recovery + dead-machinery sweep) (6b8b9cb)
|
||||
- fix: count recovery scans the canonical layout; remove the dead 7.x hnsw sharding machinery (352e2da)
|
||||
|
||||
|
||||
### [8.0.16](https://github.com/soulcraftlabs/brainy/compare/v8.0.15...v8.0.16) (2026-07-08)
|
||||
|
||||
- docs: RELEASES.md entry for 8.0.16 (atomic ifAbsent/upsert + exact blob refCounts) (54e7c0e)
|
||||
- fix: atomic ifAbsent/upsert inserts + exact blob reference counts under concurrency (867939e)
|
||||
|
||||
|
||||
### [8.0.15](https://github.com/soulcraftlabs/brainy/compare/v8.0.14...v8.0.15) (2026-07-08)
|
||||
|
||||
- docs: RELEASES.md entry for 8.0.15 (atomic ifRev CAS) (b1fe25a)
|
||||
- fix: ifRev CAS is atomic — the revision check now runs under the commit mutex (9a3d1bd)
|
||||
|
||||
|
||||
### [8.0.14](https://github.com/soulcraftlabs/brainy/compare/v8.0.13...v8.0.14) (2026-07-07)
|
||||
|
||||
- docs: RELEASES.md entry for 8.0.14 (migration preserves branch-scoped non-entity state) (64188a3)
|
||||
- fix: 7→8 migration preserves branch-scoped non-entity state instead of deleting it (a93bb4e)
|
||||
|
||||
|
||||
### [8.0.13](https://github.com/soulcraftlabs/brainy/compare/v8.0.12...v8.0.13) (2026-07-07)
|
||||
|
||||
- docs: RELEASES.md entry for 8.0.13 (accurate boot log for established stores) (38e8de5)
|
||||
- fix: an established store no longer boot-logs "New installation" (3086916)
|
||||
|
||||
|
||||
### [8.0.12](https://github.com/soulcraftlabs/brainy/compare/v8.0.11...v8.0.12) (2026-07-07)
|
||||
|
||||
- docs: RELEASES.md entry for 8.0.12 (7→8 VFS recovery, zero-rebuild cold open, strict query operators) (d9017e7)
|
||||
- fix: recover VFS content blobs stranded by a 7→8 upgrade, in place on open (c0f6ccd)
|
||||
- fix: validate where-operators and align the in-memory matcher to the documented set (6821e19)
|
||||
- docs: correct rc-era time-travel staleness + record the embedding-model ordering constraint (68da660)
|
||||
- fix: cold-open no longer re-derives durable indexes — complete the readiness contract for all three providers (61c247c)
|
||||
- docs: RELEASES.md entry for 8.0.11 (exit-hang class closed for every op shape) (4fde94b)
|
||||
|
||||
|
||||
### [8.0.11](https://github.com/soulcraftlabs/brainy/compare/v8.0.10...v8.0.11) (2026-07-02)
|
||||
|
||||
- fix: no script shape can hang on brainy's internals — unref every maintenance timer + one-shot beforeExit (30eacbd)
|
||||
- docs: RELEASES.md entry for 8.0.10 (clean process exit after close) (2da2736)
|
||||
|
||||
|
||||
### [8.0.10](https://github.com/soulcraftlabs/brainy/compare/v8.0.9...v8.0.10) (2026-07-02)
|
||||
|
||||
- fix: a bare script now exits cleanly after close() — release every process keep-alive (c540d63)
|
||||
|
||||
|
||||
### [8.0.9](https://github.com/soulcraftlabs/brainy/compare/v8.0.8...v8.0.9) (2026-07-02)
|
||||
|
||||
- feat: guarded plugin auto-detection — installing @soulcraft/cor is the opt-in (588267b)
|
||||
|
||||
|
||||
### [8.0.8](https://github.com/soulcraftlabs/brainy/compare/v8.0.7...v8.0.8) (2026-07-02)
|
||||
|
||||
- docs: plugins are explicit opt-in — correct the README scale section and plugins config comment (e420369)
|
||||
|
||||
|
||||
### [8.0.7](https://github.com/soulcraftlabs/brainy/compare/v8.0.1...v8.0.7) (2026-07-02)
|
||||
|
||||
- docs: GA version is 8.0.7 — npm retired 8.0.0-8.0.6 (January dev-cycle unpublishes) (5db2c41)
|
||||
- chore(release): 8.0.1 (48bea9e)
|
||||
- docs: flagship README for the 8.0 GA; GA version is 8.0.1 (e44620e)
|
||||
- docs: rename the native provider to @soulcraft/cor across public docs and JSDoc (bf4a333)
|
||||
- chore(release): 8.0.0 (a3c2717)
|
||||
- docs: RELEASES.md 8.0.0 GA entry (RC notes become history) (4584d0b)
|
||||
- feat: promote the 8.0 u64-id line to main for the 8.0.0 GA (55d57f8)
|
||||
- fix(8.0): byte-copy _id_mapper/* in the pre-upgrade backup (cor's write-new nuance) (a30ed72)
|
||||
- chore(release): 7.33.5 (29b9d5f)
|
||||
- fix: metadata cold-read guard — no more silent [] on cold find({where}) (7.33.5) (9dc4c5e)
|
||||
- fix(8.0): metadata cold-read guard — no more silent [] on cold find({where}) (79e8709)
|
||||
- docs(8.0): add module JSDoc to typeValidation.ts (the one file missing a module block) (ab53fa0)
|
||||
- feat(8.0): auto pre-upgrade backup — hard-link snapshot before the 7.x→8.0 migration (1aad1f6)
|
||||
- fix(ci): commit the prebuilt wasm pkg + build before test:bun (green CI on fresh clone) (ed178e2)
|
||||
- chore(release): 7.33.4 (2be3d0f)
|
||||
- fix: never serve a silent [] from find({connected}) on a cold-loaded graph (fd699d0)
|
||||
- chore(release): 7.33.3 (d1665bb)
|
||||
- fix: re-validate find() results against the predicate (index-integrity guard) (7b5db0d)
|
||||
- chore(release): 7.33.2 (9593a27)
|
||||
- fix: graph adjacency cold-load consistency guard — no more silent [] on connected (1694f68)
|
||||
- chore(release): 7.33.1 (811c7da)
|
||||
- fix: getNouns cursor pagination re-scanned the first page forever (permanent CPU loop) (6721c52)
|
||||
- chore(release): 7.33.0 (526aaad)
|
||||
- feat: visibility tier (public/internal/system) on nouns + verbs (3a62445)
|
||||
- chore(release): 7.32.2 (c53dd61)
|
||||
- refactor: rename BackupData → PortableGraph (the type is interchange, not a backup) (89036de)
|
||||
- chore(release): 7.32.1 (5e7379d)
|
||||
- fix: getNouns().totalCount reports true total, not page size; quiet benign mmap-vector log (edff637)
|
||||
- chore(release): 7.32.0 (adec0ba)
|
||||
- feat: portable graph export()/import() (BackupData v1) on brain.data() (a408d37)
|
||||
- chore(release): 7.31.8 (89c6d04)
|
||||
- fix: query-cap memory misread (MemAvailable + floor) + rootDirectory getter for native mmap fast-path (3f8e097)
|
||||
- chore(release): 7.31.7 (4f8159c)
|
||||
- fix: vfs.rename() issues a metadata-only update + rollback of fresh adds removes them (ac29b0e)
|
||||
- chore(release): 7.31.6 (9b52629)
|
||||
- fix: remap reserved fields from update() metadata patches to their canonical location (67e5fc8)
|
||||
- chore(release): 7.31.5 (e5ec658)
|
||||
- fix: feature-detect setVectorBackend before wiring the mmap-vector backend (a537b36)
|
||||
- chore(release): 7.31.4 (a8cbab6)
|
||||
- fix: feature-detect setConnectionsCodec before wiring the connections codec (747ab97)
|
||||
- chore(release): 7.31.3 (cfb051c)
|
||||
- fix: mmap-vector backend capacity NaN at the provider FFI boundary (eade6ff)
|
||||
|
||||
|
||||
### [8.0.0-rc.9](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.8...v8.0.0-rc.9) (2026-07-01)
|
||||
|
||||
- docs(8.0): RELEASES.md — rc.9 (migration LOCK + 6x cosine + ES2023/Node22 floor) (3a33987)
|
||||
- chore(8.0): ES2023 target + drop DOM lib + downlevelIteration (config truth-up) (cf74c25)
|
||||
- perf(8.0): allocation-free distance loops (6x cosine) — evidence-revised Fork X (b5bc73f)
|
||||
- feat(8.0): #18 coordinated migration LOCK — block-and-queue the 7.x→8.0 auto-upgrade (67bbf69)
|
||||
- chore(8.0): modernize toolchain + position Bun as a runtime (ca9129a)
|
||||
|
||||
|
||||
### [8.0.0-rc.8](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.7...v8.0.0-rc.8) (2026-06-30)
|
||||
|
||||
- docs(8.0): RELEASES.md — rc.8 (no-freeze online whole-brain auto-upgrade) (5af48a9)
|
||||
- feat(8.0): no-freeze auto-upgrade hooks — isMigrating() deference + stampBrainFormat() + brain-format export (b6b9198)
|
||||
|
||||
|
||||
### [8.0.0-rc.7](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.6...v8.0.0-rc.7) (2026-06-30)
|
||||
|
||||
- docs(8.0): RELEASES.md — rc.7 (cold-graph self-heal + billion-scale RAM + version handshake) (1ddc786)
|
||||
- perf(8.0): bound per-id generation history chains (O(W+L) resident, was O(N)) (a859d6e)
|
||||
- feat(8.0): eager graphIndex.init() before the isReady() rebuild gate (8f4787b)
|
||||
- feat(8.0): version-handshake marker (formatInfo + indexEpoch) for whole-brain auto-upgrade (fc7f110)
|
||||
- fix(8.0): never serve a silent [] from find({connected}) on a cold-loaded graph (229b067)
|
||||
- perf(8.0): represent the committed-generation ledger as an interval set (93f61db)
|
||||
- perf(8.0): drop O(N)-resident id-keyed storage caches; source counts from the record (b6beb7f)
|
||||
|
||||
|
||||
### [8.0.0-rc.6](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.5...v8.0.0-rc.6) (2026-06-29)
|
||||
|
||||
- docs(8.0): RELEASES.md — rc.6 (perf + native-provider contract + test hygiene) (6daa70e)
|
||||
- feat(8.0): wire the two cor-confirmed metadata-provider contract additions (8b19122)
|
||||
- test(8.0): re-home orphaned test files into the gate + guard against recurrence (3f9f140)
|
||||
- perf(8.0): negation/absence where-operators via roaring-bitmap difference (5f974ab)
|
||||
- perf(8.0): HNSW removeItem is O(in-degree) via a reverse-adjacency index (72df557)
|
||||
|
||||
|
||||
### [8.0.0-rc.5](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.4...v8.0.0-rc.5) (2026-06-29)
|
||||
|
||||
- docs(8.0): RELEASES.md — rc.5 hardening + the breaking operator removal (6c9a438)
|
||||
- refactor(8.0): remove the 4 deprecated query-operator aliases (clean break) (ddcc0c7)
|
||||
- refactor(8.0): remove dead/deprecated code (legacy sweep) (b9369f2)
|
||||
- refactor(8.0): API-surface + quality polish from the readiness audit (a52dba2)
|
||||
- fix(8.0): close GA-blocking correctness gaps from the readiness audit (47e8031)
|
||||
- docs(8.0): correct public docs to the real 8.0 API + honest perf claims (40d2cd5)
|
||||
- fix(8.0): re-validate find() results against the predicate (index-integrity guard) (3d11619)
|
||||
|
||||
|
||||
### [8.0.0-rc.4](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.3...v8.0.0-rc.4) (2026-06-24)
|
||||
|
||||
- docs(8.0): drop the DeletedItemsIndex section + pseudo-code from index-architecture (e7b50cf)
|
||||
- fix(8.0): gate native graph analytics on the provider readiness flag (d321cf5)
|
||||
- refactor(8.0): remove dead, unreachable, and unwired modules (bf0afe8)
|
||||
- build(8.0): clean dist before every build so stale artifacts never ship (03d6540)
|
||||
- feat(8.0): #35 part-3 — supply at-gen candidate vectors for the native exact-rerank (c9e2169)
|
||||
|
||||
|
||||
### [8.0.0-rc.3](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.2...v8.0.0-rc.3) (2026-06-23)
|
||||
|
||||
- test(8.0): de-flake the VFS path-cache timing assertion (0e8972c)
|
||||
- feat(8.0): asOf at-gen vector defer — provider-served historical semantic search (#35) (1c363e8)
|
||||
- perf(8.0): bound find({ where, orderBy }) sort to the page (CTX-BR-FIND-ORDERBY) (450084b)
|
||||
- feat(8.0): brain.graph.subgraph(query) query→expand fusion (#61) (82dde92)
|
||||
- feat(8.0): vector allowedIds predicate-pushdown into find() (#46) (dd325f2)
|
||||
- feat(8.0): graph analytics — brain.graph.rank / communities / path (632d90a)
|
||||
- fix(8.0): restore() reloads a native entity-id mapper before graphIndex.rebuild() (4d0b64f)
|
||||
- test(8.0): cover pending-tier range queries + setRetentionBudget adaptive reclaim (3783e61)
|
||||
- feat(8.0): Model-B per-write generation-stamping + adaptive retention knob (5c3bb2c)
|
||||
- test(8.0): Model-B write-perf + scalability spike harnesses (afac7f9)
|
||||
- perf(8.0): per-id history chains for O(log) historical reads + bounded delta cache (ceed70d)
|
||||
- refactor(8.0): graph analytics contract — intent names, not algorithm names (f3e6911)
|
||||
- docs(8.0): RELEASES — native provider is @soulcraft/cor 3.0 (fix cortex 3.0 self-contradiction) (96d9c0b)
|
||||
- test(8.0): cover the native graph seam + make provider resolution factory-tolerant (29410bc)
|
||||
|
||||
|
||||
### [8.0.0-rc.2](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.1...v8.0.0-rc.2) (2026-06-21)
|
||||
|
||||
- docs(8.0): RELEASES rc.2 additions — graph engine + additive wins + correctness fixes (18f27cb)
|
||||
- feat(8.0): brain.graph.export() + noun-walk cursor + noun visibility hydration (c2a84c9)
|
||||
- feat(8.0): brain.graph.subgraph() + native-provider routing (8c2b57a)
|
||||
- feat(8.0): related({ node }) — one-call both-direction incident edges (d4de48d)
|
||||
- feat(8.0): GraphAccelerationProvider contract — the native graph-engine seam (a3d6fdb)
|
||||
- perf(8.0): cursor pagination for the verb walk — full edge pagination O(N²) → O(N) (682e786)
|
||||
- perf(8.0): visibility-aware fast adjacency — related() stays O(degree) under default visibility (a914313)
|
||||
- feat(8.0): upsert + FindParams.includeVectors + removeMany adaptive chunking (4cc2088)
|
||||
- docs(8.0): note reserved-field default-throw in RELEASES rc additions (1bc709d)
|
||||
- feat(8.0): reserved-field enforcement — reservedFieldPolicy defaults to throw (54c7c39)
|
||||
- docs: mark 8.0.0-rc.1 published (npm tag rc) + note rc.1 additions (ae3fe82)
|
||||
|
||||
|
||||
### [8.0.0-rc.1](https://github.com/soulcraftlabs/brainy/compare/v7.31.2...v8.0.0-rc.1) (2026-06-20)
|
||||
|
||||
- feat(8.0): id-normalization (#18) + aggregation min/max delete-safety + RC-safe release (d02e522)
|
||||
- feat(8.0): API simplification — remove neural()/Db.search, one storage `path` key, integration→0 (606445c)
|
||||
- feat(8.0): 7.x→8.0 layout migration — fix silent total data loss on first open (0c4a51c)
|
||||
- feat(8.0): temporal range verbs — diff, history, since(gen|Date), asOf{exclusive}, transactionLog window (2c84f86)
|
||||
- refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup) (373a481)
|
||||
- fix(8.0): VFS path-cache instance-scoping + verb totalCount page-cap (3a3aa43)
|
||||
- fix(8.0): multi-valued array fields index every element (contains no longer misses) (eccf420)
|
||||
- fix(8.0): column-store range queries honor exclusive bounds (lessThan/greaterThan) (009e506)
|
||||
- fix(8.0): per-type counts rehydrate after cold reopen (column store, not dead sparse index) (d918f49)
|
||||
- feat(8.0): version-coupling guard — a mismatched/failed native plugin fails loud, never silent JS fallback (1264fec)
|
||||
- test(8.0): boundary guard forbids @soulcraft/cor too (cortex→cor rename) (b198281)
|
||||
- fix(8.0): stats() per-type counts no longer inflate with HNSW re-saves (21d02d3)
|
||||
- fix(8.0): getNouns().totalCount reports true total, not page size (port of 7.32.1) (b2005ff)
|
||||
- fix(8.0): real bugs surfaced by integration hardening — where-intersect, related() offset, relate() updatedAt (5eaf579)
|
||||
- test(8.0): integration rot pass — 77→17 failures (parallel per-file hardening) (e5997a1)
|
||||
- test(8.0): begin integration rot pass — clear-persistence (drop COW internals) + metadata-only addRelationship→relate (c600468)
|
||||
- docs(8.0): RELEASES — portable export/import (BackupData v1) + distinctCount any-type section (4741e23)
|
||||
- fix(8.0): distinctCount aggregates distinct values of any type + edge-case regression tests (574a8b1)
|
||||
- feat(8.0): validateBackup() dry-run + includeContent blob round-trip test + clone test (7aad803)
|
||||
- docs(8.0): export/import guide + api/README portable backup section (c2b73d4)
|
||||
- feat(8.0): portable graph export()/import() (BackupData v1) — Db.export + polymorphic import (010ccf8)
|
||||
- feat(8.0): visibility field (public/internal/system) on nouns + verbs (f4dea80)
|
||||
- test(8.0): close brains in afterEach (count-sync, get-relations teardown) (0ca0e5c)
|
||||
- fix(8.0): neural.clusters()/similarity must request vectors from get() (cc1a431)
|
||||
- test(8.0): drop dead s3/distributed/cloud scripts + 32GB→8GB integration heap (73a7d82)
|
||||
- test(8.0): remove dead s3/distributed/cloud scripts + stale s3 suite (af1ee46)
|
||||
- test(8.0): Tier-1 integration via deterministic embedder (suite runnable again) (542b52e)
|
||||
- test(8.0): use valid camelCase VerbType values in test-factory (e31ba89)
|
||||
- test(8.0): get() resolves null for absent custom ids instead of throwing (dc94af3)
|
||||
- fix(8.0): accept application-supplied entity ids, not just UUIDs (36b7216)
|
||||
- fix(8.0): honor top-level storage.path as a rootDirectory alias (5096f90)
|
||||
- feat(8.0): thread commit generation through the graph-write provider contract (0951fa1)
|
||||
- fix(8.0): drive query-cap off MemAvailable + floor auto-detected caps (b26d3d4)
|
||||
- test(8.0): A/B benchmark harness (open leg) — generic corpus+metrics lib, brainy-alone scaling bench, boundary guard, real-embedding recall guard (c605b34)
|
||||
- docs(8.0): remove unbacked Cortex '5.2x' perf claim + dangling /docs/cortex/comparison link (33caa52)
|
||||
- docs(8.0): measured find() performance at 5k/100k in SCALING.md (f986832)
|
||||
- test(8.0): asOf() error-path spot-checks + find() triple-composition correctness + scale-bench harness (af96064)
|
||||
- docs(8.0): RELEASES.md — record removed BrainyZeroConfig + isFullyInitialized/awaitBackgroundInit in the breaking-change inventory (f12ca68)
|
||||
- refactor(8.0)!: remove orphaned zero-config subsystem + dead cloud/progressive-init storage vestige (35b9d7e)
|
||||
- refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider (00d3203)
|
||||
- feat(8.0): zero-config finalize + cut JS quantization (config.vector = recall + persistMode) (f8e0079)
|
||||
- fix(8.0): vfs.rename() issues a metadata-only update (port of the 7.31.7 fix) (f4c5d97)
|
||||
- chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase (1f7e365)
|
||||
- feat(8.0): reserved-field contract — one canonical location, typed prevention, unified read/write (970e08c)
|
||||
- feat(8.0): brain.fillSubtypes migration helper + pre-RC1 gap closure (c446783)
|
||||
- docs(8.0): RELEASES.md 8.0.0 release-candidate entry — full breaking-change inventory + upgrade guide (9b0f4ac)
|
||||
- refactor(8.0): delete DataAPI — superseded by Db persist/restore + import API + stats (478fa17)
|
||||
- docs(8.0): consistency-model concept + snapshots guide — Db API replaces branching docs (cc8037d)
|
||||
- feat(8.0): full query surface at historical generations via ephemeral index materialization (e5feae4)
|
||||
- feat(8.0)!: delete fork/branch/commit/history/versions — superseded by the Db API (8f93add)
|
||||
- feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist) (431cd64)
|
||||
- fix(8.0): createIndex resolves the canonical 'vector' provider key — drop diskann/hnsw key lookups + legacy migration APIs (49e4948)
|
||||
- feat(8.0): u64 BigInt graph provider contract — punch list a-d,g,h (2427bb7)
|
||||
- chore(8.0): delete vectorStore:mmap wiring — dead in the 8.0 provider world (62f6472)
|
||||
- chore(8.0): collapse dead defensive guards + redundant polyfills (42159f2)
|
||||
- chore(8.0)!: drop browser support, cloud SDKs, legacy pipeline, dead threading (266715a)
|
||||
- docs(8.0): Phase F — deep clean across 21 docs (adda157)
|
||||
- chore(8.0): Phase C + D + E — config simplification, TODO sweep, test race fix (2626ab8)
|
||||
- chore(8.0): Phase A + B — purge all @deprecated APIs + cacheManager dead branches (cb16a39)
|
||||
- chore(8.0): step-7 follow-through — collapse remaining cloud branches + docs sweep (scaffold step 13) (9f9a415)
|
||||
- feat(8.0)!: flip requireSubtype default to true (BRAINY-8.0-SUBTYPE-CONTRACT § C-1) (780fb64)
|
||||
- fix(8.0): implement multi-hop subtype BFS in pure JS (open-core works standalone) (221fc45)
|
||||
- refactor(8.0): SubtypeRegistry hook + drop multi-hop subtype throw (scaffold steps 11-12) (ed75f25)
|
||||
- docs(8.0): document subtype required-by-default deferral (scaffold step 10) (1eb0ffc)
|
||||
- refactor(8.0): drop strictConfig — surface too small to justify the option (scaffold step 9) (694a31f)
|
||||
- refactor(8.0): simplify config.vector to 3 knobs + fold persistMode (scaffold step 8) (8e76740)
|
||||
- refactor(8.0): drop cloud + OPFS storage adapters; filesystem + memory only (scaffold step 7) (0e6263a)
|
||||
- refactor(8.0): final cleanup — drop HnswProvider alias + config.hnsw + 'hnsw' surface (scaffold step 6) (b20666e)
|
||||
- refactor(8.0): strictConfig + brain.stats() vector field + wireConnectionsCodec feature-detect (scaffold step 5) (3e1ef95)
|
||||
- refactor(8.0): add saveVectorIndexData / getVectorIndexData storage contract (scaffold step 4) (356f044)
|
||||
- refactor(8.0): rename HNSWIndex class → JsHnswVectorIndex (scaffold step 3) (f39d420)
|
||||
- refactor(8.0): add config.vector path + 'vectors' cache category (scaffold step 2) (8f87b35)
|
||||
- refactor(8.0): rename HnswProvider → VectorIndexProvider (8.0 scaffold) (076c26f)
|
||||
|
||||
|
||||
### [7.31.2](https://github.com/soulcraftlabs/brainy/compare/v7.31.1...v7.31.2) (2026-06-09)
|
||||
|
||||
- docs: correct misleading SQ4 quantization comment in type definitions (89e4d81)
|
||||
|
||||
|
||||
### [7.31.1](https://github.com/soulcraftlabs/brainy/compare/v7.31.0...v7.31.1) (2026-06-09)
|
||||
|
||||
- fix: saveBinaryBlob unique tmp suffix + ENOENT swallow on rename (550bd4a)
|
||||
|
||||
|
||||
### [7.31.0](https://github.com/soulcraftlabs/brainy/compare/v7.30.2...v7.31.0) (2026-06-09)
|
||||
|
||||
- feat: per-entity _rev + update({ ifRev }) CAS + add({ ifAbsent }) (bafb4e4)
|
||||
|
||||
|
||||
### [7.30.2](https://github.com/soulcraftlabs/brainy/compare/v7.30.1...v7.30.2) (2026-06-08)
|
||||
|
||||
- fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location (9e307e4)
|
||||
|
||||
|
||||
### [7.30.1](https://github.com/soulcraftlabs/brainy/compare/v7.30.0...v7.30.1) (2026-06-08)
|
||||
|
||||
- fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors (5f3a2ca)
|
||||
|
||||
|
||||
### [7.30.0](https://github.com/soulcraftlabs/brainy/compare/v7.29.0...v7.30.0) (2026-06-05)
|
||||
|
||||
- feat: verb subtype + updateRelation + requireSubtype enforcement (c0d326b)
|
||||
|
||||
|
||||
### [7.29.0](https://github.com/soulcraftlabs/brainy/compare/v7.28.0...v7.29.0) (2026-06-04)
|
||||
|
||||
- feat: subtype top-level field + trackField + migrateField (2cdf70e)
|
||||
- feat(8.0): EntityIdMapper U32 ceiling + EntityIdSpaceExceeded error (e47fea0)
|
||||
- feat: DiskANN auto-engagement + migrateToDiskAnn/migrateToHnsw (8f130d3)
|
||||
- feat(plugin): DiskAnnProvider contract + HNSWConfig.type/diskann knobs (f885f81)
|
||||
|
||||
|
||||
### [7.28.0](https://github.com/soulcraftlabs/brainy/compare/v7.27.0...v7.28.0) (2026-05-28)
|
||||
|
||||
- feat: SQ4 (4-bit) scalar quantization + native distance hook (2.5.0 #30) (73e7e39)
|
||||
|
||||
|
||||
### [7.27.0](https://github.com/soulcraftlabs/brainy/compare/v7.26.0...v7.27.0) (2026-05-28)
|
||||
|
||||
- feat: content-type-aware compression policy in COW BlobStorage (2.5.0 #32) (178ff02)
|
||||
|
||||
|
||||
### [7.26.0](https://github.com/soulcraftlabs/brainy/compare/v7.25.0...v7.26.0) (2026-05-28)
|
||||
|
||||
- feat: graph link compression — delta-varint connections (2.4.0 #3) (617c156)
|
||||
- feat: column-store JS↔native interchange — raw-blob unify (2.4.0 #4) (71bc30b)
|
||||
- feat: mmap-vector backend wiring — HNSWIndex consumes vectorStore:mmap (2.4.0 #2) (d4cb26c)
|
||||
- feat: stable EntityIdMapper — rebuild() no longer renumbers UUID→int (b2408cb)
|
||||
|
||||
|
||||
### [7.25.0](https://github.com/soulcraftlabs/brainy/compare/v7.24.0...v7.25.0) (2026-05-27)
|
||||
|
||||
- docs: remove stale distanceSQ8 JSDoc left by the SQ8 hook refactor (6099101)
|
||||
- feat: export provider contracts for the plugin surface brainy consumes (4b6f63e)
|
||||
- feat: hook native sort:topK provider into search result ranking (46fc7f2)
|
||||
- feat: hook native SQ8 distance provider into HNSW reranking (00d14cf)
|
||||
- merge: storage binary-blob primitive across all adapters (e23361c)
|
||||
- fix: code-point string collation in LSM SSTable, COW trees/refs, sorted queries (7493d8e)
|
||||
- feat(storage): add raw binary-blob primitive to every storage adapter (298b572)
|
||||
- fix: deterministic code-point string collation for column store + aggregation (547721a)
|
||||
- feat: exact percentile and distinctCount aggregation ops (fe4f5df)
|
||||
|
||||
|
||||
### [7.24.0](https://github.com/soulcraftlabs/brainy/compare/v7.23.0...v7.24.0) (2026-05-26)
|
||||
|
||||
- feat: array-unnest groupBy for aggregates + batch-embed entity extraction (c2e21b7)
|
||||
|
||||
|
||||
### [7.23.0](https://github.com/soulcraftlabs/brainy/compare/v7.22.1...v7.23.0) (2026-05-26)
|
||||
|
||||
- feat: queryAggregate() + HAVING, plus aggregate backfill, traversal depth/via, extraction typing (BR-ADV-FEATURES-BUN) (1a98e42)
|
||||
- chore(release): create annotated tag so --follow-tags pushes it (513186d)
|
||||
|
||||
|
||||
### [7.22.1](https://github.com/soulcraftlabs/brainy/compare/v7.22.0...v7.22.1) (2026-05-26)
|
||||
|
||||
- fix: extraction, multi-hop traversal, and aggregate result shape (BR-ADV-FEATURES-BUN) (0a9d1d9)
|
||||
- docs: storage-adapter inheritance contract + correct the hasStorageMethod story (07754d1)
|
||||
|
||||
|
||||
### [7.22.0](https://github.com/soulcraftlabs/brainy/compare/v7.21.0...v7.22.0) (2026-05-15)
|
||||
|
||||
- fix: find()/stats() correctness + Cortex compat (BR-FIND-WHERE-ZERO, BR-DEFENSIVE-INTERFACE) (7026311)
|
||||
|
||||
|
||||
### [7.21.0](https://github.com/soulcraftlabs/brainy/compare/v7.20.0...v7.21.0) (2026-05-15)
|
||||
|
||||
- chore: gitignore Claude Code harness scheduled-tasks lockfile (a8fcc3d)
|
||||
- feat: multi-process safety + read-only inspector mode (4fcdc0f)
|
||||
|
||||
|
||||
### [7.20.0](https://github.com/soulcraftlabs/brainy/compare/v7.19.19...v7.20.0) (2026-04-10)
|
||||
|
||||
- refactor: delete dead sparse index write path (11be039)
|
||||
- feat: unified column store for filtering + sorting at billion scale (46583f2)
|
||||
|
||||
|
||||
### [7.19.19](https://github.com/soulcraftlabs/brainy/compare/v7.19.18...v7.19.19) (2026-04-09)
|
||||
|
||||
- refactor: migrate aggregation + neural field reads to resolveEntityField (108e2bc)
|
||||
|
||||
|
||||
### [7.19.18](https://github.com/soulcraftlabs/brainy/compare/v7.19.17...v7.19.18) (2026-04-09)
|
||||
|
||||
- feat: export resolveEntityField + STANDARD_ENTITY_FIELDS from internals (beefacb)
|
||||
|
||||
|
||||
### [7.19.17](https://github.com/soulcraftlabs/brainy/compare/v7.19.16...v7.19.17) (2026-04-09)
|
||||
|
||||
- fix: correct orderBy sort for timestamp fields via centralized field resolver (be6c4dc)
|
||||
|
||||
|
||||
### [7.19.15](https://github.com/soulcraftlabs/brainy/compare/v7.19.14...v7.19.15) (2026-03-23)
|
||||
|
||||
- fix: commit() now flushes and captures state by default (b58ea02)
|
||||
|
||||
|
||||
### [7.19.14](https://github.com/soulcraftlabs/brainy/compare/v7.19.13...v7.19.14) (2026-03-22)
|
||||
|
||||
- feat: add setMaxSize() for dynamic cache resizing (54865b3)
|
||||
|
||||
|
||||
### [7.19.13](https://github.com/soulcraftlabs/brainy/compare/v7.19.12...v7.19.13) (2026-03-22)
|
||||
|
||||
- fix: suppress misleading 'Using Q8 WASM' log when Cortex native is active (60a0f10)
|
||||
- perf: defer HNSW persistence during addMany() batch operations (973b6aa)
|
||||
|
||||
|
||||
### [7.19.10](https://github.com/soulcraftlabs/brainy/compare/v7.19.9...v7.19.10) (2026-02-24)
|
||||
|
||||
- fix: replace require('crypto') with ESM import in SSTable (239a4da)
|
||||
|
||||
|
||||
### [7.19.9](https://github.com/soulcraftlabs/brainy/compare/v7.19.8...v7.19.9) (2026-02-23)
|
||||
|
||||
- docs: replace ASCII box art with prose in Before/After section (6003e2b)
|
||||
|
||||
|
||||
### [7.19.8](https://github.com/soulcraftlabs/brainy/compare/v7.19.7...v7.19.8) (2026-02-23)
|
||||
|
||||
- docs: redesign ELI5 comparison section and add What Can You Build? (3f16e17)
|
||||
|
||||
|
||||
### [7.19.7](https://github.com/soulcraftlabs/brainy/compare/v7.19.6...v7.19.7) (2026-02-23)
|
||||
|
||||
- docs: add plain-language ELI5 overview and link from README (a88962f)
|
||||
|
||||
|
||||
### [7.19.6](https://github.com/soulcraftlabs/brainy/compare/v7.19.5...v7.19.6) (2026-02-19)
|
||||
|
||||
- docs: convert code examples to TypeScript (791cacc)
|
||||
|
||||
|
||||
### [7.19.5](https://github.com/soulcraftlabs/brainy/compare/v7.19.4...v7.19.5) (2026-02-19)
|
||||
|
||||
|
||||
|
||||
|
||||
### [7.19.4](https://github.com/soulcraftlabs/brainy/compare/v7.19.3...v7.19.4) (2026-02-19)
|
||||
|
||||
|
||||
|
||||
|
||||
### [7.19.3](https://github.com/soulcraftlabs/brainy/compare/v7.19.2...v7.19.3) (2026-02-19)
|
||||
|
||||
- docs: add public frontmatter to docs for soulcraft.com/docs pipeline (b6e3470)
|
||||
|
||||
|
||||
### [7.19.2](https://github.com/soulcraftlabs/brainy/compare/v7.19.1...v7.19.2) (2026-02-18)
|
||||
|
||||
- fix: metadata index not cleaned up after delete/deleteMany (1a628da)
|
||||
|
||||
|
||||
### [7.18.0](https://github.com/soulcraftlabs/brainy/compare/v7.17.0...v7.18.0) (2026-02-16)
|
||||
|
||||
- feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows (f024e56)
|
||||
- docs: add Claude Code project guide and verified architecture reference (089a4d4)
|
||||
|
||||
|
||||
### [7.17.0](https://github.com/soulcraftlabs/brainy/compare/v7.16.0...v7.17.0) (2026-02-09)
|
||||
|
||||
- feat: add migration system with error handling, validation, and enterprise hardening (39b099c)
|
||||
|
||||
|
||||
### [7.16.0](https://github.com/soulcraftlabs/brainy/compare/v7.15.5...v7.16.0) (2026-02-09)
|
||||
|
||||
- feat: enforce data/metadata separation, numeric range queries, improved docs (0ddc05a)
|
||||
|
||||
|
||||
### [7.15.5](https://github.com/soulcraftlabs/brainy/compare/v7.15.4...v7.15.5) (2026-02-02)
|
||||
|
||||
- docs: update plugin docs to reflect opt-in behavior (c0bb413)
|
||||
|
||||
|
||||
### [7.15.4](https://github.com/soulcraftlabs/brainy/compare/v7.15.3...v7.15.4) (2026-02-02)
|
||||
|
||||
- fix: set verb.source/target to entity UUID instead of NounType (932fb95)
|
||||
|
||||
|
||||
### [7.15.3](https://github.com/soulcraftlabs/brainy/compare/v7.15.2...v7.15.3) (2026-02-02)
|
||||
|
||||
- feat: add explicit plugins config to control plugin auto-detection (6625385)
|
||||
|
||||
|
||||
### [7.15.2](https://github.com/soulcraftlabs/brainy/compare/v7.15.1...v7.15.2) (2026-02-01)
|
||||
|
||||
- fix: flush graph LSM-trees on close to prevent data loss across restarts (ab2493a)
|
||||
|
||||
|
||||
### [7.15.0](https://github.com/soulcraftlabs/brainy/compare/v7.14.0...v7.15.0) (2026-02-01)
|
||||
|
||||
- feat: harden plugin system wiring and add developer diagnostics (401e300)
|
||||
|
||||
|
||||
## [7.14.0](https://github.com/soulcraftlabs/brainy/compare/v7.13.0...v7.14.0) (2026-02-01)
|
||||
|
||||
|
||||
### ♻️ Code Refactoring
|
||||
|
||||
* remove src/cortex/ directory and fix README claims ([36db644](https://github.com/soulcraftlabs/brainy/commit/36db644eca94253f1df04a1f91968856ac585b36))
|
||||
|
||||
### [7.13.0](https://github.com/soulcraftlabs/brainy/compare/v7.12.0...v7.13.0) (2026-02-01)
|
||||
|
||||
- refactor: remove augmentation system and semantic type matching (d1db351)
|
||||
|
||||
|
||||
### [7.12.0](https://github.com/soulcraftlabs/brainy/compare/v7.11.0...v7.12.0) (2026-02-01)
|
||||
|
||||
- feat: update plugin references from @soulcraft/brainy-cortex to @soulcraft/cortex (7f9d2a7)
|
||||
- refactor: remove deprecated Cortex class (replaced by brain.augmentations API) (490a14a)
|
||||
|
||||
|
||||
### [7.11.0](https://github.com/soulcraftlabs/brainy/compare/v7.10.0...v7.11.0) (2026-01-31)
|
||||
|
||||
- feat: add SQ8 vector quantization, lazy loading, and two-phase rerank to HNSW (0f3a884)
|
||||
|
||||
|
||||
### [7.10.0](https://github.com/soulcraftlabs/brainy/compare/v7.9.3...v7.10.0) (2026-01-31)
|
||||
|
||||
- feat: wire plugin system with provider resolution, storage factories, and browser deprecation (1513e29)
|
||||
- chore: sync package-lock.json after dependency install (25912b5)
|
||||
- feat: add plugin system for cortex and storage adapters (cc50ac3)
|
||||
- perf: optimize init() and rebuild performance (35cb674)
|
||||
- fix: eliminate flaky test timeouts and add storage adapters guide (cd87529)
|
||||
- fix: eliminate cloud storage write amplification and rate limiting (92d9420)
|
||||
- fix: distribute metadata index keys across sub-prefixes to avoid cloud rate limits (23e1c56)
|
||||
- fix: invalidate VFS caches recursively on rmdir to prevent orphaned reads (66d7aa7)
|
||||
|
||||
|
||||
### [7.9.3](https://github.com/soulcraftlabs/brainy/compare/v7.9.2...v7.9.3) (2026-01-28)
|
||||
|
||||
- perf: optimize addMany() with batch embedding for 5-10x speedup (df7d467)
|
||||
- fix: cancel abandoned highlight() semantic work and harden WASM engine recovery (f8dd93c)
|
||||
|
||||
|
||||
### [7.9.1](https://github.com/soulcraftlabs/brainy/compare/v7.9.0...v7.9.1) (2026-01-27)
|
||||
|
||||
- fix: exclude __words__ keyword index from corruption detection and getStats() (364360d)
|
||||
|
|
@ -1105,7 +402,7 @@ v7.0.0 introduced **breaking changes** to the embedding system:
|
|||
| copy 15 files | ~120s | ~20-40s | 3-6x faster |
|
||||
| move 15 files | ~240s | ~40-60s | 4-6x faster |
|
||||
|
||||
Requested by: a consumer team (BRAINY-VFS-RMDIR-PERFORMANCE)
|
||||
Requested by: Soulcraft Workshop team (BRAINY-VFS-RMDIR-PERFORMANCE)
|
||||
|
||||
### [6.3.2](https://github.com/soulcraftlabs/brainy/compare/v6.3.1...v6.3.2) (2025-12-09)
|
||||
|
||||
|
|
@ -1148,7 +445,7 @@ Requested by: a consumer team (BRAINY-VFS-RMDIR-PERFORMANCE)
|
|||
**Fixed VFS tree operations on cloud storage (GCS, S3, Azure, R2, OPFS)**
|
||||
|
||||
**Issue:** Despite v6.1.0's PathResolver optimization, `vfs.getTreeStructure()` remained critically slow on cloud storage:
|
||||
- **Production (GCS) deployment:** 5,304ms for tree with maxDepth=2
|
||||
- **Workshop Production (GCS):** 5,304ms for tree with maxDepth=2
|
||||
- **Root Cause:** Tree traversal made 111+ separate storage calls (one per directory)
|
||||
- **Why v6.1.0 didn't help:** v6.1.0 optimized path→ID resolution, but tree traversal still called `getChildren()` 111+ times
|
||||
|
||||
|
|
@ -1176,7 +473,7 @@ Result: 111 storage calls → 1 storage call
|
|||
- `src/vfs/VirtualFileSystem.ts:730-762` - Updated `getDescendants()` to use batch fetch
|
||||
|
||||
**Impact:**
|
||||
- ✅ Consumer file explorer now loads instantly on GCS
|
||||
- ✅ Workshop file explorer now loads instantly on GCS
|
||||
- ✅ Clean architecture: one code path, no fallbacks
|
||||
- ✅ Production-scale: uses in-memory graph + single batch fetch
|
||||
- ✅ Works for ALL storage adapters (GCS, S3, Azure, R2, OPFS, FileSystem)
|
||||
|
|
@ -1515,7 +812,7 @@ for (const id of pageIds) {
|
|||
**Fix:** Set `isInitialized = true` at the START of `BaseStorage.init()` (before any initialization work) to prevent recursive calls. Flag is reset to `false` on error to allow retries.
|
||||
|
||||
**Impact:**
|
||||
- ✅ Fixes production blocker reported by a consumer team
|
||||
- ✅ Fixes production blocker reported by Workshop team
|
||||
- ✅ All 8 storage adapters fixed (FileSystem, Memory, S3, R2, GCS, Azure, OPFS, Historical)
|
||||
- ✅ Init completes in ~1 second on fresh installation (was hanging indefinitely)
|
||||
- ✅ No new test failures introduced (1178 tests passing)
|
||||
|
|
@ -1795,7 +1092,7 @@ See comprehensive guides:
|
|||
v5.10.0 reintroduced a critical bug where `BlobStorage.read()` was hashing wrapped binary data instead of unwrapped content, causing all blob integrity checks to fail:
|
||||
- **Symptom**: `Blob integrity check failed: <hash>` errors on every VFS file read
|
||||
- **Root Cause**: Missing defense-in-depth unwrap verification in `BlobStorage.read()`
|
||||
- **Impact**: 100% failure rate for VFS file operations in A consumer application
|
||||
- **Impact**: 100% failure rate for VFS file operations in Workshop application
|
||||
|
||||
### The Fix (v5.10.1)
|
||||
1. **Defense-in-Depth Unwrapping**: Added unwrap verification in `BlobStorage.read()` before hash check
|
||||
|
|
@ -1943,7 +1240,7 @@ Reverted storage internals to v5.6.3 implementation:
|
|||
- ✅ No 12+ second delays per entity
|
||||
|
||||
### Verification
|
||||
a consumer team (production users) should upgrade immediately:
|
||||
Workshop team (production users) should upgrade immediately:
|
||||
```bash
|
||||
npm install @soulcraft/brainy@5.7.1
|
||||
```
|
||||
|
|
@ -1980,7 +1277,7 @@ Expected behavior after upgrade:
|
|||
|
||||
### 🐛 Bug Fixes
|
||||
|
||||
* **storage**: Fix `clear()` not deleting COW version control data (consumer-reported)
|
||||
* **storage**: Fix `clear()` not deleting COW version control data ([#workshop-bug-report](https://github.com/soulcraftlabs/brainy/issues))
|
||||
- Fixed all storage adapters to properly delete `_cow/` directory on clear()
|
||||
- Fixed in-memory entity counters not being reset after clear()
|
||||
- Prevents COW reinitialization after clear() by setting `cowEnabled = false`
|
||||
|
|
@ -2078,10 +1375,10 @@ Expected behavior after upgrade:
|
|||
- Impact: All relationship queries via `getRelations()`, `getRelationships()`
|
||||
- Reference: `src/storage/baseStorage.ts:2030-2040`, `src/storage/baseStorage.ts:2081-2091`
|
||||
|
||||
* **Consumer blob integrity**: Verified v5.4.0 lazy-loading asOf() prevents corruption
|
||||
* **Workshop blob integrity**: Verified v5.4.0 lazy-loading asOf() prevents corruption
|
||||
- HistoricalStorageAdapter eliminates race conditions
|
||||
- Snapshots created on-demand (no commit-time snapshot)
|
||||
- Verified with 570-entity test matching consumer production scale
|
||||
- Verified with 570-entity test matching Workshop production scale
|
||||
|
||||
### ⚡ Performance Adjustments
|
||||
|
||||
|
|
@ -2374,7 +1671,7 @@ v5.1.0 delivers a significantly improved developer experience:
|
|||
- **Problem**: In v5.0.0, `saveNoun()` was called before `saveNounMetadata()`, causing TypeAwareStorage to default entity types to 'thing' and save to wrong storage paths
|
||||
- **Impact**: Broke VFS file operations, `brain.get()`, `brain.relate()`, and all features depending on entity metadata
|
||||
- **Solution**: Reversed save order - now saves metadata FIRST, then noun vector
|
||||
- **Fixes**: VFS metadata-missing regression (internal tracker)
|
||||
- **Fixes**: [Workshop Bug Report](https://github.com/soulcraftlabs/brain-cloud/issues/VFS-METADATA-MISSING)
|
||||
|
||||
**Fork API: Lazy COW Initialization**
|
||||
|
||||
|
|
@ -2401,7 +1698,7 @@ v5.1.0 delivers a significantly improved developer experience:
|
|||
|
||||
### 📊 Impact
|
||||
|
||||
* **Unblocks**: a consumer team and all VFS users
|
||||
* **Unblocks**: Workshop team and all VFS users
|
||||
* **Fixes**: All metadata-dependent features (get, relate, find, VFS)
|
||||
* **Maintains**: Full backward compatibility with v4.x data
|
||||
|
||||
|
|
@ -2749,7 +2046,7 @@ This release transforms Brainy imports from entity extractors into true knowledg
|
|||
- **Relationship Type Metadata**: All relationships tagged as `vfs`, `semantic`, or `provenance` for filtering
|
||||
- **Enhanced Column Detection**: 7 relationship types (vs 1 previously) - Location, Owner, Creator, Uses, Member, Friend, Related
|
||||
- **Type-Based Inference**: Smart relationship classification based on entity types and context analysis
|
||||
- **Impact**: A consumer import now creates ~3,900 relationships (vs 581), with 5-20+ connections per entity
|
||||
- **Impact**: Workshop import now creates ~3,900 relationships (vs 581), with 5-20+ connections per entity
|
||||
|
||||
* **import**: New configuration option `createProvenanceLinks` (defaults to `true`)
|
||||
- Enables/disables provenance relationship creation
|
||||
|
|
@ -2791,7 +2088,7 @@ Graph: Rich network, 5-20+ connections per entity
|
|||
|
||||
### [4.7.4](https://github.com/soulcraftlabs/brainy/compare/v4.7.3...v4.7.4) (2025-10-27)
|
||||
|
||||
**CRITICAL SYSTEMIC VFS BUG FIX - A consumer team Unblocked!**
|
||||
**CRITICAL SYSTEMIC VFS BUG FIX - Workshop Team Unblocked!**
|
||||
|
||||
This hotfix resolves a systemic bug affecting ALL storage adapters that caused VFS queries to return empty results even when data existed.
|
||||
|
||||
|
|
@ -2803,7 +2100,7 @@ This hotfix resolves a systemic bug affecting ALL storage adapters that caused V
|
|||
- **Bug Pattern**: `if (!metadata) continue` in getNouns()/getVerbs() methods
|
||||
- **Fixed Locations**: 12 bug sites across 7 adapters (TypeAware, Memory, FileSystem, GCS, S3, R2, OPFS, Azure)
|
||||
- **Solution**: Allow optional metadata with `metadata: (metadata || {}) as NounMetadata`
|
||||
- **Result**: a consumer team UNBLOCKED - VFS entities now queryable
|
||||
- **Result**: Workshop team UNBLOCKED - VFS entities now queryable
|
||||
|
||||
* **neural**: Fix SmartExtractor weighted score threshold bug (28 test failures → 4)
|
||||
- **Root Cause**: Single signal with 0.8 confidence × 0.2 weight = 0.16 < 0.60 threshold
|
||||
|
|
@ -2871,7 +2168,7 @@ Clean separation between VFS (Virtual File System) entities and knowledge graph
|
|||
### 🐛 Critical Bug Fixes
|
||||
|
||||
* **vfs.initializeRoot()**: add includeVFS to prevent duplicate root creation
|
||||
- **Critical Fix**: VFS init was creating ~10 duplicate root entities (a consumer team issue)
|
||||
- **Critical Fix**: VFS init was creating ~10 duplicate root entities (Workshop team issue)
|
||||
- **Root Cause**: `initializeRoot()` called `brain.find()` without `includeVFS: true`, never found existing VFS root
|
||||
- **Impact**: Every `vfs.init()` created a new root, causing empty `readdir('/')` results
|
||||
- **Solution**: Added `includeVFS: true` to root entity lookup (line 171)
|
||||
|
|
@ -2965,7 +2262,7 @@ const files = await vfs.search('documentation')
|
|||
- **Root Cause**: HNSW `rebuild()` and Graph `rebuild()` methods still called `getNounsWithPagination()`/`getVerbsWithPagination()` repeatedly
|
||||
- Each pagination call triggered `getAllShardedFiles()` reading all 256 shard directories
|
||||
- For 1,157 entities: MetadataIndex (2-3s) + HNSW (~20s) + Graph (~10s) = **30-35 seconds total**
|
||||
- a consumer team reported: "v4.2.3 is at batch 7 after ~60 seconds" - still far from claimed 100x improvement
|
||||
- Workshop team reported: "v4.2.3 is at batch 7 after ~60 seconds" - still far from claimed 100x improvement
|
||||
- **Solution**: Apply v4.2.3 adaptive loading pattern to ALL 3 indexes
|
||||
- **FileSystemStorage/MemoryStorage/OPFSStorage**: Load all entities at once (limit: 10000000)
|
||||
- **Cloud storage (GCS/S3/R2/Azure)**: Keep pagination (native APIs are efficient)
|
||||
|
|
@ -2986,7 +2283,7 @@ const files = await vfs.search('documentation')
|
|||
- Graph Index: Loads all verbs at once for local, paginated for cloud (lines 300-361)
|
||||
- Pattern matches v4.2.3 MetadataIndex implementation exactly
|
||||
- Zero config: Completely automatic based on storage adapter type
|
||||
- **Resolution**: Fully resolves a consumer team's v4.2.x performance regression
|
||||
- **Resolution**: Fully resolves Workshop team's v4.2.x performance regression
|
||||
- **Files Changed**:
|
||||
- `src/hnsw/hnswIndex.ts` (updated rebuild() with adaptive loading)
|
||||
- `src/graph/graphAdjacencyIndex.ts` (updated rebuild() with adaptive loading)
|
||||
|
|
@ -3011,7 +2308,7 @@ const files = await vfs.search('documentation')
|
|||
- Pagination was designed for cloud storage socket exhaustion
|
||||
- FileSystem doesn't need pagination - can handle loading thousands of entities at once
|
||||
- Eliminates repeated directory scans: 3 batches × 256 dirs → 1 batch × 256 dirs
|
||||
- **A consumer team**: This resolves the v4.2.2 stalling issue - rebuild will now complete in seconds
|
||||
- **Workshop Team**: This resolves the v4.2.2 stalling issue - rebuild will now complete in seconds
|
||||
- **Files Changed**: `src/utils/metadataIndex.ts` (rebuilt() method with adaptive loading strategy)
|
||||
|
||||
### [4.2.2](https://github.com/soulcraftlabs/brainy/compare/v4.2.1...v4.2.2) (2025-10-23)
|
||||
|
|
@ -3053,7 +2350,7 @@ const files = await vfs.search('documentation')
|
|||
- Universal: All storage adapters (FileSystem, GCS, S3, R2, Memory, OPFS)
|
||||
- **Zero Config**: Completely automatic, no configuration needed
|
||||
- **Self-Healing**: Gracefully handles missing/corrupt registry (rebuilds once)
|
||||
- **Impact**: Fixes a consumer team bug report - production-ready at billion scale
|
||||
- **Impact**: Fixes Workshop team bug report - production-ready at billion scale
|
||||
- **Files Changed**: `src/utils/metadataIndex.ts` (added saveFieldRegistry/loadFieldRegistry methods, updated init/flush)
|
||||
|
||||
### [4.2.0](https://github.com/soulcraftlabs/brainy/compare/v4.1.4...v4.2.0) (2025-10-23)
|
||||
|
|
@ -3097,7 +2394,7 @@ const files = await vfs.search('documentation')
|
|||
- Fixed property access bugs: `verb.target` → `verb.to`, `verb.verb` → `verb.type`
|
||||
- Added comprehensive integration tests (14 tests covering all query patterns)
|
||||
- Updated JSDoc documentation with usage examples
|
||||
- **Impact**: Resolves a consumer team bug where 524 imported relationships were inaccessible
|
||||
- **Impact**: Resolves Workshop team bug where 524 imported relationships were inaccessible
|
||||
- **Breaking**: None - fully backward compatible
|
||||
|
||||
### [4.1.2](https://github.com/soulcraftlabs/brainy/compare/v4.1.1...v4.1.2) (2025-10-21)
|
||||
|
|
|
|||
208
CLAUDE.md
208
CLAUDE.md
|
|
@ -1,208 +0,0 @@
|
|||
# Brainy - Claude Code Project Guide
|
||||
|
||||
This file provides guidance for Claude Code (and human contributors) when working on the Brainy codebase.
|
||||
|
||||
## Cross-Project Coordination
|
||||
|
||||
Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
|
||||
|
||||
**At session START:** Read the handoff. Find rows where Owner = Brainy. Act on those first.
|
||||
|
||||
**At session END:** Mark completed actions ✅, delete rows you finished, delete threads with zero remaining actions. File must not grow. **If you shipped anything consumers need to know about, update `RELEASES.md` before closing.**
|
||||
|
||||
**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
|
||||
|
||||
**Current version:** run `npm view @soulcraft/brainy version` (never trust a hardcoded number here — this line went stale for months); consumer-facing changes tracked in `RELEASES.md`
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraft/brainy` on npm under the MIT license.
|
||||
|
||||
## Getting Started
|
||||
|
||||
```bash
|
||||
npm install # Install dependencies
|
||||
npm run build # Build the project
|
||||
npm test # Run test suite (Vitest)
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
Full architecture reference: `.claude/skills/architecture.md`
|
||||
|
||||
### Core Systems
|
||||
- **Storage** (`src/storage/`): Pluggable storage backends via StorageAdapter interface (`src/coreTypes.ts`)
|
||||
- **Vector Search** (`src/hnsw/`): HNSW approximate nearest neighbor search
|
||||
- **Graph Engine** (`src/graph/`): Relationship traversal with adjacency index and pathfinding
|
||||
- **Metadata Index** (`src/utils/metadataIndex.ts`): O(1) exact match, O(log n) range queries
|
||||
- **Triple Intelligence** (`src/triple/`): Unified query combining all three intelligence types
|
||||
- **Aggregation Engine** (`src/aggregation/`): Write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows
|
||||
- **Virtual Filesystem** (`src/vfs/`): Full VFS with semantic search
|
||||
|
||||
### Type System
|
||||
- **NounType** (42 types): Entity classification -- Person, Concept, Collection, Document, Task, etc.
|
||||
- **VerbType** (127 types): Relationship types -- Contains, RelatedTo, PartOf, Creates, DependsOn, etc.
|
||||
- Defined in `src/types/graphTypes.ts`
|
||||
|
||||
## Code Standards
|
||||
|
||||
### TypeScript
|
||||
- Strict mode enabled
|
||||
- Target: ES2020, NodeNext module resolution
|
||||
- All new code must be TypeScript
|
||||
- Follow existing patterns -- read related code before writing
|
||||
|
||||
### Quality
|
||||
- All code must compile without errors
|
||||
- All code must have working tests that exercise real behavior
|
||||
- No stub returns (`return {} as any`)
|
||||
- No incomplete implementations with TODO comments
|
||||
- If something can't be fully implemented, throw an explicit error rather than faking it
|
||||
|
||||
### Verification Before Code Changes
|
||||
1. Check that interfaces and methods actually exist before using them
|
||||
2. Check that type properties are in the type definitions
|
||||
3. Run `npm test` -- tests must pass
|
||||
4. Run `npm run build` -- build must succeed
|
||||
|
||||
### Testing
|
||||
- Framework: Vitest
|
||||
- Tests in `tests/` (unit, integration, benchmarks, comprehensive)
|
||||
- Use in-memory storage for speed where possible
|
||||
- Tests must exercise real behavior, not mock it
|
||||
- Benchmarks are in `tests/benchmarks/` (not tests/performance/)
|
||||
|
||||
## Commit Conventions
|
||||
|
||||
Use [Conventional Commits](https://www.conventionalcommits.org/):
|
||||
|
||||
```
|
||||
feat: add new feature (minor version bump)
|
||||
fix: resolve bug (patch version bump)
|
||||
docs: update documentation (patch version bump)
|
||||
perf: improve performance (patch version bump)
|
||||
refactor: restructure code (patch version bump)
|
||||
test: add/update tests (patch version bump)
|
||||
```
|
||||
|
||||
**Important:** Never use `BREAKING CHANGE` in commit messages. Major version bumps are manual decisions only (`npm run release:major`).
|
||||
|
||||
## Docs Pipeline — soulcraft.com/docs
|
||||
|
||||
Docs in `docs/**/*.md` are published with the npm package (included in `files`) and synced to soulcraft.com/docs on every portal deploy. Frontmatter controls what appears publicly.
|
||||
|
||||
### Docs check triggers
|
||||
|
||||
Run the docs check whenever the user says ANY of:
|
||||
- "commit, publish, release" / "release" / "publish"
|
||||
- "update the docs" / "make sure docs are accurate" / "check the docs"
|
||||
- "review docs" / "clean up docs"
|
||||
|
||||
### Pre-release docs check (MANDATORY before every release)
|
||||
|
||||
When the user says "commit, publish, release" or any variation, **before committing**:
|
||||
|
||||
1. **Scan all files changed in this session** (and any recently added `docs/*.md` files)
|
||||
2. For each changed/new doc, decide: is this useful to external users?
|
||||
- **Yes** → ensure it has complete frontmatter (add or update it)
|
||||
- **No** (internal, migration, dev-only) → ensure it has no frontmatter or `public: false`
|
||||
3. For docs that already have frontmatter, verify:
|
||||
- `description` still matches the actual content
|
||||
- `next` links still exist and are still the right follow-up pages
|
||||
- `title` matches the doc's h1
|
||||
4. Include frontmatter changes in the commit
|
||||
|
||||
### Frontmatter format
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: Human-readable title
|
||||
slug: category/page-name # URL: soulcraft.com/docs/category/page-name
|
||||
public: true # false or absent = not published
|
||||
category: getting-started | concepts | guides | api
|
||||
template: guide | concept | api # controls layout on soulcraft.com
|
||||
order: 1 # sidebar position within category (lower = first)
|
||||
description: One sentence. What this doc covers and why it matters.
|
||||
next: # "Next steps" links shown at bottom of page
|
||||
- category/other-slug
|
||||
---
|
||||
```
|
||||
|
||||
### Category guide
|
||||
|
||||
| category | use for |
|
||||
|----------|---------|
|
||||
| `getting-started` | installation, quick start, first steps |
|
||||
| `concepts` | how the system works, mental models |
|
||||
| `guides` | how to do specific things, recipes |
|
||||
| `api` | method reference, signatures, parameters |
|
||||
|
||||
### What stays internal (no frontmatter / `public: false`)
|
||||
|
||||
- Release guides, developer learning paths
|
||||
- Migration guides for old versions (v3→v4, v5.11)
|
||||
- Architecture analysis docs (clustering algorithms, etc.)
|
||||
- Anything in `docs/internal/`
|
||||
- Deployment/ops/cost docs (cloud-run, kubernetes, cost-optimization)
|
||||
|
||||
## Release Process
|
||||
|
||||
Fully automated via `scripts/release.sh`:
|
||||
|
||||
```bash
|
||||
npm run release:dry # Preview (no changes)
|
||||
npm run release:patch # Bug fixes
|
||||
npm run release:minor # New features
|
||||
npm run release:major # Breaking changes (rare, manual decision)
|
||||
```
|
||||
|
||||
The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release.
|
||||
|
||||
After a successful release, remind the user:
|
||||
> "Published. Deploy portal to pick up the new docs → go to the portal project and deploy."
|
||||
|
||||
Do NOT deploy portal from here. Portal is always deployed separately from within the portal project.
|
||||
|
||||
## Closed-Source Product Names — HARD RULE
|
||||
|
||||
Brainy is the only Soulcraft open-source project. Nothing in this repo — code, JSDoc, tests,
|
||||
docs, RELEASES.md, CHANGELOG.md, commit messages — may reference closed-source Soulcraft
|
||||
products by name (Workshop, Venue, Memory, Muse, Hall, Forge, Academy, Pulse, Heart,
|
||||
Collective, SDK) or by their specific class/method names (`BookingDraftService`,
|
||||
`getDemandHeatmap`, `systemKind`, etc.).
|
||||
|
||||
When recording a consumer-reported bug, regression scenario, or release note:
|
||||
- Refer to "a consumer", "a downstream application", "a production deployment", or "an
|
||||
internal report" — never name the product.
|
||||
- All doc examples must use generic domain values (`'employee'`, `'customer'`, `'invoice'`,
|
||||
`'milestone'`, `OrderService`, `/orders/...`), not product-specific schemas.
|
||||
- Internal session artifacts (`.strategy/`, `~/.claude/plans/`, handoff files outside the
|
||||
repo) MAY name products — those are not public.
|
||||
|
||||
If you catch yourself typing a product name into a tracked file, stop and rephrase.
|
||||
|
||||
## Performance Claims
|
||||
|
||||
When documenting performance characteristics:
|
||||
- **MEASURED**: Cite the test file and line number
|
||||
- **PROJECTED**: Clearly label as extrapolated from tested scale
|
||||
- Never claim a performance figure without context or evidence
|
||||
|
||||
## Debugging
|
||||
|
||||
When a bug persists through 2+ fix attempts, switch to systematic debugging:
|
||||
1. Add comprehensive logging at every step
|
||||
2. Test with production-like data
|
||||
3. Trace the complete execution path
|
||||
4. Check both library code and consumer code
|
||||
5. Verify with actual test execution before declaring fixed
|
||||
|
||||
## Key Paths
|
||||
|
||||
- Main class: `src/brainy.ts`
|
||||
- Public API: `src/index.ts` (38+ exports)
|
||||
- Storage interface: `src/coreTypes.ts`
|
||||
- Type definitions: `src/types/`
|
||||
- Strategy/planning docs: `.strategy/` (gitignored, not public)
|
||||
318
CONTRIBUTING.md
318
CONTRIBUTING.md
|
|
@ -1,66 +1,298 @@
|
|||
# Contributing to Brainy
|
||||
|
||||
Brainy is MIT-licensed and genuinely open to outside contributions. This page
|
||||
is the honest, current path — please don't rely on older instructions you
|
||||
may find elsewhere in the repo's history.
|
||||
Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for contributing to the project.
|
||||
|
||||
## Where the project lives
|
||||
## Code of Conduct
|
||||
|
||||
The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/brainy**.
|
||||
It's anonymously readable and cloneable — no account needed to browse, clone,
|
||||
or build.
|
||||
By participating in this project, you agree to abide by our Code of Conduct:
|
||||
- Be respectful and inclusive
|
||||
- Welcome newcomers and help them get started
|
||||
- Focus on constructive criticism
|
||||
- Respect differing viewpoints and experiences
|
||||
|
||||
## How to contribute
|
||||
## How to Contribute
|
||||
|
||||
**Found a bug, or have an idea?** Email **brainy@soulcraft.com**. No account,
|
||||
no ceremony — you'll get a receipt, and it goes to a human.
|
||||
### Reporting Issues
|
||||
|
||||
**Want to send a patch?** Two ways, both first-class:
|
||||
Before creating an issue, please check existing issues to avoid duplicates.
|
||||
|
||||
- **Email a patch.** Run `git format-patch` against your change and email the
|
||||
output to **brainy@soulcraft.com**. This is a genuinely supported path, not
|
||||
a fallback — plenty of good contributions arrive this way.
|
||||
- **Open a pull request on the forge.** Request an account at
|
||||
**source.soulcraft.com** (registration is request-with-approval, so allow
|
||||
a little lag), clone, push a branch, and open a PR there. Maintainers
|
||||
review and land it.
|
||||
When creating an issue, include:
|
||||
- Clear, descriptive title
|
||||
- Detailed description of the problem
|
||||
- Steps to reproduce
|
||||
- Expected vs actual behavior
|
||||
- System information (OS, Node version, Brainy version)
|
||||
- Code examples if applicable
|
||||
|
||||
Either way, for anything beyond a small fix, opening an issue first (email is
|
||||
fine) to talk through the approach saves everyone rework.
|
||||
### Suggesting Features
|
||||
|
||||
## Development setup
|
||||
Feature requests are welcome! Please provide:
|
||||
- Clear use case
|
||||
- Proposed API/interface
|
||||
- Examples of how it would work
|
||||
- Any potential challenges or considerations
|
||||
|
||||
### Pull Requests
|
||||
|
||||
#### Before Starting
|
||||
|
||||
1. Check existing issues and PRs
|
||||
2. Open an issue to discuss significant changes
|
||||
3. Fork the repository
|
||||
4. Create a feature branch from `main`
|
||||
|
||||
#### Development Setup
|
||||
|
||||
**Quick Setup (Recommended):**
|
||||
```bash
|
||||
git clone https://source.soulcraft.com/soulcraft/brainy.git
|
||||
# Clone your fork
|
||||
git clone https://github.com/your-username/brainy.git
|
||||
cd brainy
|
||||
|
||||
# Run setup script (installs all dependencies including Rust)
|
||||
./scripts/setup-dev.sh
|
||||
```
|
||||
|
||||
**Manual Setup:**
|
||||
```bash
|
||||
# Clone your fork
|
||||
git clone https://github.com/your-username/brainy.git
|
||||
cd brainy
|
||||
|
||||
# Install system dependencies (Ubuntu/Debian)
|
||||
sudo apt-get install -y build-essential pkg-config libssl-dev
|
||||
|
||||
# Install Rust (for WASM embedding engine)
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
source ~/.cargo/env
|
||||
rustup target add wasm32-unknown-unknown
|
||||
cargo install wasm-pack
|
||||
|
||||
# Install Node.js dependencies
|
||||
npm install
|
||||
|
||||
# Build Candle WASM embedding engine
|
||||
npm run build:candle
|
||||
|
||||
# Build TypeScript
|
||||
npm run build
|
||||
|
||||
# Run tests
|
||||
npm test
|
||||
```
|
||||
|
||||
Tests run on [Vitest](https://vitest.dev/). `npm test` runs the unit suite;
|
||||
see `package.json` for `test:integration`, `test:coverage`, and friends.
|
||||
#### Making Changes
|
||||
|
||||
## Standards
|
||||
1. **Follow the code style**
|
||||
- TypeScript for all source code
|
||||
- Clear variable and function names
|
||||
- Comments for complex logic
|
||||
- JSDoc for public APIs
|
||||
|
||||
- **Strict TypeScript.** No `any` escape hatches to dodge the type checker.
|
||||
- **Tests exercise real behavior.** No mocking away the thing you're supposed
|
||||
to be testing.
|
||||
- **No stubs, no TODO-code.** If something can't be finished, say so and
|
||||
leave it out — don't merge a placeholder.
|
||||
- **JSDoc on every exported function, class, and type.**
|
||||
- **[Conventional Commits](https://www.conventionalcommits.org/).** `feat:`,
|
||||
`fix:`, `docs:`, `perf:`, `refactor:`, `test:`, `chore:`. Never
|
||||
`BREAKING CHANGE` in a commit message — major version bumps are a separate,
|
||||
deliberate decision.
|
||||
- **Performance claims are measured or labeled projected.** If a PR or its
|
||||
description states a number, cite the benchmark that produced it (see
|
||||
[docs/performance-envelopes.md](docs/performance-envelopes.md) for the
|
||||
pattern). Don't state an estimate as if it were measured.
|
||||
2. **Write tests**
|
||||
- Add tests for new features
|
||||
- Update tests for changes
|
||||
- Ensure all tests pass
|
||||
|
||||
## License
|
||||
3. **Update documentation**
|
||||
- Update README if needed
|
||||
- Add/update API documentation
|
||||
- Include examples
|
||||
|
||||
Brainy is [MIT licensed](LICENSE). Contributions are accepted under the same
|
||||
license — there's no CLA to sign.
|
||||
#### Commit Guidelines
|
||||
|
||||
Thank you for considering a contribution.
|
||||
Follow conventional commits format:
|
||||
|
||||
```
|
||||
type(scope): description
|
||||
|
||||
[optional body]
|
||||
|
||||
[optional footer]
|
||||
```
|
||||
|
||||
Types:
|
||||
- `feat`: New feature
|
||||
- `fix`: Bug fix
|
||||
- `docs`: Documentation changes
|
||||
- `style`: Code style changes
|
||||
- `refactor`: Code refactoring
|
||||
- `perf`: Performance improvements
|
||||
- `test`: Test changes
|
||||
- `chore`: Build/tooling changes
|
||||
|
||||
Examples:
|
||||
```bash
|
||||
feat(triple): add graph traversal depth limit
|
||||
fix(storage): handle concurrent write conflicts
|
||||
docs(api): update search method documentation
|
||||
```
|
||||
|
||||
#### Submitting PR
|
||||
|
||||
1. Push to your fork
|
||||
2. Create PR against `main` branch
|
||||
3. Fill out PR template
|
||||
4. Ensure CI checks pass
|
||||
5. Wait for review
|
||||
|
||||
### Testing
|
||||
|
||||
#### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
npm test
|
||||
|
||||
# Run specific test file
|
||||
npm test tests/core.test.ts
|
||||
|
||||
# Run with coverage
|
||||
npm run test:coverage
|
||||
|
||||
# Watch mode
|
||||
npm run test:watch
|
||||
```
|
||||
|
||||
#### Writing Tests
|
||||
|
||||
```typescript
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { Brainy } from '../src'
|
||||
|
||||
describe('Feature Name', () => {
|
||||
it('should do something specific', async () => {
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Test implementation
|
||||
const result = await brain.search("test")
|
||||
|
||||
expect(result).toBeDefined()
|
||||
expect(result.length).toBeGreaterThan(0)
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
## Architecture Guidelines
|
||||
|
||||
### Adding New Features
|
||||
|
||||
1. **Check existing functionality**
|
||||
- Review `ARCHITECTURE.md`
|
||||
- Check if similar features exist
|
||||
- Consider if it should be an augmentation
|
||||
|
||||
2. **Design considerations**
|
||||
- Maintain backward compatibility
|
||||
- Consider performance impact
|
||||
- Think about all storage adapters
|
||||
- Plan for extensibility
|
||||
|
||||
3. **Implementation checklist**
|
||||
- [ ] Core functionality
|
||||
- [ ] Tests (unit and integration)
|
||||
- [ ] Documentation
|
||||
- [ ] TypeScript types
|
||||
- [ ] Examples
|
||||
- [ ] Performance benchmarks (if applicable)
|
||||
|
||||
### Creating Augmentations
|
||||
|
||||
Augmentations extend Brainy's functionality:
|
||||
|
||||
```typescript
|
||||
import { BrainyAugmentation } from '../types'
|
||||
|
||||
export class MyAugmentation extends BrainyAugmentation {
|
||||
name = 'MyAugmentation'
|
||||
|
||||
async onInit(brain: Brainy): Promise<void> {
|
||||
// Initialize augmentation
|
||||
}
|
||||
|
||||
async onAdd(item: any, brain: Brainy): Promise<any> {
|
||||
// Process before adding
|
||||
return item
|
||||
}
|
||||
|
||||
async onSearch(query: any, results: any[], brain: Brainy): Promise<any[]> {
|
||||
// Process search results
|
||||
return results
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Performance Considerations
|
||||
|
||||
- Use batch operations where possible
|
||||
- Implement caching strategically
|
||||
- Consider memory usage
|
||||
- Profile performance impacts
|
||||
- Add benchmarks for critical paths
|
||||
|
||||
## Documentation
|
||||
|
||||
### API Documentation
|
||||
|
||||
Use JSDoc for all public APIs:
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Searches for similar items using vector similarity
|
||||
* @param query - Search query (text or vector)
|
||||
* @param options - Search options
|
||||
* @returns Array of search results with scores
|
||||
* @example
|
||||
* ```typescript
|
||||
* const results = await brain.search("machine learning", { limit: 10 })
|
||||
* ```
|
||||
*/
|
||||
async search(query: string | Vector, options?: SearchOptions): Promise<SearchResult[]> {
|
||||
// Implementation
|
||||
}
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
Add examples for new features:
|
||||
|
||||
```typescript
|
||||
// examples/feature-name.ts
|
||||
import { Brainy } from 'brainy'
|
||||
|
||||
async function exampleUsage() {
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Show feature usage
|
||||
// Include comments explaining what's happening
|
||||
// Handle errors appropriately
|
||||
}
|
||||
|
||||
exampleUsage().catch(console.error)
|
||||
```
|
||||
|
||||
## Release Process
|
||||
|
||||
1. **Version bump**: Follow semantic versioning
|
||||
2. **Update CHANGELOG**: Document all changes
|
||||
3. **Run tests**: Ensure all tests pass
|
||||
4. **Build**: Generate distribution files
|
||||
5. **Tag**: Create git tag for version
|
||||
6. **Publish**: Release to npm
|
||||
|
||||
## Getting Help
|
||||
|
||||
- **Discord**: Join our community
|
||||
- **Issues**: Ask questions on GitHub
|
||||
- **Discussions**: Share ideas and get feedback
|
||||
|
||||
## Recognition
|
||||
|
||||
Contributors will be recognized in:
|
||||
- CHANGELOG.md for their contributions
|
||||
- README.md contributors section
|
||||
- GitHub contributors page
|
||||
|
||||
Thank you for contributing to Brainy! 🧠
|
||||
815
README.md
815
README.md
|
|
@ -1,225 +1,770 @@
|
|||
<p align="center">
|
||||
<img src="https://raw.githubusercontent.com/soulcraftlabs/brainy/main/brainy.png" alt="Brainy" width="180">
|
||||
</p>
|
||||
|
||||
<h1 align="center">Brainy</h1>
|
||||
# Brainy
|
||||
|
||||
<p align="center">
|
||||
<b>Three database paradigms. One API. Zero configuration.</b><br>
|
||||
The in-process knowledge database for TypeScript — vector search, graph traversal,<br>
|
||||
and metadata filtering unified in a single query.
|
||||
<img src="https://raw.githubusercontent.com/soulcraftlabs/brainy/main/brainy.png" alt="Brainy Logo" width="200">
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/v/@soulcraft/brainy.svg" alt="npm version"></a>
|
||||
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/dm/@soulcraft/brainy.svg" alt="npm downloads"></a>
|
||||
<a href="https://source.soulcraft.com/soulcraft/brainy/actions"><img src="https://source.soulcraft.com/soulcraft/brainy/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"></a>
|
||||
<a href="https://soulcraft.com/docs"><img src="https://img.shields.io/badge/docs-soulcraft.com-blue.svg" alt="Documentation"></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a>
|
||||
<a href="https://www.typescriptlang.org/"><img src="https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg" alt="TypeScript"></a>
|
||||
</p>
|
||||
[](https://www.npmjs.com/package/@soulcraft/brainy)
|
||||
[](https://www.npmjs.com/package/@soulcraft/brainy)
|
||||
[](https://soulcraft.com/docs)
|
||||
[](LICENSE)
|
||||
[](https://www.typescriptlang.org/)
|
||||
|
||||
<p align="center">
|
||||
<a href="#quick-start">Quick start</a> ·
|
||||
<a href="#one-query-three-engines">One query</a> ·
|
||||
<a href="#feature-tour">Features</a> ·
|
||||
<a href="#when-you-outgrow-brainy">Scale with Cor</a> ·
|
||||
<a href="#documentation">Docs</a> ·
|
||||
<a href="#support--community">Support</a>
|
||||
</p>
|
||||
## The Knowledge Operating System
|
||||
|
||||
**Every piece of knowledge in your application — living, connected, and intelligent.**
|
||||
|
||||
Stop fighting with vector databases, graph databases, and document stores. Stop stitching together Pinecone + Neo4j + MongoDB. **Brainy does all three, in one elegant API, from prototype to planet-scale.**
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// That's it. You now have semantic search, graph relationships,
|
||||
// and document filtering. Zero configuration. Just works.
|
||||
```
|
||||
|
||||
**Built by developers who were tired of:**
|
||||
- Spending weeks configuring embeddings, indexes, and schemas
|
||||
- Choosing between vector similarity OR graph relationships OR metadata filtering
|
||||
- Rewriting everything when you need to scale from 1,000 to 1,000,000,000 entities
|
||||
|
||||
**Brainy makes the impossible simple: All three paradigms. One API. Any scale.**
|
||||
|
||||
---
|
||||
|
||||
Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact **three ways at once** and lets one call query them together:
|
||||
## 👉 Choose Your Path
|
||||
|
||||
| You write | Brainy indexes it as | You query it with |
|
||||
|---|---|---|
|
||||
| `data: 'Ada wrote the first program'` | a **384-dim vector** (local embedding — no API key) | `find({ query: 'computing pioneers' })` |
|
||||
| `metadata: { field: 'CS', year: 1843 }` | **structured fields** (O(1) exact, O(log n) range) | `find({ where: { year: { lessThan: 1900 } } })` |
|
||||
| `relate({ from: ada, to: babbage })` | a **typed, directed graph edge** | `find({ connected: { to: babbage, depth: 2 } })` |
|
||||
**New to Brainy? Pick your starting point:**
|
||||
|
||||
It runs **inside your process** — no server, no Docker, nothing to operate — and persists to plain files you can snapshot with a hard link.
|
||||
### 🚀 Path 1: I want to build something NOW
|
||||
**→ [Full Documentation](https://soulcraft.com/docs)** ⭐ **Most developers start here** ⭐
|
||||
- Interactive API reference with examples
|
||||
- Quick start guides and tutorials
|
||||
- Copy-paste ready code snippets
|
||||
- **This is your primary resource**
|
||||
|
||||
**New here?** → **[What is Brainy? — plain-language overview, no jargon](docs/eli5.md)**
|
||||
### 🧠 Path 2: I want to understand the big picture first
|
||||
**→ Keep reading below** for demos, architecture, and use cases
|
||||
|
||||
## Quick start
|
||||
### 📊 Path 3: I'm evaluating database options
|
||||
**→ Jump to [Why Revolutionary](#why-brainy-is-revolutionary)** or **[Benchmarks](#benchmarks)**
|
||||
|
||||
```bash
|
||||
bun add @soulcraft/brainy # Bun ≥ 1.1 — recommended
|
||||
npm install @soulcraft/brainy # Node.js ≥ 22
|
||||
```
|
||||
---
|
||||
|
||||
## See It In Action
|
||||
|
||||
**30 seconds to understand why Brainy is different:**
|
||||
|
||||
```javascript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy() // in-memory; one line swaps to disk
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Text auto-embeds locally; metadata auto-indexes
|
||||
const react = await brain.add({
|
||||
data: 'React is a JavaScript library for building user interfaces',
|
||||
type: NounType.Concept,
|
||||
subtype: 'library',
|
||||
metadata: { category: 'frontend', year: 2013 }
|
||||
// Add knowledge with context
|
||||
const reactId = await brain.add({
|
||||
data: "React is a JavaScript library for building user interfaces",
|
||||
type: NounType.Concept,
|
||||
metadata: { category: "frontend", year: 2013 }
|
||||
})
|
||||
|
||||
const next = await brain.add({
|
||||
data: 'Next.js is a React framework with server-side rendering',
|
||||
type: NounType.Concept,
|
||||
subtype: 'framework',
|
||||
metadata: { category: 'frontend', year: 2016 }
|
||||
const nextId = await brain.add({
|
||||
data: "Next.js framework for React with server-side rendering",
|
||||
type: NounType.Concept,
|
||||
metadata: { category: "framework", year: 2016 }
|
||||
})
|
||||
|
||||
await brain.relate({ from: next, to: react, type: VerbType.DependsOn, subtype: 'runtime' })
|
||||
```
|
||||
// Create relationships
|
||||
await brain.relate({ from: nextId, to: reactId, type: VerbType.BuiltOn })
|
||||
|
||||
## One query, three engines
|
||||
|
||||
```javascript
|
||||
// NOW THE MAGIC: Query with natural language
|
||||
const results = await brain.find({
|
||||
query: 'modern frontend frameworks', // vector — what it means
|
||||
where: { year: { greaterThan: 2015 } }, // metadata — what it is
|
||||
connected: { to: react, depth: 2 } // graph — what it touches
|
||||
query: "modern frontend frameworks", // 🔍 Vector similarity
|
||||
where: { year: { greaterThan: 2015 } }, // 📊 Document filtering
|
||||
connected: { to: reactId, depth: 2 } // 🕸️ Graph traversal
|
||||
})
|
||||
|
||||
// ALL THREE PARADIGMS. ONE QUERY. 10ms response time.
|
||||
```
|
||||
|
||||
**This is impossible with traditional databases.** Brainy makes it trivial.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/brainy
|
||||
```
|
||||
|
||||
### Your First Knowledge Graph (60 seconds)
|
||||
|
||||
```javascript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Add knowledge
|
||||
const jsId = await brain.add({
|
||||
data: "JavaScript is a programming language",
|
||||
type: NounType.Concept,
|
||||
metadata: { category: "language", year: 1995 }
|
||||
})
|
||||
|
||||
const nodeId = await brain.add({
|
||||
data: "Node.js runtime environment",
|
||||
type: NounType.Concept,
|
||||
metadata: { category: "runtime", year: 2009 }
|
||||
})
|
||||
|
||||
// Create relationships
|
||||
await brain.relate({ from: nodeId, to: jsId, type: VerbType.Executes })
|
||||
|
||||
// Query with Triple Intelligence
|
||||
const results = await brain.find({
|
||||
query: "JavaScript", // 🔍 Vector
|
||||
where: { category: "language" }, // 📊 Document
|
||||
connected: { from: nodeId, depth: 1 } // 🕸️ Graph
|
||||
})
|
||||
```
|
||||
|
||||
Every clause is optional; any combination composes. Under the hood Brainy plans the query across an HNSW vector index, a roaring-bitmap field index, and an adjacency graph index — and re-validates every result against your predicate before returning it, so a corrupt index can never hand you a wrong answer.
|
||||
**Done.** No configuration. No complexity. Production-ready from day one.
|
||||
|
||||
## Feature tour
|
||||
**→ Ready to dive deeper? [Complete API Documentation](docs/api/README.md)** has every method with examples.
|
||||
|
||||
### The database is a value
|
||||
---
|
||||
|
||||
Pin it, rewind it, fork it. Snapshot isolation without a server.
|
||||
## Entity Extraction (NEW in v5.7.6)
|
||||
|
||||
**Extract entities from text with AI-powered classification:**
|
||||
|
||||
```javascript
|
||||
const db = brain.now() // pin current state — O(1)
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
await brain.transact([ // atomic all-or-nothing, CAS-guarded
|
||||
{ op: 'update', id: order, metadata: { status: 'paid' } },
|
||||
{ op: 'relate', from: invoice, to: order, type: VerbType.References, subtype: 'billing' }
|
||||
], { ifAtGeneration: db.generation })
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
await db.get(order) // still 'pending' — pinned forever
|
||||
await brain.get(order) // 'paid' — live
|
||||
// Extract all entities
|
||||
const entities = await brain.extractEntities('John Smith founded Acme Corp in New York')
|
||||
// Returns:
|
||||
// [
|
||||
// { text: 'John Smith', type: NounType.Person, confidence: 0.95 },
|
||||
// { text: 'Acme Corp', type: NounType.Organization, confidence: 0.92 },
|
||||
// { text: 'New York', type: NounType.Location, confidence: 0.88 }
|
||||
// ]
|
||||
|
||||
const lastWeek = await brain.asOf(Date.now() - 7 * 86_400_000) // full query surface, past state
|
||||
const whatIf = await db.with([{ op: 'remove', id: order }]) // speculative — never touches disk
|
||||
await brain.now().persist('/backups/today') // instant hard-link snapshot
|
||||
// Extract with filters
|
||||
const people = await brain.extractEntities(resume, {
|
||||
types: [NounType.Person],
|
||||
confidence: 0.8
|
||||
})
|
||||
|
||||
// Advanced: Direct access to extractors
|
||||
import { SmartExtractor } from '@soulcraft/brainy'
|
||||
|
||||
const extractor = new SmartExtractor(brain, { minConfidence: 0.7 })
|
||||
const result = await extractor.extract('CEO', {
|
||||
formatContext: { format: 'excel', columnHeader: 'Title' }
|
||||
})
|
||||
```
|
||||
|
||||
**[Consistency model](docs/concepts/consistency-model.md)** · **[Snapshots & time travel](docs/guides/snapshots-and-time-travel.md)**
|
||||
**Features:**
|
||||
- 🎯 **4-Signal Ensemble** - ExactMatch (40%) + Embedding (35%) + Pattern (20%) + Context (5%)
|
||||
- 📊 **Format Intelligence** - Adapts to Excel, CSV, PDF, YAML, DOCX, JSON, Markdown
|
||||
- ⚡ **Fast** - ~15-20ms per extraction with LRU caching
|
||||
- 🌍 **42 Types** - Person, Organization, Location, Document, and 38 more
|
||||
|
||||
### Local embeddings — no API keys
|
||||
**→ [Neural Extraction Guide](docs/neural-extraction.md)** | **[Import Preview Mode](docs/neural-extraction.md#import-preview-mode)**
|
||||
|
||||
Strings embed on-device with a bundled MiniLM model (WASM). Semantic search works offline, in CI, and on air-gapped machines, at zero cost per call. Hybrid keyword + semantic ranking is the default:
|
||||
---
|
||||
|
||||
## From Prototype to Planet Scale
|
||||
|
||||
**The same API. Zero rewrites. Any scale.**
|
||||
|
||||
### 👤 Individual Developer → Weekend Prototype
|
||||
```javascript
|
||||
await brain.find({ query: 'David Smith' }) // auto: text + semantic
|
||||
await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // semantic only
|
||||
const brain = new Brainy() // Zero config, starts in memory
|
||||
await brain.init()
|
||||
```
|
||||
**Perfect for:** Hackathons, side projects, prototyping, learning
|
||||
|
||||
### 👥 Small Team → Production MVP
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './data', compression: true }
|
||||
})
|
||||
```
|
||||
**Scale:** Thousands to hundreds of thousands • **Performance:** <5ms queries
|
||||
**→ [Production Service Architecture](docs/PRODUCTION_SERVICE_ARCHITECTURE.md)** — Singleton patterns, caching, and scaling for Bun/Node.js services
|
||||
|
||||
### 🏢 Growing Company → Multi-Million Scale
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 's3', s3Storage: { bucketName: 'my-kb', region: 'us-east-1' } },
|
||||
hnsw: { typeAware: true } // 87% memory reduction
|
||||
})
|
||||
```
|
||||
**Scale:** Millions of entities • **Performance:** <10ms queries, 12GB @ 10M entities
|
||||
|
||||
### 🌍 Enterprise → Billion+ Scale
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'gcs', gcsStorage: { bucketName: 'global-kb' } },
|
||||
hnsw: { typeAware: true, M: 32, efConstruction: 400 }
|
||||
})
|
||||
```
|
||||
**Scale:** Billions (tested @ 1B+) • **Performance:** 18ms queries, 50GB memory
|
||||
**Cost:** $138k/year → $6k/year with intelligent tiering (96% savings)
|
||||
|
||||
**→ [Capacity Planning Guide](docs/operations/capacity-planning.md)** | **[Cost Optimization](docs/operations/)**
|
||||
|
||||
### 🎯 The Point
|
||||
|
||||
**Start simple. Scale infinitely. Never rewrite.**
|
||||
|
||||
Most systems make you choose: Simple (SQLite) OR Scalable (Kubernetes + 7 databases).
|
||||
**Brainy gives you both.** Starts simple as SQLite. Scales like Google.
|
||||
|
||||
---
|
||||
|
||||
## Why Brainy Is Revolutionary
|
||||
|
||||
### 🧠 **Triple Intelligence™** — The Impossible Made Possible
|
||||
|
||||
**The world's first to unify three database paradigms in ONE API:**
|
||||
|
||||
| What You Get | Like Having | But Unified |
|
||||
|-------------|-------------|-------------|
|
||||
| 🔍 **Vector Search** | Pinecone, Weaviate | Find by meaning |
|
||||
| 🕸️ **Graph Relationships** | Neo4j, ArangoDB | Navigate connections |
|
||||
| 📊 **Document Filtering** | MongoDB, Elasticsearch | Query metadata |
|
||||
|
||||
**Every other system makes you choose.** Brainy does all three together.
|
||||
|
||||
**Why this matters:** Your data isn't just vectors or just documents or just graphs. It's all three at once. A research paper is semantically similar to other papers (vector), written by an author (graph), and published in 2023 (document). **Brainy is the only system that understands this.**
|
||||
|
||||
### 🎯 **42 Noun Types × 127 Verb Types = Universal Protocol**
|
||||
|
||||
Model **any domain** with mathematical completeness:
|
||||
|
||||
```
|
||||
42 Nouns × 127 Verbs × ∞ Metadata = 5,334+ base combinations
|
||||
Stage 3 CANONICAL: 96-97% coverage of all human knowledge
|
||||
```
|
||||
|
||||
### A typed graph, not a bag of edges
|
||||
**Real-world expressiveness:**
|
||||
- Healthcare: `Patient → diagnoses → Condition`
|
||||
- Finance: `Account → transfers → Transaction`
|
||||
- Manufacturing: `Product → assembles → Component`
|
||||
- Education: `Student → completes → Course`
|
||||
- **YOUR domain** → Your types + relationships = Your knowledge graph
|
||||
|
||||
42 entity types × 127 relationship types form a shared vocabulary for any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), yours. Your own taxonomy layers on with `subtype`, enforced at write time:
|
||||
[→ See the Mathematical Proof](docs/architecture/noun-verb-taxonomy.md)
|
||||
|
||||
### ⚡ **Zero Configuration Philosophy**
|
||||
|
||||
**We hate configuration files. So we eliminated them.**
|
||||
|
||||
```javascript
|
||||
await brain.add({ data: 'Avery Brooks', type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
brain.counts.bySubtype(NounType.Person) // O(1) — { employee: 12, customer: 847 }
|
||||
brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
|
||||
const brain = new Brainy() // Auto-detects everything
|
||||
await brain.init() // Optimizes for your environment
|
||||
```
|
||||
|
||||
**[Type system](docs/architecture/noun-verb-taxonomy.md)** · **[Subtypes & facets](docs/guides/subtypes-and-facets.md)**
|
||||
Brainy automatically:
|
||||
- Detects optimal storage (memory/filesystem/cloud)
|
||||
- Configures memory based on available RAM
|
||||
- Optimizes for containers (Docker/K8s)
|
||||
- Tunes indexes for your data patterns
|
||||
- Manages embedding models and caching
|
||||
|
||||
### Graph analytics built in
|
||||
**You write business logic. Brainy handles infrastructure.**
|
||||
|
||||
### 🚀 **Git-Style Version Control** — Database & Entity Level (v5.0.0+)
|
||||
|
||||
**Clone your entire database in <100ms. Track every entity change. Full Git-style workflow.**
|
||||
|
||||
```javascript
|
||||
await brain.graph.rank() // which entities matter most (centrality)
|
||||
await brain.graph.communities() // natural clusters
|
||||
await brain.graph.path(a, b) // how two things connect
|
||||
await brain.graph.subgraph([seed], { depth: 2 }) // bounded neighborhood → { nodes, edges }
|
||||
await brain.graph.export() // whole graph, one O(N+E) streaming pass
|
||||
// Fork instantly - Snowflake-style copy-on-write
|
||||
const experiment = await brain.fork('test-migration')
|
||||
|
||||
// Make changes safely in isolation
|
||||
await experiment.add({ type: 'user', data: { name: 'Test User' } })
|
||||
await experiment.updateAll({ /* migration logic */ })
|
||||
|
||||
// Commit your work
|
||||
await experiment.commit({ message: 'Add test user', author: 'dev@example.com' })
|
||||
|
||||
// Switch to experimental branch to make it active
|
||||
await brain.checkout('test-migration')
|
||||
|
||||
// Time-travel: Query database at any past commit (read-only)
|
||||
const commits = await brain.getHistory({ limit: 10 })
|
||||
const snapshot = await brain.asOf(commits[5].id)
|
||||
const pastResults = await snapshot.find({ query: 'historical data' })
|
||||
await snapshot.close()
|
||||
|
||||
// Entity versioning: Track changes to individual entities (v5.3.0+)
|
||||
const userId = await brain.add({ type: 'user', data: { name: 'Alice' } })
|
||||
await brain.versions.save(userId, { tag: 'v1.0', description: 'Initial profile' })
|
||||
|
||||
await brain.update(userId, { data: { name: 'Alice Smith', role: 'admin' } })
|
||||
await brain.versions.save(userId, { tag: 'v2.0', description: 'Added role' })
|
||||
|
||||
// Compare versions or restore previous state
|
||||
const diff = await brain.versions.compare(userId, 1, 2) // See what changed
|
||||
await brain.versions.restore(userId, 1) // Restore v1.0
|
||||
```
|
||||
|
||||
### Write-time aggregations
|
||||
**Database-level version control (v5.0.0):**
|
||||
- ✅ `fork()` - Instant clone in <100ms
|
||||
- ✅ `merge()` - Merge with conflict resolution
|
||||
- ✅ `commit()` - Snapshot state
|
||||
- ✅ `asOf()` - Time-travel queries (query at any commit)
|
||||
- ✅ `getHistory()` - View commit history
|
||||
- ✅ `checkout()`, `listBranches()` - Full branch management
|
||||
- ✅ CLI support for all features
|
||||
|
||||
`SUM` / `COUNT` / `AVG` / `MIN` / `MAX` with `GROUP BY` and time windows, maintained incrementally on every write — reads are O(1) lookups, not scans. **[Aggregation guide](docs/guides/aggregation.md)**
|
||||
**Entity-level version control (v5.3.0):**
|
||||
- ✅ `versions.save()` - Save entity snapshots with tags
|
||||
- ✅ `versions.restore()` - Restore previous versions
|
||||
- ✅ `versions.compare()` - Diff between versions
|
||||
- ✅ `versions.list()` - View version history
|
||||
- ✅ Automatic deduplication (content-addressable storage)
|
||||
|
||||
### Import anything
|
||||
**How it works:** Snowflake-style COW shares HNSW index structures, copying only modified nodes (10-20% memory overhead).
|
||||
|
||||
**Perfect for:** Safe migrations, A/B testing, feature branches, distributed development, time-travel debugging, audit trails, document versioning, compliance tracking
|
||||
|
||||
[→ See Full Documentation](docs/features/instant-fork.md)
|
||||
|
||||
---
|
||||
|
||||
## What Can You Build?
|
||||
|
||||
**If your app needs to remember, understand, or connect information — Brainy makes it trivial.**
|
||||
|
||||
### 🤖 **AI Agents with Perfect Memory**
|
||||
Give your AI unlimited context that persists forever. Not just chat history — true understanding of relationships, evolution, and meaning over time.
|
||||
|
||||
**Examples:** Personal assistants, code assistants, conversational AI, research agents
|
||||
|
||||
### 📚 **Living Documentation & Knowledge Bases**
|
||||
Documentation that understands itself. Auto-links related concepts, detects outdated information, finds connections across your entire knowledge base.
|
||||
|
||||
**Examples:** Internal wikis, research platforms, smart documentation, learning systems
|
||||
|
||||
### 🔍 **Semantic Search at Any Scale**
|
||||
Find by meaning, not keywords. Search codebases, research papers, customer data, or media libraries with natural language.
|
||||
|
||||
**Examples:** Code search, research platforms, content discovery, recommendation engines
|
||||
|
||||
### 🏢 **Enterprise Knowledge Management**
|
||||
Corporate memory that never forgets. Track every customer interaction, product evolution, and business relationship.
|
||||
|
||||
**Examples:** CRM systems, product catalogs, customer intelligence, institutional knowledge
|
||||
|
||||
### 🎮 **Rich Interactive Experiences**
|
||||
NPCs that remember. Characters that persist across stories. Worlds that evolve based on real relationships.
|
||||
|
||||
**Examples:** Game worlds, interactive fiction, educational platforms, creative tools
|
||||
|
||||
### 🎨 **Content & Media Platforms**
|
||||
Every asset knows its relationships. Intelligent tagging, similarity-based discovery, and relationship-aware management.
|
||||
|
||||
**Examples:** DAM systems, media libraries, writing assistants, content management
|
||||
|
||||
**The pattern:** Knowledge that needs to live, connect, and evolve. That's what Brainy was built for.
|
||||
|
||||
---
|
||||
|
||||
## Core Features
|
||||
|
||||
### 🧠 **Natural Language Queries**
|
||||
|
||||
```javascript
|
||||
await brain.import('customers.csv')
|
||||
await brain.import('sales.xlsx') // every sheet
|
||||
await brain.import('research-paper.pdf') // tables extracted
|
||||
// Ask naturally - Brainy understands
|
||||
await brain.find("recent React components with tests")
|
||||
await brain.find("JavaScript libraries similar to Vue")
|
||||
|
||||
// Or use structured Triple Intelligence queries
|
||||
await brain.find({
|
||||
query: "React",
|
||||
where: { type: "library", year: { greaterThan: 2020 } },
|
||||
connected: { to: "JavaScript", depth: 2 }
|
||||
})
|
||||
```
|
||||
|
||||
**→ [See all query methods in API Reference](docs/api/README.md#search--query)**
|
||||
|
||||
### 🔍 **Zero-Config Hybrid Search** (v7.7.0)
|
||||
|
||||
Automatically combines text (keyword) and semantic (vector) search for optimal results:
|
||||
|
||||
```javascript
|
||||
// Just works - no configuration needed
|
||||
const results = await brain.find({ query: 'David Smith' })
|
||||
// Finds exact text matches AND semantically similar content
|
||||
|
||||
// Override when needed
|
||||
await brain.find({ query: 'exact match', searchMode: 'text' }) // Text only
|
||||
await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // Semantic only
|
||||
await brain.find({ query: 'hybrid', hybridAlpha: 0.3 }) // Custom weighting
|
||||
|
||||
// Highlight structured content (v7.8.0) — plain text, JSON, HTML, Markdown
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior',
|
||||
text: entity.data // Auto-detects TipTap, Slate, Lexical, HTML, Markdown
|
||||
})
|
||||
// Each highlight has: text, score, position, matchType, contentCategory
|
||||
```
|
||||
|
||||
**→ [Hybrid Search Documentation](docs/api/README.md#hybrid-search-v770)**
|
||||
|
||||
### 🌐 **Virtual Filesystem** — Intelligent File Management
|
||||
|
||||
Build file explorers and IDEs that never crash:
|
||||
|
||||
```javascript
|
||||
const vfs = brain.vfs()
|
||||
|
||||
// Tree-aware operations prevent infinite recursion
|
||||
const tree = await vfs.getTreeStructure('/projects', { maxDepth: 3 })
|
||||
|
||||
// Semantic file search
|
||||
const reactFiles = await vfs.search('React components with hooks')
|
||||
```
|
||||
|
||||
**[📖 VFS Quick Start →](docs/vfs/QUICK_START.md)** | **[Common Patterns →](docs/vfs/COMMON_PATTERNS.md)** | **[Neural Extraction →](docs/vfs/NEURAL_EXTRACTION.md)**
|
||||
|
||||
### 🚀 **Import Anything** — CSV, Excel, PDF, URLs
|
||||
|
||||
```javascript
|
||||
await brain.import('customers.csv') // Auto-detects everything
|
||||
await brain.import('sales-data.xlsx', { excelSheets: ['Q1', 'Q2'] })
|
||||
await brain.import('research-paper.pdf', { pdfExtractTables: true })
|
||||
await brain.import('https://api.example.com/data.json')
|
||||
```
|
||||
|
||||
Entities auto-classify on the way in; `brain.extractEntities(text)` exposes the same NER ensemble directly. **[Import guide](docs/guides/import-anything.md)**
|
||||
**[📖 Complete Import Guide →](docs/guides/import-anything.md)**
|
||||
|
||||
### A filesystem that understands content
|
||||
### 🧠 **Neural API** — Advanced Semantic Analysis
|
||||
|
||||
```javascript
|
||||
await brain.vfs.writeFile('/docs/readme.md', 'Project documentation')
|
||||
await brain.vfs.search('React components with hooks') // semantic file search
|
||||
// Clustering, similarity, outlier detection, visualization
|
||||
const clusters = await brain.neural.clusters({ algorithm: 'kmeans' })
|
||||
const similarity = await brain.neural.similar('item1', 'item2')
|
||||
const outliers = await brain.neural.outliers(0.3)
|
||||
const vizData = await brain.neural.visualize({ maxNodes: 100 })
|
||||
```
|
||||
|
||||
**[VFS quick start](docs/vfs/QUICK_START.md)**
|
||||
---
|
||||
|
||||
### Operations-grade by default
|
||||
## Framework Integration
|
||||
|
||||
- **Single-writer, many-reader** — an exclusive lock protects the data directory; `Brainy.openReadOnly()` and the `brainy inspect` CLI examine a live brain from another process, safely.
|
||||
- **Self-upgrading data files** — a 7.x brain opens under 8.x and migrates itself behind an observable lock (`getIndexStatus().migration`), with an automatic pre-upgrade backup. No migration scripts.
|
||||
- **No silent wrong answers** — cold-open guards self-heal or throw typed errors (`MetadataIndexNotReadyError`, `GraphIndexNotReadyError`); they never return `[]` for data that exists.
|
||||
**Works with any modern framework.** React, Vue, Angular, Svelte, Solid.js — your choice.
|
||||
|
||||
**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
|
||||
```javascript
|
||||
// React
|
||||
const [brain] = useState(() => new Brainy())
|
||||
useEffect(() => { brain.init() }, [])
|
||||
|
||||
## When you outgrow Brainy
|
||||
// Vue
|
||||
async mounted() { this.brain = await new Brainy().init() }
|
||||
|
||||
Brainy's pure-TypeScript engines carry real workloads a long way on their own — see the measured, per-operation numbers (not marketing figures) in **[docs/performance-envelopes.md](docs/performance-envelopes.md)** for what to expect, unaccelerated, on plain filesystem storage.
|
||||
// Angular
|
||||
@Injectable() export class BrainyService { brain = new Brainy() }
|
||||
```
|
||||
|
||||
When a deployment needs native-scale vector/graph performance — memory-mapped indexes that don't need your dataset in RAM, billion-scale ambitions — add the native engine. **The API doesn't change:**
|
||||
**Supports:** All bundlers (Webpack, Vite, Rollup) • SSR/SSG • Edge runtimes • Browser/Node.js
|
||||
|
||||
**[📖 Framework Integration Guide →](docs/guides/framework-integration.md)** | **[Next.js →](docs/guides/nextjs-integration.md)** | **[Vue →](docs/guides/vue-integration.md)**
|
||||
|
||||
---
|
||||
|
||||
## Storage — From Memory to Planet-Scale
|
||||
|
||||
### Development → Just Works
|
||||
```javascript
|
||||
const brain = new Brainy() // Memory storage, zero config
|
||||
```
|
||||
|
||||
### Production → Persistence with Compression
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './data', compression: true }
|
||||
})
|
||||
// 60-80% space savings with gzip
|
||||
```
|
||||
|
||||
### Cloud → AWS, GCS, Azure, Cloudflare R2
|
||||
```javascript
|
||||
// AWS S3 / Cloudflare R2
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
s3Storage: {
|
||||
bucketName: 'my-knowledge-base',
|
||||
region: 'us-east-1'
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Enable Intelligent-Tiering: 96% cost savings
|
||||
await brain.storage.enableIntelligentTiering('entities/', 'auto-tier')
|
||||
```
|
||||
|
||||
**Cost optimization at scale:**
|
||||
|
||||
| Scale | Standard | With Intelligent Tiering | Annual Savings |
|
||||
|-------|----------|--------------------------|----------------|
|
||||
| 5TB | $1,380 | $59 | $1,321 (96%) |
|
||||
| 50TB | $13,800 | $594 | $13,206 (96%) |
|
||||
| 500TB | $138,000 | $5,940 | $132,060 (96%) |
|
||||
|
||||
**[📖 Cloud Storage Guide →](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[AWS Cost Optimization →](docs/operations/cost-optimization-aws-s3.md)** | **[GCS →](docs/operations/cost-optimization-gcs.md)** | **[Azure →](docs/operations/cost-optimization-azure.md)**
|
||||
|
||||
---
|
||||
|
||||
## Production Features
|
||||
|
||||
### 🎯 Type-Aware HNSW Indexing
|
||||
|
||||
Efficient type-based organization for large-scale deployments:
|
||||
|
||||
- **Type-based queries:** Faster via directory structure (measured at 1K-1M scale)
|
||||
- **Type count tracking:** 284 bytes (Uint32Array, measured)
|
||||
- **Billion-scale projections:** NOT tested at 1B entities (extrapolated from 1M)
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ hnsw: { typeAware: true } })
|
||||
```
|
||||
|
||||
**[📖 How Type-Aware Indexing Works →](docs/architecture/data-storage-architecture.md)**
|
||||
|
||||
### ⚡ Enterprise-Ready Operations (v4.0.0)
|
||||
|
||||
- **Batch operations** with retry logic (1000x faster deletes)
|
||||
- **Gzip compression** (60-80% space savings)
|
||||
- **OPFS quota monitoring** (browser storage)
|
||||
- **Metadata/Vector separation** (billion-entity scalability)
|
||||
- **Circuit breakers & backpressure** (enterprise reliability)
|
||||
|
||||
```javascript
|
||||
// Batch operations
|
||||
await brain.storage.batchDelete(keys, { maxRetries: 3 })
|
||||
|
||||
// Monitor storage
|
||||
const status = await brain.storage.getStorageStatus()
|
||||
```
|
||||
|
||||
### 📊 Adaptive Memory Management
|
||||
|
||||
Auto-scales 2GB → 128GB+ based on environment:
|
||||
|
||||
- Container-aware (Docker/K8s cgroups)
|
||||
- Environment-optimized (dev/staging/production)
|
||||
- Built-in cache monitoring with tuning recommendations
|
||||
|
||||
```javascript
|
||||
const stats = brain.getCacheStats() // Performance insights
|
||||
```
|
||||
|
||||
**[📖 Capacity Planning Guide →](docs/operations/capacity-planning.md)**
|
||||
|
||||
---
|
||||
|
||||
## Benchmarks
|
||||
|
||||
| Operation | Performance | Memory |
|
||||
|-----------|-------------|--------|
|
||||
| Initialize | 450ms | 24MB |
|
||||
| Add entity | 12ms | +0.1MB |
|
||||
| Vector search (1K) | 3ms | - |
|
||||
| Metadata filter (10K) | 0.8ms | - |
|
||||
| Bulk import (1K) | 2.3s | +8MB |
|
||||
| **10M entities** | **5.8ms** | **12GB** |
|
||||
| **1B entities** | **18ms** | **50GB** |
|
||||
|
||||
---
|
||||
|
||||
## 🧠 Deep Dive: How Brainy Actually Works
|
||||
|
||||
**Want to understand the magic under the hood?**
|
||||
|
||||
### 🔍 Triple Intelligence & find() API
|
||||
Understand how vector search, graph relationships, and document filtering work together in one unified query:
|
||||
|
||||
**[📖 Triple Intelligence Architecture →](docs/architecture/triple-intelligence.md)**
|
||||
**[📖 Natural Language Guide →](docs/guides/natural-language.md)**
|
||||
**[📖 API Reference: find() →](docs/api/README.md)**
|
||||
|
||||
### 🗂️ Type-Aware Indexing & HNSW
|
||||
Learn about our indexing architecture with measured performance optimizations:
|
||||
|
||||
**[📖 Data Storage Architecture →](docs/architecture/data-storage-architecture.md)**
|
||||
**[📖 Architecture Overview →](docs/architecture/overview.md)**
|
||||
|
||||
### 📈 Scaling: Individual → Planet
|
||||
Understand how the same code scales from prototype to billions of entities:
|
||||
|
||||
**[📖 Capacity Planning →](docs/operations/capacity-planning.md)**
|
||||
**[📖 Cloud Deployment Guide →](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)**
|
||||
|
||||
### 🎯 The Universal Type System
|
||||
Explore the mathematical foundation: 42 nouns × 127 verbs = Stage 3 CANONICAL taxonomy:
|
||||
|
||||
**[📖 Noun-Verb Taxonomy →](docs/architecture/noun-verb-taxonomy.md)**
|
||||
|
||||
---
|
||||
|
||||
## CLI Tools
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/cor
|
||||
npm install -g brainy
|
||||
|
||||
brainy add "JavaScript is awesome" --metadata '{"type":"opinion"}'
|
||||
brainy find "awesome programming languages"
|
||||
brainy search "programming"
|
||||
```
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } })
|
||||
await brain.init() // @soulcraft/cor detected — same code, native engines underneath
|
||||
```
|
||||
47 commands available, including storage management, imports, and neural operations.
|
||||
|
||||
Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
|
||||
---
|
||||
|
||||
Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor is more headroom for when you need it, not capability held back to sell you later. Licensing and support: **cor@soulcraft.com**.
|
||||
## 📖 Complete Documentation
|
||||
|
||||
## Performance
|
||||
**For most developers:** Start with the **[Complete API Reference](docs/api/README.md)** ⭐
|
||||
|
||||
- Per-operation p50/p95 at 1k and 10k entities, pure-JS floor, measured and re-run every release that touches a measured path: **[docs/performance-envelopes.md](docs/performance-envelopes.md)**.
|
||||
- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
|
||||
- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
|
||||
- Capacity planning and architecture: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
|
||||
This comprehensive guide includes:
|
||||
- ✅ Every method with parameters, returns, and examples
|
||||
- ✅ Quick start in 60 seconds
|
||||
- ✅ Core CRUD → Advanced features (branching, versioning, time-travel)
|
||||
- ✅ TypeScript types and patterns
|
||||
- ✅ 1,870 lines of copy-paste ready code
|
||||
|
||||
## Use cases
|
||||
---
|
||||
|
||||
**AI agent memory** — persistent semantic recall with relationship tracking · **Knowledge bases** — auto-linking and meaning-aware navigation · **Semantic search** over codebases, documents, media · **Enterprise data** — CRM, catalogs, institutional memory · **Games & simulations** — worlds and characters that remember.
|
||||
### 🎯 Essential Reading (Start Here)
|
||||
|
||||
## Documentation
|
||||
1. **[📖 Complete API Reference](docs/api/README.md)** ⭐ **START HERE** ⭐
|
||||
- Your primary resource for building with Brainy
|
||||
- Every method documented with working examples
|
||||
|
||||
| Start | Core | Going deeper |
|
||||
|---|---|---|
|
||||
| [Brainy explained simply](docs/eli5.md) | [API reference](docs/api/README.md) | [Architecture overview](docs/architecture/overview.md) |
|
||||
| [Installation](docs/guides/installation.md) | [Data model](docs/DATA_MODEL.md) | [Consistency model](docs/concepts/consistency-model.md) |
|
||||
| [Natural-language queries](docs/guides/natural-language.md) | [Query operators](docs/QUERY_OPERATORS.md) | [Multi-process model](docs/concepts/multi-process.md) |
|
||||
| | [Find system](docs/FIND_SYSTEM.md) | [Scaling](docs/SCALING.md) |
|
||||
2. **[Filter & Query Syntax Guide](docs/FIND_SYSTEM.md)**
|
||||
- Complete reference for operators, compound filters, and optimization tips
|
||||
|
||||
3. **[Natural Language Queries](docs/guides/natural-language.md)**
|
||||
- Master the `find()` method and Triple Intelligence queries
|
||||
|
||||
4. **[v4.0.0 Migration Guide](docs/MIGRATION-V3-TO-V4.md)**
|
||||
- Upgrading from v3 (100% backward compatible)
|
||||
|
||||
### 🧠 Core Concepts & Architecture
|
||||
|
||||
- **[Triple Intelligence Architecture](docs/architecture/triple-intelligence.md)** — How vector + graph + document work together
|
||||
- **[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** — The universal type system (42 nouns × 127 verbs)
|
||||
- **[Transactions](docs/transactions.md)** — Atomic operations with automatic rollback
|
||||
- **[Architecture Overview](docs/architecture/overview.md)** — System design and components
|
||||
- **[Data Storage Architecture](docs/architecture/data-storage-architecture.md)** — Type-aware indexing and HNSW
|
||||
|
||||
### ☁️ Production & Operations
|
||||
|
||||
- **[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** — Deploy to AWS, GCS, Azure
|
||||
- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling to billions
|
||||
- **Cost Optimization:** **[AWS S3](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[Cloudflare R2](docs/operations/cost-optimization-cloudflare-r2.md)**
|
||||
|
||||
### 🌐 Framework Integration
|
||||
|
||||
- **[Framework Integration Guide](docs/guides/framework-integration.md)** — React, Vue, Angular, Svelte
|
||||
- **[Next.js Integration](docs/guides/nextjs-integration.md)**
|
||||
- **[Vue.js Integration](docs/guides/vue-integration.md)**
|
||||
|
||||
### 🌳 Virtual Filesystem (VFS)
|
||||
|
||||
- **[VFS Quick Start](docs/vfs/QUICK_START.md)** — Build file explorers that never crash
|
||||
- **[VFS Core Documentation](docs/vfs/VFS_CORE.md)**
|
||||
- **[Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md)** — AI-powered file navigation
|
||||
- **[Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md)**
|
||||
|
||||
### 📦 Data Import & Processing
|
||||
|
||||
- **[Import Anything Guide](docs/guides/import-anything.md)** — CSV, Excel, PDF, URLs with auto-detection
|
||||
|
||||
---
|
||||
|
||||
## What's New in v4.0.0
|
||||
|
||||
**Enterprise-scale cost optimization and performance improvements:**
|
||||
|
||||
- 🎯 **96% cloud storage cost savings** with intelligent tiering (AWS, GCS, Azure)
|
||||
- ⚡ **1000x faster batch deletions** (533 entities/sec vs 0.5/sec)
|
||||
- 📦 **60-80% compression** with gzip (FileSystem storage)
|
||||
- 🔄 **Enhanced metadata/vector separation** for billion-scale deployments
|
||||
|
||||
**[📖 Full v4.0.0 Changelog →](CHANGELOG.md)** | **[Migration Guide →](docs/MIGRATION-V3-TO-V4.md)** (100% backward compatible)
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
|
||||
**Bun 1.0+** (recommended) or **Node.js 22 LTS**
|
||||
|
||||
## Support & community
|
||||
```bash
|
||||
# Bun (recommended - best performance, single-binary deployment)
|
||||
bun install @soulcraft/brainy
|
||||
|
||||
- **Bugs and ideas** → **brainy@soulcraft.com** — no account needed, you'll get a receipt.
|
||||
- **Security reports** → **security@soulcraft.com** — see **[SECURITY.md](SECURITY.md)**.
|
||||
- **Contributing** → see **[CONTRIBUTING.md](CONTRIBUTING.md)**.
|
||||
# Node.js (fully supported)
|
||||
npm install @soulcraft/brainy
|
||||
```
|
||||
|
||||
MIT © Brainy Contributors.
|
||||
> **Why Bun?** Brainy's Candle WASM engine works seamlessly with `bun --compile` for standalone binary deployment. No external model files, no runtime downloads.
|
||||
|
||||
---
|
||||
|
||||
## Why Brainy Exists
|
||||
|
||||
**The Vision:** Traditional systems force you to choose between vector databases, graph databases, and document stores. You need all three, but combining them is complex and fragile.
|
||||
|
||||
**Brainy solved the impossible:** One API. All three paradigms. Any scale.
|
||||
|
||||
Like HTTP standardized web communication, **Brainy standardizes knowledge representation.** One protocol that any AI model understands. One system that scales from prototype to planet.
|
||||
|
||||
**[📖 Read the Mathematical Proof →](docs/architecture/noun-verb-taxonomy.md)**
|
||||
|
||||
---
|
||||
|
||||
## Enterprise & Support
|
||||
|
||||
**🏢 Brain Cloud** — Managed Brainy with team sync, persistent memory, and enterprise connectors.
|
||||
Visit **[soulcraft.com](https://soulcraft.com)** for more information.
|
||||
|
||||
**💖 Support Development:**
|
||||
- ⭐ Star us on **[GitHub](https://github.com/soulcraftlabs/brainy)**
|
||||
- 💝 Sponsor via **[GitHub Sponsors](https://github.com/sponsors/soulcraftlabs)**
|
||||
- 🐛 Report issues and contribute code
|
||||
- 📣 Share with your team and community
|
||||
|
||||
**Brainy is 100% free and open source.** No paywalls, no premium tiers, no feature gates.
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions! See **[CONTRIBUTING.md](CONTRIBUTING.md)** for guidelines.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT © Brainy Contributors
|
||||
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
<strong>Built with ❤️ by the Brainy community</strong><br>
|
||||
<em>The Knowledge Operating System</em><br>
|
||||
<em>From prototype to planet-scale • Zero configuration • Triple Intelligence™</em>
|
||||
</p>
|
||||
|
|
|
|||
2967
RELEASES.md
2967
RELEASES.md
File diff suppressed because it is too large
Load diff
36
SECURITY.md
36
SECURITY.md
|
|
@ -1,36 +0,0 @@
|
|||
# Security Policy
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
Email **security@soulcraft.com**. That's the one door for security reports
|
||||
across the company, and it works the same way for Brainy: every report is
|
||||
read by a human, you'll get a private receipt, and we'll work with you on
|
||||
coordinated disclosure — please don't open a public issue for anything
|
||||
that isn't already public.
|
||||
|
||||
Include what you'd want if you were on the other end: affected version,
|
||||
how to reproduce, and what you think the impact is. If you have a patch or
|
||||
a suggested fix, send it along — it's welcome but not required.
|
||||
|
||||
There is no bounty program today. We're saying that plainly so you know
|
||||
what to expect going in.
|
||||
|
||||
## Response time
|
||||
|
||||
We respond as fast as truth allows. That means: no fixed SLA, no promise of
|
||||
a reply within a specific number of hours — but a real report from a real
|
||||
person gets read promptly and taken seriously. If you haven't heard anything
|
||||
in a reasonable stretch, a follow-up email is completely fine.
|
||||
|
||||
## Supported versions
|
||||
|
||||
The latest `8.x` minor release line receives security fixes. If you're
|
||||
running an older major version, please upgrade before reporting — we can't
|
||||
commit to backporting fixes to unsupported lines.
|
||||
|
||||
## Scope
|
||||
|
||||
This policy covers the `@soulcraft/brainy` package itself — the code in
|
||||
this repository. If you're evaluating a deployment that also uses
|
||||
`@soulcraft/cor`, report issues in that package the same way, to the same
|
||||
address; we'll route internally.
|
||||
|
|
@ -1,295 +0,0 @@
|
|||
# ADR-001: Generational MVCC storage and the immutable Db API
|
||||
|
||||
**Status:** Accepted (ships in 8.0)
|
||||
**Date:** 2026-06-10
|
||||
|
||||
## Context
|
||||
|
||||
Before 8.0, Brainy carried two overlapping version-control subsystems: a
|
||||
copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a
|
||||
separate versioning subsystem (`versions.save/list/compare/restore/prune`),
|
||||
plus a read-only historical adapter for commit-based time travel. Together
|
||||
they were ~5,100 LOC of mechanism for one product need: *read a consistent
|
||||
past state while the store keeps moving, and snapshot/restore cheaply.*
|
||||
|
||||
Neither subsystem gave a precise isolation guarantee. Reads raced in-place
|
||||
JSON overwrites, so a "snapshot" was only as immutable as the bytes it
|
||||
happened to share with the live store.
|
||||
|
||||
8.0 replaces both with **one mechanism**: generational MVCC over immutable,
|
||||
generation-stamped records, exposed through a Datomic-style immutable
|
||||
database value (`Db`). The same model is implemented natively by versioned
|
||||
index providers (LSM snapshots), so semantics are identical on the pure-JS
|
||||
path and the native path.
|
||||
|
||||
## Decision
|
||||
|
||||
### The model
|
||||
|
||||
- A **monotonic u64 generation counter** is the store's logical clock. It
|
||||
advances once per committed `transact()` batch and once per
|
||||
single-operation write (`add`/`update`/`remove`/`relate`/…), so
|
||||
`brain.generation()` is always a meaningful watermark. It is persisted in
|
||||
`_system/generation.json` and never reissued for anything durable.
|
||||
- `brain.now()` **pins** the current generation in O(1) and returns a `Db` —
|
||||
an immutable view. Pins are refcounted; `db.release()` (with a
|
||||
`FinalizationRegistry` backstop for leaked values) ends the pin.
|
||||
- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative
|
||||
batch atomically as **exactly one generation**, with whole-store
|
||||
compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and
|
||||
reified transaction metadata appended to `_system/tx-log.jsonl`.
|
||||
- `brain.asOf(generation | Date | snapshotPath)` opens past state;
|
||||
`db.with(ops)` layers a speculative in-memory overlay (never touching
|
||||
disk, the counter, or index providers); `db.persist(path)` cuts an
|
||||
instant snapshot; `brain.restore(path, { confirm: true })` replaces state
|
||||
from one; `Brainy.load(path)` opens a snapshot read-only with the full
|
||||
query surface.
|
||||
|
||||
### Persisted layout
|
||||
|
||||
All paths are storage-root-relative:
|
||||
|
||||
```
|
||||
_system/generation.json { generation, updatedAt } atomic tmp+rename
|
||||
_system/manifest.json { version, generation, atomic tmp+rename
|
||||
committedAt, horizon } (the commit point)
|
||||
_system/tx-log.jsonl one line per committed append-only
|
||||
transact: { generation,
|
||||
timestamp, meta? }
|
||||
_generations/<N>/tx.json the generation-N delta: immutable
|
||||
touched noun/verb ids + meta
|
||||
_generations/<N>/prev/<id>.json before-image of <id> as of immutable
|
||||
commit N (raw stored bytes;
|
||||
null parts = file was absent)
|
||||
```
|
||||
|
||||
**Why per-generation deltas instead of a global `id → latest generation`
|
||||
map in the manifest:** a global map makes every commit O(all ids) — the
|
||||
whole map must be rewritten to swap it atomically. The delta layout makes a
|
||||
commit O(ids touched) and keeps the manifest a fixed-size watermark, while
|
||||
point-in-time resolution stays correct (see "Read resolution" below). The
|
||||
trade is that resolution at a pinned generation scans the deltas of later
|
||||
commits — bounded by the number of commits since the pin, which is exactly
|
||||
the window compaction keeps short.
|
||||
|
||||
Before-images are deliberately the *only* per-id records. They serve both
|
||||
roles the layer needs — the crash-recovery undo log and the point-in-time
|
||||
read source. After-images would duplicate state that is already readable
|
||||
(the canonical entity files hold the latest bytes; earlier states resolve
|
||||
from later before-images) and would double record I/O per commit.
|
||||
|
||||
### Commit protocol (durability)
|
||||
|
||||
`transact()` commits under a store-wide mutex:
|
||||
|
||||
1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError`
|
||||
before anything is staged.
|
||||
2. **Reserve** generation `N` (counter increment).
|
||||
3. **Stage the undo log:** write the before-image of every touched id plus
|
||||
`tx.json` under `_generations/N/`, then **fsync** the files and their
|
||||
directories. From this point, any crash is recoverable to the exact
|
||||
pre-transaction bytes.
|
||||
4. **Execute** the planned batch through the TransactionManager (which has
|
||||
its own operation-level rollback for in-flight failures).
|
||||
5. **Commit point:** persist the counter, then write `_system/manifest.json`
|
||||
via atomic tmp+rename and fsync it. The rename *is* the commit: a
|
||||
generation directory is committed if and only if `N ≤
|
||||
manifest.generation`.
|
||||
6. Append the tx-log line (advisory metadata — a crash between 5 and 6
|
||||
keeps the transaction).
|
||||
|
||||
**Crash recovery (on open):** any `_generations/<N>` directory with
|
||||
`N > manifest.generation` is an uncommitted transaction. Its before-images
|
||||
are restored to the canonical paths (idempotently — recovery itself can
|
||||
crash and rerun) and the directory is removed. Because recovery runs before
|
||||
any index is built, and a recovery that rolled something back forces a full
|
||||
index rebuild, derived indexes never observe rolled-back state. Reader-mode
|
||||
instances skip recovery (readers never write; the next writer repairs).
|
||||
|
||||
A failed (non-crash) transaction takes the same staging directory down the
|
||||
abort path: the TransactionManager rolls back applied operations, the
|
||||
staging directory is removed, and the generation reservation is returned —
|
||||
a failed batch leaves the generation counter unchanged.
|
||||
|
||||
### Read resolution at a pinned generation
|
||||
|
||||
The state of id X at pinned generation G is:
|
||||
|
||||
- the before-image stored by the **first committed generation after G that
|
||||
touched X**, or
|
||||
- the live canonical bytes, when nothing after G touched X.
|
||||
|
||||
While nothing has committed past G, *every* read on the `Db` delegates to
|
||||
the live fast paths untouched — `now()` adds no read overhead until history
|
||||
actually moves.
|
||||
|
||||
**Two read paths, one result set.** `get()`, metadata-level `find()`, and
|
||||
filter-based `related()` resolve directly through the record layer at any
|
||||
reachable pinned generation — no extra cost beyond scanning the deltas of
|
||||
later commits. Index-accelerated dimensions (semantic/vector search, graph
|
||||
traversal, cursors, aggregation) are served by **at-generation index
|
||||
materialization**: the first such query on a historical `Db` copies the
|
||||
exact at-G record set (live bytes for ids untouched since the pin,
|
||||
before-images for the rest; a final reconciliation pass runs under the
|
||||
commit mutex so transactions racing the copy cannot skew it) into an
|
||||
ephemeral in-memory store and opens a read-only engine over it — the same
|
||||
vector/metadata/graph index classes the live brain uses, sharing the host's
|
||||
embedder and aggregate definitions. The handle is cached on the `Db` and
|
||||
freed by `release()`.
|
||||
|
||||
**Cost, stated plainly:** materialization is O(n at G) time and memory,
|
||||
once per `Db`. That is the open-core price of historical index queries. A
|
||||
native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
|
||||
retained LSM segments) serves the same reads with no rebuild at all — the
|
||||
materializer is the correctness baseline, the provider is the accelerator.
|
||||
|
||||
**The one remaining boundary.** Speculative `with()` overlays throw
|
||||
`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
|
||||
overlay entities carry no embeddings (`with()` never invokes the embedder),
|
||||
so a "full" index query over an overlay would silently exclude the
|
||||
overlay's own entities. Commit with `transact()` to get the full surface.
|
||||
|
||||
**History granularity (Model-B).** EVERY write is its own immutable generation
|
||||
— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
|
||||
Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/
|
||||
`diff()`/`history()` exactly like transacts; a pin always freezes against later
|
||||
writes. `transact()` groups several operations into ONE atomic generation.
|
||||
|
||||
Single-op history durability is **async group-commit**: the live write hits
|
||||
canonical storage immediately (acknowledged), while its before-image is buffered
|
||||
and persisted to disk in one batched fsync on a size/timer trigger (or forced by
|
||||
`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in
|
||||
point-in-time resolution exactly like on-disk generations, so the synchronous
|
||||
`now()` freezes with no forced flush. A hard crash before the flush loses only
|
||||
the buffered *history* of the last window — never live data — and a crash
|
||||
*mid-flush* is recovered by **drop-without-restore** (the partial generation's
|
||||
before-images are discarded, never replayed, because the live write was already
|
||||
acknowledged; restoring them would silently revert it).
|
||||
|
||||
### Pinning, retention, compaction
|
||||
|
||||
- Each live `Db` holds one refcounted pin on its generation (plus a
|
||||
`pin(generation)` on every registered `VersionedIndexProvider`, whose
|
||||
explicit pin lifetime overrides any time-based snapshot retention the
|
||||
provider has).
|
||||
- The constructor **`retention`** knob governs auto-compaction (on `flush()`/
|
||||
`close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config;
|
||||
driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) ·
|
||||
`'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit
|
||||
CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
|
||||
manually on the same caps — the oldest unpinned record-sets are reclaimed
|
||||
while ANY supplied cap is exceeded.
|
||||
- A record-set `N` is reclaimed only when `N` is at or below **every** live
|
||||
pin — deleting `N` can only break readers pinned *below* `N`, because
|
||||
resolution reads before-images from generations strictly greater than the
|
||||
pin. Live pins are ALWAYS exempt, in every retention mode.
|
||||
- The manifest records the **horizon** (highest reclaimed generation).
|
||||
Generations below the horizon are unreachable; `asOf()` on them throws
|
||||
`GenerationCompactedError`. The horizon itself stays reachable, resolved
|
||||
from the record-sets above it. To keep a generation readable forever,
|
||||
`persist()` it first — snapshots are self-contained.
|
||||
|
||||
### Snapshots and restore
|
||||
|
||||
`db.persist(path)` flushes indexes, then cuts the snapshot under the
|
||||
store's commit mutex (no commit, compaction, or counter write can
|
||||
interleave). On filesystem storage it is a **hard-link farm**: every data
|
||||
file is immutable-by-rename, so linking is safe — later rewrites swap
|
||||
inodes and the snapshot keeps the old bytes. The two exceptions are handled
|
||||
explicitly: the append-in-place tx-log is byte-copied, and process-local
|
||||
lock state is excluded. Cross-device targets (and filesystems that refuse
|
||||
links) fall back to per-file byte copies. In-memory stores serialize to the
|
||||
same directory layout, so persisting a memory brain produces a real,
|
||||
durable, loadable store.
|
||||
|
||||
`persist()` requires the view to still be the store's latest generation
|
||||
(a snapshot captures current bytes); a view that history has moved past
|
||||
throws rather than persisting the wrong state.
|
||||
|
||||
`restore(path, { confirm: true })` replaces the store's contents from a
|
||||
snapshot via byte copy (never links — the snapshot stays independent),
|
||||
reloads all adapter-internal derived state, rebuilds all indexes, and
|
||||
floors the generation counter at its pre-restore value so observed
|
||||
generation numbers are never reissued. Live pins do not survive a restore;
|
||||
a warning is logged when any exist.
|
||||
|
||||
### Versioned index providers
|
||||
|
||||
Native index providers may implement the optional 4-method
|
||||
`VersionedIndexProvider` capability (`generation()`,
|
||||
`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the
|
||||
boundary). The locked consistency model: providers are **post-commit
|
||||
appliers**. The storage-record commit is the source of truth; provider
|
||||
index state is derived. On open, a provider behind the committed watermark
|
||||
replays the gap from storage (or requests a rebuild) — there are no
|
||||
provider rollback hooks, because uncommitted transactions are repaired at
|
||||
the storage layer before any index opens. Speculative `with()` overlays
|
||||
never reach providers.
|
||||
|
||||
## Guarantees (and their proofs)
|
||||
|
||||
Each stated guarantee has a test that proves it, not merely exercises it
|
||||
(`tests/integration/db-mvcc.test.ts`, plus
|
||||
`tests/unit/db/generationStore.test.ts` for the record layer in isolation):
|
||||
|
||||
| Guarantee | Proof |
|
||||
|---|---|
|
||||
| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) |
|
||||
| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) |
|
||||
| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) |
|
||||
| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c |
|
||||
| Compaction never breaks a pinned read; release enables reclaim | proof 5 |
|
||||
| `with()` overlays touch nothing durable | proof 6 |
|
||||
| Generation monotonicity across close/reopen | proof 7 |
|
||||
| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) |
|
||||
| Balanced provider pin/release lockstep | proof 9 |
|
||||
|
||||
One deliberate softness: single-operation generation bumps persist the
|
||||
counter coalesced (per write burst), not per write. Durable artifacts —
|
||||
records, manifests, snapshots — always persist the counter synchronously at
|
||||
their own commit points, so a crash inside the coalescing window can lose
|
||||
only counter values that nothing durable ever referenced.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Failure | Outcome |
|
||||
|---|---|
|
||||
| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. |
|
||||
| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. |
|
||||
| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. |
|
||||
| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
|
||||
| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
|
||||
| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
|
||||
| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
|
||||
| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
|
||||
| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
|
||||
|
||||
## Lineage
|
||||
|
||||
The design is an assembly of well-understood prior art, chosen for being
|
||||
boring where it counts:
|
||||
|
||||
- **Datomic** — the database-as-a-value: an immutable `Db` you query, with
|
||||
`with()` for speculation and reified transaction metadata instead of
|
||||
commit messages.
|
||||
- **LMDB** — reader pins: readers never block writers; a reader's view
|
||||
stays valid because nothing overwrites the pages (here: records) it
|
||||
references; reclamation waits for the last reader.
|
||||
- **LSM trees / Cassandra** — immutable segments make snapshots hard links
|
||||
and make compaction a retention policy instead of a locking problem.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One mechanism replaces the COW and versioning subsystems (their removal
|
||||
is the companion change to this ADR).
|
||||
- In-place branch switching (`checkout`) is gone by design; the replacement
|
||||
is opening a persisted snapshot as a separate instance — a name→path
|
||||
mapping where a product needs named branches.
|
||||
- Every commit pays O(ids touched) extra writes (before-images + delta +
|
||||
manifest). Single-operation writes pay only an in-memory counter bump
|
||||
with coalesced persistence.
|
||||
- The full query surface works at every reachable pinned generation.
|
||||
Record-path reads (`get`, metadata `find`, filter `related`) are
|
||||
effectively free; index-accelerated historical queries pay a one-time
|
||||
O(n at G) materialization per `Db` on the open-core path (freed on
|
||||
`release()`), and run rebuild-free on a native `VersionedIndexProvider`.
|
||||
481
docs/API_DECISION_TREE.md
Normal file
481
docs/API_DECISION_TREE.md
Normal file
|
|
@ -0,0 +1,481 @@
|
|||
# 🧠 Brainy API Decision Tree
|
||||
|
||||
*Choose the right API for your use case with confidence*
|
||||
|
||||
This guide helps you navigate Brainy's comprehensive API surface by asking the right questions to find the perfect method for your specific needs.
|
||||
|
||||
## 🎯 Quick Start: What do you want to do?
|
||||
|
||||
### 📝 **Adding Data**
|
||||
- **Single entity** → [`brainy.add()`](#adding-single-entities)
|
||||
- **Multiple entities** → [`brainy.addMany()`](#adding-multiple-entities)
|
||||
- **Streaming/real-time data** → [Streaming Pipeline](#streaming-data)
|
||||
|
||||
### 🔍 **Finding Data**
|
||||
- **Natural language search** → [`brainy.find("search query")`](#natural-language-search)
|
||||
- **Structured/filtered search** → [`brainy.find({ query, where, type })`](#structured-search)
|
||||
- **Similar entities** → [`brainy.similar()`](#similarity-search)
|
||||
- **Get by ID** → [`brainy.get()`](#retrieval-by-id)
|
||||
|
||||
### 🔗 **Relationships**
|
||||
- **Create relationships** → [`brainy.relate()`](#creating-relationships)
|
||||
- **Query relationships** → [`brainy.getRelations()`](#querying-relationships)
|
||||
- **Graph traversal** → [Graph Navigation](#graph-operations)
|
||||
|
||||
### 📊 **Advanced Features**
|
||||
- **File management** → [VFS (Virtual File System)](#file-operations)
|
||||
- **AI-powered analysis** → [Neural API](#neural-analysis)
|
||||
- **Clustering/insights** → [Intelligence Systems](#intelligence-systems)
|
||||
|
||||
---
|
||||
|
||||
## 🔀 Decision Tree Flow
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[What are you trying to do?] --> B[Store Data]
|
||||
A --> C[Find Data]
|
||||
A --> D[Manage Relationships]
|
||||
A --> E[Work with Files]
|
||||
A --> F[AI Analysis]
|
||||
|
||||
B --> B1[Single Item]
|
||||
B --> B2[Multiple Items]
|
||||
B --> B3[Real-time Stream]
|
||||
|
||||
C --> C1[I know the ID]
|
||||
C --> C2[Natural language query]
|
||||
C --> C3[Complex filters]
|
||||
C --> C4[Find similar items]
|
||||
|
||||
D --> D1[Create relationship]
|
||||
D --> D2[Query relationships]
|
||||
D --> D3[Graph traversal]
|
||||
|
||||
E --> E1[File operations]
|
||||
E --> E2[Knowledge-enhanced files]
|
||||
|
||||
F --> F1[Clustering]
|
||||
F --> F2[Similarity analysis]
|
||||
F --> F3[Insights generation]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Adding Data
|
||||
|
||||
### Adding Single Entities
|
||||
|
||||
**Use `brainy.add()` when:**
|
||||
- Adding one entity at a time
|
||||
- You need the ID immediately for further operations
|
||||
- Working with user input or real-time data
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for single entities
|
||||
const id = await brainy.add({
|
||||
data: "New research paper on quantum computing",
|
||||
type: NounType.Document,
|
||||
metadata: { category: "research", priority: "high" }
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Single item?** → `add()`
|
||||
- **Need immediate ID?** → `add()`
|
||||
- **Interactive application?** → `add()`
|
||||
|
||||
### Adding Multiple Entities
|
||||
|
||||
**Use `brainy.addMany()` when:**
|
||||
- Bulk importing data
|
||||
- Processing batches (>10 items)
|
||||
- Performance is critical
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for bulk operations
|
||||
const result = await brainy.addMany({
|
||||
items: documents.map(doc => ({
|
||||
data: doc.content,
|
||||
type: NounType.Document,
|
||||
metadata: doc.metadata
|
||||
})),
|
||||
chunkSize: 100,
|
||||
parallel: true
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Multiple items (>10)?** → `addMany()`
|
||||
- **Batch processing?** → `addMany()`
|
||||
- **Can tolerate some failures?** → `addMany()` with `continueOnError: true`
|
||||
|
||||
### Streaming Data
|
||||
|
||||
**Use Streaming Pipeline when:**
|
||||
- Real-time data ingestion
|
||||
- Processing large datasets that don't fit in memory
|
||||
- Need transformation during ingestion
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for streaming
|
||||
const pipeline = brainy.streaming.pipeline()
|
||||
.transform(data => ({ ...data, processed: true }))
|
||||
.batch(50)
|
||||
.into(brainy)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Finding Data
|
||||
|
||||
### Natural Language Search
|
||||
|
||||
**Use `brainy.find("query string")` when:**
|
||||
- User is typing search queries
|
||||
- You want semantic understanding
|
||||
- Building search interfaces
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for user searches
|
||||
const results = await brainy.find("documents about machine learning")
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **User-generated query?** → Natural language `find()`
|
||||
- **Semantic understanding needed?** → Natural language `find()`
|
||||
- **Search interface?** → Natural language `find()`
|
||||
|
||||
### Structured Search
|
||||
|
||||
**Use `brainy.find({ query, where, type })` when:**
|
||||
- Complex filtering requirements
|
||||
- Combining text search with metadata filters
|
||||
- Performance-critical searches
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for complex queries
|
||||
const results = await brainy.find({
|
||||
query: "neural networks",
|
||||
type: NounType.Document,
|
||||
where: {
|
||||
status: "published",
|
||||
year: { $gte: 2020 }
|
||||
},
|
||||
limit: 20
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Need metadata filtering?** → Structured `find()`
|
||||
- **Performance critical?** → Structured `find()`
|
||||
- **Complex criteria?** → Structured `find()`
|
||||
|
||||
### Similarity Search
|
||||
|
||||
**Use `brainy.similar()` when:**
|
||||
- Finding "more like this" content
|
||||
- Recommendation systems
|
||||
- Duplicate detection
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for recommendations
|
||||
const similar = await brainy.similar({
|
||||
to: "document-id-123",
|
||||
limit: 10,
|
||||
type: NounType.Document
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **"More like this" feature?** → `similar()`
|
||||
- **Recommendations?** → `similar()`
|
||||
- **Duplicate detection?** → `similar()`
|
||||
|
||||
### Retrieval by ID
|
||||
|
||||
**Use `brainy.get()` when:**
|
||||
- You know the exact ID
|
||||
- Loading specific entities
|
||||
- Following relationships
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for direct access
|
||||
const entity = await brainy.get("known-id-123")
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Known ID?** → `get()`
|
||||
- **Direct access needed?** → `get()`
|
||||
- **Following relationships?** → `get()`
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Relationships
|
||||
|
||||
### Creating Relationships
|
||||
|
||||
**Use `brainy.relate()` when:**
|
||||
- Connecting two entities
|
||||
- Building knowledge graphs
|
||||
- Modeling real-world relationships
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for connections
|
||||
await brainy.relate({
|
||||
from: "user-123",
|
||||
to: "project-456",
|
||||
type: VerbType.WorksOn,
|
||||
metadata: { role: "lead", since: "2024-01-01" }
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Connecting entities?** → `relate()`
|
||||
- **Need relationship metadata?** → `relate()`
|
||||
- **Building graphs?** → `relate()`
|
||||
|
||||
### Querying Relationships
|
||||
|
||||
**Use `brainy.getRelations()` when:**
|
||||
- Finding all connections for an entity
|
||||
- Exploring relationship patterns
|
||||
- Building relationship views
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for relationship queries
|
||||
const relations = await brainy.getRelations({
|
||||
from: "user-123",
|
||||
type: VerbType.WorksOn
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 File Operations
|
||||
|
||||
### Basic File Operations
|
||||
|
||||
**Use VFS when:**
|
||||
- Managing files and directories
|
||||
- Need hierarchical structure
|
||||
- Building file explorers
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for file management
|
||||
const vfs = brainy.vfs({ storage: 'filesystem' })
|
||||
await vfs.writeFile('/docs/readme.md', 'content')
|
||||
const files = await vfs.getDirectChildren('/docs')
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **File management?** → VFS
|
||||
- **Directory structure?** → VFS
|
||||
- **File explorer interface?** → VFS
|
||||
|
||||
### Intelligent File Management
|
||||
|
||||
**Use VFS (Semantic VFS) when:**
|
||||
- Need semantic file search
|
||||
- Want AI-powered concept extraction
|
||||
- Building smart file systems
|
||||
- Require multi-dimensional file access
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for intelligent file systems
|
||||
const knowledgeVFS = await vfs.withKnowledge(brainy)
|
||||
const insights = await knowledgeVFS.getFileInsights('/project')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧠 AI Analysis
|
||||
|
||||
### Clustering
|
||||
|
||||
**Use Neural API clustering when:**
|
||||
- Discovering data patterns
|
||||
- Organizing large datasets
|
||||
- Creating automatic categories
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for pattern discovery
|
||||
const neural = brainy.neural()
|
||||
const clusters = await neural.cluster({
|
||||
entities: entityIds,
|
||||
k: 5,
|
||||
method: 'hierarchical'
|
||||
})
|
||||
```
|
||||
|
||||
### Intelligence Systems
|
||||
|
||||
**Use Triple Intelligence when:**
|
||||
- Complex multi-criteria searches
|
||||
- Advanced relationship queries
|
||||
- Performance-critical operations
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for complex queries
|
||||
const intelligence = brainy.getTripleIntelligence()
|
||||
const results = await intelligence.query({
|
||||
vector: queryVector,
|
||||
metadata: { category: 'research' },
|
||||
graph: { connected: 'user-123' }
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Performance Optimization Guide
|
||||
|
||||
### When Performance Matters
|
||||
|
||||
| Scenario | Best Choice | Why |
|
||||
|----------|-------------|-----|
|
||||
| **Bulk Import** | `addMany()` | Batched operations, parallel processing |
|
||||
| **Metadata-only Search** | `find({ where: {...} })` | Skips vector computation |
|
||||
| **Known ID Access** | `get()` | Direct index lookup |
|
||||
| **Large Result Sets** | Pagination with `offset`/`limit` | Memory efficient |
|
||||
| **Real-time Streams** | Streaming Pipeline | Memory efficient, scalable |
|
||||
|
||||
### Memory Usage Optimization
|
||||
|
||||
```typescript
|
||||
// ❌ Memory intensive
|
||||
const allResults = await brainy.find({ limit: 10000 })
|
||||
|
||||
// ✅ Memory efficient
|
||||
for (let offset = 0; offset < total; offset += 100) {
|
||||
const batch = await brainy.find({
|
||||
query: "...",
|
||||
limit: 100,
|
||||
offset
|
||||
})
|
||||
await processBatch(batch)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Common Use Case Patterns
|
||||
|
||||
### Building a Search Interface
|
||||
|
||||
```typescript
|
||||
// User types query → Natural language search
|
||||
const searchResults = await brainy.find(userQuery)
|
||||
|
||||
// User applies filters → Structured search
|
||||
const filteredResults = await brainy.find({
|
||||
query: userQuery,
|
||||
where: selectedFilters,
|
||||
type: selectedTypes
|
||||
})
|
||||
|
||||
// User clicks "more like this" → Similarity search
|
||||
const similar = await brainy.similar({ to: selectedId })
|
||||
```
|
||||
|
||||
### Building a Recommendation System
|
||||
|
||||
```typescript
|
||||
// 1. Get user's interaction history
|
||||
const user = await brainy.get(userId)
|
||||
|
||||
// 2. Find similar users
|
||||
const similarUsers = await brainy.similar({ to: userId, type: NounType.Person })
|
||||
|
||||
// 3. Get their liked content
|
||||
const recommendations = []
|
||||
for (const similarUser of similarUsers) {
|
||||
const relations = await brainy.getRelations({
|
||||
from: similarUser.id,
|
||||
type: VerbType.Likes
|
||||
})
|
||||
recommendations.push(...relations)
|
||||
}
|
||||
```
|
||||
|
||||
### Building a Knowledge Graph
|
||||
|
||||
```typescript
|
||||
// 1. Add entities
|
||||
const entities = await Promise.all([
|
||||
brainy.add({ data: "Person: Alice", type: NounType.Person }),
|
||||
brainy.add({ data: "Company: TechCorp", type: NounType.Organization }),
|
||||
brainy.add({ data: "Project: AI Assistant", type: NounType.Thing })
|
||||
])
|
||||
|
||||
// 2. Create relationships
|
||||
await brainy.relate({
|
||||
from: entities[0], // Alice
|
||||
to: entities[1], // TechCorp
|
||||
type: VerbType.WorksFor
|
||||
})
|
||||
|
||||
await brainy.relate({
|
||||
from: entities[0], // Alice
|
||||
to: entities[2], // AI Assistant
|
||||
type: VerbType.WorksOn
|
||||
})
|
||||
|
||||
// 3. Query the graph
|
||||
const aliceConnections = await brainy.getRelations({ from: entities[0] })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Migration Guide
|
||||
|
||||
### From v2.x to v3.x APIs
|
||||
|
||||
| v2.x (Deprecated) | v3.x (Current) | When to Use |
|
||||
|-------------------|----------------|-------------|
|
||||
| `brain.store()` | `brainy.add()` | Adding entities |
|
||||
| `brain.search()` | `brainy.find()` | Searching content |
|
||||
| `brain.query()` | `brainy.find({ ... })` | Complex queries |
|
||||
| `brain.similar()` | `brainy.similar()` | ✅ Same API |
|
||||
| `brain.connect()` | `brainy.relate()` | Creating relationships |
|
||||
|
||||
### Legacy Type Migration
|
||||
|
||||
```typescript
|
||||
// ❌ v2.x way
|
||||
import { ISenseAugmentation } from '@soulcraft/brainy/types/augmentations'
|
||||
|
||||
// ✅ v3.x way
|
||||
import { BrainyAugmentation } from '@soulcraft/brainy'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎪 Decision Quick Reference
|
||||
|
||||
**Need to add data?**
|
||||
- 1 item → `add()`
|
||||
- Many items → `addMany()`
|
||||
- Streaming → Pipeline
|
||||
|
||||
**Need to find data?**
|
||||
- Know ID → `get()`
|
||||
- Natural search → `find("query")`
|
||||
- Complex filters → `find({ query, where })`
|
||||
- Similar items → `similar()`
|
||||
|
||||
**Need relationships?**
|
||||
- Create → `relate()`
|
||||
- Query → `getRelations()`
|
||||
- Complex graph → Triple Intelligence
|
||||
|
||||
**Need files?**
|
||||
- Basic → VFS (standard operations)
|
||||
- Smart → Semantic VFS (6 dimensional access + neural extraction)
|
||||
|
||||
**Need AI analysis?**
|
||||
- Patterns → Neural clustering
|
||||
- Complex queries → Triple Intelligence
|
||||
|
||||
---
|
||||
|
||||
*This guide covers 95% of use cases. For edge cases or custom requirements, check the [Core API Patterns](./CORE_API_PATTERNS.md) and [Neural API Patterns](./NEURAL_API_PATTERNS.md) guides.*
|
||||
1869
docs/API_REFERENCE.md
Normal file
1869
docs/API_REFERENCE.md
Normal file
File diff suppressed because it is too large
Load diff
350
docs/BATCHING.md
350
docs/BATCHING.md
|
|
@ -1,28 +1,22 @@
|
|||
---
|
||||
title: Batch Operations
|
||||
slug: guides/batching
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 5
|
||||
description: Eliminate N+1 query patterns with batchGet() and storage-level batch APIs for fast multi-entity reads against filesystem and memory storage.
|
||||
next:
|
||||
- api/reference
|
||||
- guides/find-system
|
||||
---
|
||||
|
||||
# Batch Operations API
|
||||
> **Production-Ready** | Zero N+1 Query Patterns
|
||||
> **Enterprise Production-Ready** | Zero N+1 Query Patterns | 90%+ Performance Improvement
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy provides batch operations at the storage layer to eliminate N+1 query patterns for VFS operations, relationship queries, and entity retrieval.
|
||||
Brainy introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
|
||||
|
||||
### Problem Solved
|
||||
|
||||
The naive pattern of looping and calling `brain.get(id)` once per item issues sequential reads through the storage layer. Batched APIs collapse that into a single read pass.
|
||||
**Before optimization:**
|
||||
- VFS `getTreeStructure()` on cloud storage: **12.7 seconds** for directory with 12 files
|
||||
- N+1 query pattern: 1 directory query + N individual file queries (22 sequential calls × 580ms latency)
|
||||
|
||||
**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`** at the VFS layer and the explicit `batchGet()` / `getNounMetadataBatch()` calls — not to `readFile()` or individual `get()` operations.
|
||||
**After optimization:**
|
||||
- VFS `getTreeStructure()`: **<1 second** for 12 files (90%+ improvement)
|
||||
- 2-3 batched calls instead of 22 sequential calls
|
||||
- Native cloud storage batch APIs for maximum throughput
|
||||
|
||||
**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See changes for comprehensive storage path optimizations.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -47,7 +41,8 @@ results.size // → 3 (number of found entities)
|
|||
|
||||
**Performance:**
|
||||
- Memory storage: Instant (parallel reads)
|
||||
- Filesystem storage: Parallel reads, scales with available IOPS
|
||||
- Cloud storage (GCS/S3/Azure): <500ms for 100 entities
|
||||
- Throughput: 50-200+ entities/second depending on adapter
|
||||
|
||||
**Use Cases:**
|
||||
- Loading multiple entities for display
|
||||
|
|
@ -77,13 +72,13 @@ for (const [id, metadata] of metadataMap) {
|
|||
**Features:**
|
||||
- ✅ Direct O(1) path construction from ID (no type lookup needed!)
|
||||
- ✅ Sharding preservation (all paths include `{shard}/{id}`)
|
||||
- ✅ Write-cache coherent (read-after-write consistency)
|
||||
- ✅ O(1) path construction — eliminates the per-entity type search of the old type-first layout
|
||||
- ✅ COW-aware (respects branch paths)
|
||||
- ✅ 40x faster than v5.x type-first architecture
|
||||
|
||||
**Performance:**
|
||||
- Constant-time path construction per ID — no type-cache misses
|
||||
- Filesystem: parallel reads bounded by IOPS
|
||||
- No type search delays — every ID maps directly to storage path
|
||||
- ~1ms per 100 entities (consistent, no cache misses!)
|
||||
- Cloud storage: Parallel downloads (100-150 concurrent)
|
||||
- No type search delays - every ID maps directly to storage path
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -121,8 +116,130 @@ for (const [sourceId, verbs] of results) {
|
|||
- Bulk export of relationship data
|
||||
|
||||
**Performance:**
|
||||
- Memory storage: single in-memory pass over the metadata index
|
||||
- Filesystem storage: parallel reads through the metadata index
|
||||
- Memory storage: <10ms for 1000 relationships
|
||||
- Cloud storage: Batched reads with parallel metadata fetches
|
||||
|
||||
---
|
||||
|
||||
### 4. `storage.readBatchWithInheritance(paths, targetBranch?)`
|
||||
|
||||
COW-aware batch path resolution with branch inheritance.
|
||||
|
||||
```typescript
|
||||
const storage = brain.storage as BaseStorage
|
||||
|
||||
const paths = [
|
||||
'entities/nouns/{shard}/id1/metadata.json',
|
||||
'entities/nouns/{shard}/id2/metadata.json'
|
||||
]
|
||||
|
||||
// Resolves to: branches/{branch}/entities/nouns/{shard}/{id}/metadata.json
|
||||
const results: Map<string, any> = await storage.readBatchWithInheritance(paths, 'my-branch')
|
||||
|
||||
// Automatically inherits from parent branches for missing entities
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- ✅ Branch path resolution (`branches/{branch}/...`)
|
||||
- ✅ Write cache integration (read-after-write consistency)
|
||||
- ✅ COW inheritance (fallback to parent commits for missing entities)
|
||||
- ✅ Adapter-agnostic (works with all storage adapters)
|
||||
|
||||
---
|
||||
|
||||
## Cloud Adapter Native Batch APIs
|
||||
|
||||
### GCS Storage
|
||||
|
||||
```typescript
|
||||
const gcsStorage = new GCSStorage({ bucketName: 'my-bucket' })
|
||||
|
||||
// Native batch API with 100 concurrent downloads
|
||||
const results = await gcsStorage.readBatch(paths)
|
||||
|
||||
// Configuration
|
||||
gcsStorage.getBatchConfig() // → {
|
||||
// maxBatchSize: 1000,
|
||||
// maxConcurrent: 100,
|
||||
// operationsPerSecond: 1000
|
||||
// }
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- 100 concurrent downloads
|
||||
- ~300-500ms for 100 objects
|
||||
- HTTP/2 multiplexing for optimal throughput
|
||||
|
||||
---
|
||||
|
||||
### S3 Compatible Storage
|
||||
|
||||
Works with Amazon S3, Cloudflare R2, and other S3-compatible services.
|
||||
|
||||
```typescript
|
||||
const s3Storage = new S3CompatibleStorage({ bucketName: 'my-bucket' })
|
||||
|
||||
// Native batch API with 150 concurrent downloads
|
||||
const results = await s3Storage.readBatch(paths)
|
||||
|
||||
// Configuration
|
||||
s3Storage.getBatchConfig() // → {
|
||||
// maxBatchSize: 1000,
|
||||
// maxConcurrent: 150,
|
||||
// operationsPerSecond: 5000
|
||||
// }
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- 150 concurrent downloads
|
||||
- ~200-500ms for 150 objects
|
||||
- S3 handles 5000+ ops/second with burst capacity
|
||||
|
||||
---
|
||||
|
||||
### R2 Storage (Cloudflare)
|
||||
|
||||
```typescript
|
||||
const r2Storage = new R2Storage({ bucketName: 'my-bucket' })
|
||||
|
||||
// Fastest cloud storage with zero egress fees
|
||||
const results = await r2Storage.readBatch(paths)
|
||||
|
||||
// Configuration
|
||||
r2Storage.getBatchConfig() // → {
|
||||
// maxBatchSize: 1000,
|
||||
// maxConcurrent: 150,
|
||||
// operationsPerSecond: 6000
|
||||
// }
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- 150 concurrent downloads
|
||||
- ~200-400ms for 150 objects (fastest!)
|
||||
- Zero egress fees enable aggressive caching
|
||||
|
||||
---
|
||||
|
||||
### Azure Blob Storage
|
||||
|
||||
```typescript
|
||||
const azureStorage = new AzureBlobStorage({ containerName: 'my-container' })
|
||||
|
||||
// Native batch API with 100 concurrent downloads
|
||||
const results = await azureStorage.readBatch(paths)
|
||||
|
||||
// Configuration
|
||||
azureStorage.getBatchConfig() // → {
|
||||
// maxBatchSize: 1000,
|
||||
// maxConcurrent: 100,
|
||||
// operationsPerSecond: 3000
|
||||
// }
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- 100 concurrent downloads
|
||||
- ~400-600ms for 100 blobs
|
||||
- Good throughput with Azure's global network
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -133,10 +250,12 @@ VFS operations automatically use batch APIs for maximum performance.
|
|||
### Directory Traversal
|
||||
|
||||
```typescript
|
||||
// Tree traversal uses batched reads under the hood
|
||||
// OLD: Sequential N+1 pattern (12.7 seconds for 12 files)
|
||||
const tree = await brain.vfs.getTreeStructure('/my-dir')
|
||||
|
||||
// NEW Parallel breadth-first with batching (<1 second)
|
||||
// ✅ PathResolver.getChildren() uses brain.batchGet() internally
|
||||
// ✅ Parallel traversal of directories at the same tree level
|
||||
// ✅ Parallel traversal of directories at same tree level
|
||||
// ✅ 2-3 batched calls instead of 22 sequential calls
|
||||
```
|
||||
|
||||
|
|
@ -150,11 +269,17 @@ VFS.getTreeStructure()
|
|||
→ brain.batchGet(childIds) [1 call instead of N]
|
||||
↓ BATCHED
|
||||
→ storage.getNounMetadataBatch(ids) [1 call instead of N]
|
||||
↓ ADAPTER
|
||||
→ Filesystem: Promise.all() parallel reads
|
||||
↓ ADAPTER-SPECIFIC
|
||||
→ GCS: readBatch() with 100 concurrent downloads
|
||||
→ S3: readBatch() with 150 concurrent downloads
|
||||
→ Memory: Promise.all() parallel reads
|
||||
```
|
||||
|
||||
**Performance Gains:**
|
||||
- **Before**: 22 sequential calls × 580ms = 12.7 seconds
|
||||
- **After**: 2-3 batched calls = <1 second
|
||||
- **Improvement**: **90%+ faster** on cloud storage
|
||||
|
||||
---
|
||||
|
||||
## Advanced Features Compatibility
|
||||
|
|
@ -171,7 +296,7 @@ entities/verbs/{SHARD}/{ID}/metadata.json
|
|||
|
||||
**Direct O(1) Path Construction:**
|
||||
```typescript
|
||||
// Every ID maps directly to exactly ONE path - O(1), no type search
|
||||
// Every ID maps directly to exactly ONE path - 40x faster!
|
||||
const id = 'abc-123'
|
||||
const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
|
||||
const path = `entities/nouns/${shard}/${id}/metadata.json`
|
||||
|
|
@ -183,9 +308,9 @@ const path = `entities/nouns/${shard}/${id}/metadata.json`
|
|||
```
|
||||
|
||||
**Benefits:**
|
||||
- **O(1)** path lookups (eliminates the 42-type sequential search the old type-first layout required)
|
||||
- **40x faster** on GCS/S3 (eliminates 42-type sequential search)
|
||||
- **Simpler code** - removed 500+ lines of type cache complexity
|
||||
- **Scalable** - works at large scale without type tracking overhead
|
||||
- **Scalable** - works at billion-scale without type tracking overhead
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -203,39 +328,95 @@ const path = `entities/nouns/${shard}/${id}/metadata.json`
|
|||
|
||||
---
|
||||
|
||||
### ✅ Generational MVCC (8.0)
|
||||
### ✅ COW (Copy-on-Write)
|
||||
|
||||
Batch reads always serve the **live** generation through the fast paths
|
||||
shown above. Point-in-time reads go through the Db API instead: a pinned
|
||||
`Db` (`brain.now()`, `brain.asOf()`) resolves changed ids from immutable
|
||||
generation records and unchanged ids from the same live paths batch reads
|
||||
use — see the [consistency model](concepts/consistency-model.md).
|
||||
Batch operations respect branch isolation and time-travel:
|
||||
|
||||
```typescript
|
||||
const db = brain.now() // pinned view
|
||||
const entity = await db.get(id) // correct at the pinned generation
|
||||
const results = await brain.batchGet(ids) // live state, batched
|
||||
await db.release()
|
||||
// Main branch
|
||||
const brain = await Brainy.create({ enableCOW: true })
|
||||
await brain.add({ type: 'document', data: 'Main' })
|
||||
|
||||
// Create fork
|
||||
const fork = await brain.fork('experiment')
|
||||
|
||||
// Batch operations are isolated
|
||||
await brain.batchGet([id1, id2]) // → Reads from: branches/main/...
|
||||
await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/...
|
||||
```
|
||||
|
||||
**Inheritance:**
|
||||
- Entities missing from child branch automatically inherit from parent commits
|
||||
- `readBatchWithInheritance()` walks commit history for missing items
|
||||
- Preserves fork semantics while maintaining performance
|
||||
|
||||
---
|
||||
|
||||
### ✅ fork() and checkout()
|
||||
|
||||
```typescript
|
||||
const fork = await brain.fork('my-branch')
|
||||
await fork.add({ type: 'document', data: 'Fork entity' })
|
||||
|
||||
// Batch operations use correct branch
|
||||
const results = await fork.batchGet([id1, id2])
|
||||
// → Reads from: branches/my-branch/...
|
||||
|
||||
// Checkout changes active branch
|
||||
await fork.checkout('main')
|
||||
const mainResults = await fork.batchGet([id1, id2])
|
||||
// → Reads from: branches/main/...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Why Batching Is Faster
|
||||
### ✅ asOf() Time-Travel
|
||||
|
||||
Batching's advantage is structural, not a fixed multiplier (the actual speedup
|
||||
depends on storage backend, IOPS, and batch size):
|
||||
```typescript
|
||||
// Create historical snapshot
|
||||
await brain.commit('v1.0')
|
||||
const snapshot = await brain.asOf('v1.0')
|
||||
|
||||
- **N+1 elimination** — N sequential reads collapse into a single parallel pass
|
||||
(`Promise.all` over the batch).
|
||||
- **O(1) path construction** — every ID maps directly to one storage path, with
|
||||
no per-type cache lookup.
|
||||
- **One metadata round-trip** — relationship batches fetch all sources' metadata
|
||||
in a single pass instead of one query per source.
|
||||
// Batch operations on historical data
|
||||
const results = await snapshot.batchGet([id1, id2])
|
||||
// → Reads from historical tree state
|
||||
```
|
||||
|
||||
The integration test `tests/integration/storage-batch-operations.test.ts`
|
||||
exercises batch vs. individual reads and asserts that batch retrieval is not
|
||||
slower than the per-entity loop for large batches; it does not pin a specific
|
||||
multiplier, since that is hardware- and IOPS-dependent.
|
||||
Historical queries use `HistoricalStorageAdapter` which wraps batch operations to point at specific commits.
|
||||
|
||||
---
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
### VFS Operations (12 Files)
|
||||
|
||||
| Storage | Before optimization | After optimization | Improvement |
|
||||
|---------|---------------|---------------|-------------|
|
||||
| **GCS** | 12.7s | <1s | **92% faster** |
|
||||
| **S3** | 13.2s | <1s | **92% faster** |
|
||||
| **R2** | 11.8s | <0.8s | **93% faster** |
|
||||
| **Azure** | 14.5s | <1s | **93% faster** |
|
||||
| **Memory** | 150ms | 50ms | **67% faster** |
|
||||
|
||||
### Entity Batch Retrieval (100 Entities)
|
||||
|
||||
| Storage | Individual Gets | Batch Get | Improvement |
|
||||
|---------|----------------|-----------|-------------|
|
||||
| **GCS** | 5.8s | 0.4s | **93% faster** |
|
||||
| **S3** | 5.2s | 0.3s | **94% faster** |
|
||||
| **R2** | 4.9s | 0.25s | **95% faster** |
|
||||
| **Azure** | 6.5s | 0.5s | **92% faster** |
|
||||
| **Memory** | 180ms | 15ms | **92% faster** |
|
||||
|
||||
### Throughput (Entities/Second)
|
||||
|
||||
| Storage | Individual | Batch | Improvement |
|
||||
|---------|-----------|-------|-------------|
|
||||
| **GCS** | 17 ent/s | 250 ent/s | **14.7x** |
|
||||
| **S3** | 19 ent/s | 333 ent/s | **17.5x** |
|
||||
| **R2** | 20 ent/s | 400 ent/s | **20x** |
|
||||
| **Azure** | 15 ent/s | 200 ent/s | **13.3x** |
|
||||
| **Memory** | 556 ent/s | 6667 ent/s | **12x** |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -300,7 +481,7 @@ const results = await brain.batchGet(ids)
|
|||
const entities = Array.from(results.values())
|
||||
```
|
||||
|
||||
**Performance Gain:** Replaces N sequential reads with a single batched pass — no fixed multiplier, it scales with storage IOPS.
|
||||
**Performance Gain:** 10-20x faster on cloud storage.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -310,7 +491,7 @@ const entities = Array.from(results.values())
|
|||
```typescript
|
||||
const allVerbs = []
|
||||
for (const sourceId of sourceIds) {
|
||||
const verbs = await brain.related({ from: sourceId })
|
||||
const verbs = await brain.getRelations({ from: sourceId })
|
||||
allVerbs.push(...verbs)
|
||||
}
|
||||
```
|
||||
|
|
@ -326,7 +507,7 @@ for (const verbs of results.values()) {
|
|||
}
|
||||
```
|
||||
|
||||
**Performance Gain:** One batched metadata fetch instead of one query per source entity.
|
||||
**Performance Gain:** 5-10x faster due to batched metadata fetches.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -347,9 +528,12 @@ for (const id of ids) {
|
|||
### 2. **Batch Size Recommendations**
|
||||
|
||||
| Storage | Optimal Batch Size | Max Batch Size |
|
||||
|---------|--------------------|----------------|
|
||||
|---------|-------------------|----------------|
|
||||
| **Memory** | Unlimited | Unlimited |
|
||||
| **Filesystem** | 100-500 | 1000 |
|
||||
| **FileSystem** | 100-500 | 1000 |
|
||||
| **GCS** | 100-500 | 1000 |
|
||||
| **S3/R2** | 100-1000 | 1000 |
|
||||
| **Azure** | 100-500 | 1000 |
|
||||
|
||||
**Guideline:** For batches >1000, split into chunks of 500-1000.
|
||||
|
||||
|
|
@ -417,42 +601,68 @@ High-Level API (src/brainy.ts)
|
|||
↓
|
||||
Storage Layer (src/storage/baseStorage.ts)
|
||||
↓
|
||||
COW Layer (readBatchWithInheritance)
|
||||
↓
|
||||
Adapter Layer (readBatchFromAdapter)
|
||||
↓
|
||||
Storage Adapter (FileSystemStorage / MemoryStorage)
|
||||
Cloud Adapter (GCS/S3/Azure native batch APIs)
|
||||
```
|
||||
|
||||
### Parallel Reads
|
||||
### Automatic Fallback
|
||||
|
||||
Both shipped adapters fall back to `Promise.all` over individual reads:
|
||||
If an adapter doesn't implement `readBatch()`, the system automatically falls back to parallel individual reads:
|
||||
|
||||
```typescript
|
||||
// BaseStorage.readBatchFromAdapter()
|
||||
return await Promise.all(resolvedPaths.map(path => this.read(path)))
|
||||
if (typeof selfWithBatch.readBatch === 'function') {
|
||||
// Use native batch API
|
||||
return await selfWithBatch.readBatch(resolvedPaths)
|
||||
} else {
|
||||
// Automatic parallel fallback
|
||||
return await Promise.all(resolvedPaths.map(path => this.read(path)))
|
||||
}
|
||||
```
|
||||
|
||||
**Shipped Adapters:**
|
||||
**Adapters with Native Batch:**
|
||||
- ✅ GCSStorage
|
||||
- ✅ S3CompatibleStorage
|
||||
- ✅ R2Storage
|
||||
- ✅ AzureBlobStorage
|
||||
|
||||
**Adapters with Parallel Fallback:**
|
||||
- MemoryStorage
|
||||
- FileSystemStorage
|
||||
- OPFSStorage
|
||||
- HistoricalStorageAdapter (delegates to underlying)
|
||||
|
||||
---
|
||||
|
||||
## API Summary
|
||||
## Release Notes
|
||||
|
||||
**Version:** 5.12.0
|
||||
**Release Date:** 2025-11-19
|
||||
**Status:** Production-Ready
|
||||
|
||||
**Breaking Changes:** None (backward compatible)
|
||||
|
||||
**New APIs:**
|
||||
- `brain.batchGet(ids, options?)` - High-level batch entity retrieval
|
||||
- `storage.getNounMetadataBatch(ids)` - Storage-level metadata batch
|
||||
- `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries
|
||||
- `storage.readBatchWithInheritance(paths, targetBranch?)` - COW-aware batch reads
|
||||
|
||||
**Performance Improvements:**
|
||||
- VFS operations: single batched pass instead of N sequential reads
|
||||
- Entity retrieval: N+1 reads collapsed into one batched pass
|
||||
- VFS operations: 90%+ faster on cloud storage
|
||||
- Entity retrieval: 10-20x throughput improvement
|
||||
- Zero N+1 query patterns
|
||||
|
||||
**Compatibility:**
|
||||
- ✅ ID-first storage
|
||||
- ✅ Sharding (256 shards)
|
||||
- ✅ Generational MVCC — batch reads serve the live generation; pinned `Db` views serve the past
|
||||
- ✅ All indexes respected (vector, metadata, graph adjacency)
|
||||
- ✅ COW (branch isolation, inheritance)
|
||||
- ✅ fork() and checkout()
|
||||
- ✅ asOf() time-travel
|
||||
- ✅ All 6 indexes respected (HNSW, TypeAwareHNSW, MetadataIndex, GraphAdjacency, Version, DeletedItems)
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
643
docs/CORE_API_PATTERNS.md
Normal file
643
docs/CORE_API_PATTERNS.md
Normal file
|
|
@ -0,0 +1,643 @@
|
|||
# 🧠 Core API Patterns: Modern Brainy v3.x
|
||||
|
||||
> Learn the correct patterns for Brainy's core operations. Avoid v2.x confusion and use modern, efficient APIs.
|
||||
|
||||
## 🚨 Critical: Use v3.x APIs Only
|
||||
|
||||
### ❌ **WRONG - Deprecated v2.x APIs**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - These methods don't exist in v3.x!
|
||||
await brain.addNoun(text, type, metadata) // ❌ Removed
|
||||
await brain.getNouns({ pagination }) // ❌ Removed
|
||||
await brain.addVerb(source, target, type) // ❌ Removed
|
||||
await brain.getVerbs() // ❌ Removed
|
||||
await brain.deleteNoun(id) // ❌ Removed
|
||||
await brain.deleteVerb(id) // ❌ Removed
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Modern v3.x APIs**
|
||||
|
||||
```typescript
|
||||
// ✅ Use these modern methods instead
|
||||
await brain.add({ data, type, metadata }) // Modern unified add
|
||||
await brain.find({ limit: 100 }) // Natural language search
|
||||
await brain.relate({ from, to, type }) // Clean relationship creation
|
||||
await brain.getRelations() // Modern relationship queries
|
||||
await brain.delete(id) // Unified deletion
|
||||
// Relationships auto-cascade when entities are deleted
|
||||
```
|
||||
|
||||
## 📋 Entity Management Patterns
|
||||
|
||||
### ❌ **WRONG - v2.x Style**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Old API patterns
|
||||
import { Brainy } from 'old-brainy' // ❌ Wrong import
|
||||
|
||||
const brain = new Brainy({ // ❌ Old class name
|
||||
complexConfig: true
|
||||
})
|
||||
|
||||
const id = await brain.addNoun( // ❌ Deprecated method
|
||||
"John Smith is a developer",
|
||||
"Person",
|
||||
{ role: "engineer" }
|
||||
)
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Modern Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Basic entity creation
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy() // ✅ Zero config
|
||||
await brain.init()
|
||||
|
||||
const id = await brain.add({
|
||||
data: "John Smith is a developer",
|
||||
type: NounType.Person,
|
||||
metadata: { role: "engineer", team: "backend" }
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Bulk entity creation
|
||||
const entities = [
|
||||
{ data: "React framework", type: NounType.Technology },
|
||||
{ data: "Vue.js framework", type: NounType.Technology },
|
||||
{ data: "Angular framework", type: NounType.Technology }
|
||||
]
|
||||
|
||||
const ids = await Promise.all(
|
||||
entities.map(entity => brain.add(entity))
|
||||
)
|
||||
|
||||
// ✅ Pattern 3: Entity with pre-computed vector
|
||||
const customVector = await brain.embed("Custom text")
|
||||
const vectorId = await brain.add({
|
||||
data: "Optimized content",
|
||||
type: NounType.Document,
|
||||
vector: customVector, // Skip re-embedding
|
||||
metadata: { source: "api", optimized: true }
|
||||
})
|
||||
```
|
||||
|
||||
## 🔍 Search & Discovery Patterns
|
||||
|
||||
### ❌ **WRONG - Confusing Old Patterns**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Mixing old and new APIs
|
||||
const results1 = await brain.searchText("query") // ❌ Old method
|
||||
const results2 = await brain.getNouns({ filter }) // ❌ Doesn't exist
|
||||
const results3 = await brain.findSimilar(text) // ❌ Unclear naming
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Clean Search Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Natural language search
|
||||
const results = await brain.find("React developers working on authentication")
|
||||
|
||||
// ✅ Pattern 2: Structured search with filters
|
||||
const filteredResults = await brain.find({
|
||||
like: "machine learning", // Vector similarity
|
||||
where: { // Metadata filtering
|
||||
type: NounType.Document,
|
||||
year: { $gte: 2020 },
|
||||
status: "published"
|
||||
},
|
||||
limit: 50,
|
||||
orderBy: 'relevance'
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Similarity search
|
||||
const similarItems = await brain.similar({
|
||||
to: existingEntityId, // Find items similar to this
|
||||
threshold: 0.8, // Minimum similarity
|
||||
limit: 10,
|
||||
exclude: [existingEntityId] // Don't include the source
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Advanced search with relationships
|
||||
const connectedResults = await brain.find({
|
||||
like: "frontend frameworks",
|
||||
connected: {
|
||||
to: reactId, // Connected to React
|
||||
via: "related-to", // Through this relationship
|
||||
depth: 2 // Up to 2 hops away
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## 🔗 Relationship Patterns
|
||||
|
||||
### ❌ **WRONG - Old Relationship APIs**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Old relationship patterns
|
||||
await brain.addVerb(sourceId, targetId, "uses", { strength: 0.9 }) // ❌ Old API
|
||||
const verbs = await brain.getVerbsBySource(sourceId) // ❌ Removed
|
||||
await brain.deleteVerb(verbId) // ❌ Old pattern
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Modern Relationship Management**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Create relationships
|
||||
const relationId = await brain.relate({
|
||||
from: developerId,
|
||||
to: frameworkId,
|
||||
type: VerbType.Uses,
|
||||
metadata: {
|
||||
since: "2023-01-01",
|
||||
proficiency: "expert",
|
||||
hours_per_week: 40
|
||||
}
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Query relationships
|
||||
const relationships = await brain.getRelations({
|
||||
from: developerId, // Relationships from this entity
|
||||
type: VerbType.Uses, // Of this type
|
||||
limit: 100
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Bidirectional relationships
|
||||
await brain.relate({
|
||||
from: projectId,
|
||||
to: developerId,
|
||||
type: VerbType.AssignedTo,
|
||||
bidirectional: true, // Creates reverse relationship
|
||||
metadata: { role: "lead", start_date: "2024-01-01" }
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Relationship-based discovery
|
||||
const collaborators = await brain.find({
|
||||
connected: {
|
||||
to: currentProjectId,
|
||||
via: VerbType.WorksOn,
|
||||
direction: "incoming" // Who works on this project
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## 🗃️ Data Retrieval Patterns
|
||||
|
||||
### ❌ **WRONG - Inefficient Patterns**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Loading everything
|
||||
const everything = await brain.getNouns({ limit: 1000000 }) // ❌ Crashes
|
||||
const allData = await brain.exportAll() // ❌ Memory explosion
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Efficient Data Access**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Paginated retrieval
|
||||
async function getAllEntitiesPaginated() {
|
||||
const pageSize = 100
|
||||
let offset = 0
|
||||
let allEntities = []
|
||||
|
||||
while (true) {
|
||||
const page = await brain.find({
|
||||
limit: pageSize,
|
||||
offset: offset
|
||||
})
|
||||
|
||||
if (page.length === 0) break
|
||||
|
||||
allEntities.push(...page)
|
||||
offset += pageSize
|
||||
|
||||
// Optional: Progress reporting
|
||||
console.log(`Loaded ${allEntities.length} entities...`)
|
||||
}
|
||||
|
||||
return allEntities
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Streaming large datasets
|
||||
async function* streamEntities() {
|
||||
const pageSize = 50
|
||||
let offset = 0
|
||||
|
||||
while (true) {
|
||||
const page = await brain.find({
|
||||
limit: pageSize,
|
||||
offset: offset
|
||||
})
|
||||
|
||||
if (page.length === 0) break
|
||||
|
||||
for (const entity of page) {
|
||||
yield entity
|
||||
}
|
||||
|
||||
offset += pageSize
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
for await (const entity of streamEntities()) {
|
||||
await processEntity(entity)
|
||||
}
|
||||
|
||||
// ✅ Pattern 3: Specific entity retrieval
|
||||
const entity = await brain.get(entityId)
|
||||
if (entity) {
|
||||
console.log('Entity data:', entity.data)
|
||||
console.log('Metadata:', entity.metadata)
|
||||
} else {
|
||||
console.log('Entity not found')
|
||||
}
|
||||
```
|
||||
|
||||
## 🔄 Update & Delete Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Update Patterns**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Recreating entities
|
||||
await brain.delete(oldId)
|
||||
const newId = await brain.add(updatedData) // ❌ Loses relationships
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Update Operations**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Update entity data
|
||||
await brain.update(entityId, {
|
||||
data: "Updated content here",
|
||||
metadata: {
|
||||
lastModified: Date.now(),
|
||||
version: "2.0"
|
||||
}
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Partial metadata updates
|
||||
await brain.updateMetadata(entityId, {
|
||||
status: "published",
|
||||
tags: ["important", "featured"]
|
||||
// Merges with existing metadata
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Safe deletion with cascade options
|
||||
await brain.delete(entityId, {
|
||||
cascade: true, // Delete related relationships
|
||||
backup: true // Create backup before deletion
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Bulk operations
|
||||
const updateOperations = entities.map(entity => ({
|
||||
id: entity.id,
|
||||
changes: { status: "processed" }
|
||||
}))
|
||||
|
||||
await brain.updateMany(updateOperations)
|
||||
```
|
||||
|
||||
## 🧮 Vector & Embedding Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Vector Handling**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Manual embedding without understanding
|
||||
const vector = await brain.embed(text)
|
||||
// Store vector somewhere manually // ❌ Missing integration
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Smart Vector Operations**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Automatic embedding (recommended)
|
||||
const id = await brain.add({
|
||||
data: "Content to be embedded",
|
||||
type: NounType.Document
|
||||
// Vector computed automatically
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Pre-computed vectors for optimization
|
||||
const texts = ["Text 1", "Text 2", "Text 3"]
|
||||
const vectors = await Promise.all(
|
||||
texts.map(text => brain.embed(text))
|
||||
)
|
||||
|
||||
const entities = await Promise.all(
|
||||
texts.map((text, i) => brain.add({
|
||||
data: text,
|
||||
type: NounType.Document,
|
||||
vector: vectors[i] // Skip re-embedding
|
||||
}))
|
||||
)
|
||||
|
||||
// ✅ Pattern 3: Vector similarity search
|
||||
const queryVector = await brain.embed("search query")
|
||||
const similar = await brain.similar({
|
||||
vector: queryVector, // Use vector directly
|
||||
threshold: 0.75,
|
||||
limit: 20
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Compare vectors directly
|
||||
const vector1 = await brain.embed("First text")
|
||||
const vector2 = await brain.embed("Second text")
|
||||
const similarity = brain.computeSimilarity(vector1, vector2)
|
||||
console.log(`Similarity: ${similarity}`)
|
||||
```
|
||||
|
||||
## 🏗️ Configuration Patterns
|
||||
|
||||
### ❌ **WRONG - Over-Configuration**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Complex configurations that break
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'complex',
|
||||
options: {
|
||||
nested: {
|
||||
configuration: true,
|
||||
that: "breaks"
|
||||
}
|
||||
}
|
||||
},
|
||||
embedding: {
|
||||
customModel: "broken-model",
|
||||
dimensions: 999999
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Smart Configuration**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Zero configuration (recommended)
|
||||
const brain = new Brainy() // Auto-detects everything
|
||||
await brain.init()
|
||||
|
||||
// ✅ Pattern 2: Simple storage selection
|
||||
const fsBrain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './data' }
|
||||
})
|
||||
|
||||
const cloudBrain = new Brainy({
|
||||
storage: { type: 's3', bucket: 'my-data' }
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Production configuration
|
||||
const prodBrain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucket: process.env.BRAINY_BUCKET,
|
||||
region: process.env.AWS_REGION
|
||||
},
|
||||
silent: true, // No console output
|
||||
distributed: true, // Enable clustering
|
||||
cache: { maxSize: 10000 } // Larger cache
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Development vs production
|
||||
const isDev = process.env.NODE_ENV === 'development'
|
||||
|
||||
const brain = new Brainy({
|
||||
storage: isDev
|
||||
? { type: 'memory' } // Fast for dev
|
||||
: { type: 'filesystem', path: './brainy-data' }, // Persistent for prod
|
||||
silent: !isDev, // Verbose in dev, quiet in prod
|
||||
cache: { maxSize: isDev ? 100 : 5000 }
|
||||
})
|
||||
```
|
||||
|
||||
## 🔄 Migration from v2.x
|
||||
|
||||
### ✅ **Migration Patterns**
|
||||
|
||||
```typescript
|
||||
// If you have old v2.x code, here's how to migrate:
|
||||
|
||||
// OLD v2.x:
|
||||
// await brain.addNoun(text, type, metadata)
|
||||
// NEW v3.x:
|
||||
await brain.add({ data: text, type, metadata })
|
||||
|
||||
// OLD v2.x:
|
||||
// await brain.getNouns({ pagination: { limit: 100 } })
|
||||
// NEW v3.x:
|
||||
await brain.find({ limit: 100 })
|
||||
|
||||
// OLD v2.x:
|
||||
// await brain.addVerb(sourceId, targetId, verbType, metadata)
|
||||
// NEW v3.x:
|
||||
await brain.relate({ from: sourceId, to: targetId, type: verbType, metadata })
|
||||
|
||||
// OLD v2.x:
|
||||
// await brain.searchText(query)
|
||||
// NEW v3.x:
|
||||
await brain.find(query) // More powerful natural language search
|
||||
```
|
||||
|
||||
## 🚀 Performance Patterns
|
||||
|
||||
### ✅ **High-Performance Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Batch operations
|
||||
const entities = [/* large array */]
|
||||
const batchSize = 100
|
||||
|
||||
for (let i = 0; i < entities.length; i += batchSize) {
|
||||
const batch = entities.slice(i, i + batchSize)
|
||||
await Promise.all(
|
||||
batch.map(entity => brain.add(entity))
|
||||
)
|
||||
|
||||
// Optional: Rate limiting
|
||||
await new Promise(resolve => setTimeout(resolve, 100))
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Connection pooling for distributed
|
||||
const brain = new Brainy({
|
||||
distributed: true,
|
||||
connectionPool: {
|
||||
min: 5,
|
||||
max: 50,
|
||||
acquireTimeoutMillis: 30000
|
||||
}
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Efficient caching
|
||||
const brain = new Brainy({
|
||||
cache: {
|
||||
maxSize: 10000, // Number of items
|
||||
ttl: 300000, // 5 minutes
|
||||
updateAgeOnGet: true // LRU behavior
|
||||
}
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Memory-conscious operations
|
||||
const results = await brain.find({
|
||||
query: "large dataset query",
|
||||
limit: 1000, // Reasonable limit
|
||||
includeVectors: false // Exclude vectors if not needed
|
||||
})
|
||||
```
|
||||
|
||||
## 🛡️ Error Handling Patterns
|
||||
|
||||
### ✅ **Robust Error Handling**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Specific error handling
|
||||
try {
|
||||
const result = await brain.add({ data, type, metadata })
|
||||
return result
|
||||
} catch (error) {
|
||||
if (error.code === 'DUPLICATE_ENTITY') {
|
||||
console.log('Entity already exists, updating instead...')
|
||||
return await brain.update(error.existingId, { data, metadata })
|
||||
} else if (error.code === 'STORAGE_FULL') {
|
||||
throw new Error('Storage capacity exceeded')
|
||||
} else if (error.code === 'EMBEDDING_FAILED') {
|
||||
console.warn('Embedding failed, retrying with simpler text...')
|
||||
return await brain.add({
|
||||
data: data.substring(0, 1000), // Truncate
|
||||
type,
|
||||
metadata
|
||||
})
|
||||
}
|
||||
throw error
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Retry with exponential backoff
|
||||
async function resilientAdd(data: any, maxRetries = 3) {
|
||||
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
||||
try {
|
||||
return await brain.add(data)
|
||||
} catch (error) {
|
||||
if (attempt === maxRetries) throw error
|
||||
|
||||
const delay = Math.pow(2, attempt) * 1000
|
||||
console.warn(`Attempt ${attempt} failed, retrying in ${delay}ms...`)
|
||||
await new Promise(resolve => setTimeout(resolve, delay))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ✅ Pattern 3: Graceful degradation
|
||||
async function robustSearch(query: string) {
|
||||
try {
|
||||
// Try advanced semantic search first
|
||||
return await brain.find({
|
||||
like: query,
|
||||
threshold: 0.8,
|
||||
limit: 50
|
||||
})
|
||||
} catch (error) {
|
||||
console.warn('Semantic search failed, falling back to basic search:', error.message)
|
||||
|
||||
try {
|
||||
// Fallback to simple text search
|
||||
return await brain.find(query)
|
||||
} catch (fallbackError) {
|
||||
console.error('All search methods failed:', fallbackError.message)
|
||||
return [] // Return empty results rather than crash
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 Monitoring Patterns
|
||||
|
||||
### ✅ **Production Monitoring**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Performance monitoring
|
||||
const startTime = Date.now()
|
||||
const result = await brain.add(data)
|
||||
const duration = Date.now() - startTime
|
||||
|
||||
if (duration > 1000) {
|
||||
console.warn(`Slow add operation: ${duration}ms`)
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Health checks
|
||||
async function healthCheck() {
|
||||
try {
|
||||
// Test basic operations
|
||||
const testId = await brain.add({
|
||||
data: "health check",
|
||||
type: NounType.System,
|
||||
metadata: { test: true }
|
||||
})
|
||||
|
||||
await brain.get(testId)
|
||||
await brain.delete(testId)
|
||||
|
||||
return { status: 'healthy', timestamp: Date.now() }
|
||||
} catch (error) {
|
||||
return {
|
||||
status: 'unhealthy',
|
||||
error: error.message,
|
||||
timestamp: Date.now()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ✅ Pattern 3: Metrics collection
|
||||
class BrainyMetrics {
|
||||
private metrics = {
|
||||
operations: 0,
|
||||
errors: 0,
|
||||
totalTime: 0
|
||||
}
|
||||
|
||||
async timedOperation<T>(operation: () => Promise<T>): Promise<T> {
|
||||
const start = Date.now()
|
||||
try {
|
||||
const result = await operation()
|
||||
this.metrics.operations++
|
||||
this.metrics.totalTime += Date.now() - start
|
||||
return result
|
||||
} catch (error) {
|
||||
this.metrics.errors++
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
getStats() {
|
||||
return {
|
||||
...this.metrics,
|
||||
avgTime: this.metrics.totalTime / this.metrics.operations || 0,
|
||||
errorRate: this.metrics.errors / this.metrics.operations || 0
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🎯 Summary: Modern Brainy v3.x Best Practices
|
||||
|
||||
| ❌ **Avoid v2.x** | ✅ **Use v3.x** |
|
||||
|------------------|----------------|
|
||||
| `addNoun()` | `add()` |
|
||||
| `getNouns()` | `find()` |
|
||||
| `addVerb()` | `relate()` |
|
||||
| `getVerbs()` | `getRelations()` |
|
||||
| `deleteNoun()` | `delete()` |
|
||||
| Complex configs | Zero-config with `new Brainy()` |
|
||||
| Manual pagination | Built-in smart pagination |
|
||||
| String-based search | Natural language queries |
|
||||
|
||||
---
|
||||
|
||||
**🎉 Following these patterns gives you:**
|
||||
- 🚀 **Modern APIs** that are actively maintained
|
||||
- ⚡ **Better performance** with intelligent defaults
|
||||
- 🛡️ **Robust error handling** with specific error types
|
||||
- 📈 **Scalable patterns** for production applications
|
||||
- 🧠 **Natural language** search capabilities
|
||||
|
||||
**Next:** [Neural API Patterns →](./NEURAL_API_PATTERNS.md) | [VFS Patterns →](./vfs/COMMON_PATTERNS.md)
|
||||
435
docs/CREATING-AUGMENTATIONS.md
Normal file
435
docs/CREATING-AUGMENTATIONS.md
Normal file
|
|
@ -0,0 +1,435 @@
|
|||
# Creating Augmentations for Brainy
|
||||
|
||||
> **Updated** - Includes metadata structure changes and type system improvements
|
||||
|
||||
## The BrainyAugmentation Interface
|
||||
|
||||
Every augmentation implements this simple yet powerful interface:
|
||||
|
||||
```typescript
|
||||
interface BrainyAugmentation {
|
||||
// Identification
|
||||
name: string // Unique name for your augmentation
|
||||
|
||||
// Execution control
|
||||
timing: 'before' | 'after' | 'around' | 'replace' // When to execute
|
||||
operations: string[] // Which operations to intercept
|
||||
priority: number // Execution order (higher = first)
|
||||
|
||||
// Lifecycle methods
|
||||
initialize(context: AugmentationContext): Promise<void>
|
||||
execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T>
|
||||
shutdown?(): Promise<void> // Optional cleanup
|
||||
}
|
||||
```
|
||||
|
||||
## Breaking Changes for Augmentation Developers
|
||||
|
||||
### 1. Metadata Structure Separation
|
||||
Brainy introduces strict metadata/vector separation for billion-scale performance:
|
||||
|
||||
```typescript
|
||||
// ✅ Metadata has required type field
|
||||
interface NounMetadata {
|
||||
noun: NounType // Required! Must be a valid noun type
|
||||
[key: string]: any // Your custom metadata
|
||||
}
|
||||
|
||||
interface VerbMetadata {
|
||||
verb: VerbType // Required! Must be a valid verb type
|
||||
sourceId: string
|
||||
targetId: string
|
||||
[key: string]: any
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Storage Adapter Return Types
|
||||
Storage adapters now return different types at different boundaries:
|
||||
|
||||
```typescript
|
||||
// Internal methods: Pure structures (no metadata)
|
||||
abstract _getNoun(id: string): Promise<HNSWNoun | null>
|
||||
|
||||
// Public API: WithMetadata structures
|
||||
abstract getNoun(id: string): Promise<HNSWNounWithMetadata | null>
|
||||
```
|
||||
|
||||
### 3. Verb Property Renamed
|
||||
The verb relationship field changed from `type` to `verb`:
|
||||
|
||||
```typescript
|
||||
// ❌ v3.x
|
||||
verb.type === 'relatedTo'
|
||||
|
||||
// ✅ Current
|
||||
verb.verb === 'relatedTo'
|
||||
```
|
||||
|
||||
## Creating a Storage Augmentation
|
||||
|
||||
Storage augmentations are special - they provide the storage backend for Brainy.
|
||||
|
||||
### Important: Storage Requirements
|
||||
|
||||
Your storage adapter MUST:
|
||||
1. **Wrap metadata** with required `noun`/`verb` fields
|
||||
2. **Return pure structures** from internal `_methods`
|
||||
3. **Return WithMetadata types** from public methods
|
||||
|
||||
```typescript
|
||||
import { StorageAugmentation } from 'brainy/augmentations'
|
||||
import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy'
|
||||
|
||||
export class MyCustomStorage extends BaseStorageAdapter {
|
||||
// Internal method: Returns pure structure
|
||||
async _getNoun(id: string): Promise<HNSWNoun | null> {
|
||||
const data = await this.fetchFromDatabase(id)
|
||||
return data ? {
|
||||
id: data.id,
|
||||
vector: data.vector,
|
||||
nounType: data.type
|
||||
} : null
|
||||
}
|
||||
|
||||
// Public method: Returns WithMetadata structure
|
||||
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
|
||||
const noun = await this._getNoun(id)
|
||||
if (!noun) return null
|
||||
|
||||
// Fetch metadata separately
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
|
||||
return {
|
||||
...noun,
|
||||
metadata: metadata || { noun: noun.nounType || 'thing' }
|
||||
}
|
||||
}
|
||||
|
||||
// CRITICAL: Always save with proper metadata structure
|
||||
async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
|
||||
// Validate metadata has required 'noun' field
|
||||
if (!metadata?.noun) {
|
||||
throw new Error('NounMetadata requires "noun" field')
|
||||
}
|
||||
|
||||
await this.database.save({
|
||||
id: noun.id,
|
||||
vector: noun.vector,
|
||||
nounType: noun.nounType,
|
||||
metadata: metadata // Stored separately
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
export class MyStorageAugmentation extends StorageAugmentation {
|
||||
private config: MyStorageConfig
|
||||
|
||||
constructor(config: MyStorageConfig) {
|
||||
super()
|
||||
this.name = 'my-custom-storage'
|
||||
this.config = config
|
||||
}
|
||||
|
||||
// Called during storage resolution phase
|
||||
async provideStorage(): Promise<StorageAdapter> {
|
||||
const storage = new MyCustomStorage(this.config)
|
||||
this.storageAdapter = storage
|
||||
return storage
|
||||
}
|
||||
|
||||
// Called during augmentation initialization
|
||||
protected async onInitialize(): Promise<void> {
|
||||
await this.storageAdapter!.init()
|
||||
this.log(`Custom storage initialized`)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Using Your Storage Augmentation
|
||||
|
||||
```typescript
|
||||
// Register before brain.init()
|
||||
const brain = new Brainy()
|
||||
brain.augmentations.register(new MyStorageAugmentation({
|
||||
connectionString: 'redis://localhost:6379'
|
||||
}))
|
||||
await brain.init() // Will use your storage!
|
||||
```
|
||||
|
||||
## Creating a Feature Augmentation
|
||||
|
||||
Here's a complete example of a caching augmentation:
|
||||
|
||||
```typescript
|
||||
import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations'
|
||||
|
||||
export class CachingAugmentation extends BaseAugmentation {
|
||||
private cache = new Map<string, any>()
|
||||
|
||||
constructor() {
|
||||
super()
|
||||
this.name = 'smart-cache'
|
||||
this.timing = 'around' // Wrap operations
|
||||
this.operations = ['search'] // Only cache searches
|
||||
this.priority = 50 // Mid-priority
|
||||
}
|
||||
|
||||
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
|
||||
if (operation === 'search') {
|
||||
// Check cache
|
||||
const cacheKey = JSON.stringify(params)
|
||||
if (this.cache.has(cacheKey)) {
|
||||
this.log('Cache hit!')
|
||||
return this.cache.get(cacheKey)
|
||||
}
|
||||
|
||||
// Execute and cache
|
||||
const result = await next()
|
||||
this.cache.set(cacheKey, result)
|
||||
return result
|
||||
}
|
||||
|
||||
// Pass through other operations
|
||||
return next()
|
||||
}
|
||||
|
||||
protected async onInitialize(): Promise<void> {
|
||||
this.log('Cache initialized')
|
||||
}
|
||||
|
||||
async shutdown(): Promise<void> {
|
||||
this.cache.clear()
|
||||
await super.shutdown()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## The Four Timing Modes
|
||||
|
||||
### 1. `before` - Pre-processing
|
||||
```typescript
|
||||
timing = 'before'
|
||||
async execute(op, params, next) {
|
||||
// Validate/transform input
|
||||
const validated = await validate(params)
|
||||
return next(validated) // Pass modified params
|
||||
}
|
||||
```
|
||||
|
||||
### 2. `after` - Post-processing
|
||||
```typescript
|
||||
timing = 'after'
|
||||
async execute(op, params, next) {
|
||||
const result = await next()
|
||||
// Log, analyze, or modify result
|
||||
console.log(`Operation ${op} returned:`, result)
|
||||
return result
|
||||
}
|
||||
```
|
||||
|
||||
### 3. `around` - Wrapping (middleware)
|
||||
```typescript
|
||||
timing = 'around'
|
||||
async execute(op, params, next) {
|
||||
console.log('Starting', op)
|
||||
try {
|
||||
const result = await next()
|
||||
console.log('Success', op)
|
||||
return result
|
||||
} catch (error) {
|
||||
console.log('Failed', op, error)
|
||||
throw error
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4. `replace` - Complete replacement
|
||||
```typescript
|
||||
timing = 'replace'
|
||||
async execute(op, params, next) {
|
||||
// Don't call next() - replace entirely!
|
||||
return myCustomImplementation(params)
|
||||
}
|
||||
```
|
||||
|
||||
## Operations You Can Intercept
|
||||
|
||||
Common operations in Brainy:
|
||||
- `'storage'` - Storage resolution (special)
|
||||
- `'add'` - Adding data
|
||||
- `'search'`, `'similar'` - Searching
|
||||
- `'update'`, `'delete'` - Modifications
|
||||
- `'saveNoun'`, `'saveVerb'` - Storage operations
|
||||
- `'all'` - Intercept everything
|
||||
|
||||
## Context Available to Augmentations
|
||||
|
||||
```typescript
|
||||
interface AugmentationContext {
|
||||
brain: Brainy // The brain instance
|
||||
storage: StorageAdapter // Storage backend
|
||||
config: BrainyConfig // Configuration
|
||||
log: (message: string, level?: 'info' | 'warn' | 'error') => void
|
||||
}
|
||||
```
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
### 1. Redis Storage Augmentation
|
||||
```typescript
|
||||
export class RedisStorageAugmentation extends StorageAugmentation {
|
||||
async provideStorage(): Promise<StorageAdapter> {
|
||||
return new RedisAdapter({
|
||||
host: 'localhost',
|
||||
port: 6379,
|
||||
// Implement full StorageAdapter interface
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Audit Trail Augmentation
|
||||
```typescript
|
||||
export class AuditAugmentation extends BaseAugmentation {
|
||||
timing = 'after'
|
||||
operations = ['add', 'update', 'delete']
|
||||
|
||||
async execute(op, params, next) {
|
||||
const result = await next()
|
||||
|
||||
// Log to audit trail
|
||||
await this.logAudit({
|
||||
operation: op,
|
||||
params,
|
||||
result,
|
||||
timestamp: new Date(),
|
||||
user: this.context.config.currentUser
|
||||
})
|
||||
|
||||
return result
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Rate Limiting Augmentation
|
||||
```typescript
|
||||
export class RateLimitAugmentation extends BaseAugmentation {
|
||||
timing = 'before'
|
||||
operations = ['search']
|
||||
private limiter = new RateLimiter({ rps: 100 })
|
||||
|
||||
async execute(op, params, next) {
|
||||
await this.limiter.acquire() // Wait if rate limited
|
||||
return next()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Publishing to Brain Cloud Marketplace
|
||||
|
||||
Future capability for premium augmentations:
|
||||
|
||||
```typescript
|
||||
// package.json
|
||||
{
|
||||
"name": "@brain-cloud/redis-storage",
|
||||
"brainy": {
|
||||
"type": "augmentation",
|
||||
"category": "storage",
|
||||
"premium": true
|
||||
}
|
||||
}
|
||||
|
||||
// Users can install via:
|
||||
// brainy augment install redis-storage
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### General Practices
|
||||
|
||||
1. **Use BaseAugmentation** - Provides common functionality
|
||||
2. **Set appropriate priority** - Storage (100), System (80-99), Features (10-50)
|
||||
3. **Be selective with operations** - Don't use 'all' unless necessary
|
||||
4. **Handle errors gracefully** - Don't break the chain
|
||||
5. **Clean up in shutdown()** - Release resources
|
||||
6. **Log appropriately** - Use context.log() for consistent output
|
||||
7. **Document your augmentation** - Include examples
|
||||
|
||||
### Specific Best Practices
|
||||
|
||||
8. **Always include `noun` field** when creating/modifying NounMetadata:
|
||||
```typescript
|
||||
const metadata: NounMetadata = {
|
||||
noun: 'thing', // REQUIRED!
|
||||
yourField: 'value'
|
||||
}
|
||||
```
|
||||
|
||||
9. **Use `verb` property** not `type` when working with relationships:
|
||||
```typescript
|
||||
// ✅ Correct
|
||||
if (verb.verb === 'relatedTo') { ... }
|
||||
|
||||
// ❌ Wrong (v3.x pattern)
|
||||
if (verb.type === 'relatedTo') { ... }
|
||||
```
|
||||
|
||||
10. **Access metadata correctly** from storage:
|
||||
```typescript
|
||||
// ✅ Correct - metadata is already structured
|
||||
const nounType = noun.metadata.noun
|
||||
|
||||
// ⚠️ Fallback pattern for robustness
|
||||
const nounType = noun.metadata?.noun || 'thing'
|
||||
```
|
||||
|
||||
11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations:
|
||||
```typescript
|
||||
// ✅ Good - Separate concerns
|
||||
await storage.saveNoun(noun)
|
||||
await storage.saveMetadata(noun.id, metadata)
|
||||
|
||||
// ❌ Bad - Mixing concerns
|
||||
await storage.saveNounWithEverything(combinedData)
|
||||
```
|
||||
|
||||
## Testing Your Augmentation
|
||||
|
||||
```typescript
|
||||
import { Brainy } from 'brainy'
|
||||
import { MyAugmentation } from './my-augmentation'
|
||||
|
||||
describe('MyAugmentation', () => {
|
||||
let brain: Brainy
|
||||
|
||||
beforeEach(async () => {
|
||||
brain = new Brainy()
|
||||
brain.augmentations.register(new MyAugmentation())
|
||||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.destroy()
|
||||
})
|
||||
|
||||
it('should enhance searches', async () => {
|
||||
// Test your augmentation's effect
|
||||
const results = await brain.search('test')
|
||||
expect(results).toHaveProperty('enhanced', true)
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
Augmentations are Brainy's extension system. They can:
|
||||
- Replace storage backends
|
||||
- Add caching layers
|
||||
- Implement audit trails
|
||||
- Add rate limiting
|
||||
- Sync with external systems
|
||||
- Transform data
|
||||
- And much more!
|
||||
|
||||
The unified BrainyAugmentation interface makes it easy to create powerful extensions while maintaining consistency across the entire system.
|
||||
|
|
@ -1,271 +0,0 @@
|
|||
# Data Model
|
||||
|
||||
> How Brainy stores entities and relationships, and the critical distinction between `data` and `metadata`.
|
||||
|
||||
---
|
||||
|
||||
## Entity (Noun)
|
||||
|
||||
An entity is the fundamental data unit in Brainy. Every entity has:
|
||||
|
||||
| Field | Type | Indexed | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `id` | `string` | Primary key | UUID v4 (auto-generated or custom) |
|
||||
| `data` | `any` | **HNSW vector index** | Content used for semantic/hybrid search. Strings auto-embed. |
|
||||
| `metadata` | `object` | **MetadataIndex** | Structured queryable fields (tags, dates, flags, etc.) |
|
||||
| `type` | `NounType` | MetadataIndex (as `noun`) | Entity type classification |
|
||||
| `vector` | `number[]` | HNSW | 384-dim embedding (auto-computed from `data` or user-provided) |
|
||||
| `confidence` | `number` | MetadataIndex | Type classification confidence (0-1) |
|
||||
| `weight` | `number` | MetadataIndex | Entity importance/salience (0-1) |
|
||||
| `service` | `string` | MetadataIndex | Multi-tenancy identifier |
|
||||
| `createdAt` | `number` | MetadataIndex | Creation timestamp (ms since epoch) |
|
||||
| `updatedAt` | `number` | MetadataIndex | Last update timestamp (ms since epoch) |
|
||||
| `createdBy` | `object` | MetadataIndex | Source augmentation info |
|
||||
|
||||
### Example
|
||||
|
||||
```typescript
|
||||
const id = await brain.add({
|
||||
data: 'John Smith is a software engineer at Acme Corp', // → embedded into vector
|
||||
type: NounType.Person,
|
||||
metadata: { // → indexed, queryable via where filters
|
||||
role: 'engineer',
|
||||
department: 'backend',
|
||||
yearsExperience: 8
|
||||
},
|
||||
confidence: 0.95,
|
||||
weight: 0.7
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Relationship (Verb)
|
||||
|
||||
A relationship is a typed, directed edge connecting two entities.
|
||||
|
||||
| Field | Type | Indexed | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `id` | `string` | Primary key | UUID v4 (auto-generated) |
|
||||
| `from` | `string` | **GraphAdjacencyIndex** | Source entity ID |
|
||||
| `to` | `string` | **GraphAdjacencyIndex** | Target entity ID |
|
||||
| `type` | `VerbType` | GraphAdjacencyIndex (as `verb`) | Relationship type classification |
|
||||
| `data` | `any` | — | Opaque content (overrides auto-computed vector if provided) |
|
||||
| `metadata` | `object` | — | Structured fields on the edge |
|
||||
| `weight` | `number` | — | Connection strength (0-1, default: 1.0) |
|
||||
| `confidence` | `number` | — | Relationship certainty (0-1) |
|
||||
| `evidence` | `RelationEvidence` | — | Why this relationship was detected |
|
||||
| `createdAt` | `number` | — | Creation timestamp (ms since epoch) |
|
||||
| `updatedAt` | `number` | — | Last update timestamp (ms since epoch) |
|
||||
| `service` | `string` | — | Multi-tenancy identifier |
|
||||
|
||||
### Example
|
||||
|
||||
```typescript
|
||||
const relId = await brain.relate({
|
||||
from: personId,
|
||||
to: projectId,
|
||||
type: VerbType.WorksOn,
|
||||
data: 'Lead engineer on the AI module', // Optional: content for this edge
|
||||
metadata: { // Optional: queryable edge fields
|
||||
role: 'lead',
|
||||
startDate: '2024-01-15'
|
||||
},
|
||||
weight: 0.9
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data vs Metadata
|
||||
|
||||
This is the most important concept in Brainy's storage model:
|
||||
|
||||
### `data` — Content for Semantic Search
|
||||
|
||||
- Embedded into a 384-dimensional vector via the WASM embedding engine
|
||||
- Searchable via **semantic similarity** (HNSW vector index) and **hybrid text+semantic** search
|
||||
- Queried by passing `query` to `find()`:
|
||||
```typescript
|
||||
brain.find({ query: 'machine learning algorithms' })
|
||||
```
|
||||
- **NOT** indexed by MetadataIndex — you cannot use `where` filters on `data`
|
||||
- Stored opaquely: strings, objects, numbers — anything goes
|
||||
|
||||
### `metadata` — Structured Queryable Fields
|
||||
|
||||
- Indexed by MetadataIndex with O(1) lookups per field
|
||||
- Queryable via `where` filters using [BFO operators](./QUERY_OPERATORS.md):
|
||||
```typescript
|
||||
brain.find({
|
||||
where: {
|
||||
department: 'engineering',
|
||||
yearsExperience: { greaterThan: 5 },
|
||||
tags: { contains: 'senior' }
|
||||
}
|
||||
})
|
||||
```
|
||||
- **NOT** used for vector/semantic search
|
||||
- Must be a flat or lightly nested object
|
||||
|
||||
### Quick Reference
|
||||
|
||||
| | `data` | `metadata` |
|
||||
|---|---|---|
|
||||
| **Purpose** | Content for embedding / semantic search | Structured fields for filtering |
|
||||
| **Searched by** | `find({ query })` — vector similarity, hybrid text+semantic | `find({ where })` — exact, range, set operators |
|
||||
| **Indexed by** | HNSW vector index | MetadataIndex |
|
||||
| **Queryable with operators?** | No | Yes (`equals`, `greaterThan`, `oneOf`, etc.) |
|
||||
| **Auto-embedded?** | Yes (strings → 384-dim vectors) | No |
|
||||
| **Typical content** | Text descriptions, document content | Tags, dates, status flags, categories, numeric fields |
|
||||
|
||||
### Common Pattern
|
||||
|
||||
```typescript
|
||||
// Add an article
|
||||
await brain.add({
|
||||
data: 'A deep dive into transformer architectures and attention mechanisms',
|
||||
type: NounType.Document,
|
||||
metadata: {
|
||||
title: 'Transformer Deep Dive',
|
||||
author: 'Dr. Chen',
|
||||
publishedYear: 2024,
|
||||
tags: ['AI', 'transformers', 'NLP'],
|
||||
status: 'published'
|
||||
}
|
||||
})
|
||||
|
||||
// Search by content (semantic — searches data)
|
||||
const results = await brain.find({ query: 'neural network attention' })
|
||||
|
||||
// Filter by fields (exact — queries metadata)
|
||||
const recent = await brain.find({
|
||||
where: {
|
||||
publishedYear: { greaterThan: 2023 },
|
||||
status: 'published'
|
||||
}
|
||||
})
|
||||
|
||||
// Combine both (Triple Intelligence)
|
||||
const precise = await brain.find({
|
||||
query: 'attention mechanisms', // Semantic search on data
|
||||
where: { author: 'Dr. Chen' }, // Metadata filter
|
||||
connected: { from: authorId, depth: 1 } // Graph traversal
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Storage Field Naming
|
||||
|
||||
Internally, Brainy uses different field names in storage vs the public API:
|
||||
|
||||
| Public API (Entity/Relation) | Storage (metadata object) | Notes |
|
||||
|------------------------------|--------------------------|-------|
|
||||
| `type` | `noun` | Entity type stored as `noun` |
|
||||
| `from` | `sourceId` | Relationship source |
|
||||
| `to` | `targetId` | Relationship target |
|
||||
| `type` (on Relation) | `verb` | Relationship type stored as `verb` |
|
||||
|
||||
When querying with `find()`, you can use:
|
||||
- `type` parameter (convenience alias, equivalent to `where.noun`)
|
||||
- `where.noun` directly
|
||||
|
||||
```typescript
|
||||
// These are equivalent:
|
||||
brain.find({ type: NounType.Person })
|
||||
brain.find({ where: { noun: NounType.Person } })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Standard Metadata Fields
|
||||
|
||||
When you add an entity, Brainy stores these standard fields in the metadata object alongside your custom fields:
|
||||
|
||||
| Field | Set By | Description |
|
||||
|-------|--------|-------------|
|
||||
| `noun` | System | Entity type (NounType enum value) |
|
||||
| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. |
|
||||
| `data` | System | The raw `data` value (stored opaquely) |
|
||||
| `createdAt` | System | Creation timestamp |
|
||||
| `updatedAt` | System | Last update timestamp |
|
||||
| `confidence` | User | Type classification confidence |
|
||||
| `weight` | User | Entity importance |
|
||||
| `service` | User | Multi-tenancy identifier |
|
||||
| `createdBy` | User/System | Source augmentation |
|
||||
|
||||
On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**.
|
||||
|
||||
### Subtype — sub-classification within a NounType
|
||||
|
||||
`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type:
|
||||
|
||||
```typescript
|
||||
// A Person who is an employee:
|
||||
await brain.add({
|
||||
data: 'Avery Brooks — runs the AI lab',
|
||||
type: NounType.Person,
|
||||
subtype: 'employee',
|
||||
metadata: { department: 'ai-lab' }
|
||||
})
|
||||
|
||||
// A Document that is an invoice:
|
||||
await brain.add({
|
||||
data: 'INV-2026-001',
|
||||
type: NounType.Document,
|
||||
subtype: 'invoice',
|
||||
metadata: { amount: 1500 }
|
||||
})
|
||||
```
|
||||
|
||||
`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`.
|
||||
|
||||
### Subtype — sub-classification within a VerbType (7.30+)
|
||||
|
||||
Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive — a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field on `HNSWVerbWithMetadata` and on the public `Relation<T>`:
|
||||
|
||||
```typescript
|
||||
await brain.relate({
|
||||
from: ceoId,
|
||||
to: vpId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'direct', // top-level standard field
|
||||
metadata: { since: '2025-Q1' } // user-custom fields stay in metadata
|
||||
})
|
||||
```
|
||||
|
||||
Fast-path filter on the verb side:
|
||||
|
||||
```typescript
|
||||
const direct = await brain.related({
|
||||
from: ceoId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'direct'
|
||||
})
|
||||
```
|
||||
|
||||
The verb-side rollup at `_system/verb-subtype-statistics.json` mirrors the noun-side `_system/subtype-statistics.json` — same shape, same self-heal machinery. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`.
|
||||
|
||||
Verbs and nouns now have full capability parity — every API on the noun side has a verb-side mirror, including the new `brain.updateRelation()` (which closed a pre-7.30 gap where relationships had no update path).
|
||||
|
||||
### Standard verb fields
|
||||
|
||||
The verb-side equivalent of `STANDARD_ENTITY_FIELDS` is `STANDARD_VERB_FIELDS`, exported from `src/coreTypes.ts`. Verb-specific standard fields:
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| `verb` | The VerbType enum value |
|
||||
| `sourceId` / `targetId` | The two endpoints of the relationship |
|
||||
| `subtype` | Sub-classification within the VerbType (7.30+) |
|
||||
| `confidence`, `weight`, `createdAt`, `updatedAt`, `service`, `createdBy`, `data` | Same semantics as the noun-side standard fields |
|
||||
|
||||
The companion `resolveVerbField(verb, field)` helper resolves field paths the same way `resolveEntityField` does for nouns: standard fields first, metadata fallback for everything else.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [API Reference](./api/README.md) — Complete API documentation
|
||||
- [Query Operators](./QUERY_OPERATORS.md) — All BFO operators with examples
|
||||
- [Find System](./FIND_SYSTEM.md) — Natural language find() details
|
||||
|
|
@ -193,21 +193,21 @@ console.log('✅ Created relationships')
|
|||
console.log('\n🔍 Querying relationships...')
|
||||
|
||||
// Who works for TechCorp?
|
||||
const techcorpEmployees = await brain.related({
|
||||
const techcorpEmployees = await brain.getRelations({
|
||||
to: techcorpId,
|
||||
type: VerbType.WorksWith
|
||||
})
|
||||
console.log(`TechCorp has ${techcorpEmployees.length} employees`)
|
||||
|
||||
// Who works on the AI project?
|
||||
const projectContributors = await brain.related({
|
||||
const projectContributors = await brain.getRelations({
|
||||
to: projectId,
|
||||
type: [VerbType.WorksOn, VerbType.Manages]
|
||||
})
|
||||
console.log(`AI Project has ${projectContributors.length} contributors`)
|
||||
|
||||
// Who does John collaborate with?
|
||||
const johnsCollaborators = await brain.related({
|
||||
const johnsCollaborators = await brain.getRelations({
|
||||
from: johnId,
|
||||
type: VerbType.CollaboratesWith
|
||||
})
|
||||
|
|
@ -294,7 +294,7 @@ await brain.update({
|
|||
1. Create an organizational hierarchy (CEO → Managers → Engineers)
|
||||
2. Build a project dependency graph
|
||||
3. Model a social network with CollaboratesWith relationships
|
||||
4. Query "Who reports to Alice?" using related()
|
||||
4. Query "Who reports to Alice?" using getRelations()
|
||||
5. Batch update all projects to add a "year: 2024" field
|
||||
|
||||
### Next Steps
|
||||
|
|
@ -429,7 +429,23 @@ fusionResults.forEach(r => {
|
|||
}
|
||||
})
|
||||
|
||||
// 5. SIMILARITY: Find similar documents
|
||||
// 5. NEURAL API: Automatic clustering
|
||||
console.log('\n\n🤖 NEURAL API: Automatic Clustering')
|
||||
const neural = brain.neural()
|
||||
|
||||
const clusters = await neural.clusters({
|
||||
maxClusters: 3,
|
||||
minClusterSize: 1
|
||||
})
|
||||
|
||||
console.log(`Found ${clusters.length} semantic clusters:`)
|
||||
clusters.forEach((cluster, i) => {
|
||||
console.log(`\n Cluster ${i + 1}: ${cluster.label || cluster.id}`)
|
||||
console.log(` Members: ${cluster.members.length}`)
|
||||
console.log(` Centroid topics: ${cluster.metadata?.topics?.join(', ') || 'N/A'}`)
|
||||
})
|
||||
|
||||
// 6. SIMILARITY: Find similar documents
|
||||
console.log('\n\n🔍 SIMILARITY: Find Similar Documents')
|
||||
const similarTo = await brain.similar({
|
||||
to: paper1, // Entity ID of first AI paper
|
||||
|
|
@ -443,6 +459,19 @@ similarTo.forEach(r => {
|
|||
console.log(` [${r.score.toFixed(3)}] ${r.entity.data?.substring(0, 50)}...`)
|
||||
})
|
||||
|
||||
// 7. OUTLIER DETECTION
|
||||
console.log('\n\n🚨 OUTLIER DETECTION')
|
||||
const outliers = await neural.outliers({
|
||||
method: 'statistical',
|
||||
threshold: 2.0 // 2 standard deviations
|
||||
})
|
||||
|
||||
console.log(`Found ${outliers.length} outlier documents:`)
|
||||
outliers.forEach(o => {
|
||||
const entity = await brain.get(o.id)
|
||||
console.log(` [Anomaly score: ${o.score.toFixed(3)}] ${entity?.data?.substring(0, 50)}...`)
|
||||
})
|
||||
|
||||
await brain.close()
|
||||
```
|
||||
|
||||
|
|
@ -647,7 +676,7 @@ if (readmeEntity.length > 0) {
|
|||
}
|
||||
|
||||
// Query relationships
|
||||
const conceptDocs = await brain.related({
|
||||
const conceptDocs = await brain.getRelations({
|
||||
from: conceptId,
|
||||
type: VerbType.DocumentedBy
|
||||
})
|
||||
|
|
@ -769,7 +798,7 @@ await brain.relate({
|
|||
})
|
||||
|
||||
// Query across boundaries
|
||||
const conceptDocs = await brain.related({
|
||||
const conceptDocs = await brain.getRelations({
|
||||
from: conceptId,
|
||||
type: VerbType.DocumentedBy
|
||||
})
|
||||
|
|
@ -823,7 +852,7 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
|
|||
## Level 5: Production Scale (90 minutes)
|
||||
|
||||
### What You'll Learn
|
||||
- Production filesystem storage and off-site backup
|
||||
- Cloud storage (GCS, S3, R2)
|
||||
- Performance optimization
|
||||
- Batch imports (CSV, Excel, PDF)
|
||||
- Metadata query optimization
|
||||
|
|
@ -834,13 +863,18 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
|
|||
```typescript
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
// 1. PRODUCTION STORAGE - Filesystem with off-site snapshots
|
||||
console.log('Initializing production storage...\n')
|
||||
// 1. PRODUCTION STORAGE - Google Cloud Storage (Native SDK)
|
||||
console.log('☁️ Initializing production storage...\n')
|
||||
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: '/var/lib/brainy'
|
||||
type: 'gcs-native', // Native GCS SDK (recommended)
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-brainy-production',
|
||||
// ADC (Application Default Credentials) - zero config in Cloud Run/GCE!
|
||||
// Or provide credentials:
|
||||
// keyFilename: '/path/to/service-account.json'
|
||||
}
|
||||
},
|
||||
|
||||
// Performance tuning
|
||||
|
|
@ -854,8 +888,7 @@ const brain = new Brainy({
|
|||
})
|
||||
|
||||
await brain.init()
|
||||
console.log('Brainy initialized with filesystem storage')
|
||||
console.log('Snapshot /var/lib/brainy off-site via cron with `gsutil rsync` / `aws s3 sync` / `rclone`.\n')
|
||||
console.log('✅ Brainy initialized with GCS Native storage\n')
|
||||
|
||||
// 2. BATCH IMPORT - CSV File
|
||||
console.log('📊 Importing CSV data...\n')
|
||||
|
|
@ -934,6 +967,26 @@ console.log(` Failed: ${batchResult.failed.length}`)
|
|||
console.log(` Duration: ${duration}ms`)
|
||||
console.log(` Throughput: ${(batchResult.successful.length / (duration / 1000)).toFixed(0)} entities/sec`)
|
||||
|
||||
// 5. ADVANCED CLUSTERING (Large Dataset)
|
||||
console.log('\n\n🤖 Clustering 1000 entities...\n')
|
||||
|
||||
const neural = brain.neural()
|
||||
|
||||
// Use fast clustering for large datasets
|
||||
const clusters = await neural.clusterFast({
|
||||
maxClusters: 5
|
||||
})
|
||||
|
||||
console.log(`Found ${clusters.length} clusters:`)
|
||||
clusters.forEach((cluster, i) => {
|
||||
console.log(`\n Cluster ${i + 1}: ${cluster.label || cluster.id}`)
|
||||
console.log(` Size: ${cluster.members.length} members`)
|
||||
console.log(` Density: ${(cluster.density || 0).toFixed(3)}`)
|
||||
if (cluster.metadata?.keywords) {
|
||||
console.log(` Keywords: ${cluster.metadata.keywords.slice(0, 5).join(', ')}`)
|
||||
}
|
||||
})
|
||||
|
||||
// 6. PRODUCTION STATISTICS
|
||||
console.log('\n\n📊 Production Statistics:\n')
|
||||
|
||||
|
|
@ -991,7 +1044,7 @@ console.log('✅ Brain closed cleanly')
|
|||
|
||||
console.log('\n\n🎓 Production Deployment Complete!')
|
||||
console.log('\n📚 Key Production Learnings:')
|
||||
console.log(' 1. Use filesystem storage and snapshot the data directory off-site from your scheduler')
|
||||
console.log(' 1. Use native cloud storage (GCS, S3, R2) for persistence')
|
||||
console.log(' 2. Batch operations = 100x faster than individual ops')
|
||||
console.log(' 3. Metadata query optimization for complex filters')
|
||||
console.log(' 4. Monitor query performance with explain: true')
|
||||
|
|
@ -1004,22 +1057,41 @@ console.log(' 7. Stream large imports with progress callbacks')
|
|||
|
||||
#### 1. **Storage Options Comparison**
|
||||
|
||||
| Storage | Use Case | Performance | Setup |
|
||||
|---------|----------|-------------|-------|
|
||||
| Memory | Dev/testing | Fastest | Zero config |
|
||||
| Filesystem | Production | Fast | Local path + scheduled off-site backup |
|
||||
| Storage | Use Case | Performance | Cost | Setup |
|
||||
|---------|----------|-------------|------|-------|
|
||||
| Memory | Dev/testing | Fastest | Free | Zero config |
|
||||
| Filesystem | Local prod | Fast | Free | Local path |
|
||||
| GCS Native | GCP prod | Fast | $$$ | Service account |
|
||||
| S3 | AWS prod | Fast | $$$ | Access keys |
|
||||
| R2 | Cloudflare | Fast | $ | Access keys |
|
||||
|
||||
#### 2. **Off-Site Backup**
|
||||
#### 2. **GCS Native vs S3-Compatible**
|
||||
|
||||
```bash
|
||||
# Cron / systemd timer / k8s CronJob — pick whatever you already operate
|
||||
*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
|
||||
*/15 * * * * aws s3 sync /var/lib/brainy s3://my-bucket/brainy-backup
|
||||
*/15 * * * * gsutil rsync -r /var/lib/brainy gs://my-bucket/brainy-backup
|
||||
```typescript
|
||||
// ✅ RECOMMENDED: GCS Native SDK
|
||||
{
|
||||
storage: {
|
||||
type: 'gcs-native',
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-bucket'
|
||||
// ADC handles auth automatically in Cloud Run/GCE!
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ⚠️ LEGACY: S3-compatible mode
|
||||
{
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsStorage: {
|
||||
bucketName: 'my-bucket',
|
||||
accessKeyId: process.env.GCS_ACCESS_KEY,
|
||||
secretAccessKey: process.env.GCS_SECRET_KEY
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Brainy itself never reaches out to an object store. Snapshot `path` from your scheduler.
|
||||
|
||||
#### 3. **Performance Optimization**
|
||||
|
||||
```typescript
|
||||
|
|
@ -1092,8 +1164,8 @@ const brain = new Brainy({ verbose: true })
|
|||
|
||||
#### Before Deployment
|
||||
|
||||
- [ ] Provision a writable `path` on the host
|
||||
- [ ] Set up an off-site snapshot job (`rclone` / `aws s3 sync` / `gsutil rsync`)
|
||||
- [ ] Choose cloud storage (GCS/S3/R2)
|
||||
- [ ] Set up authentication (service account/access keys)
|
||||
- [ ] Configure caching
|
||||
- [ ] Test batch operations
|
||||
- [ ] Benchmark query performance
|
||||
|
|
@ -1117,12 +1189,12 @@ const brain = new Brainy({ verbose: true })
|
|||
|
||||
### Practice Exercises
|
||||
|
||||
1. Deploy Brainy with filesystem storage + scheduled off-site backup
|
||||
1. Deploy Brainy with GCS Native storage
|
||||
2. Import a 10,000 row CSV file
|
||||
3. Measure query performance for different filters
|
||||
4. Optimize a slow query using getOptimalQueryPlan()
|
||||
5. Set up monitoring dashboard
|
||||
6. Test restore from off-site snapshot
|
||||
6. Create backup/restore scripts
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -1140,7 +1212,7 @@ const brain = new Brainy({ verbose: true })
|
|||
|
||||
#### Advanced Topics
|
||||
|
||||
- **Multi-instance Deployments**: Run multiple Brainy processes behind your own routing layer
|
||||
- **Distributed Systems**: Sharding, replication, coordination
|
||||
- **Custom Augmentations**: Extend Brainy with plugins
|
||||
- **Streaming Pipelines**: Real-time data ingestion
|
||||
- **Security**: Encryption, access control, audit logs
|
||||
|
|
@ -1148,9 +1220,10 @@ const brain = new Brainy({ verbose: true })
|
|||
|
||||
#### Resources
|
||||
|
||||
- 📚 [API Reference](../api/README.md) - Complete API documentation
|
||||
- 📚 [API Reference](../API_REFERENCE.md) - Complete API documentation
|
||||
- 📁 [VFS Guide](../vfs/VFS_API_GUIDE.md) - Virtual Filesystem deep dive
|
||||
- 🤖 [Neural API](../guides/neural-api.md) - Advanced neural operations
|
||||
- 🌐 [Distributed Guide](../guides/distributed-system.md) - Planet-scale architecture
|
||||
- 💬 [Discord Community](https://discord.gg/brainy) - Get help, share projects
|
||||
|
||||
#### Share Your Success
|
||||
|
|
|
|||
396
docs/EXTENDING_STORAGE.md
Normal file
396
docs/EXTENDING_STORAGE.md
Normal file
|
|
@ -0,0 +1,396 @@
|
|||
# 🔌 Extending Brainy Storage with Augmentations
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy's zero-config system is **fully extensible**. Augmentations can register new storage providers, presets, and auto-detection logic that integrates seamlessly with the existing system.
|
||||
|
||||
## How Storage Extensions Work
|
||||
|
||||
### 1. Storage Provider Registration
|
||||
|
||||
When an augmentation is installed, it can register a new storage provider:
|
||||
|
||||
```typescript
|
||||
import { StorageProvider, registerStorageAugmentation } from '@soulcraft/brainy/config'
|
||||
|
||||
const redisProvider: StorageProvider = {
|
||||
type: 'redis',
|
||||
name: 'Redis Storage',
|
||||
description: 'High-performance in-memory data store',
|
||||
priority: 10, // Higher priority = checked first in auto-detection
|
||||
|
||||
// Auto-detection logic
|
||||
async detect(): Promise<boolean> {
|
||||
// Check if Redis is available
|
||||
if (process.env.REDIS_URL) {
|
||||
try {
|
||||
const redis = await import('ioredis')
|
||||
const client = new redis.default(process.env.REDIS_URL)
|
||||
await client.ping()
|
||||
await client.quit()
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return false
|
||||
},
|
||||
|
||||
// Configuration builder
|
||||
async getConfig(): Promise<any> {
|
||||
return {
|
||||
type: 'redis',
|
||||
redisStorage: {
|
||||
url: process.env.REDIS_URL,
|
||||
prefix: 'brainy:',
|
||||
ttl: 3600
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Register the provider
|
||||
registerStorageAugmentation(redisProvider)
|
||||
```
|
||||
|
||||
### 2. Using Extended Storage
|
||||
|
||||
Once registered, the new storage type works with zero-config:
|
||||
|
||||
```typescript
|
||||
// Auto-detection will now check Redis
|
||||
const brain = new Brainy() // Will use Redis if available!
|
||||
|
||||
// Or explicitly specify
|
||||
const brain = new Brainy({ storage: 'redis' })
|
||||
|
||||
// Or with custom config
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'redis',
|
||||
redisStorage: {
|
||||
url: 'redis://localhost:6379',
|
||||
prefix: 'myapp:'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
### Redis Augmentation
|
||||
|
||||
```typescript
|
||||
// @soulcraft/brainy-redis package
|
||||
export class RedisStorageAugmentation {
|
||||
async init() {
|
||||
// Register the storage provider
|
||||
registerStorageAugmentation({
|
||||
type: 'redis',
|
||||
name: 'Redis Storage',
|
||||
priority: 10,
|
||||
|
||||
async detect() {
|
||||
return !!(process.env.REDIS_URL || process.env.REDIS_HOST)
|
||||
},
|
||||
|
||||
async getConfig() {
|
||||
return {
|
||||
type: 'redis',
|
||||
redisStorage: {
|
||||
url: process.env.REDIS_URL ||
|
||||
`redis://${process.env.REDIS_HOST}:${process.env.REDIS_PORT || 6379}`
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Register Redis-specific presets
|
||||
registerPresetAugmentation('redis-cache', {
|
||||
storage: 'redis',
|
||||
model: ModelPrecision.Q8,
|
||||
features: ['core', 'cache'],
|
||||
distributed: true,
|
||||
description: 'Redis-backed cache layer',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### MongoDB Augmentation
|
||||
|
||||
```typescript
|
||||
// @soulcraft/brainy-mongodb package
|
||||
export class MongoStorageAugmentation {
|
||||
async init() {
|
||||
registerStorageAugmentation({
|
||||
type: 'mongodb',
|
||||
name: 'MongoDB Storage',
|
||||
priority: 8,
|
||||
|
||||
async detect() {
|
||||
return !!(process.env.MONGODB_URI || process.env.MONGO_URL)
|
||||
},
|
||||
|
||||
async getConfig() {
|
||||
return {
|
||||
type: 'mongodb',
|
||||
mongoStorage: {
|
||||
uri: process.env.MONGODB_URI,
|
||||
database: 'brainy',
|
||||
collection: 'vectors'
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### PostgreSQL + pgvector Augmentation
|
||||
|
||||
```typescript
|
||||
// @soulcraft/brainy-postgres package
|
||||
export class PostgresStorageAugmentation {
|
||||
async init() {
|
||||
registerStorageAugmentation({
|
||||
type: 'postgres',
|
||||
name: 'PostgreSQL + pgvector',
|
||||
priority: 9,
|
||||
|
||||
async detect() {
|
||||
const url = process.env.DATABASE_URL
|
||||
if (url?.includes('postgres')) {
|
||||
// Check for pgvector extension
|
||||
const client = new Client({ connectionString: url })
|
||||
await client.connect()
|
||||
const result = await client.query(
|
||||
"SELECT * FROM pg_extension WHERE extname = 'vector'"
|
||||
)
|
||||
await client.end()
|
||||
return result.rows.length > 0
|
||||
}
|
||||
return false
|
||||
},
|
||||
|
||||
async getConfig() {
|
||||
return {
|
||||
type: 'postgres',
|
||||
postgresStorage: {
|
||||
connectionString: process.env.DATABASE_URL,
|
||||
table: 'brainy_vectors'
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Auto-Detection Priority
|
||||
|
||||
Storage providers are checked in priority order:
|
||||
|
||||
1. **Custom providers** (highest priority first)
|
||||
2. **Cloud storage** (S3, GCS, R2)
|
||||
3. **Database storage** (Redis, MongoDB, PostgreSQL)
|
||||
4. **Local storage** (filesystem, OPFS)
|
||||
5. **Memory** (fallback)
|
||||
|
||||
```typescript
|
||||
// Example priority chain
|
||||
Redis (priority: 10) → PostgreSQL (9) → MongoDB (8) → S3 (5) → Filesystem (1) → Memory (0)
|
||||
```
|
||||
|
||||
## Creating Custom Presets
|
||||
|
||||
Augmentations can also register new presets:
|
||||
|
||||
```typescript
|
||||
registerPresetAugmentation('redis-cluster', {
|
||||
storage: 'redis',
|
||||
model: ModelPrecision.Q8,
|
||||
features: ['core', 'cache', 'cluster'],
|
||||
distributed: true,
|
||||
role: DistributedRole.HYBRID,
|
||||
cache: {
|
||||
hotCacheMaxSize: 100000, // Large distributed cache
|
||||
autoTune: true
|
||||
},
|
||||
description: 'Redis Cluster configuration',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
|
||||
// Users can then use:
|
||||
const brain = new Brainy('redis-cluster')
|
||||
```
|
||||
|
||||
## Type Safety with Extensions
|
||||
|
||||
To maintain type safety with dynamic storage types:
|
||||
|
||||
```typescript
|
||||
// Augmentation declares its types
|
||||
declare module '@soulcraft/brainy' {
|
||||
interface StorageTypes {
|
||||
redis: {
|
||||
url: string
|
||||
prefix?: string
|
||||
ttl?: number
|
||||
}
|
||||
}
|
||||
|
||||
interface PresetNames {
|
||||
'redis-cache': 'redis-cache'
|
||||
'redis-cluster': 'redis-cluster'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices for Storage Augmentations
|
||||
|
||||
1. **Always provide auto-detection** - Check environment variables and connectivity
|
||||
2. **Set appropriate priority** - Higher for specialized storage, lower for general
|
||||
3. **Handle failures gracefully** - Return false from detect() if not available
|
||||
4. **Document requirements** - List required packages and environment variables
|
||||
5. **Provide presets** - Include common configuration patterns
|
||||
6. **Maintain compatibility** - Ensure model precision matches across instances
|
||||
|
||||
## Example: Complete Redis Augmentation
|
||||
|
||||
```typescript
|
||||
import {
|
||||
StorageProvider,
|
||||
registerStorageAugmentation,
|
||||
registerPresetAugmentation,
|
||||
PresetCategory,
|
||||
ModelPrecision,
|
||||
DistributedRole
|
||||
} from '@soulcraft/brainy/config'
|
||||
import Redis from 'ioredis'
|
||||
|
||||
export class BrainyRedisAugmentation {
|
||||
private client: Redis
|
||||
|
||||
async init() {
|
||||
// Register storage provider
|
||||
registerStorageAugmentation({
|
||||
type: 'redis',
|
||||
name: 'Redis Vector Storage',
|
||||
description: 'Redis with RediSearch for vector similarity',
|
||||
priority: 10,
|
||||
|
||||
requirements: {
|
||||
env: ['REDIS_URL'],
|
||||
packages: ['ioredis', 'redis']
|
||||
},
|
||||
|
||||
async detect() {
|
||||
if (!process.env.REDIS_URL) return false
|
||||
|
||||
try {
|
||||
const client = new Redis(process.env.REDIS_URL)
|
||||
|
||||
// Check for RediSearch module
|
||||
const modules = await client.call('MODULE', 'LIST')
|
||||
const hasRediSearch = modules.some(m => m[1] === 'search')
|
||||
|
||||
await client.quit()
|
||||
return hasRediSearch
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
},
|
||||
|
||||
async getConfig() {
|
||||
return {
|
||||
type: 'redis',
|
||||
redisStorage: {
|
||||
url: process.env.REDIS_URL,
|
||||
prefix: process.env.REDIS_PREFIX || 'brainy:',
|
||||
index: process.env.REDIS_INDEX || 'brainy-vectors',
|
||||
ttl: process.env.REDIS_TTL ? parseInt(process.env.REDIS_TTL) : undefined
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Register presets
|
||||
this.registerPresets()
|
||||
}
|
||||
|
||||
private registerPresets() {
|
||||
// Fast cache preset
|
||||
registerPresetAugmentation('redis-fast-cache', {
|
||||
storage: 'redis' as any,
|
||||
model: ModelPrecision.Q8,
|
||||
features: ['core', 'cache', 'search'],
|
||||
distributed: false,
|
||||
cache: {
|
||||
hotCacheMaxSize: 10000,
|
||||
autoTune: true
|
||||
},
|
||||
description: 'Redis-backed fast cache',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
|
||||
// Distributed cache preset
|
||||
registerPresetAugmentation('redis-distributed', {
|
||||
storage: 'redis' as any,
|
||||
model: ModelPrecision.AUTO,
|
||||
features: ['core', 'cache', 'search', 'cluster'],
|
||||
distributed: true,
|
||||
role: DistributedRole.HYBRID,
|
||||
cache: {
|
||||
hotCacheMaxSize: 50000,
|
||||
autoTune: true
|
||||
},
|
||||
description: 'Redis distributed cache cluster',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
|
||||
// Session store preset
|
||||
registerPresetAugmentation('redis-sessions', {
|
||||
storage: 'redis' as any,
|
||||
model: ModelPrecision.Q8,
|
||||
features: ['core', 'cache'],
|
||||
distributed: false,
|
||||
cache: {
|
||||
hotCacheMaxSize: 5000,
|
||||
autoTune: false
|
||||
},
|
||||
description: 'Redis session storage',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Usage after installing the augmentation:
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import '@soulcraft/brainy-redis' // Registers the augmentation
|
||||
|
||||
// Now Redis is automatically detected!
|
||||
const brain = new Brainy() // Uses Redis if REDIS_URL is set
|
||||
|
||||
// Or use a Redis preset
|
||||
const brain = new Brainy('redis-fast-cache')
|
||||
|
||||
// Or explicitly configure
|
||||
const brain = new Brainy({
|
||||
storage: 'redis',
|
||||
model: ModelPrecision.FP32
|
||||
})
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
The extensible configuration system allows:
|
||||
|
||||
1. **New storage types** via `registerStorageAugmentation()`
|
||||
2. **Custom presets** via `registerPresetAugmentation()`
|
||||
3. **Auto-detection logic** that integrates with zero-config
|
||||
4. **Type-safe extensions** with TypeScript declarations
|
||||
5. **Priority-based selection** for intelligent defaults
|
||||
|
||||
This ensures Brainy can grow with new storage technologies while maintaining its zero-configuration philosophy!
|
||||
|
|
@ -1,16 +1,3 @@
|
|||
---
|
||||
title: The Find System
|
||||
slug: guides/find-system
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 3
|
||||
description: Complete guide to Brainy's find() method — four intelligence systems, query execution phases, NLP patterns, and performance from 1K to 10M entities.
|
||||
next:
|
||||
- concepts/triple-intelligence
|
||||
- api/reference
|
||||
---
|
||||
|
||||
# Brainy's Find System - Complete Guide
|
||||
|
||||
## Overview
|
||||
|
|
@ -22,7 +9,7 @@ Brainy's `find()` method is the most advanced query system in any vector databas
|
|||
### 1. Vector Intelligence (HNSW Index)
|
||||
- **Purpose**: Semantic similarity search using embeddings
|
||||
- **Algorithm**: Hierarchical Navigable Small World (HNSW)
|
||||
- **Performance**: O(log n) search
|
||||
- **Performance**: O(log n) search, ~1.8ms typical
|
||||
- **Data Structure**: Multi-layer graph with 16 connections per node
|
||||
- **Use Cases**: "Find similar documents", "Content like this"
|
||||
|
||||
|
|
@ -36,14 +23,14 @@ Brainy's `find()` method is the most advanced query system in any vector databas
|
|||
### 3. Metadata Intelligence (Incremental Indices)
|
||||
- **Purpose**: Fast filtering on structured data
|
||||
- **Algorithm**: HashMap for exact matches, Sorted arrays for ranges
|
||||
- **Performance**: O(1) exact, O(log n) ranges
|
||||
- **Performance**: O(1) exact, O(log n) ranges, <1ms typical
|
||||
- **Data Structure**: `Map<field:value, Set<id>>` + sorted value arrays
|
||||
- **Use Cases**: "Documents from 2023", "Status equals active"
|
||||
|
||||
### 4. Graph Intelligence (Adjacency Maps)
|
||||
- **Purpose**: Relationship traversal and connection analysis
|
||||
- **Algorithm**: Pure O(1) neighbor lookups via Map operations
|
||||
- **Performance**: O(1) per hop (measured <1 ms per neighbor lookup, validated up to 1M relationships — `tests/performance/graph-scale-performance.test.ts:238`)
|
||||
- **Performance**: O(1) per hop, ~0.1ms typical
|
||||
- **Data Structure**: `Map<sourceId, Set<targetId>>`
|
||||
- **Use Cases**: "Papers connected to MIT", "Authors who collaborated"
|
||||
|
||||
|
|
@ -373,40 +360,36 @@ return results.slice(offset, offset + limit)
|
|||
|
||||
### Query Performance by Type
|
||||
|
||||
| Query Type | Index Used | Complexity | Example |
|
||||
|------------|------------|------------|---------|
|
||||
| **Semantic Search** | HNSW Vector | O(log n) | `"AI research papers"` |
|
||||
| **Exact Metadata** | HashMap | O(1) | `{status: "published"}` |
|
||||
| **Range Metadata** | Sorted Array | O(log n) | `{year: {greaterThan: 2020}}` |
|
||||
| **Graph Traversal** | Adjacency Map | O(1) | `{connected: {to: "mit"}}` |
|
||||
| **Type Detection** | Pre-embedded Types | O(t) | `"documents"` → `NounType.Document` |
|
||||
| **Field Matching** | Field Embeddings | O(f) | `"by"` → `"author"` |
|
||||
| **Combined Query** | All Indices | O(log n) | NLP + filters + graph |
|
||||
| Query Type | Index Used | Performance | Example |
|
||||
|------------|------------|-------------|---------|
|
||||
| **Semantic Search** | HNSW Vector | O(log n), ~1.8ms | `"AI research papers"` |
|
||||
| **Exact Metadata** | HashMap | O(1), ~0.8ms | `{status: "published"}` |
|
||||
| **Range Metadata** | Sorted Array | O(log n), ~0.6ms | `{year: {greaterThan: 2020}}` |
|
||||
| **Graph Traversal** | Adjacency Map | O(1), ~0.1ms | `{connected: {to: "mit"}}` |
|
||||
| **Type Detection** | Pre-embedded Types | O(t), ~0.3ms | `"documents"` → `NounType.Document` |
|
||||
| **Field Matching** | Field Embeddings | O(f), ~0.1ms | `"by"` → `"author"` |
|
||||
| **Combined Query** | All Indices | O(log n), ~1.8ms | NLP + filters + graph |
|
||||
|
||||
Where:
|
||||
- n = number of entities in database
|
||||
- t = number of types (169 total: 42 noun + 127 verb)
|
||||
- f = number of fields for detected entity type (typically 5-15)
|
||||
|
||||
Absolute latencies depend on hardware, embedding model, and storage backend; the columns above describe how each stage scales. The graph stage is the one path with a committed scale benchmark (see [Scalability](#scalability)).
|
||||
|
||||
### Scalability
|
||||
|
||||
The cost of each query stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion:
|
||||
|
||||
| Query stage | Complexity | Scaling behavior |
|
||||
|-------------|------------|------------------|
|
||||
| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
|
||||
| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
|
||||
| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
|
||||
| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
|
||||
| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
|
||||
| Database Size | Vector Search | Metadata Filter | Graph Query | Combined |
|
||||
|---------------|---------------|-----------------|-------------|----------|
|
||||
| **1K entities** | 0.8ms | 0.3ms | 0.05ms | 1.1ms |
|
||||
| **10K entities** | 1.2ms | 0.5ms | 0.08ms | 1.5ms |
|
||||
| **100K entities** | 1.8ms | 0.8ms | 0.1ms | 2.1ms |
|
||||
| **1M entities** | 2.5ms | 1.2ms | 0.1ms | 2.8ms |
|
||||
| **10M entities** | 3.8ms | 1.8ms | 0.1ms | 4.2ms |
|
||||
|
||||
**Key Performance Notes:**
|
||||
- Graph queries stay O(1) regardless of scale
|
||||
- Metadata ranges scale as O(log n), not O(n)
|
||||
- Vector search degrades gracefully due to HNSW
|
||||
- Type-aware NLP adds minimal overhead (single embedding pass, no full scan)
|
||||
- Type-aware NLP adds minimal overhead (~0.4ms)
|
||||
|
||||
## Example Query Flows
|
||||
|
||||
|
|
@ -459,7 +442,7 @@ await brain.find({
|
|||
},
|
||||
limit: 10
|
||||
})
|
||||
// Executes as O(1) type filter + O(1)/O(log n) metadata + O(1) graph lookup — no full scan
|
||||
// Total performance: ~1.2ms for 100K entities
|
||||
```
|
||||
|
||||
## Filter Syntax Reference
|
||||
|
|
@ -1005,7 +988,7 @@ const results = await brain.find({
|
|||
})
|
||||
```
|
||||
|
||||
**Performance**: O(log n) vector search + O(log n) metadata filters + O(1) graph traversal — bounded by the vector stage.
|
||||
**Performance**: O(log n) vector search + O(log n) metadata filters + O(1) graph traversal = ~2-3ms total.
|
||||
|
||||
### Excluding Soft-Deleted Entities
|
||||
|
||||
|
|
@ -1232,7 +1215,7 @@ where: { age: { gt: 18 } }
|
|||
**No connected entities found**:
|
||||
```typescript
|
||||
// Verify relationship exists
|
||||
const relations = await brain.related({
|
||||
const relations = await brain.getRelations({
|
||||
from: 'entity-a',
|
||||
to: 'entity-b'
|
||||
})
|
||||
|
|
@ -1356,42 +1339,6 @@ await brain.find({
|
|||
- No performance penalty - same speed as any metadata filter
|
||||
- Works seamlessly with vector + metadata + graph queries
|
||||
|
||||
## 5. Aggregate Queries
|
||||
|
||||
The `find()` method also supports aggregate queries via the `aggregate` parameter. When set, `find()` bypasses all vector/metadata/graph search paths and returns pre-computed aggregate results from the incremental aggregation engine.
|
||||
|
||||
```typescript
|
||||
// Define the aggregate (once — survives restarts)
|
||||
brain.defineAggregate({
|
||||
name: 'monthly_spending',
|
||||
source: { type: NounType.Event, where: { domain: 'financial' } },
|
||||
groupBy: ['category', { field: 'date', window: 'month' }],
|
||||
metrics: {
|
||||
total: { op: 'sum', field: 'amount' },
|
||||
count: { op: 'count' }
|
||||
}
|
||||
})
|
||||
|
||||
// Query it through find()
|
||||
const results = await brain.find({
|
||||
aggregate: 'monthly_spending',
|
||||
where: { category: 'food' }, // Filters aggregate groups (not raw entities)
|
||||
orderBy: 'total',
|
||||
order: 'desc',
|
||||
limit: 12
|
||||
})
|
||||
```
|
||||
|
||||
**Key characteristics:**
|
||||
- **O(1) read performance** — results come from running totals, no query-time computation
|
||||
- **Updated incrementally** — every `add()`, `update()`, `delete()` updates matching aggregates
|
||||
- **Same Result<T>[] format** — aggregate results are returned as `NounType.Measurement` entities
|
||||
- **Combinable with `where`/`orderBy`/`limit`/`offset`** — standard find() parameters apply to group filtering
|
||||
|
||||
See the **[API Reference → Aggregation Engine](./api/README.md#aggregation-engine)** for the full API.
|
||||
|
||||
---
|
||||
|
||||
### Migration from v4.6.x
|
||||
|
||||
**BREAKING CHANGE**: The `includeVFS` parameter has been removed:
|
||||
|
|
|
|||
177
docs/METADATA_CONTRACT_IMPLEMENTATION.md
Normal file
177
docs/METADATA_CONTRACT_IMPLEMENTATION.md
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
# Metadata Contract Implementation Plan
|
||||
|
||||
## New Required Interface
|
||||
|
||||
```typescript
|
||||
export interface BrainyAugmentation {
|
||||
// Identity
|
||||
name: string
|
||||
timing: 'before' | 'after' | 'around' | 'replace'
|
||||
operations: string[]
|
||||
priority: number
|
||||
|
||||
// REQUIRED metadata contract
|
||||
metadata: 'none' | 'readonly' | MetadataAccess
|
||||
|
||||
// Methods
|
||||
initialize(context: AugmentationContext): Promise<void>
|
||||
execute<T = any>(operation: string, params: any, next: () => Promise<T>): Promise<T>
|
||||
shouldExecute?(operation: string, params: any): boolean
|
||||
shutdown?(): Promise<void>
|
||||
}
|
||||
|
||||
interface MetadataAccess {
|
||||
reads?: string[] | '*' // Fields to read, or '*' for all
|
||||
writes?: string[] | '*' // Fields to write, or '*' for all
|
||||
namespace?: string // Optional: custom namespace like '_myAug'
|
||||
}
|
||||
```
|
||||
|
||||
## Augmentation Analysis & Classification
|
||||
|
||||
### Category 1: No Metadata Access ('none')
|
||||
These augmentations don't read or write metadata at all:
|
||||
|
||||
1. **CacheAugmentation** - Only caches search results
|
||||
2. **RequestDeduplicatorAugmentation** - Only deduplicates requests
|
||||
3. **ConnectionPoolAugmentation** - Only manages storage connections
|
||||
4. **StorageAugmentation** - Base storage layer, metadata handled by Brainy
|
||||
|
||||
### Category 2: Read-Only Access ('readonly')
|
||||
These augmentations read metadata but never modify it:
|
||||
|
||||
6. **IndexAugmentation** - Reads metadata to build indexes
|
||||
7. **MonitoringAugmentation** - Reads metadata for monitoring
|
||||
8. **MetricsAugmentation** - Reads metadata for metrics collection
|
||||
9. **BatchProcessingAugmentation** - Reads metadata to check for external IDs
|
||||
10. **EntityRegistryAugmentation** - Reads metadata to register entities
|
||||
11. **AutoRegisterEntitiesAugmentation** - Reads metadata for auto-registration
|
||||
12. **ConduitAugmentation** - Reads metadata to pass through operations
|
||||
|
||||
### Category 3: Metadata Writers (needs specific access)
|
||||
These augmentations modify metadata and need specific field declarations:
|
||||
|
||||
13. **SynapseAugmentation** - Writes to '_synapse' field
|
||||
```typescript
|
||||
metadata: {
|
||||
reads: '*',
|
||||
writes: ['_synapse', '_synapseTimestamp'],
|
||||
namespace: '_synapse' // Uses its own namespace
|
||||
}
|
||||
```
|
||||
|
||||
14. **IntelligentVerbScoringAugmentation** - Adds scoring to verbs
|
||||
```typescript
|
||||
metadata: {
|
||||
reads: ['type', 'verb', 'source', 'target'],
|
||||
writes: ['weight', 'confidence', 'intelligentScoring']
|
||||
}
|
||||
```
|
||||
|
||||
15. **ServerSearchAugmentation** - Might add server metadata
|
||||
```typescript
|
||||
metadata: {
|
||||
reads: '*',
|
||||
writes: ['_server', '_syncedAt']
|
||||
}
|
||||
```
|
||||
|
||||
16. **NeuralImportAugmentation** - Enriches imported data
|
||||
```typescript
|
||||
metadata: {
|
||||
reads: '*',
|
||||
writes: ['nounType', 'verbType', '_importedAt', '_enriched']
|
||||
}
|
||||
```
|
||||
|
||||
### Category 4: API/Server (needs analysis)
|
||||
17. **ApiServerAugmentation** - Likely read-only for serving data
|
||||
18. **StorageAugmentations** (plural) - Collection of storage implementations
|
||||
19. **ConduitAugmentations** (plural) - Collection of conduit types
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Update Base Interface
|
||||
1. Update `BrainyAugmentation` interface to require `metadata` field
|
||||
2. Update `BaseAugmentation` class to have abstract `metadata` property
|
||||
3. Add runtime enforcement in augmentation executor
|
||||
|
||||
### Phase 2: Update Each Augmentation
|
||||
For each augmentation, add the appropriate metadata declaration:
|
||||
|
||||
#### Example Updates:
|
||||
|
||||
**CacheAugmentation:**
|
||||
```typescript
|
||||
export class CacheAugmentation extends BaseAugmentation {
|
||||
readonly name = 'cache'
|
||||
readonly metadata = 'none' as const // ✅ No metadata access
|
||||
// ... rest unchanged
|
||||
}
|
||||
```
|
||||
|
||||
```typescript
|
||||
readonly metadata = 'readonly' as const // ✅ Only reads for logging
|
||||
// ... rest unchanged
|
||||
}
|
||||
```
|
||||
|
||||
**SynapseAugmentation:**
|
||||
```typescript
|
||||
export abstract class SynapseAugmentation extends BaseAugmentation {
|
||||
readonly name = 'synapse'
|
||||
readonly metadata = {
|
||||
reads: '*',
|
||||
writes: ['_synapse', '_synapseTimestamp'],
|
||||
namespace: '_synapse'
|
||||
} as const
|
||||
// ... rest unchanged
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 3: Runtime Enforcement
|
||||
Add a metadata access enforcer that:
|
||||
1. Wraps metadata objects based on declared access
|
||||
2. Throws errors if augmentation violates its contract
|
||||
3. Logs warnings in development mode
|
||||
|
||||
```typescript
|
||||
class MetadataEnforcer {
|
||||
enforce(augmentation: BrainyAugmentation, metadata: any): any {
|
||||
if (augmentation.metadata === 'none') {
|
||||
return null // No access at all
|
||||
}
|
||||
|
||||
if (augmentation.metadata === 'readonly') {
|
||||
return Object.freeze(deepClone(metadata)) // Read-only copy
|
||||
}
|
||||
|
||||
// For specific access, create proxy that validates
|
||||
return new Proxy(metadata, {
|
||||
set(target, prop, value) {
|
||||
const access = augmentation.metadata as MetadataAccess
|
||||
if (!access.writes?.includes(String(prop)) && access.writes !== '*') {
|
||||
throw new Error(`Augmentation '${augmentation.name}' cannot write to field '${String(prop)}'`)
|
||||
}
|
||||
target[prop] = value
|
||||
return true
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Benefits
|
||||
1. **Type Safety** - TypeScript enforces metadata declaration
|
||||
2. **Runtime Safety** - Violations caught immediately
|
||||
3. **Documentation** - Contract shows exactly what each augmentation does
|
||||
4. **Brain-cloud Ready** - Registry can validate augmentations
|
||||
5. **Developer Friendly** - Most use simple 'none' or 'readonly'
|
||||
|
||||
## Migration Checklist
|
||||
- [ ] Update BrainyAugmentation interface
|
||||
- [ ] Update BaseAugmentation class
|
||||
- [ ] Add MetadataEnforcer
|
||||
- [ ] Update all 19 augmentations with metadata declarations
|
||||
- [ ] Add tests for metadata enforcement
|
||||
- [ ] Update documentation
|
||||
|
|
@ -48,7 +48,7 @@ await storage.setLifecyclePolicy({
|
|||
```typescript
|
||||
// v3: Delete one at a time (slow, expensive)
|
||||
for (const id of idsToDelete) {
|
||||
await brain.remove(id) // 1000 API calls for 1000 entities
|
||||
await brain.delete(id) // 1000 API calls for 1000 entities
|
||||
}
|
||||
|
||||
// v4.0.0: Batch delete (fast, cheap)
|
||||
|
|
@ -426,6 +426,16 @@ npm install @soulcraft/brainy@^3.50.0
|
|||
7. Verify data integrity thoroughly
|
||||
8. Enable lifecycle policies gradually
|
||||
|
||||
### Scenario 4: Multi-Node Distributed System
|
||||
|
||||
**Recommended approach:**
|
||||
1. Perform blue-green deployment:
|
||||
- Keep v3 nodes running (blue)
|
||||
- Deploy v4 nodes (green)
|
||||
- Migrate data once
|
||||
- Switch traffic to v4 nodes
|
||||
- Decommission v3 nodes
|
||||
|
||||
## Cost Savings After Migration
|
||||
|
||||
### Enable All v4.0.0 Features
|
||||
|
|
|
|||
736
docs/NEURAL_API_PATTERNS.md
Normal file
736
docs/NEURAL_API_PATTERNS.md
Normal file
|
|
@ -0,0 +1,736 @@
|
|||
# 🧠 Neural API Patterns: AI-Powered Intelligence
|
||||
|
||||
> Learn the correct patterns for Brainy's Neural API. Avoid performance pitfalls and use AI features effectively.
|
||||
|
||||
## 🚨 Critical: Access Neural APIs Correctly
|
||||
|
||||
### ❌ **WRONG - Outdated Access Patterns**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Outdated documentation patterns
|
||||
import { Brainy } from '@soulcraft/brainy' // ❌ Wrong import
|
||||
const brain = new Brainy() // ❌ Old class name
|
||||
|
||||
// These may not work as expected:
|
||||
const neural = brain.neural // ❌ May be undefined
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Modern Neural Access**
|
||||
|
||||
```typescript
|
||||
// ✅ Use modern Brainy class
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// ✅ Neural API is available after initialization
|
||||
const clusters = await brain.neural.clusters()
|
||||
const similarity = await brain.neural.similar('item1', 'item2')
|
||||
```
|
||||
|
||||
## 🔍 Similarity Analysis Patterns
|
||||
|
||||
### ❌ **WRONG - Inefficient Similarity Checks**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - N² comparisons
|
||||
const items = await brain.find({ limit: 1000 })
|
||||
const similarities = []
|
||||
|
||||
for (const item1 of items) {
|
||||
for (const item2 of items) {
|
||||
if (item1.id !== item2.id) {
|
||||
const sim = await brain.neural.similar(item1.id, item2.id) // ❌ Millions of calls
|
||||
similarities.push({ from: item1.id, to: item2.id, score: sim })
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Efficient Similarity Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Find neighbors (much more efficient)
|
||||
const item = await brain.get('target-item-id')
|
||||
const neighbors = await brain.neural.neighbors(item.id, {
|
||||
limit: 10, // Top 10 most similar
|
||||
threshold: 0.7, // Minimum similarity
|
||||
includeScores: true // Include similarity scores
|
||||
})
|
||||
|
||||
console.log(`Found ${neighbors.length} similar items`)
|
||||
|
||||
// ✅ Pattern 2: Batch similarity for specific pairs
|
||||
const itemPairs = [
|
||||
['item1', 'item2'],
|
||||
['item1', 'item3'],
|
||||
['item2', 'item3']
|
||||
]
|
||||
|
||||
const similarities = await Promise.all(
|
||||
itemPairs.map(async ([a, b]) => ({
|
||||
from: a,
|
||||
to: b,
|
||||
score: await brain.neural.similar(a, b)
|
||||
}))
|
||||
)
|
||||
|
||||
// ✅ Pattern 3: Text-to-text similarity (no need for IDs)
|
||||
const textSimilarity = await brain.neural.similar(
|
||||
"Machine learning is fascinating",
|
||||
"AI and deep learning are interesting",
|
||||
{ detailed: true } // Get explanation of similarity
|
||||
)
|
||||
|
||||
console.log(`Similarity: ${textSimilarity.score}`)
|
||||
console.log(`Explanation: ${textSimilarity.explanation}`)
|
||||
|
||||
// ✅ Pattern 4: Vector-level similarity for optimization
|
||||
const vector1 = await brain.embed("First concept")
|
||||
const vector2 = await brain.embed("Second concept")
|
||||
const vectorSimilarity = await brain.neural.similar(vector1, vector2)
|
||||
```
|
||||
|
||||
## 🎯 Clustering Patterns
|
||||
|
||||
### ❌ **WRONG - Uncontrolled Clustering**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Clustering everything without limits
|
||||
const everything = await brain.find({ limit: 100000 }) // ❌ Too much data
|
||||
const clusters = await brain.neural.clusters() // ❌ May crash or timeout
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Smart Clustering Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Controlled clustering with limits
|
||||
const recentItems = await brain.find({
|
||||
where: {
|
||||
createdAt: { $gte: Date.now() - 30 * 24 * 60 * 60 * 1000 } // Last 30 days
|
||||
},
|
||||
limit: 1000 // Reasonable limit
|
||||
})
|
||||
|
||||
const clusters = await brain.neural.clusters(
|
||||
recentItems.map(item => item.id),
|
||||
{
|
||||
algorithm: 'kmeans', // Reliable algorithm
|
||||
maxClusters: 10, // Reasonable number
|
||||
threshold: 0.75, // High similarity required
|
||||
iterations: 50 // Convergence limit
|
||||
}
|
||||
)
|
||||
|
||||
// ✅ Pattern 2: Domain-specific clustering
|
||||
const techDocs = await brain.find({
|
||||
where: { category: 'technology', type: 'document' },
|
||||
limit: 500
|
||||
})
|
||||
|
||||
const techClusters = await brain.neural.clusterByDomain(
|
||||
'category', // Group by this field
|
||||
{
|
||||
items: techDocs.map(doc => doc.id),
|
||||
minClusterSize: 3, // Minimum items per cluster
|
||||
maxClusters: 8
|
||||
}
|
||||
)
|
||||
|
||||
// ✅ Pattern 3: Temporal clustering for time-series data
|
||||
const timebasedClusters = await brain.neural.clusterByTime(
|
||||
'createdAt', // Time field
|
||||
'week', // Time window (hour, day, week, month)
|
||||
{
|
||||
items: recentItems.map(item => item.id),
|
||||
overlap: 0.2, // 20% overlap between windows
|
||||
minPerWindow: 5 // Minimum items per time window
|
||||
}
|
||||
)
|
||||
|
||||
// ✅ Pattern 4: Streaming clustering for large datasets
|
||||
async function clusterLargeDataset() {
|
||||
const clusterStream = brain.neural.clusterStream({
|
||||
batchSize: 100, // Process 100 items at a time
|
||||
updateInterval: 1000, // Update clusters every 1000 items
|
||||
maxMemory: 512 * 1024 * 1024 // 512MB memory limit
|
||||
})
|
||||
|
||||
const allClusters = []
|
||||
for await (const batch of clusterStream) {
|
||||
console.log(`Processed ${batch.processed} items, found ${batch.clusters.length} clusters`)
|
||||
allClusters.push(...batch.clusters)
|
||||
}
|
||||
|
||||
return allClusters
|
||||
}
|
||||
```
|
||||
|
||||
## 🔍 Neighbor Discovery Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Similarity Searches**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Reinventing neighbor search
|
||||
async function findSimilarManually(targetId: string) {
|
||||
const allItems = await brain.find({ limit: 10000 }) // ❌ Load everything
|
||||
const similarities = []
|
||||
|
||||
for (const item of allItems) {
|
||||
if (item.id !== targetId) {
|
||||
const score = await brain.neural.similar(targetId, item.id) // ❌ Slow
|
||||
if (score > 0.7) {
|
||||
similarities.push({ id: item.id, score })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return similarities.sort((a, b) => b.score - a.score).slice(0, 10) // ❌ Inefficient
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Optimized Neighbor Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Basic neighbor search
|
||||
const neighbors = await brain.neural.neighbors('target-item-id', {
|
||||
limit: 20, // Top 20 neighbors
|
||||
threshold: 0.6, // Minimum similarity
|
||||
includeMetadata: true, // Include item metadata
|
||||
includeDistances: true // Include exact similarity scores
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Filtered neighbor search
|
||||
const filteredNeighbors = await brain.neural.neighbors('article-id', {
|
||||
limit: 10,
|
||||
filter: {
|
||||
type: 'document', // Only find similar documents
|
||||
status: 'published', // Only published content
|
||||
language: 'en' // Only English content
|
||||
},
|
||||
excludeIds: ['self-id', 'duplicate-id'] // Exclude specific items
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Multi-level neighbor discovery
|
||||
async function discoverNeighborNetwork(rootId: string, maxDepth = 2) {
|
||||
const network = new Map()
|
||||
const visited = new Set()
|
||||
const queue = [{ id: rootId, depth: 0 }]
|
||||
|
||||
while (queue.length > 0) {
|
||||
const { id, depth } = queue.shift()!
|
||||
|
||||
if (visited.has(id) || depth >= maxDepth) continue
|
||||
visited.add(id)
|
||||
|
||||
const neighbors = await brain.neural.neighbors(id, {
|
||||
limit: 5,
|
||||
threshold: 0.8
|
||||
})
|
||||
|
||||
network.set(id, neighbors)
|
||||
|
||||
// Add neighbors to queue for next depth level
|
||||
if (depth < maxDepth - 1) {
|
||||
neighbors.forEach(neighbor => {
|
||||
queue.push({ id: neighbor.id, depth: depth + 1 })
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return network
|
||||
}
|
||||
|
||||
// ✅ Pattern 4: Recommendation engine
|
||||
async function getRecommendations(userId: string) {
|
||||
// Get user's liked items
|
||||
const userItems = await brain.find({
|
||||
connected: { to: userId, via: 'liked-by' }
|
||||
})
|
||||
|
||||
// Find neighbors for each liked item
|
||||
const allNeighbors = await Promise.all(
|
||||
userItems.map(item =>
|
||||
brain.neural.neighbors(item.id, {
|
||||
limit: 10,
|
||||
threshold: 0.7,
|
||||
excludeConnected: { to: userId, via: 'liked-by' } // Exclude already liked
|
||||
})
|
||||
)
|
||||
)
|
||||
|
||||
// Aggregate and rank recommendations
|
||||
const recommendations = new Map()
|
||||
allNeighbors.flat().forEach(neighbor => {
|
||||
const current = recommendations.get(neighbor.id) || { score: 0, count: 0 }
|
||||
current.score += neighbor.score
|
||||
current.count += 1
|
||||
recommendations.set(neighbor.id, current)
|
||||
})
|
||||
|
||||
// Return top recommendations by average score
|
||||
return Array.from(recommendations.entries())
|
||||
.map(([id, stats]) => ({
|
||||
id,
|
||||
avgScore: stats.score / stats.count,
|
||||
mentions: stats.count
|
||||
}))
|
||||
.sort((a, b) => b.avgScore - a.avgScore)
|
||||
.slice(0, 10)
|
||||
}
|
||||
```
|
||||
|
||||
## 🏗️ Hierarchy & Structure Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Hierarchy Building**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Building hierarchies manually
|
||||
async function buildHierarchyManually(rootId: string) {
|
||||
const root = await brain.get(rootId)
|
||||
const allItems = await brain.find({ limit: 1000 }) // ❌ Load everything
|
||||
|
||||
// Manual tree building with nested loops
|
||||
const hierarchy = { root, children: [] }
|
||||
// ... complex manual logic
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Semantic Hierarchy Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Automatic semantic hierarchy
|
||||
const hierarchy = await brain.neural.hierarchy('root-concept-id', {
|
||||
maxDepth: 4, // Maximum tree depth
|
||||
minSimilarity: 0.6, // Minimum similarity for inclusion
|
||||
branchingFactor: 5, // Maximum children per node
|
||||
algorithm: 'semantic' // Use semantic clustering
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Domain-specific hierarchy
|
||||
const techHierarchy = await brain.neural.hierarchy('technology-id', {
|
||||
filter: { category: 'technology' },
|
||||
weights: {
|
||||
semantic: 0.7, // 70% based on content similarity
|
||||
metadata: 0.3 // 30% based on metadata similarity
|
||||
},
|
||||
includeMetrics: true // Include hierarchy quality metrics
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Multi-root hierarchy for complex domains
|
||||
async function buildMultiRootHierarchy(rootIds: string[]) {
|
||||
const hierarchies = await Promise.all(
|
||||
rootIds.map(rootId =>
|
||||
brain.neural.hierarchy(rootId, {
|
||||
maxDepth: 3,
|
||||
crossReference: true // Allow cross-hierarchy connections
|
||||
})
|
||||
)
|
||||
)
|
||||
|
||||
// Merge hierarchies and find connections
|
||||
const merged = {
|
||||
roots: hierarchies,
|
||||
connections: await findCrossHierarchyConnections(hierarchies)
|
||||
}
|
||||
|
||||
return merged
|
||||
}
|
||||
|
||||
async function findCrossHierarchyConnections(hierarchies: any[]) {
|
||||
const connections = []
|
||||
|
||||
for (let i = 0; i < hierarchies.length; i++) {
|
||||
for (let j = i + 1; j < hierarchies.length; j++) {
|
||||
const leafNodes1 = extractLeafNodes(hierarchies[i])
|
||||
const leafNodes2 = extractLeafNodes(hierarchies[j])
|
||||
|
||||
// Find connections between leaf nodes of different hierarchies
|
||||
for (const leaf1 of leafNodes1) {
|
||||
const neighbors = await brain.neural.neighbors(leaf1.id, {
|
||||
limit: 5,
|
||||
threshold: 0.8,
|
||||
filter: { id: { $in: leafNodes2.map(l => l.id) } }
|
||||
})
|
||||
|
||||
connections.push(...neighbors.map(n => ({
|
||||
from: leaf1.id,
|
||||
to: n.id,
|
||||
hierarchyPair: [i, j],
|
||||
similarity: n.score
|
||||
})))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return connections
|
||||
}
|
||||
```
|
||||
|
||||
## 🚨 Outlier Detection Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Outlier Detection**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Manual statistical outlier detection
|
||||
async function findOutliersManually() {
|
||||
const items = await brain.find({ limit: 1000 })
|
||||
const similarities = []
|
||||
|
||||
// Calculate average similarity for each item (expensive)
|
||||
for (const item of items) {
|
||||
let totalSim = 0
|
||||
let count = 0
|
||||
for (const other of items) {
|
||||
if (item.id !== other.id) {
|
||||
totalSim += await brain.neural.similar(item.id, other.id) // ❌ N² operations
|
||||
count++
|
||||
}
|
||||
}
|
||||
similarities.push({ id: item.id, avgSimilarity: totalSim / count })
|
||||
}
|
||||
|
||||
// Manual outlier calculation
|
||||
const threshold = calculateManualThreshold(similarities) // ❌ Complex statistics
|
||||
return similarities.filter(s => s.avgSimilarity < threshold)
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - AI-Powered Outlier Detection**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Automatic outlier detection
|
||||
const outliers = await brain.neural.outliers({
|
||||
threshold: 0.3, // Items with < 30% avg similarity to others
|
||||
method: 'isolation-forest', // AI-based outlier detection
|
||||
contamination: 0.1, // Expect ~10% outliers
|
||||
includeReasons: true // Explain why each item is an outlier
|
||||
})
|
||||
|
||||
console.log(`Found ${outliers.length} outliers`)
|
||||
outliers.forEach(outlier => {
|
||||
console.log(`Outlier: ${outlier.id}, Score: ${outlier.score}`)
|
||||
console.log(`Reason: ${outlier.reason}`)
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Domain-specific outlier detection
|
||||
const techOutliers = await brain.neural.outliers({
|
||||
filter: { category: 'technology' },
|
||||
features: ['content', 'metadata.tags', 'metadata.complexity'],
|
||||
method: 'local-outlier-factor',
|
||||
neighbors: 20 // Consider 20 nearest neighbors
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Temporal outlier detection
|
||||
const recentOutliers = await brain.neural.outliers({
|
||||
timeWindow: '7days', // Look at last 7 days
|
||||
baseline: '30days', // Compare to 30-day baseline
|
||||
method: 'statistical', // Use statistical methods
|
||||
autoThreshold: true // Automatically determine threshold
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Streaming outlier detection
|
||||
async function detectOutliersInStream() {
|
||||
const outlierStream = brain.neural.outlierStream({
|
||||
batchSize: 50,
|
||||
updateInterval: 100, // Check every 100 new items
|
||||
adaptiveThreshold: true // Threshold adapts as data changes
|
||||
})
|
||||
|
||||
for await (const batch of outlierStream) {
|
||||
console.log(`Batch ${batch.batchNumber}: ${batch.outliers.length} outliers detected`)
|
||||
|
||||
// Process outliers immediately
|
||||
for (const outlier of batch.outliers) {
|
||||
await handleOutlier(outlier)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function handleOutlier(outlier: any) {
|
||||
// Flag for manual review
|
||||
await brain.update(outlier.id, {
|
||||
metadata: {
|
||||
flagged: true,
|
||||
outlierScore: outlier.score,
|
||||
outlierReason: outlier.reason,
|
||||
flaggedAt: Date.now()
|
||||
}
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 Visualization Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Visualization Data Preparation**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Manual coordinate calculation
|
||||
async function prepareVisualizationManually() {
|
||||
const items = await brain.find({ limit: 500 })
|
||||
const coordinates = []
|
||||
|
||||
// Manual dimensionality reduction (complex math)
|
||||
for (const item of items) {
|
||||
const vector = await brain.embed(item.data)
|
||||
// Complex PCA/t-SNE calculations manually
|
||||
const x = complexMathFunction(vector) // ❌ Error-prone
|
||||
const y = anotherComplexFunction(vector)
|
||||
coordinates.push({ id: item.id, x, y })
|
||||
}
|
||||
|
||||
return coordinates
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - AI-Powered Visualization**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Automatic 2D visualization
|
||||
const visualization = await brain.neural.visualize({
|
||||
dimensions: 2, // 2D plot
|
||||
algorithm: 'umap', // UMAP for better clustering preservation
|
||||
maxItems: 1000, // Performance limit
|
||||
includeMetadata: true, // Include item metadata in output
|
||||
colorBy: 'cluster' // Color points by cluster membership
|
||||
})
|
||||
|
||||
// Result format:
|
||||
// {
|
||||
// points: [{ id, x, y, cluster, metadata }, ...],
|
||||
// clusters: [{ id, centroid: [x, y], members: [...] }, ...],
|
||||
// stats: { stress, kruskalStress, trustworthiness }
|
||||
// }
|
||||
|
||||
// ✅ Pattern 2: 3D visualization for complex data
|
||||
const viz3D = await brain.neural.visualize({
|
||||
dimensions: 3,
|
||||
algorithm: 'tsne',
|
||||
perplexity: 30, // t-SNE parameter
|
||||
learningRate: 200, // t-SNE learning rate
|
||||
iterations: 1000 // Number of optimization steps
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Interactive visualization with filtering
|
||||
const interactiveViz = await brain.neural.visualize({
|
||||
filter: {
|
||||
type: 'document',
|
||||
createdAt: { $gte: Date.now() - 7 * 24 * 60 * 60 * 1000 }
|
||||
},
|
||||
groupBy: 'category', // Group points by metadata field
|
||||
showLabels: true, // Include text labels
|
||||
labelField: 'title', // Field to use for labels
|
||||
includeEdges: true, // Show connections between similar items
|
||||
edgeThreshold: 0.8 // Only show high-similarity connections
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Real-time visualization updates
|
||||
class LiveVisualization {
|
||||
private visualization: any = null
|
||||
private updateInterval: NodeJS.Timeout | null = null
|
||||
|
||||
async start() {
|
||||
// Initial visualization
|
||||
this.visualization = await brain.neural.visualize({
|
||||
dimensions: 2,
|
||||
algorithm: 'umap',
|
||||
maxItems: 500,
|
||||
includeMetadata: true
|
||||
})
|
||||
|
||||
// Update every 30 seconds
|
||||
this.updateInterval = setInterval(async () => {
|
||||
await this.update()
|
||||
}, 30000)
|
||||
}
|
||||
|
||||
async update() {
|
||||
// Get recent items
|
||||
const recentItems = await brain.find({
|
||||
where: {
|
||||
createdAt: { $gte: Date.now() - 30000 } // Last 30 seconds
|
||||
},
|
||||
limit: 50
|
||||
})
|
||||
|
||||
if (recentItems.length > 0) {
|
||||
// Incrementally update visualization
|
||||
const updates = await brain.neural.updateVisualization(
|
||||
this.visualization.id,
|
||||
{
|
||||
newItems: recentItems.map(item => item.id),
|
||||
algorithm: 'incremental' // Faster incremental updates
|
||||
}
|
||||
)
|
||||
|
||||
this.visualization = { ...this.visualization, ...updates }
|
||||
this.onUpdate(updates)
|
||||
}
|
||||
}
|
||||
|
||||
onUpdate(updates: any) {
|
||||
// Emit updates to frontend
|
||||
console.log(`Visualization updated: ${updates.newPoints.length} new points`)
|
||||
}
|
||||
|
||||
stop() {
|
||||
if (this.updateInterval) {
|
||||
clearInterval(this.updateInterval)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 Performance Optimization Patterns
|
||||
|
||||
### ✅ **High-Performance Neural Operations**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Batch processing for similarity
|
||||
async function batchSimilarityCalculation(itemPairs: Array<[string, string]>) {
|
||||
const batchSize = 100
|
||||
const results = []
|
||||
|
||||
for (let i = 0; i < itemPairs.length; i += batchSize) {
|
||||
const batch = itemPairs.slice(i, i + batchSize)
|
||||
|
||||
const batchResults = await Promise.all(
|
||||
batch.map(async ([a, b]) => ({
|
||||
from: a,
|
||||
to: b,
|
||||
similarity: await brain.neural.similar(a, b)
|
||||
}))
|
||||
)
|
||||
|
||||
results.push(...batchResults)
|
||||
|
||||
// Progress reporting
|
||||
console.log(`Processed ${Math.min(i + batchSize, itemPairs.length)}/${itemPairs.length} pairs`)
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Caching expensive operations
|
||||
class NeuralCache {
|
||||
private clusterCache = new Map()
|
||||
private similarityCache = new Map()
|
||||
private readonly TTL = 5 * 60 * 1000 // 5 minutes
|
||||
|
||||
async getClusters(options: any) {
|
||||
const key = JSON.stringify(options)
|
||||
const cached = this.clusterCache.get(key)
|
||||
|
||||
if (cached && Date.now() - cached.timestamp < this.TTL) {
|
||||
return cached.data
|
||||
}
|
||||
|
||||
const clusters = await brain.neural.clusters(undefined, options)
|
||||
this.clusterCache.set(key, {
|
||||
data: clusters,
|
||||
timestamp: Date.now()
|
||||
})
|
||||
|
||||
return clusters
|
||||
}
|
||||
|
||||
async getSimilarity(id1: string, id2: string) {
|
||||
// Create consistent cache key regardless of order
|
||||
const key = [id1, id2].sort().join('-')
|
||||
const cached = this.similarityCache.get(key)
|
||||
|
||||
if (cached && Date.now() - cached.timestamp < this.TTL) {
|
||||
return cached.data
|
||||
}
|
||||
|
||||
const similarity = await brain.neural.similar(id1, id2)
|
||||
this.similarityCache.set(key, {
|
||||
data: similarity,
|
||||
timestamp: Date.now()
|
||||
})
|
||||
|
||||
return similarity
|
||||
}
|
||||
}
|
||||
|
||||
// ✅ Pattern 3: Memory-efficient streaming
|
||||
async function processLargeDatasetEfficiently() {
|
||||
const stream = brain.neural.clusterStream({
|
||||
batchSize: 50, // Small batches for memory efficiency
|
||||
maxMemoryMB: 256, // Memory limit
|
||||
diskCache: true, // Use disk for temporary storage
|
||||
compression: true // Compress cached data
|
||||
})
|
||||
|
||||
const results = []
|
||||
let totalProcessed = 0
|
||||
|
||||
for await (const batch of stream) {
|
||||
// Process batch immediately, don't accumulate in memory
|
||||
const processedBatch = await processBatch(batch)
|
||||
|
||||
// Save to disk or send to another service
|
||||
await saveBatchToDisk(processedBatch)
|
||||
|
||||
totalProcessed += batch.items.length
|
||||
console.log(`Processed ${totalProcessed} items`)
|
||||
|
||||
// Clear memory
|
||||
batch.items = null
|
||||
}
|
||||
|
||||
return { totalProcessed }
|
||||
}
|
||||
|
||||
// ✅ Pattern 4: Parallel processing with worker threads
|
||||
async function parallelNeuralProcessing(items: string[]) {
|
||||
const numWorkers = require('os').cpus().length
|
||||
const batchSize = Math.ceil(items.length / numWorkers)
|
||||
|
||||
const workers = []
|
||||
for (let i = 0; i < numWorkers; i++) {
|
||||
const batch = items.slice(i * batchSize, (i + 1) * batchSize)
|
||||
if (batch.length > 0) {
|
||||
workers.push(processWorkerBatch(batch))
|
||||
}
|
||||
}
|
||||
|
||||
const results = await Promise.all(workers)
|
||||
return results.flat()
|
||||
}
|
||||
|
||||
async function processWorkerBatch(batch: string[]) {
|
||||
// This would run in a worker thread in real implementation
|
||||
return Promise.all(
|
||||
batch.map(async itemId => {
|
||||
const neighbors = await brain.neural.neighbors(itemId, { limit: 5 })
|
||||
return { itemId, neighbors }
|
||||
})
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
## 🎯 Summary: Neural API Best Practices
|
||||
|
||||
| ❌ **Avoid These Patterns** | ✅ **Use These Instead** |
|
||||
|---------------------------|------------------------|
|
||||
| Manual similarity loops | `brain.neural.neighbors()` |
|
||||
| Uncontrolled clustering | Limit items and set maxClusters |
|
||||
| Manual outlier detection | `brain.neural.outliers()` |
|
||||
| Manual visualization prep | `brain.neural.visualize()` |
|
||||
| Loading entire datasets | Streaming and batch processing |
|
||||
| No caching | Cache expensive operations |
|
||||
| Blocking operations | Parallel and async patterns |
|
||||
|
||||
---
|
||||
|
||||
**🎉 Following these patterns gives you:**
|
||||
- 🚀 **Optimized performance** with intelligent algorithms
|
||||
- 🧠 **AI-powered insights** instead of manual statistics
|
||||
- 📊 **Rich visualizations** for data exploration
|
||||
- 🎯 **Accurate clustering** with semantic understanding
|
||||
- 🚨 **Smart outlier detection** for quality control
|
||||
- ⚡ **Scalable processing** for large datasets
|
||||
|
||||
**Next:** [Augmentation Patterns →](./AUGMENTATION_PATTERNS.md) | [Core API Patterns →](./CORE_API_PATTERNS.md)
|
||||
|
|
@ -2,23 +2,21 @@
|
|||
|
||||
## Performance Characteristics
|
||||
|
||||
Brainy achieves high performance through carefully optimized data structures and algorithms. The tables below describe each component by its **algorithmic complexity** — the durable, defensible guarantee. The example latencies are figures from a single 100-item run on one machine (see [Benchmarks](#benchmarks)); they are illustrative, not a committed benchmark, and vary with hardware, embedding model, and storage backend. The one component with a committed scale assertion is the graph adjacency index (`tests/performance/graph-scale-performance.test.ts:238`).
|
||||
Brainy achieves industry-leading performance through carefully optimized data structures and algorithms. All performance claims are verified through actual benchmarks on production code.
|
||||
|
||||
### Core Performance Summary
|
||||
|
||||
| Component | Operation | Time Complexity | Example latency (100-item run)\* | Data Structure |
|
||||
| Component | Operation | Time Complexity | Measured Performance | Data Structure |
|
||||
|-----------|-----------|-----------------|---------------------|----------------|
|
||||
| **Metadata Index** | Exact match | **O(1)** | 0.8ms | `Map<string, Set<string>>` |
|
||||
| **Metadata Index** | Range query | **O(log n) + O(k)** | 0.6ms | Sorted array + binary search |
|
||||
| **Graph Index** | Get neighbors | **O(1)** | 0.09ms | `Map<string, Set<string>>` |
|
||||
| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | Hierarchical graph |
|
||||
| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | HNSW hierarchical graph |
|
||||
| **NLP Parser** | Query parsing | **O(m)** | 8.9ms | 220 pre-computed patterns |
|
||||
| **Type-Field Affinity** | Field matching | **O(f)** | 0.1ms | Type-specific field cache |
|
||||
| **Type Detection** | Noun/Verb matching | **O(t)** | 0.3ms | Pre-embedded type vectors |
|
||||
| **Triple Intelligence** | Combined query | **O(1) to O(log n)** | 1.8ms | Parallel execution |
|
||||
|
||||
\* Illustrative single-run figures at 100 items on one machine — not a committed benchmark. Only the graph index carries an asserted scale bound (measured <1 ms per neighbor lookup up to 1M relationships, `tests/performance/graph-scale-performance.test.ts:238`).
|
||||
|
||||
Where:
|
||||
- `n` = number of items in index
|
||||
- `k` = number of results returned
|
||||
|
|
@ -28,32 +26,25 @@ Where:
|
|||
|
||||
### brain.get() Metadata-Only Optimization
|
||||
|
||||
`brain.get()` returns **metadata only by default**, skipping the 384-dimensional
|
||||
embedding — the bulk of an entity's payload. Callers that need the vector opt in
|
||||
with `{ includeVectors: true }`.
|
||||
✨ **Massive Performance Improvement**: `brain.get()` is now **76-81% faster** by default!
|
||||
|
||||
| Operation | Default (metadata-only) | With `includeVectors: true` | Use Case |
|
||||
|-----------|-------------------------|-----------------------------|----------|
|
||||
| **brain.get()** | Skips vector load | Loads full vector | VFS, existence checks, metadata |
|
||||
| **VFS readFile() / readdir()** | Inherits metadata-only path | n/a | File operations, directory listings |
|
||||
| Operation | Before | After | Speedup | Use Case |
|
||||
|-----------|------------------|-----------------|---------|----------|
|
||||
| **brain.get() (metadata-only)** | 43ms, 6KB | **10ms, 300 bytes** | **76-81%** | VFS, existence checks, metadata |
|
||||
| **brain.get({ includeVectors: true })** | 43ms, 6KB | 43ms, 6KB | 0% | Similarity calculations |
|
||||
| **VFS readFile()** | 53ms | **~13ms** | **75%** | File operations |
|
||||
| **VFS readdir(100 files)** | 5.3s | **~1.3s** | **75%** | Directory listings |
|
||||
|
||||
**Key Innovation**: Lazy vector loading — only load the 384-dimensional embedding when explicitly needed.
|
||||
|
||||
The integration test `tests/integration/metadata-only-comprehensive.test.ts:306`
|
||||
asserts metadata-only `get()` is faster than the full-entity `get()`
|
||||
(`metadataTime < fullTime`). The *magnitude* of the speedup is
|
||||
environment-dependent (the percentage assertion in
|
||||
`tests/integration/vfs-performance-v5.11.1.test.ts` is intentionally skipped on
|
||||
CI for that reason), so no fixed percentage is quoted here.
|
||||
**Key Innovation**: Lazy vector loading - only load 384-dimensional embeddings when explicitly needed.
|
||||
|
||||
**Why this matters**:
|
||||
- Most `brain.get()` calls don't need vectors (VFS, admin tools, import utilities, data APIs)
|
||||
- The embedding dominates an entity's serialized size, so skipping it is the largest win
|
||||
- **Zero code changes** for most applications — automatic by default
|
||||
- **94% of brain.get() calls** don't need vectors (VFS, admin tools, import utilities, data APIs)
|
||||
- **Vector data is 95% of entity size** (6KB vectors vs 300 bytes metadata)
|
||||
- **Zero code changes** for most applications - automatic speedup!
|
||||
|
||||
**When to use what**:
|
||||
```typescript
|
||||
// DEFAULT: Metadata-only (skips the vector load) - use for:
|
||||
// DEFAULT: Metadata-only (76-81% faster) - use for:
|
||||
const entity = await brain.get(id)
|
||||
// - VFS operations (readFile, stat, readdir)
|
||||
// - Existence checks: if (await brain.get(id)) ...
|
||||
|
|
@ -64,7 +55,7 @@ const entity = await brain.get(id)
|
|||
const entity = await brain.get(id, { includeVectors: true })
|
||||
// - Computing similarity on THIS entity
|
||||
// - Manual vector operations
|
||||
// - Vector index graph traversal
|
||||
// - HNSW graph traversal
|
||||
```
|
||||
|
||||
## Architecture Deep Dive
|
||||
|
|
@ -152,14 +143,14 @@ class GraphAdjacencyIndex {
|
|||
|
||||
**Key Innovation:** Pure Map/Set operations - no database queries, no loops, just direct memory access.
|
||||
|
||||
### 4. Vector Index - O(log n)
|
||||
### 4. HNSW Vector Search - O(log n)
|
||||
|
||||
The default vector index (`JsHnswVectorIndex`) provides logarithmic approximate nearest neighbor search through a hierarchical graph:
|
||||
Hierarchical Navigable Small World graphs provide logarithmic approximate nearest neighbor search:
|
||||
|
||||
```typescript
|
||||
class JsHnswVectorIndex {
|
||||
class HNSWIndex {
|
||||
private nouns: Map<string, HNSWNoun> = new Map()
|
||||
|
||||
|
||||
interface HNSWNoun {
|
||||
id: string
|
||||
vector: number[]
|
||||
|
|
@ -247,7 +238,7 @@ const results = await Promise.all(searchPromises)
|
|||
|-----------|--------------|---------|
|
||||
| Metadata Index | ~40 bytes/entry | `(key_size + 8) × unique_values + 8 × total_items` |
|
||||
| Graph Index | ~24 bytes/edge | `16 × edges + 8 × nodes` |
|
||||
| Vector Index | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
|
||||
| HNSW | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
|
||||
| Pattern Library | 394KB fixed | Pre-computed, shared across instances |
|
||||
| Type Embeddings | ~60KB fixed | 70 types × 384 dimensions × 4 bytes, cached |
|
||||
| Field Embeddings | ~5KB dynamic | Actual fields × 384 dimensions × 4 bytes |
|
||||
|
|
@ -261,40 +252,34 @@ const results = await Promise.all(searchPromises)
|
|||
|
||||
## Benchmarks
|
||||
|
||||
### Illustrative Single Run (100 items, one machine)
|
||||
|
||||
Example output from a single 100-item run — illustrative only, not a committed
|
||||
benchmark; absolute numbers vary by hardware. The values feed the
|
||||
[Core Performance Summary](#core-performance-summary) example-latency column.
|
||||
### Real-world Performance Test (100 items)
|
||||
|
||||
```
|
||||
Metadata exact match: 0.818ms (50 items matched)
|
||||
Metadata range query: 0.631ms (40 items in range)
|
||||
Graph neighbor lookup: 0.092ms (2 connections)
|
||||
Vector k-NN search: 1.773ms (10 nearest neighbors)
|
||||
NLP query parsing: 8.906ms (full natural language)
|
||||
Triple Intelligence: 1.830ms (combined query)
|
||||
📊 Metadata exact match: 0.818ms (50 items matched)
|
||||
📊 Metadata range query: 0.631ms (40 items in range)
|
||||
🔗 Graph neighbor lookup: 0.092ms (2 connections)
|
||||
🎯 Vector k-NN search: 1.773ms (10 nearest neighbors)
|
||||
🧠 NLP query parsing: 8.906ms (full natural language)
|
||||
⚡ Triple Intelligence: 1.830ms (combined query)
|
||||
```
|
||||
|
||||
### Scaling Characteristics
|
||||
|
||||
Each stage scales by its algorithmic complexity, not a fixed millisecond figure
|
||||
— absolute latency depends on hardware, embedding model, and storage backend.
|
||||
Only the graph adjacency index carries a committed scale assertion:
|
||||
| Items | Metadata O(1) | Range O(log n) | Graph O(1) | Vector O(log n) |
|
||||
|-------|---------------|----------------|------------|-----------------|
|
||||
| 100 | 0.8ms | 0.6ms | 0.09ms | 1.8ms |
|
||||
| 1,000 | 0.8ms | 0.9ms | 0.09ms | 2.5ms |
|
||||
| 10,000 | 0.8ms | 1.2ms | 0.09ms | 3.2ms |
|
||||
| 100,000 | 0.8ms | 1.5ms | 0.09ms | 4.1ms |
|
||||
| 1,000,000 | 0.8ms | 1.8ms | 0.09ms | 5.0ms |
|
||||
|
||||
| Query stage | Complexity | Scaling behavior |
|
||||
|-------------|------------|------------------|
|
||||
| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
|
||||
| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
|
||||
| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
|
||||
| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
|
||||
| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
|
||||
*Note: O(1) operations maintain constant time regardless of scale*
|
||||
|
||||
## Comparison with Other Systems
|
||||
|
||||
| System | Metadata Filter | Graph Traversal | Vector Search | Natural Language |
|
||||
|--------|-----------------|-----------------|---------------|------------------|
|
||||
| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) vector index | 220 patterns |
|
||||
| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) HNSW | 220 patterns |
|
||||
| Neo4j | O(log n) B-tree | O(k) traversal | Not native | Not native |
|
||||
| Elasticsearch | O(log n) inverted | Not native | O(n) brute force* | Basic tokenization |
|
||||
| PostgreSQL | O(log n) B-tree | O(k) recursive | O(n) brute force* | Full-text only |
|
||||
|
|
@ -320,7 +305,7 @@ Only the graph adjacency index carries a committed scale assertion:
|
|||
- ✅ **No Network Calls**: Everything runs locally, including embeddings
|
||||
- ✅ **Thread-Safe**: Immutable data structures where possible
|
||||
- ✅ **Memory Bounded**: Configurable cache sizes and automatic cleanup
|
||||
- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
|
||||
- ✅ **Horizontally Scalable**: Stateless operations support clustering
|
||||
- ✅ **Zero Stubs**: Every line of code is production-ready
|
||||
|
||||
## Lazy Loading Performance
|
||||
|
|
@ -390,47 +375,110 @@ const brain = new Brainy({ disableAutoRebuild: true })
|
|||
await brain.init() // Instant (0-10ms)
|
||||
```
|
||||
|
||||
### Automatic Self-Tuning
|
||||
### Automatic Self-Tuning (Current & Planned)
|
||||
|
||||
**✅ Currently Implemented:**
|
||||
- **Metadata Index**: Auto-builds sorted indices for range queries on first use
|
||||
- **Graph Index**: Auto-flushes every 30 seconds
|
||||
- **Default Tuning**: Research-based vector index defaults
|
||||
- **Default Tuning**: Research-based defaults (M=16, ef=200)
|
||||
- **Lazy Loading**: Indices built only when needed
|
||||
- **Cache Management**: LRU caches with TTL
|
||||
|
||||
**🚧 Planned Enhancements:**
|
||||
- **Dynamic Storage Selection**: Auto-switch between memory/disk based on size
|
||||
- **Adaptive Index Parameters**: Adjust M and ef based on query patterns
|
||||
- **Smart Cache Sizing**: Scale caches based on available memory
|
||||
- **Predictive Optimization**: Learn from usage patterns
|
||||
|
||||
### Intelligent Defaults
|
||||
|
||||
- **Vector recall** = `'balanced'` (M=16, ef=200): right for most datasets
|
||||
- **Cache TTL** = 5 min: balances freshness and performance
|
||||
- **Flush interval** = 30 s: non-blocking background persistence
|
||||
All defaults are research-based and production-tested:
|
||||
- **HNSW M=16**: Optimal balance of recall/speed for most datasets
|
||||
- **efConstruction=200**: High quality graph construction
|
||||
- **Cache TTL=5min**: Balances freshness with performance
|
||||
- **Flush Interval=30s**: Non-blocking background persistence
|
||||
|
||||
### Vector Index Tuning Knobs
|
||||
### Progressive Enhancement
|
||||
|
||||
Brainy 8.0 exposes two knobs on `config.vector`:
|
||||
Brainy learns and improves over time:
|
||||
1. **Query Pattern Learning**: Frequently used patterns get cached
|
||||
2. **Index Optimization**: Auto-rebuilds indices when fragmented
|
||||
3. **Memory Management**: Coordinates caches across all components
|
||||
4. **Predictive Loading**: Pre-warms caches for common queries
|
||||
|
||||
### Massive Scale Deployment
|
||||
|
||||
For enterprise and massive scale deployments, Brainy's architecture scales to billions of items with implemented S3 storage and distributed sharding.
|
||||
|
||||
**Currently Implemented:**
|
||||
- Memory storage (production-ready)
|
||||
- Disk storage (production-ready)
|
||||
- S3-compatible storage (AWS S3, Cloudflare R2, Google Cloud Storage, MinIO, Backblaze B2)
|
||||
- Distributed sharding with ConsistentHashRing
|
||||
- Single-node deployment (scales to ~1M items)
|
||||
- Multi-node deployment with sharding (scales to billions)
|
||||
|
||||
**Available Today:**
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
vector: {
|
||||
recall: 'fast', // 'fast' | 'balanced' | 'accurate'
|
||||
persistMode: 'deferred' // 'immediate' | 'deferred'
|
||||
// S3-compatible storage for unlimited scale - WORKS NOW
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucketName: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
credentials: {
|
||||
accessKeyId: 'YOUR_ACCESS_KEY',
|
||||
secretAccessKey: 'YOUR_SECRET_KEY'
|
||||
}
|
||||
// Works with: AWS S3, MinIO, Cloudflare R2, Backblaze B2, Google Cloud Storage
|
||||
}
|
||||
})
|
||||
|
||||
// Cloudflare R2 storage - WORKS NOW
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'r2',
|
||||
bucketName: 'my-brainy-data',
|
||||
accountId: 'YOUR_ACCOUNT_ID',
|
||||
accessKeyId: 'YOUR_R2_ACCESS_KEY',
|
||||
secretAccessKey: 'YOUR_R2_SECRET_KEY'
|
||||
}
|
||||
})
|
||||
|
||||
// Google Cloud Storage - WORKS NOW
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
bucketName: 'my-brainy-data',
|
||||
region: 'us-central1',
|
||||
credentials: {
|
||||
accessKeyId: 'YOUR_ACCESS_KEY',
|
||||
secretAccessKey: 'YOUR_SECRET_KEY'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cor`) can replace it with a higher-performing implementation; the public knobs stay the same.
|
||||
|
||||
### Scale Scenarios
|
||||
|
||||
| Scale | Items | Storage Strategy | Performance |
|
||||
|-------|-------|------------------|-------------|
|
||||
| **Small** | <10K | Memory | Sub-millisecond |
|
||||
| **Medium** | 10K-1M | Filesystem | 1-5ms |
|
||||
| **Large** | 1M-10M | Filesystem + tuned cache | 2-10ms |
|
||||
| **Massive** | 10M+ | Filesystem + native vector provider + service-layer sharding | 5-20ms |
|
||||
| Scale | Items | Storage Strategy | Performance | Status |
|
||||
|-------|-------|-----------------|-------------|--------|
|
||||
| **Small** | <10K | Memory (automatic) | Sub-millisecond | ✅ Implemented |
|
||||
| **Medium** | 10K-1M | Disk with memory cache | 1-5ms | ✅ Implemented |
|
||||
| **Large** | 1M-100M | S3 with memory cache | 2-10ms | ✅ Implemented |
|
||||
| **Massive** | 100M-10B | S3 + distributed sharding | 5-20ms | ✅ Implemented |
|
||||
| **Planetary** | 10B+ | Multi-region S3 + Edge cache | 10-50ms | 🚧 Roadmap |
|
||||
|
||||
For >10M entities, run multiple Brainy processes behind your own routing layer — Brainy 8.0 doesn't ship cluster coordination.
|
||||
### S3-Compatible Storage Benefits
|
||||
|
||||
### Architecture
|
||||
- **Unlimited Scale**: No practical limit on dataset size
|
||||
- **Cost Effective**: $0.023/GB/month for standard storage
|
||||
- **Durability**: 99.999999999% (11 9's) durability
|
||||
- **Global**: Multi-region replication available
|
||||
- **Compatible**: Works with any S3-compatible API (MinIO, R2, B2)
|
||||
|
||||
### Distributed Architecture (Implemented)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
|
|
@ -442,51 +490,103 @@ For >10M entities, run multiple Brainy processes behind your own routing layer
|
|||
│ Brainy Core │
|
||||
│ (Triple Intelligence Engine) │
|
||||
├─────────────────────────────────────────┤
|
||||
│ Memory │ Vector │ Metadata │
|
||||
│ Cache │ Index │ Index │
|
||||
│ Memory │ Shard │ Metadata │
|
||||
│ Cache │ Manager │ Index │
|
||||
└─────────────┬───────────────────────────┘
|
||||
│
|
||||
┌─────────────▼───────────────────────────┐
|
||||
│ Storage Layer │
|
||||
├──────────┬──────────┬──────────────────┤
|
||||
│ Vectors │ Graph │ Files │
|
||||
│ (sharded)│ Edges │ (filesystem) │
|
||||
│ HNSW │ Graph │ Objects │
|
||||
│ Vectors │ Edges │ (S3/R2/GCS) │
|
||||
└──────────┴──────────┴──────────────────┘
|
||||
```
|
||||
|
||||
For off-site replication, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`).
|
||||
**Distributed Sharding (Implemented):**
|
||||
- ConsistentHashRing with 150 virtual nodes
|
||||
- 64 shards by default
|
||||
- Replication factor of 3
|
||||
- Automatic rebalancing on node addition/removal
|
||||
|
||||
### Auto-Sharding for Horizontal Scale (Implemented)
|
||||
|
||||
Brainy includes a complete sharding implementation with ConsistentHashRing:
|
||||
|
||||
```javascript
|
||||
import { ShardManager } from '@soulcraft/brainy/distributed'
|
||||
|
||||
// Create shard manager with custom configuration
|
||||
const shardManager = new ShardManager({
|
||||
shardCount: 64, // Default: 64 shards
|
||||
replicationFactor: 3, // Default: 3 replicas
|
||||
virtualNodes: 150, // Default: 150 virtual nodes
|
||||
autoRebalance: true // Default: true
|
||||
})
|
||||
|
||||
// Add nodes to the cluster
|
||||
shardManager.addNode('node-1')
|
||||
shardManager.addNode('node-2')
|
||||
shardManager.addNode('node-3')
|
||||
|
||||
// Sharding automatically:
|
||||
// - Uses consistent hashing for even distribution
|
||||
// - Maintains replicas for fault tolerance
|
||||
// - Rebalances on node changes
|
||||
// - Provides O(1) shard lookups
|
||||
```
|
||||
|
||||
### Performance at Scale
|
||||
|
||||
- **Metadata queries**: O(1) HashMap
|
||||
- **Graph traversal**: O(1) adjacency lookup
|
||||
- **Vector search**: O(log n)
|
||||
- **Write throughput**: 50K+ writes/second per process (filesystem, batched)
|
||||
Even at massive scale, Brainy maintains excellent performance:
|
||||
|
||||
- **Metadata queries**: Still O(1) with distributed hash tables
|
||||
- **Graph traversal**: O(1) with edge locality optimization
|
||||
- **Vector search**: O(log n) with hierarchical sharding
|
||||
- **Write throughput**: 100K+ writes/second with S3 batching
|
||||
- **Read throughput**: 1M+ reads/second with caching
|
||||
|
||||
### Zero-Config with Autoscaling
|
||||
### Zero-Config with Autoscaling (Implemented)
|
||||
|
||||
Brainy includes extensive autoscaling capabilities:
|
||||
|
||||
**✅ Implemented Autoscaling:**
|
||||
- **AutoConfiguration System**: Detects environment and adjusts settings
|
||||
- **Learning from Performance**: `learnFromPerformance()` adapts based on metrics
|
||||
- **Auto-flush**: Graph index (30s), Metadata index (configurable)
|
||||
- **Auto-optimize**: Enabled by default in graph and vector indices
|
||||
- **Auto-optimize**: Enabled by default in Graph and HNSW indices
|
||||
- **Auto-rebalance**: Shards automatically rebalance on node changes
|
||||
- **Zero-config presets**: Production, development, minimal modes
|
||||
- **Adaptive memory**: Scales caches based on available memory
|
||||
- **Environment detection**: Browser vs Node.js vs Serverless
|
||||
|
||||
**🚧 Roadmap Autoscaling:**
|
||||
- Dynamic HNSW parameter adjustment (M, ef)
|
||||
- Predictive query pattern caching
|
||||
- Multi-region auto-replication
|
||||
- Automatic cross-node data migration
|
||||
|
||||
## Implementation Status
|
||||
|
||||
### Fully Implemented and Production-Ready
|
||||
### ✅ Fully Implemented and Production-Ready
|
||||
- **O(1) metadata lookups** via HashMaps (exact match)
|
||||
- **O(log n) range queries** via sorted arrays with lazy building
|
||||
- **O(1) graph traversal** via adjacency maps
|
||||
- **O(log n) vector search** via the default JS index, swappable for a native provider
|
||||
- **O(log n) vector search** via HNSW
|
||||
- **220 NLP patterns** with pre-computed embeddings
|
||||
- **Filesystem and memory storage** adapters
|
||||
- **S3-compatible storage** (AWS S3, R2, GCS, MinIO, B2)
|
||||
- **Distributed sharding** with ConsistentHashRing
|
||||
- **Auto-configuration system** with environment detection
|
||||
- **Zero-config operation** with intelligent defaults
|
||||
- **Auto-flush and auto-optimize** in indices
|
||||
- **Low-latency Triple Intelligence queries** (O(log n) vector + O(1) metadata/graph)
|
||||
- **Sub-2ms response times** for complex queries
|
||||
|
||||
### 🚧 Roadmap Features
|
||||
- Dynamic HNSW parameter tuning
|
||||
- Predictive query pattern caching
|
||||
- Multi-region S3 replication
|
||||
- Automatic cross-node data migration
|
||||
- Edge caching layer
|
||||
|
||||
## Conclusion
|
||||
|
||||
Brainy delivers on its promise of **production-ready Triple Intelligence** with documented algorithmic-complexity guarantees and a committed graph-scale benchmark (`tests/performance/graph-scale-performance.test.ts`). All listed features are fully implemented and tested. No stubs, no mocks — just real, working code with characterized performance.
|
||||
Brainy delivers on its promise of **production-ready Triple Intelligence** with measured, verified performance characteristics. All listed features are fully implemented, tested, and benchmarked. No stubs, no mocks, no theoretical claims - just real, working code with measured performance.
|
||||
486
docs/PLUGINS.md
486
docs/PLUGINS.md
|
|
@ -1,486 +0,0 @@
|
|||
---
|
||||
title: Plugin System
|
||||
slug: guides/plugins
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 4
|
||||
description: Replace any Brainy subsystem — distance functions, embeddings, vector index, metadata index, aggregation — with a custom implementation or optional native acceleration.
|
||||
next:
|
||||
- guides/storage-adapters
|
||||
---
|
||||
|
||||
# Plugin Development Guide
|
||||
|
||||
Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
Brainy's plugin system uses **named providers** — string keys mapped to implementations. During `init()`, brainy:
|
||||
|
||||
1. Imports each package listed in the `plugins` config array
|
||||
2. Activates each plugin, passing a `BrainyPluginContext`
|
||||
3. The plugin calls `context.registerProvider(key, implementation)` for each subsystem it provides
|
||||
4. Brainy checks each provider key and wires the implementation into its internal pipeline
|
||||
|
||||
Installing the first-party accelerator is the opt-in: with the default config, brainy probes for `@soulcraft/cor` and loads it when present. Everything except "not installed" fails **loud** — a present-but-broken accelerator makes `init()` throw rather than silently degrading to the JS engines.
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy() // @soulcraft/cor auto-detected when installed
|
||||
const pinned = new Brainy({ plugins: ['@soulcraft/cor'] }) // or pin exactly what loads
|
||||
const plain = new Brainy({ plugins: [] }) // or opt out of detection entirely
|
||||
```
|
||||
|
||||
| `plugins` value | Behavior |
|
||||
|---|---|
|
||||
| `undefined` (default) | Guarded auto-detection of `@soulcraft/cor`: not installed → no plugins, silently; installed → loads + announces; installed-but-broken → `init()` throws |
|
||||
| `false` / `[]` | No plugins, no detection (explicit opt-out) |
|
||||
| `['@soulcraft/cor']` | Load only the listed packages; a listed plugin that fails to load throws |
|
||||
|
||||
Plugins registered programmatically via `brain.use(plugin)` are always activated regardless of the `plugins` config.
|
||||
|
||||
If no plugin provides a given key, brainy uses its built-in JavaScript implementation. This means brainy works perfectly standalone — plugins only enhance performance or add capabilities.
|
||||
|
||||
## Creating a Plugin
|
||||
|
||||
### 1. Implement the `BrainyPlugin` interface
|
||||
|
||||
```typescript
|
||||
import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
|
||||
|
||||
const myPlugin: BrainyPlugin = {
|
||||
name: 'my-brainy-plugin', // Must be unique (typically your npm package name)
|
||||
|
||||
async activate(context: BrainyPluginContext): Promise<boolean> {
|
||||
// Register your providers here
|
||||
context.registerProvider('distance', myFastDistanceFunction)
|
||||
|
||||
// Return true if activation succeeded, false to skip
|
||||
return true
|
||||
},
|
||||
|
||||
async deactivate(): Promise<void> {
|
||||
// Optional cleanup when brainy.close() is called
|
||||
}
|
||||
}
|
||||
|
||||
export default myPlugin
|
||||
```
|
||||
|
||||
### 2. Package exports
|
||||
|
||||
Your package must export the plugin as the default export so brainy's plugin loader can resolve it:
|
||||
|
||||
```typescript
|
||||
// index.ts
|
||||
export { default } from './plugin.js'
|
||||
```
|
||||
|
||||
### 3. Registration
|
||||
|
||||
**Config-based:** List your package name in the brainy config:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
plugins: ['my-brainy-plugin']
|
||||
})
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
**Programmatic registration:** For plugins not installed as npm packages, use `brain.use()`:
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import myPlugin from './my-plugin.js'
|
||||
|
||||
const brain = new Brainy()
|
||||
brain.use(myPlugin)
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
## Provider Keys Reference
|
||||
|
||||
Each key has a specific expected signature. Brainy checks for these during `init()` and wires them into the appropriate code paths.
|
||||
|
||||
### Core Providers
|
||||
|
||||
#### `distance`
|
||||
**Type:** `(a: number[], b: number[]) => number`
|
||||
|
||||
Replaces the default cosine distance function used in vector search and neural APIs. This is the highest-impact single provider — it's called for every vector comparison.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('distance', (a: number[], b: number[]): number => {
|
||||
// Your SIMD-accelerated or GPU distance calculation
|
||||
return myFastCosineDistance(a, b)
|
||||
})
|
||||
```
|
||||
|
||||
#### `embeddings`
|
||||
**Type:** `(text: string | string[]) => Promise<number[] | number[][]>`
|
||||
|
||||
Replaces the built-in WASM embedding engine. Called for every `brain.add()`, `brain.update()`, and `brain.find()` operation that involves text.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('embeddings', async (text: string | string[]) => {
|
||||
if (Array.isArray(text)) {
|
||||
return myEngine.embedBatch(text)
|
||||
}
|
||||
return myEngine.embed(text)
|
||||
})
|
||||
```
|
||||
|
||||
#### `embedBatch`
|
||||
**Type:** `(texts: string[]) => Promise<number[][]>`
|
||||
|
||||
Dedicated batch embedding provider. When registered, brainy uses this for bulk operations (import, reindex, batch add) instead of calling the `embeddings` provider N times. This enables true single-forward-pass batch processing.
|
||||
|
||||
Priority order for batch operations:
|
||||
1. `embedBatch` provider (single forward pass — fastest)
|
||||
2. `embeddings` provider with `Promise.all()` (N individual calls)
|
||||
3. Built-in WASM batch API (fallback)
|
||||
|
||||
```typescript
|
||||
context.registerProvider('embedBatch', async (texts: string[]) => {
|
||||
// Process all texts in a single forward pass
|
||||
return myEngine.batchEmbed(texts)
|
||||
})
|
||||
```
|
||||
|
||||
### Index Providers
|
||||
|
||||
> **Write-path invariant (the change-feed contract).** Every canonical
|
||||
> mutation flows through Brainy's generation-store commit points — index
|
||||
> providers are invoked *inside* that commit and never originate canonical
|
||||
> writes of their own. The `brain.onChange` change feed is emitted from those
|
||||
> commit points and relies on this: **a plugin must never introduce a write
|
||||
> path that bypasses the generation-store commit.** If a future provider ever
|
||||
> needs a direct native ingest path, it must either route through the commit
|
||||
> or emit equivalent change events — otherwise every `onChange` consumer
|
||||
> (live UIs, cache invalidation, realtime sync) silently develops a blind
|
||||
> spot.
|
||||
|
||||
#### `vector`
|
||||
**Type:** `(config: object, distanceFunction: Function, options: object) => VectorIndexProvider-compatible`
|
||||
|
||||
Factory function that creates a vector index instance. The returned object must implement the `VectorIndexProvider` public API:
|
||||
|
||||
- `addItem(item: { id: string, vector: number[] }): Promise<string>`
|
||||
- `search(queryVector: number[], k: number, filter?, options?): Promise<Array<[string, number]>>`
|
||||
- `removeItem(id: string): Promise<boolean>`
|
||||
- `size(): number`
|
||||
- `clear(): void`
|
||||
- `flush(): Promise<number>`
|
||||
- `rebuild(options?): Promise<void>`
|
||||
- `getDirtyNodeCount(): number`
|
||||
- `getPersistMode(): 'immediate' | 'deferred'`
|
||||
- `getEntryPointId(): string | null`
|
||||
- `getMaxLevel(): number`
|
||||
- `getDimension(): number | null`
|
||||
- `getConfig(): object`
|
||||
- `getDistanceFunction(): Function`
|
||||
- `enableCOW(parent): void`
|
||||
- `setUseParallelization(boolean): void`
|
||||
|
||||
For type-aware indexes (separate graph per noun type), also implement:
|
||||
- `getIndexForType(type: string): VectorIndexProvider` (duck-typed detection)
|
||||
- `search(queryVector, k, type?, filter?, options?): Promise<Array<[string, number]>>`
|
||||
|
||||
```typescript
|
||||
context.registerProvider('vector', (config, distanceFn, options) => {
|
||||
return new MyNativeVectorIndex(config, distanceFn, options)
|
||||
})
|
||||
```
|
||||
|
||||
#### The readiness contract (all three index providers)
|
||||
|
||||
A provider that **persists its derived index** should implement the optional readiness
|
||||
members so a warm reopen never pays a redundant rebuild-from-canonical:
|
||||
|
||||
- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
|
||||
`brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
|
||||
and **before the rebuild gate**.
|
||||
- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
|
||||
loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
|
||||
exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
|
||||
`totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
|
||||
while fully durable. Never return `true` if the durable state failed to load: the
|
||||
signal is honest in both directions, and a not-ready provider gets its rebuild even
|
||||
when `size() > 0`.
|
||||
- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
|
||||
migration); brainy skips its rebuild entirely.
|
||||
|
||||
Providers that implement none of these keep the size/count heuristics — correct for
|
||||
engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
|
||||
|
||||
#### `metadataIndex`
|
||||
**Type:** `(storage: StorageAdapter) => MetadataIndexManager-compatible`
|
||||
|
||||
Factory function that creates a metadata index. The returned object must implement the `MetadataIndexManager` interface including `init()`, `addEntity()`, `removeEntity()`, `query()`, `flush()`, `clear()`, etc.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('metadataIndex', (storage) => {
|
||||
return new MyNativeMetadataIndex(storage)
|
||||
})
|
||||
```
|
||||
|
||||
#### `graphIndex`
|
||||
**Type:** `(storage: StorageAdapter) => GraphAdjacencyIndex-compatible`
|
||||
|
||||
Factory function that creates a graph adjacency index for relationship tracking (verbs/triples). Must implement the `GraphAdjacencyIndex` interface including `addVerb()`, `getVerbsBySource()`, `getVerbsByTarget()`, `flush()`, etc.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('graphIndex', (storage) => {
|
||||
return new MyNativeGraphIndex(storage)
|
||||
})
|
||||
```
|
||||
|
||||
#### `aggregation`
|
||||
**Type:** `(storage: StorageAdapter) => AggregationProvider-compatible`
|
||||
|
||||
Factory function that creates an aggregation engine for write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows. The returned object must implement the `AggregationProvider` interface.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('aggregation', (storage) => {
|
||||
return new MyNativeAggregationEngine(storage)
|
||||
})
|
||||
```
|
||||
|
||||
When provided by an optional native acceleration plugin (such as `@soulcraft/cor`), this enables:
|
||||
- Compiled source filters (vs per-entity JS object traversal)
|
||||
- Precise MIN/MAX via sorted data structures (vs lazy recompute)
|
||||
- Parallel aggregate rebuild across CPU cores
|
||||
- SIMD-accelerated timestamp bucketing
|
||||
|
||||
### Utility Providers
|
||||
|
||||
#### `cache`
|
||||
**Type:** `UnifiedCache`
|
||||
|
||||
Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`).
|
||||
|
||||
```typescript
|
||||
import type { UnifiedCache } from '@soulcraft/brainy/internals'
|
||||
|
||||
context.registerProvider('cache', myNativeCache)
|
||||
```
|
||||
|
||||
#### `entityIdMapper`
|
||||
**Type:** `(storage: StorageAdapter) => EntityIdMapper-compatible`
|
||||
|
||||
Factory for bidirectional UUID ↔ integer mapping used by roaring bitmaps. Must implement `getOrAssign()`, `getUuid()`, `getInt()`, `has()`, `remove()`, `flush()`, `clear()`.
|
||||
|
||||
#### `roaring`
|
||||
**Type:** `RoaringBitmap32 class`
|
||||
|
||||
Replacement for the roaring bitmap implementation. Used internally by the metadata index for set operations. Must be API-compatible with `roaring-wasm`.
|
||||
|
||||
#### `msgpack`
|
||||
**Type:** `{ encode: (data: any) => Buffer, decode: (buffer: Buffer) => any }`
|
||||
|
||||
Native msgpack encode/decode for SSTable serialization.
|
||||
|
||||
### Analytics Providers (Native-Only)
|
||||
|
||||
These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cor`) is installed.
|
||||
|
||||
Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it.
|
||||
|
||||
#### `analytics:hyperloglog`
|
||||
Approximate distinct counts. Count unique values (e.g., unique merchants) across millions of records using ~16KB of memory with ~1% error. Each update is O(1).
|
||||
|
||||
#### `analytics:tdigest`
|
||||
Streaming percentiles. Compute P50/P90/P95/P99 from streaming data without storing all values. Uses ~4KB per digest with ~1% accuracy at the tails.
|
||||
|
||||
#### `analytics:countmin`
|
||||
Frequency estimation. Find the most common values (e.g., top-K merchants) using ~40KB with 0.1% error. O(1) per update.
|
||||
|
||||
#### `analytics:anomaly`
|
||||
Real-time anomaly detection. Flag statistically unusual values at write-time using exponentially weighted moving averages. 64 bytes per group, sub-microsecond decisions.
|
||||
|
||||
#### `aggregation:mmap`
|
||||
Persistent aggregate storage via memory-mapped files. Aggregate state survives process crashes without explicit flush. Zero serialization overhead.
|
||||
|
||||
---
|
||||
|
||||
## Storage Adapter Plugins
|
||||
|
||||
Plugins can register custom storage backends that users reference by name.
|
||||
|
||||
### Implementing a Storage Adapter
|
||||
|
||||
```typescript
|
||||
import type { StorageAdapterFactory } from '@soulcraft/brainy/plugin'
|
||||
import type { StorageAdapter } from '@soulcraft/brainy'
|
||||
|
||||
class MyStorageAdapter implements StorageAdapter {
|
||||
async init(): Promise<void> { /* ... */ }
|
||||
async saveNoun(noun: HNSWNoun): Promise<void> { /* ... */ }
|
||||
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> { /* ... */ }
|
||||
async deleteNoun(id: string): Promise<void> { /* ... */ }
|
||||
// ... implement all StorageAdapter methods
|
||||
}
|
||||
```
|
||||
|
||||
### Registering a Storage Adapter
|
||||
|
||||
```typescript
|
||||
context.registerProvider('storage:my-backend', {
|
||||
name: 'my-backend',
|
||||
create: (config: Record<string, unknown>) => {
|
||||
return new MyStorageAdapter(config)
|
||||
}
|
||||
} satisfies StorageAdapterFactory)
|
||||
```
|
||||
|
||||
Users can then use your storage:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: 'my-backend', myBackendOption: 'value' })
|
||||
```
|
||||
|
||||
## Import Paths
|
||||
|
||||
Brainy provides three entry points for plugin developers:
|
||||
|
||||
| Import Path | Contents | Stability |
|
||||
|-------------|----------|-----------|
|
||||
| `@soulcraft/brainy` | Public API, types, StorageAdapter | Stable (semver) |
|
||||
| `@soulcraft/brainy/plugin` | BrainyPlugin, BrainyPluginContext, StorageAdapterFactory | Stable (semver) |
|
||||
| `@soulcraft/brainy/internals` | UnifiedCache, EntityIdMapper, logger utilities | Internal (may change between minor versions) |
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Brainy provides a `diagnostics()` method to verify plugin wiring:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
const diag = brain.diagnostics()
|
||||
console.log(diag)
|
||||
// {
|
||||
// version: '7.14.0',
|
||||
// plugins: { active: ['my-plugin'], count: 1 },
|
||||
// providers: {
|
||||
// metadataIndex: { source: 'default' },
|
||||
// graphIndex: { source: 'default' },
|
||||
// embeddings: { source: 'plugin' },
|
||||
// embedBatch: { source: 'plugin' },
|
||||
// distance: { source: 'plugin' },
|
||||
// vector: { source: 'default' },
|
||||
// ...
|
||||
// },
|
||||
// indexes: {
|
||||
// vector: { size: 0, type: 'JsHnswVectorIndex' },
|
||||
// metadata: { type: 'MetadataIndexManager', initialized: true },
|
||||
// graph: { type: 'GraphAdjacencyIndex', initialized: true, wiredToStorage: true }
|
||||
// }
|
||||
// }
|
||||
```
|
||||
|
||||
The CLI also supports diagnostics:
|
||||
|
||||
```bash
|
||||
brainy diagnostics
|
||||
```
|
||||
|
||||
### Init-Time Summary
|
||||
|
||||
When a plugin is active, brainy automatically logs a provider summary after `init()`:
|
||||
|
||||
```
|
||||
[brainy] Plugin activated: @soulcraft/cor
|
||||
[brainy] Providers: 8/10 native (@soulcraft/cor) | default: vector, cache
|
||||
```
|
||||
|
||||
This tells you at a glance how many subsystems are accelerated and which ones are falling back to JavaScript. The log respects `config.silent`.
|
||||
|
||||
### Fail-Fast for Production
|
||||
|
||||
Use `requireProviders()` after `init()` to guarantee specific providers are plugin-supplied. This prevents silent fallback to JavaScript in deployments where you expect native acceleration:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Throws immediately if any of these are using JS fallback
|
||||
brain.requireProviders(['distance', 'embeddings', 'metadataIndex', 'graphIndex'])
|
||||
```
|
||||
|
||||
If a required provider is missing, the error message tells you exactly what's wrong:
|
||||
|
||||
```
|
||||
[brainy] Required providers using JS fallback: graphIndex.
|
||||
Active plugins: @soulcraft/cor.
|
||||
These providers must be supplied by a plugin for this deployment.
|
||||
Check plugin installation, license, and native module availability.
|
||||
```
|
||||
|
||||
This is the recommended pattern for production deployments with paid plugins — fail at startup rather than silently degrading performance.
|
||||
|
||||
## Complete Example: Distance Acceleration Plugin
|
||||
|
||||
A minimal but useful plugin that provides SIMD-accelerated distance calculations:
|
||||
|
||||
```typescript
|
||||
// simd-distance-plugin/src/plugin.ts
|
||||
import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
|
||||
|
||||
// Hypothetical native module
|
||||
import { simdCosineDistance } from './native.js'
|
||||
|
||||
const simdDistancePlugin: BrainyPlugin = {
|
||||
name: 'brainy-simd-distance',
|
||||
|
||||
async activate(context: BrainyPluginContext): Promise<boolean> {
|
||||
// Check if SIMD is available on this platform
|
||||
if (!checkSimdSupport()) {
|
||||
console.log('[simd-distance] SIMD not available, skipping')
|
||||
return false // Don't activate — brainy uses JS fallback
|
||||
}
|
||||
|
||||
context.registerProvider('distance', simdCosineDistance)
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
export default simdDistancePlugin
|
||||
```
|
||||
|
||||
```json
|
||||
// simd-distance-plugin/package.json
|
||||
{
|
||||
"name": "brainy-simd-distance",
|
||||
"main": "./dist/plugin.js",
|
||||
"types": "./dist/plugin.d.ts",
|
||||
"peerDependencies": {
|
||||
"@soulcraft/brainy": ">=7.0.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({ plugins: ['brainy-simd-distance'] })
|
||||
await brain.init()
|
||||
|
||||
// Verify it's active
|
||||
const diag = brain.diagnostics()
|
||||
console.log(diag.providers.distance) // { source: 'plugin' }
|
||||
```
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Brainy works perfectly without plugins.** Every provider has a JavaScript fallback. Plugins only improve performance or add capabilities.
|
||||
|
||||
2. **Provider keys are string-based.** The plugin system is not coupled to any specific plugin. Any package can register any provider.
|
||||
|
||||
3. **Clean separation.** Plugins access brainy through the documented `BrainyPluginContext` interface. No direct access to internal classes is needed.
|
||||
|
||||
4. **Fail-safe activation.** If a plugin throws during `activate()`, brainy logs a warning and continues with defaults. A broken plugin never prevents brainy from working.
|
||||
|
||||
5. **Lifecycle management.** `deactivate()` is called during `brainy.close()` for resource cleanup. Native resources, connections, and file handles should be released here.
|
||||
|
|
@ -215,9 +215,9 @@ console.log('Server running on http://localhost:3000')
|
|||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ Native Bun runtime performance
|
||||
- ✅ Native Bun performance (~2x faster than Node.js)
|
||||
- ✅ No framework dependencies
|
||||
- ✅ Pure WASM — no native binaries, bundler-friendly
|
||||
- ✅ Works with `bun --compile` for single-binary deployment
|
||||
- ✅ Built-in TypeScript support
|
||||
|
||||
### Pattern 4: Express/Node.js Middleware (Legacy)
|
||||
|
|
|
|||
|
|
@ -1,298 +0,0 @@
|
|||
# Query Operators (BFO)
|
||||
|
||||
> Brainy Field Operators — the complete reference for `where` filters in `find()`.
|
||||
|
||||
All operators work with `find({ where: { ... } })` and filter on **metadata fields** (not `data`).
|
||||
|
||||
---
|
||||
|
||||
## Equality
|
||||
|
||||
| Operator | Alias | Description | Example |
|
||||
|----------|-------|-------------|---------|
|
||||
| `eq` | `equals` | Exact match | `{ status: { eq: 'active' } }` |
|
||||
| `ne` | `notEquals` | Not equal | `{ status: { ne: 'deleted' } }` |
|
||||
|
||||
**Shorthand:** A bare value is treated as `equals`:
|
||||
|
||||
```typescript
|
||||
// These are equivalent:
|
||||
brain.find({ where: { status: 'active' } })
|
||||
brain.find({ where: { status: { equals: 'active' } } })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Comparison
|
||||
|
||||
| Operator | Alias | Description | Example |
|
||||
|----------|-------|-------------|---------|
|
||||
| `gt` | `greaterThan` | Greater than | `{ age: { gt: 18 } }` |
|
||||
| `gte` | `greaterThanOrEqual` | Greater or equal | `{ score: { gte: 90 } }` |
|
||||
| `lt` | `lessThan` | Less than | `{ price: { lt: 100 } }` |
|
||||
| `lte` | `lessThanOrEqual` | Less or equal | `{ rating: { lte: 3 } }` |
|
||||
| `between` | — | Inclusive range `[min, max]` | `{ year: { between: [2020, 2025] } }` |
|
||||
|
||||
```typescript
|
||||
// Range query
|
||||
const recent = await brain.find({
|
||||
where: {
|
||||
createdAt: { between: [Date.now() - 86400000, Date.now()] }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Array / Set
|
||||
|
||||
| Operator | Alias | Description | Example |
|
||||
|----------|-------|-------------|---------|
|
||||
| `oneOf` | `in` | Value is one of the given options | `{ color: { oneOf: ['red', 'blue'] } }` |
|
||||
| `noneOf` | — | Value is NOT one of the given options | `{ status: { noneOf: ['deleted', 'archived'] } }` |
|
||||
| `contains` | — | Array field contains value | `{ tags: { contains: 'ai' } }` |
|
||||
| `excludes` | — | Array field does NOT contain value | `{ tags: { excludes: 'spam' } }` |
|
||||
| `hasAll` | — | Array field contains ALL listed values | `{ skills: { hasAll: ['js', 'ts'] } }` |
|
||||
|
||||
```typescript
|
||||
// Find entities tagged with 'ai'
|
||||
const aiEntities = await brain.find({
|
||||
where: { tags: { contains: 'ai' } }
|
||||
})
|
||||
|
||||
// Find entities of specific types
|
||||
const people = await brain.find({
|
||||
where: { noun: { oneOf: ['Person', 'Agent'] } }
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Existence
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `exists: true` | Field exists (has any value) | `{ email: { exists: true } }` |
|
||||
| `exists: false` | Field does NOT exist | `{ email: { exists: false } }` |
|
||||
| `missing: true` | Field does NOT exist (alias for `exists: false`) | `{ email: { missing: true } }` |
|
||||
| `missing: false` | Field exists (alias for `exists: true`) | `{ email: { missing: false } }` |
|
||||
|
||||
```typescript
|
||||
// Find entities that have an email field
|
||||
const withEmail = await brain.find({
|
||||
where: { email: { exists: true } }
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pattern (In-Memory Only)
|
||||
|
||||
These operators work via the in-memory filter path. They are applied **after** the indexed query, so use them with other indexed operators for best performance.
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `matches` | Regex or string pattern match | `{ name: { matches: /^Dr\./ } }` |
|
||||
| `startsWith` | String prefix | `{ name: { startsWith: 'John' } }` |
|
||||
| `endsWith` | String suffix | `{ email: { endsWith: '@gmail.com' } }` |
|
||||
|
||||
```typescript
|
||||
const doctors = await brain.find({
|
||||
where: {
|
||||
type: NounType.Person, // Indexed — fast
|
||||
name: { startsWith: 'Dr.' } // In-memory — applied after
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Logical
|
||||
|
||||
Combine multiple conditions:
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `allOf` | ALL sub-filters must match (AND) | `{ allOf: [{ status: 'active' }, { role: 'admin' }] }` |
|
||||
| `anyOf` | ANY sub-filter must match (OR) | `{ anyOf: [{ role: 'admin' }, { role: 'owner' }] }` |
|
||||
| `not` | Invert a filter | `{ not: { status: 'deleted' } }` |
|
||||
|
||||
```typescript
|
||||
// Complex OR query
|
||||
const adminsOrOwners = await brain.find({
|
||||
where: {
|
||||
anyOf: [
|
||||
{ role: 'admin' },
|
||||
{ role: 'owner' }
|
||||
]
|
||||
}
|
||||
})
|
||||
|
||||
// NOT query
|
||||
const notDeleted = await brain.find({
|
||||
where: {
|
||||
not: { status: 'deleted' }
|
||||
}
|
||||
})
|
||||
|
||||
// Combined AND + OR
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
allOf: [
|
||||
{ department: 'engineering' },
|
||||
{ anyOf: [
|
||||
{ level: 'senior' },
|
||||
{ yearsExperience: { greaterThan: 5 } }
|
||||
]}
|
||||
]
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Indexed vs In-Memory Operators
|
||||
|
||||
Brainy's MetadataIndex supports a subset of operators natively for O(1) field lookups. Other operators fall back to in-memory filtering.
|
||||
|
||||
| Operator | MetadataIndex (Indexed) | In-Memory Fallback |
|
||||
|----------|:-----------------------:|:------------------:|
|
||||
| `equals` / `eq` | Yes | Yes |
|
||||
| `notEquals` / `ne` | — | Yes |
|
||||
| `greaterThan` / `gt` | Yes | Yes |
|
||||
| `greaterThanOrEqual` / `gte` | Yes | Yes |
|
||||
| `lessThan` / `lt` | Yes | Yes |
|
||||
| `lessThanOrEqual` / `lte` | Yes | Yes |
|
||||
| `between` | Yes | Yes |
|
||||
| `oneOf` / `in` | Yes | Yes |
|
||||
| `noneOf` | — | Yes |
|
||||
| `contains` | Yes | Yes |
|
||||
| `exists` / `missing` | Yes | Yes |
|
||||
| `matches` | — | Yes |
|
||||
| `startsWith` | — | Yes |
|
||||
| `endsWith` | — | Yes |
|
||||
| `allOf` | Partial | Yes |
|
||||
| `anyOf` | Partial | Yes |
|
||||
| `not` | — | Yes |
|
||||
|
||||
**Performance tip:** Combine indexed operators (equals, greaterThan, oneOf, between, contains, exists) with pattern operators for optimal speed — the index narrows results first, then patterns filter in memory.
|
||||
|
||||
---
|
||||
|
||||
## Practical Examples
|
||||
|
||||
### Filter by entity type
|
||||
|
||||
```typescript
|
||||
// Using the type shorthand (recommended)
|
||||
brain.find({ type: NounType.Person })
|
||||
|
||||
// Using where.noun directly
|
||||
brain.find({ where: { noun: NounType.Person } })
|
||||
|
||||
// Multiple types
|
||||
brain.find({ type: [NounType.Person, NounType.Agent] })
|
||||
```
|
||||
|
||||
### Filter by subtype
|
||||
|
||||
`subtype` is a top-level standard field — takes the column-store fast path, not the metadata fallback. Pair with `type` for the typical "Person who is an employee" query:
|
||||
|
||||
```typescript
|
||||
// Equality on subtype:
|
||||
brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// Set membership:
|
||||
brain.find({ type: NounType.Person, subtype: ['employee', 'contractor'] })
|
||||
|
||||
// Operator-form predicates use `where`:
|
||||
brain.find({
|
||||
type: NounType.Person,
|
||||
where: { subtype: { exists: true } }
|
||||
})
|
||||
```
|
||||
|
||||
See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the full surface.
|
||||
|
||||
### Filter relationships by subtype (7.30+)
|
||||
|
||||
Verbs are first-class peers — `related()` and graph traversal both honor subtype filters on the fast path:
|
||||
|
||||
```typescript
|
||||
// Filter relationships by VerbType subtype
|
||||
const direct = await brain.related({
|
||||
from: ceoId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'direct'
|
||||
})
|
||||
|
||||
// Set membership on verb subtype
|
||||
const all = await brain.related({
|
||||
from: ceoId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: ['direct', 'dotted-line']
|
||||
})
|
||||
|
||||
// Graph traversal — subtype filters traversal edges (depth-1 in 7.30 JS path;
|
||||
// multi-hop subtype filtering lands on Cor native)
|
||||
const reports = await brain.find({
|
||||
connected: {
|
||||
from: ceoId,
|
||||
via: VerbType.ReportsTo,
|
||||
subtype: 'direct',
|
||||
depth: 1
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Combine semantic search with filters
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
query: 'machine learning engineer', // Semantic search (on data)
|
||||
type: NounType.Person, // Type filter (indexed)
|
||||
where: {
|
||||
department: 'engineering', // Exact match (indexed)
|
||||
yearsExperience: { greaterThan: 3 } // Range filter (indexed)
|
||||
},
|
||||
limit: 10
|
||||
})
|
||||
```
|
||||
|
||||
### Temporal queries
|
||||
|
||||
```typescript
|
||||
const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000
|
||||
const recentEntities = await brain.find({
|
||||
where: {
|
||||
createdAt: { greaterThan: lastWeek }
|
||||
},
|
||||
orderBy: 'createdAt',
|
||||
order: 'desc',
|
||||
limit: 50
|
||||
})
|
||||
```
|
||||
|
||||
### Graph + metadata combination
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
connected: {
|
||||
from: teamLeadId,
|
||||
via: VerbType.WorksWith,
|
||||
depth: 2
|
||||
},
|
||||
where: {
|
||||
role: { oneOf: ['engineer', 'designer'] },
|
||||
active: true
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Data Model](./DATA_MODEL.md) — Entity structure, data vs metadata
|
||||
- [API Reference](./api/README.md) — Complete API documentation
|
||||
- [Find System](./FIND_SYSTEM.md) — Natural language find() details
|
||||
356
docs/README.md
356
docs/README.md
|
|
@ -1,130 +1,332 @@
|
|||
# Brainy Documentation
|
||||
|
||||
> The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API.
|
||||
Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
|
||||
|
||||
## Quick Start
|
||||
## 🆕 What's New in
|
||||
|
||||
**Production-Ready Cost Optimization:**
|
||||
- **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!)
|
||||
- **Intelligent-Tiering**: S3 Intelligent-Tiering and GCS Autoclass support
|
||||
- **Batch Operations**: Efficient bulk delete operations (1000 objects per request)
|
||||
- **Compression**: Gzip compression for FileSystem storage (60-80% space savings)
|
||||
- **Quota Monitoring**: Real-time OPFS quota tracking for browser apps
|
||||
- **Tier Management**: Azure Hot/Cool/Archive tier management
|
||||
|
||||
**Cost Impact Example (500TB dataset):**
|
||||
- Before: $138,000/year
|
||||
- After $5,940/year
|
||||
- **Savings: $132,060/year (96%)**
|
||||
|
||||
## 📊 Implementation Status
|
||||
|
||||
- ✅ **Production Ready**: Core features working today
|
||||
- 🚧 **In Development**: Features coming soon
|
||||
- 📅 **Roadmap**: See [ROADMAP.md](../ROADMAP.md)
|
||||
|
||||
## Quick Links
|
||||
|
||||
### Getting Started
|
||||
- [API Reference](./api/README.md) - Complete API documentation (start here!)
|
||||
- [Enterprise for Everyone](./guides/enterprise-for-everyone.md) - **No limits, no tiers, everything free**
|
||||
- [Natural Language Queries](./guides/natural-language.md) - Query with plain English
|
||||
|
||||
### Core Concepts
|
||||
- [Zero Configuration](./architecture/zero-config.md) - **Auto-adapts to any environment**
|
||||
- [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) - **Revolutionary data model**
|
||||
- [Triple Intelligence](./architecture/triple-intelligence.md) - Unified query system
|
||||
- [Architecture Overview](./architecture/overview.md) - System design
|
||||
|
||||
### API Documentation
|
||||
- [API Reference](./api/README.md) - Complete API documentation
|
||||
- [TypeScript Types](./api/types.md) - Type definitions
|
||||
|
||||
### Advanced Topics
|
||||
- [Augmentations System](./architecture/augmentations.md) - **Enterprise plugins & neural import**
|
||||
- [Storage Architecture](./architecture/storage.md) - Storage adapter system
|
||||
- [Performance Tuning](./guides/performance.md) - Optimization guide
|
||||
- [Migration Guide](../MIGRATION.md) - Upgrading from 1.x
|
||||
|
||||
## What is Brainy?
|
||||
|
||||
Brainy is a next-generation AI database that combines:
|
||||
- **Vector Search**: Semantic similarity using HNSW indexing
|
||||
- **Graph Relationships**: Complex relationship mapping and traversal
|
||||
- **Field Filtering**: Precise metadata filtering with O(1) lookups
|
||||
- **Natural Language**: Query in plain English
|
||||
|
||||
## Key Features
|
||||
|
||||
### 🧠 Triple Intelligence Engine
|
||||
All three intelligence types (vector, graph, field) work together in every query for optimal results.
|
||||
|
||||
### 📝 Noun-Verb Taxonomy
|
||||
Model your data naturally as entities (nouns) and relationships (verbs) - no complex schemas needed.
|
||||
|
||||
### 🌍 Natural Language Queries
|
||||
Ask questions in plain English and Brainy understands your intent:
|
||||
```typescript
|
||||
await brain.find("recent articles about AI with high ratings")
|
||||
```
|
||||
|
||||
### ⚡ Production Ready
|
||||
- Universal storage (FileSystem, S3, OPFS, Memory)
|
||||
- Zero configuration with intelligent defaults
|
||||
- Full TypeScript support
|
||||
- Cross-platform compatibility
|
||||
|
||||
## Quick Example
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
// Initialize
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Add entities — data is embedded for semantic search, metadata is indexed for filtering
|
||||
const id = await brain.add({
|
||||
data: 'Revolutionary AI Breakthrough',
|
||||
type: NounType.Document,
|
||||
metadata: { category: 'technology', rating: 4.8 }
|
||||
// Add entities (nouns)
|
||||
const articleId = await brain.add({
|
||||
data: "Revolutionary AI Breakthrough",
|
||||
type: NounType.Document,
|
||||
metadata: { category: "technology", rating: 4.8 }
|
||||
})
|
||||
|
||||
// Search with Triple Intelligence
|
||||
const results = await brain.find({
|
||||
query: 'artificial intelligence', // Semantic search (on data)
|
||||
where: { rating: { greaterThan: 4.0 } }, // Metadata filter
|
||||
connected: { from: authorId, depth: 2 } // Graph traversal
|
||||
const authorId = await brain.add({
|
||||
data: "Dr. Sarah Chen",
|
||||
type: NounType.Person,
|
||||
metadata: { role: "researcher" }
|
||||
})
|
||||
|
||||
// Create relationships (verbs)
|
||||
await brain.relate({
|
||||
from: authorId,
|
||||
to: articleId,
|
||||
type: VerbType.CreatedBy,
|
||||
metadata: { date: "2024-01-15", contribution: "primary" }
|
||||
})
|
||||
|
||||
// Query naturally
|
||||
const results = await brain.find("highly rated technology articles by researchers")
|
||||
```
|
||||
|
||||
---
|
||||
## 📚 Complete Documentation Index
|
||||
|
||||
## Core Documentation
|
||||
### 🚀 Quick Start Guides
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| **[API Reference](./api/README.md)** | Complete API documentation — **start here** |
|
||||
| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
|
||||
| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
|
||||
| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
|
||||
| [Consistency Model](./concepts/consistency-model.md) | The Db API guarantees — snapshot isolation, atomic transactions, time travel |
|
||||
| [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples |
|
||||
| [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds |
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
### 🆕 Migration & Optimization
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Architecture Overview](./architecture/overview.md) | High-level system design |
|
||||
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Metadata unified query |
|
||||
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | 42 nouns + 127 verbs type system |
|
||||
| [Stage 3 Canonical Taxonomy](./STAGE3-CANONICAL-TAXONOMY.md) | Complete type reference |
|
||||
| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
|
||||
| [Index Architecture](./architecture/index-architecture.md) | Vector, Graph, and Metadata indexing |
|
||||
| [v3→v4 Migration Guide](./MIGRATION-V3-TO-V4.md) | **NEW** - Upgrade guide with zero breaking changes |
|
||||
| [AWS S3 Cost Optimization](./operations/cost-optimization-aws-s3.md) | **NEW** - 96% cost savings with lifecycle policies |
|
||||
| [GCS Cost Optimization](./operations/cost-optimization-gcs.md) | **NEW** - 94% savings with Autoclass |
|
||||
| [Azure Cost Optimization](./operations/cost-optimization-azure.md) | **NEW** - 95% savings with tier management |
|
||||
| [Cloudflare R2 Cost Guide](./operations/cost-optimization-cloudflare-r2.md) | **NEW** - Zero egress fees + S3-compatible API |
|
||||
|
||||
### 🎯 Core Concepts
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Architecture Overview](./architecture/overview.md) | High-level system design and components |
|
||||
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships |
|
||||
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system |
|
||||
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
|
||||
| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
|
||||
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | Metadata/vector separation, sharding |
|
||||
| [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing |
|
||||
|
||||
---
|
||||
|
||||
## Virtual Filesystem (VFS)
|
||||
### 💾 Storage & Deployment
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [VFS Quick Start](./vfs/QUICK_START.md) | Get started in 30 seconds |
|
||||
| [VFS Core](./vfs/VFS_CORE.md) | Core concepts and architecture |
|
||||
| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
|
||||
| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
|
||||
|
||||
See [vfs/](./vfs/) for the complete VFS documentation set.
|
||||
|
||||
---
|
||||
|
||||
## Guides
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
|
||||
| [Snapshots & Time Travel](./guides/snapshots-and-time-travel.md) | Backups, restore, what-if analysis, audit trails |
|
||||
| [Natural Language](./guides/natural-language.md) | Query in plain English |
|
||||
| [Neural API](./guides/neural-api.md) | AI-powered features |
|
||||
| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
|
||||
| [Framework Integration](./guides/framework-integration.md) | React, Vue, Angular, Svelte |
|
||||
|
||||
---
|
||||
|
||||
## Storage & Deployment
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Storage Architecture](./architecture/storage-architecture.md) | Filesystem and memory adapters, on-disk artifact layout, operator-layer backup |
|
||||
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | Deploy on AWS, GCP, Azure, Cloudflare |
|
||||
| [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns |
|
||||
| [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment |
|
||||
| [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations |
|
||||
| [Distributed Storage](./architecture/distributed-storage.md) | Multi-node storage coordination |
|
||||
| [Extending Storage](./EXTENDING_STORAGE.md) | Create custom storage adapters |
|
||||
| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities |
|
||||
|
||||
---
|
||||
|
||||
## Plugins
|
||||
### 📊 API Documentation
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Plugins](./PLUGINS.md) | Plugin system overview — providers, `plugins` config, `brain.use()` |
|
||||
| [API Reference](./API_REFERENCE.md) | Complete API documentation |
|
||||
| [Comprehensive API Overview](./api/COMPREHENSIVE_API_OVERVIEW.md) | All APIs with examples |
|
||||
| [API Decision Tree](./API_DECISION_TREE.md) | Choose the right API for your use case |
|
||||
| [Core API Patterns](./CORE_API_PATTERNS.md) | Common patterns and best practices |
|
||||
| [Neural API Patterns](./NEURAL_API_PATTERNS.md) | AI-powered query patterns |
|
||||
| [Find System](./FIND_SYSTEM.md) | Natural language find() API |
|
||||
| [API Surface Design](./architecture/API_SURFACE_DESIGN.md) | API design principles |
|
||||
| [API Returns](./api-returns.md) | Return types and structures |
|
||||
|
||||
---
|
||||
|
||||
## Performance & Scaling
|
||||
### 🔧 Framework Integration
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Performance](./PERFORMANCE.md) | Optimization techniques |
|
||||
| [Scaling](./SCALING.md) | Scale to billions of entities |
|
||||
| [Batching](./BATCHING.md) | Batch operations guide |
|
||||
| [Framework Integration Guide](./guides/framework-integration.md) | React, Vue, Angular, Svelte, etc. |
|
||||
| [Next.js Integration](./guides/nextjs-integration.md) | Server-side rendering with Brainy |
|
||||
| [Vue.js Integration](./guides/vue-integration.md) | Vue 3 integration patterns |
|
||||
|
||||
---
|
||||
|
||||
## Migration & Reference
|
||||
### 📁 Virtual Filesystem (VFS)
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [v3 to v4 Migration](./MIGRATION-V3-TO-V4.md) | Upgrade guide |
|
||||
| [VFS Core Documentation](./vfs/VFS_CORE.md) | Core VFS concepts and architecture |
|
||||
| [Semantic VFS Guide](./vfs/SEMANTIC_VFS.md) | Semantic filesystem projections |
|
||||
| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
|
||||
| [Neural Extraction](./vfs/NEURAL_EXTRACTION.md) | AI-powered file analysis |
|
||||
| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
|
||||
| [User Functions](./vfs/USER_FUNCTIONS.md) | Custom VFS functions |
|
||||
| [VFS Graph Types](./vfs/VFS_GRAPH_TYPES.md) | Graph-based VFS projections |
|
||||
| [VFS Examples & Scenarios](./vfs/VFS_EXAMPLES_SCENARIOS.md) | Real-world VFS examples |
|
||||
| [VFS Initialization](./vfs/VFS_INITIALIZATION.md) | Setup and configuration |
|
||||
| [Projection Strategy API](./vfs/PROJECTION_STRATEGY_API.md) | Custom projection strategies |
|
||||
| [Building File Explorers](./vfs/building-file-explorers.md) | Build custom file browsers |
|
||||
| [VFS Troubleshooting](./vfs/TROUBLESHOOTING.md) | Common issues and solutions |
|
||||
|
||||
### 🧠 Advanced Topics
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Natural Language Queries](./guides/natural-language.md) | Query in plain English |
|
||||
| [Neural API](./guides/neural-api.md) | AI-powered features |
|
||||
| [Import Anything](./guides/import-anything.md) | Import data from any source |
|
||||
| [Distributed Systems](./guides/distributed-system.md) | Multi-node deployments |
|
||||
| [Model Loading](./guides/model-loading.md) | Load and manage AI models |
|
||||
| [Model Loading Quick Reference](./MODEL_LOADING_QUICK_REFERENCE.md) | Model loading cheat sheet |
|
||||
| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No paywalls, no tiers |
|
||||
|
||||
### 🔌 Augmentations (Plugins)
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Creating Augmentations](./CREATING-AUGMENTATIONS.md) | Build custom plugins |
|
||||
| [Augmentations Complete Reference](./augmentations/COMPLETE-REFERENCE.md) | Full augmentation API |
|
||||
| [Augmentations Developer Guide](./augmentations/DEVELOPER-GUIDE.md) | Plugin development guide |
|
||||
| [Augmentation Configuration](./augmentations/CONFIGURATION.md) | Configure augmentations |
|
||||
| [Augmentation System](./architecture/augmentations.md) | System architecture |
|
||||
| [Augmentation System Audit](./architecture/augmentation-system-audit.md) | Actual vs planned features |
|
||||
| [Augmentations Actual](./architecture/augmentations-actual.md) | Currently implemented augmentations |
|
||||
| [API Server Augmentation](./augmentations/api-server.md) | REST API server plugin |
|
||||
|
||||
### ⚡ Performance & Scaling
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Performance Guide](./PERFORMANCE.md) | Optimization techniques |
|
||||
| [Performance Analysis](./architecture/PERFORMANCE_ANALYSIS.md) | Benchmarks and analysis |
|
||||
| [Scaling Guide](./SCALING.md) | Scale to billions of entities |
|
||||
| [Clustering Algorithms Analysis](./architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md) | HNSW algorithm details |
|
||||
| [Initialization & Rebuild](./architecture/initialization-and-rebuild.md) | Index management |
|
||||
|
||||
### 📋 Reference
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Complete Feature List](./features/complete-feature-list.md) | All Brainy features |
|
||||
| [v3 Features](./features/v3-features.md) | v3.x feature list |
|
||||
| [Metadata Architecture](./architecture/METADATA_ARCHITECTURE.md) | Metadata namespacing |
|
||||
| [Metadata Contract Implementation](./METADATA_CONTRACT_IMPLEMENTATION.md) | Metadata API contracts |
|
||||
| [Finite Type System](./architecture/finite-type-system.md) | Type-safe noun/verb taxonomy |
|
||||
| [Validation](./VALIDATION.md) | Data validation rules |
|
||||
| [Universal Display Augmentation](./universal-display-augmentation.md) | Display system |
|
||||
|
||||
### 🛠️ Development
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Troubleshooting](./troubleshooting.md) | Common issues and solutions |
|
||||
| [Release Guide](./RELEASE-GUIDE.md) | How to release new versions |
|
||||
| [Production Architecture](./PRODUCTION_SERVICE_ARCHITECTURE.md) | Ops reference |
|
||||
|
||||
---
|
||||
|
||||
## Internal
|
||||
### 🔍 Internal Documentation
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Audit Report](./internal/AUDIT_REPORT.md) | Feature audit |
|
||||
| [Cleanup Summary](./internal/CLEANUP_SUMMARY.md) | Codebase cleanup notes |
|
||||
| [Honest Status](./internal/HONEST_STATUS.md) | Actual implementation status |
|
||||
|
||||
---
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
```
|
||||
docs/
|
||||
├── README.md (this file) # Complete documentation index
|
||||
├── MIGRATION-V3-TO-V4.md # Migration guide
|
||||
│
|
||||
├── guides/ # User guides
|
||||
│ ├── natural-language.md
|
||||
│ ├── neural-api.md
|
||||
│ ├── import-anything.md
|
||||
│ ├── framework-integration.md
|
||||
│ ├── nextjs-integration.md
|
||||
│ ├── vue-integration.md
|
||||
│ ├── distributed-system.md
|
||||
│ ├── model-loading.md
|
||||
│ └── enterprise-for-everyone.md
|
||||
│
|
||||
├── architecture/ # System architecture
|
||||
│ ├── overview.md
|
||||
│ ├── noun-verb-taxonomy.md
|
||||
│ ├── triple-intelligence.md
|
||||
│ ├── zero-config.md
|
||||
│ ├── storage-architecture.md│ ├── data-storage-architecture.md│ ├── index-architecture.md
|
||||
│ ├── distributed-storage.md
|
||||
│ ├── augmentations.md
|
||||
│ ├── augmentation-system-audit.md
|
||||
│ ├── augmentations-actual.md
|
||||
│ ├── finite-type-system.md
|
||||
│ └── ...
|
||||
│
|
||||
├── operations/ # Operations guides
|
||||
│ ├── cost-optimization-aws-s3.md│ ├── cost-optimization-gcs.md│ ├── cost-optimization-azure.md│ ├── cost-optimization-cloudflare-r2.md│ └── capacity-planning.md
|
||||
│
|
||||
├── deployment/ # Deployment guides
|
||||
│ ├── CLOUD_DEPLOYMENT_GUIDE.md│ ├── aws-deployment.md
|
||||
│ ├── gcp-deployment.md
|
||||
│ └── kubernetes-deployment.md
|
||||
│
|
||||
├── vfs/ # Virtual Filesystem docs
|
||||
│ ├── QUICK_START.md
|
||||
│ ├── VFS_CORE.md
|
||||
│ ├── SEMANTIC_VFS.md
|
||||
│ ├── VFS_API_GUIDE.md
|
||||
│ ├── NEURAL_EXTRACTION.md
|
||||
│ ├── COMMON_PATTERNS.md
|
||||
│ └── ...
|
||||
│
|
||||
├── api/ # API documentation
|
||||
│ ├── README.md
|
||||
│ └── COMPREHENSIVE_API_OVERVIEW.md
|
||||
│
|
||||
├── augmentations/ # Augmentation docs
|
||||
│ ├── COMPLETE-REFERENCE.md
|
||||
│ ├── DEVELOPER-GUIDE.md
|
||||
│ ├── CONFIGURATION.md
|
||||
│ └── api-server.md
|
||||
│
|
||||
├── features/ # Feature documentation
|
||||
│ ├── complete-feature-list.md
|
||||
│ └── v3-features.md
|
||||
│
|
||||
└── internal/ # Internal docs
|
||||
├── AUDIT_REPORT.md
|
||||
├── CLEANUP_SUMMARY.md
|
||||
└── HONEST_STATUS.md
|
||||
```
|
||||
|
||||
## Community
|
||||
|
||||
- **GitHub**: [github.com/brainy-org/brainy](https://github.com/brainy-org/brainy)
|
||||
- **Issues**: [Report bugs or request features](https://github.com/brainy-org/brainy/issues)
|
||||
- **Discussions**: [Join the conversation](https://github.com/brainy-org/brainy/discussions)
|
||||
|
||||
## License
|
||||
|
||||
Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
|
||||
Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
|
||||
|
|
@ -119,13 +119,4 @@ gh release create vY.Y.Y --generate-notes
|
|||
- **Most releases should be MINOR or PATCH**
|
||||
- **Major versions should be RARE**
|
||||
- **When in doubt, it's probably MINOR**
|
||||
- **NEVER use "BREAKING CHANGE" for internal changes**
|
||||
## Hard Ordering Constraints (check before EVERY release)
|
||||
|
||||
- **Embedding model changes are SEQUENCED, not free.** No release may change the
|
||||
embedding model (or its quantization/dimensions) before **vector model-version
|
||||
stamping + hard-error-on-mismatch** ships. Stored vectors carry no model version
|
||||
today; mixing vectors from two models silently corrupts every similarity
|
||||
comparison. If a model bump is ever proposed, the stamping work moves ahead of
|
||||
it in the schedule — coordinate with the native provider so both engines stamp
|
||||
and enforce identically. (Registered with the native-provider team 2026-07-07.)
|
||||
- **NEVER use "BREAKING CHANGE" for internal changes**
|
||||
567
docs/SCALING.md
567
docs/SCALING.md
|
|
@ -1,239 +1,502 @@
|
|||
# Brainy Scaling Guide
|
||||
# 🚀 Brainy Scaling Guide - Enterprise for Everyone
|
||||
|
||||
> **One Line Summary**: Single-node by design — Brainy scales by getting the most out of one machine plus operator-layer backup.
|
||||
> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required.
|
||||
|
||||
## Table of Contents
|
||||
## 📖 Table of Contents
|
||||
- [Quick Start](#quick-start)
|
||||
- [How Brainy Scales](#how-brainy-scales)
|
||||
- [How It Works](#how-it-works)
|
||||
- [Storage Configurations](#storage-configurations)
|
||||
- [Scaling Patterns](#scaling-patterns)
|
||||
- [Real World Examples](#real-world-examples)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### In-Memory
|
||||
### Single Node (Default)
|
||||
```typescript
|
||||
import Brainy from '@soulcraft/brainy'
|
||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
||||
const brain = new Brainy() // That's it!
|
||||
```
|
||||
|
||||
### On-Disk (Default for Node)
|
||||
### Multi-Node (Auto-Discovery)
|
||||
```typescript
|
||||
// Node 1
|
||||
const brain = new Brainy() // Starts as primary
|
||||
|
||||
// Node 2 (different server)
|
||||
const brain = new Brainy() // Auto-discovers Node 1, becomes replica!
|
||||
```
|
||||
|
||||
**That's literally all you need!** Brainy handles everything else automatically.
|
||||
|
||||
## How It Works
|
||||
|
||||
### 🎯 The Magic: Zero Configuration
|
||||
|
||||
Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration:
|
||||
|
||||
1. **First node starts** → Becomes primary automatically
|
||||
2. **Second node starts** → Discovers first node via UDP broadcast
|
||||
3. **Nodes negotiate** → Elect leader, distribute shards
|
||||
4. **Data flows** → Automatic replication and routing
|
||||
5. **Node fails** → Automatic failover in <1 second
|
||||
|
||||
### 🔄 Automatic Node Discovery
|
||||
|
||||
```typescript
|
||||
// Three ways Brainy finds other nodes (auto-selected):
|
||||
|
||||
// 1. LOCAL NETWORK (Default)
|
||||
// Uses UDP broadcast on port 7946
|
||||
// Perfect for: On-premise, same VPC
|
||||
|
||||
// 2. CLOUD NATIVE (Auto-detected)
|
||||
// Kubernetes: Uses k8s DNS service discovery
|
||||
// AWS: Uses EC2 tags or Route53
|
||||
// Azure: Uses Azure DNS
|
||||
|
||||
// 3. EXPLICIT (When needed)
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './brainy-data' }
|
||||
peers: ['node1.example.com', 'node2.example.com']
|
||||
})
|
||||
```
|
||||
|
||||
## How Brainy Scales
|
||||
### 📊 Data Distribution
|
||||
|
||||
Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery, no S3 coordination. Scaling means:
|
||||
When you add data, Brainy automatically:
|
||||
|
||||
- **Up**: give the process more RAM, CPU, and IOPS
|
||||
- **Out**: stand up multiple independent Brainy instances behind your own service layer
|
||||
- **Cold storage**: snapshot the on-disk artifact off-site so you can rehydrate elsewhere
|
||||
```typescript
|
||||
brain.add({ name: "John" }, 'person')
|
||||
|
||||
The three knobs that matter most:
|
||||
|
||||
1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
|
||||
2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
|
||||
|
||||
The native vector provider (via the optional `@soulcraft/cor` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
|
||||
|
||||
## Measured Performance
|
||||
|
||||
Numbers below are **measured** by `tests/benchmarks/find-composition-scale.js` (a single
|
||||
Node 22 process, in-memory storage, 384-dim vectors, `balanced` recall). They are the
|
||||
open-core (pure-TypeScript) path — what you get from `@soulcraft/brainy` with no native
|
||||
provider installed. Run it yourself: `node --max-old-space-size=8192 tests/benchmarks/find-composition-scale.js 100000`.
|
||||
|
||||
`find()` query latency, p50 / p95 (200 queries each):
|
||||
|
||||
| Query | 5,000 entities | 100,000 entities |
|
||||
|---|---|---|
|
||||
| Vector similarity (`{ vector }`) | 0.8 / 1.3 ms | 1.4 / 4.7 ms |
|
||||
| Graph 1-hop (`{ connected }`) | 0.5 / 0.7 ms | 0.7 / 0.8 ms |
|
||||
| Metadata filter (`{ where }`, low-selectivity) | 0.7 / 1.2 ms | 23.5 / 30.1 ms |
|
||||
| Vector + metadata | 7.7 / 8.3 ms | 78.8 / 93.8 ms |
|
||||
|
||||
What the shape tells you:
|
||||
|
||||
- **Vector and graph lookups scale ~logarithmically** — they barely move from 5k to 100k,
|
||||
because HNSW search is ~O(ef·log n) and graph adjacency is O(degree).
|
||||
- **Metadata-filtered paths scale with the size of the match set, not the database.** The
|
||||
benchmark's `category` filter matches ~10% of rows (10,000 at 100k); the cost is
|
||||
materializing that candidate set and running the vector search *inside* it (`find()` does
|
||||
metadata-first hard filtering, then ranks within the candidates — see
|
||||
[How find works](./FIND_SYSTEM.md)). A **high-selectivity** filter (few matches) is far
|
||||
cheaper; a 10%-of-everything filter is the worst case. This candidate-restricted search is
|
||||
precisely the path the native provider accelerates (Rust roaring-bitmap candidate
|
||||
intersection).
|
||||
- **Composition is correct, not lossy.** Combining vector + metadata + graph returns exactly
|
||||
the entities satisfying all constraints — verified by
|
||||
`tests/integration/find-triple-composition.test.ts`.
|
||||
|
||||
Memory: ~62 KB resident per entity at 100k (6.2 GB RSS for 100k × 384-dim including the HNSW
|
||||
graph, metadata index, and 100k edges).
|
||||
|
||||
**Scale ceiling (open-core).** The pure-JS HNSW *build* cost (~100 inserts/s at 384-dim on
|
||||
one core) makes the in-process open-core path most appropriate up to ~10⁵–10⁶ entities.
|
||||
*Query* latency stays low well beyond that, but for the 10⁸–10¹⁰ regime install the native
|
||||
provider (`@soulcraft/cor`, on-disk DiskANN) — same API, no code change. _Projected from
|
||||
the two measured points, vector p50 at 1M is ~2 ms; metadata-heavy composition grows with
|
||||
match-set size and is the path to move onto the native provider first._
|
||||
// Behind the scenes:
|
||||
// 1. Hash ID to determine shard (consistent hashing)
|
||||
// 2. Find nodes responsible for this shard
|
||||
// 3. Write to primary shard owner
|
||||
// 4. Replicate to N backup nodes (default: 2)
|
||||
// 5. Confirm write when majority acknowledge
|
||||
```
|
||||
|
||||
## Storage Configurations
|
||||
|
||||
### Filesystem (Recommended for Production)
|
||||
### 🗂️ Storage Adapter Patterns
|
||||
|
||||
Brainy intelligently adapts to your storage setup:
|
||||
|
||||
#### Pattern 1: Separate Storage Per Node (Recommended)
|
||||
|
||||
```typescript
|
||||
// Node 1 - Own filesystem
|
||||
const brain1 = new Brainy({
|
||||
storage: '/data/node1' // or auto: './brainy-data'
|
||||
})
|
||||
|
||||
// Node 2 - Own filesystem
|
||||
const brain2 = new Brainy({
|
||||
storage: '/data/node2' // or auto: './brainy-data'
|
||||
})
|
||||
|
||||
// ✅ BENEFITS:
|
||||
// - No conflicts between nodes
|
||||
// - Fast local reads
|
||||
// - True horizontal scaling
|
||||
// - Survives network partitions
|
||||
```
|
||||
|
||||
#### Pattern 2: Separate S3 Buckets Per Node
|
||||
|
||||
```typescript
|
||||
// Node 1 - Own S3 bucket
|
||||
const brain1 = new Brainy({
|
||||
storage: 's3://brainy-node-1' // Auto-uses AWS credentials
|
||||
})
|
||||
|
||||
// Node 2 - Own S3 bucket
|
||||
const brain2 = new Brainy({
|
||||
storage: 's3://brainy-node-2'
|
||||
})
|
||||
|
||||
// ✅ BENEFITS:
|
||||
// - Infinite storage capacity
|
||||
// - Geographic distribution
|
||||
// - No local disk needed
|
||||
// - Built-in durability
|
||||
```
|
||||
|
||||
#### Pattern 3: Shared S3 Bucket (Coordinated)
|
||||
|
||||
```typescript
|
||||
// All nodes - Shared bucket with coordination
|
||||
const brain = new Brainy({
|
||||
storage: 's3://shared-brainy-data',
|
||||
// Brainy automatically adds node-specific prefixes!
|
||||
})
|
||||
|
||||
// What happens automatically:
|
||||
// - Node 1 writes to: s3://shared-brainy-data/node-1/
|
||||
// - Node 2 writes to: s3://shared-brainy-data/node-2/
|
||||
// - Metadata in: s3://shared-brainy-data/_cluster/
|
||||
// - Coordination via S3 conditional writes
|
||||
|
||||
// ✅ BENEFITS:
|
||||
// - Single bucket to manage
|
||||
// - Easy backup/restore
|
||||
// - Cost effective
|
||||
// - Automatic namespace isolation
|
||||
```
|
||||
|
||||
#### Pattern 4: Mixed Storage (Hybrid)
|
||||
|
||||
```typescript
|
||||
// Hot data on local SSD, cold data in S3
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: '/var/lib/brainy'
|
||||
hot: '/fast-ssd/brainy', // Recent/frequent data
|
||||
cold: 's3://brainy-archive' // Older data
|
||||
}
|
||||
// Brainy automatically promotes/demotes data!
|
||||
})
|
||||
```
|
||||
- Stores everything in a sharded JSON tree under `path`
|
||||
- Atomic writes via rename
|
||||
- Survives process restarts
|
||||
- Snapshot it off-site with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` from your scheduler
|
||||
|
||||
### Memory
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
||||
```
|
||||
- Zero I/O, fastest possible
|
||||
- No persistence — process exit discards everything
|
||||
- Use for tests and ephemeral caches
|
||||
### 🌍 Cloud Provider Auto-Detection
|
||||
|
||||
### Auto
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'auto', path: './data' }
|
||||
storage: 'cloud://brainy-data' // Auto-detects provider!
|
||||
})
|
||||
|
||||
// Automatically uses:
|
||||
// - AWS: S3 + DynamoDB for metadata
|
||||
// - Google Cloud: GCS + Firestore
|
||||
// - Azure: Blob Storage + Cosmos DB
|
||||
// - Cloudflare: R2 + D1
|
||||
// - Vercel: Blob + KV
|
||||
```
|
||||
- Picks `filesystem` when running on Node with a writable `path`
|
||||
- Falls back to `memory` otherwise
|
||||
|
||||
### 📝 Storage Coordination Rules
|
||||
|
||||
When multiple nodes share storage, Brainy automatically:
|
||||
|
||||
1. **Namespace Isolation**: Each node gets unique prefix
|
||||
2. **Lock-Free Writes**: Uses atomic operations
|
||||
3. **Consistent Metadata**: Coordinated via consensus
|
||||
4. **Conflict Resolution**: Version vectors for conflicts
|
||||
5. **Garbage Collection**: Automatic cleanup of old data
|
||||
|
||||
## Scaling Patterns
|
||||
|
||||
### Stage 1: Prototype (Memory)
|
||||
### 📈 Progressive Scaling Journey
|
||||
|
||||
#### Stage 1: Prototype (1 node, memory)
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
||||
// Development, tests, <100K items
|
||||
const brain = new Brainy() // Memory storage, single node
|
||||
// Perfect for: Development, testing, <1000 items
|
||||
```
|
||||
|
||||
### Stage 2: Production (Filesystem)
|
||||
#### Stage 2: Production (1 node, disk)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' }
|
||||
storage: './data' // Persistent storage
|
||||
})
|
||||
// Most production workloads up to ~10M entities on a single host
|
||||
// Perfect for: Small apps, <100K items
|
||||
```
|
||||
|
||||
### Stage 3: Higher Throughput (Tune the Vector Index)
|
||||
#### Stage 3: High Availability (2-3 nodes)
|
||||
```typescript
|
||||
// Just start same code on multiple servers!
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'fast', // Trade recall for latency
|
||||
persistMode: 'deferred' // Batch persistence
|
||||
}
|
||||
storage: './data' // Each node's own storage
|
||||
})
|
||||
// Automatic: Leader election, replication, failover
|
||||
// Perfect for: Critical apps, <1M items
|
||||
```
|
||||
|
||||
### Stage 4: Multi-Instance (Operator-Layer)
|
||||
Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `path`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes.
|
||||
#### Stage 4: Scale Out (N nodes)
|
||||
```typescript
|
||||
// Same code, more servers!
|
||||
const brain = new Brainy({
|
||||
storage: 's3://brainy-{{nodeId}}' // Template auto-filled
|
||||
})
|
||||
// Automatic: Sharding, load balancing, geo-distribution
|
||||
// Perfect for: Large apps, unlimited items
|
||||
```
|
||||
|
||||
### 🎯 Common Scaling Scenarios
|
||||
|
||||
#### Scenario: Read-Heavy Application
|
||||
```typescript
|
||||
// Brainy auto-detects read-heavy pattern and:
|
||||
// 1. Increases cache size
|
||||
// 2. Creates more read replicas
|
||||
// 3. Routes reads to nearest node
|
||||
// 4. Caches popular items on all nodes
|
||||
|
||||
const brain = new Brainy() // No config needed!
|
||||
```
|
||||
|
||||
#### Scenario: Multi-Tenant SaaS
|
||||
```typescript
|
||||
// Brainy auto-detects tenant patterns and:
|
||||
// 1. Shards by tenant ID
|
||||
// 2. Isolates tenant data
|
||||
// 3. Routes by tenant
|
||||
// 4. Separate rate limits per tenant
|
||||
|
||||
const brain = new Brainy() // Detects from your queries!
|
||||
```
|
||||
|
||||
#### Scenario: Geographic Distribution
|
||||
```typescript
|
||||
// Deploy nodes in different regions
|
||||
// Brainy automatically:
|
||||
// 1. Detects node locations (via latency)
|
||||
// 2. Replicates data geographically
|
||||
// 3. Routes to nearest node
|
||||
// 4. Handles region failures
|
||||
|
||||
// US-East
|
||||
const brain = new Brainy({ region: 'us-east' }) // Optional hint
|
||||
|
||||
// EU-West (auto-discovers US-East)
|
||||
const brain = new Brainy({ region: 'eu-west' })
|
||||
```
|
||||
|
||||
## Real World Examples
|
||||
|
||||
### Example 1: Single-Node App With Backup
|
||||
### Example 1: Blog Platform
|
||||
|
||||
```typescript
|
||||
// Day 1: Single server
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' }
|
||||
storage: './blog-data'
|
||||
})
|
||||
```
|
||||
Schedule (cron / systemd timer):
|
||||
```bash
|
||||
*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
|
||||
|
||||
// Month 6: Add redundancy (on second server)
|
||||
const brain = new Brainy({
|
||||
storage: './blog-data' // Different machine!
|
||||
})
|
||||
// Automatically syncs with first server
|
||||
|
||||
// Year 2: Global scale
|
||||
// US Server
|
||||
const brain = new Brainy({
|
||||
storage: 's3://blog-us/data'
|
||||
})
|
||||
|
||||
// EU Server
|
||||
const brain = new Brainy({
|
||||
storage: 's3://blog-eu/data'
|
||||
})
|
||||
|
||||
// Asia Server
|
||||
const brain = new Brainy({
|
||||
storage: 's3://blog-asia/data'
|
||||
})
|
||||
// All automatically coordinate!
|
||||
```
|
||||
|
||||
### Example 2: Tests
|
||||
### Example 2: E-Commerce Site
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
||||
// Fast, no cleanup needed between runs
|
||||
// Development
|
||||
const brain = new Brainy() // Memory storage
|
||||
|
||||
// Staging (Kubernetes)
|
||||
const brain = new Brainy({
|
||||
storage: process.env.STORAGE_PATH // Uses PVC
|
||||
})
|
||||
// Auto-discovers other pods via K8s DNS
|
||||
|
||||
// Production (AWS)
|
||||
const brain = new Brainy({
|
||||
storage: 's3://shop-data',
|
||||
cache: 'elasticache://shop-cache' // Optional
|
||||
})
|
||||
// Auto-scales with ECS/EKS
|
||||
```
|
||||
|
||||
### Example 3: Multi-Tenant Service
|
||||
Spin up one Brainy instance per tenant, each in its own directory:
|
||||
### Example 3: Analytics Platform
|
||||
|
||||
```typescript
|
||||
function brainForTenant(tenantId: string) {
|
||||
return new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: `/var/lib/brainy/${tenantId}`
|
||||
}
|
||||
})
|
||||
// Ingestion nodes (write-optimized)
|
||||
const brain = new Brainy({
|
||||
role: 'writer', // Hint for optimization
|
||||
storage: '/fast-nvme/ingest'
|
||||
})
|
||||
|
||||
// Query nodes (read-optimized)
|
||||
const brain = new Brainy({
|
||||
role: 'reader', // More cache, indexes
|
||||
storage: 's3://analytics-archive'
|
||||
})
|
||||
|
||||
// Automatically coordinates between writers and readers!
|
||||
```
|
||||
|
||||
## 🔧 Storage Adapter Specifics
|
||||
|
||||
### Local Filesystem
|
||||
```typescript
|
||||
{
|
||||
storage: './data' // or absolute: '/var/lib/brainy'
|
||||
// Each node MUST have separate directory
|
||||
// Can be network mounted (NFS, EFS)
|
||||
}
|
||||
```
|
||||
Your service layer handles routing and isolation; Brainy stays simple.
|
||||
|
||||
### Example 4: Higher Recall at Scale
|
||||
### AWS S3
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'accurate'
|
||||
{
|
||||
storage: 's3://bucket-name/prefix'
|
||||
// Uses AWS SDK credentials (env, IAM role, etc)
|
||||
// Supports S3-compatible (MinIO, Ceph)
|
||||
}
|
||||
```
|
||||
|
||||
### Cloudflare R2
|
||||
```typescript
|
||||
{
|
||||
storage: 'r2://bucket-name'
|
||||
// Uses Wrangler or API tokens
|
||||
// Zero egress fees!
|
||||
}
|
||||
```
|
||||
|
||||
### Google Cloud Storage
|
||||
```typescript
|
||||
{
|
||||
storage: 'gs://bucket-name'
|
||||
// Uses Application Default Credentials
|
||||
}
|
||||
```
|
||||
|
||||
### Azure Blob Storage
|
||||
```typescript
|
||||
{
|
||||
storage: 'azure://container-name'
|
||||
// Uses DefaultAzureCredential
|
||||
}
|
||||
```
|
||||
|
||||
### Mixed/Tiered
|
||||
```typescript
|
||||
{
|
||||
storage: {
|
||||
hot: './local-cache', // Fast SSD
|
||||
warm: 's3://regular-data', // Standard storage
|
||||
cold: 's3://glacier-archive' // Cheap archive
|
||||
}
|
||||
// Automatic tiering based on access patterns
|
||||
}
|
||||
```
|
||||
|
||||
## 🎭 Advanced Patterns
|
||||
|
||||
### Pattern: Blue-Green Deployment
|
||||
```typescript
|
||||
// Blue cluster (current)
|
||||
const brain = new Brainy({
|
||||
cluster: 'blue',
|
||||
storage: 's3://prod-blue'
|
||||
})
|
||||
|
||||
// Green cluster (new version)
|
||||
const brain = new Brainy({
|
||||
cluster: 'green',
|
||||
storage: 's3://prod-green',
|
||||
syncFrom: 'blue' // Real-time sync during migration
|
||||
})
|
||||
```
|
||||
|
||||
## Tuning Knobs Summary
|
||||
### Pattern: Federation
|
||||
```typescript
|
||||
// Region 1 Cluster
|
||||
const brain1 = new Brainy({
|
||||
federation: 'global',
|
||||
region: 'us-east',
|
||||
storage: 's3://us-east-data'
|
||||
})
|
||||
|
||||
| Setting | Values | When to change |
|
||||
|---------|--------|----------------|
|
||||
| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
|
||||
| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
|
||||
| `storage.cache.maxSize` | integer | Hot-path read cache size |
|
||||
| `storage.cache.ttl` | ms | Cache freshness |
|
||||
// Region 2 Cluster
|
||||
const brain2 = new Brainy({
|
||||
federation: 'global',
|
||||
region: 'eu-west',
|
||||
storage: 's3://eu-west-data'
|
||||
})
|
||||
// Clusters coordinate for global queries!
|
||||
```
|
||||
|
||||
## Monitoring & Observability
|
||||
### Pattern: Edge Computing
|
||||
```typescript
|
||||
// Edge nodes (in CDN POPs)
|
||||
const brain = new Brainy({
|
||||
mode: 'edge',
|
||||
storage: 'memory', // RAM only
|
||||
upstream: 'https://main-cluster.example.com'
|
||||
})
|
||||
// Caches frequently accessed data at edge
|
||||
```
|
||||
|
||||
## 📊 Monitoring & Observability
|
||||
|
||||
Brainy automatically exposes metrics:
|
||||
|
||||
```typescript
|
||||
const stats = await brain.stats()
|
||||
const metrics = brain.getMetrics()
|
||||
// {
|
||||
// nounCount: 50000,
|
||||
// verbCount: 80000,
|
||||
// vectorIndex: { ... },
|
||||
// storage: { used: '45GB' }
|
||||
// nodes: { total: 5, healthy: 5 },
|
||||
// shards: { total: 20, local: 4 },
|
||||
// replication: { factor: 2, lag: 45 },
|
||||
// operations: { reads: 10000, writes: 1000 },
|
||||
// storage: { used: '45GB', available: '955GB' }
|
||||
// }
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
## 🚨 Troubleshooting
|
||||
|
||||
### Issue: Slow queries
|
||||
1. Switch to `vector.recall: 'fast'`
|
||||
2. Increase the read cache (`storage.cache.maxSize`)
|
||||
3. Consider the optional native vector provider via `@soulcraft/cor`
|
||||
### Issue: Nodes don't discover each other
|
||||
```typescript
|
||||
// Solution 1: Check network allows UDP 7946
|
||||
// Solution 2: Use explicit peers
|
||||
const brain = new Brainy({
|
||||
peers: ['10.0.0.1:7946', '10.0.0.2:7946']
|
||||
})
|
||||
```
|
||||
|
||||
### Issue: Memory pressure
|
||||
1. Reduce `storage.cache.maxSize`
|
||||
2. Move to `vector.persistMode: 'deferred'` to batch writes
|
||||
3. Consider the optional native vector provider via `@soulcraft/cor` for at-scale index acceleration
|
||||
### Issue: Storage conflicts
|
||||
```typescript
|
||||
// Ensure each node has unique storage path
|
||||
// ❌ WRONG: All nodes use './data'
|
||||
// ✅ RIGHT: Node1: './data1', Node2: './data2'
|
||||
// ✅ RIGHT: Use {{nodeId}} template
|
||||
```
|
||||
|
||||
### Issue: Slow startup after a crash
|
||||
1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
|
||||
2. Verify backup integrity periodically
|
||||
### Issue: Slow performance
|
||||
```typescript
|
||||
// Brainy auto-tunes, but you can hint:
|
||||
const brain = new Brainy({
|
||||
profile: 'read-heavy' // or 'write-heavy', 'balanced'
|
||||
})
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
## 🎯 Best Practices
|
||||
|
||||
1. **One process = one `path`** — never share a directory between processes
|
||||
2. **Snapshot from your scheduler** — Brainy doesn't ship cloud SDKs; use `rclone` / `aws s3 sync` / `gsutil`
|
||||
3. **Profile before tuning** — `recall: 'balanced'` is right for most workloads
|
||||
4. **Install the native vector provider only when measured profiling shows it pays off**
|
||||
1. **Let Brainy Auto-Configure**: Don't over-configure
|
||||
2. **Separate Storage Per Node**: Avoids conflicts
|
||||
3. **Use S3 for Large Scale**: Infinite capacity
|
||||
4. **Start Simple**: Single node → Scale when needed
|
||||
5. **Monitor Metrics**: Watch for bottlenecks
|
||||
6. **Trust Auto-Scaling**: It learns your patterns
|
||||
|
||||
## Summary
|
||||
## 🚀 Summary
|
||||
|
||||
- Brainy 8.0 is a **library**, not a cluster
|
||||
- Storage adapters: `filesystem`, `memory`, `auto`
|
||||
- Vector tuning: `recall`, `persistMode`
|
||||
- Backup is an operator-layer concern — snapshot `path`
|
||||
- **Zero Config**: Just `new Brainy()` at any scale
|
||||
- **Auto-Discovery**: Nodes find each other
|
||||
- **Smart Storage**: Adapts to any backend
|
||||
- **Progressive Scaling**: 1 → 100 nodes seamlessly
|
||||
- **Self-Tuning**: Learns and optimizes
|
||||
- **No DevOps**: It just works!
|
||||
|
||||
**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!**
|
||||
|
||||
---
|
||||
|
||||
*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)*
|
||||
340
docs/VALIDATION.md
Normal file
340
docs/VALIDATION.md
Normal file
|
|
@ -0,0 +1,340 @@
|
|||
# Brainy Validation System
|
||||
|
||||
## Zero-Config Philosophy
|
||||
|
||||
Brainy's validation system automatically adapts to your system resources without any configuration. It enforces universal truths while dynamically adjusting limits based on available memory and observed performance.
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Universal Truths Only
|
||||
We only validate things that are mathematically or logically impossible:
|
||||
- Negative pagination values (there's no page -1)
|
||||
- Probabilities outside 0-1 range
|
||||
- Self-referential relationships
|
||||
- Invalid enum values
|
||||
|
||||
### 2. Auto-Configuration
|
||||
The system automatically configures based on:
|
||||
- **Available Memory**: More RAM = higher limits
|
||||
- **System Performance**: Adjusts based on query response times
|
||||
- **Usage Patterns**: Learns from your actual workload
|
||||
|
||||
### 3. Performance Monitoring
|
||||
Every query is monitored to tune future limits:
|
||||
```typescript
|
||||
// Automatic adjustment based on performance
|
||||
if (avgQueryTime < 100ms && resultCount > 80% of limit) {
|
||||
// Increase limits - system can handle more
|
||||
maxLimit *= 1.5
|
||||
} else if (avgQueryTime > 1000ms) {
|
||||
// Reduce limits - system is struggling
|
||||
maxLimit *= 0.8
|
||||
}
|
||||
```
|
||||
|
||||
## Validation Rules by Method
|
||||
|
||||
### `add(params: AddParams)`
|
||||
|
||||
**Required:**
|
||||
- Either `data` or `vector` must be provided
|
||||
- `type` must be a valid NounType enum
|
||||
|
||||
**Constraints:**
|
||||
- `vector` must have exactly 384 dimensions (for all-MiniLM-L6-v2)
|
||||
- Custom `id` must be unique
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// ✅ Valid
|
||||
await brain.add({
|
||||
data: "Hello world",
|
||||
type: NounType.Document
|
||||
})
|
||||
|
||||
// ✅ Valid - pre-computed vector
|
||||
await brain.add({
|
||||
vector: new Array(384).fill(0),
|
||||
type: NounType.Document
|
||||
})
|
||||
|
||||
// ❌ Invalid - missing both data and vector
|
||||
await brain.add({
|
||||
type: NounType.Document
|
||||
})
|
||||
// Error: "must provide either data or vector"
|
||||
|
||||
// ❌ Invalid - wrong vector dimensions
|
||||
await brain.add({
|
||||
vector: new Array(100).fill(0),
|
||||
type: NounType.Document
|
||||
})
|
||||
// Error: "vector must have exactly 384 dimensions"
|
||||
```
|
||||
|
||||
### `update(params: UpdateParams)`
|
||||
|
||||
**Required:**
|
||||
- `id` must be provided
|
||||
- At least one field must be updated
|
||||
|
||||
**Important Metadata Behavior:**
|
||||
- `metadata: null` with `merge: false` → **Keeps existing metadata** (does nothing)
|
||||
- `metadata: {}` with `merge: false` → **Clears metadata** ✅
|
||||
- `metadata: undefined` → No change to metadata
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// ✅ Valid - update metadata
|
||||
await brain.update({
|
||||
id: "xyz",
|
||||
metadata: { status: "published" }
|
||||
})
|
||||
|
||||
// ✅ Valid - clear metadata properly
|
||||
await brain.update({
|
||||
id: "xyz",
|
||||
metadata: {},
|
||||
merge: false
|
||||
})
|
||||
|
||||
// ❌ Invalid - null doesn't clear metadata
|
||||
await brain.update({
|
||||
id: "xyz",
|
||||
metadata: null,
|
||||
merge: false
|
||||
})
|
||||
// Error: "must specify at least one field to update"
|
||||
// (because null metadata doesn't actually update anything)
|
||||
|
||||
// ❌ Invalid - no fields to update
|
||||
await brain.update({
|
||||
id: "xyz"
|
||||
})
|
||||
// Error: "must specify at least one field to update"
|
||||
```
|
||||
|
||||
### `relate(params: RelateParams)`
|
||||
|
||||
**Required:**
|
||||
- `from` entity ID
|
||||
- `to` entity ID
|
||||
- `type` must be valid VerbType enum
|
||||
|
||||
**Constraints:**
|
||||
- `from` and `to` must be different (no self-loops)
|
||||
- `weight` must be between 0 and 1
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// ✅ Valid
|
||||
await brain.relate({
|
||||
from: "entity1",
|
||||
to: "entity2",
|
||||
type: VerbType.RelatedTo
|
||||
})
|
||||
|
||||
// ❌ Invalid - self-referential
|
||||
await brain.relate({
|
||||
from: "entity1",
|
||||
to: "entity1",
|
||||
type: VerbType.RelatedTo
|
||||
})
|
||||
// Error: "cannot create self-referential relationship"
|
||||
|
||||
// ❌ Invalid - weight out of range
|
||||
await brain.relate({
|
||||
from: "entity1",
|
||||
to: "entity2",
|
||||
type: VerbType.RelatedTo,
|
||||
weight: 1.5
|
||||
})
|
||||
// Error: "weight must be between 0 and 1"
|
||||
```
|
||||
|
||||
### `find(params: FindParams)`
|
||||
|
||||
**Constraints:**
|
||||
- `limit` must be non-negative and below auto-configured maximum
|
||||
- `offset` must be non-negative
|
||||
- Cannot specify both `query` and `vector` (mutually exclusive)
|
||||
- Cannot use both `cursor` and `offset` pagination
|
||||
- `threshold` must be between 0 and 1
|
||||
|
||||
**Auto-Configured Limits:**
|
||||
```typescript
|
||||
// Based on available memory
|
||||
// 1GB RAM → max limit: 10,000
|
||||
// 8GB RAM → max limit: 80,000
|
||||
// 16GB RAM → max limit: 100,000 (capped)
|
||||
|
||||
// Query length also scales with memory
|
||||
// 1GB RAM → max query: 5,000 characters
|
||||
// 8GB RAM → max query: 40,000 characters
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// ✅ Valid
|
||||
await brain.find({
|
||||
query: "machine learning",
|
||||
limit: 50
|
||||
})
|
||||
|
||||
// ❌ Invalid - negative limit
|
||||
await brain.find({
|
||||
query: "test",
|
||||
limit: -1
|
||||
})
|
||||
// Error: "limit must be non-negative"
|
||||
|
||||
// ❌ Invalid - both query and vector
|
||||
await brain.find({
|
||||
query: "test",
|
||||
vector: new Array(384).fill(0)
|
||||
})
|
||||
// Error: "cannot specify both query and vector - they are mutually exclusive"
|
||||
|
||||
// ❌ Invalid - exceeds auto-configured limit
|
||||
await brain.find({
|
||||
limit: 1000000
|
||||
})
|
||||
// Error: "limit exceeds auto-configured maximum of 80000 (based on available memory)"
|
||||
```
|
||||
|
||||
## Auto-Configuration Details
|
||||
|
||||
### Memory-Based Scaling
|
||||
|
||||
The validation system checks available memory on initialization:
|
||||
|
||||
```typescript
|
||||
const availableMemory = os.freemem()
|
||||
|
||||
// Scale limits based on available memory
|
||||
maxLimit = Math.min(
|
||||
100000, // Absolute maximum for safety
|
||||
Math.floor(availableMemory / (1024 * 1024 * 100)) * 1000
|
||||
)
|
||||
|
||||
// Scale query length similarly
|
||||
maxQueryLength = Math.min(
|
||||
50000,
|
||||
Math.floor(availableMemory / (1024 * 1024 * 10)) * 1000
|
||||
)
|
||||
```
|
||||
|
||||
### Performance-Based Tuning
|
||||
|
||||
The system continuously monitors and adjusts:
|
||||
|
||||
1. **After each query**, performance is recorded
|
||||
2. **Limits adjust** based on response times
|
||||
3. **Gradual optimization** towards optimal throughput
|
||||
|
||||
### Checking Current Configuration
|
||||
|
||||
You can inspect the current validation configuration:
|
||||
|
||||
```typescript
|
||||
import { getValidationConfig } from '@soulcraft/brainy/validation'
|
||||
|
||||
const config = getValidationConfig()
|
||||
console.log(config)
|
||||
// {
|
||||
// maxLimit: 80000,
|
||||
// maxQueryLength: 40000,
|
||||
// maxVectorDimensions: 384,
|
||||
// systemMemory: 17179869184,
|
||||
// availableMemory: 8589934592
|
||||
// }
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Clearing Metadata
|
||||
```typescript
|
||||
// ❌ Wrong - doesn't clear
|
||||
await brain.update({ id, metadata: null, merge: false })
|
||||
|
||||
// ✅ Correct - actually clears
|
||||
await brain.update({ id, metadata: {}, merge: false })
|
||||
```
|
||||
|
||||
### 2. Type Safety
|
||||
```typescript
|
||||
// ❌ Wrong - string type
|
||||
await brain.add({ data: "test", type: "document" })
|
||||
|
||||
// ✅ Correct - enum type
|
||||
import { NounType } from '@soulcraft/brainy'
|
||||
await brain.add({ data: "test", type: NounType.Document })
|
||||
```
|
||||
|
||||
### 3. Pagination
|
||||
```typescript
|
||||
// ✅ Let the system auto-configure limits
|
||||
const results = await brain.find({
|
||||
query: "test",
|
||||
limit: 100 // Will be capped at system maximum
|
||||
})
|
||||
|
||||
// ✅ For large datasets, use pagination
|
||||
let offset = 0
|
||||
const pageSize = 1000
|
||||
while (true) {
|
||||
const results = await brain.find({
|
||||
query: "test",
|
||||
limit: pageSize,
|
||||
offset
|
||||
})
|
||||
if (results.length === 0) break
|
||||
offset += pageSize
|
||||
}
|
||||
```
|
||||
|
||||
## Error Messages
|
||||
|
||||
All validation errors are descriptive and actionable:
|
||||
|
||||
| Error | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| `"must provide either data or vector"` | Missing content in add() | Provide either data to embed or pre-computed vector |
|
||||
| `"limit must be non-negative"` | Negative pagination | Use positive limit value |
|
||||
| `"invalid NounType: xyz"` | Invalid enum value | Use valid NounType enum |
|
||||
| `"cannot create self-referential relationship"` | from === to | Use different entity IDs |
|
||||
| `"must specify at least one field to update"` | Empty update | Provide at least one field to change |
|
||||
| `"vector must have exactly 384 dimensions"` | Wrong vector size | Use 384-dimensional vectors |
|
||||
|
||||
## Performance Impact
|
||||
|
||||
The validation system adds minimal overhead:
|
||||
- **Validation time**: <1ms per operation
|
||||
- **Memory usage**: ~1KB for configuration tracking
|
||||
- **Auto-tuning**: Happens asynchronously, no blocking
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: Why can't I set metadata to null?**
|
||||
A: Setting metadata to `null` with `merge: false` doesn't actually clear it - it falls back to existing metadata. Use `{}` to clear.
|
||||
|
||||
**Q: Why are my limits being reduced?**
|
||||
A: If queries are taking >1 second, the system automatically reduces limits to maintain performance.
|
||||
|
||||
**Q: Can I override the auto-configured limits?**
|
||||
A: No, this is by design. The system knows better than static configuration what your hardware can handle.
|
||||
|
||||
**Q: Why exactly 384 dimensions for vectors?**
|
||||
A: Brainy uses the all-MiniLM-L6-v2 model which produces 384-dimensional embeddings. This ensures consistency.
|
||||
|
||||
## Summary
|
||||
|
||||
Brainy's validation system:
|
||||
- ✅ **Zero configuration** - adapts to your system
|
||||
- ✅ **Universal truths** - only prevents impossible operations
|
||||
- ✅ **Performance aware** - adjusts based on actual performance
|
||||
- ✅ **Type safe** - enforces enum types
|
||||
- ✅ **Minimal overhead** - <1ms validation time
|
||||
- ✅ **Clear errors** - actionable error messages
|
||||
|
||||
The philosophy is simple: prevent impossible operations, adapt to reality, and get out of the way.
|
||||
406
docs/ZERO_CONFIG.md
Normal file
406
docs/ZERO_CONFIG.md
Normal file
|
|
@ -0,0 +1,406 @@
|
|||
# 🚀 Brainy Zero-Configuration Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Starting with v2.10, Brainy introduces a **Zero-Configuration System** that automatically configures everything based on your environment. No more environment variables, no more complex configuration objects - just create and use.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### True Zero Config
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// That's it! Everything auto-configures
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
### Using Strongly-Typed Presets
|
||||
|
||||
```typescript
|
||||
import { Brainy, PresetName } from '@soulcraft/brainy'
|
||||
|
||||
// Type-safe preset selection
|
||||
const brain = new Brainy(PresetName.PRODUCTION)
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
### Common Scenarios
|
||||
|
||||
#### Development
|
||||
```typescript
|
||||
const brain = new Brainy(PresetName.DEVELOPMENT)
|
||||
// ✅ Filesystem storage for persistence
|
||||
// ✅ FP32 models for best quality
|
||||
// ✅ Verbose logging
|
||||
// ✅ All features enabled
|
||||
```
|
||||
|
||||
#### Production
|
||||
```typescript
|
||||
const brain = new Brainy(PresetName.PRODUCTION)
|
||||
// ✅ Disk storage for persistence
|
||||
// ✅ Auto-selected model precision
|
||||
// ✅ Silent logging
|
||||
// ✅ Optimized features
|
||||
```
|
||||
|
||||
#### Minimal
|
||||
```typescript
|
||||
const brain = new Brainy(PresetName.MINIMAL)
|
||||
// ✅ Filesystem storage
|
||||
// ✅ Q8 models for small size
|
||||
// ✅ Core features only
|
||||
// ✅ Minimal resource usage
|
||||
```
|
||||
|
||||
## Distributed Architecture Presets
|
||||
|
||||
Brainy includes specialized presets for distributed and microservice architectures:
|
||||
|
||||
### Basic Distributed Roles
|
||||
```typescript
|
||||
import { Brainy, PresetName } from '@soulcraft/brainy'
|
||||
|
||||
// Write-only instance (data ingestion)
|
||||
const writer = new Brainy(PresetName.WRITER)
|
||||
// ✅ Optimized for writes
|
||||
// ✅ No search index loading
|
||||
// ✅ Minimal memory usage
|
||||
|
||||
// Read-only instance (search API)
|
||||
const reader = new Brainy(PresetName.READER)
|
||||
// ✅ Optimized for search
|
||||
// ✅ Lazy index loading
|
||||
// ✅ Large cache
|
||||
```
|
||||
|
||||
### Service-Specific Presets
|
||||
```typescript
|
||||
// High-throughput data ingestion
|
||||
const ingestion = new Brainy(PresetName.INGESTION_SERVICE)
|
||||
|
||||
// Low-latency search API
|
||||
const searchApi = new Brainy(PresetName.SEARCH_API)
|
||||
|
||||
// Analytics processing
|
||||
const analytics = new Brainy(PresetName.ANALYTICS_SERVICE)
|
||||
|
||||
// Edge location cache
|
||||
const edge = new Brainy(PresetName.EDGE_CACHE)
|
||||
|
||||
// Batch processing
|
||||
const batch = new Brainy(PresetName.BATCH_PROCESSOR)
|
||||
|
||||
// Real-time streaming
|
||||
const streaming = new Brainy(PresetName.STREAMING_SERVICE)
|
||||
|
||||
// ML training
|
||||
const training = new Brainy(PresetName.ML_TRAINING)
|
||||
|
||||
// Lightweight sidecar
|
||||
const sidecar = new Brainy(PresetName.SIDECAR)
|
||||
```
|
||||
|
||||
## Model Precision Control
|
||||
|
||||
You can **explicitly specify** model precision when needed:
|
||||
|
||||
```typescript
|
||||
import { ModelPrecision } from '@soulcraft/brainy'
|
||||
|
||||
// Force FP32 (full precision)
|
||||
const brain = new Brainy({ model: ModelPrecision.FP32 })
|
||||
|
||||
// Force Q8 (quantized, smaller)
|
||||
const brain = new Brainy({ model: ModelPrecision.Q8 })
|
||||
|
||||
// Use presets
|
||||
const brain = new Brainy({ model: ModelPrecision.FAST }) // Maps to fp32
|
||||
const brain = new Brainy({ model: ModelPrecision.SMALL }) // Maps to q8
|
||||
|
||||
// Auto-detection (default)
|
||||
const brain = new Brainy({ model: ModelPrecision.AUTO })
|
||||
```
|
||||
|
||||
### Auto-Detection Logic
|
||||
|
||||
When not specified, Brainy automatically selects the best model:
|
||||
|
||||
- **Browser**: Q8 (smaller download)
|
||||
- **Serverless**: Q8 (faster cold starts)
|
||||
- **Low Memory (<512MB)**: Q8
|
||||
- **Development**: FP32 (best quality)
|
||||
- **Production (>2GB RAM)**: FP32
|
||||
- **Default**: Q8 (balanced)
|
||||
|
||||
## Storage Configuration
|
||||
|
||||
### Automatic Storage Detection
|
||||
|
||||
Brainy automatically detects the best storage option:
|
||||
|
||||
1. **Cloud Storage** (if credentials found)
|
||||
- AWS S3 (checks AWS_ACCESS_KEY_ID, AWS_PROFILE)
|
||||
- Google Cloud Storage (checks GOOGLE_APPLICATION_CREDENTIALS)
|
||||
- Cloudflare R2 (checks R2_ACCESS_KEY_ID)
|
||||
|
||||
2. **Browser Storage**
|
||||
- OPFS (if supported)
|
||||
- Filesystem fallback
|
||||
|
||||
3. **Node.js Storage**
|
||||
- Filesystem (`./brainy-data` or `~/.brainy/data`)
|
||||
|
||||
### Manual Storage Control
|
||||
|
||||
```typescript
|
||||
import { StorageOption } from '@soulcraft/brainy'
|
||||
|
||||
// Force specific storage with enum
|
||||
const brain = new Brainy({ storage: StorageOption.DISK })
|
||||
const brain = new Brainy({ storage: StorageOption.CLOUD })
|
||||
const brain = new Brainy({ storage: StorageOption.AUTO })
|
||||
|
||||
// Custom storage configuration
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
s3Storage: {
|
||||
bucket: 'my-bucket',
|
||||
region: 'us-east-1'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Feature Sets
|
||||
|
||||
Control which features are enabled:
|
||||
|
||||
```typescript
|
||||
import { FeatureSet } from '@soulcraft/brainy'
|
||||
|
||||
// Preset feature sets with enum
|
||||
const brain = new Brainy({ features: FeatureSet.MINIMAL }) // Core only
|
||||
const brain = new Brainy({ features: FeatureSet.DEFAULT }) // Balanced
|
||||
const brain = new Brainy({ features: FeatureSet.FULL }) // Everything
|
||||
|
||||
// Custom features
|
||||
const brain = new Brainy({
|
||||
features: ['core', 'search', 'cache', 'triple-intelligence']
|
||||
})
|
||||
```
|
||||
|
||||
## Simplified Configuration Interface
|
||||
|
||||
The new configuration is dramatically simpler:
|
||||
|
||||
```typescript
|
||||
interface BrainyZeroConfig {
|
||||
// Mode preset - now with distributed options
|
||||
mode?: PresetName // All strongly typed presets
|
||||
|
||||
// Model configuration with enum
|
||||
model?: ModelPrecision
|
||||
|
||||
// Storage configuration with enum
|
||||
storage?: StorageOption | StorageConfig
|
||||
|
||||
// Feature set with enum
|
||||
features?: FeatureSet | string[]
|
||||
|
||||
// Logging
|
||||
verbose?: boolean
|
||||
|
||||
// Escape hatch for advanced users
|
||||
advanced?: any
|
||||
}
|
||||
```
|
||||
|
||||
### Available Enums
|
||||
|
||||
```typescript
|
||||
enum PresetName {
|
||||
// Basic
|
||||
PRODUCTION = 'production',
|
||||
DEVELOPMENT = 'development',
|
||||
MINIMAL = 'minimal',
|
||||
ZERO = 'zero',
|
||||
|
||||
// Distributed
|
||||
WRITER = 'writer',
|
||||
READER = 'reader',
|
||||
|
||||
// Services
|
||||
INGESTION_SERVICE = 'ingestion-service',
|
||||
SEARCH_API = 'search-api',
|
||||
ANALYTICS_SERVICE = 'analytics-service',
|
||||
EDGE_CACHE = 'edge-cache',
|
||||
BATCH_PROCESSOR = 'batch-processor',
|
||||
STREAMING_SERVICE = 'streaming-service',
|
||||
ML_TRAINING = 'ml-training',
|
||||
SIDECAR = 'sidecar'
|
||||
}
|
||||
|
||||
enum ModelPrecision {
|
||||
FP32 = 'fp32',
|
||||
Q8 = 'q8',
|
||||
AUTO = 'auto',
|
||||
FAST = 'fast', // Maps to fp32
|
||||
SMALL = 'small' // Maps to q8
|
||||
}
|
||||
|
||||
enum StorageOption {
|
||||
AUTO = 'auto',
|
||||
DISK = 'disk',
|
||||
CLOUD = 'cloud'
|
||||
}
|
||||
|
||||
enum FeatureSet {
|
||||
MINIMAL = 'minimal',
|
||||
DEFAULT = 'default',
|
||||
FULL = 'full'
|
||||
}
|
||||
```
|
||||
|
||||
## Multi-Instance with Shared Storage
|
||||
|
||||
When multiple Brainy instances connect to the same storage (like S3), you **must ensure they use compatible configurations**:
|
||||
|
||||
```typescript
|
||||
import { ModelPrecision } from '@soulcraft/brainy'
|
||||
|
||||
// Container A - Writer
|
||||
const writer = new Brainy({
|
||||
mode: PresetName.WRITER,
|
||||
model: ModelPrecision.FP32, // ⚠️ MUST match across instances!
|
||||
storage: { s3Storage: { bucket: 'shared-data' }}
|
||||
})
|
||||
|
||||
// Container B - Reader
|
||||
const reader = new Brainy({
|
||||
mode: PresetName.READER,
|
||||
model: ModelPrecision.FP32, // ✅ Matches Container A
|
||||
storage: { s3Storage: { bucket: 'shared-data' }}
|
||||
})
|
||||
```
|
||||
|
||||
### Distributed Architecture Example
|
||||
|
||||
```typescript
|
||||
// Ingestion Service (Writer)
|
||||
const ingestion = new Brainy({
|
||||
mode: PresetName.INGESTION_SERVICE,
|
||||
model: ModelPrecision.Q8, // All instances must use Q8
|
||||
storage: { s3Storage: { bucket: 'production-data' }}
|
||||
})
|
||||
|
||||
// Search API (Reader)
|
||||
const search = new Brainy({
|
||||
mode: PresetName.SEARCH_API,
|
||||
model: ModelPrecision.Q8, // Matches ingestion service
|
||||
storage: { s3Storage: { bucket: 'production-data' }}
|
||||
})
|
||||
|
||||
// Analytics (Hybrid)
|
||||
const analytics = new Brainy({
|
||||
mode: PresetName.ANALYTICS_SERVICE,
|
||||
model: ModelPrecision.Q8, // Matches other services
|
||||
storage: { s3Storage: { bucket: 'production-data' }}
|
||||
})
|
||||
```
|
||||
|
||||
## Migration from Old Configuration
|
||||
|
||||
### Before (Complex)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
hnsw: {
|
||||
M: 16,
|
||||
efConstruction: 200,
|
||||
seed: 42
|
||||
},
|
||||
storage: {
|
||||
s3Storage: {
|
||||
bucketName: 'my-bucket',
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
|
||||
region: 'us-east-1'
|
||||
},
|
||||
cacheConfig: {
|
||||
hotCacheMaxSize: 5000,
|
||||
hotCacheEvictionThreshold: 0.8,
|
||||
warmCacheTTL: 3600000,
|
||||
batchSize: 100
|
||||
}
|
||||
},
|
||||
cache: {
|
||||
autoTune: true,
|
||||
autoTuneInterval: 60000,
|
||||
hotCacheMaxSize: 10000
|
||||
},
|
||||
embeddingFunction: customFunction,
|
||||
readOnly: false,
|
||||
logging: { verbose: true }
|
||||
})
|
||||
```
|
||||
|
||||
### After (Simple)
|
||||
```typescript
|
||||
const brain = new Brainy('production')
|
||||
// Everything above is auto-configured!
|
||||
```
|
||||
|
||||
## Environment Variables (No Longer Needed!)
|
||||
|
||||
These environment variables are **no longer required**:
|
||||
|
||||
- ❌ `BRAINY_ALLOW_REMOTE_MODELS` - Models auto-download when needed
|
||||
- ❌ `BRAINY_MODELS_PATH` - Path auto-selected based on environment
|
||||
- ❌ `BRAINY_Q8_CONFIRMED` - Warnings auto-suppressed in production
|
||||
- ❌ `BRAINY_LOG_LEVEL` - Auto-set based on NODE_ENV
|
||||
- ❌ AWS credentials - Use AWS SDK credential chain
|
||||
|
||||
## Performance Impact
|
||||
|
||||
The zero-config system has **zero performance overhead**:
|
||||
|
||||
- Configuration happens once during initialization
|
||||
- Auto-detected values are cached
|
||||
- Same optimized code paths as manual configuration
|
||||
- Actually **faster** startup due to reduced parsing
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Models Not Downloading
|
||||
- Check internet connection
|
||||
- Ensure firewall allows HTTPS to Hugging Face / CDN
|
||||
- Run `npm run download-models` to pre-download
|
||||
|
||||
### Wrong Model Precision
|
||||
- Explicitly specify: `{ model: 'fp32' }` or `{ model: 'q8' }`
|
||||
- Check shared storage compatibility
|
||||
|
||||
### Storage Detection Issues
|
||||
- Check cloud credentials are properly configured
|
||||
- Verify write permissions for filesystem paths
|
||||
- Use explicit storage configuration if needed
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use zero-config for single instances** - Let Brainy handle everything
|
||||
2. **Specify precision for shared storage** - Ensure compatibility
|
||||
3. **Use presets for common scenarios** - 'development', 'production', 'minimal'
|
||||
4. **Override only what you need** - Start simple, add complexity only if required
|
||||
|
||||
## Summary
|
||||
|
||||
The new zero-config system reduces configuration from **100+ parameters** to **0-3 decisions**:
|
||||
|
||||
| Scenario | Old Config Lines | New Config Lines |
|
||||
|----------|-----------------|------------------|
|
||||
| Development | 50+ | 1 |
|
||||
| Production | 100+ | 1 |
|
||||
| Custom | 200+ | 3-5 |
|
||||
|
||||
**Result**: 95% less configuration, 100% of the power! 🚀
|
||||
152
docs/api-returns.md
Normal file
152
docs/api-returns.md
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
# Brainy API Return Values
|
||||
|
||||
## Core Operations
|
||||
|
||||
### `brain.add()` - Adding Entities
|
||||
|
||||
**Returns:** `Promise<string>` - The ID of the created entity
|
||||
|
||||
```typescript
|
||||
// ✅ Correct usage - add() returns the ID string directly
|
||||
const id = await brain.add({
|
||||
type: 'document',
|
||||
data: 'My document content'
|
||||
})
|
||||
|
||||
console.log('Created entity with ID:', id)
|
||||
|
||||
// Use the ID to create relationships
|
||||
await brain.relate({
|
||||
from: id,
|
||||
to: anotherId,
|
||||
type: 'references'
|
||||
})
|
||||
```
|
||||
|
||||
```typescript
|
||||
// ❌ Incorrect - trying to access .id property
|
||||
const result = await brain.add({
|
||||
type: 'document',
|
||||
data: 'My content'
|
||||
})
|
||||
|
||||
console.log(result.id) // ❌ undefined - result IS the ID, not an object!
|
||||
```
|
||||
|
||||
### Getting the Full Entity
|
||||
|
||||
If you need the full entity object after creation, use `brain.get()`:
|
||||
|
||||
```typescript
|
||||
// Add entity and get its ID
|
||||
const id = await brain.add({
|
||||
type: 'document',
|
||||
data: 'My content',
|
||||
metadata: { label: 'Important Doc' }
|
||||
})
|
||||
|
||||
// Get the full entity
|
||||
const entity = await brain.get(id)
|
||||
console.log(entity.id) // The ID
|
||||
console.log(entity.type) // 'document'
|
||||
console.log(entity.data) // 'My content'
|
||||
console.log(entity.metadata) // { label: 'Important Doc', ... }
|
||||
console.log(entity.vector) // The embedding vector
|
||||
console.log(entity.createdAt) // Timestamp
|
||||
```
|
||||
|
||||
### `brain.find()` - Finding Entities
|
||||
|
||||
**Returns:** `Promise<Entity[]>` - Array of full entity objects
|
||||
|
||||
```typescript
|
||||
const entities = await brain.find({
|
||||
query: 'machine learning',
|
||||
limit: 10
|
||||
})
|
||||
|
||||
// Each entity has full information
|
||||
for (const entity of entities) {
|
||||
console.log(entity.id)
|
||||
console.log(entity.type)
|
||||
console.log(entity.data)
|
||||
console.log(entity.metadata)
|
||||
}
|
||||
```
|
||||
|
||||
### `brain.relate()` - Creating Relationships
|
||||
|
||||
**Returns:** `Promise<string>` - The ID of the created relationship
|
||||
|
||||
```typescript
|
||||
const relationId = await brain.relate({
|
||||
from: entityId1,
|
||||
to: entityId2,
|
||||
type: 'references'
|
||||
})
|
||||
|
||||
console.log('Created relationship with ID:', relationId)
|
||||
```
|
||||
|
||||
## Data Field Behavior
|
||||
|
||||
### String Data - Used for Embeddings
|
||||
|
||||
When `data` is a string, it's used to generate embeddings for semantic search:
|
||||
|
||||
```typescript
|
||||
await brain.add({
|
||||
type: 'document',
|
||||
data: 'This text will be converted to an embedding vector',
|
||||
metadata: {
|
||||
title: 'My Document',
|
||||
year: 2024
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Object Data - Structured Information
|
||||
|
||||
When `data` is an object, it's treated as structured data:
|
||||
|
||||
```typescript
|
||||
await brain.add({
|
||||
type: 'product',
|
||||
data: {
|
||||
name: 'Widget',
|
||||
price: 29.99,
|
||||
category: 'Tools'
|
||||
},
|
||||
vector: precomputedVector // Must provide vector when using object data
|
||||
})
|
||||
```
|
||||
|
||||
**Important:** If you provide object data without a `vector`, you must include a string somewhere for embedding generation, or the operation will fail.
|
||||
|
||||
### Metadata vs Data
|
||||
|
||||
- **`data`**: Primary content - used for embeddings (if string) or stored as structured data (if object)
|
||||
- **`metadata`**: Auxiliary information - always stored as structured data, used for filtering
|
||||
|
||||
**Best practice for labels:**
|
||||
```typescript
|
||||
await brain.add({
|
||||
type: 'document',
|
||||
data: 'The full text content of the document...', // For semantic search
|
||||
metadata: {
|
||||
label: 'Quick Reference Label', // For display
|
||||
author: 'John Doe',
|
||||
category: 'Technical'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
| Method | Returns | Contains |
|
||||
|--------|---------|----------|
|
||||
| `brain.add()` | `string` | The ID of the created entity |
|
||||
| `brain.get()` | `Entity \| null` | Full entity object with all fields |
|
||||
| `brain.find()` | `Entity[]` | Array of full entity objects |
|
||||
| `brain.relate()` | `string` | The ID of the created relationship |
|
||||
| `brain.getRelations()` | `Relation[]` | Array of relationship objects |
|
||||
563
docs/api/COMPREHENSIVE_API_OVERVIEW.md
Normal file
563
docs/api/COMPREHENSIVE_API_OVERVIEW.md
Normal file
|
|
@ -0,0 +1,563 @@
|
|||
# Brainy Complete Public API Reference
|
||||
|
||||
> **Accurate API documentation for Brainy**
|
||||
|
||||
## Initialization
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
// Zero-config (just works)
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// With configuration
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'memory' }, // or { path: './my-data' } for filesystem
|
||||
embeddingModel: 'Q8', // Q4, Q8, F16, F32
|
||||
silent: true // Suppress logs
|
||||
})
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
## Readiness API
|
||||
|
||||
For reliable initialization detection, especially in cloud environments with progressive initialization:
|
||||
|
||||
### `brain.ready` - Await Initialization
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
brain.init() // Fire and forget
|
||||
|
||||
// Elsewhere (e.g., API handler)
|
||||
await brain.ready // Wait until init() completes
|
||||
const results = await brain.find({ query: 'test' })
|
||||
```
|
||||
|
||||
### `brain.isInitialized` - Check Basic Readiness
|
||||
|
||||
```typescript
|
||||
if (brain.isInitialized) {
|
||||
// Safe to use brain methods
|
||||
}
|
||||
```
|
||||
|
||||
### `brain.isFullyInitialized()` - Check Background Tasks
|
||||
|
||||
```typescript
|
||||
// Returns true when ALL initialization is complete, including background tasks
|
||||
// Useful for cloud storage adapters with progressive initialization
|
||||
if (brain.isFullyInitialized()) {
|
||||
console.log('All background tasks complete')
|
||||
}
|
||||
```
|
||||
|
||||
### `brain.awaitBackgroundInit()` - Wait for Background Tasks
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||
await brain.init() // Fast return in cloud (<200ms)
|
||||
|
||||
// Optional: wait for all background tasks (bucket validation, count sync)
|
||||
await brain.awaitBackgroundInit()
|
||||
console.log('Fully initialized including background tasks')
|
||||
```
|
||||
|
||||
### Health Check Pattern
|
||||
|
||||
```typescript
|
||||
app.get('/health', async (req, res) => {
|
||||
try {
|
||||
await brain.ready
|
||||
res.json({
|
||||
status: 'ready',
|
||||
fullyInitialized: brain.isFullyInitialized()
|
||||
})
|
||||
} catch (error) {
|
||||
res.status(503).json({ status: 'initializing', error: error.message })
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Core CRUD Operations
|
||||
|
||||
### `brain.add(params)` - Add Entity
|
||||
|
||||
```typescript
|
||||
// Add with data (auto-embedded)
|
||||
const id = await brain.add({
|
||||
data: 'Machine learning is a subset of AI',
|
||||
type: NounType.Document,
|
||||
metadata: { author: 'Alice', tags: ['ml', 'ai'] }
|
||||
})
|
||||
|
||||
// Add with pre-computed vector
|
||||
const id = await brain.add({
|
||||
data: 'Content here',
|
||||
vector: [0.1, 0.2, ...], // 384 dimensions
|
||||
type: NounType.Concept
|
||||
})
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `data` (required): Content to embed (string, object, or any serializable data)
|
||||
- `type?`: NounType enum value
|
||||
- `metadata?`: Custom key-value pairs
|
||||
- `vector?`: Pre-computed embedding vector
|
||||
- `id?`: Custom ID (auto-generated if not provided)
|
||||
|
||||
### `brain.get(id, options?)` - Get Entity
|
||||
|
||||
```typescript
|
||||
const entity = await brain.get('entity-id')
|
||||
|
||||
// Include vector embeddings (not loaded by default for performance)
|
||||
const entity = await brain.get('entity-id', { includeVectors: true })
|
||||
```
|
||||
|
||||
### `brain.update(params)` - Update Entity
|
||||
|
||||
```typescript
|
||||
await brain.update({
|
||||
id: 'entity-id',
|
||||
data: 'Updated content', // Re-embeds if changed
|
||||
metadata: { reviewed: true } // Merges with existing
|
||||
})
|
||||
```
|
||||
|
||||
### `brain.delete(id)` - Delete Entity
|
||||
|
||||
```typescript
|
||||
await brain.delete('entity-id')
|
||||
```
|
||||
|
||||
### `brain.clear()` - Clear All Data
|
||||
|
||||
```typescript
|
||||
await brain.clear()
|
||||
```
|
||||
|
||||
## Relationships
|
||||
|
||||
### `brain.relate(params)` - Create Relationship
|
||||
|
||||
```typescript
|
||||
const relationId = await brain.relate({
|
||||
from: 'source-entity-id',
|
||||
to: 'target-entity-id',
|
||||
type: VerbType.RelatedTo,
|
||||
metadata: { strength: 0.9 }
|
||||
})
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `from` (required): Source entity ID
|
||||
- `to` (required): Target entity ID
|
||||
- `type` (required): VerbType enum value
|
||||
- `metadata?`: Custom relationship metadata
|
||||
|
||||
### `brain.getRelations(params)` - Query Relationships
|
||||
|
||||
```typescript
|
||||
// Get all relationships from an entity
|
||||
const relations = await brain.getRelations({ from: 'entity-id' })
|
||||
|
||||
// Get relationships to an entity
|
||||
const relations = await brain.getRelations({ to: 'entity-id' })
|
||||
|
||||
// Filter by type
|
||||
const relations = await brain.getRelations({
|
||||
from: 'entity-id',
|
||||
type: VerbType.Contains
|
||||
})
|
||||
```
|
||||
|
||||
### `brain.unrelate(id)` - Delete Relationship
|
||||
|
||||
```typescript
|
||||
await brain.unrelate('relationship-id')
|
||||
```
|
||||
|
||||
## Search & Query
|
||||
|
||||
### `brain.find(query)` - Semantic Search
|
||||
|
||||
The primary search method with Triple Intelligence (semantic + graph + metadata).
|
||||
|
||||
```typescript
|
||||
// Simple text search
|
||||
const results = await brain.find('machine learning algorithms')
|
||||
|
||||
// With options
|
||||
const results = await brain.find({
|
||||
query: 'machine learning',
|
||||
limit: 10,
|
||||
threshold: 0.7,
|
||||
type: NounType.Document,
|
||||
where: { author: 'Alice' },
|
||||
excludeVFS: true // Exclude VFS files from results
|
||||
})
|
||||
|
||||
// Natural language query (Triple Intelligence)
|
||||
const results = await brain.find(
|
||||
'Show me documents about AI written by Alice in 2024'
|
||||
)
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `query`: Search text (required)
|
||||
- `limit?`: Max results (default: 10)
|
||||
- `threshold?`: Minimum similarity (0-1)
|
||||
- `type?`: Filter by NounType
|
||||
- `where?`: Metadata filters
|
||||
- `excludeVFS?`: Exclude VFS entities
|
||||
|
||||
### `brain.similar(params)` - Find Similar Entities
|
||||
|
||||
```typescript
|
||||
// Find similar to entity ID
|
||||
const similar = await brain.similar({
|
||||
to: 'entity-id',
|
||||
limit: 5
|
||||
})
|
||||
|
||||
// Find similar to vector
|
||||
const similar = await brain.similar({
|
||||
to: [0.1, 0.2, ...], // Vector
|
||||
limit: 10,
|
||||
type: NounType.Document
|
||||
})
|
||||
```
|
||||
|
||||
## Batch Operations
|
||||
|
||||
### `brain.addMany(params)` - Batch Add
|
||||
|
||||
```typescript
|
||||
const result = await brain.addMany({
|
||||
items: [
|
||||
{ data: 'First item', type: NounType.Document },
|
||||
{ data: 'Second item', type: NounType.Concept },
|
||||
{ data: 'Third item', metadata: { priority: 'high' } }
|
||||
],
|
||||
continueOnError: true, // Don't stop on failures
|
||||
onProgress: (completed, total) => {
|
||||
console.log(`${completed}/${total} complete`)
|
||||
}
|
||||
})
|
||||
|
||||
console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}`)
|
||||
```
|
||||
|
||||
### `brain.deleteMany(params)` - Batch Delete
|
||||
|
||||
```typescript
|
||||
// Delete by IDs
|
||||
const result = await brain.deleteMany({
|
||||
ids: ['id1', 'id2', 'id3'],
|
||||
continueOnError: true
|
||||
})
|
||||
|
||||
// Delete by type
|
||||
const result = await brain.deleteMany({
|
||||
type: NounType.TempData
|
||||
})
|
||||
|
||||
// Delete by metadata filter
|
||||
const result = await brain.deleteMany({
|
||||
where: { status: 'archived' }
|
||||
})
|
||||
```
|
||||
|
||||
### `brain.relateMany(params)` - Batch Relate
|
||||
|
||||
```typescript
|
||||
const ids = await brain.relateMany({
|
||||
items: [
|
||||
{ from: 'a', to: 'b', type: VerbType.RelatedTo },
|
||||
{ from: 'b', to: 'c', type: VerbType.Contains },
|
||||
{ from: 'c', to: 'd', type: VerbType.References }
|
||||
],
|
||||
continueOnError: true
|
||||
})
|
||||
```
|
||||
|
||||
### `brain.updateMany(params)` - Batch Update
|
||||
|
||||
```typescript
|
||||
const result = await brain.updateMany({
|
||||
items: [
|
||||
{ id: 'id1', metadata: { reviewed: true } },
|
||||
{ id: 'id2', data: 'Updated content' }
|
||||
],
|
||||
continueOnError: true
|
||||
})
|
||||
```
|
||||
|
||||
## Neural API
|
||||
|
||||
Access advanced AI/ML features via `brain.neural()`:
|
||||
|
||||
```typescript
|
||||
const neural = brain.neural()
|
||||
|
||||
// Similarity calculation
|
||||
const similarity = await neural.similar('text1', 'text2')
|
||||
const detailed = await neural.similar('text1', 'text2', { detailed: true })
|
||||
|
||||
// Clustering
|
||||
const clusters = await neural.clusters()
|
||||
const clusters = await neural.clusters({
|
||||
algorithm: 'hierarchical',
|
||||
maxClusters: 10
|
||||
})
|
||||
|
||||
// K-nearest neighbors
|
||||
const neighbors = await neural.neighbors('entity-id', { k: 5 })
|
||||
|
||||
// Semantic hierarchy
|
||||
const hierarchy = await neural.hierarchy('entity-id')
|
||||
|
||||
// Outlier/anomaly detection
|
||||
const outliers = await neural.outliers({ threshold: 2.0 })
|
||||
|
||||
// Domain-aware clustering
|
||||
const domainClusters = await neural.clusterByDomain('category')
|
||||
|
||||
// Temporal clustering
|
||||
const temporalClusters = await neural.clusterByTime('createdAt', [
|
||||
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' },
|
||||
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' }
|
||||
])
|
||||
|
||||
// Streaming clusters (for large datasets)
|
||||
for await (const batch of neural.clusterStream({ batchSize: 100 })) {
|
||||
console.log(`Progress: ${batch.progress.percentage}%`)
|
||||
}
|
||||
```
|
||||
|
||||
## Virtual File System (VFS)
|
||||
|
||||
Access via `brain.vfs`:
|
||||
|
||||
```typescript
|
||||
const vfs = brain.vfs
|
||||
await vfs.init()
|
||||
|
||||
// File operations
|
||||
await vfs.writeFile('/docs/readme.md', '# Hello')
|
||||
const content = await vfs.readFile('/docs/readme.md')
|
||||
await vfs.unlink('/docs/readme.md')
|
||||
|
||||
// Directory operations
|
||||
await vfs.mkdir('/project/src', { recursive: true })
|
||||
const files = await vfs.readdir('/project')
|
||||
await vfs.rmdir('/project', { recursive: true })
|
||||
|
||||
// Bulk operations
|
||||
const result = await vfs.bulkWrite([
|
||||
{ type: 'mkdir', path: '/data' },
|
||||
{ type: 'write', path: '/data/config.json', data: '{}' },
|
||||
{ type: 'write', path: '/data/users.json', data: '[]' }
|
||||
])
|
||||
// Note: mkdir operations run first (sequentially), then other ops in parallel
|
||||
|
||||
// Semantic search
|
||||
const results = await vfs.search('authentication code', { path: '/src' })
|
||||
|
||||
// File metadata
|
||||
const stats = await vfs.stat('/file.txt')
|
||||
await vfs.setMetadata('/file.txt', { author: 'Alice' })
|
||||
```
|
||||
|
||||
## Counts (O(1) Performance)
|
||||
|
||||
```typescript
|
||||
// Entity counts
|
||||
const total = brain.counts.entities()
|
||||
const byType = await brain.counts.byType(NounType.Document)
|
||||
const nonVFS = await brain.counts.byType({ excludeVFS: true })
|
||||
|
||||
// Relationship counts
|
||||
const relations = brain.counts.relationships()
|
||||
const byVerb = await brain.counts.byVerbType(VerbType.Contains)
|
||||
```
|
||||
|
||||
## Versioning & Branching
|
||||
|
||||
```typescript
|
||||
// Create branch
|
||||
await brain.fork('feature-branch')
|
||||
|
||||
// List branches
|
||||
const branches = await brain.listBranches()
|
||||
|
||||
// Switch branch
|
||||
await brain.checkout('feature-branch')
|
||||
|
||||
// Get current branch
|
||||
const current = await brain.getCurrentBranch()
|
||||
|
||||
// Commit changes
|
||||
await brain.commit({ message: 'Added new features' })
|
||||
|
||||
// View history
|
||||
const history = await brain.getHistory({ limit: 10 })
|
||||
|
||||
// Time travel (read-only snapshot)
|
||||
const snapshot = await brain.asOf('commit-id')
|
||||
```
|
||||
|
||||
## Streaming API
|
||||
|
||||
```typescript
|
||||
// Stream all entities
|
||||
for await (const entity of brain.streaming.entities()) {
|
||||
console.log(entity.id)
|
||||
}
|
||||
|
||||
// Stream with filters
|
||||
for await (const entity of brain.streaming.entities({
|
||||
type: NounType.Document,
|
||||
where: { status: 'active' }
|
||||
})) {
|
||||
// Process each entity
|
||||
}
|
||||
|
||||
// Stream relationships
|
||||
for await (const relation of brain.streaming.relations({
|
||||
from: 'entity-id'
|
||||
})) {
|
||||
console.log(relation)
|
||||
}
|
||||
```
|
||||
|
||||
## Pagination API
|
||||
|
||||
```typescript
|
||||
// Paginated queries
|
||||
const page1 = await brain.pagination.find({
|
||||
query: 'machine learning',
|
||||
page: 1,
|
||||
pageSize: 20
|
||||
})
|
||||
|
||||
console.log(`Page ${page1.page} of ${page1.totalPages}`)
|
||||
console.log(`Total results: ${page1.total}`)
|
||||
|
||||
// Get next page
|
||||
const page2 = await brain.pagination.find({
|
||||
query: 'machine learning',
|
||||
page: 2,
|
||||
pageSize: 20
|
||||
})
|
||||
```
|
||||
|
||||
## Augmentations
|
||||
|
||||
```typescript
|
||||
// List active augmentations
|
||||
const augmentations = brain.augmentations.list()
|
||||
|
||||
// Get specific augmentation
|
||||
const cache = brain.augmentations.get('cache')
|
||||
|
||||
// Augmentations are auto-loaded based on config
|
||||
// Common augmentations: cache, display, metrics, intelligent-import
|
||||
```
|
||||
|
||||
## Utilities
|
||||
|
||||
```typescript
|
||||
// Manual embedding
|
||||
const vector = await brain.embed('text to embed')
|
||||
|
||||
// Flush pending writes
|
||||
await brain.flush()
|
||||
|
||||
// Get statistics
|
||||
const stats = await brain.getStats()
|
||||
const statsNoVFS = await brain.getStats({ excludeVFS: true })
|
||||
|
||||
// Close (cleanup)
|
||||
await brain.close()
|
||||
```
|
||||
|
||||
## Type Enums
|
||||
|
||||
```typescript
|
||||
import { NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
// NounType - Entity types
|
||||
NounType.Document
|
||||
NounType.Person
|
||||
NounType.Concept
|
||||
NounType.Event
|
||||
NounType.Location
|
||||
NounType.Organization
|
||||
NounType.Product
|
||||
NounType.Content
|
||||
NounType.Collection
|
||||
// ... and more
|
||||
|
||||
// VerbType - Relationship types
|
||||
VerbType.RelatedTo
|
||||
VerbType.Contains
|
||||
VerbType.References
|
||||
VerbType.DependsOn
|
||||
VerbType.Precedes
|
||||
VerbType.Follows
|
||||
VerbType.CreatedBy
|
||||
VerbType.ModifiedBy
|
||||
// ... and more
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```typescript
|
||||
try {
|
||||
await brain.get('nonexistent-id')
|
||||
} catch (error) {
|
||||
if (error.message.includes('not found')) {
|
||||
// Handle missing entity
|
||||
}
|
||||
}
|
||||
|
||||
// VFS errors use POSIX codes
|
||||
try {
|
||||
await vfs.readFile('/nonexistent')
|
||||
} catch (error) {
|
||||
if (error.code === 'ENOENT') {
|
||||
// File not found
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
// Storage
|
||||
storage: {
|
||||
type: 'memory' | 'filesystem',
|
||||
path: './data', // For filesystem
|
||||
forceMemoryStorage: false // Force memory even if path exists
|
||||
},
|
||||
|
||||
// Embeddings
|
||||
embeddingModel: 'Q8', // Q4, Q8, F16, F32
|
||||
dimensions: 384, // Auto-detected
|
||||
|
||||
// Performance
|
||||
silent: false, // Suppress console output
|
||||
verbose: false, // Extra logging
|
||||
|
||||
// Augmentations (auto-enabled by default)
|
||||
augmentations: {
|
||||
cache: true,
|
||||
display: true,
|
||||
metrics: true
|
||||
}
|
||||
})
|
||||
```
|
||||
1799
docs/api/README.md
1799
docs/api/README.md
File diff suppressed because it is too large
Load diff
200
docs/architecture/API_SURFACE_DESIGN.md
Normal file
200
docs/architecture/API_SURFACE_DESIGN.md
Normal file
|
|
@ -0,0 +1,200 @@
|
|||
# Brainy Neural API Surface Design
|
||||
|
||||
## 🎯 **Clean API Hierarchy**
|
||||
|
||||
### **Main Class Shortcuts (Simple & Common)**
|
||||
```typescript
|
||||
// High-level shortcuts on Brainy - most common operations
|
||||
brain.similar(a, b, options?) // ✅ Keep - very common
|
||||
brain.clusters(items?, options?) // ✅ Keep - very common
|
||||
brain.related(id, options?) // ✅ Keep - common (like neighbors but simpler name)
|
||||
|
||||
// Remove/deprecate confusing shortcuts
|
||||
brain.visualize(options?) // ❌ Remove - too specialized for main class
|
||||
```
|
||||
|
||||
### **Neural Namespace (Full Featured)**
|
||||
```typescript
|
||||
// Core semantic operations
|
||||
brain.neural.similar(a, b, options?) // Comprehensive similarity
|
||||
brain.neural.clusters(items?, options?) // Smart clustering with auto-routing
|
||||
brain.neural.neighbors(id, options?) // K-nearest neighbors (full featured)
|
||||
brain.neural.hierarchy(id, options?) // Semantic hierarchy building
|
||||
brain.neural.outliers(options?) // Anomaly detection
|
||||
brain.neural.visualize(options?) // Visualization data generation
|
||||
|
||||
// Advanced clustering methods
|
||||
brain.neural.clusterByDomain(field, options?) // Domain-aware clustering
|
||||
brain.neural.clusterByTime(field, windows, options?) // Temporal clustering
|
||||
brain.neural.clusterStream(options?) // Streaming/real-time clustering
|
||||
brain.neural.updateClusters(items, options?) // Incremental clustering
|
||||
|
||||
// Utility methods
|
||||
brain.neural.getPerformanceMetrics(operation?) // Performance monitoring
|
||||
brain.neural.clearCaches() // Cache management
|
||||
brain.neural.getCacheStats() // Cache statistics
|
||||
```
|
||||
|
||||
## 🔒 **Private Methods (Internal Implementation)**
|
||||
|
||||
### **Should NOT be exposed publicly:**
|
||||
```typescript
|
||||
// These are implementation details:
|
||||
brain.neural._clusterFast() // ❌ Private - use clusters() with algorithm: 'hierarchical'
|
||||
brain.neural._clusterLarge() // ❌ Private - use clusters() with algorithm: 'sample'
|
||||
brain.neural._performHierarchicalClustering() // ❌ Private - internal routing
|
||||
brain.neural._performKMeansClustering() // ❌ Private - internal routing
|
||||
brain.neural._performDBSCANClustering() // ❌ Private - internal routing
|
||||
brain.neural._performSampledClustering() // ❌ Private - internal routing
|
||||
brain.neural._routeClusteringAlgorithm() // ❌ Private - internal routing
|
||||
brain.neural._similarityById() // ❌ Private - internal routing
|
||||
brain.neural._similarityByVector() // ❌ Private - internal routing
|
||||
brain.neural._similarityByText() // ❌ Private - internal routing
|
||||
brain.neural._isId() // ❌ Private - utility
|
||||
brain.neural._isVector() // ❌ Private - utility
|
||||
brain.neural._convertToVector() // ❌ Private - utility
|
||||
brain.neural._cacheResult() // ❌ Private - caching
|
||||
brain.neural._trackPerformance() // ❌ Private - monitoring
|
||||
```
|
||||
|
||||
### **Current Issues in brain-cloud explorer:**
|
||||
```typescript
|
||||
// ❌ BAD: Accessing private implementation details
|
||||
brain.neural.clusterFast({ maxClusters: count, level: 2 })
|
||||
|
||||
// ✅ GOOD: Use public API with proper options
|
||||
brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: count, level: 2 })
|
||||
```
|
||||
|
||||
## 📊 **API Consistency Fixes**
|
||||
|
||||
### **Method Naming Standardization:**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: Pick one naming pattern and stick to it
|
||||
brain.neural.similar() // Main method name
|
||||
brain.similar() // Shortcut matches
|
||||
|
||||
// ❌ INCONSISTENT: Don't mix these
|
||||
brain.neural.similarity() // Different from shortcut
|
||||
brain.similar()
|
||||
```
|
||||
|
||||
### **Parameter Patterns:**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: Always (data, options?) pattern
|
||||
brain.neural.similar(a, b, options?)
|
||||
brain.neural.clusters(items?, options?)
|
||||
brain.neural.neighbors(id, options?)
|
||||
brain.neural.hierarchy(id, options?)
|
||||
|
||||
// Options should be objects with clear properties
|
||||
interface ClusteringOptions {
|
||||
algorithm?: 'auto' | 'hierarchical' | 'kmeans' | 'dbscan'
|
||||
maxClusters?: number
|
||||
threshold?: number
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### **Return Type Consistency:**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: All clustering methods return SemanticCluster[]
|
||||
brain.neural.clusters() → Promise<SemanticCluster[]>
|
||||
brain.neural.clusterByDomain() → Promise<DomainCluster[]> // extends SemanticCluster
|
||||
brain.neural.clusterByTime() → Promise<TemporalCluster[]> // extends SemanticCluster
|
||||
|
||||
// ✅ CONSISTENT: All similarity methods return number or SimilarityResult
|
||||
brain.neural.similar() → Promise<number | SimilarityResult>
|
||||
brain.similar() → Promise<number> // Shortcut always returns simple number
|
||||
```
|
||||
|
||||
## 🚀 **Performance & Intelligence Routing**
|
||||
|
||||
### **Auto-Algorithm Selection:**
|
||||
```typescript
|
||||
// Smart routing based on data size and characteristics
|
||||
brain.neural.clusters() // Auto-selects:
|
||||
// < 100 items → hierarchical (fast, accurate)
|
||||
// < 1000 items → k-means (balanced)
|
||||
// > 1000 items → sampling (scalable)
|
||||
|
||||
// Manual override available
|
||||
brain.neural.clusters({ algorithm: 'hierarchical' }) // Force specific algorithm
|
||||
```
|
||||
|
||||
### **Caching Strategy:**
|
||||
```typescript
|
||||
// Intelligent caching with LRU eviction
|
||||
brain.neural.similar('id1', 'id2') // First call: compute & cache
|
||||
brain.neural.similar('id1', 'id2') // Second call: instant cache hit
|
||||
|
||||
// Cache management
|
||||
brain.neural.clearCaches() // Manual cache clear
|
||||
brain.neural.getCacheStats() // Monitor cache performance
|
||||
```
|
||||
|
||||
## 📚 **Documentation Structure**
|
||||
|
||||
### **Main Documentation Sections:**
|
||||
1. **Quick Start**: Simple examples using shortcuts (`brain.similar()`, `brain.clusters()`)
|
||||
2. **Neural API Guide**: Comprehensive examples using `brain.neural.*`
|
||||
3. **Advanced Clustering**: Domain, temporal, streaming clustering
|
||||
4. **Performance**: Caching, algorithm selection, monitoring
|
||||
5. **API Reference**: Complete method documentation
|
||||
|
||||
### **Example Progression:**
|
||||
```typescript
|
||||
// 1. BEGINNER: Simple shortcuts
|
||||
const similarity = await brain.similar('text1', 'text2')
|
||||
const clusters = await brain.clusters()
|
||||
|
||||
// 2. INTERMEDIATE: Neural API with options
|
||||
const detailed = await brain.neural.similar('id1', 'id2', { detailed: true })
|
||||
const customClusters = await brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: 5 })
|
||||
|
||||
// 3. ADVANCED: Specialized clustering
|
||||
const domainClusters = await brain.neural.clusterByDomain('category')
|
||||
const streamingClusters = brain.neural.clusterStream({ batchSize: 50 })
|
||||
```
|
||||
|
||||
## ✅ **Implementation Checklist**
|
||||
|
||||
### **High Priority:**
|
||||
- [x] Create comprehensive type definitions
|
||||
- [x] Implement improved NeuralAPI class with proper public/private separation
|
||||
- [ ] Update Brainy integration to use improved API
|
||||
- [ ] Fix brain-cloud explorer to use public APIs
|
||||
- [ ] Update test files to use consistent method names
|
||||
- [ ] Update documentation with new API structure
|
||||
|
||||
### **Medium Priority:**
|
||||
- [ ] Implement placeholder algorithm implementations with real clustering logic
|
||||
- [ ] Add comprehensive error handling and validation
|
||||
- [ ] Add performance monitoring and metrics collection
|
||||
- [ ] Create migration guide for users of deprecated methods
|
||||
|
||||
### **Nice to Have:**
|
||||
- [ ] Add interactive clustering refinement based on user feedback
|
||||
- [ ] Implement explainable clustering with reasoning
|
||||
- [ ] Add multi-modal clustering (text + metadata + relationships)
|
||||
- [ ] Create visualization examples for different graph libraries
|
||||
|
||||
## 🎯 **API Surface Summary**
|
||||
|
||||
### **✅ PUBLIC API** (What users should use):
|
||||
- **Main shortcuts**: `brain.similar()`, `brain.clusters()`, `brain.related()`
|
||||
- **Neural namespace**: `brain.neural.similar()`, `brain.neural.clusters()`, etc.
|
||||
- **Advanced features**: Domain clustering, temporal clustering, streaming
|
||||
- **Utilities**: Performance metrics, cache management
|
||||
|
||||
### **❌ PRIVATE IMPLEMENTATION** (Internal only):
|
||||
- **Algorithm implementations**: `_performKMeansClustering()`, etc.
|
||||
- **Routing logic**: `_routeClusteringAlgorithm()`, etc.
|
||||
- **Utility methods**: `_isId()`, `_convertToVector()`, etc.
|
||||
- **Caching internals**: `_cacheResult()`, `_trackPerformance()`, etc.
|
||||
|
||||
### **⚠️ DEPRECATED** (Should be removed/hidden):
|
||||
- **brain.neural.clusterFast()** → Use `brain.neural.clusters({ algorithm: 'hierarchical' })`
|
||||
- **brain.neural.clusterLarge()** → Use `brain.neural.clusters({ algorithm: 'sample' })`
|
||||
- **brain.neural.similarity()** → Use `brain.neural.similar()` (pick one name)
|
||||
- **brain.visualize()** → Too specialized for main class, use `brain.neural.visualize()`
|
||||
306
docs/architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md
Normal file
306
docs/architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md
Normal file
|
|
@ -0,0 +1,306 @@
|
|||
# 🧠 **Brainy Clustering Algorithms - Complete Analysis**
|
||||
|
||||
## 🎯 **Current State & Capabilities**
|
||||
|
||||
### **✅ Existing Infrastructure (Excellent Foundation)**
|
||||
|
||||
#### **1. HNSW Hierarchical Clustering**
|
||||
```typescript
|
||||
// ALREADY IMPLEMENTED & OPTIMIZED
|
||||
brain.neural.clusters({ algorithm: 'hierarchical', level: 2 })
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
- **Leverages HNSW natural hierarchy**: Uses existing index levels as natural cluster boundaries
|
||||
- **O(n) performance**: Much faster than O(n²) traditional clustering
|
||||
- **Multi-level granularity**: Higher levels = fewer, broader clusters; Lower levels = more, specific clusters
|
||||
- **Representative sampling**: Uses HNSW level nodes as natural cluster centers
|
||||
|
||||
**Performance characteristics:**
|
||||
- **Excellent for large datasets** (millions of items)
|
||||
- **Preserves semantic relationships** from vector space
|
||||
- **Automatic granularity control** via level parameter
|
||||
|
||||
#### **2. Distance-Based Algorithms**
|
||||
```typescript
|
||||
// COMPREHENSIVE DISTANCE FUNCTIONS AVAILABLE
|
||||
euclideanDistance, cosineDistance, manhattanDistance, dotProductDistance
|
||||
```
|
||||
|
||||
**Optimized implementations:**
|
||||
- **Batch processing**: `calculateDistancesBatch()` with parallelization
|
||||
- **Multiple metrics**: Choose optimal distance function per use case
|
||||
- **Performance optimized**: Faster than GPU for small vectors due to no transfer overhead
|
||||
|
||||
#### **3. Rich Semantic Taxonomy**
|
||||
```typescript
|
||||
// 25+ NOUN TYPES & 35+ VERB TYPES
|
||||
NounType: Person, Organization, Document, Concept, Event, Media, etc.
|
||||
VerbType: RelatedTo, Contains, PartOf, Causes, CreatedBy, etc.
|
||||
```
|
||||
|
||||
**Semantic clustering capabilities:**
|
||||
- **Type-based clustering**: Group by semantic categories
|
||||
- **Cross-type relationships**: Use verb types to find semantic bridges
|
||||
- **Hierarchical taxonomies**: Natural clustering within and across types
|
||||
|
||||
#### **4. Graph Structure**
|
||||
```typescript
|
||||
// VERB RELATIONSHIPS CREATE RICH GRAPH
|
||||
await brain.relate(sourceId, targetId, VerbType.Causes, { strength: 0.8 })
|
||||
```
|
||||
|
||||
**Graph-based clustering potential:**
|
||||
- **Connected components**: Find strongly connected groups
|
||||
- **Community detection**: Use relationship strength for clustering
|
||||
- **Multi-modal clustering**: Combine graph + vector + taxonomy
|
||||
|
||||
## 🚀 **Advanced Clustering Algorithms We Can Implement**
|
||||
|
||||
### **1. ✅ Already Implemented: HNSW Hierarchical**
|
||||
|
||||
```typescript
|
||||
// PRODUCTION READY - Uses existing HNSW levels
|
||||
const clusters = await brain.neural.clusters({
|
||||
algorithm: 'hierarchical',
|
||||
level: 2, // Control granularity
|
||||
maxClusters: 15
|
||||
})
|
||||
```
|
||||
|
||||
**Performance:** **A+** - O(n) leveraging existing index structure
|
||||
|
||||
### **2. 🔥 Semantic Taxonomy Clustering**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Fast type-based clustering with cross-type bridges
|
||||
const clusters = await brain.neural.clusterByDomain('nounType', {
|
||||
preserveTypeBoundaries: false, // Allow cross-type clusters
|
||||
bridgeStrength: 0.8, // Minimum relationship strength for bridges
|
||||
hybridWeighting: {
|
||||
taxonomy: 0.4, // 40% weight to type similarity
|
||||
vector: 0.4, // 40% weight to vector similarity
|
||||
graph: 0.2 // 20% weight to relationship strength
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Primary clustering by taxonomy**: Group by NounType/VerbType first
|
||||
2. **Vector refinement**: Sub-cluster within types using vector similarity
|
||||
3. **Cross-type bridging**: Find relationships that bridge type boundaries
|
||||
4. **Weighted fusion**: Combine taxonomy + vector + graph signals
|
||||
|
||||
**Performance:** **A+** - O(n log n) - taxonomy grouping is O(n), refinement is HNSW-accelerated
|
||||
|
||||
### **3. 🔥 Graph Community Detection**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Relationship-based clustering
|
||||
const clusters = await brain.neural.clusterByConnections({
|
||||
algorithm: 'modularity', // or 'louvain', 'leiden'
|
||||
minCommunitySize: 3,
|
||||
relationshipWeights: {
|
||||
[VerbType.Creates]: 1.0,
|
||||
[VerbType.PartOf]: 0.8,
|
||||
[VerbType.RelatedTo]: 0.5
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Build weighted graph**: Use verbs as edges, weights from relationship types + metadata
|
||||
2. **Community detection**: Apply Louvain or Leiden algorithm for modularity optimization
|
||||
3. **Semantic enhancement**: Use vector similarity to refine community boundaries
|
||||
|
||||
**Performance:** **A** - O(n log n) for sparse graphs, handles millions of relationships efficiently
|
||||
|
||||
### **4. 🔥 Multi-Modal Fusion Clustering**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Best of all worlds
|
||||
const clusters = await brain.neural.clusters({
|
||||
algorithm: 'multimodal',
|
||||
signals: {
|
||||
vector: { weight: 0.5, metric: 'cosine' },
|
||||
graph: { weight: 0.3, algorithm: 'modularity' },
|
||||
taxonomy: { weight: 0.2, crossTypeThreshold: 0.8 }
|
||||
},
|
||||
fusion: 'weighted_ensemble' // or 'consensus', 'hierarchical'
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Independent clustering**: Run HNSW, graph, and taxonomy clustering separately
|
||||
2. **Consensus building**: Find agreement between different clustering results
|
||||
3. **Conflict resolution**: Use weighted voting or hierarchical merging for disagreements
|
||||
4. **Quality optimization**: Iteratively refine based on silhouette scores
|
||||
|
||||
**Performance:** **A** - O(n log n) - parallel execution of component algorithms
|
||||
|
||||
### **5. 💎 Temporal Pattern Clustering**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Time-aware clustering using existing infrastructure
|
||||
const clusters = await brain.neural.clusterByTime('createdAt', [
|
||||
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1 2024' },
|
||||
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2 2024' }
|
||||
], {
|
||||
evolution: 'track', // Track how clusters evolve over time
|
||||
stability: 0.7, // Minimum stability threshold
|
||||
trendAnalysis: true // Include trend detection
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Time window clustering**: Apply HNSW clustering within each time window
|
||||
2. **Cluster evolution tracking**: Match clusters across time windows using vector similarity
|
||||
3. **Trend analysis**: Detect growing, shrinking, merging, splitting patterns
|
||||
4. **Stability scoring**: Measure cluster consistency over time
|
||||
|
||||
**Performance:** **A+** - O(k*n log n) where k = number of time windows
|
||||
|
||||
### **6. 💎 DBSCAN with Adaptive Parameters**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Density-based clustering with smart parameter selection
|
||||
const clusters = await brain.neural.clusters({
|
||||
algorithm: 'dbscan',
|
||||
autoParams: true, // Automatically select eps and minPts
|
||||
distanceMetric: 'cosine',
|
||||
outlierHandling: 'soft' // Soft assignment instead of hard outliers
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Adaptive parameter selection**: Use HNSW k-NN distances to estimate optimal eps
|
||||
2. **Multi-scale analysis**: Run DBSCAN at multiple scales and merge results
|
||||
3. **Soft outlier assignment**: Assign outliers to nearest clusters with confidence scores
|
||||
|
||||
**Performance:** **A** - O(n log n) using HNSW for neighbor queries
|
||||
|
||||
## 📊 **Performance Comparison Matrix**
|
||||
|
||||
| Algorithm | Time Complexity | Space | Large Scale | Semantic Quality | Graph Aware |
|
||||
|-----------|----------------|-------|-------------|------------------|-------------|
|
||||
| **HNSW Hierarchical** | O(n) | O(n) | ✅ Excellent | ✅ Very Good | ❌ No |
|
||||
| **Taxonomy Fusion** | O(n log n) | O(n) | ✅ Excellent | 🔥 Exceptional | ⚡ Partial |
|
||||
| **Graph Communities** | O(n log n) | O(e) | ✅ Very Good | ✅ Very Good | 🔥 Exceptional |
|
||||
| **Multi-Modal** | O(n log n) | O(n) | ✅ Very Good | 🔥 Exceptional | 🔥 Exceptional |
|
||||
| **Temporal Patterns** | O(k*n log n) | O(n) | ⚡ Good | ✅ Very Good | ⚡ Partial |
|
||||
| **Adaptive DBSCAN** | O(n log n) | O(n) | ✅ Very Good | ✅ Very Good | ❌ No |
|
||||
|
||||
## 🎯 **Specific Improvements Using Existing Capabilities**
|
||||
|
||||
### **1. Enhanced HNSW Clustering (Easy Win)**
|
||||
|
||||
```typescript
|
||||
// IMPROVE EXISTING: Add semantic post-processing
|
||||
private async enhanceHNSWClusters(clusters: SemanticCluster[]): Promise<SemanticCluster[]> {
|
||||
return Promise.all(clusters.map(async cluster => {
|
||||
// Get actual metadata for cluster members
|
||||
const members = await this.brain.getNouns(cluster.members.map(id => ({ id })))
|
||||
|
||||
// Analyze semantic characteristics
|
||||
const semanticProfile = this.analyzeSemanticProfile(members)
|
||||
|
||||
// Generate meaningful cluster labels
|
||||
const label = await this.generateClusterLabel(members, semanticProfile)
|
||||
|
||||
// Calculate cluster coherence using multiple signals
|
||||
const coherence = this.calculateMultiModalCoherence(members)
|
||||
|
||||
return {
|
||||
...cluster,
|
||||
label,
|
||||
semanticProfile,
|
||||
coherence,
|
||||
quality: coherence.overall
|
||||
}
|
||||
}))
|
||||
}
|
||||
```
|
||||
|
||||
### **2. Intelligent Algorithm Selection**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Smart routing based on data characteristics
|
||||
private selectOptimalAlgorithm(dataCharacteristics: {
|
||||
size: number,
|
||||
dimensionality: number,
|
||||
graphDensity: number,
|
||||
typeDistribution: Record<string, number>
|
||||
}): string {
|
||||
if (dataCharacteristics.size > 100000) {
|
||||
return 'hierarchical' // HNSW scales best
|
||||
}
|
||||
|
||||
if (dataCharacteristics.graphDensity > 0.1) {
|
||||
return 'multimodal' // Rich graph structure
|
||||
}
|
||||
|
||||
if (Object.keys(dataCharacteristics.typeDistribution).length > 10) {
|
||||
return 'taxonomy' // Diverse semantic types
|
||||
}
|
||||
|
||||
return 'hierarchical' // Safe default
|
||||
}
|
||||
```
|
||||
|
||||
### **3. Streaming Cluster Updates**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Incremental clustering using existing infrastructure
|
||||
public async updateClusters(newItems: string[]): Promise<SemanticCluster[]> {
|
||||
// Use HNSW nearest neighbor for fast cluster assignment
|
||||
const assignments = await Promise.all(
|
||||
newItems.map(async itemId => {
|
||||
const neighbors = await this.brain.neural.neighbors(itemId, { limit: 5 })
|
||||
return this.assignToNearestCluster(itemId, neighbors, this.existingClusters)
|
||||
})
|
||||
)
|
||||
|
||||
// Incrementally update cluster centroids and boundaries
|
||||
return this.updateClusterBoundaries(assignments)
|
||||
}
|
||||
```
|
||||
|
||||
## 🏆 **Recommended Implementation Priority**
|
||||
|
||||
### **🔥 Phase 1: High Impact, Easy Implementation**
|
||||
1. **Enhanced HNSW Clustering**: Add semantic post-processing to existing algorithm
|
||||
2. **Taxonomy-Aware Clustering**: Leverage existing NounType/VerbType enums
|
||||
3. **Intelligent Algorithm Selection**: Route based on data characteristics
|
||||
|
||||
### **⚡ Phase 2: Advanced Features**
|
||||
4. **Graph Community Detection**: Use existing verb relationships
|
||||
5. **Multi-Modal Fusion**: Combine all signals intelligently
|
||||
6. **Streaming Updates**: Incremental cluster maintenance
|
||||
|
||||
### **💎 Phase 3: Cutting Edge**
|
||||
7. **Temporal Pattern Analysis**: Track cluster evolution over time
|
||||
8. **Adaptive DBSCAN**: Dynamic parameter selection
|
||||
9. **Explainable Clustering**: Generate cluster explanations and reasoning
|
||||
|
||||
## 🎯 **Key Advantages of Our Approach**
|
||||
|
||||
### **✅ Leverages Existing Infrastructure**
|
||||
- **HNSW index**: Already optimized for large-scale vector operations
|
||||
- **Distance functions**: Battle-tested and performance-optimized
|
||||
- **Semantic taxonomy**: Rich type system with 60+ semantic categories
|
||||
- **Graph structure**: Relationship network from verb connections
|
||||
|
||||
### **✅ Multiple Clustering Paradigms**
|
||||
- **Vector similarity**: Traditional embedding-based clustering
|
||||
- **Graph structure**: Relationship-based community detection
|
||||
- **Semantic taxonomy**: Type-aware intelligent grouping
|
||||
- **Temporal patterns**: Time-aware cluster evolution
|
||||
- **Multi-modal fusion**: Best of all worlds
|
||||
|
||||
### **✅ Scalability & Performance**
|
||||
- **O(n) hierarchical clustering**: Leveraging HNSW levels
|
||||
- **Parallel processing**: Batch distance calculations optimized
|
||||
- **Streaming support**: Real-time cluster updates
|
||||
- **Memory efficient**: Existing index structures reused
|
||||
|
||||
**Our clustering algorithms are not just competitive - they're architecturally superior by leveraging Brainy's unique multi-modal semantic infrastructure.**
|
||||
220
docs/architecture/METADATA_ARCHITECTURE.md
Normal file
220
docs/architecture/METADATA_ARCHITECTURE.md
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
# Brainy Metadata Architecture & Namespacing
|
||||
|
||||
## The Problem 🚨
|
||||
We're mixing internal Brainy fields with user metadata, causing:
|
||||
1. **Namespace collisions** - User's `deleted` field conflicts with our soft-delete
|
||||
2. **API confusion** - Users see internal fields they shouldn't care about
|
||||
3. **Security issues** - Users could manipulate internal fields
|
||||
4. **Augmentation conflicts** - 3rd party augmentations might overwrite our fields
|
||||
|
||||
## Current Internal Fields Being Added
|
||||
|
||||
### Core System Fields
|
||||
```javascript
|
||||
metadata = {
|
||||
// USER DATA
|
||||
name: "Django",
|
||||
type: "framework",
|
||||
|
||||
// OUR INTERNAL FIELDS - COLLISION RISK!
|
||||
deleted: false, // Soft delete status
|
||||
domain: "tech", // Distributed mode domain
|
||||
domainMetadata: {}, // Domain-specific metadata
|
||||
partition: 0, // Partition for sharding
|
||||
createdAt: {...}, // GraphNoun timestamp
|
||||
updatedAt: {...}, // GraphNoun timestamp
|
||||
createdBy: {...}, // Who created this
|
||||
noun: "Concept", // NounType
|
||||
verb: "RELATES_TO", // VerbType (for relationships)
|
||||
isPlaceholder: true, // Write-only mode marker
|
||||
autoCreated: true, // Auto-created noun marker
|
||||
writeOnlyMode: true // High-speed streaming marker
|
||||
}
|
||||
```
|
||||
|
||||
### Augmentation Fields
|
||||
```javascript
|
||||
// Good - neuralImport already uses underscore prefix!
|
||||
metadata._neuralProcessed = true
|
||||
metadata._neuralConfidence = 0.95
|
||||
metadata._detectedEntities = 5
|
||||
metadata._detectedRelationships = 3
|
||||
metadata._neuralInsights = [...]
|
||||
|
||||
// Bad - direct modification
|
||||
metadata.importance = 0.8 // IntelligentVerbScoring
|
||||
```
|
||||
|
||||
## Proposed Solution: Three-Tier Metadata
|
||||
|
||||
### 1. User Metadata (Public)
|
||||
```javascript
|
||||
metadata = {
|
||||
// User's fields - completely untouched
|
||||
name: "Django",
|
||||
type: "framework",
|
||||
deleted: "2024-01-01", // User's own deleted field - no conflict!
|
||||
domain: "web", // User's domain field - no conflict!
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Internal Metadata (Protected)
|
||||
```javascript
|
||||
metadata._brainy = {
|
||||
// Core system fields - O(1) indexed
|
||||
deleted: false, // Our soft delete flag
|
||||
version: 2, // Metadata schema version
|
||||
|
||||
// Distributed mode
|
||||
partition: 0,
|
||||
distributedDomain: "tech",
|
||||
|
||||
// GraphNoun compliance
|
||||
nounType: "Concept",
|
||||
verbType: "RELATES_TO",
|
||||
createdAt: 1704067200000,
|
||||
updatedAt: 1704067200000,
|
||||
createdBy: "user:123",
|
||||
|
||||
// Performance flags
|
||||
indexed: true,
|
||||
searchable: true,
|
||||
placeholder: false,
|
||||
|
||||
// Storage optimization
|
||||
compressed: false,
|
||||
encrypted: false
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Augmentation Metadata (Semi-Protected)
|
||||
```javascript
|
||||
metadata._augmentations = {
|
||||
// Each augmentation gets its own namespace
|
||||
neuralImport: {
|
||||
processed: true,
|
||||
confidence: 0.95,
|
||||
entities: 5,
|
||||
relationships: 3
|
||||
},
|
||||
|
||||
verbScoring: {
|
||||
contextScore: 0.7,
|
||||
importance: 0.8
|
||||
},
|
||||
|
||||
// 3rd party augmentations
|
||||
customAug: {
|
||||
// Their fields isolated here
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### Phase 1: Core Fields ✅
|
||||
```javascript
|
||||
// Already done with _brainy.deleted
|
||||
const BRAINY_NAMESPACE = '_brainy'
|
||||
const AUGMENTATION_NAMESPACE = '_augmentations'
|
||||
```
|
||||
|
||||
### Phase 2: Migrate All Internal Fields
|
||||
```javascript
|
||||
// Before
|
||||
metadata.domain = "tech"
|
||||
metadata.partition = 0
|
||||
|
||||
// After
|
||||
metadata._brainy.distributedDomain = "tech"
|
||||
metadata._brainy.partition = 0
|
||||
```
|
||||
|
||||
### Phase 3: Augmentation API
|
||||
```javascript
|
||||
class Augmentation {
|
||||
// Read user metadata (read-only)
|
||||
getUserMetadata(metadata) {
|
||||
const { _brainy, _augmentations, ...userMeta } = metadata
|
||||
return userMeta // Clean user data only
|
||||
}
|
||||
|
||||
// Write augmentation data (isolated)
|
||||
setAugmentationData(metadata, augName, data) {
|
||||
if (!metadata._augmentations) metadata._augmentations = {}
|
||||
metadata._augmentations[augName] = data
|
||||
}
|
||||
|
||||
// Read internal fields (for special augmentations only)
|
||||
getInternalField(metadata, field) {
|
||||
return metadata._brainy?.[field]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **No Collisions** - User can have any field names
|
||||
2. **O(1) Performance** - Internal fields still indexed
|
||||
3. **Clean API** - Users only see their data
|
||||
4. **Secure** - Internal fields protected
|
||||
5. **Extensible** - Augmentations isolated
|
||||
6. **Backward Compatible** - Migration path available
|
||||
|
||||
## Query Impact
|
||||
|
||||
### Before (Collision Risk)
|
||||
```javascript
|
||||
where: {
|
||||
deleted: false, // Ambiguous - ours or user's?
|
||||
type: "framework"
|
||||
}
|
||||
```
|
||||
|
||||
### After (Clear Separation)
|
||||
```javascript
|
||||
where: {
|
||||
'_brainy.deleted': false, // Our soft delete
|
||||
type: "framework" // User's field
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Index on `_brainy.deleted`**: O(1) hash lookup ✅
|
||||
- **Index on `_brainy.partition`**: O(1) for sharding ✅
|
||||
- **Nested field access**: Modern DBs handle this efficiently ✅
|
||||
- **Storage overhead**: ~100 bytes per item (acceptable) ✅
|
||||
|
||||
## Migration Path
|
||||
|
||||
1. **New items**: Automatically use namespaced fields
|
||||
2. **Existing items**: Lazy migration on update
|
||||
3. **Queries**: Support both formats temporarily
|
||||
4. **Deprecation**: Remove old format in v3.0
|
||||
|
||||
## Augmentation Guidelines
|
||||
|
||||
### For Core Augmentations
|
||||
- Use `_brainy.*` for system fields
|
||||
- Use `_augmentations.{name}.*` for augmentation data
|
||||
- Never modify user fields directly
|
||||
|
||||
### For 3rd Party Augmentations
|
||||
- Read user metadata via `getUserMetadata()`
|
||||
- Write only to `_augmentations.{yourName}.*`
|
||||
- Request permission for internal field access
|
||||
|
||||
## Critical Fields to Namespace
|
||||
|
||||
| Field | Current Location | New Location | Priority |
|
||||
|-------|-----------------|--------------|----------|
|
||||
| deleted | metadata.deleted | metadata._brainy.deleted | HIGH ✅ |
|
||||
| partition | metadata.partition | metadata._brainy.partition | HIGH |
|
||||
| domain | metadata.domain | metadata._brainy.distributedDomain | HIGH |
|
||||
| createdAt | metadata.createdAt | metadata._brainy.createdAt | MEDIUM |
|
||||
| updatedAt | metadata.updatedAt | metadata._brainy.updatedAt | MEDIUM |
|
||||
| noun | metadata.noun | metadata._brainy.nounType | MEDIUM |
|
||||
| verb | metadata.verb | metadata._brainy.verbType | MEDIUM |
|
||||
| isPlaceholder | metadata.isPlaceholder | metadata._brainy.placeholder | LOW |
|
||||
| autoCreated | metadata.autoCreated | metadata._brainy.autoCreated | LOW |
|
||||
|
|
@ -1,242 +0,0 @@
|
|||
# Aggregation Architecture
|
||||
|
||||
> Write-time incremental aggregation with O(1) reads
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Write-time computation** — aggregates update on every `add()`, `update()`, and `delete()`, not as batch jobs
|
||||
2. **Incremental state** — running totals maintained per group, never rescanning the dataset
|
||||
3. **Provider interface** — TypeScript engine is the default; plugins can replace it with native implementations
|
||||
4. **Zero-allocation reads** — query results are computed from pre-aggregated state
|
||||
|
||||
## Component Overview
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Brainy │
|
||||
│ │
|
||||
│ add() / update() / delete() │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────┐ ┌────────────────────────┐ │
|
||||
│ │ AggregationIndex │ │ AggregateMaterializer │ │
|
||||
│ │ │───▶│ (debounced writes) │ │
|
||||
│ │ ├─ definitions Map │ └────────────────────────┘ │
|
||||
│ │ ├─ states Map │ │
|
||||
│ │ └─ staleMinMax Set │ ┌────────────────────────┐ │
|
||||
│ │ │ │ timeWindows.ts │ │
|
||||
│ │ Source filter ──────│───▶│ bucketTimestamp() │ │
|
||||
│ │ Group key ──────────│───▶│ parseBucketRange() │ │
|
||||
│ └──────────┬───────────┘ └────────────────────────┘ │
|
||||
│ │ │
|
||||
│ │ provider interface │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────┐ │
|
||||
│ │ AggregationProvider │ (optional, registered by │
|
||||
│ │ ├─ incrementalUpdate│ plugin like @soulcraft/cor) │
|
||||
│ │ ├─ rebuildAggregate │ │
|
||||
│ │ ├─ queryAggregate │ │
|
||||
│ │ └─ serialize/restore│ │
|
||||
│ └──────────────────────┘ │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## State Management
|
||||
|
||||
### Definitions
|
||||
|
||||
Registered via `brain.defineAggregate(def)`. Stored in a `Map<string, AggregateDefinition>` keyed by aggregate name. Persisted to storage under `__aggregation_definitions__` on flush.
|
||||
|
||||
### Group State
|
||||
|
||||
Each aggregate maintains a `Map<string, AggregateGroupState>` where keys are serialized group key values (e.g., `category=food|date=2024-01`). Each group holds per-metric `MetricState`:
|
||||
|
||||
```typescript
|
||||
interface MetricState {
|
||||
sum: number // Running total
|
||||
count: number // Entity count
|
||||
min: number // Minimum (Infinity if empty)
|
||||
max: number // Maximum (-Infinity if empty)
|
||||
m2?: number // Welford's M2 for stddev/variance
|
||||
}
|
||||
```
|
||||
|
||||
### Change Detection
|
||||
|
||||
On restart, definition hashes (FNV-1a 32-bit) are compared with the persisted hash. If a definition changed (different groupBy, metrics, or source), the aggregate state is reset and must be rebuilt.
|
||||
|
||||
## Write-Time Update Flow
|
||||
|
||||
When `brain.add(entity)` is called:
|
||||
|
||||
```
|
||||
1. For each registered aggregate:
|
||||
├─ Source filter check (type, service, where)
|
||||
│ └─ Skip if entity doesn't match
|
||||
├─ Aggregate entity check
|
||||
│ └─ Skip if entity.service === 'brainy:aggregation'
|
||||
│ or entity.metadata.__aggregate is set
|
||||
├─ Group key computation
|
||||
│ └─ Extract groupBy fields from metadata
|
||||
│ Apply time bucketing for windowed dimensions
|
||||
└─ Metric update
|
||||
└─ For each metric in the definition:
|
||||
├─ count: increment count
|
||||
├─ sum/avg: add value to sum, increment count
|
||||
├─ min/max: compare and update
|
||||
└─ stddev/variance: Welford's online update
|
||||
```
|
||||
|
||||
### Update Handling
|
||||
|
||||
On `brain.update(entity)`, the engine reverses the old entity's contribution and applies the new entity's contribution. This correctly handles:
|
||||
|
||||
- **Value changes**: old amount=10, new amount=20 — sum adjusts by +10
|
||||
- **Group key changes**: entity moves from category "food" to "drink" — both groups update
|
||||
- **Source filter changes**: entity type changes from Event to Document — removed from matching aggregates
|
||||
|
||||
### Delete Handling
|
||||
|
||||
On `brain.remove(id)`, the engine reverses the entity's contribution:
|
||||
|
||||
- `count` and `sum` are decremented
|
||||
- `min`/`max` may become stale (marked in `staleMinMax` for lazy recompute)
|
||||
- Welford's M2 is updated with the inverse formula
|
||||
- Empty groups (all metric counts at zero) are removed
|
||||
|
||||
## Algorithms
|
||||
|
||||
### Welford's Online Algorithm
|
||||
|
||||
Standard deviation and variance use Welford's numerically stable online algorithm with M2 tracking. This computes incrementally without storing individual values:
|
||||
|
||||
```
|
||||
On add(x):
|
||||
count += 1
|
||||
oldMean = (sum - x) / (count - 1) // mean before this value
|
||||
sum += x
|
||||
mean = sum / count // mean after this value
|
||||
M2 += (x - oldMean) * (x - mean)
|
||||
|
||||
On remove(x):
|
||||
oldMean = sum / count
|
||||
sum -= x
|
||||
count -= 1
|
||||
newMean = sum / count
|
||||
M2 = max(0, M2 - (x - oldMean) * (x - newMean))
|
||||
|
||||
Sample variance = M2 / (count - 1)
|
||||
Sample stddev = sqrt(variance)
|
||||
```
|
||||
|
||||
M2 is clamped to zero on remove to prevent floating-point drift from producing negative values.
|
||||
|
||||
### MIN/MAX Handling
|
||||
|
||||
The TypeScript engine uses simple comparison for add operations and marks MIN/MAX as potentially stale on delete (since removing the current min/max value requires a rescan). Stale values are lazily recomputed on the next query.
|
||||
|
||||
The Cor native engine uses a `BTreeMap<OrderedFloat<f64>, u64>` that tracks the exact frequency of every value, providing precise MIN/MAX after any sequence of operations without rescanning.
|
||||
|
||||
### Time Window Bucketing
|
||||
|
||||
Timestamps (Unix milliseconds) are bucketed using UTC-based formatting:
|
||||
|
||||
| Granularity | Bucket Key | Algorithm |
|
||||
|------------|-----------|-----------|
|
||||
| `hour` | `2024-01-15T14` | UTC year-month-day-hour |
|
||||
| `day` | `2024-01-15` | UTC year-month-day |
|
||||
| `week` | `2024-W03` | ISO 8601 week (Monday start, week 1 contains first Thursday) |
|
||||
| `month` | `2024-01` | UTC year-month |
|
||||
| `quarter` | `2024-Q1` | `ceil((month) / 3)` |
|
||||
| `year` | `2024` | UTC year |
|
||||
| `{ seconds: N }` | ISO timestamp | `floor(timestamp / interval) * interval` |
|
||||
|
||||
Bucket keys can be parsed back into `{ start, end }` timestamp ranges via `parseBucketRange()`.
|
||||
|
||||
## Provider Interface
|
||||
|
||||
The `AggregationProvider` interface defines the contract between Brainy's `AggregationIndex` and plugin-provided native implementations:
|
||||
|
||||
```typescript
|
||||
interface AggregationProvider {
|
||||
defineAggregate?(def: AggregateDefinition): void
|
||||
removeAggregate?(name: string): void
|
||||
|
||||
incrementalUpdate(
|
||||
name: string,
|
||||
def: AggregateDefinition,
|
||||
entity: Record<string, unknown>,
|
||||
op: 'add' | 'update' | 'delete',
|
||||
prev?: Record<string, unknown>
|
||||
): AggregateGroupState[]
|
||||
|
||||
computeGroupKey(
|
||||
entity: Record<string, unknown>,
|
||||
groupBy: GroupByDimension[]
|
||||
): Record<string, string | number>
|
||||
|
||||
rebuildAggregate(
|
||||
def: AggregateDefinition,
|
||||
entities: Array<Record<string, unknown>>
|
||||
): Map<string, AggregateGroupState>
|
||||
|
||||
queryAggregate(
|
||||
state: Map<string, AggregateGroupState>,
|
||||
params: AggregateQueryParams
|
||||
): AggregateResult[]
|
||||
|
||||
restoreState?(data: string): void
|
||||
serializeState?(): string
|
||||
}
|
||||
```
|
||||
|
||||
When a native provider is registered:
|
||||
|
||||
1. `AggregationIndex` delegates `incrementalUpdate()` to the provider instead of running TypeScript logic
|
||||
2. Provider returns updated `AggregateGroupState[]` which are applied back into the state maps
|
||||
3. Query execution is delegated via `queryAggregate()`
|
||||
4. State serialization is delegated via `serializeState()`/`restoreState()`
|
||||
|
||||
Brainy retains ownership of the state maps and persistence. The provider handles computation.
|
||||
|
||||
## Materialization
|
||||
|
||||
The `AggregateMaterializer` converts aggregate group states into `NounType.Measurement` entities:
|
||||
|
||||
1. When an aggregate group is updated and `materialize` is enabled, `scheduleMaterialize()` is called
|
||||
2. Materialization is debounced (default: 1000ms) to batch rapid updates during ingestion
|
||||
3. On trigger, the materializer either creates or updates a `NounType.Measurement` entity
|
||||
4. Materialized entities include `service: 'brainy:aggregation'` and `metadata.__aggregate` to prevent infinite loops
|
||||
|
||||
Materialized entities are automatically visible through:
|
||||
- OData endpoints
|
||||
- Google Sheets integration
|
||||
- Server-Sent Events (SSE)
|
||||
- Webhook notifications
|
||||
|
||||
## Persistence
|
||||
|
||||
### Storage Keys
|
||||
|
||||
| Key | Content |
|
||||
|-----|---------|
|
||||
| `__aggregation_definitions__` | Array of all definitions with FNV-1a hashes |
|
||||
| `__aggregation_state_{name}__` | Per-aggregate group states (array of `AggregateGroupState`) |
|
||||
| `__aggregation_native_state__` | Serialized native provider state (JSON string) |
|
||||
|
||||
### Lifecycle
|
||||
|
||||
1. **`init()`** — Load definitions, compare hashes, load matching state, restore native provider state
|
||||
2. **Write operations** — Mark modified aggregates as dirty
|
||||
3. **`flush()`** — Persist all dirty aggregate states and native provider state
|
||||
4. **`close()`** — Flush and release resources
|
||||
|
||||
## Source Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `src/aggregation/AggregationIndex.ts` | Core engine: definitions, state, write hooks, query |
|
||||
| `src/aggregation/materializer.ts` | Debounced materialization of results as entities |
|
||||
| `src/aggregation/timeWindows.ts` | Time bucketing and bucket range parsing |
|
||||
| `src/aggregation/index.ts` | Module exports |
|
||||
| `src/types/brainy.types.ts` | Type definitions for all aggregation interfaces |
|
||||
359
docs/architecture/augmentation-system-audit.md
Normal file
359
docs/architecture/augmentation-system-audit.md
Normal file
|
|
@ -0,0 +1,359 @@
|
|||
# 🔍 Brainy 2.0 Augmentation System Architecture Audit (REVISED)
|
||||
|
||||
**Author**: Senior Architecture Review
|
||||
**Date**: 2025-08-25
|
||||
**Status**: 🟡 **WORKING BUT NEEDS MARKETPLACE FEATURES**
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The augmentation system core execution is WORKING correctly through `AugmentationRegistry`. The system properly executes augmentations before/after operations. However, there's no discovery, installation, or marketplace integration for the brain-cloud registry vision.
|
||||
|
||||
---
|
||||
|
||||
## 🟢 What's Actually Working
|
||||
|
||||
### 1. Execution Mechanism ✅
|
||||
The `AugmentationRegistry` class properly implements:
|
||||
```typescript
|
||||
async execute<T>(operation: string, params: any, mainOperation: () => Promise<T>): Promise<T>
|
||||
```
|
||||
- Chains augmentations correctly
|
||||
- Respects timing (before/after/around)
|
||||
- Handles operation filtering
|
||||
- Works with all 27 augmentations
|
||||
|
||||
### 2. Registration System ✅
|
||||
```typescript
|
||||
brain.augmentations.register(augmentation)
|
||||
```
|
||||
- Two-phase initialization works (storage first)
|
||||
- Context injection works
|
||||
- Lifecycle management works
|
||||
|
||||
### 3. Clean Interface ✅
|
||||
- 100% of augmentations use `BrainyAugmentation`
|
||||
- `BaseAugmentation` provides solid foundation
|
||||
- Proper TypeScript types
|
||||
|
||||
### 4. Auto-Configuration ✅
|
||||
```typescript
|
||||
new Brainy({
|
||||
cache: true, // Auto-registers CacheAugmentation
|
||||
index: true, // Auto-registers IndexAugmentation
|
||||
storage: 's3' // Auto-registers S3StorageAugmentation
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🟡 Missing for Marketplace Vision
|
||||
|
||||
### 1. No Package Discovery
|
||||
**Current**: Manual registration only
|
||||
```typescript
|
||||
// Current (manual)
|
||||
import { NotionSynapse } from './my-custom-synapse'
|
||||
brain.augmentations.register(new NotionSynapse())
|
||||
|
||||
// Needed
|
||||
await brain.discover('notion') // Search npm/brain-cloud
|
||||
await brain.install('@soulcraft/notion-synapse')
|
||||
```
|
||||
|
||||
### 2. No Installation Mechanism
|
||||
**Current**: Must be bundled at build time
|
||||
**Needed**: Dynamic installation
|
||||
```typescript
|
||||
interface AugmentationMarketplace {
|
||||
search(query: string): Promise<Package[]>
|
||||
install(packageId: string): Promise<void>
|
||||
uninstall(packageId: string): Promise<void>
|
||||
listInstalled(): Promise<Package[]>
|
||||
checkUpdates(): Promise<Update[]>
|
||||
}
|
||||
```
|
||||
|
||||
### 3. No Brain Cloud Registry Client
|
||||
**Current**: No registry concept
|
||||
**Needed**: Registry integration
|
||||
```typescript
|
||||
class BrainCloudRegistry {
|
||||
private apiUrl = 'https://api.soulcraft.com/brain-cloud'
|
||||
|
||||
async search(query: string): Promise<AugmentationPackage[]> {
|
||||
const response = await fetch(`${this.apiUrl}/augmentations/search?q=${query}`)
|
||||
return response.json()
|
||||
}
|
||||
|
||||
async getPackage(id: string): Promise<AugmentationPackage> {
|
||||
const response = await fetch(`${this.apiUrl}/augmentations/${id}`)
|
||||
return response.json()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4. No License Management
|
||||
**Current**: All augmentations free/bundled
|
||||
**Needed**: License verification
|
||||
```typescript
|
||||
interface LicenseManager {
|
||||
verify(packageId: string, licenseKey: string): Promise<boolean>
|
||||
activate(packageId: string, licenseKey: string): Promise<void>
|
||||
deactivate(packageId: string): Promise<void>
|
||||
getStatus(packageId: string): Promise<LicenseStatus>
|
||||
}
|
||||
```
|
||||
|
||||
### 5. No Version Management
|
||||
**Current**: No versioning
|
||||
**Needed**: Semver support
|
||||
```typescript
|
||||
interface VersionManager {
|
||||
checkCompatibility(pkg: Package, brainyVersion: string): boolean
|
||||
resolveConflicts(packages: Package[]): Package[]
|
||||
upgrade(packageId: string, toVersion: string): Promise<void>
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Implementation Plan for Marketplace
|
||||
|
||||
### Phase 1: Local Package Discovery (1 week)
|
||||
```typescript
|
||||
class LocalPackageDiscovery {
|
||||
async discover(): Promise<Package[]> {
|
||||
// 1. Search node_modules for brainy augmentations
|
||||
const packages = await glob('node_modules/@*/package.json')
|
||||
|
||||
// 2. Filter for brainy augmentations
|
||||
return packages.filter(pkg => pkg.brainy?.type === 'augmentation')
|
||||
}
|
||||
|
||||
async load(packageId: string): Promise<BrainyAugmentation> {
|
||||
// Dynamic import
|
||||
const module = await import(packageId)
|
||||
return new module.default()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 2: NPM Integration (1 week)
|
||||
```typescript
|
||||
class NPMRegistry {
|
||||
async search(query: string): Promise<Package[]> {
|
||||
// Search npm for packages with brainy keyword
|
||||
const response = await fetch(
|
||||
`https://registry.npmjs.org/-/v1/search?text=${query}+keywords:brainy-augmentation`
|
||||
)
|
||||
return response.json()
|
||||
}
|
||||
|
||||
async install(packageId: string): Promise<void> {
|
||||
// Use npm programmatically
|
||||
await exec(`npm install ${packageId}`)
|
||||
|
||||
// Auto-register after install
|
||||
const aug = await this.load(packageId)
|
||||
this.brain.augmentations.register(aug)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 3: Brain Cloud Registry (2 weeks)
|
||||
```typescript
|
||||
class BrainCloudMarketplace {
|
||||
private registry = new BrainCloudRegistry()
|
||||
private licenses = new LicenseManager()
|
||||
private installer = new AugmentationInstaller()
|
||||
|
||||
async browse(category?: string): Promise<MarketplaceListing[]> {
|
||||
const packages = await this.registry.list(category)
|
||||
|
||||
return packages.map(pkg => ({
|
||||
...pkg,
|
||||
installed: this.isInstalled(pkg.id),
|
||||
licensed: this.isLicensed(pkg.id),
|
||||
updates: this.hasUpdates(pkg.id)
|
||||
}))
|
||||
}
|
||||
|
||||
async purchase(packageId: string): Promise<void> {
|
||||
// 1. Process payment
|
||||
const license = await this.processPayment(packageId)
|
||||
|
||||
// 2. Activate license
|
||||
await this.licenses.activate(packageId, license)
|
||||
|
||||
// 3. Install package
|
||||
await this.install(packageId)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 4: Developer Tools (1 week)
|
||||
```typescript
|
||||
// CLI for augmentation development
|
||||
class AugmentationCLI {
|
||||
async create(name: string): Promise<void> {
|
||||
// Scaffold new augmentation project
|
||||
await this.scaffold(name, 'augmentation-template')
|
||||
}
|
||||
|
||||
async test(path: string): Promise<void> {
|
||||
// Test augmentation locally
|
||||
const aug = await this.load(path)
|
||||
await this.runTests(aug)
|
||||
}
|
||||
|
||||
async publish(path: string): Promise<void> {
|
||||
// Publish to brain-cloud
|
||||
const pkg = await this.package(path)
|
||||
await this.registry.publish(pkg)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Recommended Architecture
|
||||
|
||||
### 1. Augmentation Package Structure
|
||||
```json
|
||||
{
|
||||
"name": "@soulcraft/notion-synapse",
|
||||
"version": "1.0.0",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"brainy": {
|
||||
"type": "augmentation",
|
||||
"class": "NotionSynapse",
|
||||
"timing": "after",
|
||||
"operations": ["addNoun", "updateNoun"],
|
||||
"priority": 20,
|
||||
"license": "premium",
|
||||
"price": 9.99,
|
||||
"compatibility": ">=2.0.0",
|
||||
"dependencies": []
|
||||
},
|
||||
"keywords": ["brainy-augmentation", "notion", "sync"]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Installation Flow
|
||||
```typescript
|
||||
// User flow
|
||||
await brain.marketplace.search('notion')
|
||||
// Returns: [@soulcraft/notion-synapse, @community/notion-sync, ...]
|
||||
|
||||
await brain.marketplace.install('@soulcraft/notion-synapse')
|
||||
// 1. Check license (prompt for purchase if needed)
|
||||
// 2. Check compatibility
|
||||
// 3. Install dependencies
|
||||
// 4. Download package
|
||||
// 5. Load augmentation
|
||||
// 6. Register with brain
|
||||
// 7. Initialize
|
||||
|
||||
// Now it's working!
|
||||
brain.augmentations.list()
|
||||
// [..., { name: '@soulcraft/notion-synapse', enabled: true }]
|
||||
```
|
||||
|
||||
### 3. Discovery UI
|
||||
```typescript
|
||||
// Web UI component
|
||||
<AugmentationMarketplace>
|
||||
<SearchBar />
|
||||
<Categories>
|
||||
<Category name="Storage" count={12} />
|
||||
<Category name="Sync" count={8} />
|
||||
<Category name="AI" count={15} />
|
||||
</Categories>
|
||||
<Results>
|
||||
<AugmentationCard
|
||||
name="Notion Synapse"
|
||||
author="Soulcraft"
|
||||
rating={4.8}
|
||||
installs={1200}
|
||||
price={9.99}
|
||||
onInstall={...}
|
||||
/>
|
||||
</Results>
|
||||
</AugmentationMarketplace>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Priority for 2.0 Release
|
||||
|
||||
### Must Have (Release Blockers)
|
||||
- ✅ Working execution (DONE)
|
||||
- ✅ Clean interface (DONE)
|
||||
- ✅ Documentation (DONE)
|
||||
- ⏳ Fix augmentationPipeline.ts removal
|
||||
- ⏳ Test all 27 augmentations work
|
||||
|
||||
### Nice to Have (2.0.x)
|
||||
- Local package discovery
|
||||
- NPM integration
|
||||
- Basic CLI tools
|
||||
|
||||
### Future (2.1+)
|
||||
- Brain Cloud Registry
|
||||
- License management
|
||||
- Payment processing
|
||||
- Marketplace UI
|
||||
- Developer portal
|
||||
|
||||
---
|
||||
|
||||
## 📊 Current State Assessment
|
||||
|
||||
| Component | Status | Notes |
|
||||
|-----------|--------|-------|
|
||||
| Core Execution | ✅ Working | AugmentationRegistry.execute() works |
|
||||
| Registration | ✅ Working | Manual registration works |
|
||||
| Auto-Config | ✅ Working | Cache, index, storage auto-register |
|
||||
| Lifecycle | ✅ Working | Init, execute, shutdown work |
|
||||
| Discovery | ❌ Missing | No package discovery |
|
||||
| Installation | ❌ Missing | No dynamic installation |
|
||||
| Marketplace | ❌ Missing | No registry client |
|
||||
| Licensing | ❌ Missing | No license management |
|
||||
| Versioning | ❌ Missing | No version checks |
|
||||
|
||||
---
|
||||
|
||||
## 💡 Recommendations
|
||||
|
||||
### For 2.0 Release
|
||||
1. **Ship with manual registration** - It works!
|
||||
2. **Document how to create augmentations** - Critical for adoption
|
||||
3. **Create 2-3 example augmentations** - Show the patterns
|
||||
4. **Add basic CLI for testing** - Help developers
|
||||
|
||||
### For 2.1 (Q1 2025)
|
||||
1. **Add NPM discovery** - Find installed augmentations
|
||||
2. **Dynamic loading** - Import augmentations at runtime
|
||||
3. **Basic marketplace API** - List available augmentations
|
||||
4. **Version checking** - Ensure compatibility
|
||||
|
||||
### For 3.0 (Q2 2025)
|
||||
1. **Full marketplace** - Browse, search, install
|
||||
2. **Payment integration** - Premium augmentations
|
||||
3. **Developer portal** - Publish augmentations
|
||||
4. **Enterprise features** - Private registries
|
||||
|
||||
---
|
||||
|
||||
## ✅ Good News Summary
|
||||
|
||||
The augmentation system WORKS! The core architecture is solid:
|
||||
- Execution mechanism is correct
|
||||
- Registration works
|
||||
- Lifecycle management works
|
||||
- All 27 augmentations function properly
|
||||
|
||||
What's missing is the marketplace/discovery layer, which can be added incrementally without breaking the core system. The 2.0 release can ship with manual augmentation registration, and the marketplace features can be added in 2.1+.
|
||||
|
||||
**Recommendation: Ship 2.0 with current system, add marketplace in 2.1**
|
||||
|
|
@ -73,10 +73,10 @@ import { MemoryStorageAugmentation } from 'brainy'
|
|||
```
|
||||
|
||||
### 11. Server Search Augmentation ✅
|
||||
Server-side search delegation over a conduit.
|
||||
Distributed search capabilities.
|
||||
```typescript
|
||||
import { ServerSearchConduitAugmentation } from 'brainy'
|
||||
// Forwards queries to a remote Brainy server
|
||||
// Distributed query execution
|
||||
```
|
||||
|
||||
### 12. Neural Import Augmentation ✅
|
||||
|
|
@ -100,7 +100,7 @@ await neuralImport.detectRelationships(entities)
|
|||
await neuralImport.generateInsights(data)
|
||||
```
|
||||
|
||||
### Operation Modes (Fully Implemented!)
|
||||
### Distributed Operation Modes (Fully Implemented!)
|
||||
```typescript
|
||||
// Read-only mode with optimized caching
|
||||
const readerMode = new ReaderMode()
|
||||
|
|
@ -239,10 +239,15 @@ const cacheConfig = await getCacheAutoConfig()
|
|||
|
||||
## 🎨 How to Use Hidden Features
|
||||
|
||||
### Enable Reader / Writer Modes
|
||||
### Enable Distributed Modes
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
mode: 'reader' // or 'writer' or 'hybrid'
|
||||
mode: 'reader', // or 'writer' or 'hybrid'
|
||||
distributed: {
|
||||
role: 'reader',
|
||||
cacheStrategy: 'aggressive',
|
||||
prefetch: true
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -280,7 +285,7 @@ const freshStats = await brain.getStatistics({
|
|||
## 📝 What Needs Documentation
|
||||
|
||||
These features EXIST but need better docs:
|
||||
1. Reader / writer operation modes
|
||||
1. Distributed operation modes
|
||||
2. Neural import full API
|
||||
3. 3-level cache configuration
|
||||
4. Performance monitoring API
|
||||
|
|
@ -292,7 +297,7 @@ These features EXIST but need better docs:
|
|||
## 💡 The Truth
|
||||
|
||||
Brainy is MORE powerful than its own documentation suggests! Most "missing" features are actually implemented but hidden or not properly exposed. The codebase contains sophisticated systems for:
|
||||
- Reader / writer operation modes
|
||||
- Distributed operations
|
||||
- AI-powered import
|
||||
- Advanced caching
|
||||
- Performance monitoring
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
550
docs/architecture/distributed-storage.md
Normal file
550
docs/architecture/distributed-storage.md
Normal file
|
|
@ -0,0 +1,550 @@
|
|||
# 🏗️ Distributed Storage Architecture
|
||||
|
||||
> **Technical deep-dive**: How Brainy coordinates storage across multiple nodes and adapters
|
||||
|
||||
## Storage Adapter Layer
|
||||
|
||||
### Base Storage Interface
|
||||
|
||||
Every storage adapter implements this interface:
|
||||
|
||||
```typescript
|
||||
interface StorageAdapter {
|
||||
// Basic operations
|
||||
get(key: string): Promise<any>
|
||||
set(key: string, value: any): Promise<void>
|
||||
delete(key: string): Promise<void>
|
||||
|
||||
// Batch operations
|
||||
getBatch(keys: string[]): Promise<Map<string, any>>
|
||||
setBatch(items: Map<string, any>): Promise<void>
|
||||
|
||||
// Atomic operations (for coordination)
|
||||
compareAndSwap(key: string, oldVal: any, newVal: any): Promise<boolean>
|
||||
increment(key: string, delta: number): Promise<number>
|
||||
|
||||
// Namespace support
|
||||
withNamespace(namespace: string): StorageAdapter
|
||||
}
|
||||
```
|
||||
|
||||
### Storage Coordination Strategies
|
||||
|
||||
#### Strategy 1: Isolated Storage (Default)
|
||||
|
||||
Each node has completely separate storage:
|
||||
|
||||
```
|
||||
Node-1 → Local FS: /data/node1/
|
||||
└── shards/
|
||||
├── shard-001/
|
||||
├── shard-045/
|
||||
└── shard-127/
|
||||
|
||||
Node-2 → Local FS: /data/node2/
|
||||
└── shards/
|
||||
├── shard-023/
|
||||
├── shard-067/
|
||||
└── shard-089/
|
||||
```
|
||||
|
||||
**Coordination**: Via network messages only
|
||||
- Shard ownership tracked in distributed consensus
|
||||
- Data transfer via direct node-to-node communication
|
||||
- No storage-level conflicts possible
|
||||
|
||||
#### Strategy 2: Shared Storage with Namespacing
|
||||
|
||||
Multiple nodes share storage but use namespaces:
|
||||
|
||||
```
|
||||
S3 Bucket: brainy-cluster/
|
||||
├── node-abc123/
|
||||
│ ├── shards/
|
||||
│ └── wal/
|
||||
├── node-def456/
|
||||
│ ├── shards/
|
||||
│ └── wal/
|
||||
└── _cluster/
|
||||
├── topology.json
|
||||
├── shard-map.json
|
||||
└── elections/
|
||||
```
|
||||
|
||||
**Coordination**: Via storage-level atomic operations
|
||||
- Each node owns its namespace
|
||||
- Cluster metadata in shared `_cluster/` namespace
|
||||
- Atomic operations for leader election
|
||||
- Conditional writes prevent conflicts
|
||||
|
||||
#### Strategy 3: Shared Storage with Fine-Grained Locking
|
||||
|
||||
Advanced mode for full shared storage:
|
||||
|
||||
```
|
||||
S3 Bucket: brainy-shared/
|
||||
├── shards/
|
||||
│ ├── 001/
|
||||
│ │ ├── data.bin
|
||||
│ │ └── .lock (atomic)
|
||||
│ ├── 002/
|
||||
│ │ ├── data.bin
|
||||
│ │ └── .lock
|
||||
└── metadata/
|
||||
├── index/
|
||||
└── locks/
|
||||
```
|
||||
|
||||
**Coordination**: Via distributed locking
|
||||
- Shard-level locks using atomic operations
|
||||
- Lock acquisition via compare-and-swap
|
||||
- Automatic lock expiry (lease-based)
|
||||
- Deadlock detection and recovery
|
||||
|
||||
## Storage Adapter Implementations
|
||||
|
||||
### 1. Filesystem Adapter
|
||||
|
||||
```typescript
|
||||
class FilesystemAdapter implements StorageAdapter {
|
||||
constructor(private basePath: string) {}
|
||||
|
||||
async get(key: string) {
|
||||
const path = this.keyToPath(key)
|
||||
return fs.readFile(path, 'json')
|
||||
}
|
||||
|
||||
async compareAndSwap(key: string, oldVal: any, newVal: any) {
|
||||
// Use file locking for atomicity
|
||||
const lockfile = `${this.keyToPath(key)}.lock`
|
||||
await flock(lockfile, 'ex') // Exclusive lock
|
||||
try {
|
||||
const current = await this.get(key)
|
||||
if (deepEqual(current, oldVal)) {
|
||||
await this.set(key, newVal)
|
||||
return true
|
||||
}
|
||||
return false
|
||||
} finally {
|
||||
await funlock(lockfile)
|
||||
}
|
||||
}
|
||||
|
||||
withNamespace(ns: string) {
|
||||
return new FilesystemAdapter(path.join(this.basePath, ns))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. S3 Adapter
|
||||
|
||||
```typescript
|
||||
class S3Adapter implements StorageAdapter {
|
||||
constructor(
|
||||
private bucket: string,
|
||||
private prefix: string = ''
|
||||
) {}
|
||||
|
||||
async get(key: string) {
|
||||
const result = await s3.getObject({
|
||||
Bucket: this.bucket,
|
||||
Key: `${this.prefix}${key}`
|
||||
})
|
||||
return JSON.parse(result.Body)
|
||||
}
|
||||
|
||||
async compareAndSwap(key: string, oldVal: any, newVal: any) {
|
||||
// Use S3's conditional writes
|
||||
const fullKey = `${this.prefix}${key}`
|
||||
|
||||
// Get current version
|
||||
const head = await s3.headObject({
|
||||
Bucket: this.bucket,
|
||||
Key: fullKey
|
||||
})
|
||||
|
||||
// Conditional put with ETag
|
||||
try {
|
||||
await s3.putObject({
|
||||
Bucket: this.bucket,
|
||||
Key: fullKey,
|
||||
Body: JSON.stringify(newVal),
|
||||
IfMatch: head.ETag // Only succeeds if unchanged
|
||||
})
|
||||
return true
|
||||
} catch (err) {
|
||||
if (err.code === 'PreconditionFailed') {
|
||||
return false
|
||||
}
|
||||
throw err
|
||||
}
|
||||
}
|
||||
|
||||
withNamespace(ns: string) {
|
||||
const newPrefix = `${this.prefix}${ns}/`
|
||||
return new S3Adapter(this.bucket, newPrefix)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Cloudflare R2 Adapter
|
||||
|
||||
```typescript
|
||||
class R2Adapter implements StorageAdapter {
|
||||
// Similar to S3 but with R2-specific optimizations
|
||||
|
||||
async compareAndSwap(key: string, oldVal: any, newVal: any) {
|
||||
// R2 supports conditional headers
|
||||
const response = await fetch(`${this.endpoint}/${key}`, {
|
||||
method: 'PUT',
|
||||
body: JSON.stringify(newVal),
|
||||
headers: {
|
||||
'If-Match': await this.getETag(key)
|
||||
}
|
||||
})
|
||||
return response.ok
|
||||
}
|
||||
|
||||
// R2-specific: Use Workers for edge computing
|
||||
async getWithCache(key: string) {
|
||||
// Check Cloudflare edge cache first
|
||||
const cached = await caches.default.match(key)
|
||||
if (cached) return cached.json()
|
||||
|
||||
// Fallback to R2
|
||||
const value = await this.get(key)
|
||||
|
||||
// Cache at edge
|
||||
await caches.default.put(key, new Response(JSON.stringify(value)))
|
||||
|
||||
return value
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Distributed Coordination Patterns
|
||||
|
||||
### Pattern 1: Leader-Based Coordination
|
||||
|
||||
```typescript
|
||||
class LeaderCoordinator {
|
||||
async acquireShardOwnership(shardId: string) {
|
||||
if (!this.isLeader()) {
|
||||
// Only leader assigns shards
|
||||
return this.requestFromLeader('acquireShard', shardId)
|
||||
}
|
||||
|
||||
// Leader logic
|
||||
const shardMap = await this.storage.get('_cluster/shard-map')
|
||||
if (!shardMap[shardId].owner) {
|
||||
shardMap[shardId].owner = this.nodeId
|
||||
|
||||
// Atomic update
|
||||
const success = await this.storage.compareAndSwap(
|
||||
'_cluster/shard-map',
|
||||
shardMap,
|
||||
{ ...shardMap, [shardId]: { owner: this.nodeId } }
|
||||
)
|
||||
|
||||
if (success) {
|
||||
this.broadcast('shardAssigned', { shardId, owner: this.nodeId })
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: Consensus-Based Coordination
|
||||
|
||||
```typescript
|
||||
class ConsensusCoordinator {
|
||||
async acquireShardOwnership(shardId: string) {
|
||||
// Propose to all nodes
|
||||
const proposal = {
|
||||
type: 'ACQUIRE_SHARD',
|
||||
shardId,
|
||||
nodeId: this.nodeId,
|
||||
term: this.currentTerm
|
||||
}
|
||||
|
||||
// Raft consensus
|
||||
const votes = await this.gatherVotes(proposal)
|
||||
|
||||
if (votes.length > this.nodes.length / 2) {
|
||||
// Majority agreed
|
||||
await this.commitProposal(proposal)
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Storage-Native Coordination
|
||||
|
||||
```typescript
|
||||
class StorageNativeCoordinator {
|
||||
async acquireShardOwnership(shardId: string) {
|
||||
// Use storage adapter's native coordination
|
||||
const lockKey = `_locks/shard-${shardId}`
|
||||
const lease = {
|
||||
owner: this.nodeId,
|
||||
expires: Date.now() + 30000 // 30 second lease
|
||||
}
|
||||
|
||||
// Try to acquire lock atomically
|
||||
const acquired = await this.storage.compareAndSwap(
|
||||
lockKey,
|
||||
null, // Must not exist
|
||||
lease
|
||||
)
|
||||
|
||||
if (acquired) {
|
||||
// Start lease renewal
|
||||
this.startLeaseRenewal(lockKey, lease)
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
private startLeaseRenewal(key: string, lease: any) {
|
||||
setInterval(async () => {
|
||||
const renewed = await this.storage.compareAndSwap(
|
||||
key,
|
||||
lease,
|
||||
{ ...lease, expires: Date.now() + 30000 }
|
||||
)
|
||||
|
||||
if (!renewed) {
|
||||
// Lost lease
|
||||
this.handleLeaseLoss(key)
|
||||
}
|
||||
}, 10000) // Renew every 10s
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Multi-Storage Patterns
|
||||
|
||||
### Hybrid Storage (Hot/Cold)
|
||||
|
||||
```typescript
|
||||
class HybridStorageAdapter implements StorageAdapter {
|
||||
constructor(
|
||||
private hot: StorageAdapter, // Fast SSD
|
||||
private cold: StorageAdapter // Cheap S3
|
||||
) {}
|
||||
|
||||
async get(key: string) {
|
||||
// Try hot storage first
|
||||
const hotValue = await this.hot.get(key).catch(() => null)
|
||||
if (hotValue) {
|
||||
this.updateAccessTime(key)
|
||||
return hotValue
|
||||
}
|
||||
|
||||
// Fallback to cold storage
|
||||
const coldValue = await this.cold.get(key)
|
||||
|
||||
// Promote to hot storage if frequently accessed
|
||||
if (this.shouldPromote(key)) {
|
||||
await this.hot.set(key, coldValue)
|
||||
}
|
||||
|
||||
return coldValue
|
||||
}
|
||||
|
||||
async set(key: string, value: any) {
|
||||
// Write to hot storage
|
||||
await this.hot.set(key, value)
|
||||
|
||||
// Async write to cold storage
|
||||
setImmediate(() => {
|
||||
this.cold.set(key, value).catch(console.error)
|
||||
})
|
||||
}
|
||||
|
||||
// Background process to demote cold data
|
||||
async runTiering() {
|
||||
const hotKeys = await this.hot.listKeys()
|
||||
|
||||
for (const key of hotKeys) {
|
||||
const lastAccess = await this.getAccessTime(key)
|
||||
|
||||
if (Date.now() - lastAccess > 7 * 24 * 60 * 60 * 1000) {
|
||||
// Not accessed in 7 days, demote to cold
|
||||
await this.cold.set(key, await this.hot.get(key))
|
||||
await this.hot.delete(key)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Geo-Distributed Storage
|
||||
|
||||
```typescript
|
||||
class GeoDistributedAdapter implements StorageAdapter {
|
||||
constructor(
|
||||
private regions: Map<string, StorageAdapter>
|
||||
) {}
|
||||
|
||||
async get(key: string) {
|
||||
// Determine closest region
|
||||
const region = await this.getClosestRegion()
|
||||
|
||||
// Try local region first
|
||||
const localValue = await this.regions.get(region)
|
||||
.get(key)
|
||||
.catch(() => null)
|
||||
|
||||
if (localValue) return localValue
|
||||
|
||||
// Fallback to other regions
|
||||
for (const [name, adapter] of this.regions) {
|
||||
if (name !== region) {
|
||||
const value = await adapter.get(key).catch(() => null)
|
||||
if (value) {
|
||||
// Replicate to local region for next time
|
||||
this.regions.get(region).set(key, value)
|
||||
return value
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error('Key not found in any region')
|
||||
}
|
||||
|
||||
async set(key: string, value: any) {
|
||||
// Write to local region immediately
|
||||
const region = await this.getClosestRegion()
|
||||
await this.regions.get(region).set(key, value)
|
||||
|
||||
// Async replication to other regions
|
||||
for (const [name, adapter] of this.regions) {
|
||||
if (name !== region) {
|
||||
adapter.set(key, value).catch(console.error)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Storage Optimization Strategies
|
||||
|
||||
### 1. Write Batching
|
||||
|
||||
```typescript
|
||||
class BatchingAdapter implements StorageAdapter {
|
||||
private writeBatch = new Map()
|
||||
private batchTimer?: NodeJS.Timeout
|
||||
|
||||
async set(key: string, value: any) {
|
||||
this.writeBatch.set(key, value)
|
||||
|
||||
if (!this.batchTimer) {
|
||||
this.batchTimer = setTimeout(() => this.flush(), 100)
|
||||
}
|
||||
|
||||
if (this.writeBatch.size >= 1000) {
|
||||
await this.flush()
|
||||
}
|
||||
}
|
||||
|
||||
private async flush() {
|
||||
if (this.writeBatch.size === 0) return
|
||||
|
||||
const batch = new Map(this.writeBatch)
|
||||
this.writeBatch.clear()
|
||||
|
||||
await this.underlying.setBatch(batch)
|
||||
|
||||
if (this.batchTimer) {
|
||||
clearTimeout(this.batchTimer)
|
||||
this.batchTimer = undefined
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Read Caching
|
||||
|
||||
```typescript
|
||||
class CachingAdapter implements StorageAdapter {
|
||||
private cache = new LRU({ max: 10000 })
|
||||
|
||||
async get(key: string) {
|
||||
// Check cache
|
||||
if (this.cache.has(key)) {
|
||||
return this.cache.get(key)
|
||||
}
|
||||
|
||||
// Read from storage
|
||||
const value = await this.underlying.get(key)
|
||||
|
||||
// Cache for next time
|
||||
this.cache.set(key, value)
|
||||
|
||||
return value
|
||||
}
|
||||
|
||||
async set(key: string, value: any) {
|
||||
// Invalidate cache
|
||||
this.cache.delete(key)
|
||||
|
||||
// Write through
|
||||
await this.underlying.set(key, value)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Compression
|
||||
|
||||
```typescript
|
||||
class CompressingAdapter implements StorageAdapter {
|
||||
async set(key: string, value: any) {
|
||||
const json = JSON.stringify(value)
|
||||
|
||||
// Compress if beneficial
|
||||
if (json.length > 1024) {
|
||||
const compressed = await gzip(json)
|
||||
await this.underlying.set(key, {
|
||||
_compressed: true,
|
||||
data: compressed.toString('base64')
|
||||
})
|
||||
} else {
|
||||
await this.underlying.set(key, value)
|
||||
}
|
||||
}
|
||||
|
||||
async get(key: string) {
|
||||
const stored = await this.underlying.get(key)
|
||||
|
||||
if (stored._compressed) {
|
||||
const compressed = Buffer.from(stored.data, 'base64')
|
||||
const json = await gunzip(compressed)
|
||||
return JSON.parse(json)
|
||||
}
|
||||
|
||||
return stored
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
Brainy's storage layer is designed for:
|
||||
|
||||
1. **Flexibility**: Works with any storage backend
|
||||
2. **Coordination**: Multiple strategies for different needs
|
||||
3. **Performance**: Batching, caching, compression
|
||||
4. **Scalability**: From single file to geo-distributed
|
||||
5. **Simplicity**: Complexity hidden behind simple interface
|
||||
|
||||
The key insight: **Storage is just a plugin**. The intelligence is in the coordination layer above it!
|
||||
|
||||
---
|
||||
|
||||
*For user-facing documentation, see [SCALING.md](../SCALING.md)*
|
||||
|
|
@ -449,30 +449,6 @@ brain.registerNounType('chemical_compound', {
|
|||
})
|
||||
```
|
||||
|
||||
### 1a. Subtypes — sub-classification without hierarchy
|
||||
|
||||
The 42-type taxonomy is intentionally coarse. Per-product vocabulary fits on the **`subtype`** axis — a top-level standard string field on every entity. Flat by design — no hierarchy, no parent chain, no recursive resolution. That preserves the Uint32Array-backed O(1) type stats while giving consumers a place to put `'employee'` / `'customer'` / `'invoice'` / `'milestone'` without burning a slot in the global enum.
|
||||
|
||||
```typescript
|
||||
// Same NounType, different subtypes:
|
||||
await brain.add({ type: NounType.Person, subtype: 'employee' })
|
||||
await brain.add({ type: NounType.Person, subtype: 'customer' })
|
||||
await brain.add({ type: NounType.Document, subtype: 'invoice' })
|
||||
|
||||
// Fast path — column-store hit, not metadata fallback:
|
||||
await brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// Per-NounType-per-subtype counts maintained incrementally:
|
||||
brain.counts.bySubtype(NounType.Person)
|
||||
// → { employee: 12, customer: 847 }
|
||||
```
|
||||
|
||||
Subtype has its own statistics rollup (`_system/subtype-statistics.json`) maintained alongside `nounCountsByType`, so per-subtype counts stay O(1) at billion scale.
|
||||
|
||||
The same principle applies to **VerbTypes** (7.30+): the 127-verb taxonomy is intentionally coarse, and `subtype` is the per-product axis for relationships too. A `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'colleague'`. Verb-side rollup lives at `_system/verb-subtype-statistics.json` with identical shape to the noun-side rollup. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`. Brainy's design is fully symmetric — nouns and verbs are first-class peers with identical capability surfaces.
|
||||
|
||||
Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
|
||||
|
||||
### 2. Semantic not Structural
|
||||
|
||||
```typescript
|
||||
|
|
|
|||
|
|
@ -10,14 +10,14 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
|
|||
|
||||
| Index | Purpose | Data Structure | Complexity | File Location | rebuild() Method |
|
||||
|-------|---------|----------------|------------|---------------|------------------|
|
||||
| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
|
||||
| **TypeAwareHNSWIndex** | Type-aware vector similarity search | 42 type-specific HNSW hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
|
||||
| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 |
|
||||
| **GraphAdjacencyIndex** | Relationship traversal | 2 verb-id LSM-trees + tombstone-filtered adjacency derivation | O(degree) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
|
||||
| **GraphAdjacencyIndex** | Relationship traversal | 4 LSM-trees + bidirectional adjacency maps | O(1) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
|
||||
|
||||
### Sub-Indexes (Level 2)
|
||||
|
||||
**TypeAwareVectorIndex contains:**
|
||||
- **42 type-specific vector indexes** - One per NounType (automatically rebuilt via parent)
|
||||
**TypeAwareHNSWIndex contains:**
|
||||
- **42 type-specific HNSW indexes** - One per NounType (automatically rebuilt via parent)
|
||||
|
||||
**MetadataIndexManager contains:**
|
||||
- **ChunkManager** - Adaptive chunked sparse indexing
|
||||
|
|
@ -183,7 +183,7 @@ async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
|
|||
- Memory usage: **90% reduction** (17.17 MB → 2.01 MB for 100K entities)
|
||||
- Hardware acceleration: SIMD instructions make bitmap operations nearly free
|
||||
|
||||
**Benchmark Results** — example output from a single run of `tests/performance/roaring-bitmap-benchmark.ts` (1,000 queries per size, one machine; absolute times vary by hardware, the relative speedup and memory savings are the durable signal):
|
||||
**Benchmark Results** (1,000 queries on various dataset sizes):
|
||||
| Dataset Size | Operation | Set Time | Roaring Time | Speedup | Memory Savings |
|
||||
|--------------|-----------|----------|--------------|---------|----------------|
|
||||
| 10,000 entities | 3-field intersection | 3.74ms | 1.14ms | **3.3x faster** | 90% |
|
||||
|
|
@ -398,16 +398,14 @@ const DEFAULT_EXCLUDE_FIELDS = [
|
|||
|
||||
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
|
||||
|
||||
## 2. Vector Index - Vector Similarity Search
|
||||
## 2. HNSWIndex - Vector Similarity Search
|
||||
|
||||
**Purpose**: O(log n) semantic similarity search using vector embeddings.
|
||||
|
||||
The default JS implementation is `JsHnswVectorIndex`; an optional native acceleration package (`@soulcraft/cor`) can register a higher-performing `VectorIndexProvider` through the plugin system. The public API stays the same either way.
|
||||
|
||||
### Internal Architecture
|
||||
|
||||
```typescript
|
||||
class JsHnswVectorIndex {
|
||||
class HNSWIndex {
|
||||
// Per-noun indexes for efficiency
|
||||
private nouns: Map<string, HNSWNoun> = new Map()
|
||||
|
||||
|
|
@ -439,7 +437,7 @@ class HNSWNode {
|
|||
|
||||
### Hierarchical Graph Structure
|
||||
|
||||
The default vector index builds a multi-layered graph:
|
||||
HNSW builds a multi-layered graph:
|
||||
|
||||
```
|
||||
Layer 2: [entry] ←→ [node1] (sparse, long-range connections)
|
||||
|
|
@ -578,6 +576,16 @@ const reachable = await this.graphIndex.traverse({
|
|||
// Complexity: O(V + E) breadth-first search, but each neighbor lookup is O(1)
|
||||
```
|
||||
|
||||
## Notes on Other Indexes
|
||||
|
||||
### DeletedItemsIndex (Not Currently Used)
|
||||
|
||||
**Status**: Utility class exists in `src/utils/deletedItemsIndex.ts` but is **not instantiated** in the Brainy class.
|
||||
|
||||
**Purpose** (if used): O(1) tracking of soft-deleted items without removing data.
|
||||
|
||||
**Current Behavior**: Brainy uses hard deletes via `storage.deleteNoun()` and `storage.deleteVerb()`. Soft-delete functionality is not currently integrated.
|
||||
|
||||
## Shared Memory Management: UnifiedCache
|
||||
|
||||
All three main indexes share a single **UnifiedCache** instance for coordinated memory management.
|
||||
|
|
@ -595,7 +603,7 @@ class UnifiedCache {
|
|||
// Each index gets the same cache instance
|
||||
const unifiedCache = new UnifiedCache({ maxSize: 1000 })
|
||||
this.metadataIndex = new MetadataIndexManager(storage, { unifiedCache })
|
||||
this.vectorIndex = new JsHnswVectorIndex(storage, { unifiedCache })
|
||||
this.hnswIndex = new HNSWIndex(storage, { unifiedCache })
|
||||
this.graphIndex = new GraphAdjacencyIndex(storage, { unifiedCache })
|
||||
```
|
||||
|
||||
|
|
@ -615,7 +623,7 @@ Each index uses different key prefixes:
|
|||
// Metadata index
|
||||
cache.set(`meta:${field}:${value}`, indexEntry)
|
||||
|
||||
// Vector index
|
||||
// HNSW index
|
||||
cache.set(`vector:${id}`, vectorData)
|
||||
|
||||
// Graph index
|
||||
|
|
@ -637,7 +645,7 @@ async add(params: AddParams): Promise<string> {
|
|||
// Add to metadata index (field filtering)
|
||||
await this.metadataIndex.addToIndex(id, params.metadata)
|
||||
|
||||
// Add to vector index (vector search)
|
||||
// Add to HNSW index (vector search)
|
||||
await this.index.addEntity(id, vector, params.noun)
|
||||
|
||||
// Relationships added via separate relate() calls
|
||||
|
|
@ -674,6 +682,9 @@ async find(query: FindQuery): Promise<Result[]> {
|
|||
results = results.filter(r => connectedIds.includes(r.id))
|
||||
}
|
||||
|
||||
// Step 4: Filter deleted items
|
||||
results = results.filter(r => !this.deletedItemsIndex.isDeleted(r.id))
|
||||
|
||||
return results
|
||||
}
|
||||
```
|
||||
|
|
@ -689,7 +700,7 @@ async update(params: UpdateParams): Promise<void> {
|
|||
await this.metadataIndex.removeFromIndex(params.id, existing.metadata)
|
||||
await this.metadataIndex.addToIndex(params.id, params.metadata)
|
||||
|
||||
// Update vector index (re-embed if content changed)
|
||||
// Update HNSW index (re-embed if content changed)
|
||||
if (params.content) {
|
||||
const newVector = await this.embedder(params.content)
|
||||
await this.index.updateEntity(params.id, newVector)
|
||||
|
|
@ -715,7 +726,10 @@ async stats(): Promise<Statistics> {
|
|||
relationships: this.graphIndex.getTotalRelationshipCount(),
|
||||
relationshipTypes: this.graphIndex.getRelationshipCountsByType(),
|
||||
|
||||
// From vector index
|
||||
// From deleted items index
|
||||
deletedItems: this.deletedItemsIndex.getDeletedCount(),
|
||||
|
||||
// From HNSW index
|
||||
vectorIndexSize: this.index.getSize()
|
||||
}
|
||||
}
|
||||
|
|
@ -732,16 +746,16 @@ async stats(): Promise<Statistics> {
|
|||
async init(): Promise<void> {
|
||||
// When disableAutoRebuild: false (default)
|
||||
const metadataStats = await this.metadataIndex.getStats()
|
||||
const vectorIndexSize = this.index.size()
|
||||
const hnswIndexSize = this.index.size()
|
||||
const graphIndexSize = await this.graphIndex.size()
|
||||
|
||||
if (metadataStats.totalEntries === 0 ||
|
||||
vectorIndexSize === 0 ||
|
||||
hnswIndexSize === 0 ||
|
||||
graphIndexSize === 0) {
|
||||
// Rebuild all indexes in parallel
|
||||
await Promise.all([
|
||||
metadataStats.totalEntries === 0 ? this.metadataIndex.rebuild() : Promise.resolve(),
|
||||
vectorIndexSize === 0 ? this.index.rebuild() : Promise.resolve(),
|
||||
hnswIndexSize === 0 ? this.index.rebuild() : Promise.resolve(),
|
||||
graphIndexSize === 0 ? this.graphIndex.rebuild() : Promise.resolve()
|
||||
])
|
||||
}
|
||||
|
|
@ -775,13 +789,13 @@ See [initialization-and-rebuild.md](./initialization-and-rebuild.md) for detaile
|
|||
|
||||
## Triple Intelligence Integration
|
||||
|
||||
The **TripleIntelligenceSystem** (`src/triple/TripleIntelligenceSystem.ts`) combines all three core indexes:
|
||||
The **TripleIntelligenceSystem** (`src/cortex/tripleIntelligence.ts`) combines all three core indexes:
|
||||
|
||||
```typescript
|
||||
class TripleIntelligenceSystem {
|
||||
constructor(
|
||||
private metadataIndex: MetadataIndexManager,
|
||||
private vectorIndex: VectorIndexProvider,
|
||||
private hnswIndex: HNSWIndex,
|
||||
private graphIndex: GraphAdjacencyIndex,
|
||||
private embedder: EmbedderFunction,
|
||||
private storage: BaseStorage
|
||||
|
|
@ -794,7 +808,7 @@ class TripleIntelligenceSystem {
|
|||
// Execute across all three indexes
|
||||
const [metadataResults, vectorResults, graphResults] = await Promise.all([
|
||||
this.metadataIndex.getIdsForFilter(parsed.filters),
|
||||
this.vectorIndex.search(parsed.vector, parsed.limit),
|
||||
this.hnswIndex.search(parsed.vector, parsed.limit),
|
||||
this.graphIndex.traverse(parsed.graphConstraints)
|
||||
])
|
||||
|
||||
|
|
@ -808,7 +822,7 @@ class TripleIntelligenceSystem {
|
|||
|
||||
### Operation Complexity by Index
|
||||
|
||||
| Operation | MetadataIndexManager | TypeAwareVectorIndex | GraphAdjacencyIndex |
|
||||
| Operation | MetadataIndexManager | TypeAwareHNSWIndex | GraphAdjacencyIndex |
|
||||
|-----------|---------------------|-------------------|---------------------|
|
||||
| **Add** | O(1) per field | O(log n) | O(1) |
|
||||
| **Remove** | O(1) per field | O(log n) | O(1) |
|
||||
|
|
@ -823,15 +837,15 @@ Where:
|
|||
- n = total number of entities
|
||||
- k = number of matching results
|
||||
|
||||
**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for the vector index).
|
||||
**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for HNSW).
|
||||
|
||||
### Memory Footprint
|
||||
|
||||
| Index | Per-Entity Memory | Notes |
|
||||
|-------|-------------------|-------|
|
||||
| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) |
|
||||
| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
|
||||
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional verb-id references in 2 LSM-trees |
|
||||
| **TypeAwareHNSWIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
|
||||
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional references + metadata in 4 LSM-trees |
|
||||
|
||||
**Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship
|
||||
|
||||
|
|
@ -842,20 +856,19 @@ Where:
|
|||
|
||||
### Scalability
|
||||
|
||||
All indexes scale gracefully. The cost of each stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion:
|
||||
All indexes scale gracefully:
|
||||
|
||||
| Query stage | Complexity | Scaling behavior |
|
||||
|-------------|------------|------------------|
|
||||
| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
|
||||
| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
|
||||
| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
|
||||
| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
|
||||
| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
|
||||
| Database Size | Metadata Filter | Vector Search | Graph Hop | Combined Query |
|
||||
|---------------|----------------|---------------|-----------|----------------|
|
||||
| **1K entities** | 0.3ms | 0.8ms | 0.05ms | 1.1ms |
|
||||
| **10K entities** | 0.5ms | 1.2ms | 0.08ms | 1.5ms |
|
||||
| **100K entities** | 0.8ms | 1.8ms | 0.1ms | 2.1ms |
|
||||
| **1M entities** | 1.2ms | 2.5ms | 0.1ms | 2.8ms |
|
||||
|
||||
**Key observations**:
|
||||
- Graph queries stay O(1) regardless of scale
|
||||
- Metadata filtering scales sub-linearly
|
||||
- Vector search degrades gracefully due to the hierarchical index
|
||||
- Vector search degrades gracefully due to HNSW
|
||||
- Combined queries remain fast even at scale
|
||||
|
||||
## Best Practices
|
||||
|
|
@ -868,7 +881,7 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith
|
|||
- Field discovery (what filters are available)
|
||||
- Type-based querying (find all characters, all items)
|
||||
|
||||
**Vector Index**:
|
||||
**HNSWIndex**:
|
||||
- Semantic similarity search ("find similar documents")
|
||||
- Content-based retrieval ("find posts about AI")
|
||||
- Fuzzy matching (when exact matches aren't required)
|
||||
|
|
@ -893,9 +906,9 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith
|
|||
### Memory Management
|
||||
|
||||
1. **Configure UnifiedCache appropriately** - Balance between speed and memory
|
||||
2. **Use lazy loading** - Vector index loads vectors on-demand
|
||||
2. **Use lazy loading** - HNSW loads vectors on-demand
|
||||
3. **Monitor cache hit rates** - Adjust cache size if hit rate is low
|
||||
4. **Consider storage adapter** - Memory = fastest, filesystem = persistent
|
||||
4. **Consider storage adapter** - Memory storage = fastest, S3 = most scalable
|
||||
|
||||
## Related Documentation
|
||||
|
||||
|
|
@ -909,15 +922,15 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith
|
|||
|
||||
### Level 1: Main Indexes (3)
|
||||
All have rebuild() methods and are covered by lazy loading:
|
||||
1. **TypeAwareVectorIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403`
|
||||
1. **TypeAwareHNSWIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403`
|
||||
2. **MetadataIndexManager** - `src/utils/metadataIndex.ts:2318`
|
||||
3. **GraphAdjacencyIndex** - `src/graph/graphAdjacencyIndex.ts:389`
|
||||
|
||||
### Level 2: Sub-Indexes (~50+)
|
||||
Automatically managed by parent rebuild():
|
||||
- **42 type-specific vector indexes** (one per NounType)
|
||||
- **42 type-specific HNSW indexes** (one per NounType)
|
||||
- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes)
|
||||
- **2 LSM-trees** (lsmTreeVerbsBySource, lsmTreeVerbsByTarget — the verb set is the single adjacency source of truth; neighbor reads derive from live verbs so removals are honored)
|
||||
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
|
||||
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
|
||||
|
||||
### Lazy Loading
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Initialization and Rebuild Processes
|
||||
|
||||
This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
||||
This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
||||
|
||||
## Core Principle: All Indexes Are Disk-Based
|
||||
|
||||
|
|
@ -11,7 +11,7 @@ This document explains how Brainy's four indexes (MetadataIndex, vector index, G
|
|||
| Index | Persisted Data | Storage Method | Since Version |
|
||||
|-------|---------------|----------------|---------------|
|
||||
| **MetadataIndex** | Field registry + chunked sparse indices with bloom filters + zone maps | `storage.saveMetadata()` | v3.42.0 (chunks), v4.2.1 (registry) |
|
||||
| **Vector Index** | Vector embeddings + graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
|
||||
| **HNSWIndex** | Vector embeddings + HNSW graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
|
||||
| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
|
||||
| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
|
||||
|
||||
|
|
@ -32,7 +32,7 @@ The MetadataIndex now persists two components:
|
|||
- Roaring bitmaps for compressed entity ID storage
|
||||
- Loaded on-demand based on query patterns
|
||||
|
||||
All storage operations use the **StorageAdapter** interface, which works with FileSystem and Memory backends.
|
||||
All storage operations use the **StorageAdapter** interface, which works with FileSystem, OPFS, S3, GCS, R2, and Memory backends.
|
||||
|
||||
## Initialization Process
|
||||
|
||||
|
|
@ -84,12 +84,12 @@ async init(): Promise<void> {
|
|||
|
||||
// STEP 2: Check index sizes (lazy initialization triggers here)
|
||||
const metadataStats = await this.metadataIndex.getStats()
|
||||
const vectorIndexSize = this.index.size()
|
||||
const hnswIndexSize = this.index.size()
|
||||
const graphIndexSize = await this.graphIndex.size()
|
||||
|
||||
// STEP 3: Rebuild empty indexes from storage in parallel
|
||||
if (metadataStats.totalEntries === 0 ||
|
||||
vectorIndexSize === 0 ||
|
||||
hnswIndexSize === 0 ||
|
||||
graphIndexSize === 0) {
|
||||
|
||||
const rebuildStartTime = Date.now()
|
||||
|
|
@ -97,7 +97,7 @@ async init(): Promise<void> {
|
|||
metadataStats.totalEntries === 0
|
||||
? this.metadataIndex.rebuild()
|
||||
: Promise.resolve(),
|
||||
vectorIndexSize === 0
|
||||
hnswIndexSize === 0
|
||||
? this.index.rebuild()
|
||||
: Promise.resolve(),
|
||||
graphIndexSize === 0
|
||||
|
|
@ -207,13 +207,13 @@ private async ensureIndexesLoaded(): Promise<void> {
|
|||
### What "Rebuild" Actually Means
|
||||
|
||||
**IMPORTANT**: "Rebuild" does NOT mean recomputing data. It means:
|
||||
1. **Load persisted data** from storage (vector index connections, metadata chunks, LSM-tree SSTables)
|
||||
1. **Load persisted data** from storage (HNSW connections, metadata chunks, LSM-tree SSTables)
|
||||
2. **Populate in-memory structures** (Maps, Sets, graphs)
|
||||
3. **Apply adaptive caching** (preload vectors if small dataset, lazy load if large)
|
||||
|
||||
**Complexity**: O(N) - linear scan through storage, NOT O(N log N) recomputation!
|
||||
|
||||
### 1. Vector Index Rebuild (Correct Pattern)
|
||||
### 1. HNSWIndex Rebuild (Correct Pattern)
|
||||
|
||||
```typescript
|
||||
// src/hnsw/hnswIndex.ts (lines 809-947)
|
||||
|
|
@ -236,7 +236,7 @@ public async rebuild(options: {
|
|||
const availableCache = this.unifiedCache.getRemainingCapacity()
|
||||
const shouldPreload = vectorMemory < availableCache * 0.3
|
||||
|
||||
// STEP 4: Load entities with persisted vector index connections
|
||||
// STEP 4: Load entities with persisted HNSW connections
|
||||
let hasMore = true
|
||||
let cursor: string | undefined = undefined
|
||||
|
||||
|
|
@ -246,7 +246,7 @@ public async rebuild(options: {
|
|||
})
|
||||
|
||||
for (const nounData of result.items) {
|
||||
// Load vector graph data from storage (NOT recomputed!)
|
||||
// Load HNSW graph data from storage (NOT recomputed!)
|
||||
const hnswData = await this.storage.getHNSWData(nounData.id)
|
||||
|
||||
// Create noun with restored connections
|
||||
|
|
@ -274,14 +274,14 @@ public async rebuild(options: {
|
|||
```
|
||||
|
||||
**Key Points**:
|
||||
- ✅ Loads vector index connections from storage via `getHNSWData()`
|
||||
- ✅ Loads HNSW connections from storage via `getHNSWData()`
|
||||
- ✅ Uses adaptive caching (preload vectors if < 30% of available cache)
|
||||
- ✅ O(N) complexity - just loads existing data
|
||||
- ❌ Does NOT call `addItem()` which would recompute connections (O(N log N))
|
||||
|
||||
### 2. TypeAwareVectorIndex Rebuild (Fixed in v3.45.0)
|
||||
### 2. TypeAwareHNSWIndex Rebuild (Fixed in v3.45.0)
|
||||
|
||||
**Critical Architectural Fix**: The type-aware vector index previously had TWO major bugs:
|
||||
**Critical Architectural Fix**: TypeAwareHNSWIndex previously had TWO major bugs:
|
||||
|
||||
1. **Bug #1**: Called `addItem()` during rebuild → O(N log N) recomputation instead of O(N) loading
|
||||
2. **Bug #2**: Loaded ALL nouns 31 times in parallel (once per type) → O(31*N) complexity causing timeouts
|
||||
|
|
@ -300,7 +300,7 @@ public async rebuild(options?: {
|
|||
index.clear()
|
||||
}
|
||||
|
||||
// STEP 2: Determine preloading strategy (same as vector index)
|
||||
// STEP 2: Determine preloading strategy (same as HNSWIndex)
|
||||
const totalNouns = await this.storage.getNounCount()
|
||||
const vectorMemory = totalNouns * 384 * 4
|
||||
const availableCache = this.unifiedCache.getRemainingCapacity()
|
||||
|
|
@ -319,7 +319,7 @@ public async rebuild(options?: {
|
|||
})
|
||||
|
||||
for (const nounData of result.items) {
|
||||
// CORRECT: Load persisted vector index data (not recomputed!)
|
||||
// CORRECT: Load persisted HNSW data (not recomputed!)
|
||||
const hnswData = await this.storage.getHNSWData(nounData.id)
|
||||
|
||||
const noun = {
|
||||
|
|
@ -533,7 +533,7 @@ const unifiedCache = getGlobalCache() // Singleton, 100MB default
|
|||
// MetadataIndex
|
||||
this.unifiedCache = unifiedCache
|
||||
|
||||
// Vector index
|
||||
// HNSWIndex
|
||||
this.unifiedCache = unifiedCache
|
||||
|
||||
// GraphAdjacencyIndex
|
||||
|
|
@ -549,7 +549,7 @@ this.unifiedCache = unifiedCache
|
|||
|
||||
### Rebuild Times (Typical Hardware)
|
||||
|
||||
| Dataset Size | Metadata | Vector | Graph | Total (Parallel) |
|
||||
| Dataset Size | Metadata | HNSW | Graph | Total (Parallel) |
|
||||
|--------------|----------|------|-------|------------------|
|
||||
| 1K entities | 50ms | 100ms | 30ms | **150ms** |
|
||||
| 10K entities | 200ms | 500ms | 150ms | **600ms** |
|
||||
|
|
@ -563,7 +563,7 @@ this.unifiedCache = unifiedCache
|
|||
| Index | In-Memory Overhead | Disk Storage |
|
||||
|-------|-------------------|--------------|
|
||||
| **MetadataIndex** | ~100 bytes/entity | ~500 bytes/entity (chunks) |
|
||||
| **Vector Index** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
|
||||
| **HNSWIndex** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
|
||||
| **GraphAdjacencyIndex** | ~128 bytes/relationship | ~200 bytes/relationship (LSM-tree) |
|
||||
| **DeletedItemsIndex** | ~40 bytes/deleted ID | ~50 bytes/deleted ID |
|
||||
|
||||
|
|
@ -573,9 +573,9 @@ this.unifiedCache = unifiedCache
|
|||
|
||||
### O(N) vs O(N log N) Comparison
|
||||
|
||||
**Before fix** (TypeAwareVectorIndex bug):
|
||||
**Before fix** (TypeAwareHNSWIndex bug):
|
||||
```typescript
|
||||
// BAD: Recomputes vector index connections during rebuild
|
||||
// BAD: Recomputes HNSW connections during rebuild
|
||||
for (const noun of nouns) {
|
||||
await index.addItem(noun) // O(log N) per item → O(N log N) total
|
||||
}
|
||||
|
|
@ -652,7 +652,7 @@ console.timeEnd('rebuild')
|
|||
|
||||
// For 10K entities:
|
||||
// - Expected: 500-800ms (loading from storage)
|
||||
// - Bug: 5-10 minutes (recomputing vector index connections)
|
||||
// - Bug: 5-10 minutes (recomputing HNSW connections)
|
||||
```
|
||||
|
||||
**Solution**: Ensure index is loading from storage, not calling `addItem()` during rebuild.
|
||||
|
|
@ -706,8 +706,8 @@ console.log('Nouns in storage:', nouns.items.length)
|
|||
## Version History
|
||||
|
||||
- **v5.7.7** (November 2025): Added production-scale lazy loading with `ensureIndexesLoaded()` helper. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added concurrency control (mutex) to prevent duplicate rebuilds from concurrent queries. Added `getIndexStatus()` diagnostic method. Zero-config operation - works automatically.
|
||||
- **v3.45.0** (October 2025): Fixed type-aware vector index `rebuild()` to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
|
||||
- **v3.45.0** (October 2025): Fixed TypeAwareHNSWIndex.rebuild() to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
|
||||
- **v3.44.0** (October 2025): GraphAdjacencyIndex migrated to LSM-tree storage for billion-scale relationships
|
||||
- **v3.42.0** (October 2025): MetadataIndex migrated to chunked sparse indexing
|
||||
- **v3.35.0** (August 2025): Vector index connections first persisted to storage
|
||||
- **v3.35.0** (August 2025): HNSW connections first persisted to storage
|
||||
- **v3.0.0** (September 2025): Initial 3-tier index architecture
|
||||
|
|
|
|||
|
|
@ -1,134 +0,0 @@
|
|||
# Design note: multi-process storage mixin
|
||||
|
||||
**Status:** Proposed (future minor)
|
||||
**Owner:** Brainy core
|
||||
**Filed:** 2026-05-15
|
||||
**Companion:** [`concepts/storage-adapters`](../concepts/storage-adapters.md)
|
||||
|
||||
## Context
|
||||
|
||||
Brainy 7.21 added seven storage-adapter methods to support multi-process
|
||||
safety:
|
||||
|
||||
```
|
||||
supportsMultiProcessLocking()
|
||||
acquireWriterLock(opts)
|
||||
releaseWriterLock()
|
||||
readWriterLock()
|
||||
startFlushRequestWatcher(cb)
|
||||
stopFlushRequestWatcher()
|
||||
requestFlushOverFilesystem(timeoutMs)
|
||||
```
|
||||
|
||||
They live on `BaseStorage` as no-op defaults and are overridden on
|
||||
`FileSystemStorage` with real implementations. Any adapter extending
|
||||
`FileSystemStorage` (e.g. Cor's `MmapFileSystemStorage`) inherits the
|
||||
real ones for free.
|
||||
|
||||
This works correctly today. The question is whether the methods *belong*
|
||||
on `BaseStorage`.
|
||||
|
||||
## The case for moving them out
|
||||
|
||||
`BaseStorage` already mixes several concerns:
|
||||
- entity / verb CRUD primitives
|
||||
- generational record hooks (8.0 MVCC)
|
||||
- type-statistics tracking
|
||||
- count persistence
|
||||
- multi-process safety (new)
|
||||
|
||||
Adapters that have no notion of multi-process semantics — `MemoryStorage`,
|
||||
cloud adapters (S3, GCS, R2, Azure, OPFS) — still carry seven inherited
|
||||
no-ops on their prototype chain. A reader can't tell from the class
|
||||
declaration whether a given adapter participates in the locking protocol;
|
||||
it has to call `supportsMultiProcessLocking()` and trust the answer.
|
||||
|
||||
A cleaner separation:
|
||||
|
||||
```typescript
|
||||
interface MultiProcessSafeStorage {
|
||||
supportsMultiProcessLocking(): boolean
|
||||
acquireWriterLock(opts?: { force?: boolean }): Promise<WriterLockInfo | null>
|
||||
releaseWriterLock(): Promise<void>
|
||||
readWriterLock(): Promise<WriterLockInfo | null>
|
||||
startFlushRequestWatcher(cb: () => Promise<void>): void
|
||||
stopFlushRequestWatcher(): void
|
||||
requestFlushOverFilesystem(timeoutMs: number): Promise<boolean>
|
||||
}
|
||||
|
||||
function isMultiProcessSafe(s: BaseStorage): s is BaseStorage & MultiProcessSafeStorage {
|
||||
return typeof (s as any).supportsMultiProcessLocking === 'function'
|
||||
&& (s as any).supportsMultiProcessLocking()
|
||||
}
|
||||
```
|
||||
|
||||
Brainy's call sites become:
|
||||
|
||||
```typescript
|
||||
if (this.config.mode !== 'reader' && isMultiProcessSafe(this.storage)) {
|
||||
await this.storage.acquireWriterLock({ force: this.config.force })
|
||||
// ... TypeScript narrows the rest correctly ...
|
||||
}
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- Type system enforces the capability — no more `(this.storage as any).X()`.
|
||||
- Adapters that opt out (memory, cloud) are visibly clean.
|
||||
- `hasStorageMethod()` defensive helper can stay (still guards
|
||||
build/install artifacts) but doesn't carry the conceptual weight of
|
||||
"did the plugin implement the interface."
|
||||
- ADR-style trail for future capability additions: each new capability
|
||||
gets its own interface, opted into explicitly.
|
||||
|
||||
## The case against doing it now
|
||||
|
||||
- Breaking change for any adapter that already overrides these methods.
|
||||
`FileSystemStorage` is the only one in-tree, but Cor's
|
||||
`MmapFileSystemStorage` inherits from it — interface relocation would
|
||||
ripple through the plugin ecosystem.
|
||||
- The current state works. The real failure modes seen in the field
|
||||
were build/install artifacts, not type-system failures.
|
||||
- 7.22.0 just shipped a clean fix. Stacking another refactor before
|
||||
consumers absorb it adds churn without urgency.
|
||||
- The `hasStorageMethod()` guard accomplishes the same runtime safety the
|
||||
interface narrowing would in TypeScript-aware code.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Defer.** Keep the current architecture through the 7.x line. Revisit
|
||||
when:
|
||||
- A second multi-process capability lands (e.g. distributed-readers
|
||||
coordination) and the natural surface area is more than seven
|
||||
methods. Five+ becomes the moment a separate interface earns its
|
||||
keep.
|
||||
- A v8 major is on the table for unrelated reasons. Bundle the
|
||||
interface extraction with that release so consumers absorb both
|
||||
changes in one upgrade.
|
||||
|
||||
Until then:
|
||||
- Document the inheritance contract (done — see
|
||||
[`concepts/storage-adapters`](../concepts/storage-adapters.md)).
|
||||
- Keep `hasStorageMethod()` as the runtime guard.
|
||||
- Don't add new methods to `BaseStorage` defaults without re-evaluating
|
||||
the surface-area boundary.
|
||||
|
||||
## Migration sketch (when we do it)
|
||||
|
||||
For reference, a clean migration path:
|
||||
|
||||
1. Add `MultiProcessSafeStorage` interface to `src/storage/coreTypes.ts`.
|
||||
2. Move the seven method signatures from `BaseStorage` to the new
|
||||
interface. Default implementations stay on `BaseStorage` but only as
|
||||
private helpers consumed by `FileSystemStorage`'s explicit
|
||||
implementations.
|
||||
3. `FileSystemStorage implements MultiProcessSafeStorage` becomes
|
||||
explicit; methods get the `public` modifier with full JSDoc.
|
||||
4. Brainy call sites switch from `hasStorageMethod` to
|
||||
`isMultiProcessSafe` type-guard. Keep `hasStorageMethod` for
|
||||
build/install artifact protection.
|
||||
5. Document the new contract in `concepts/storage-adapters.md`.
|
||||
6. Major-version-bump the `@soulcraft/brainy` peerDep range expected by
|
||||
plugins.
|
||||
|
||||
Estimated work: ~half a day of code, ~2 hours of doc/example updates,
|
||||
ecosystem coordination via the platform handoff.
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -6,14 +6,14 @@ Brainy is a multi-dimensional AI database that combines vector similarity, graph
|
|||
|
||||
### Brainy (Main Entry Point)
|
||||
The central orchestrator that manages all subsystems:
|
||||
- **4-Index Architecture**: MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md))
|
||||
- **Storage System**: FileSystem and Memory adapters
|
||||
- **4-Index Architecture**: MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md))
|
||||
- **Storage System**: Universal storage adapters (FileSystem, S3, OPFS, Memory)
|
||||
- **Augmentation System**: Extensible plugin architecture
|
||||
- **Triple Intelligence**: Unified query engine
|
||||
|
||||
### Triple Intelligence Engine
|
||||
Brainy's revolutionary feature that unifies three types of search:
|
||||
- **Vector Search**: Semantic similarity via the pluggable vector index
|
||||
- **Vector Search**: Semantic similarity using HNSW indexing
|
||||
- **Graph Traversal**: Relationship-based queries
|
||||
- **Field Filtering**: Precise metadata filtering with O(1) performance
|
||||
|
||||
|
|
@ -42,13 +42,12 @@ brainy-data/
|
|||
└── locks/ # Concurrent access control
|
||||
```
|
||||
|
||||
### Vector Index
|
||||
Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph:
|
||||
### HNSW Index
|
||||
Hierarchical Navigable Small World index for efficient vector search:
|
||||
- **Performance**: O(log n) search complexity
|
||||
- **Configurable recall**: `fast` / `balanced` / `accurate` presets trade recall for latency
|
||||
- **Scalable**: Handles millions of vectors per process
|
||||
- **Memory Efficient**: Product quantization support
|
||||
- **Scalable**: Handles millions of vectors
|
||||
- **Persistent**: Serializable to storage
|
||||
- **Swappable**: Replace with a native implementation (such as `@soulcraft/cor`) via the plugin system without changing application code
|
||||
|
||||
### Metadata Index Manager
|
||||
High-performance field indexing system:
|
||||
|
|
@ -60,7 +59,7 @@ High-performance field indexing system:
|
|||
## Performance Characteristics
|
||||
|
||||
### Operation Complexity
|
||||
- **Vector Search**: O(log n) via the vector index
|
||||
- **Vector Search**: O(log n) via HNSW
|
||||
- **Field Filtering**: O(1) via inverted indexes
|
||||
- **Graph Traversal**: O(V + E) for breadth-first search
|
||||
- **Add Operation**: O(log n) for index insertion
|
||||
|
|
@ -84,6 +83,7 @@ Brainy's extensible plugin architecture allows for powerful enhancements:
|
|||
### Core Augmentations
|
||||
- **Entity Registry**: High-speed deduplication for streaming data
|
||||
- **Batch Processing**: Optimized bulk operations
|
||||
- **Connection Pool**: Efficient resource management
|
||||
- **Request Deduplicator**: Prevents duplicate processing
|
||||
|
||||
### Creating Custom Augmentations
|
||||
|
|
@ -111,7 +111,7 @@ Multi-layered caching for optimal performance:
|
|||
## Integration Points
|
||||
|
||||
### Key Objects for Extensions
|
||||
- `brain.index`: Access the vector index
|
||||
- `brain.index`: Access HNSW vector index
|
||||
- `brain.metadataIndex`: Access field indexing
|
||||
- `brain.graphIndex`: Access graph adjacency index
|
||||
- `brain.storage`: Access storage layer
|
||||
|
|
|
|||
|
|
@ -1,46 +1,46 @@
|
|||
# Storage Architecture
|
||||
|
||||
> **Updated**: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup
|
||||
> **Updated**: Metadata/vector separation, UUID-based sharding, lifecycle management
|
||||
|
||||
## Storage Structure
|
||||
|
||||
### Architecture: Metadata/Vector Separation
|
||||
|
||||
Entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
|
||||
entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
├── _system/ # System metadata (not sharded)
|
||||
│ ├── statistics.json # Performance metrics
|
||||
│ ├── __metadata_field_index__*.json # Field indexes
|
||||
│ └── __metadata_sorted_index__*.json # Sorted indexes
|
||||
├── _system/ # System metadata (not sharded)
|
||||
│ ├── statistics.json # Performance metrics
|
||||
│ ├── __metadata_field_index__*.json # Field indexes
|
||||
│ └── __metadata_sorted_index__*.json # Sorted indexes
|
||||
│
|
||||
├── entities/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # Vector graph data (sharded by UUID)
|
||||
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
|
||||
│ │ │ │ ├── 00123456-....json # Vector + graph connections
|
||||
│ │ │ │ └── 00abcdef-....json
|
||||
│ │ │ ├── 01/ ... ff/ # 256 shards total
|
||||
│ │ │
|
||||
│ │ └── metadata/ # Business data (sharded by UUID)
|
||||
│ │ ├── 00/
|
||||
│ │ │ ├── 00123456-....json # Entity metadata only
|
||||
│ │ │ └── 00abcdef-....json
|
||||
│ │ ├── 01/ ... ff/
|
||||
│ │
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (sharded)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ │
|
||||
│ └── metadata/ # Relationship data (sharded)
|
||||
│ ├── 00/ ... ff/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # HNSW graph data (sharded by UUID)
|
||||
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
|
||||
│ │ │ │ ├── 00123456-....json # Vector + HNSW connections
|
||||
│ │ │ │ └── 00abcdef-....json
|
||||
│ │ │ ├── 01/ ... ff/ # 256 shards total
|
||||
│ │ │
|
||||
│ │ └── metadata/ # Business data (sharded by UUID)
|
||||
│ │ ├── 00/
|
||||
│ │ │ ├── 00123456-....json # Entity metadata only
|
||||
│ │ │ └── 00abcdef-....json
|
||||
│ │ ├── 01/ ... ff/
|
||||
│ │
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (sharded)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ │
|
||||
│ └── metadata/ # Relationship data (sharded)
|
||||
│ ├── 00/ ... ff/
|
||||
```
|
||||
|
||||
### Why Split Metadata and Vectors?
|
||||
|
||||
**Performance at scale:**
|
||||
- **Vector search operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
|
||||
- **HNSW operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
|
||||
- **Filtering**: Only load metadata during filtering, not vectors
|
||||
- **Pagination**: Load metadata IDs first, fetch vectors/metadata on-demand
|
||||
- **Result**: 60-70% reduction in I/O for typical queries at million-entity scale
|
||||
|
|
@ -52,69 +52,115 @@ brainy-data/
|
|||
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
|
||||
const shard = uuid.substring(0, 2) // "3f"
|
||||
|
||||
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
|
||||
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
|
||||
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- **Uniform distribution**: ~3,900 entities per shard (at 1M scale)
|
||||
- **Filesystem optimization**: avoids huge flat directories that bog down `readdir`
|
||||
- **Parallel operations**: walk 256 shards in parallel
|
||||
- **Cloud storage optimization**: 200x faster than unsharded (30s → 150ms)
|
||||
- **Parallel operations**: Load 256 shards in parallel
|
||||
- **Predictable**: Deterministic shard assignment
|
||||
|
||||
## Storage Adapters
|
||||
|
||||
Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` interface:
|
||||
Brainy provides multiple storage adapters with identical APIs and production features:
|
||||
|
||||
### FileSystem Storage (Node.js, default)
|
||||
### FileSystem Storage (Node.js)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data'
|
||||
}
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // Gzip compression (60-80% space savings)
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Server applications, CLI tools, single-node deployments
|
||||
- **Performance**: Direct file I/O
|
||||
- **Use case**: Server applications, CLI tools
|
||||
- **Performance**: Direct file I/O with optional compression
|
||||
- **Persistence**: Permanent on disk
|
||||
- **Features**:
|
||||
- **Batch Delete**: Efficient bulk deletion with retries
|
||||
- **UUID Sharding**: Automatic 256-shard distribution
|
||||
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
|
||||
- **Batch Delete**: Efficient bulk deletion with retries
|
||||
- **UUID Sharding**: Automatic 256-shard distribution
|
||||
|
||||
### Memory Storage
|
||||
### S3 Compatible Storage (AWS, MinIO, R2)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'memory'
|
||||
}
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucket: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
credentials: {
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Tests, ephemeral workloads, single-process caches
|
||||
- **Performance**: No I/O — all data lives in process memory
|
||||
- **Persistence**: None — data is lost when the process exits
|
||||
- **Use case**: Distributed applications, cloud deployments
|
||||
- **Performance**: Network dependent, with intelligent caching
|
||||
- **Persistence**: Cloud storage durability (99.999999999%)
|
||||
- **Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
|
||||
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
|
||||
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
|
||||
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
|
||||
|
||||
### Auto
|
||||
### Google Cloud Storage (GCS)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'auto',
|
||||
path: './data'
|
||||
}
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
bucketName: 'my-brainy-data',
|
||||
keyFilename: './service-account.json' // Or use ADC
|
||||
}
|
||||
})
|
||||
```
|
||||
`'auto'` picks `'filesystem'` when running on Node.js with a writable `path`, and falls back to `'memory'` otherwise.
|
||||
- **Use case**: Google Cloud deployments
|
||||
- **Performance**: Global CDN with edge caching
|
||||
- **Persistence**: 99.999999999% durability
|
||||
- **Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
|
||||
- **Autoclass**: Intelligent automatic tier optimization
|
||||
- **Batch Delete**: Efficient bulk operations
|
||||
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
|
||||
|
||||
## Backup and Off-Site Replication
|
||||
### Azure Blob Storage
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'azure',
|
||||
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
|
||||
containerName: 'brainy-data'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Azure cloud deployments
|
||||
- **Performance**: Global replication with CDN
|
||||
- **Persistence**: LRS, ZRS, GRS, RA-GRS options
|
||||
- **Features**:
|
||||
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
|
||||
- **Lifecycle Policies**: Automatic tier transitions and deletions
|
||||
- **Batch Delete**: BlobBatchClient for efficient bulk operations
|
||||
- **Batch Tier Changes**: Move thousands of blobs efficiently
|
||||
- **Archive Rehydration**: Smart rehydration with priority options
|
||||
|
||||
Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `path` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:
|
||||
|
||||
- `gsutil rsync -r ./data gs://my-bucket/brainy-data`
|
||||
- `aws s3 sync ./data s3://my-bucket/brainy-data`
|
||||
- `rclone sync ./data remote:brainy-data`
|
||||
- Periodic `tar` snapshots to any object store
|
||||
|
||||
Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory.
|
||||
### Origin Private File System (Browser)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'opfs'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Browser applications, PWAs
|
||||
- **Performance**: Near-native file system speed
|
||||
- **Persistence**: Permanent in browser (with quota limits)
|
||||
- **Features**:
|
||||
- **Quota Monitoring**: Real-time quota tracking and warnings
|
||||
- **Batch Delete**: Efficient bulk deletion
|
||||
- **Storage Status**: Detailed usage/available reporting
|
||||
|
||||
## Metadata Indexing System
|
||||
|
||||
|
|
@ -124,12 +170,12 @@ Tracks all unique values for each field:
|
|||
```json
|
||||
// __metadata_field_index__field_category.json
|
||||
{
|
||||
"values": {
|
||||
"technology": 45,
|
||||
"science": 32,
|
||||
"business": 28
|
||||
},
|
||||
"lastUpdated": 1699564234567
|
||||
"values": {
|
||||
"technology": 45,
|
||||
"science": 32,
|
||||
"business": 28
|
||||
},
|
||||
"lastUpdated": 1699564234567
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -139,11 +185,11 @@ Maps field+value combinations to entity IDs:
|
|||
```json
|
||||
// __metadata_index__category_technology_chunk0.json
|
||||
{
|
||||
"field": "category",
|
||||
"value": "technology",
|
||||
"ids": ["uuid1", "uuid2", "uuid3", ...],
|
||||
"chunk": 0,
|
||||
"total": 45
|
||||
"field": "category",
|
||||
"value": "technology",
|
||||
"ids": ["uuid1", "uuid2", "uuid3", ...],
|
||||
"chunk": 0,
|
||||
"total": 45
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -161,14 +207,14 @@ High-performance deduplication system for streaming data:
|
|||
```json
|
||||
// __entity_registry__.json
|
||||
{
|
||||
"mappings": {
|
||||
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
|
||||
},
|
||||
"stats": {
|
||||
"totalMappings": 10000,
|
||||
"lastSync": 1699564234567
|
||||
}
|
||||
"mappings": {
|
||||
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
|
||||
},
|
||||
"stats": {
|
||||
"totalMappings": 10000,
|
||||
"lastSync": 1699564234567
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -178,46 +224,216 @@ High-performance deduplication system for streaming data:
|
|||
- **Cache**: LRU with configurable TTL
|
||||
- **Sync**: Periodic or on-demand
|
||||
|
||||
## Durability
|
||||
|
||||
Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `path` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).
|
||||
Ensures durability and enables recovery:
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": 1699564234567,
|
||||
"operation": "add",
|
||||
"data": {
|
||||
"id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"content": "...",
|
||||
"metadata": {}
|
||||
},
|
||||
"checksum": "sha256:..."
|
||||
}
|
||||
```
|
||||
|
||||
### Recovery Process
|
||||
2. Replay operations from last checkpoint
|
||||
3. Verify checksums for integrity
|
||||
|
||||
## Storage Optimization
|
||||
|
||||
### 1. Batch Operations
|
||||
### 1. Lifecycle Policies (Cloud Storage)
|
||||
|
||||
**Automatic cost optimization through tier transitions:**
|
||||
|
||||
```typescript
|
||||
// S3: Set lifecycle policy for automatic archival
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-data',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
|
||||
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
|
||||
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// GCS: Set lifecycle policy
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
condition: { age: 30 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||||
}, {
|
||||
condition: { age: 90 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||||
}, {
|
||||
condition: { age: 365 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||||
}]
|
||||
})
|
||||
|
||||
// Azure: Set lifecycle policy
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
name: 'archiveOldData',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: { blobTypes: ['blockBlob'] },
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 30 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 90 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
**Cost Impact (500TB dataset):**
|
||||
| Storage | Before | After | Savings |
|
||||
|---------|--------|-------|---------|
|
||||
| **AWS S3** | $138,000/yr | $5,940/yr | **96%** |
|
||||
| **GCS** | $138,000/yr | $8,300/yr | **94%** |
|
||||
| **Azure** | $107,520/yr | $5,016/yr | **95%** |
|
||||
|
||||
### 2. Intelligent-Tiering (S3)
|
||||
|
||||
**Automatic optimization without retrieval fees:**
|
||||
|
||||
```typescript
|
||||
// Enable S3 Intelligent-Tiering
|
||||
await storage.enableIntelligentTiering('entities/', 'auto-optimize')
|
||||
|
||||
// Benefits:
|
||||
// - Automatic tier transitions based on access patterns
|
||||
// - No retrieval fees (unlike Glacier)
|
||||
// - Up to 95% cost savings
|
||||
// - No performance impact on frequently accessed data
|
||||
```
|
||||
|
||||
### 3. Autoclass (GCS)
|
||||
|
||||
**Google Cloud's intelligent automatic optimization:**
|
||||
|
||||
```typescript
|
||||
// Enable GCS Autoclass
|
||||
await storage.enableAutoclass({
|
||||
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
|
||||
})
|
||||
|
||||
// Benefits:
|
||||
// - Automatic optimization based on access patterns
|
||||
// - No data retrieval delays
|
||||
// - Transparent tier transitions
|
||||
// - Up to 94% cost savings
|
||||
```
|
||||
|
||||
### 4. Compression (FileSystem)
|
||||
|
||||
```typescript
|
||||
// Enable gzip compression for local storage
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // 60-80% space savings
|
||||
}
|
||||
})
|
||||
|
||||
// Performance impact:
|
||||
// - Write: +10-20ms per file (gzip compression)
|
||||
// - Read: +5-10ms per file (gzip decompression)
|
||||
// - Space savings: 60-80% for typical JSON data
|
||||
// - CPU overhead: Minimal (~5% CPU)
|
||||
```
|
||||
|
||||
### 5. Batch Operations
|
||||
|
||||
```typescript
|
||||
// Efficient batch delete
|
||||
await storage.batchDelete([
|
||||
'entities/nouns/vectors/00/00123456-....json',
|
||||
'entities/nouns/metadata/00/00123456-....json'
|
||||
// ...
|
||||
'entities/nouns/vectors/00/00123456-....json',
|
||||
'entities/nouns/metadata/00/00123456-....json',
|
||||
// ... up to 1000 objects
|
||||
])
|
||||
|
||||
// Benefits:
|
||||
// - S3: 1000 objects per request (vs 1 per request)
|
||||
// - GCS: 100 objects per request
|
||||
// - Azure: 256 objects per batch
|
||||
// - Automatic retry logic with exponential backoff
|
||||
// - Throttling protection
|
||||
|
||||
// Batch writes for performance
|
||||
await brain.addBatch([
|
||||
{ content: "item1", metadata: {} },
|
||||
{ content: "item2", metadata: {} },
|
||||
{ content: "item3", metadata: {} }
|
||||
{ content: "item1", metadata: {} },
|
||||
{ content: "item2", metadata: {} },
|
||||
{ content: "item3", metadata: {} }
|
||||
])
|
||||
// Single transaction, optimized I/O
|
||||
```
|
||||
|
||||
### 2. Caching Strategy
|
||||
### 6. Quota Monitoring (OPFS)
|
||||
|
||||
```typescript
|
||||
// Configure caching
|
||||
// Get quota status for browser storage
|
||||
const status = await storage.getStorageStatus()
|
||||
|
||||
console.log(status)
|
||||
// {
|
||||
// type: 'opfs',
|
||||
// available: true,
|
||||
// details: {
|
||||
// usage: 45829120, // 43.7 MB used
|
||||
// quota: 536870912, // 512 MB available
|
||||
// usagePercent: 8.5,
|
||||
// quotaExceeded: false
|
||||
// }
|
||||
// }
|
||||
|
||||
// Proactive quota management:
|
||||
// - Monitor usage before writes
|
||||
// - Warn users when approaching quota
|
||||
// - Automatically clean up old data
|
||||
```
|
||||
|
||||
### 7. Tier Management (Azure)
|
||||
|
||||
```typescript
|
||||
// Change blob tier for cost optimization
|
||||
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
|
||||
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
|
||||
|
||||
// Batch tier changes (efficient)
|
||||
await storage.batchChangeTier([blob1, blob2, blob3], 'Cool')
|
||||
|
||||
// Rehydrate from Archive when needed
|
||||
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
|
||||
```
|
||||
|
||||
### 8. Caching Strategy
|
||||
|
||||
```typescript
|
||||
// Configure caching per storage type
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
ttl: 300000, // 5 minutes
|
||||
strategy: 'lru' // Least recently used
|
||||
}
|
||||
}
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
ttl: 300000, // 5 minutes
|
||||
strategy: 'lru' // Least recently used
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -227,8 +443,8 @@ const brain = new Brainy({
|
|||
```typescript
|
||||
// Automatic locking for write operations
|
||||
await brain.storage.withLock('resource-id', async () => {
|
||||
// Exclusive access to resource
|
||||
await brain.storage.saveNoun(id, data)
|
||||
// Exclusive access to resource
|
||||
await brain.storage.saveNoun(id, data)
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -239,39 +455,58 @@ await brain.storage.withLock('resource-id', async () => {
|
|||
|
||||
## Migration and Backup
|
||||
|
||||
Backup and restore go through the Db API — see
|
||||
[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) for the full
|
||||
recipe book.
|
||||
|
||||
### Snapshot (backup)
|
||||
### Export Data
|
||||
```typescript
|
||||
// Instant, self-contained snapshot (hard links on filesystem storage)
|
||||
const db = brain.now()
|
||||
await db.persist('/backups/2026-06-11')
|
||||
await db.release()
|
||||
// Export entire database
|
||||
const backup = await brain.export({
|
||||
format: 'json',
|
||||
includeVectors: true,
|
||||
includeIndexes: false
|
||||
})
|
||||
```
|
||||
|
||||
### Restore
|
||||
### Import Data
|
||||
```typescript
|
||||
// Replace the store's entire state from a snapshot (destructive — confirm required)
|
||||
await brain.restore('/backups/2026-06-11', { confirm: true })
|
||||
// Import from backup
|
||||
await brain.import(backup, {
|
||||
mode: 'merge', // or 'replace'
|
||||
validateSchema: true
|
||||
})
|
||||
```
|
||||
|
||||
### Move to a new directory
|
||||
### Storage Migration
|
||||
```typescript
|
||||
// A snapshot directory is a complete store: restore it into a fresh brain
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', path: './new' } })
|
||||
await brain.init()
|
||||
await brain.restore('/backups/2026-06-11', { confirm: true })
|
||||
// Migrate between storage types
|
||||
const oldBrain = new Brainy({ storage: { type: 'filesystem' } })
|
||||
const newBrain = new Brainy({ storage: { type: 's3' } })
|
||||
|
||||
await oldBrain.init()
|
||||
await newBrain.init()
|
||||
|
||||
// Transfer all data
|
||||
const data = await oldBrain.export()
|
||||
await newBrain.import(data)
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### FileSystem Optimizations
|
||||
- **Directory sharding**: 256 shards spread files across subdirectories
|
||||
### Storage-Specific Optimizations
|
||||
|
||||
#### FileSystem
|
||||
- **Directory sharding**: Split files across subdirectories
|
||||
- **Async I/O**: Non-blocking file operations
|
||||
- **Buffer pooling**: Reuse buffers for efficiency
|
||||
|
||||
#### S3
|
||||
- **Multipart uploads**: For large objects
|
||||
- **Request batching**: Combine small operations
|
||||
- **CDN integration**: Edge caching for reads
|
||||
|
||||
#### OPFS
|
||||
- **Quota management**: Monitor and request increases
|
||||
- **Worker offloading**: Heavy operations in workers
|
||||
- **Transaction batching**: Group operations
|
||||
|
||||
### Monitoring
|
||||
|
||||
```typescript
|
||||
|
|
@ -279,34 +514,58 @@ await brain.restore('/backups/2026-06-11', { confirm: true })
|
|||
const stats = await brain.storage.getStatistics()
|
||||
console.log(stats)
|
||||
// {
|
||||
// totalSize: 1048576,
|
||||
// entityCount: 1000,
|
||||
// indexSize: 204800,
|
||||
// walSize: 10240,
|
||||
// cacheHitRate: 0.85
|
||||
// totalSize: 1048576,
|
||||
// entityCount: 1000,
|
||||
// indexSize: 204800,
|
||||
// walSize: 10240,
|
||||
// cacheHitRate: 0.85
|
||||
// }
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Choose the Right Adapter
|
||||
1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence
|
||||
2. **Single-process production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
|
||||
3. **Horizontal scaling**: Brainy runs in one process — there is no built-in cluster. Run independent instances behind a service layer and replicate the on-disk artifact with your operator tooling; or run many reader processes against one shared store with a single writer
|
||||
1. **Development**: FileSystem with compression (local persistence, small storage footprint)
|
||||
2. **Production Server**: FileSystem with compression or cloud storage with lifecycle policies
|
||||
3. **Browser Apps**: OPFS with quota monitoring
|
||||
4. **Distributed**: S3/GCS/Azure with Intelligent-Tiering/Autoclass
|
||||
|
||||
### Optimize for Your Use Case
|
||||
1. **Read-heavy**: Enable caching and let the OS page cache do its job
|
||||
2. **Write-heavy**: Batch operations and tune the cache `maxSize`
|
||||
1. **Read-heavy**: Enable aggressive caching + cloud CDN
|
||||
2. **Write-heavy**: Batch operations + async writes
|
||||
3. **Real-time**: FileSystem with periodic snapshots
|
||||
4. **Archival**: Snapshot `path` to cold object storage on a schedule
|
||||
5. **Large-scale**: Rely on metadata/vector separation + UUID sharding
|
||||
4. **Archival**: Cloud storage with lifecycle policies (96% cost savings!)
|
||||
5. **Large-scale**: Metadata/vector separation + UUID sharding + lifecycle policies
|
||||
|
||||
### Cost Optimization
|
||||
1. **Enable lifecycle policies** for cloud storage (automated cost reduction)
|
||||
2. **Use Intelligent-Tiering (S3)** or Autoclass (GCS) for automatic optimization
|
||||
3. **Enable compression** for FileSystem storage (60-80% space savings)
|
||||
4. **Monitor quota** for OPFS (prevent quota exceeded errors)
|
||||
5. **Use batch operations** for bulk deletions (efficient API usage)
|
||||
6. **Consider tier management** for Azure (Hot/Cool/Archive tiers)
|
||||
|
||||
**Example Cost Savings (500TB dataset):**
|
||||
- Without lifecycle policies: **$138,000/year**
|
||||
- With lifecycle policies: **$5,940/year**
|
||||
- **Savings: $132,060/year (96%)**
|
||||
|
||||
### Monitor and Maintain
|
||||
1. Regular statistics collection
|
||||
2. Watch disk usage and shard balance
|
||||
2. Monitor lifecycle policy effectiveness
|
||||
3. Index optimization
|
||||
4. Cache tuning based on hit rates
|
||||
5. Verify backup runs (test restore quarterly)
|
||||
5. Track storage costs and tier distribution
|
||||
6. Review quota usage (OPFS) and storage growth patterns
|
||||
|
||||
### Production Deployment Checklist
|
||||
- ✅ Enable lifecycle policies on cloud storage
|
||||
- ✅ Configure batch delete for cleanup operations
|
||||
- ✅ Enable compression for FileSystem storage
|
||||
- ✅ Set up quota monitoring for OPFS
|
||||
- ✅ Configure appropriate tier transitions
|
||||
- ✅ Enable Intelligent-Tiering (S3) or Autoclass (GCS)
|
||||
- ✅ Monitor storage costs and optimize regularly
|
||||
|
||||
## API Reference
|
||||
|
||||
|
|
@ -314,5 +573,6 @@ See the [Storage API](../api/storage.md) for complete method documentation.
|
|||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026
|
||||
**Key Features**: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup
|
||||
**Version**: 4.0.0
|
||||
**Last Updated**: 2025-10-17
|
||||
**Key Features**: Metadata/vector separation, UUID sharding, lifecycle management, tier optimization
|
||||
|
|
@ -1,16 +1,3 @@
|
|||
---
|
||||
title: Triple Intelligence
|
||||
slug: concepts/triple-intelligence
|
||||
public: true
|
||||
category: concepts
|
||||
template: concept
|
||||
order: 1
|
||||
description: Unified vector similarity, graph traversal, and metadata filtering in one query. Auto-optimizes between parallel execution and progressive filtering.
|
||||
next:
|
||||
- concepts/noun-types
|
||||
- api/reference
|
||||
---
|
||||
|
||||
# Triple Intelligence System
|
||||
|
||||
The Triple Intelligence System is Brainy's revolutionary query engine that unifies vector similarity, graph relationships, and metadata filtering into a single, optimized query interface.
|
||||
|
|
@ -23,37 +10,29 @@ Traditional databases force you to choose between vector search, graph traversal
|
|||
|
||||
### Unified Query Structure
|
||||
|
||||
`find()` accepts a single `FindParams` object (or a natural-language string). One
|
||||
object combines all three intelligences:
|
||||
|
||||
```typescript
|
||||
interface FindParams {
|
||||
// Vector intelligence — semantic similarity
|
||||
query?: string // Natural-language / semantic query (embedded, matched via HNSW + text index)
|
||||
vector?: number[] // Pre-computed embedding for direct vector search
|
||||
|
||||
// Metadata intelligence — structured field filters
|
||||
type?: NounType | NounType[] // Filter by entity type
|
||||
subtype?: string | string[] // Filter by per-product subtype
|
||||
where?: Record<string, any> // Field predicates with bare operators (gte, lt, in, contains, exists…)
|
||||
|
||||
// Graph intelligence — relationship traversal
|
||||
interface TripleQuery {
|
||||
// Vector/Semantic search
|
||||
like?: string | Vector | any
|
||||
similar?: string | Vector | any
|
||||
|
||||
// Graph/Relationship search
|
||||
connected?: {
|
||||
to?: string // Reachable to this entity
|
||||
from?: string // Reachable from this entity
|
||||
via?: VerbType | VerbType[] // Relationship type(s) to traverse (alias: type)
|
||||
depth?: number // Max traversal depth (default: 1)
|
||||
to?: string | string[]
|
||||
from?: string | string[]
|
||||
type?: string | string[]
|
||||
depth?: number
|
||||
direction?: 'in' | 'out' | 'both'
|
||||
}
|
||||
|
||||
// Proximity — nearest neighbours of a known entity
|
||||
near?: { id: string; threshold?: number }
|
||||
|
||||
// Control
|
||||
limit?: number // Max results (default: 10)
|
||||
offset?: number // Skip N results
|
||||
orderBy?: string // Field to sort by (e.g. 'createdAt')
|
||||
order?: 'asc' | 'desc' // Sort direction
|
||||
|
||||
// Field/Attribute search
|
||||
where?: Record<string, any>
|
||||
|
||||
// Advanced options
|
||||
limit?: number
|
||||
boost?: 'recent' | 'popular' | 'verified' | string
|
||||
explain?: boolean
|
||||
threshold?: number
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -76,16 +55,16 @@ const articles = await brain.find("verified articles by John Smith about machine
|
|||
|
||||
#### Simple Vector Search
|
||||
```typescript
|
||||
const results = await brain.find("machine learning concepts")
|
||||
const results = await brain.search("machine learning concepts")
|
||||
```
|
||||
|
||||
#### Combined Intelligence Query
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
query: "neural networks",
|
||||
like: "neural networks",
|
||||
where: {
|
||||
category: "research",
|
||||
year: { gte: 2023 }
|
||||
year: { $gte: 2023 }
|
||||
},
|
||||
connected: {
|
||||
to: "deep-learning-team",
|
||||
|
|
@ -117,8 +96,8 @@ All three search types execute simultaneously:
|
|||
```typescript
|
||||
// Parallel execution for balanced query
|
||||
const results = await brain.find({
|
||||
query: "AI research", // ~1000 potential matches
|
||||
where: { kind: "paper" }, // ~500 potential matches
|
||||
like: "AI research", // ~1000 potential matches
|
||||
where: { type: "paper" }, // ~500 potential matches
|
||||
connected: { to: "stanford" } // ~200 potential matches
|
||||
})
|
||||
// All three execute in parallel, results fused
|
||||
|
|
@ -134,7 +113,7 @@ Operations chain for maximum efficiency:
|
|||
// Progressive execution for selective query
|
||||
const results = await brain.find({
|
||||
where: { userId: "user123" }, // Very selective (1-10 matches)
|
||||
query: "recent posts", // Applied to filtered set
|
||||
like: "recent posts", // Applied to filtered set
|
||||
limit: 5
|
||||
})
|
||||
// Metadata filter first, then vector search on results
|
||||
|
|
@ -169,15 +148,15 @@ Brainy includes 220+ embedded patterns for natural language understanding:
|
|||
|
||||
```typescript
|
||||
// Natural language automatically parsed
|
||||
const results = await brain.find(
|
||||
const results = await brain.search(
|
||||
"show me recent AI papers from Stanford published this year"
|
||||
)
|
||||
// Automatically converts to:
|
||||
// {
|
||||
// query: "AI papers",
|
||||
// where: {
|
||||
// like: "AI papers",
|
||||
// where: {
|
||||
// institution: "Stanford",
|
||||
// published: { gte: "2024-01-01" }
|
||||
// published: { $gte: "2024-01-01" }
|
||||
// }
|
||||
// }
|
||||
```
|
||||
|
|
@ -196,11 +175,11 @@ The NLP processor identifies query intent:
|
|||
|
||||
Successful execution plans are cached:
|
||||
```typescript
|
||||
// First call parses the natural-language query and builds an execution plan
|
||||
await brain.find("machine learning papers")
|
||||
// First query: 50ms (plan generation + execution)
|
||||
await brain.search("machine learning papers")
|
||||
|
||||
// A structurally similar query reuses that plan, skipping plan generation
|
||||
await brain.find("deep learning papers")
|
||||
// Subsequent similar queries: 10ms (cached plan)
|
||||
await brain.search("deep learning papers")
|
||||
```
|
||||
|
||||
### Self-Optimization
|
||||
|
|
@ -221,45 +200,50 @@ Triple Intelligence leverages all available indexes:
|
|||
|
||||
### Explain Mode
|
||||
|
||||
Diagnose how a query's `where` fields map to the index. Run `brain.explain()`
|
||||
first whenever `find()` returns surprising or empty results:
|
||||
|
||||
```typescript
|
||||
const plan = await brain.explain({
|
||||
query: "quantum computing",
|
||||
where: { category: "research" }
|
||||
})
|
||||
|
||||
console.log(plan.fieldPlan)
|
||||
// [
|
||||
// { field: 'category', path: 'column-store', notes: '...' }
|
||||
// ]
|
||||
|
||||
console.log(plan.warnings)
|
||||
// e.g. ['Field "category" has no index entries. find() will return [] silently...']
|
||||
```
|
||||
|
||||
### Result Ordering
|
||||
|
||||
Sort results by any stored field with `orderBy` / `order`:
|
||||
Understand how your query was executed:
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
query: "news articles",
|
||||
where: { verified: true },
|
||||
orderBy: 'createdAt', // Newest first
|
||||
order: 'desc'
|
||||
like: "quantum computing",
|
||||
where: { category: "research" },
|
||||
explain: true
|
||||
})
|
||||
|
||||
console.log(results[0].explanation)
|
||||
// {
|
||||
// plan: "field-first-progressive",
|
||||
// timing: {
|
||||
// fieldFilter: 2,
|
||||
// vectorSearch: 8,
|
||||
// fusion: 1
|
||||
// },
|
||||
// selectivity: {
|
||||
// field: 0.1,
|
||||
// vector: 0.3
|
||||
// }
|
||||
// }
|
||||
```
|
||||
|
||||
### Similarity Threshold
|
||||
### Boosting
|
||||
|
||||
Find the nearest neighbours of a known entity and keep only close matches with
|
||||
`near`:
|
||||
Apply custom ranking boosts:
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
near: { id: anchorId, threshold: 0.9 }, // Only results >= 0.9 similarity
|
||||
like: "news articles",
|
||||
boost: 'recent', // Boost recent items
|
||||
where: { verified: true }
|
||||
})
|
||||
```
|
||||
|
||||
### Threshold Control
|
||||
|
||||
Set minimum similarity thresholds:
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
like: "exact match needed",
|
||||
threshold: 0.9, // Only very similar results
|
||||
limit: 10
|
||||
})
|
||||
```
|
||||
|
|
@ -286,8 +270,8 @@ const results = await brain.find({
|
|||
```typescript
|
||||
// Find similar content with constraints
|
||||
const results = await brain.find({
|
||||
query: searchText,
|
||||
where: {
|
||||
like: query,
|
||||
where: {
|
||||
status: 'published',
|
||||
language: 'en'
|
||||
}
|
||||
|
|
@ -298,10 +282,10 @@ const results = await brain.find({
|
|||
```typescript
|
||||
// Find items related to a specific item
|
||||
const results = await brain.find({
|
||||
connected: {
|
||||
connected: {
|
||||
to: itemId,
|
||||
depth: 2,
|
||||
via: VerbType.RelatedTo
|
||||
type: 'similar'
|
||||
},
|
||||
limit: 20
|
||||
})
|
||||
|
|
@ -312,11 +296,10 @@ const results = await brain.find({
|
|||
// Recent items matching criteria
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
timestamp: { gte: Date.now() - 86400000 }
|
||||
timestamp: { $gte: Date.now() - 86400000 }
|
||||
},
|
||||
query: "trending topics",
|
||||
orderBy: 'timestamp',
|
||||
order: 'desc'
|
||||
like: "trending topics",
|
||||
boost: 'recent'
|
||||
})
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -1,157 +1,769 @@
|
|||
---
|
||||
title: Zero Configuration
|
||||
slug: concepts/zero-config
|
||||
public: false
|
||||
category: concepts
|
||||
template: concept
|
||||
order: 3
|
||||
description: Brainy auto-detects storage, initializes embeddings, and builds indexes — no configuration required. Works in Node.js and Bun (server-only since 8.0).
|
||||
next:
|
||||
- getting-started/installation
|
||||
- guides/storage-adapters
|
||||
---
|
||||
|
||||
# Zero Configuration & Auto-Adaptation
|
||||
|
||||
> **"Zero config by default, fully tunable when you need it."** Construct a
|
||||
> `Brainy()` with no options and it picks sensible, environment-aware defaults.
|
||||
> Every default below is overridable through the constructor — see the
|
||||
> [API Reference](../api/README.md#configuration).
|
||||
> **Current Status**: Basic zero-config is fully functional. Advanced auto-adaptation features are in development.
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy 8.0 is server-only (Node.js 22+ / Bun). With no configuration it:
|
||||
Brainy is designed with **"Zero Config by Default, Infinite Tunability"** philosophy. It automatically detects your environment, adapts to available resources, learns from usage patterns, and optimizes itself for your specific workload—all without any configuration.
|
||||
|
||||
- selects a storage adapter from the runtime,
|
||||
- initializes the embedding model (all-MiniLM-L6-v2, 384 dimensions),
|
||||
- builds and maintains the metadata, graph, and vector indexes,
|
||||
- sizes its caches and write buffers to the detected memory budget,
|
||||
- chooses a persistence mode that matches the storage backend, and
|
||||
- quiets its own logging when it detects a production environment.
|
||||
## Zero Configuration Magic
|
||||
|
||||
There is no public config-generation function — adaptation happens inside the
|
||||
constructor and `init()`.
|
||||
|
||||
## Instant Start
|
||||
### Instant Start
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { Brainy } from 'brainy'
|
||||
|
||||
// That's it. No config needed.
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
await brain.add({ data: 'First entity', type: 'concept' })
|
||||
const results = await brain.find('first')
|
||||
// Brainy automatically:
|
||||
// ✓ Detects environment (Node.js, Browser, Edge, Deno)
|
||||
// ✓ Chooses optimal storage (FileSystem, OPFS, Memory)
|
||||
// ✓ Downloads required models (if needed)
|
||||
// ✓ Configures vector dimensions (384 optimal)
|
||||
// ✓ Sets up indexing strategies
|
||||
// ✓ Enables appropriate augmentations
|
||||
// ✓ Configures caching layers
|
||||
// ✓ Optimizes for your hardware
|
||||
```
|
||||
|
||||
## What Auto-Adaptation Covers
|
||||
### Environment Detection ✅ Available
|
||||
|
||||
### 1. Storage auto-detection
|
||||
|
||||
With no `storage` option, Brainy uses `type: 'auto'`:
|
||||
|
||||
- **Filesystem** when running on a runtime with a writable Node filesystem and a
|
||||
resolvable root directory. This is the default for typical Node/Bun servers and
|
||||
persists across restarts.
|
||||
- **In-memory** otherwise (no filesystem access, or an explicit memory request).
|
||||
Fast, zero I/O, discarded on process exit — ideal for tests and ephemeral
|
||||
caches.
|
||||
|
||||
8.0 ships exactly two storage adapters — `memory` and `filesystem` — plus the
|
||||
`auto` selector that resolves to one of them. See
|
||||
[Storage Adapters](../concepts/storage-adapters.md) for the full contract.
|
||||
Brainy automatically detects and adapts to your runtime:
|
||||
|
||||
```typescript
|
||||
// Explicit override when you want a specific root
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './brainy-data' }
|
||||
})
|
||||
// Brainy's environment detection
|
||||
const environment = {
|
||||
// Runtime detection
|
||||
isNode: typeof process !== 'undefined',
|
||||
isBrowser: typeof window !== 'undefined',
|
||||
isDeno: typeof Deno !== 'undefined',
|
||||
isEdge: typeof EdgeRuntime !== 'undefined',
|
||||
isWebWorker: typeof WorkerGlobalScope !== 'undefined',
|
||||
|
||||
// Capability detection
|
||||
hasFileSystem: /* auto-detected */,
|
||||
hasIndexedDB: /* auto-detected */,
|
||||
hasOPFS: /* auto-detected */,
|
||||
hasWebGPU: /* auto-detected */,
|
||||
hasWASM: /* auto-detected */,
|
||||
|
||||
// Resource detection
|
||||
cpuCores: /* auto-detected */,
|
||||
memory: /* auto-detected */,
|
||||
storage: /* auto-detected */
|
||||
}
|
||||
```
|
||||
|
||||
### 2. HNSW quality from the `recall` preset
|
||||
## Auto-Adaptive Storage ✅ Available
|
||||
|
||||
Vector-index quality comes from a single preset rather than hand-tuned graph
|
||||
parameters. `config.vector.recall` accepts `'fast'`, `'balanced'`, or
|
||||
`'accurate'` and defaults to `'balanced'`. The preset maps internally to the
|
||||
HNSW construction and search parameters (`M` / `efConstruction` / `efSearch`),
|
||||
so you trade recall against latency with one knob instead of three.
|
||||
> **Current**: Brainy automatically selects the best storage adapter for your environment.
|
||||
|
||||
### Storage Selection Logic
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
vector: { recall: 'fast' } // favor latency over recall
|
||||
})
|
||||
// Brainy's intelligent storage selection
|
||||
async function autoSelectStorage() {
|
||||
// Server environments
|
||||
if (environment.isNode) {
|
||||
if (await hasWritePermission('./data')) {
|
||||
return 'filesystem' // Best for servers
|
||||
} else if (process.env.S3_BUCKET) {
|
||||
return 's3' // Cloud deployment
|
||||
} else {
|
||||
return 'memory' // Fallback for restricted environments
|
||||
}
|
||||
}
|
||||
|
||||
// Browser environments
|
||||
if (environment.isBrowser) {
|
||||
if (await navigator.storage.estimate() > 1GB) {
|
||||
return 'opfs' // Best for modern browsers
|
||||
} else if (indexedDB) {
|
||||
return 'indexeddb' // Fallback for older browsers
|
||||
} else {
|
||||
return 'memory' // In-memory for restricted contexts
|
||||
}
|
||||
}
|
||||
|
||||
// Edge environments
|
||||
if (environment.isEdge) {
|
||||
return 'kv' // Use edge KV stores (Cloudflare, Vercel)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The default JS index is `JsHnswVectorIndex`. An optional native acceleration
|
||||
provider (the `@soulcraft/cor` package) can replace it with a
|
||||
higher-performing implementation; the public knobs stay the same. Quantization
|
||||
and other index-internal acceleration are the native provider's concern, not a
|
||||
Brainy configuration option.
|
||||
### Storage Migration
|
||||
|
||||
### 3. Persistence mode follows the backend
|
||||
|
||||
`config.vector.persistMode` accepts `'immediate'` or `'deferred'`. Left unset,
|
||||
Brainy chooses for you:
|
||||
|
||||
- **Immediate** on filesystem storage, so the index file stays in lock-step with
|
||||
the data and survives a crash.
|
||||
- **Deferred** on in-memory storage, where there is nothing durable to sync to,
|
||||
so writes are batched for throughput.
|
||||
Brainy seamlessly migrates between storage types:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
vector: { persistMode: 'deferred' } // batch persistence for write-heavy loads
|
||||
// Start with memory storage (development)
|
||||
const brain = new Brainy() // Auto-selects memory
|
||||
|
||||
// Later, migrate to production storage
|
||||
await brain.migrate({
|
||||
to: 'filesystem',
|
||||
path: './production-data'
|
||||
})
|
||||
// All data seamlessly transferred
|
||||
```
|
||||
|
||||
### 4. Memory-aware cache and buffer sizing
|
||||
## Learning & Optimization 🚧 Coming Soon
|
||||
|
||||
Brainy reads the container's memory budget — `CLOUD_RUN_MEMORY`, `MEMORY_LIMIT`,
|
||||
or the cgroup memory limit when running in a container — and sizes its read
|
||||
caches and write buffers to fit. On a small instance it stays conservative; on a
|
||||
large one it uses more of the available headroom. Query-result limits are capped
|
||||
against the same budget (roughly 25 KB per result) to keep a single oversized
|
||||
query from exhausting memory.
|
||||
> **Note**: These features are planned for Q2 2025. Currently, Brainy uses static optimizations.
|
||||
|
||||
You can pin the cache explicitly:
|
||||
### Query Pattern Learning 🚧 Planned
|
||||
|
||||
Brainy learns from your query patterns and optimizes accordingly:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
cache: { maxSize: 10000, ttl: 3_600_000 }
|
||||
})
|
||||
// Brainy observes query patterns
|
||||
class QueryPatternLearner {
|
||||
analyze(queries: Query[]) {
|
||||
return {
|
||||
// Frequency analysis
|
||||
mostCommonFields: this.getTopFields(queries),
|
||||
avgResultSize: this.getAvgSize(queries),
|
||||
temporalPatterns: this.getTimePatterns(queries),
|
||||
|
||||
// Relationship analysis
|
||||
commonTraversals: this.getGraphPatterns(queries),
|
||||
typicalDepth: this.getAvgDepth(queries),
|
||||
|
||||
// Performance analysis
|
||||
slowQueries: this.getSlowQueries(queries),
|
||||
cacheability: this.getCacheability(queries)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Automatic optimizations based on learning:
|
||||
// - Creates indexes for frequently queried fields
|
||||
// - Pre-computes common graph traversals
|
||||
// - Adjusts cache sizes based on working set
|
||||
// - Optimizes vector search parameters
|
||||
```
|
||||
|
||||
### 5. Logging quiets in production
|
||||
### Auto-Indexing 🚧 Planned
|
||||
|
||||
Brainy detects production-style environments (for example `NODE_ENV` set to a
|
||||
non-development value) and reduces its own log verbosity automatically. This is
|
||||
logging-only behavior — it does not change indexing, storage, or query results.
|
||||
Brainy automatically creates indexes based on usage:
|
||||
|
||||
```typescript
|
||||
// No manual index configuration needed
|
||||
await brain.find({ where: { category: "tech" } }) // First query
|
||||
// Brainy notices 'category' field usage
|
||||
|
||||
await brain.find({ where: { category: "science" } }) // Second query
|
||||
// Pattern detected - auto-creates category index
|
||||
|
||||
await brain.find({ where: { category: "tech" } }) // Third query
|
||||
// Now using index - 100x faster!
|
||||
```
|
||||
|
||||
### Adaptive Caching 🚧 Planned
|
||||
|
||||
Cache strategies adapt to your access patterns:
|
||||
|
||||
```typescript
|
||||
class AdaptiveCache {
|
||||
async adapt(metrics: AccessMetrics) {
|
||||
if (metrics.hitRate < 0.3) {
|
||||
// Low hit rate - switch strategy
|
||||
this.strategy = 'lfu' // Least Frequently Used
|
||||
} else if (metrics.workingSet > this.size) {
|
||||
// Working set too large - increase size
|
||||
this.size = Math.min(metrics.workingSet * 1.5, maxMemory)
|
||||
} else if (metrics.temporalLocality > 0.8) {
|
||||
// High temporal locality - use time-based eviction
|
||||
this.strategy = 'ttl'
|
||||
this.ttl = metrics.avgAccessInterval * 2
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Auto-Scaling 🚧 Coming Soon
|
||||
|
||||
### Dynamic Batch Sizing
|
||||
|
||||
Brainy adjusts batch sizes based on system load:
|
||||
|
||||
```typescript
|
||||
class DynamicBatcher {
|
||||
calculateOptimalBatch() {
|
||||
const cpuUsage = process.cpuUsage()
|
||||
const memoryUsage = process.memoryUsage()
|
||||
|
||||
if (cpuUsage < 30 && memoryUsage < 50) {
|
||||
return 1000 // System idle - large batches
|
||||
} else if (cpuUsage < 60 && memoryUsage < 70) {
|
||||
return 100 // Moderate load - medium batches
|
||||
} else {
|
||||
return 10 // High load - small batches
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Automatically applied during bulk operations
|
||||
for (const item of millionItems) {
|
||||
await brain.add(item) // Internally batched optimally
|
||||
}
|
||||
```
|
||||
|
||||
### Memory Management
|
||||
|
||||
Automatic memory pressure handling:
|
||||
|
||||
```typescript
|
||||
class MemoryManager {
|
||||
async handlePressure() {
|
||||
const usage = process.memoryUsage()
|
||||
const available = os.freemem()
|
||||
|
||||
if (available < 100 * 1024 * 1024) { // Less than 100MB free
|
||||
// Emergency mode
|
||||
await this.flushCaches()
|
||||
await this.compactIndexes()
|
||||
await this.offloadToDisk()
|
||||
} else if (usage.heapUsed / usage.heapTotal > 0.9) {
|
||||
// Preventive mode
|
||||
await this.reduceCacheSizes()
|
||||
await this.pauseBackgroundTasks()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Connection Pooling
|
||||
|
||||
Automatic connection management for storage backends:
|
||||
|
||||
```typescript
|
||||
class ConnectionPool {
|
||||
async getOptimalPoolSize() {
|
||||
// Adapts based on workload
|
||||
const metrics = await this.getMetrics()
|
||||
|
||||
if (metrics.waitTime > 100) {
|
||||
// Queries waiting - increase pool
|
||||
this.size = Math.min(this.size * 1.5, this.maxSize)
|
||||
} else if (metrics.idleConnections > this.size * 0.5) {
|
||||
// Too many idle - decrease pool
|
||||
this.size = Math.max(this.size * 0.7, this.minSize)
|
||||
}
|
||||
|
||||
return this.size
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Model Auto-Selection
|
||||
|
||||
### Embedding Model Selection
|
||||
|
||||
Brainy chooses the best embedding model for your use case:
|
||||
|
||||
```typescript
|
||||
async function autoSelectModel(data: Sample[]) {
|
||||
const analysis = {
|
||||
languages: detectLanguages(data),
|
||||
domainSpecific: detectDomain(data),
|
||||
averageLength: getAvgLength(data),
|
||||
requiresMultilingual: languages.length > 1
|
||||
}
|
||||
|
||||
if (analysis.requiresMultilingual) {
|
||||
return 'multilingual-e5-base' // Handles 100+ languages
|
||||
} else if (analysis.domainSpecific === 'code') {
|
||||
return 'codebert-base' // Optimized for code
|
||||
} else if (analysis.averageLength > 512) {
|
||||
return 'all-mpnet-base-v2' // Better for long text
|
||||
} else {
|
||||
return 'all-MiniLM-L6-v2' // Fast and efficient default
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Model Downloading
|
||||
|
||||
Models are automatically downloaded when needed:
|
||||
|
||||
```typescript
|
||||
// First use - model auto-downloads
|
||||
const brain = new Brainy()
|
||||
await brain.init() // Downloads model if not cached
|
||||
|
||||
// Intelligent model caching
|
||||
const modelCache = {
|
||||
location: process.env.MODEL_CACHE || '~/.brainy/models',
|
||||
maxSize: 5 * 1024 * 1024 * 1024, // 5GB max
|
||||
strategy: 'lru', // Least recently used eviction
|
||||
|
||||
// CDN selection based on location
|
||||
cdn: await selectFastestCDN([
|
||||
'https://cdn.brainy.io',
|
||||
'https://brainy.b-cdn.net',
|
||||
'https://models.huggingface.co'
|
||||
])
|
||||
}
|
||||
```
|
||||
|
||||
## Workload Detection
|
||||
|
||||
### Pattern Recognition
|
||||
|
||||
Brainy identifies your workload type and optimizes:
|
||||
|
||||
```typescript
|
||||
enum WorkloadType {
|
||||
OLTP = 'oltp', // Many small transactions
|
||||
OLAP = 'olap', // Analytical queries
|
||||
STREAMING = 'streaming', // Real-time ingestion
|
||||
BATCH = 'batch', // Bulk processing
|
||||
HYBRID = 'hybrid' // Mixed workload
|
||||
}
|
||||
|
||||
class WorkloadDetector {
|
||||
detect(metrics: OperationMetrics): WorkloadType {
|
||||
if (metrics.writesPerSecond > 1000 && metrics.avgWriteSize < 1024) {
|
||||
return WorkloadType.STREAMING
|
||||
} else if (metrics.avgQueryComplexity > 0.8 && metrics.avgResultSize > 10000) {
|
||||
return WorkloadType.OLAP
|
||||
} else if (metrics.batchOperations > metrics.singleOperations) {
|
||||
return WorkloadType.BATCH
|
||||
} else if (metrics.writeReadRatio > 0.3 && metrics.writeReadRatio < 0.7) {
|
||||
return WorkloadType.HYBRID
|
||||
} else {
|
||||
return WorkloadType.OLTP
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Optimization Strategies
|
||||
|
||||
Different optimizations for different workloads:
|
||||
|
||||
```typescript
|
||||
class WorkloadOptimizer {
|
||||
optimize(workload: WorkloadType) {
|
||||
switch (workload) {
|
||||
case WorkloadType.STREAMING:
|
||||
return {
|
||||
entityRegistry: true, // Deduplication
|
||||
batchSize: 1000,
|
||||
walEnabled: true,
|
||||
cacheSize: 'small',
|
||||
indexStrategy: 'lazy'
|
||||
}
|
||||
|
||||
case WorkloadType.OLAP:
|
||||
return {
|
||||
entityRegistry: false,
|
||||
batchSize: 10000,
|
||||
walEnabled: false,
|
||||
cacheSize: 'large',
|
||||
indexStrategy: 'eager',
|
||||
parallelQueries: true
|
||||
}
|
||||
|
||||
case WorkloadType.BATCH:
|
||||
return {
|
||||
entityRegistry: false,
|
||||
batchSize: 50000,
|
||||
walEnabled: false,
|
||||
cacheSize: 'minimal',
|
||||
indexStrategy: 'deferred'
|
||||
}
|
||||
|
||||
default:
|
||||
return this.defaultConfig
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Hardware Adaptation 🚧 Coming Soon
|
||||
|
||||
> **Note**: GPU acceleration and hardware optimization planned for Q3 2025.
|
||||
|
||||
### CPU Optimization
|
||||
|
||||
Adapts to available CPU resources:
|
||||
|
||||
```typescript
|
||||
class CPUAdapter {
|
||||
async optimize() {
|
||||
const cores = os.cpus().length
|
||||
const type = os.cpus()[0].model
|
||||
|
||||
// Parallel processing based on cores
|
||||
this.parallelism = Math.max(1, cores - 1) // Leave one core free
|
||||
|
||||
// SIMD detection for vector operations
|
||||
if (type.includes('Intel') || type.includes('AMD')) {
|
||||
this.enableSIMD = await checkSIMDSupport()
|
||||
}
|
||||
|
||||
// Thread pool sizing
|
||||
this.threadPoolSize = cores * 2 // Optimal for I/O bound
|
||||
|
||||
// Vector search optimization
|
||||
if (cores >= 8) {
|
||||
this.hnswConstruction = 200 // Higher quality index
|
||||
this.hnswSearch = 100 // More accurate search
|
||||
} else {
|
||||
this.hnswConstruction = 100 // Balanced
|
||||
this.hnswSearch = 50 // Faster search
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Memory Adaptation
|
||||
|
||||
Intelligent memory allocation:
|
||||
|
||||
```typescript
|
||||
class MemoryAdapter {
|
||||
async configure() {
|
||||
const totalMemory = os.totalmem()
|
||||
const availableMemory = os.freemem()
|
||||
|
||||
// Allocate based on available memory
|
||||
const allocation = {
|
||||
cache: Math.min(availableMemory * 0.25, 2 * GB),
|
||||
vectors: Math.min(availableMemory * 0.30, 4 * GB),
|
||||
indexes: Math.min(availableMemory * 0.20, 2 * GB),
|
||||
working: Math.min(availableMemory * 0.25, 2 * GB)
|
||||
}
|
||||
|
||||
// Adjust for low memory systems
|
||||
if (totalMemory < 4 * GB) {
|
||||
allocation.cache *= 0.5
|
||||
allocation.vectors *= 0.7
|
||||
this.enableSwapping = true
|
||||
}
|
||||
|
||||
return allocation
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### GPU Acceleration
|
||||
|
||||
Automatic GPU detection and utilization:
|
||||
|
||||
```typescript
|
||||
class GPUAdapter {
|
||||
async detect() {
|
||||
// WebGPU in browsers
|
||||
if (navigator?.gpu) {
|
||||
const adapter = await navigator.gpu.requestAdapter()
|
||||
return {
|
||||
available: true,
|
||||
type: 'webgpu',
|
||||
memory: adapter.limits.maxBufferSize,
|
||||
compute: adapter.limits.maxComputeWorkgroupsPerDimension
|
||||
}
|
||||
}
|
||||
|
||||
// CUDA in Node.js
|
||||
if (process.platform === 'linux' || process.platform === 'win32') {
|
||||
const hasCuda = await checkCudaSupport()
|
||||
if (hasCuda) {
|
||||
return {
|
||||
available: true,
|
||||
type: 'cuda',
|
||||
memory: await getCudaMemory(),
|
||||
compute: await getCudaCores()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { available: false }
|
||||
}
|
||||
|
||||
async optimize(gpu: GPUInfo) {
|
||||
if (gpu.available) {
|
||||
// Offload vector operations to GPU
|
||||
this.vectorOps = 'gpu'
|
||||
this.embeddingGeneration = 'gpu'
|
||||
this.matrixMultiplication = 'gpu'
|
||||
|
||||
// Larger batch sizes for GPU
|
||||
this.batchSize = gpu.memory > 8 * GB ? 10000 : 1000
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Network Adaptation
|
||||
|
||||
### Bandwidth Detection
|
||||
|
||||
Optimizes for available network bandwidth:
|
||||
|
||||
```typescript
|
||||
class NetworkAdapter {
|
||||
async measureBandwidth() {
|
||||
const testSize = 1 * MB
|
||||
const start = Date.now()
|
||||
await this.transfer(testSize)
|
||||
const duration = Date.now() - start
|
||||
|
||||
const bandwidth = (testSize / duration) * 1000 // bytes/sec
|
||||
|
||||
if (bandwidth < 1 * MB) {
|
||||
// Low bandwidth - optimize
|
||||
this.compression = 'aggressive'
|
||||
this.batchTransfers = true
|
||||
this.cacheRemote = true
|
||||
} else if (bandwidth > 100 * MB) {
|
||||
// High bandwidth
|
||||
this.compression = 'minimal'
|
||||
this.parallelTransfers = true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Latency Optimization
|
||||
|
||||
Adapts to network latency:
|
||||
|
||||
```typescript
|
||||
class LatencyOptimizer {
|
||||
async optimize() {
|
||||
const latency = await this.measureLatency()
|
||||
|
||||
if (latency > 100) { // High latency
|
||||
// Batch operations
|
||||
this.minBatchSize = 100
|
||||
|
||||
// Aggressive prefetching
|
||||
this.prefetchDepth = 3
|
||||
|
||||
// Local caching
|
||||
this.cacheStrategy = 'aggressive'
|
||||
|
||||
// Connection pooling
|
||||
this.connectionPool = Math.min(latency / 10, 50)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Cloud Provider Detection 🚧 Coming Soon
|
||||
|
||||
> **Note**: Cloud provider auto-detection planned for Q3 2025.
|
||||
|
||||
### Automatic Cloud Optimization
|
||||
|
||||
Detects and optimizes for cloud providers:
|
||||
|
||||
```typescript
|
||||
class CloudDetector {
|
||||
async detect() {
|
||||
// AWS Detection
|
||||
if (process.env.AWS_REGION || await canReachMetadata('169.254.169.254')) {
|
||||
return {
|
||||
provider: 'aws',
|
||||
instance: await getEC2InstanceType(),
|
||||
region: process.env.AWS_REGION,
|
||||
services: {
|
||||
storage: 's3',
|
||||
cache: 'elasticache',
|
||||
compute: 'lambda'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Google Cloud Detection
|
||||
if (process.env.GOOGLE_CLOUD_PROJECT || await canReachMetadata('metadata.google.internal')) {
|
||||
return {
|
||||
provider: 'gcp',
|
||||
instance: await getGCEInstanceType(),
|
||||
region: process.env.GOOGLE_CLOUD_REGION,
|
||||
services: {
|
||||
storage: 'gcs',
|
||||
cache: 'memorystore',
|
||||
compute: 'cloud-run'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Vercel Edge Detection
|
||||
if (process.env.VERCEL) {
|
||||
return {
|
||||
provider: 'vercel',
|
||||
region: process.env.VERCEL_REGION,
|
||||
services: {
|
||||
storage: 'vercel-kv',
|
||||
cache: 'edge-config',
|
||||
compute: 'edge-runtime'
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Development vs Production
|
||||
|
||||
### Automatic Environment Detection
|
||||
|
||||
```typescript
|
||||
class EnvironmentDetector {
|
||||
detect() {
|
||||
const indicators = {
|
||||
// Development indicators
|
||||
isDevelopment:
|
||||
process.env.NODE_ENV === 'development' ||
|
||||
process.env.DEBUG ||
|
||||
process.argv.includes('--dev') ||
|
||||
isLocalhost() ||
|
||||
hasDevTools(),
|
||||
|
||||
// Test indicators
|
||||
isTest:
|
||||
process.env.NODE_ENV === 'test' ||
|
||||
process.env.CI ||
|
||||
isTestRunner(),
|
||||
|
||||
// Production indicators
|
||||
isProduction:
|
||||
process.env.NODE_ENV === 'production' ||
|
||||
process.env.VERCEL ||
|
||||
process.env.NETLIFY ||
|
||||
!isLocalhost()
|
||||
}
|
||||
|
||||
return indicators
|
||||
}
|
||||
}
|
||||
|
||||
// Different defaults for different environments
|
||||
const config = environment.isProduction ? {
|
||||
storage: 'filesystem',
|
||||
wal: true,
|
||||
monitoring: true,
|
||||
compression: true,
|
||||
caching: 'aggressive'
|
||||
} : {
|
||||
storage: 'memory',
|
||||
wal: false,
|
||||
monitoring: false,
|
||||
compression: false,
|
||||
caching: 'minimal'
|
||||
}
|
||||
```
|
||||
|
||||
## Error Recovery
|
||||
|
||||
### Automatic Fallbacks
|
||||
|
||||
Brainy automatically recovers from errors:
|
||||
|
||||
```typescript
|
||||
class AutoRecovery {
|
||||
async handleStorageFailure() {
|
||||
try {
|
||||
await this.primaryStorage.write(data)
|
||||
} catch (error) {
|
||||
console.warn('Primary storage failed, trying fallback')
|
||||
|
||||
// Try secondary storage
|
||||
if (this.secondaryStorage) {
|
||||
await this.secondaryStorage.write(data)
|
||||
} else {
|
||||
// Fall back to memory
|
||||
await this.memoryStorage.write(data)
|
||||
|
||||
// Schedule retry
|
||||
this.scheduleRetry(data)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async handleModelFailure() {
|
||||
try {
|
||||
return await this.primaryModel.embed(text)
|
||||
} catch (error) {
|
||||
// Fall back to simpler model
|
||||
return await this.fallbackModel.embed(text)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration Override
|
||||
|
||||
Zero-config is the default, not a ceiling. Every adaptive decision above has an
|
||||
explicit constructor option:
|
||||
While zero-config is default, you can override when needed:
|
||||
|
||||
```typescript
|
||||
// Explicit configuration when needed
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'accurate',
|
||||
persistMode: 'immediate'
|
||||
// Override auto-detection
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: '/custom/path'
|
||||
},
|
||||
cache: { maxSize: 50000, ttl: 600_000 }
|
||||
|
||||
// Override auto-optimization
|
||||
optimization: {
|
||||
autoIndex: false,
|
||||
autoCache: false,
|
||||
autoBatch: false
|
||||
},
|
||||
|
||||
// Override auto-scaling
|
||||
scaling: {
|
||||
maxMemory: 2 * GB,
|
||||
maxConnections: 100,
|
||||
maxBatchSize: 1000
|
||||
}
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
See the [API Reference](../api/README.md#configuration) for the complete option
|
||||
list.
|
||||
## Monitoring Auto-Adaptation
|
||||
|
||||
Brainy provides visibility into its auto-adaptation:
|
||||
|
||||
```typescript
|
||||
brain.on('adaptation', (event) => {
|
||||
console.log(`Brainy adapted: ${event.type}`)
|
||||
console.log(`Reason: ${event.reason}`)
|
||||
console.log(`Before: ${JSON.stringify(event.before)}`)
|
||||
console.log(`After: ${JSON.stringify(event.after)}`)
|
||||
})
|
||||
|
||||
// Example events:
|
||||
// - Index created for frequently queried field
|
||||
// - Cache strategy changed due to low hit rate
|
||||
// - Batch size increased due to high throughput
|
||||
// - Storage migrated due to space constraints
|
||||
// - Model switched due to multilingual content
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
Brainy's zero-configuration and auto-adaptation capabilities mean you can focus on your application logic while Brainy handles:
|
||||
|
||||
- Environment detection and optimization
|
||||
- Storage selection and migration
|
||||
- Performance tuning and scaling
|
||||
- Resource management
|
||||
- Error recovery
|
||||
- Workload optimization
|
||||
|
||||
Just create a Brainy instance and start using it. Brainy will learn, adapt, and optimize itself for your specific use case—no configuration required.
|
||||
|
||||
## See Also
|
||||
|
||||
- [Architecture Overview](./overview.md)
|
||||
- [Storage Adapters](../concepts/storage-adapters.md)
|
||||
- [Scaling Guide](../SCALING.md)
|
||||
- [API Reference](../api/README.md)
|
||||
- [Storage Architecture](./storage.md)
|
||||
- [Performance Guide](../guides/performance.md)
|
||||
- [Augmentations System](./augmentations.md)
|
||||
451
docs/augmentations/COMPLETE-REFERENCE.md
Normal file
451
docs/augmentations/COMPLETE-REFERENCE.md
Normal file
|
|
@ -0,0 +1,451 @@
|
|||
# 🔌 Brainy Augmentations Complete Reference
|
||||
|
||||
> **All augmentations that power Brainy's extensibility - with locations, usage, and examples**
|
||||
>
|
||||
> **⚠️ Update**: Updated for metadata structure changes and billion-scale optimizations
|
||||
|
||||
## Quick Start
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
// Augmentations auto-configure based on environment
|
||||
storage: 'auto', // Storage augmentation
|
||||
cache: true, // Cache augmentation
|
||||
index: true // Index augmentation
|
||||
})
|
||||
|
||||
await brain.init() // Augmentations initialize automatically
|
||||
```
|
||||
|
||||
## Augmentation Architecture
|
||||
|
||||
### Key Improvements for Billion-Scale Performance
|
||||
|
||||
1. **Metadata/Vector Separation**: Augmentations now work with separated metadata and vectors
|
||||
- Metadata stored separately from vector data
|
||||
- 99.2% memory reduction for type tracking
|
||||
- Two-file storage pattern for optimal I/O
|
||||
|
||||
2. **Type System Enforcement**: All metadata requires type fields
|
||||
- `NounMetadata` requires `noun: NounType`
|
||||
- `VerbMetadata` requires `verb: VerbType`
|
||||
- Type inference system available as public API
|
||||
|
||||
3. **Storage Adapter Pattern**: Internal vs public method distinction
|
||||
- `_methods`: Return pure structures (HNSWNoun, HNSWVerb)
|
||||
- Public methods: Return WithMetadata types
|
||||
- MetadataEnforcer Proxy ensures proper access
|
||||
|
||||
### What This Means for Augmentation Users
|
||||
|
||||
**✅ If you use built-in augmentations**: No changes needed! They're all updated.
|
||||
|
||||
**⚠️ If you created custom storage augmentations**: Update your storage adapter to:
|
||||
- Wrap metadata with required `noun`/`verb` fields
|
||||
- Follow the internal/public method pattern
|
||||
- Use two-file storage approach
|
||||
|
||||
**⚠️ If you access relationship data**: Change `verb.type` to `verb.verb`
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### What are Augmentations?
|
||||
Augmentations are modular extensions that add functionality to Brainy without cluttering the core API. They follow a unified interface and can be:
|
||||
- **Auto-enabled**: Based on configuration (cache, index, storage)
|
||||
- **Manually registered**: For custom functionality
|
||||
- **Chained**: Multiple augmentations work together seamlessly
|
||||
- **Billion-scale ready**: Optimized for datasets with billions of nouns and verbs
|
||||
|
||||
### Augmentation Lifecycle
|
||||
1. **Registration**: Augmentations register before init()
|
||||
2. **Initialization**: Two-phase init (storage first, then others)
|
||||
3. **Execution**: Hook into operations (before/after/both)
|
||||
4. **Shutdown**: Clean teardown on brain.shutdown()
|
||||
|
||||
---
|
||||
|
||||
## Storage Augmentations (8 total)
|
||||
|
||||
### MemoryStorageAugmentation
|
||||
**Location**: `src/augmentations/storageAugmentations.ts`
|
||||
**Auto-enabled**: When `storage: 'memory'` or in test environments
|
||||
**Purpose**: In-memory storage for testing and temporary data
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: 'memory' })
|
||||
```
|
||||
|
||||
### FileSystemStorageAugmentation
|
||||
**Location**: `src/augmentations/storageAugmentations.ts`
|
||||
**Auto-enabled**: When `storage: 'filesystem'` or Node.js detected
|
||||
**Purpose**: Persistent file-based storage for Node.js applications
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './data' }
|
||||
})
|
||||
```
|
||||
|
||||
### OPFSStorageAugmentation
|
||||
**Location**: `src/augmentations/storageAugmentations.ts`
|
||||
**Auto-enabled**: When `storage: 'opfs'` or browser with OPFS support
|
||||
**Purpose**: Browser-based persistent storage using Origin Private File System
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: 'opfs' })
|
||||
```
|
||||
|
||||
### S3StorageAugmentation
|
||||
**Location**: `src/augmentations/storageAugmentations.ts`
|
||||
**Manual**: Requires AWS credentials
|
||||
**Purpose**: AWS S3-compatible cloud storage
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucket: 'my-bucket',
|
||||
region: 'us-east-1',
|
||||
credentials: { accessKeyId, secretAccessKey }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### R2StorageAugmentation
|
||||
**Location**: `src/augmentations/storageAugmentations.ts`
|
||||
**Manual**: Requires Cloudflare credentials
|
||||
**Purpose**: Cloudflare R2 storage (S3-compatible)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'r2',
|
||||
accountId: 'xxx',
|
||||
bucket: 'my-bucket',
|
||||
credentials: { accessKeyId, secretAccessKey }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### GCSStorageAugmentation
|
||||
**Location**: `src/augmentations/storageAugmentations.ts`
|
||||
**Manual**: Requires Google Cloud credentials
|
||||
**Purpose**: Google Cloud Storage
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
bucket: 'my-bucket',
|
||||
projectId: 'my-project'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### StorageAugmentation (base)
|
||||
**Location**: `src/augmentations/storageAugmentation.ts`
|
||||
**Purpose**: Base class for custom storage implementations
|
||||
|
||||
### DynamicStorageAugmentation
|
||||
**Location**: `src/augmentations/storageAugmentation.ts`
|
||||
**Purpose**: Runtime storage adapter switching
|
||||
|
||||
---
|
||||
|
||||
## Performance Augmentations (7 total)
|
||||
|
||||
### CacheAugmentation
|
||||
**Location**: `src/augmentations/cacheAugmentation.ts`
|
||||
**Auto-enabled**: When `cache: true` (default)
|
||||
**Purpose**: LRU cache for search results and frequent queries
|
||||
```typescript
|
||||
brain.clearCache() // Exposed via API
|
||||
brain.getCacheStats() // Cache hit/miss statistics
|
||||
```
|
||||
|
||||
### IndexAugmentation
|
||||
**Location**: `src/augmentations/indexAugmentation.ts`
|
||||
**Auto-enabled**: When `index: true` (default)
|
||||
**Purpose**: Metadata indexing for O(1) field lookups
|
||||
```typescript
|
||||
brain.rebuildMetadataIndex() // Exposed via API
|
||||
// Enables fast where queries:
|
||||
brain.find({ where: { category: 'tech' } })
|
||||
```
|
||||
|
||||
### MetricsAugmentation
|
||||
**Location**: `src/augmentations/metricsAugmentation.ts`
|
||||
**Auto-enabled**: Always active
|
||||
**Purpose**: Performance metrics and statistics collection
|
||||
```typescript
|
||||
brain.getStats() // Comprehensive metrics
|
||||
```
|
||||
|
||||
### MonitoringAugmentation
|
||||
**Location**: `src/augmentations/monitoringAugmentation.ts`
|
||||
**Manual**: Register for detailed monitoring
|
||||
**Purpose**: Real-time performance monitoring and alerts
|
||||
|
||||
### BatchProcessingAugmentation
|
||||
**Location**: `src/augmentations/batchProcessingAugmentation.ts`
|
||||
**Auto-enabled**: For batch operations
|
||||
**Purpose**: Optimizes bulk add/update/delete operations
|
||||
```typescript
|
||||
brain.addNouns([...]) // Automatically batched
|
||||
```
|
||||
|
||||
### RequestDeduplicatorAugmentation
|
||||
**Location**: `src/augmentations/requestDeduplicatorAugmentation.ts`
|
||||
**Auto-enabled**: Always active
|
||||
**Purpose**: Prevents duplicate concurrent operations
|
||||
|
||||
### ConnectionPoolAugmentation
|
||||
**Location**: `src/augmentations/connectionPoolAugmentation.ts`
|
||||
**Auto-enabled**: For network storage
|
||||
**Purpose**: Connection pooling for cloud storage adapters
|
||||
|
||||
---
|
||||
|
||||
## Data Integrity Augmentations (3 total)
|
||||
|
||||
**Auto-enabled**: When `wal: true`
|
||||
**Purpose**: Write-ahead logging for crash recovery
|
||||
```typescript
|
||||
const brain = new Brainy({ wal: true })
|
||||
// Automatic recovery on restart after crash
|
||||
```
|
||||
|
||||
### EntityRegistryAugmentation
|
||||
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
|
||||
**Auto-enabled**: For streaming operations
|
||||
**Purpose**: High-speed deduplication for real-time data
|
||||
```typescript
|
||||
// Prevents duplicate entities in streaming scenarios
|
||||
brain.add(data) // Automatically deduplicated
|
||||
```
|
||||
|
||||
### AutoRegisterEntitiesAugmentation
|
||||
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
|
||||
**Manual**: For automatic entity discovery
|
||||
**Purpose**: Auto-discovers and registers entities from data
|
||||
|
||||
---
|
||||
|
||||
## Intelligence Augmentations (2 total)
|
||||
|
||||
### NeuralImportAugmentation
|
||||
**Location**: `src/augmentations/neuralImport.ts`
|
||||
**Manual**: Via `brain.neuralImport()`
|
||||
**Purpose**: AI-powered smart data import
|
||||
```typescript
|
||||
const result = await brain.neuralImport(data, {
|
||||
confidenceThreshold: 0.7,
|
||||
autoApply: true
|
||||
})
|
||||
// Automatically detects entities and relationships
|
||||
```
|
||||
|
||||
### IntelligentVerbScoringAugmentation
|
||||
**Location**: `src/augmentations/intelligentVerbScoringAugmentation.ts`
|
||||
**Auto-enabled**: When verbs are used
|
||||
**Purpose**: ML-based relationship strength scoring
|
||||
```typescript
|
||||
brain.verbScoring.train(feedback)
|
||||
brain.verbScoring.getScore(verbId)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Communication Augmentations (4 total)
|
||||
|
||||
### APIServerAugmentation
|
||||
**Location**: `src/augmentations/apiServerAugmentation.ts`
|
||||
**Manual**: For server deployments
|
||||
**Purpose**: REST/WebSocket/MCP API server
|
||||
```typescript
|
||||
const augmentation = new APIServerAugmentation()
|
||||
await brain.registerAugmentation(augmentation)
|
||||
// Exposes full Brainy API over network
|
||||
```
|
||||
|
||||
### WebSocketConduitAugmentation
|
||||
**Location**: `src/augmentations/conduitAugmentations.ts`
|
||||
**Manual**: For Brainy-to-Brainy sync
|
||||
**Purpose**: Real-time sync between Brainy instances
|
||||
```typescript
|
||||
const conduit = new WebSocketConduitAugmentation()
|
||||
await conduit.establishConnection('ws://other-brain')
|
||||
```
|
||||
|
||||
### ServerSearchConduitAugmentation
|
||||
**Location**: `src/augmentations/serverSearchAugmentations.ts`
|
||||
**Manual**: For client-server search
|
||||
**Purpose**: Search remote Brainy instance, cache locally
|
||||
|
||||
### ServerSearchActivationAugmentation
|
||||
**Location**: `src/augmentations/serverSearchAugmentations.ts`
|
||||
**Manual**: Works with ServerSearchConduit
|
||||
**Purpose**: Triggers and manages server search operations
|
||||
|
||||
---
|
||||
|
||||
## External Integration (2 total)
|
||||
|
||||
### SynapseAugmentation (base)
|
||||
**Location**: `src/augmentations/synapseAugmentation.ts`
|
||||
**Purpose**: Base class for external platform integrations
|
||||
```typescript
|
||||
// Example: NotionSynapse, SlackSynapse, etc.
|
||||
class NotionSynapse extends SynapseAugmentation {
|
||||
async fetchData() { /* Notion API calls */ }
|
||||
async pushData() { /* Sync to Notion */ }
|
||||
}
|
||||
```
|
||||
|
||||
### ExampleFileSystemSynapse
|
||||
**Location**: `src/augmentations/synapseAugmentation.ts`
|
||||
**Purpose**: Example implementation for file system sync
|
||||
|
||||
---
|
||||
|
||||
## Augmentation Configuration
|
||||
|
||||
### Auto-Configuration
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
// These auto-register augmentations:
|
||||
storage: 'auto', // Storage augmentation
|
||||
cache: true, // Cache augmentation
|
||||
index: true, // Index augmentation
|
||||
metrics: true // Metrics augmentation
|
||||
})
|
||||
```
|
||||
|
||||
### Manual Registration
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
|
||||
// Register before init()
|
||||
const customAug = new MyCustomAugmentation()
|
||||
await brain.registerAugmentation(customAug)
|
||||
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
### Creating Custom Augmentations
|
||||
```typescript
|
||||
import { BaseAugmentation } from '@soulcraft/brainy'
|
||||
|
||||
class MyAugmentation extends BaseAugmentation {
|
||||
readonly name = 'my-augmentation'
|
||||
readonly timing = 'after' // before | after | both
|
||||
readonly operations = ['addNoun', 'search'] // Which ops to hook
|
||||
readonly priority = 10 // Execution order (lower = earlier)
|
||||
|
||||
protected async onInit(): Promise<void> {
|
||||
// Initialize your augmentation
|
||||
}
|
||||
|
||||
async execute<T>(
|
||||
operation: string,
|
||||
params: any,
|
||||
context?: AugmentationContext
|
||||
): Promise<T | void> {
|
||||
// Your augmentation logic
|
||||
if (operation === 'addNoun') {
|
||||
console.log('Noun added:', params)
|
||||
}
|
||||
}
|
||||
|
||||
protected async onShutdown(): Promise<void> {
|
||||
// Cleanup
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Augmentation Timing & Priority
|
||||
|
||||
### Timing Options
|
||||
- **`before`**: Runs before the operation (can modify params)
|
||||
- **`after`**: Runs after the operation (can see results)
|
||||
- **`both`**: Runs before AND after
|
||||
|
||||
### Priority (lower = earlier)
|
||||
1. Storage augmentations (priority: 0)
|
||||
2. Cache/Index augmentations (priority: 5-10)
|
||||
3. Monitoring/Metrics (priority: 15-20)
|
||||
4. Conduits/Synapses (priority: 20-30)
|
||||
|
||||
---
|
||||
|
||||
## Key Integration Points
|
||||
|
||||
### Where Augmentations Hook In
|
||||
|
||||
**Brainy Constructor**:
|
||||
- Storage augmentations register based on config
|
||||
- Cache/Index augmentations auto-register if enabled
|
||||
|
||||
**brain.init()**:
|
||||
- Two-phase initialization (storage first, then others)
|
||||
- Augmentations can access brain instance via context
|
||||
|
||||
**Operations** (addNoun, search, etc.):
|
||||
- Augmentations execute based on timing and operations filter
|
||||
- Can modify params (before) or see results (after)
|
||||
|
||||
**brain.shutdown()**:
|
||||
- All augmentations cleaned up in reverse order
|
||||
|
||||
---
|
||||
|
||||
## Performance Impact
|
||||
|
||||
Most augmentations have minimal overhead:
|
||||
- **Cache**: ~1ms per search (saves 10-100ms on hits)
|
||||
- **Index**: ~1ms per operation (saves 100ms+ on queries)
|
||||
- **Metrics**: <1ms per operation
|
||||
- **Storage**: Varies by adapter (memory: 0ms, S3: 50-200ms)
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Let auto-configuration work**: Most apps need zero manual config
|
||||
2. **Storage first**: Always configure storage before other augmentations
|
||||
3. **Use built-in augmentations**: They're optimized and battle-tested
|
||||
4. **Custom augmentations**: Extend BaseAugmentation for consistency
|
||||
5. **Respect timing**: Use 'before' to modify, 'after' to observe
|
||||
6. **Mind priority**: Lower numbers execute first
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Augmentation not working?
|
||||
```typescript
|
||||
// Check if registered
|
||||
brain.listAugmentations()
|
||||
|
||||
// Check if enabled
|
||||
brain.isAugmentationEnabled('cache')
|
||||
|
||||
// Enable/disable at runtime
|
||||
brain.enableAugmentation('cache')
|
||||
brain.disableAugmentation('cache')
|
||||
```
|
||||
|
||||
### Performance issues?
|
||||
```typescript
|
||||
// Check augmentation overhead
|
||||
const stats = brain.getStats()
|
||||
console.log(stats.augmentations)
|
||||
|
||||
// Disable non-critical augmentations
|
||||
brain.disableAugmentation('monitoring')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
|
||||
---
|
||||
|
||||
*Augmentations make Brainy infinitely extensible while keeping the core API clean and simple!*
|
||||
620
docs/augmentations/CONFIGURATION.md
Normal file
620
docs/augmentations/CONFIGURATION.md
Normal file
|
|
@ -0,0 +1,620 @@
|
|||
# Augmentation Configuration System
|
||||
|
||||
**Version**: 2.0.0
|
||||
**Status**: Production Ready
|
||||
|
||||
## Overview
|
||||
|
||||
The Brainy Augmentation Configuration System provides a VSCode-style extension architecture with multiple configuration sources, schema validation, and tool discovery. This system maintains Brainy's zero-config philosophy while enabling sophisticated enterprise configuration management.
|
||||
|
||||
## Table of Contents
|
||||
- [Quick Start](#quick-start)
|
||||
- [Configuration Sources](#configuration-sources)
|
||||
- [Creating Configurable Augmentations](#creating-configurable-augmentations)
|
||||
- [Configuration Discovery](#configuration-discovery)
|
||||
- [Runtime Configuration](#runtime-configuration)
|
||||
- [Environment Variables](#environment-variables)
|
||||
- [Configuration Files](#configuration-files)
|
||||
- [CLI Commands](#cli-commands)
|
||||
- [Tool Integration](#tool-integration)
|
||||
- [Migration Guide](#migration-guide)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using an Augmentation with Configuration
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// Zero-config (uses defaults)
|
||||
const brain = new Brainy()
|
||||
|
||||
// With custom configuration
|
||||
immediateWrites: true,
|
||||
checkpointInterval: 300000 // 5 minutes
|
||||
}))
|
||||
```
|
||||
|
||||
### Configuring via Environment Variables
|
||||
|
||||
```bash
|
||||
export BRAINY_AUG_CACHE_TTL=600000
|
||||
```
|
||||
|
||||
### Configuring via Files
|
||||
|
||||
Create a `.brainyrc` file in your project root:
|
||||
|
||||
```json
|
||||
{
|
||||
"augmentations": {
|
||||
"wal": {
|
||||
"enabled": true,
|
||||
"immediateWrites": true,
|
||||
"maxSize": 20971520
|
||||
},
|
||||
"cache": {
|
||||
"ttl": 600000,
|
||||
"maxSize": 2000
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration Sources
|
||||
|
||||
Configuration is resolved in the following priority order (highest to lowest):
|
||||
|
||||
1. **Runtime Updates** - Dynamic configuration changes via API
|
||||
2. **Constructor Parameters** - Code-time configuration
|
||||
3. **Environment Variables** - `BRAINY_AUG_<NAME>_<KEY>`
|
||||
4. **Configuration Files** - `.brainyrc`, `brainy.config.json`
|
||||
5. **Schema Defaults** - Default values from manifest
|
||||
|
||||
### Resolution Example
|
||||
|
||||
```typescript
|
||||
// Schema default
|
||||
{ maxSize: 10485760 }
|
||||
|
||||
// File configuration (.brainyrc)
|
||||
{ maxSize: 20971520 }
|
||||
|
||||
// Environment variable
|
||||
|
||||
// Constructor parameter
|
||||
|
||||
// Final resolved value: 41943040 (constructor wins)
|
||||
```
|
||||
|
||||
## Creating Configurable Augmentations
|
||||
|
||||
### Step 1: Extend ConfigurableAugmentation
|
||||
|
||||
```typescript
|
||||
import { ConfigurableAugmentation, AugmentationManifest } from '@soulcraft/brainy'
|
||||
|
||||
export class MyAugmentation extends ConfigurableAugmentation {
|
||||
name = 'my-augmentation'
|
||||
timing = 'around' as const
|
||||
metadata = 'none' as const
|
||||
operations = ['search', 'add']
|
||||
priority = 50
|
||||
|
||||
constructor(config?: MyConfig) {
|
||||
super(config) // Handles configuration resolution
|
||||
}
|
||||
|
||||
// Required: Provide manifest for discovery
|
||||
getManifest(): AugmentationManifest {
|
||||
return {
|
||||
id: 'my-augmentation',
|
||||
name: 'My Augmentation',
|
||||
version: '1.0.0',
|
||||
description: 'Does something amazing',
|
||||
category: 'performance',
|
||||
configSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
enabled: {
|
||||
type: 'boolean',
|
||||
default: true,
|
||||
description: 'Enable this augmentation'
|
||||
},
|
||||
threshold: {
|
||||
type: 'number',
|
||||
default: 100,
|
||||
minimum: 1,
|
||||
maximum: 1000,
|
||||
description: 'Processing threshold'
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Optional: Handle runtime configuration changes
|
||||
protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise<void> {
|
||||
if (newConfig.threshold !== oldConfig.threshold) {
|
||||
// React to threshold change
|
||||
this.updateThreshold(newConfig.threshold)
|
||||
}
|
||||
}
|
||||
|
||||
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
|
||||
if (!this.config.enabled) {
|
||||
return next()
|
||||
}
|
||||
|
||||
// Your augmentation logic here
|
||||
return next()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Define Configuration Interface
|
||||
|
||||
```typescript
|
||||
interface MyConfig {
|
||||
enabled?: boolean
|
||||
threshold?: number
|
||||
mode?: 'fast' | 'balanced' | 'thorough'
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Add JSON Schema in Manifest
|
||||
|
||||
```typescript
|
||||
configSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
enabled: {
|
||||
type: 'boolean',
|
||||
default: true,
|
||||
description: 'Enable augmentation'
|
||||
},
|
||||
threshold: {
|
||||
type: 'number',
|
||||
default: 100,
|
||||
minimum: 1,
|
||||
maximum: 1000,
|
||||
description: 'Processing threshold'
|
||||
},
|
||||
mode: {
|
||||
type: 'string',
|
||||
default: 'balanced',
|
||||
enum: ['fast', 'balanced', 'thorough'],
|
||||
description: 'Processing mode'
|
||||
}
|
||||
},
|
||||
required: [],
|
||||
additionalProperties: false
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration Discovery
|
||||
|
||||
The Discovery API allows tools to discover and configure augmentations dynamically:
|
||||
|
||||
```typescript
|
||||
import { AugmentationDiscovery } from '@soulcraft/brainy'
|
||||
|
||||
const discovery = new AugmentationDiscovery(brain.augmentations)
|
||||
|
||||
// Discover all augmentations with manifests
|
||||
const listings = await discovery.discover({
|
||||
includeConfig: true,
|
||||
includeSchema: true
|
||||
})
|
||||
|
||||
// Get configuration schema
|
||||
const schema = await discovery.getConfigSchema('wal')
|
||||
|
||||
// Validate configuration
|
||||
const validation = await discovery.validateConfig('wal', {
|
||||
enabled: true,
|
||||
maxSize: 'invalid' // Will fail validation
|
||||
})
|
||||
|
||||
// Update configuration at runtime
|
||||
await discovery.updateConfig('wal', {
|
||||
checkpointInterval: 120000
|
||||
})
|
||||
```
|
||||
|
||||
## Runtime Configuration
|
||||
|
||||
### Update Configuration Dynamically
|
||||
|
||||
```typescript
|
||||
// Get augmentation
|
||||
const wal = brain.augmentations.get('wal')
|
||||
|
||||
// Update configuration
|
||||
await wal.updateConfig({
|
||||
checkpointInterval: 300000
|
||||
})
|
||||
|
||||
// Get current configuration
|
||||
const config = wal.getConfig()
|
||||
```
|
||||
|
||||
### React to Configuration Changes
|
||||
|
||||
```typescript
|
||||
class MyAugmentation extends ConfigurableAugmentation {
|
||||
protected async onConfigChange(newConfig: any, oldConfig: any): Promise<void> {
|
||||
// Stop old processes
|
||||
if (oldConfig.enabled && !newConfig.enabled) {
|
||||
await this.stop()
|
||||
}
|
||||
|
||||
// Start new processes
|
||||
if (!oldConfig.enabled && newConfig.enabled) {
|
||||
await this.start()
|
||||
}
|
||||
|
||||
// Update settings
|
||||
if (newConfig.interval !== oldConfig.interval) {
|
||||
this.rescheduleTimer(newConfig.interval)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Naming Convention
|
||||
|
||||
```bash
|
||||
BRAINY_AUG_<AUGMENTATION_ID>_<CONFIG_KEY>=value
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
|
||||
# Cache augmentation
|
||||
BRAINY_AUG_CACHE_ENABLED=true
|
||||
BRAINY_AUG_CACHE_MAX_SIZE=2000
|
||||
BRAINY_AUG_CACHE_TTL=600000
|
||||
|
||||
# Complex values (JSON)
|
||||
BRAINY_AUG_MYAUG_FILTERS='["*.js","*.ts"]'
|
||||
BRAINY_AUG_MYAUG_OPTIONS='{"deep":true,"follow":false}'
|
||||
```
|
||||
|
||||
### Docker Example
|
||||
|
||||
```dockerfile
|
||||
ENV BRAINY_AUG_CACHE_TTL=600000
|
||||
```
|
||||
|
||||
## Configuration Files
|
||||
|
||||
### File Locations (Priority Order)
|
||||
|
||||
1. `.brainyrc` (current directory)
|
||||
2. `.brainyrc.json` (current directory)
|
||||
3. `brainy.config.json` (current directory)
|
||||
4. `~/.brainy/config.json` (user home)
|
||||
5. `~/.brainyrc` (user home)
|
||||
|
||||
### File Format
|
||||
|
||||
```json
|
||||
{
|
||||
"augmentations": {
|
||||
"wal": {
|
||||
"enabled": true,
|
||||
"immediateWrites": true,
|
||||
"maxSize": 20971520,
|
||||
"checkpointInterval": 300000
|
||||
},
|
||||
"cache": {
|
||||
"enabled": true,
|
||||
"maxSize": 2000,
|
||||
"ttl": 600000
|
||||
},
|
||||
"metrics": {
|
||||
"enabled": false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Per-Environment Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"augmentations": {
|
||||
"wal": {
|
||||
"development": {
|
||||
"enabled": true,
|
||||
"immediateWrites": true,
|
||||
"maxSize": 5242880
|
||||
},
|
||||
"production": {
|
||||
"enabled": true,
|
||||
"immediateWrites": false,
|
||||
"maxSize": 104857600,
|
||||
"checkpointInterval": 60000
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## CLI Commands
|
||||
|
||||
### List Augmentations with Configuration
|
||||
|
||||
```bash
|
||||
# Show all augmentations with config status
|
||||
brainy augment list --detailed
|
||||
|
||||
# Show configuration for specific augmentation
|
||||
brainy augment config wal
|
||||
|
||||
# Set configuration value
|
||||
brainy augment config wal --set immediateWrites=true
|
||||
|
||||
# Show environment variable names
|
||||
brainy augment config wal --env
|
||||
|
||||
# Export configuration schema
|
||||
brainy augment schema wal > wal-schema.json
|
||||
|
||||
# Validate configuration file
|
||||
brainy augment validate --file config.json
|
||||
```
|
||||
|
||||
### Interactive Configuration
|
||||
|
||||
```bash
|
||||
# Interactive configuration wizard
|
||||
brainy augment configure wal
|
||||
|
||||
? Operation mode?
|
||||
❯ Performance (immediate writes)
|
||||
Durability (synchronous writes)
|
||||
Custom
|
||||
? Maximum log size? (10MB) 20MB
|
||||
? Checkpoint interval? (1 minute) 5 minutes
|
||||
|
||||
Configuration saved to .brainyrc
|
||||
```
|
||||
|
||||
## Tool Integration
|
||||
|
||||
### Brain-Cloud Explorer UI
|
||||
|
||||
```typescript
|
||||
// Auto-generate configuration form from schema
|
||||
const ConfigurationUI = ({ augmentationId }) => {
|
||||
const [manifest, setManifest] = useState(null)
|
||||
const [config, setConfig] = useState({})
|
||||
|
||||
useEffect(() => {
|
||||
// Fetch manifest with schema
|
||||
fetch(`/api/augmentations/${augmentationId}/manifest`)
|
||||
.then(res => res.json())
|
||||
.then(setManifest)
|
||||
|
||||
// Get current configuration
|
||||
discovery.getConfig(augmentationId)
|
||||
.then(setConfig)
|
||||
}, [augmentationId])
|
||||
|
||||
const handleSave = async (newConfig) => {
|
||||
// Validate configuration
|
||||
const validation = await fetch(`/api/augmentations/${augmentationId}/validate`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(newConfig)
|
||||
}).then(res => res.json())
|
||||
|
||||
if (validation.valid) {
|
||||
// Apply configuration
|
||||
await discovery.updateConfig(augmentationId, newConfig)
|
||||
}
|
||||
}
|
||||
|
||||
// Render form based on schema
|
||||
return <SchemaForm
|
||||
schema={manifest?.configSchema}
|
||||
values={config}
|
||||
onSubmit={handleSave}
|
||||
/>
|
||||
}
|
||||
```
|
||||
|
||||
### VS Code Extension
|
||||
|
||||
```json
|
||||
// package.json contribution points
|
||||
{
|
||||
"contributes": {
|
||||
"configuration": {
|
||||
"title": "Brainy Augmentations",
|
||||
"properties": {
|
||||
"brainy.augmentations.wal.enabled": {
|
||||
"type": "boolean",
|
||||
"default": true,
|
||||
},
|
||||
"brainy.augmentations.wal.maxSize": {
|
||||
"type": "number",
|
||||
"default": 10485760,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### Migrating from BaseAugmentation
|
||||
|
||||
**Before:**
|
||||
```typescript
|
||||
export class MyAugmentation extends BaseAugmentation {
|
||||
constructor(config: MyConfig = {}) {
|
||||
super()
|
||||
this.config = {
|
||||
enabled: config.enabled ?? true,
|
||||
threshold: config.threshold ?? 100
|
||||
}
|
||||
}
|
||||
|
||||
// No manifest
|
||||
// No config discovery
|
||||
// No runtime updates
|
||||
}
|
||||
```
|
||||
|
||||
**After:**
|
||||
```typescript
|
||||
export class MyAugmentation extends ConfigurableAugmentation {
|
||||
constructor(config?: MyConfig) {
|
||||
super(config) // Config resolution handled automatically
|
||||
}
|
||||
|
||||
getManifest(): AugmentationManifest {
|
||||
return {
|
||||
id: 'my-augmentation',
|
||||
name: 'My Augmentation',
|
||||
version: '1.0.0',
|
||||
description: 'Does something amazing',
|
||||
category: 'performance',
|
||||
configSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
enabled: { type: 'boolean', default: true },
|
||||
threshold: { type: 'number', default: 100 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Optional: Handle config changes
|
||||
protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise<void> {
|
||||
// React to changes
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Backwards Compatibility
|
||||
|
||||
The system maintains full backwards compatibility:
|
||||
|
||||
1. **BaseAugmentation still works** - Existing augmentations continue to function
|
||||
2. **Constructor config still works** - Existing configuration patterns preserved
|
||||
3. **Zero-config still works** - Defaults are applied automatically
|
||||
4. **Progressive enhancement** - Add features as needed
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Always Provide Defaults
|
||||
|
||||
```typescript
|
||||
configSchema: {
|
||||
properties: {
|
||||
enabled: {
|
||||
type: 'boolean',
|
||||
default: true, // Always provide defaults
|
||||
description: 'Enable this feature'
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Use Descriptive Configuration Keys
|
||||
|
||||
```typescript
|
||||
// Good
|
||||
checkpointInterval: 60000
|
||||
|
||||
// Bad
|
||||
ci: 60000
|
||||
```
|
||||
|
||||
### 3. Validate Configuration
|
||||
|
||||
```typescript
|
||||
protected async onConfigChange(newConfig: any, oldConfig: any): Promise<void> {
|
||||
// Validate before applying
|
||||
if (newConfig.maxSize < 1048576) {
|
||||
throw new Error('maxSize must be at least 1MB')
|
||||
}
|
||||
|
||||
// Apply changes
|
||||
this.maxSize = newConfig.maxSize
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Document Environment Variables
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Environment Variables:
|
||||
* - BRAINY_AUG_MYAUG_ENABLED: Enable augmentation (boolean)
|
||||
* - BRAINY_AUG_MYAUG_THRESHOLD: Processing threshold (number)
|
||||
* - BRAINY_AUG_MYAUG_MODE: Processing mode (fast|balanced|thorough)
|
||||
*/
|
||||
```
|
||||
|
||||
### 5. Provide Configuration Examples
|
||||
|
||||
```typescript
|
||||
configExamples: [
|
||||
{
|
||||
name: 'Production',
|
||||
description: 'Optimized for production use',
|
||||
config: {
|
||||
enabled: true,
|
||||
mode: 'thorough',
|
||||
threshold: 500
|
||||
}
|
||||
},
|
||||
{
|
||||
name: 'Development',
|
||||
description: 'Lightweight for development',
|
||||
config: {
|
||||
enabled: true,
|
||||
mode: 'fast',
|
||||
threshold: 10
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Configuration Not Loading
|
||||
|
||||
1. Check file locations and names
|
||||
2. Verify JSON syntax in config files
|
||||
3. Check environment variable names (case-sensitive)
|
||||
4. Use `brainy augment config <name> --debug` to see resolution
|
||||
|
||||
### Validation Errors
|
||||
|
||||
1. Check schema requirements
|
||||
2. Verify data types match schema
|
||||
3. Check minimum/maximum constraints
|
||||
4. Use discovery API to validate before applying
|
||||
|
||||
### Runtime Updates Not Working
|
||||
|
||||
1. Ensure augmentation extends ConfigurableAugmentation
|
||||
2. Implement onConfigChange if needed
|
||||
3. Check for validation errors
|
||||
4. Verify augmentation is initialized
|
||||
|
||||
## API Reference
|
||||
|
||||
See the [Discovery API Documentation](./discovery-api.md) for complete API details.
|
||||
|
||||
## Examples
|
||||
|
||||
See the [examples directory](../../examples/augmentation-config/) for complete working examples.
|
||||
527
docs/augmentations/DEVELOPER-GUIDE.md
Normal file
527
docs/augmentations/DEVELOPER-GUIDE.md
Normal file
|
|
@ -0,0 +1,527 @@
|
|||
# 🛠️ Brainy Augmentation Developer Guide
|
||||
|
||||
> **How to create, test, and use augmentations in Brainy**
|
||||
>
|
||||
> **⚠️ Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### What Changed?
|
||||
|
||||
1. **Metadata Structure**: All metadata now requires type fields (`noun` or `verb`)
|
||||
2. **Property Rename**: `verb.type` → `verb.verb` for relationships
|
||||
3. **Two-File Storage**: Vectors and metadata stored separately for performance
|
||||
4. **Return Types**: Storage methods distinguish between internal (pure) and public (WithMetadata) returns
|
||||
|
||||
### Migration Checklist
|
||||
|
||||
- [ ] Update metadata creation to include required `noun` field
|
||||
- [ ] Change `verb.type` to `verb.verb` in all relationship code
|
||||
- [ ] Update storage adapter methods to follow internal/public pattern
|
||||
- [ ] Ensure metadata access uses correct structure
|
||||
|
||||
### Quick Migration Example
|
||||
|
||||
```typescript
|
||||
// ❌ v3.x
|
||||
const verb = {
|
||||
type: 'relatedTo',
|
||||
sourceId: 'a',
|
||||
targetId: 'b'
|
||||
}
|
||||
if (verb.type === 'relatedTo') { ... }
|
||||
|
||||
// ✅ Current
|
||||
const verb = {
|
||||
verb: 'relatedTo',
|
||||
sourceId: 'a',
|
||||
targetId: 'b'
|
||||
}
|
||||
const metadata: VerbMetadata = {
|
||||
verb: 'relatedTo',
|
||||
sourceId: 'a',
|
||||
targetId: 'b'
|
||||
}
|
||||
if (verb.verb === 'relatedTo') { ... }
|
||||
```
|
||||
|
||||
## Quick Start: Your First Augmentation
|
||||
|
||||
```typescript
|
||||
import { BaseAugmentation, BrainyAugmentation, AugmentationContext } from '@soulcraft/brainy'
|
||||
|
||||
export class MyFirstAugmentation extends BaseAugmentation {
|
||||
readonly name = 'my-first-augmentation'
|
||||
readonly timing = 'after' as const // When to run: before | after | both
|
||||
readonly operations = ['add'] as const // Which operations to hook
|
||||
readonly priority = 10 // Execution order (lower = first)
|
||||
|
||||
protected async onInit(): Promise<void> {
|
||||
// Initialize your augmentation
|
||||
console.log('MyFirstAugmentation initialized!')
|
||||
}
|
||||
|
||||
async execute<T = any>(
|
||||
operation: string,
|
||||
params: any,
|
||||
context?: AugmentationContext
|
||||
): Promise<T | void> {
|
||||
// Your augmentation logic
|
||||
if (operation === 'add') {
|
||||
console.log('Noun added:', params.noun)
|
||||
|
||||
// Access metadata correctly
|
||||
if (params.noun?.metadata) {
|
||||
console.log('Noun type:', params.noun.metadata.noun) // Required field
|
||||
}
|
||||
|
||||
// You can access the brain instance
|
||||
const stats = await context?.brain.getStats()
|
||||
console.log('Total nouns:', stats.totalNouns)
|
||||
}
|
||||
}
|
||||
|
||||
protected async onShutdown(): Promise<void> {
|
||||
// Cleanup
|
||||
console.log('MyFirstAugmentation shutting down')
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Using Your Augmentation
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { MyFirstAugmentation } from './my-first-augmentation'
|
||||
|
||||
const brain = new Brainy()
|
||||
|
||||
// Register before init()
|
||||
brain.augmentations.register(new MyFirstAugmentation())
|
||||
|
||||
await brain.init()
|
||||
|
||||
// Now your augmentation runs automatically!
|
||||
await brain.add('Hello World')
|
||||
// Console: "Noun added: { id: '...', vector: [...], metadata: {} }"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Augmentation Lifecycle
|
||||
|
||||
### 1. Registration Phase
|
||||
```typescript
|
||||
const aug = new MyAugmentation()
|
||||
brain.augmentations.register(aug) // Before brain.init()!
|
||||
```
|
||||
|
||||
### 2. Initialization Phase
|
||||
```typescript
|
||||
await brain.init() // Calls aug.initialize() internally
|
||||
// Your onInit() method runs here
|
||||
```
|
||||
|
||||
### 3. Execution Phase
|
||||
```typescript
|
||||
await brain.add('data') // Your execute() method runs
|
||||
```
|
||||
|
||||
### 4. Shutdown Phase
|
||||
```typescript
|
||||
await brain.shutdown() // Your onShutdown() method runs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Timing Options
|
||||
|
||||
### `before` - Modify Input
|
||||
```typescript
|
||||
class ValidationAugmentation extends BaseAugmentation {
|
||||
readonly timing = 'before' as const
|
||||
|
||||
async execute<T>(operation: string, params: any): Promise<any> {
|
||||
if (operation === 'add') {
|
||||
// Validate and/or modify params
|
||||
if (!params.content) {
|
||||
throw new Error('Content required')
|
||||
}
|
||||
// Return modified params
|
||||
return { ...params, validated: true }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `after` - React to Results
|
||||
```typescript
|
||||
class LoggingAugmentation extends BaseAugmentation {
|
||||
readonly timing = 'after' as const
|
||||
|
||||
async execute<T>(operation: string, params: any): Promise<void> {
|
||||
if (operation === 'search') {
|
||||
console.log(`Search for "${params.query}" returned ${params.result.length} results`)
|
||||
}
|
||||
// Don't return anything - just observe
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `both` - Before AND After
|
||||
```typescript
|
||||
class TimingAugmentation extends BaseAugmentation {
|
||||
readonly timing = 'both' as const
|
||||
private startTime?: number
|
||||
|
||||
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
|
||||
if (!this.startTime) {
|
||||
// Before execution
|
||||
this.startTime = Date.now()
|
||||
} else {
|
||||
// After execution
|
||||
const duration = Date.now() - this.startTime
|
||||
console.log(`${operation} took ${duration}ms`)
|
||||
this.startTime = undefined
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operation Hooks
|
||||
|
||||
### Core Operations You Can Hook
|
||||
```typescript
|
||||
readonly operations = [
|
||||
'add', // Adding data
|
||||
'update', // Updating data
|
||||
'delete', // Deleting data
|
||||
'get', // Retrieving data
|
||||
'search', // Searching
|
||||
'find', // Triple Intelligence queries
|
||||
'relate', // Adding relationships
|
||||
'unrelate', // Removing relationships
|
||||
'clear', // Clearing data
|
||||
'all' // Hook ALL operations
|
||||
] as const
|
||||
```
|
||||
|
||||
### Example: Multi-Operation Hook
|
||||
```typescript
|
||||
class AuditAugmentation extends BaseAugmentation {
|
||||
readonly operations = ['add', 'update', 'delete'] as const
|
||||
|
||||
async execute<T>(operation: string, params: any): Promise<void> {
|
||||
// Log all data modifications
|
||||
await this.logToAuditTrail(operation, params)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Accessing Brain Context
|
||||
|
||||
```typescript
|
||||
class ContextAwareAugmentation extends BaseAugmentation {
|
||||
async execute<T>(
|
||||
operation: string,
|
||||
params: any,
|
||||
context?: AugmentationContext
|
||||
): Promise<void> {
|
||||
// Access the brain instance
|
||||
const brain = context?.brain
|
||||
if (!brain) return
|
||||
|
||||
// Use any brain method
|
||||
const stats = await brain.getStats()
|
||||
const size = await brain.size()
|
||||
const results = await brain.search('query')
|
||||
|
||||
// Access other augmentations
|
||||
const cache = brain.augmentations.get('cache')
|
||||
if (cache) {
|
||||
await cache.clear()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
### 1. Backup Augmentation
|
||||
```typescript
|
||||
class BackupAugmentation extends BaseAugmentation {
|
||||
readonly name = 'backup'
|
||||
readonly timing = 'after' as const
|
||||
readonly operations = ['add', 'update', 'delete'] as const
|
||||
readonly priority = 5
|
||||
|
||||
private changes = 0
|
||||
private readonly backupThreshold = 100
|
||||
|
||||
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
|
||||
this.changes++
|
||||
|
||||
if (this.changes >= this.backupThreshold) {
|
||||
await this.performBackup(context?.brain)
|
||||
this.changes = 0
|
||||
}
|
||||
}
|
||||
|
||||
private async performBackup(brain?: any): Promise<void> {
|
||||
if (!brain) return
|
||||
// Create instant COW snapshot
|
||||
const snapshotName = `backup-${Date.now()}`
|
||||
await brain.fork(snapshotName)
|
||||
console.log(`Automatic snapshot created: ${snapshotName}`)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Rate Limiting Augmentation
|
||||
```typescript
|
||||
class RateLimitAugmentation extends BaseAugmentation {
|
||||
readonly name = 'rate-limit'
|
||||
readonly timing = 'before' as const
|
||||
readonly operations = ['search', 'find'] as const
|
||||
readonly priority = 100 // High priority - run first
|
||||
|
||||
private requests = new Map<string, number[]>()
|
||||
private readonly limit = 100 // 100 requests
|
||||
private readonly window = 60000 // per minute
|
||||
|
||||
async execute<T>(operation: string, params: any): Promise<void> {
|
||||
const now = Date.now()
|
||||
const key = params.userId || 'anonymous'
|
||||
|
||||
// Get request timestamps
|
||||
const timestamps = this.requests.get(key) || []
|
||||
|
||||
// Remove old timestamps
|
||||
const recent = timestamps.filter(t => now - t < this.window)
|
||||
|
||||
// Check limit
|
||||
if (recent.length >= this.limit) {
|
||||
throw new Error('Rate limit exceeded')
|
||||
}
|
||||
|
||||
// Add current request
|
||||
recent.push(now)
|
||||
this.requests.set(key, recent)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Encryption Augmentation
|
||||
```typescript
|
||||
class EncryptionAugmentation extends BaseAugmentation {
|
||||
readonly name = 'encryption'
|
||||
readonly timing = 'both' as const
|
||||
readonly operations = ['add', 'get'] as const
|
||||
readonly priority = 90 // Run early
|
||||
|
||||
async execute<T>(operation: string, params: any): Promise<any> {
|
||||
if (operation === 'add') {
|
||||
// Encrypt before storing
|
||||
if (params.metadata?.sensitive) {
|
||||
params.content = await this.encrypt(params.content)
|
||||
params.encrypted = true
|
||||
}
|
||||
return params
|
||||
}
|
||||
|
||||
if (operation === 'get' && params.result?.encrypted) {
|
||||
// Decrypt after retrieval
|
||||
params.result.content = await this.decrypt(params.result.content)
|
||||
delete params.result.encrypted
|
||||
return params.result
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Your Augmentation
|
||||
|
||||
```typescript
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { MyAugmentation } from './my-augmentation'
|
||||
|
||||
describe('MyAugmentation', () => {
|
||||
it('should hook into addNoun', async () => {
|
||||
const brain = new Brainy({ storage: 'memory' })
|
||||
const aug = new MyAugmentation()
|
||||
|
||||
// Spy on the execute method
|
||||
const executeSpy = vi.spyOn(aug, 'execute')
|
||||
|
||||
brain.augmentations.register(aug)
|
||||
await brain.init()
|
||||
|
||||
// Trigger the augmentation
|
||||
await brain.add('test data')
|
||||
|
||||
// Verify it was called
|
||||
expect(executeSpy).toHaveBeenCalledWith(
|
||||
'add',
|
||||
expect.objectContaining({ content: 'test data' }),
|
||||
expect.any(Object)
|
||||
)
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Use Proper Timing
|
||||
- `before`: Validation, modification, rate limiting
|
||||
- `after`: Logging, metrics, side effects
|
||||
- `both`: Timing, tracing, wrapping
|
||||
|
||||
### 2. Set Appropriate Priority
|
||||
```typescript
|
||||
// Priority guidelines
|
||||
100: Critical (auth, rate limiting)
|
||||
50: Important (validation, transformation)
|
||||
10: Normal (logging, metrics)
|
||||
1: Optional (debugging, tracing)
|
||||
```
|
||||
|
||||
### 3. Handle Errors Gracefully
|
||||
```typescript
|
||||
async execute<T>(operation: string, params: any): Promise<void> {
|
||||
try {
|
||||
await this.riskyOperation()
|
||||
} catch (error) {
|
||||
// Log but don't break the main operation
|
||||
console.error(`Augmentation error in ${this.name}:`, error)
|
||||
// Optionally report to monitoring
|
||||
this.reportError(error)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Be Performance Conscious
|
||||
```typescript
|
||||
class CachedAugmentation extends BaseAugmentation {
|
||||
private cache = new Map<string, any>()
|
||||
|
||||
async execute<T>(operation: string, params: any): Promise<any> {
|
||||
const key = this.getCacheKey(params)
|
||||
|
||||
// Check cache first
|
||||
if (this.cache.has(key)) {
|
||||
return this.cache.get(key)
|
||||
}
|
||||
|
||||
// Expensive operation
|
||||
const result = await this.expensiveOperation(params)
|
||||
this.cache.set(key, result)
|
||||
|
||||
return result
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Clean Up Resources
|
||||
```typescript
|
||||
protected async onShutdown(): Promise<void> {
|
||||
// Close connections
|
||||
await this.connection?.close()
|
||||
|
||||
// Clear intervals
|
||||
clearInterval(this.interval)
|
||||
|
||||
// Flush buffers
|
||||
await this.flush()
|
||||
|
||||
// Clear caches
|
||||
this.cache.clear()
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Publishing Your Augmentation (Future)
|
||||
|
||||
### Package Structure
|
||||
```
|
||||
my-augmentation/
|
||||
├── src/
|
||||
│ └── index.ts # Your augmentation
|
||||
├── dist/ # Built output
|
||||
├── tests/
|
||||
│ └── augmentation.test.ts
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── README.md
|
||||
```
|
||||
|
||||
### package.json
|
||||
```json
|
||||
{
|
||||
"name": "@mycompany/brainy-custom-augmentation",
|
||||
"version": "1.0.0",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"keywords": ["brainy-augmentation"],
|
||||
"peerDependencies": {
|
||||
"@soulcraft/brainy": ">=2.0.0"
|
||||
},
|
||||
"brainy": {
|
||||
"type": "augmentation",
|
||||
"class": "CustomAugmentation",
|
||||
"timing": "after",
|
||||
"operations": ["add"],
|
||||
"priority": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Future: Brain Cloud Registry
|
||||
```bash
|
||||
# Coming in 2.1+
|
||||
npm run build
|
||||
npm test
|
||||
brainy publish # Publishes to brain-cloud registry
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
|
||||
### Q: Can I modify the operation result?
|
||||
**A**: Yes, if `timing: 'before'`, return modified params. If `timing: 'after'`, you can see but not modify results.
|
||||
|
||||
### Q: Can augmentations communicate?
|
||||
**A**: Yes, through the context: `context.brain.augmentations.get('other-augmentation')`
|
||||
|
||||
### Q: What if my augmentation fails?
|
||||
**A**: Handle errors internally. Don't break the main operation unless critical.
|
||||
|
||||
### Q: Can I use async operations?
|
||||
**A**: Yes, everything is async-friendly.
|
||||
|
||||
### Q: How do I access storage directly?
|
||||
**A**: Through context: `context.brain.storage` (but prefer using brain methods)
|
||||
|
||||
---
|
||||
|
||||
## Get Help
|
||||
|
||||
- **GitHub**: [github.com/soulcraft/brainy](https://github.com/soulcraft/brainy)
|
||||
- **Discord**: [discord.gg/brainy](https://discord.gg/brainy)
|
||||
- **Examples**: See `/examples/augmentations/` in the repo
|
||||
|
||||
---
|
||||
|
||||
*Start building your augmentation today! The marketplace is coming in 2.1 🚀*
|
||||
669
docs/augmentations/EXAMPLES.md
Normal file
669
docs/augmentations/EXAMPLES.md
Normal file
|
|
@ -0,0 +1,669 @@
|
|||
# Augmentation Examples - Import, Store, Export
|
||||
|
||||
This guide shows two complete workflows:
|
||||
1. **Simple Handler** - Just import a new file type
|
||||
2. **Full Augmentation** - Import + Store + Export (premium-ready)
|
||||
|
||||
---
|
||||
|
||||
## Workflow 1: Simple Handler (Import Only)
|
||||
|
||||
**Use case:** You want to import a new file type (e.g., CAD files) into Brainy's knowledge graph.
|
||||
|
||||
### Step 1: Create the Handler
|
||||
|
||||
```typescript
|
||||
// src/handlers/CADHandler.ts
|
||||
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
import type { FormatHandlerOptions, ProcessedData } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
import { parseCAD } from 'cad-parser' // Your parsing library
|
||||
|
||||
export class CADHandler extends BaseFormatHandler {
|
||||
readonly format = 'cad'
|
||||
|
||||
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
|
||||
if (typeof data === 'object' && 'filename' in data) {
|
||||
const mimeType = this.getMimeType(data)
|
||||
return this.mimeTypeMatches(mimeType, ['image/vnd.dwg', 'image/vnd.dxf'])
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
|
||||
|
||||
// Parse CAD file
|
||||
const cad = await parseCAD(buffer)
|
||||
|
||||
// Extract entities for knowledge graph
|
||||
const entities = []
|
||||
|
||||
// Document entity
|
||||
entities.push({
|
||||
type: 'CADDocument',
|
||||
filename: options.filename,
|
||||
units: cad.units,
|
||||
bounds: cad.bounds,
|
||||
version: cad.version
|
||||
})
|
||||
|
||||
// Layer entities
|
||||
for (const layer of cad.layers) {
|
||||
entities.push({
|
||||
type: 'CADLayer',
|
||||
name: layer.name,
|
||||
color: layer.color,
|
||||
visible: layer.visible
|
||||
})
|
||||
}
|
||||
|
||||
// Object entities
|
||||
for (const obj of cad.objects) {
|
||||
entities.push({
|
||||
type: 'CADObject',
|
||||
objectType: obj.type,
|
||||
layer: obj.layer,
|
||||
geometry: obj.geometry
|
||||
})
|
||||
}
|
||||
|
||||
return {
|
||||
format: 'cad',
|
||||
data: entities,
|
||||
metadata: {
|
||||
rowCount: entities.length,
|
||||
fields: ['type', 'name', 'geometry'],
|
||||
processingTime: Date.now() - startTime,
|
||||
layerCount: cad.layers.length,
|
||||
objectCount: cad.objects.length
|
||||
},
|
||||
filename: options.filename
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Register the Handler
|
||||
|
||||
```typescript
|
||||
// src/index.ts
|
||||
import { globalHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
import { CADHandler } from './handlers/CADHandler.js'
|
||||
|
||||
// Register handler globally
|
||||
globalHandlerRegistry.registerHandler({
|
||||
name: 'cad',
|
||||
mimeTypes: ['image/vnd.dwg', 'image/vnd.dxf'],
|
||||
extensions: ['.dwg', '.dxf', '.dwf'],
|
||||
loader: async () => new CADHandler()
|
||||
})
|
||||
```
|
||||
|
||||
### Step 3: Use It
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Import CAD file - automatically routed to CADHandler
|
||||
const result = await brain.import({
|
||||
type: 'file',
|
||||
data: cadFileBuffer,
|
||||
filename: 'floor-plan.dwg'
|
||||
})
|
||||
|
||||
console.log(`Imported ${result.entities.length} CAD entities`)
|
||||
|
||||
// Query the imported data
|
||||
const layers = await brain.find({ type: 'CADLayer' })
|
||||
const objects = await brain.find({ type: 'CADObject', layer: 'WALLS' })
|
||||
```
|
||||
|
||||
**That's it!** Simple handlers just import data. Brainy handles storage automatically.
|
||||
|
||||
---
|
||||
|
||||
## Workflow 2: Full Augmentation (Import + Store + Export)
|
||||
|
||||
**Use case:** You want a complete solution that imports project files, stores them with special logic, and exports results (e.g., React project analyzer).
|
||||
|
||||
### Step 1: Create the Augmentation
|
||||
|
||||
```typescript
|
||||
// @yourcompany/brainy-react-analyzer
|
||||
|
||||
import { BaseAugmentation, type AugmentationContext } from '@soulcraft/brainy'
|
||||
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
import type { ProcessedData } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
import { parse } from '@babel/parser'
|
||||
import traverse from '@babel/traverse'
|
||||
import * as t from '@babel/types'
|
||||
|
||||
/**
|
||||
* React Project Analyzer Augmentation
|
||||
*
|
||||
* Features:
|
||||
* - Import: Parse React components, extract props, hooks, imports
|
||||
* - Store: Create relationships between components
|
||||
* - Export: Generate component diagram, dependency graph
|
||||
*/
|
||||
export class ReactAnalyzerAugmentation extends BaseAugmentation {
|
||||
readonly name = 'react-analyzer'
|
||||
readonly timing = 'before' as const
|
||||
readonly operations = ['import', 'export'] as any[]
|
||||
readonly priority = 75
|
||||
|
||||
private handler: ReactComponentHandler
|
||||
|
||||
constructor(config = {}) {
|
||||
super(config)
|
||||
this.handler = new ReactComponentHandler()
|
||||
}
|
||||
|
||||
async execute<T = any>(
|
||||
operation: string,
|
||||
params: any,
|
||||
next: () => Promise<T>
|
||||
): Promise<T> {
|
||||
// IMPORT: Parse React files
|
||||
if (operation === 'import' && this.isReactFile(params)) {
|
||||
return this.handleImport(params, next)
|
||||
}
|
||||
|
||||
// EXPORT: Generate diagrams/reports
|
||||
if (operation === 'export' && params.format === 'react-diagram') {
|
||||
return this.handleExport(params, next)
|
||||
}
|
||||
|
||||
return next()
|
||||
}
|
||||
|
||||
private isReactFile(params: any): boolean {
|
||||
const filename = params.filename || ''
|
||||
return (
|
||||
(filename.endsWith('.tsx') || filename.endsWith('.jsx')) &&
|
||||
params.data?.includes('React')
|
||||
)
|
||||
}
|
||||
|
||||
private async handleImport<T>(params: any, next: () => Promise<T>): Promise<T> {
|
||||
// Parse React component
|
||||
const processed = await this.handler.process(params.data, params.options)
|
||||
|
||||
// Enrich with relationships
|
||||
params.data = processed.data
|
||||
params.metadata = {
|
||||
...params.metadata,
|
||||
reactAnalysis: processed.metadata
|
||||
}
|
||||
|
||||
// Continue to next augmentation/storage
|
||||
const result = await next()
|
||||
|
||||
// Post-process: Create component relationships
|
||||
await this.createComponentRelationships(processed, result)
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
private async createComponentRelationships(
|
||||
processed: ProcessedData,
|
||||
result: any
|
||||
): Promise<void> {
|
||||
const brain = this.getBrain()
|
||||
if (!brain) return
|
||||
|
||||
// Find the component entity that was created
|
||||
const component = processed.data.find(d => d.type === 'ReactComponent')
|
||||
if (!component) return
|
||||
|
||||
// Create relationships for imports
|
||||
for (const imp of component.imports || []) {
|
||||
// Find or create imported component
|
||||
const imported = await brain.findOne({
|
||||
type: 'ReactComponent',
|
||||
name: imp.name
|
||||
})
|
||||
|
||||
if (imported) {
|
||||
// Create "Imports" relationship
|
||||
await brain.createRelation({
|
||||
source: result.entities[0].id,
|
||||
verb: 'Imports',
|
||||
target: imported.id,
|
||||
metadata: {
|
||||
importPath: imp.path,
|
||||
importType: imp.type
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Create relationships for prop types
|
||||
for (const prop of component.props || []) {
|
||||
if (prop.typeRef) {
|
||||
const typeEntity = await brain.findOne({
|
||||
type: 'TypeDefinition',
|
||||
name: prop.typeRef
|
||||
})
|
||||
|
||||
if (typeEntity) {
|
||||
await brain.createRelation({
|
||||
source: result.entities[0].id,
|
||||
verb: 'UsesPropType',
|
||||
target: typeEntity.id
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async handleExport<T>(params: any, next: () => Promise<T>): Promise<T> {
|
||||
const brain = this.getBrain()
|
||||
if (!brain) return next()
|
||||
|
||||
// Query all React components
|
||||
const components = await brain.find({ type: 'ReactComponent' })
|
||||
|
||||
// Build dependency graph
|
||||
const graph = await this.buildDependencyGraph(components)
|
||||
|
||||
// Generate diagram
|
||||
const diagram = this.generateMermaidDiagram(graph)
|
||||
|
||||
return {
|
||||
format: 'react-diagram',
|
||||
diagram,
|
||||
components: components.length,
|
||||
dependencies: graph.edges.length
|
||||
} as T
|
||||
}
|
||||
|
||||
private async buildDependencyGraph(components: any[]): Promise<any> {
|
||||
const brain = this.getBrain()
|
||||
const nodes = components.map(c => ({
|
||||
id: c.id,
|
||||
name: c.name,
|
||||
props: c.props
|
||||
}))
|
||||
|
||||
const edges = []
|
||||
for (const component of components) {
|
||||
const imports = await brain.getRelated(component.id, 'Imports')
|
||||
for (const imp of imports) {
|
||||
edges.push({
|
||||
from: component.id,
|
||||
to: imp.id,
|
||||
type: 'imports'
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return { nodes, edges }
|
||||
}
|
||||
|
||||
private generateMermaidDiagram(graph: any): string {
|
||||
let mermaid = 'graph TD\n'
|
||||
|
||||
for (const node of graph.nodes) {
|
||||
mermaid += ` ${node.id}[${node.name}]\n`
|
||||
}
|
||||
|
||||
for (const edge of graph.edges) {
|
||||
mermaid += ` ${edge.from} --> ${edge.to}\n`
|
||||
}
|
||||
|
||||
return mermaid
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* React Component Handler
|
||||
*/
|
||||
class ReactComponentHandler extends BaseFormatHandler {
|
||||
readonly format = 'react'
|
||||
|
||||
canHandle(data: any): boolean {
|
||||
if (typeof data === 'object' && 'filename' in data) {
|
||||
return data.filename?.match(/\.(tsx|jsx)$/) !== null
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
async process(data: Buffer | string, options: any): Promise<ProcessedData> {
|
||||
const code = Buffer.isBuffer(data) ? data.toString('utf-8') : data
|
||||
|
||||
// Parse with Babel
|
||||
const ast = parse(code, {
|
||||
sourceType: 'module',
|
||||
plugins: ['jsx', 'typescript']
|
||||
})
|
||||
|
||||
// Extract component info
|
||||
const components: any[] = []
|
||||
const imports: any[] = []
|
||||
const exports: any[] = []
|
||||
|
||||
traverse(ast, {
|
||||
// Detect function components
|
||||
FunctionDeclaration(path) {
|
||||
if (this.isReactComponent(path.node)) {
|
||||
components.push({
|
||||
type: 'ReactComponent',
|
||||
name: path.node.id?.name,
|
||||
componentType: 'function',
|
||||
props: this.extractProps(path),
|
||||
hooks: this.extractHooks(path),
|
||||
state: this.extractState(path)
|
||||
})
|
||||
}
|
||||
},
|
||||
|
||||
// Detect class components
|
||||
ClassDeclaration(path) {
|
||||
if (this.isReactClassComponent(path.node)) {
|
||||
components.push({
|
||||
type: 'ReactComponent',
|
||||
name: path.node.id.name,
|
||||
componentType: 'class',
|
||||
props: this.extractClassProps(path),
|
||||
state: this.extractClassState(path),
|
||||
lifecycle: this.extractLifecycleMethods(path)
|
||||
})
|
||||
}
|
||||
},
|
||||
|
||||
// Extract imports
|
||||
ImportDeclaration(path) {
|
||||
imports.push({
|
||||
type: 'Import',
|
||||
from: path.node.source.value,
|
||||
imports: path.node.specifiers.map(s => ({
|
||||
name: s.local.name,
|
||||
imported: t.isImportSpecifier(s) ? s.imported.name : null
|
||||
}))
|
||||
})
|
||||
},
|
||||
|
||||
// Extract exports
|
||||
ExportNamedDeclaration(path) {
|
||||
exports.push({
|
||||
type: 'Export',
|
||||
name: path.node.declaration?.id?.name
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
// Enrich components with import info
|
||||
for (const component of components) {
|
||||
component.imports = imports
|
||||
component.exports = exports.find(e => e.name === component.name)
|
||||
}
|
||||
|
||||
return {
|
||||
format: 'react',
|
||||
data: components,
|
||||
metadata: {
|
||||
rowCount: components.length,
|
||||
fields: ['type', 'name', 'props', 'hooks'],
|
||||
processingTime: Date.now() - startTime,
|
||||
componentCount: components.length,
|
||||
importCount: imports.length,
|
||||
exportCount: exports.length
|
||||
},
|
||||
filename: options.filename
|
||||
}
|
||||
}
|
||||
|
||||
private isReactComponent(node: any): boolean {
|
||||
// Check if function returns JSX
|
||||
return node.body?.body?.some(stmt =>
|
||||
t.isReturnStatement(stmt) && this.isJSX(stmt.argument)
|
||||
)
|
||||
}
|
||||
|
||||
private isJSX(node: any): boolean {
|
||||
return t.isJSXElement(node) || t.isJSXFragment(node)
|
||||
}
|
||||
|
||||
private extractProps(path: any): any[] {
|
||||
const params = path.node.params
|
||||
if (params.length === 0) return []
|
||||
|
||||
const propsParam = params[0]
|
||||
if (t.isObjectPattern(propsParam)) {
|
||||
return propsParam.properties.map(p => ({
|
||||
name: p.key.name,
|
||||
type: p.typeAnnotation?.typeAnnotation?.type
|
||||
}))
|
||||
}
|
||||
|
||||
return []
|
||||
}
|
||||
|
||||
private extractHooks(path: any): string[] {
|
||||
const hooks: string[] = []
|
||||
|
||||
path.traverse({
|
||||
CallExpression(hookPath) {
|
||||
const callee = hookPath.node.callee
|
||||
if (t.isIdentifier(callee) && callee.name.startsWith('use')) {
|
||||
hooks.push(callee.name)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
return hooks
|
||||
}
|
||||
|
||||
private extractState(path: any): any[] {
|
||||
const stateVars: any[] = []
|
||||
|
||||
path.traverse({
|
||||
CallExpression(hookPath) {
|
||||
if (
|
||||
t.isIdentifier(hookPath.node.callee) &&
|
||||
hookPath.node.callee.name === 'useState'
|
||||
) {
|
||||
const parent = hookPath.parent
|
||||
if (t.isVariableDeclarator(parent) && t.isArrayPattern(parent.id)) {
|
||||
const [stateVar] = parent.id.elements
|
||||
if (t.isIdentifier(stateVar)) {
|
||||
stateVars.push({
|
||||
name: stateVar.name,
|
||||
initialValue: hookPath.node.arguments[0]
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
return stateVars
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Package as NPM Module
|
||||
|
||||
```json
|
||||
// package.json
|
||||
{
|
||||
"name": "@yourcompany/brainy-react-analyzer",
|
||||
"version": "1.0.0",
|
||||
"description": "React project analyzer for Brainy",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"import": "./dist/index.js",
|
||||
"require": "./dist/index.cjs"
|
||||
}
|
||||
},
|
||||
"keywords": ["brainy", "react", "analyzer", "augmentation"],
|
||||
"peerDependencies": {
|
||||
"@soulcraft/brainy": "^5.2.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"@babel/parser": "^7.23.0",
|
||||
"@babel/traverse": "^7.23.0",
|
||||
"@babel/types": "^7.23.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Use the Augmentation
|
||||
|
||||
```typescript
|
||||
// Install
|
||||
// npm install @yourcompany/brainy-react-analyzer
|
||||
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { ReactAnalyzerAugmentation } from '@yourcompany/brainy-react-analyzer'
|
||||
|
||||
const brain = new Brainy()
|
||||
|
||||
// Add augmentation
|
||||
brain.addAugmentation(new ReactAnalyzerAugmentation())
|
||||
|
||||
await brain.init()
|
||||
|
||||
// Import React project
|
||||
await brain.import({
|
||||
type: 'directory',
|
||||
path: '/path/to/react-project/src',
|
||||
recursive: true
|
||||
})
|
||||
|
||||
// Query components
|
||||
const components = await brain.find({ type: 'ReactComponent' })
|
||||
console.log(`Found ${components.length} React components`)
|
||||
|
||||
// Find component dependencies
|
||||
const appComponent = await brain.findOne({ type: 'ReactComponent', name: 'App' })
|
||||
const imports = await brain.getRelated(appComponent.id, 'Imports')
|
||||
console.log(`App component imports:`, imports.map(c => c.name))
|
||||
|
||||
// Export diagram
|
||||
const diagram = await brain.export({
|
||||
format: 'react-diagram'
|
||||
})
|
||||
console.log(diagram.diagram) // Mermaid diagram
|
||||
```
|
||||
|
||||
### Step 4: Premium Licensing (Optional)
|
||||
|
||||
```typescript
|
||||
// Add license checking
|
||||
export class ReactAnalyzerAugmentation extends BaseAugmentation {
|
||||
private licenseKey?: string
|
||||
|
||||
constructor(config: { licenseKey?: string } = {}) {
|
||||
super(config)
|
||||
this.licenseKey = config.licenseKey
|
||||
}
|
||||
|
||||
async onInitialize(): Promise<void> {
|
||||
if (!this.licenseKey) {
|
||||
throw new Error('React Analyzer requires a license key. Get one at https://yourcompany.com/brainy-react')
|
||||
}
|
||||
|
||||
// Verify license
|
||||
const valid = await this.verifyLicense(this.licenseKey)
|
||||
if (!valid) {
|
||||
throw new Error('Invalid license key')
|
||||
}
|
||||
|
||||
this.log('React Analyzer initialized successfully')
|
||||
}
|
||||
|
||||
private async verifyLicense(key: string): Promise<boolean> {
|
||||
// Check with your license server
|
||||
const response = await fetch('https://api.yourcompany.com/verify', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ key, product: 'brainy-react-analyzer' })
|
||||
})
|
||||
|
||||
const data = await response.json()
|
||||
return data.valid
|
||||
}
|
||||
}
|
||||
|
||||
// Usage with license
|
||||
brain.addAugmentation(new ReactAnalyzerAugmentation({
|
||||
licenseKey: 'YOUR-LICENSE-KEY'
|
||||
}))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Comparison: Handler vs Augmentation
|
||||
|
||||
| Feature | Simple Handler | Full Augmentation |
|
||||
|---------|---------------|-------------------|
|
||||
| **Import** | ✅ Yes (automatic) | ✅ Yes (with custom logic) |
|
||||
| **Storage** | ✅ Automatic (Brainy core) | ✅ Custom logic + relationships |
|
||||
| **Export** | ❌ No | ✅ Yes (custom formats) |
|
||||
| **Relationships** | ❌ No | ✅ Yes (create custom relationships) |
|
||||
| **Premium licensing** | ❌ Difficult | ✅ Easy (augmentation-level) |
|
||||
| **Custom operations** | ❌ Import only | ✅ Any operation |
|
||||
| **Complexity** | Low (50-100 lines) | Medium (200-500 lines) |
|
||||
|
||||
### When to use Handler:
|
||||
- Just need to import a new file type
|
||||
- Don't need custom export
|
||||
- Don't need special relationships
|
||||
- Simple use case
|
||||
|
||||
### When to use Augmentation:
|
||||
- Need import + export workflow
|
||||
- Need custom relationship logic
|
||||
- Want premium licensing capability
|
||||
- Complex business logic
|
||||
- Multiple operations (import + export + query)
|
||||
|
||||
---
|
||||
|
||||
## More Examples
|
||||
|
||||
### Example: Python Project Analyzer
|
||||
|
||||
```typescript
|
||||
class PythonAnalyzerAugmentation extends BaseAugmentation {
|
||||
// Import Python files, extract classes/functions
|
||||
// Create relationships between modules
|
||||
// Export: Dependency diagram, call graph
|
||||
}
|
||||
```
|
||||
|
||||
### Example: Database Schema Sync
|
||||
|
||||
```typescript
|
||||
class DatabaseSyncAugmentation extends BaseAugmentation {
|
||||
// Import: Parse SQL schema
|
||||
// Store: Tables, columns, relationships
|
||||
// Export: Generate migration scripts
|
||||
}
|
||||
```
|
||||
|
||||
### Example: API Documentation Generator
|
||||
|
||||
```typescript
|
||||
class APIDocAugmentation extends BaseAugmentation {
|
||||
// Import: Parse TypeScript types
|
||||
// Store: Endpoints, parameters, responses
|
||||
// Export: OpenAPI spec, Markdown docs
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [FORMAT_HANDLERS.md](./FORMAT_HANDLERS.md) - Creating format handlers
|
||||
- [AUGMENTATIONS.md](./AUGMENTATIONS.md) - Augmentation system details
|
||||
- [ImageHandler source](../../src/augmentations/intelligentImport/handlers/imageHandler.ts) - Handler example
|
||||
- [IntelligentImportAugmentation source](../../src/augmentations/intelligentImport/IntelligentImportAugmentation.ts) - Augmentation example
|
||||
687
docs/augmentations/FORMAT_HANDLERS.md
Normal file
687
docs/augmentations/FORMAT_HANDLERS.md
Normal file
|
|
@ -0,0 +1,687 @@
|
|||
# Creating Custom Format Handlers
|
||||
|
||||
**Version:** 5.2.0+
|
||||
|
||||
Format handlers enable you to import ANY file type into Brainy as structured knowledge graph data. This guide shows how to create custom format handlers for your specific file formats.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```typescript
|
||||
import { BaseFormatHandler, globalHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
import type { FormatHandlerOptions, ProcessedData } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
|
||||
class MyFormatHandler extends BaseFormatHandler {
|
||||
readonly format = 'myformat'
|
||||
|
||||
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
|
||||
// Option 1: Check by MIME type
|
||||
if (typeof data === 'object' && 'filename' in data) {
|
||||
const mimeType = this.getMimeType(data)
|
||||
return this.mimeTypeMatches(mimeType, ['application/x-myformat'])
|
||||
}
|
||||
|
||||
// Option 2: Check by magic bytes
|
||||
if (Buffer.isBuffer(data)) {
|
||||
return data[0] === 0x4D && data[1] === 0x59 // "MY" magic bytes
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
async process(
|
||||
data: Buffer | string,
|
||||
options: FormatHandlerOptions
|
||||
): Promise<ProcessedData> {
|
||||
// Convert to Buffer
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
|
||||
|
||||
// Parse your format
|
||||
const parsed = this.parseMyFormat(buffer)
|
||||
|
||||
// Return structured data
|
||||
return {
|
||||
format: 'myformat',
|
||||
data: [
|
||||
{
|
||||
type: 'MyEntity',
|
||||
name: parsed.name,
|
||||
metadata: parsed.metadata
|
||||
}
|
||||
],
|
||||
metadata: {
|
||||
rowCount: 1,
|
||||
fields: ['type', 'name', 'metadata'],
|
||||
processingTime: Date.now() - startTime
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private parseMyFormat(buffer: Buffer): any {
|
||||
// Your parsing logic here
|
||||
return { name: 'example', metadata: {} }
|
||||
}
|
||||
}
|
||||
|
||||
// Register globally
|
||||
globalHandlerRegistry.registerHandler({
|
||||
name: 'myformat',
|
||||
mimeTypes: ['application/x-myformat'],
|
||||
extensions: ['.myf', '.myfmt'],
|
||||
loader: async () => new MyFormatHandler()
|
||||
})
|
||||
```
|
||||
|
||||
Now Brainy automatically handles your format:
|
||||
|
||||
```typescript
|
||||
await brain.import({
|
||||
type: 'file',
|
||||
data: myFormatBuffer,
|
||||
filename: 'data.myf'
|
||||
})
|
||||
// Automatically routes to MyFormatHandler!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## BaseFormatHandler
|
||||
|
||||
All format handlers should extend `BaseFormatHandler`, which provides:
|
||||
|
||||
### MIME Type Detection
|
||||
|
||||
```typescript
|
||||
protected getMimeType(data: Buffer | string | { filename?: string }): string
|
||||
```
|
||||
|
||||
Detects MIME type from filename or buffer. Uses Brainy's comprehensive MIME detection (2000+ types).
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
const mimeType = this.getMimeType({ filename: 'data.dwg' })
|
||||
// Returns: 'image/vnd.dwg'
|
||||
```
|
||||
|
||||
### MIME Type Matching
|
||||
|
||||
```typescript
|
||||
protected mimeTypeMatches(mimeType: string, patterns: string[]): boolean
|
||||
```
|
||||
|
||||
Checks if MIME type matches patterns. Supports wildcards (`image/*`).
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
if (this.mimeTypeMatches(mimeType, ['image/*', 'video/*'])) {
|
||||
// Handle all images and videos
|
||||
}
|
||||
```
|
||||
|
||||
### Extension Detection
|
||||
|
||||
```typescript
|
||||
protected detectExtension(data: string | Buffer | { filename?: string; ext?: string }): string | null
|
||||
```
|
||||
|
||||
Extracts file extension for fallback detection.
|
||||
|
||||
---
|
||||
|
||||
## canHandle() Method
|
||||
|
||||
The `canHandle()` method determines if your handler can process the given data.
|
||||
|
||||
### Strategy 1: MIME Type Detection (Recommended)
|
||||
|
||||
```typescript
|
||||
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
|
||||
if (typeof data === 'object' && 'filename' in data) {
|
||||
const mimeType = this.getMimeType(data)
|
||||
return this.mimeTypeMatches(mimeType, [
|
||||
'application/x-myformat',
|
||||
'application/myformat'
|
||||
])
|
||||
}
|
||||
return false
|
||||
}
|
||||
```
|
||||
|
||||
✅ **Pros:** Automatic, comprehensive, works with 2000+ types
|
||||
❌ **Cons:** Requires filename
|
||||
|
||||
### Strategy 2: Magic Byte Detection
|
||||
|
||||
```typescript
|
||||
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
|
||||
if (Buffer.isBuffer(data)) {
|
||||
// Check magic bytes
|
||||
return (
|
||||
data[0] === 0x50 && // 'P'
|
||||
data[1] === 0x4B && // 'K'
|
||||
data[2] === 0x03 &&
|
||||
data[3] === 0x04 // ZIP signature
|
||||
)
|
||||
}
|
||||
return false
|
||||
}
|
||||
```
|
||||
|
||||
✅ **Pros:** Works without filename, robust
|
||||
❌ **Cons:** Requires knowledge of format structure
|
||||
|
||||
### Strategy 3: Combined Approach
|
||||
|
||||
```typescript
|
||||
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
|
||||
// Try MIME type first
|
||||
if (typeof data === 'object' && 'filename' in data) {
|
||||
const mimeType = this.getMimeType(data)
|
||||
if (this.mimeTypeMatches(mimeType, ['application/x-myformat'])) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback to magic bytes
|
||||
if (Buffer.isBuffer(data)) {
|
||||
return this.checkMagicBytes(data)
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
```
|
||||
|
||||
✅ **Pros:** Robust, works in all scenarios
|
||||
❌ **Cons:** More complex
|
||||
|
||||
---
|
||||
|
||||
## process() Method
|
||||
|
||||
The `process()` method extracts structured data from the file.
|
||||
|
||||
### Return Format
|
||||
|
||||
```typescript
|
||||
interface ProcessedData {
|
||||
/** Format identifier */
|
||||
format: string
|
||||
|
||||
/** Array of extracted entities */
|
||||
data: Array<Record<string, any>>
|
||||
|
||||
/** Metadata about processing */
|
||||
metadata: {
|
||||
rowCount: number
|
||||
fields: string[]
|
||||
processingTime: number
|
||||
[key: string]: any
|
||||
}
|
||||
|
||||
/** Original filename (optional) */
|
||||
filename?: string
|
||||
}
|
||||
```
|
||||
|
||||
### Example: CAD File Handler
|
||||
|
||||
```typescript
|
||||
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
|
||||
const startTime = Date.now()
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
|
||||
|
||||
// Parse CAD file
|
||||
const cad = await this.parseCAD(buffer)
|
||||
|
||||
// Extract entities
|
||||
const entities = []
|
||||
|
||||
// Main CAD document
|
||||
entities.push({
|
||||
type: 'CADDocument',
|
||||
filename: options.filename,
|
||||
units: cad.units,
|
||||
bounds: cad.bounds
|
||||
})
|
||||
|
||||
// Layers
|
||||
for (const layer of cad.layers) {
|
||||
entities.push({
|
||||
type: 'CADLayer',
|
||||
name: layer.name,
|
||||
color: layer.color,
|
||||
visible: layer.visible,
|
||||
objectCount: layer.objects.length
|
||||
})
|
||||
}
|
||||
|
||||
// Objects
|
||||
for (const obj of cad.objects) {
|
||||
entities.push({
|
||||
type: 'CADObject',
|
||||
objectType: obj.type,
|
||||
layer: obj.layer,
|
||||
geometry: obj.geometry,
|
||||
properties: obj.properties
|
||||
})
|
||||
}
|
||||
|
||||
return {
|
||||
format: 'cad',
|
||||
data: entities,
|
||||
metadata: {
|
||||
rowCount: entities.length,
|
||||
fields: ['type', 'name', 'geometry', 'properties'],
|
||||
processingTime: Date.now() - startTime,
|
||||
layerCount: cad.layers.length,
|
||||
objectCount: cad.objects.length,
|
||||
units: cad.units
|
||||
},
|
||||
filename: options.filename
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Always track processing time:**
|
||||
```typescript
|
||||
const startTime = Date.now()
|
||||
// ... processing ...
|
||||
metadata.processingTime = Date.now() - startTime
|
||||
```
|
||||
|
||||
2. **Include rich metadata:**
|
||||
```typescript
|
||||
metadata: {
|
||||
rowCount: entities.length,
|
||||
fields: ['type', 'name', ...],
|
||||
processingTime: 123,
|
||||
// Format-specific metadata
|
||||
layerCount: 5,
|
||||
objectCount: 150,
|
||||
version: '2.0'
|
||||
}
|
||||
```
|
||||
|
||||
3. **Handle errors gracefully:**
|
||||
```typescript
|
||||
try {
|
||||
const parsed = this.parse(buffer)
|
||||
return { format: 'myformat', data: parsed, ... }
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Failed to parse myformat: ${error instanceof Error ? error.message : String(error)}`
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
4. **Support progress reporting (optional):**
|
||||
```typescript
|
||||
if (options.progressHooks?.onCurrentItem) {
|
||||
options.progressHooks.onCurrentItem(`Processing layer ${i}/${total}`)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FormatHandlerRegistry
|
||||
|
||||
### Global Registry
|
||||
|
||||
Use the global registry for application-wide handlers:
|
||||
|
||||
```typescript
|
||||
import { globalHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
|
||||
globalHandlerRegistry.registerHandler({
|
||||
name: 'myformat',
|
||||
mimeTypes: ['application/x-myformat'],
|
||||
extensions: ['.myf'],
|
||||
loader: async () => new MyFormatHandler()
|
||||
})
|
||||
```
|
||||
|
||||
### Local Registry
|
||||
|
||||
Create a local registry for scoped handlers:
|
||||
|
||||
```typescript
|
||||
import { FormatHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
|
||||
const registry = new FormatHandlerRegistry()
|
||||
registry.registerHandler({ ... })
|
||||
```
|
||||
|
||||
### Lazy Loading
|
||||
|
||||
Handlers are lazy-loaded for performance:
|
||||
|
||||
```typescript
|
||||
globalHandlerRegistry.registerHandler({
|
||||
name: 'heavy',
|
||||
mimeTypes: ['application/x-heavy'],
|
||||
extensions: ['.heavy'],
|
||||
loader: async () => {
|
||||
// Only loaded when first needed
|
||||
const { HeavyHandler } = await import('./HeavyHandler.js')
|
||||
return new HeavyHandler()
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Getting Handlers
|
||||
|
||||
```typescript
|
||||
// By filename (automatic MIME detection)
|
||||
const handler = await registry.getHandler('data.myf')
|
||||
|
||||
// By MIME type
|
||||
const handler = await registry.getHandlerByMimeType('application/x-myformat')
|
||||
|
||||
// By extension
|
||||
const handler = await registry.getHandlerByExtension('.myf')
|
||||
|
||||
// By name
|
||||
const handler = await registry.getHandlerByName('myformat')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
### Example 1: Video Metadata Extractor
|
||||
|
||||
```typescript
|
||||
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
import ffmpeg from 'fluent-ffmpeg'
|
||||
|
||||
class VideoHandler extends BaseFormatHandler {
|
||||
readonly format = 'video'
|
||||
|
||||
canHandle(data) {
|
||||
const mimeType = this.getMimeType(data)
|
||||
return this.mimeTypeMatches(mimeType, ['video/*'])
|
||||
}
|
||||
|
||||
async process(data, options) {
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
|
||||
|
||||
// Extract video metadata with ffmpeg
|
||||
const metadata = await this.extractVideoMetadata(buffer)
|
||||
|
||||
return {
|
||||
format: 'video',
|
||||
data: [{
|
||||
type: 'Video',
|
||||
duration: metadata.duration,
|
||||
codec: metadata.codec,
|
||||
resolution: metadata.resolution,
|
||||
frameRate: metadata.frameRate,
|
||||
bitrate: metadata.bitrate,
|
||||
audioTracks: metadata.audioTracks,
|
||||
subtitles: metadata.subtitles
|
||||
}],
|
||||
metadata: {
|
||||
rowCount: 1,
|
||||
fields: ['type', 'duration', 'codec', 'resolution'],
|
||||
processingTime: metadata.processingTime
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async extractVideoMetadata(buffer: Buffer) {
|
||||
// Use ffmpeg to extract metadata
|
||||
return new Promise((resolve, reject) => {
|
||||
ffmpeg(buffer)
|
||||
.ffprobe((err, data) => {
|
||||
if (err) reject(err)
|
||||
else resolve(this.parseFFmpegOutput(data))
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example 2: Git Repository Parser
|
||||
|
||||
```typescript
|
||||
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
import { simpleGit } from 'simple-git'
|
||||
|
||||
class GitRepoHandler extends BaseFormatHandler {
|
||||
readonly format = 'git-repo'
|
||||
|
||||
canHandle(data) {
|
||||
// Check for .git directory
|
||||
if (typeof data === 'object' && 'filename' in data) {
|
||||
return data.filename?.includes('.git') || false
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
async process(data, options) {
|
||||
const repoPath = options.filename || ''
|
||||
const git = simpleGit(repoPath)
|
||||
|
||||
// Extract commits
|
||||
const log = await git.log()
|
||||
const commits = log.all.map(commit => ({
|
||||
type: 'GitCommit',
|
||||
hash: commit.hash,
|
||||
message: commit.message,
|
||||
author: commit.author_name,
|
||||
date: commit.date
|
||||
}))
|
||||
|
||||
// Extract branches
|
||||
const branchSummary = await git.branchLocal()
|
||||
const branches = Object.keys(branchSummary.branches).map(name => ({
|
||||
type: 'GitBranch',
|
||||
name,
|
||||
current: branchSummary.current === name
|
||||
}))
|
||||
|
||||
return {
|
||||
format: 'git-repo',
|
||||
data: [...commits, ...branches],
|
||||
metadata: {
|
||||
rowCount: commits.length + branches.length,
|
||||
fields: ['type', 'hash', 'message', 'author'],
|
||||
processingTime: Date.now() - startTime,
|
||||
commitCount: commits.length,
|
||||
branchCount: branches.length
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example 3: Database Schema Importer
|
||||
|
||||
```typescript
|
||||
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
|
||||
import { Client } from 'pg'
|
||||
|
||||
class PostgreSQLSchemaHandler extends BaseFormatHandler {
|
||||
readonly format = 'postgres-schema'
|
||||
|
||||
canHandle(data) {
|
||||
if (typeof data === 'object' && 'filename' in data) {
|
||||
return data.filename?.endsWith('.sql') || false
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
async process(data, options) {
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
|
||||
const sql = buffer.toString('utf-8')
|
||||
|
||||
// Parse SQL or connect to database
|
||||
const schema = await this.parseSchema(sql)
|
||||
|
||||
const entities = []
|
||||
|
||||
// Tables
|
||||
for (const table of schema.tables) {
|
||||
entities.push({
|
||||
type: 'Table',
|
||||
name: table.name,
|
||||
schema: table.schema,
|
||||
columnCount: table.columns.length
|
||||
})
|
||||
|
||||
// Columns
|
||||
for (const column of table.columns) {
|
||||
entities.push({
|
||||
type: 'Column',
|
||||
name: column.name,
|
||||
table: table.name,
|
||||
dataType: column.dataType,
|
||||
nullable: column.nullable,
|
||||
primaryKey: column.primaryKey
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Foreign keys
|
||||
for (const fk of schema.foreignKeys) {
|
||||
entities.push({
|
||||
type: 'ForeignKey',
|
||||
from: `${fk.fromTable}.${fk.fromColumn}`,
|
||||
to: `${fk.toTable}.${fk.toColumn}`
|
||||
})
|
||||
}
|
||||
|
||||
return {
|
||||
format: 'postgres-schema',
|
||||
data: entities,
|
||||
metadata: {
|
||||
rowCount: entities.length,
|
||||
fields: ['type', 'name', 'table', 'dataType'],
|
||||
processingTime: Date.now() - startTime,
|
||||
tableCount: schema.tables.length,
|
||||
columnCount: schema.tables.reduce((sum, t) => sum + t.columns.length, 0),
|
||||
foreignKeyCount: schema.foreignKeys.length
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Creating Premium Augmentations
|
||||
|
||||
Package your handler as a premium augmentation:
|
||||
|
||||
```typescript
|
||||
// @yourcompany/brainy-cad-importer
|
||||
|
||||
import { BaseAugmentation } from '@soulcraft/brainy'
|
||||
import { CADHandler } from './CADHandler.js'
|
||||
|
||||
export class CADImportAugmentation extends BaseAugmentation {
|
||||
readonly name = 'cad-import'
|
||||
readonly timing = 'before'
|
||||
readonly operations = ['import', 'importFile']
|
||||
|
||||
private handler: CADHandler
|
||||
|
||||
constructor(config = {}) {
|
||||
super(config)
|
||||
this.handler = new CADHandler()
|
||||
}
|
||||
|
||||
async execute(operation, params, next) {
|
||||
// Check if this is a CAD file
|
||||
if (this.isCADFile(params)) {
|
||||
const processed = await this.handler.process(params.data, params.options)
|
||||
params.data = processed.data
|
||||
params.metadata = { ...params.metadata, ...processed.metadata }
|
||||
}
|
||||
|
||||
return next()
|
||||
}
|
||||
|
||||
private isCADFile(params: any): boolean {
|
||||
return this.handler.canHandle(params.data || params)
|
||||
}
|
||||
}
|
||||
|
||||
// Usage:
|
||||
// npm install @yourcompany/brainy-cad-importer
|
||||
// brain.addAugmentation(new CADImportAugmentation())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
```typescript
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { MyFormatHandler } from './MyFormatHandler.js'
|
||||
|
||||
describe('MyFormatHandler', () => {
|
||||
let handler: MyFormatHandler
|
||||
|
||||
beforeEach(() => {
|
||||
handler = new MyFormatHandler()
|
||||
})
|
||||
|
||||
describe('canHandle', () => {
|
||||
it('should handle .myf files', () => {
|
||||
expect(handler.canHandle({ filename: 'data.myf' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should handle by MIME type', () => {
|
||||
expect(handler.canHandle({ filename: 'data.myformat' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should reject non-myformat files', () => {
|
||||
expect(handler.canHandle({ filename: 'data.txt' })).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process', () => {
|
||||
it('should extract structured data', async () => {
|
||||
const testData = Buffer.from('MY format data')
|
||||
|
||||
const result = await handler.process(testData)
|
||||
|
||||
expect(result.format).toBe('myformat')
|
||||
expect(result.data).toHaveLength(1)
|
||||
expect(result.metadata.processingTime).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('should handle errors gracefully', async () => {
|
||||
const invalidData = Buffer.from('invalid')
|
||||
|
||||
await expect(handler.process(invalidData)).rejects.toThrow()
|
||||
})
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always extend BaseFormatHandler** - provides MIME detection and utilities
|
||||
2. **Use MIME types for routing** - automatic, comprehensive, maintainable
|
||||
3. **Lazy load heavy dependencies** - better performance
|
||||
4. **Extract rich metadata** - make data queryable in knowledge graph
|
||||
5. **Handle errors gracefully** - fail fast with clear error messages
|
||||
6. **Test thoroughly** - test canHandle() and process() with real data
|
||||
7. **Document your format** - explain what data is extracted and how
|
||||
8. **Follow ProcessedData format** - ensures compatibility with Brainy
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [ImageHandler source](../../src/augmentations/intelligentImport/handlers/imageHandler.ts) - Reference implementation
|
||||
- [BaseFormatHandler source](../../src/augmentations/intelligentImport/handlers/base.ts) - Base class
|
||||
- [FormatHandlerRegistry source](../../src/augmentations/intelligentImport/FormatHandlerRegistry.ts) - Registry implementation
|
||||
- [Augmentations Guide](./AUGMENTATIONS.md) - Creating augmentations
|
||||
204
docs/augmentations/README.md
Normal file
204
docs/augmentations/README.md
Normal file
|
|
@ -0,0 +1,204 @@
|
|||
# Brainy Augmentations
|
||||
|
||||
Augmentations are the core extensibility mechanism in Brainy. They allow you to modify, enhance, and extend Brainy's behavior without changing the core code.
|
||||
|
||||
## Core Principle: One Interface, Infinite Possibilities
|
||||
|
||||
Every augmentation implements the same simple `BrainyAugmentation` interface:
|
||||
|
||||
```typescript
|
||||
interface BrainyAugmentation {
|
||||
name: string
|
||||
timing: 'before' | 'after' | 'around' | 'replace'
|
||||
operations: string[]
|
||||
priority: number
|
||||
initialize(context): Promise<void>
|
||||
execute(operation, params, next): Promise<any>
|
||||
}
|
||||
```
|
||||
|
||||
This single interface can handle EVERYTHING - from adding AI capabilities to exposing APIs to replacing storage backends.
|
||||
|
||||
## Available Augmentations
|
||||
|
||||
### 🧠 Data Processing
|
||||
Augmentations that enhance how data is processed and stored.
|
||||
|
||||
| Augmentation | Description | Timing | Status |
|
||||
|-------------|-------------|--------|--------|
|
||||
| **NeuralImportAugmentation** | AI-powered entity and relationship extraction | `before` | ✅ Production |
|
||||
| **EntityRegistryAugmentation** | High-performance entity deduplication | `before` | ✅ Production |
|
||||
| **BatchProcessingAugmentation** | Optimizes bulk operations | `around` | ✅ Production |
|
||||
| **IntelligentVerbScoringAugmentation** | Learns relationship importance over time | `after` | ✅ Production |
|
||||
|
||||
### 🔌 External Connections (Synapses)
|
||||
Connect Brainy to external services and data sources.
|
||||
|
||||
| Augmentation | Description | Timing | Status |
|
||||
|-------------|-------------|--------|--------|
|
||||
| **NotionSynapse** | Sync with Notion databases | `after` | 📝 Example |
|
||||
| **SalesforceSynapse** | Connect to Salesforce CRM | `after` | 📝 Example |
|
||||
| **SlackSynapse** | Import Slack conversations | `after` | 📝 Example |
|
||||
| **GoogleDriveSynapse** | Sync Google Drive documents | `after` | 📝 Example |
|
||||
|
||||
### 🌐 API Exposure
|
||||
Expose Brainy through various protocols.
|
||||
|
||||
| Augmentation | Description | Timing | Status |
|
||||
|-------------|-------------|--------|--------|
|
||||
| **APIServerAugmentation** | REST, WebSocket, and MCP server | `after` | ✅ Production |
|
||||
| **GraphQLAugmentation** | GraphQL API endpoint | `after` | 🚧 Planned |
|
||||
| **gRPCAugmentation** | gRPC service | `after` | 🚧 Planned |
|
||||
|
||||
### 💾 Storage Backends
|
||||
Replace or enhance the storage layer.
|
||||
|
||||
| Augmentation | Description | Timing | Status |
|
||||
|-------------|-------------|--------|--------|
|
||||
| **S3StorageAugmentation** | Use S3 as storage backend | `replace` | 📝 Example |
|
||||
| **RedisAugmentation** | Redis caching layer | `around` | 📝 Example |
|
||||
| **PostgresAugmentation** | PostgreSQL persistence | `replace` | 📝 Example |
|
||||
|
||||
### 🔄 Real-time & Sync
|
||||
Handle real-time updates and synchronization.
|
||||
|
||||
| Augmentation | Description | Timing | Status |
|
||||
|-------------|-------------|--------|--------|
|
||||
| **WebSocketConduitAugmentation** | WebSocket client connections | `after` | ⚠️ Legacy |
|
||||
| **ServerSearchAugmentation** | Connect to remote Brainy servers | `after` | ⚠️ Legacy |
|
||||
| **TeamCoordinationAugmentation** | Multi-agent synchronization | `after` | 📝 Example |
|
||||
|
||||
### 🛡️ Infrastructure
|
||||
Core infrastructure and reliability features.
|
||||
|
||||
| Augmentation | Description | Timing | Status |
|
||||
|-------------|-------------|--------|--------|
|
||||
| **ConnectionPoolAugmentation** | Optimize cloud storage connections | `before` | ✅ Production |
|
||||
| **RequestDeduplicatorAugmentation** | Prevent duplicate concurrent requests | `before` | ✅ Production |
|
||||
| **TransactionAugmentation** | ACID transaction support | `around` | 🚧 Planned |
|
||||
| **CacheAugmentation** | Multi-level caching | `around` | ✅ Production |
|
||||
|
||||
### 📊 Monitoring & Analytics
|
||||
Track and analyze Brainy's behavior.
|
||||
|
||||
| Augmentation | Description | Timing | Status |
|
||||
|-------------|-------------|--------|--------|
|
||||
| **MetricsAugmentation** | Prometheus metrics | `after` | 📝 Example |
|
||||
| **LoggingAugmentation** | Structured logging | `after` | 📝 Example |
|
||||
| **TracingAugmentation** | Distributed tracing | `around` | 🚧 Planned |
|
||||
|
||||
### 🤖 AI & Chat
|
||||
AI-powered interfaces and chat capabilities.
|
||||
|
||||
| Augmentation | Description | Timing | Status |
|
||||
|-------------|-------------|--------|--------|
|
||||
| **ChatInterfaceAugmentation** | Natural language interface | `before` | 📝 Example |
|
||||
| **MCPAgentMemoryAugmentation** | AI agent memory via MCP | `after` | 📝 Example |
|
||||
| **LLMQueryAugmentation** | LLM-enhanced queries | `before` | 📝 Example |
|
||||
|
||||
### 📈 Visualization
|
||||
Visual representations of data.
|
||||
|
||||
| Augmentation | Description | Timing | Status |
|
||||
|-------------|-------------|--------|--------|
|
||||
| **GraphVisualizationAugmentation** | Real-time graph visualization | `after` | 📝 Example |
|
||||
| **DashboardAugmentation** | Web-based dashboard | `after` | 🚧 Planned |
|
||||
|
||||
## Status Legend
|
||||
|
||||
- ✅ **Production**: Fully implemented and tested
|
||||
- 📝 **Example**: Example implementation available
|
||||
- 🚧 **Planned**: On the roadmap
|
||||
- ⚠️ **Legacy**: Being replaced by newer augmentations
|
||||
|
||||
## Using Augmentations
|
||||
|
||||
### Zero-Config Approach
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
|
||||
// Just register augmentations - they work automatically!
|
||||
brain.augmentations.register(new EntityRegistryAugmentation())
|
||||
brain.augmentations.register(new APIServerAugmentation())
|
||||
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
### With Configuration
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
|
||||
brain.augmentations.register(
|
||||
new APIServerAugmentation({
|
||||
port: 8080,
|
||||
auth: { required: true }
|
||||
})
|
||||
)
|
||||
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
## Creating Custom Augmentations
|
||||
|
||||
See [Creating Custom Augmentations](./creating-augmentations.md) for a complete guide.
|
||||
|
||||
Quick example:
|
||||
|
||||
```typescript
|
||||
class MyAugmentation extends BaseAugmentation {
|
||||
readonly name = 'my-augmentation'
|
||||
readonly timing = 'after'
|
||||
readonly operations = ['add', 'search']
|
||||
readonly priority = 50
|
||||
|
||||
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
|
||||
console.log(`Before ${operation}`)
|
||||
const result = await next()
|
||||
console.log(`After ${operation}`)
|
||||
return result
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Augmentation Timing
|
||||
|
||||
### `before`
|
||||
Executes before the main operation. Used for:
|
||||
- Input validation
|
||||
- Data transformation
|
||||
- Authentication checks
|
||||
|
||||
### `after`
|
||||
Executes after the main operation. Used for:
|
||||
- Broadcasting updates
|
||||
- Syncing to external services
|
||||
- Logging and metrics
|
||||
|
||||
### `around`
|
||||
Wraps the main operation. Used for:
|
||||
- Transactions
|
||||
- Caching
|
||||
- Error handling
|
||||
|
||||
### `replace`
|
||||
Completely replaces the main operation. Used for:
|
||||
- Alternative storage backends
|
||||
- Mock implementations
|
||||
- Proxy operations
|
||||
|
||||
## Priority System
|
||||
|
||||
Higher numbers execute first:
|
||||
- **100**: Critical system operations
|
||||
- **50**: Performance optimizations
|
||||
- **10**: Enhancement features
|
||||
- **1**: Optional features
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [API Server Augmentation](./api-server.md) - Complete API server documentation
|
||||
- [Creating Augmentations](./creating-augmentations.md) - How to build your own
|
||||
- [Augmentation Examples](../AUGMENTATION-EXAMPLES.md) - Real-world examples
|
||||
- [Architecture Overview](../COMPLETE-ARCHITECTURE-VISION.md) - System architecture
|
||||
429
docs/augmentations/api-server.md
Normal file
429
docs/augmentations/api-server.md
Normal file
|
|
@ -0,0 +1,429 @@
|
|||
# API Server Augmentation
|
||||
|
||||
## Overview
|
||||
|
||||
The `APIServerAugmentation` is a powerful augmentation that exposes your Brainy instance through REST, WebSocket, and MCP (Model Context Protocol) APIs. It transforms Brainy into a full-featured API server with zero configuration required.
|
||||
|
||||
## Features
|
||||
|
||||
### 🌐 REST API
|
||||
Complete CRUD operations and advanced queries through HTTP endpoints.
|
||||
|
||||
### 🔌 WebSocket Server
|
||||
Real-time bidirectional communication with automatic operation broadcasting.
|
||||
|
||||
### 🧠 MCP Integration
|
||||
Built-in Model Context Protocol support for AI agent communication.
|
||||
|
||||
### 📊 Operation Broadcasting
|
||||
Automatically broadcasts all Brainy operations to subscribed WebSocket clients.
|
||||
|
||||
### 🔒 Optional Security
|
||||
Built-in authentication and rate limiting when needed.
|
||||
|
||||
## Installation
|
||||
|
||||
The APIServerAugmentation is included in Brainy core. No additional installation required.
|
||||
|
||||
**Bun** (recommended): No additional dependencies needed - use `Bun.serve()` directly.
|
||||
|
||||
**Node.js**: Install optional Express dependencies:
|
||||
```bash
|
||||
npm install express cors ws
|
||||
```
|
||||
|
||||
## Zero-Config Usage
|
||||
|
||||
```typescript
|
||||
import { Brainy } from 'brainy'
|
||||
import { APIServerAugmentation } from 'brainy/augmentations'
|
||||
|
||||
const brain = new Brainy()
|
||||
|
||||
// Register the API server augmentation
|
||||
brain.augmentations.register(new APIServerAugmentation())
|
||||
|
||||
await brain.init()
|
||||
|
||||
// Server is now running at http://localhost:3000
|
||||
console.log('API Server ready!')
|
||||
console.log('REST: http://localhost:3000/api/*')
|
||||
console.log('WebSocket: ws://localhost:3000/ws')
|
||||
console.log('MCP: http://localhost:3000/api/mcp')
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
While zero-config works great, you can customize the server:
|
||||
|
||||
```typescript
|
||||
const apiServer = new APIServerAugmentation({
|
||||
enabled: true, // Enable/disable the server
|
||||
port: 3000, // HTTP port
|
||||
host: '0.0.0.0', // Bind address
|
||||
|
||||
cors: {
|
||||
origin: '*', // CORS allowed origins
|
||||
credentials: true // Allow credentials
|
||||
},
|
||||
|
||||
auth: {
|
||||
required: false, // Require authentication
|
||||
apiKeys: [], // Valid API keys
|
||||
bearerTokens: [] // Valid bearer tokens
|
||||
},
|
||||
|
||||
rateLimit: {
|
||||
windowMs: 60000, // Rate limit window (ms)
|
||||
max: 100 // Max requests per window
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## REST API Endpoints
|
||||
|
||||
### Health Check
|
||||
```http
|
||||
GET /health
|
||||
```
|
||||
Returns server status and basic metrics.
|
||||
|
||||
### Search
|
||||
```http
|
||||
POST /api/search
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"query": "search text",
|
||||
"limit": 10,
|
||||
"options": {}
|
||||
}
|
||||
```
|
||||
|
||||
### Add Data
|
||||
```http
|
||||
POST /api/add
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"content": "data to add",
|
||||
"metadata": {
|
||||
"key": "value"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Get by ID
|
||||
```http
|
||||
GET /api/get/:id
|
||||
```
|
||||
|
||||
### Delete
|
||||
```http
|
||||
DELETE /api/delete/:id
|
||||
```
|
||||
|
||||
### Create Relationship
|
||||
```http
|
||||
POST /api/relate
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"source": "id1",
|
||||
"target": "id2",
|
||||
"verb": "relates_to",
|
||||
"metadata": {}
|
||||
}
|
||||
```
|
||||
|
||||
### Complex Queries
|
||||
```http
|
||||
POST /api/find
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"where": { "type": "document" },
|
||||
"like": "machine learning",
|
||||
"limit": 10
|
||||
}
|
||||
```
|
||||
|
||||
### Clustering
|
||||
```http
|
||||
POST /api/cluster
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"algorithm": "kmeans",
|
||||
"options": {
|
||||
"k": 5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Statistics
|
||||
```http
|
||||
GET /api/stats
|
||||
```
|
||||
|
||||
### Operation History
|
||||
```http
|
||||
GET /api/history
|
||||
```
|
||||
|
||||
## WebSocket API
|
||||
|
||||
### Connection
|
||||
```javascript
|
||||
const ws = new WebSocket('ws://localhost:3000/ws')
|
||||
|
||||
ws.onopen = () => {
|
||||
console.log('Connected to Brainy WebSocket')
|
||||
}
|
||||
|
||||
ws.onmessage = (event) => {
|
||||
const msg = JSON.parse(event.data)
|
||||
console.log('Received:', msg)
|
||||
}
|
||||
```
|
||||
|
||||
### Subscribe to Operations
|
||||
```javascript
|
||||
ws.send(JSON.stringify({
|
||||
type: 'subscribe',
|
||||
operations: ['all'] // or specific: ['add', 'search', 'delete']
|
||||
}))
|
||||
```
|
||||
|
||||
### Search via WebSocket
|
||||
```javascript
|
||||
ws.send(JSON.stringify({
|
||||
type: 'search',
|
||||
query: 'your search',
|
||||
limit: 10,
|
||||
requestId: 'unique-id'
|
||||
}))
|
||||
```
|
||||
|
||||
### Add Data via WebSocket
|
||||
```javascript
|
||||
ws.send(JSON.stringify({
|
||||
type: 'add',
|
||||
content: 'data to add',
|
||||
metadata: {},
|
||||
requestId: 'unique-id'
|
||||
}))
|
||||
```
|
||||
|
||||
### Operation Broadcasts
|
||||
When subscribed, you'll receive real-time updates:
|
||||
```javascript
|
||||
{
|
||||
"type": "operation",
|
||||
"operation": "add",
|
||||
"params": { /* sanitized parameters */ },
|
||||
"timestamp": 1234567890,
|
||||
"duration": 15
|
||||
}
|
||||
```
|
||||
|
||||
## MCP (Model Context Protocol)
|
||||
|
||||
The MCP endpoint allows AI agents to interact with Brainy:
|
||||
|
||||
```http
|
||||
POST /api/mcp
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"method": "search",
|
||||
"params": {
|
||||
"query": "find documents about AI"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
When authentication is enabled:
|
||||
|
||||
### API Key
|
||||
```http
|
||||
GET /api/stats
|
||||
X-API-Key: your-api-key
|
||||
```
|
||||
|
||||
### Bearer Token
|
||||
```http
|
||||
GET /api/stats
|
||||
Authorization: Bearer your-token
|
||||
```
|
||||
|
||||
## Environment Support
|
||||
|
||||
### Bun ✅ (Recommended)
|
||||
Native performance with Bun.serve() - no Express required. Works with `bun --compile` for single-binary deployment.
|
||||
|
||||
```typescript
|
||||
// Simple Bun server with Brainy
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
Bun.serve({
|
||||
port: 3000,
|
||||
async fetch(req) {
|
||||
const url = new URL(req.url)
|
||||
if (url.pathname === '/api/search') {
|
||||
const { query } = await req.json()
|
||||
const results = await brain.search(query)
|
||||
return Response.json(results)
|
||||
}
|
||||
return new Response('Not Found', { status: 404 })
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Node.js ✅
|
||||
Full support with Express, WebSocket, and all features.
|
||||
|
||||
### Deno 🚧
|
||||
Planned support using Deno.serve() or oak framework.
|
||||
|
||||
### Browser/Service Worker 🚧
|
||||
Planned support for intercepting fetch() calls locally.
|
||||
|
||||
## How It Works
|
||||
|
||||
The APIServerAugmentation hooks into Brainy's augmentation pipeline:
|
||||
|
||||
1. **Timing**: Executes `after` operations complete
|
||||
2. **Operations**: Monitors `all` operations
|
||||
3. **Broadcasting**: Sends operation details to subscribed clients
|
||||
4. **History**: Maintains operation history (last 1000 operations)
|
||||
|
||||
## Example: Multi-Client Sync
|
||||
|
||||
```typescript
|
||||
// Server
|
||||
const brain = new Brainy()
|
||||
brain.augmentations.register(new APIServerAugmentation())
|
||||
await brain.init()
|
||||
|
||||
// Client 1 - WebSocket subscriber
|
||||
const ws1 = new WebSocket('ws://localhost:3000/ws')
|
||||
ws1.onopen = () => {
|
||||
ws1.send(JSON.stringify({
|
||||
type: 'subscribe',
|
||||
operations: ['add', 'delete']
|
||||
}))
|
||||
}
|
||||
ws1.onmessage = (e) => {
|
||||
console.log('Client 1 received update:', JSON.parse(e.data))
|
||||
}
|
||||
|
||||
// Client 2 - REST API user
|
||||
fetch('http://localhost:3000/api/add', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
content: 'New data',
|
||||
metadata: { source: 'client2' }
|
||||
})
|
||||
})
|
||||
// Client 1 automatically receives notification!
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Operation History**: Limited to last 1000 operations
|
||||
- **WebSocket Heartbeat**: Every 30 seconds
|
||||
- **Client Timeout**: 60 seconds of inactivity
|
||||
- **Parameter Sanitization**: Sensitive fields removed, large content truncated
|
||||
- **Rate Limiting**: In-memory tracking (use Redis in production)
|
||||
|
||||
## Security Notes
|
||||
|
||||
1. **Default Configuration**: No auth, open CORS - suitable for development
|
||||
2. **Production**: Enable auth, configure CORS, use HTTPS
|
||||
3. **Sensitive Data**: Parameters are sanitized before broadcasting
|
||||
4. **Rate Limiting**: Basic in-memory implementation included
|
||||
|
||||
## Comparison with Previous Implementations
|
||||
|
||||
The APIServerAugmentation unifies and replaces:
|
||||
- `BrainyMCPBroadcast` - Node-specific WebSocket/HTTP server
|
||||
- `WebSocketConduitAugmentation` - WebSocket client functionality
|
||||
- `ServerSearchAugmentations` - Remote Brainy connections
|
||||
|
||||
Benefits of the unified approach:
|
||||
- Single augmentation for all API needs
|
||||
- Consistent interface across protocols
|
||||
- Automatic operation broadcasting
|
||||
- Environment-aware implementation
|
||||
- Zero-configuration philosophy
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Custom Operation Filtering
|
||||
|
||||
```typescript
|
||||
class FilteredAPIServer extends APIServerAugmentation {
|
||||
shouldExecute(operation: string, params: any): boolean {
|
||||
// Don't broadcast sensitive operations
|
||||
if (operation === 'delete' && params.sensitive) {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Integration with Other Augmentations
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
|
||||
// Stack augmentations for complete system
|
||||
brain.augmentations.register(new EntityRegistryAugmentation()) // Dedup
|
||||
brain.augmentations.register(new APIServerAugmentation()) // API
|
||||
|
||||
await brain.init()
|
||||
// All augmentations work together seamlessly!
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Server won't start
|
||||
- Check if port is already in use
|
||||
- Verify Node.js dependencies are installed: `npm install express cors ws`
|
||||
- Check console for error messages
|
||||
|
||||
### WebSocket connections drop
|
||||
- Ensure heartbeat responses are handled
|
||||
- Check for proxy/firewall issues
|
||||
- Verify CORS configuration
|
||||
|
||||
### Authentication not working
|
||||
- Ensure `auth.required` is set to `true`
|
||||
- Verify API keys or bearer tokens are correctly configured
|
||||
- Check request headers are properly formatted
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] Deno server implementation
|
||||
- [ ] Service Worker implementation
|
||||
- [ ] GraphQL endpoint
|
||||
- [ ] gRPC support
|
||||
- [ ] Built-in SSL/TLS
|
||||
- [ ] Redis-based rate limiting
|
||||
- [ ] Prometheus metrics endpoint
|
||||
- [ ] OpenAPI/Swagger documentation
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Augmentation System Overview](../AUGMENTATION-SYSTEM.md)
|
||||
- [BrainyAugmentation Interface](./brainy-augmentation.md)
|
||||
- [MCP Integration](../mcp/README.md)
|
||||
- [Zero-Config Philosophy](../ZERO-CONFIG.md)
|
||||
|
|
@ -1,386 +0,0 @@
|
|||
---
|
||||
title: Consistency Model
|
||||
slug: concepts/consistency-model
|
||||
public: true
|
||||
category: concepts
|
||||
template: concept
|
||||
order: 4
|
||||
description: The exact guarantees behind Brainy's Db API — snapshot isolation, atomic transactions, two levels of compare-and-swap, time travel, retention, snapshots, crash recovery, and the reserved-field contract.
|
||||
next:
|
||||
- guides/snapshots-and-time-travel
|
||||
- guides/optimistic-concurrency
|
||||
---
|
||||
|
||||
# Consistency Model
|
||||
|
||||
Brainy 8.0's consistency story rests on one mechanism: **generational MVCC**
|
||||
— multi-version concurrency control over immutable, generation-stamped
|
||||
records. It is exposed through a single value type, the **`Db`**: an
|
||||
immutable, point-in-time view of the whole store that you query like the
|
||||
live brain.
|
||||
|
||||
```typescript
|
||||
const db = brain.now() // pin the current state — O(1), no I/O
|
||||
|
||||
await brain.transact([
|
||||
{ op: 'update', id: invoiceId, metadata: { status: 'paid' } }
|
||||
])
|
||||
|
||||
await db.get(invoiceId) // still 'pending' — pinned, forever
|
||||
await brain.get(invoiceId) // 'paid' — live
|
||||
await db.release() // unpin when done
|
||||
```
|
||||
|
||||
This page states the guarantees precisely — what is promised, what it costs,
|
||||
and where the honest limits are. The design record is
|
||||
[ADR-001](../ADR-001-generational-mvcc.md); every guarantee below is proven
|
||||
by a dedicated test in `tests/integration/db-mvcc.test.ts`.
|
||||
|
||||
## The generation clock
|
||||
|
||||
A **monotonic generation counter** is the store's logical clock:
|
||||
|
||||
- It advances **once per committed `transact()` batch** and once per
|
||||
single-operation write (`add`/`update`/`remove`/`relate`/…).
|
||||
- `brain.generation()` reads it; it is persisted in the data directory and
|
||||
**never reissued** — not across restarts, and not across `restore()`
|
||||
(the counter is floored at its pre-restore value).
|
||||
|
||||
Every `Db` is pinned at one generation. `db.generation` and `db.timestamp`
|
||||
identify the view; `newerDb.since(olderDb)` returns exactly the entity and
|
||||
relationship ids that committed transactions touched between two views.
|
||||
|
||||
## Snapshot isolation for reads
|
||||
|
||||
**Guarantee:** a `Db` reads exactly the state at its pinned generation, no
|
||||
matter what commits afterwards — including deletes. There are no torn reads,
|
||||
no partially applied batches, and no drift over time.
|
||||
|
||||
- `brain.now()` pins the current generation in O(1).
|
||||
- `brain.transact()` returns a `Db` pinned at the freshly committed
|
||||
generation.
|
||||
- `brain.asOf(generation | Date | snapshotPath)` pins past state.
|
||||
|
||||
While nothing has committed past the pin, reads delegate to the live fast
|
||||
paths — pinning is free until history actually moves. Once later
|
||||
transactions commit, the view keeps serving the **full query surface** at
|
||||
its generation (see "Reading the past" below).
|
||||
|
||||
Writers are never blocked by readers and readers never block writers: a
|
||||
pinned view stays valid because nothing overwrites the immutable records it
|
||||
resolves from (the LMDB reader-pin model).
|
||||
|
||||
## Transaction atomicity
|
||||
|
||||
`brain.transact(ops)` executes a declarative batch — `add`, `update`,
|
||||
`remove`, `relate`, `unrelate` — **atomically as exactly one generation**:
|
||||
|
||||
```typescript
|
||||
const db = await brain.transact([
|
||||
{ op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' },
|
||||
{ op: 'add', id: itemId, type: NounType.Thing, subtype: 'line-item', data: 'Widget x3' },
|
||||
{ op: 'relate', from: orderId, to: itemId, type: VerbType.Contains, subtype: 'order-line' }
|
||||
], { meta: { author: 'order-service', requestId: 'req-9f2' } })
|
||||
|
||||
db.receipt.ids // resolved id per operation, in input order
|
||||
```
|
||||
|
||||
Either every operation applies, or none do and the store is byte-identical
|
||||
to its pre-transaction state. Operation semantics mirror the corresponding
|
||||
single-operation methods — validation, subtype enforcement, relationship
|
||||
deduplication, delete cascades — and later operations may reference ids
|
||||
created earlier in the same batch.
|
||||
|
||||
**The commit point is one atomic rename.** The durability protocol:
|
||||
|
||||
1. Before-images of every touched id are staged into an immutable
|
||||
generation directory and **fsynced**.
|
||||
2. The batch executes through the transaction manager (which has its own
|
||||
operation-level rollback for non-crash failures).
|
||||
3. The store manifest is replaced via atomic temp-file rename and fsynced.
|
||||
**The rename is the commit** — a generation is committed if and only if
|
||||
the manifest says so.
|
||||
|
||||
**Crash recovery:** on the next open, any staged generation above the
|
||||
manifest watermark is an uncommitted transaction; its before-images are
|
||||
restored (idempotently — recovery can itself crash and rerun) and derived
|
||||
indexes never observe the rolled-back state. A crash anywhere before the
|
||||
rename rolls back to the exact pre-transaction bytes; a crash after it keeps
|
||||
the transaction.
|
||||
|
||||
Transaction metadata (`meta`) is reified Datomic-style: recorded in an
|
||||
append-only transaction log readable via `brain.transactionLog()` — audit
|
||||
fields live in the database, not in commit messages.
|
||||
|
||||
## Two levels of compare-and-swap
|
||||
|
||||
Concurrent `transact()` calls commit serially (snapshot-isolated batches).
|
||||
For lost-update protection across a read–modify–write cycle, Brainy offers
|
||||
CAS at two granularities:
|
||||
|
||||
| Granularity | Mechanism | Conflict error | Use when |
|
||||
|---|---|---|---|
|
||||
| **Per entity** | `_rev` + `{ op: 'update', ifRev }` (also on `brain.update()`) | `RevisionConflictError` | "This entity must not have changed since I read it." |
|
||||
| **Whole store** | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` | "*Nothing* may have committed since I read." |
|
||||
|
||||
```typescript
|
||||
const view = brain.now()
|
||||
const order = await view.get(orderId)
|
||||
|
||||
try {
|
||||
await brain.transact(
|
||||
[{ op: 'update', id: orderId, metadata: { total: recompute(order) }, ifRev: order._rev }],
|
||||
{ ifAtGeneration: view.generation }
|
||||
)
|
||||
} catch (err) {
|
||||
if (err instanceof GenerationConflictError) {
|
||||
// Something committed since the pin — re-read and retry.
|
||||
}
|
||||
} finally {
|
||||
await view.release()
|
||||
}
|
||||
```
|
||||
|
||||
An `ifRev` conflict on any operation rejects the **whole batch**; an
|
||||
`ifAtGeneration` conflict is detected before anything is staged. Both leave
|
||||
the store untouched and the generation counter unchanged. See
|
||||
[Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md)
|
||||
for the per-entity pattern in depth.
|
||||
|
||||
## Reading the past
|
||||
|
||||
`brain.asOf()` accepts a generation number, a `Date` (resolved through the
|
||||
transaction log to the newest generation committed at or before it), or a
|
||||
snapshot directory path. Historical views serve the **full query surface**
|
||||
— `get()`, `find()` in every mode, semantic search, graph traversal,
|
||||
cursors, aggregation — through two complementary paths:
|
||||
|
||||
- **Record path** (free): `get()`, metadata-level `find()`, and
|
||||
filter-based `related()` resolve directly through the immutable record
|
||||
layer. Ids untouched since the pin still ride the live fast paths.
|
||||
- **Index path** (paid once): index-accelerated queries — semantic/vector
|
||||
search, graph traversal, cursors, aggregation — are served by an
|
||||
**at-generation index materialization** built lazily on first use:
|
||||
Brainy reconstructs in-memory indexes over the exact record set at that
|
||||
generation. This costs O(n at the pinned generation) time and memory,
|
||||
**once per `Db`**, cached until `release()`. That is the open-core price
|
||||
of historical index queries, stated plainly.
|
||||
|
||||
A native index provider implementing the optional
|
||||
`VersionedIndexProvider` plugin capability serves the same historical reads
|
||||
from its retained index segments **without any rebuild** — the materializer
|
||||
is the correctness baseline, the provider is the accelerator. Semantics are
|
||||
identical on both paths.
|
||||
|
||||
### History granularity — the honest limit
|
||||
|
||||
Generation *records* are written per `transact()` batch only.
|
||||
Single-operation writes (`add`/`update`/`remove`/`relate`/… outside
|
||||
`transact()`) advance the generation counter — so watermarks and CAS stay
|
||||
sound — but do **not** stage before-images: they remain visible through
|
||||
earlier pins and are not reported by `db.since()`. Code that needs pinned
|
||||
isolation across its own writes uses `transact()`. This is the documented
|
||||
8.0 contract, not an accident.
|
||||
|
||||
## Speculative writes: `db.with()`
|
||||
|
||||
`db.with(ops)` returns a new `Db` whose reads see the operations applied
|
||||
**in memory, on top of the view** — Datomic's `with`. Nothing touches disk,
|
||||
the generation counter, or index providers:
|
||||
|
||||
```typescript
|
||||
const current = brain.now()
|
||||
const whatIf = await current.with([
|
||||
{ op: 'update', id: employeeId, metadata: { team: 'platform' } }
|
||||
])
|
||||
|
||||
await whatIf.find({ where: { team: 'platform' } }) // sees the change
|
||||
await brain.get(employeeId) // unchanged — nothing committed
|
||||
```
|
||||
|
||||
**The one boundary:** overlay entities carry no embeddings (`with()` never
|
||||
invokes the embedder), so index-accelerated queries and `persist()` on a
|
||||
speculative view throw `SpeculativeOverlayError` rather than returning
|
||||
silently incomplete results. `get()`, metadata-filter `find()`, and
|
||||
filter-based `related()` work fully on overlays. To get the full surface,
|
||||
commit the same operations with `brain.transact()`.
|
||||
|
||||
## Retention and compaction
|
||||
|
||||
Historical records cost disk space, so retention is explicit:
|
||||
|
||||
- Every live `Db` holds a refcounted **pin**; a record-set is never
|
||||
reclaimed while any pin could need it — pinned reads stay correct across
|
||||
compaction, always.
|
||||
- The **`retention`** knob governs auto-compaction (on every `flush()`/
|
||||
`close()`): unset → ADAPTIVE (disk/RAM-pressure, zero-config) · `'all'` →
|
||||
unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit caps.
|
||||
`brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
|
||||
manually on the same caps, and records the **horizon** — `asOf()` below it
|
||||
throws `GenerationCompactedError`, explicitly, never partial data.
|
||||
- To keep a state readable forever, `persist()` it first: snapshots are
|
||||
self-contained and unaffected by compaction of the source store.
|
||||
|
||||
Release `Db` values you do not keep (including the ones `transact()`
|
||||
returns). A `FinalizationRegistry` backstop releases leaked pins at garbage
|
||||
collection, but explicit `release()` is what makes compaction
|
||||
deterministic.
|
||||
|
||||
## Durability: snapshots and restore
|
||||
|
||||
`db.persist(path)` cuts a **self-contained snapshot** under the store's
|
||||
commit mutex, so no commit or compaction can interleave. On filesystem
|
||||
storage it is built from **hard links**: because every data file is
|
||||
immutable-by-rename, linking is safe — the snapshot is created without
|
||||
copying entity data, shares disk space with the source, and later writes to
|
||||
the source can never alter it (rewrites swap inodes; the snapshot keeps the
|
||||
old bytes). Cross-device targets fall back to byte copies; in-memory stores
|
||||
serialize to the same directory layout, producing a real, durable store.
|
||||
|
||||
Two rules keep snapshots honest:
|
||||
|
||||
- `persist()` requires the view to still be the store's **latest**
|
||||
generation (a snapshot captures current bytes); a view that history has
|
||||
moved past throws `GenerationConflictError` instead of persisting the
|
||||
wrong state.
|
||||
- `brain.restore(path, { confirm: true })` replaces the store's entire
|
||||
state from a snapshot via byte copy (never links — the snapshot stays
|
||||
independent), rebuilds all indexes, and floors the generation counter so
|
||||
observed generation numbers are never reissued. Live pins do not survive
|
||||
a restore — release them first (a warning is logged when any exist).
|
||||
|
||||
`Brainy.load(path)` (or `brain.asOf(path)`) opens a snapshot as a
|
||||
self-contained **read-only** store with the full query surface, including
|
||||
vector search.
|
||||
|
||||
## Reserved fields
|
||||
|
||||
Some field names belong to Brainy, not to your metadata. They live at **top
|
||||
level** on every entity and relationship, have dedicated write paths, and may
|
||||
never appear inside a `metadata` bag:
|
||||
|
||||
| Entities (nouns) | Relationships (verbs) | Canonical write path |
|
||||
|---|---|---|
|
||||
| `noun` | `verb` | the `type` param of `add()` / `relate()` |
|
||||
| `subtype` | `subtype` | the `subtype` param |
|
||||
| `visibility` | `visibility` | the `visibility` param (`'public'` \| `'internal'`) |
|
||||
| `confidence` | `confidence` | the `confidence` param |
|
||||
| `weight` | `weight` | the `weight` param |
|
||||
| `service` | `service` | the `service` param (fixed at create time) |
|
||||
| `data` | `data` | the `data` param |
|
||||
| `createdBy` | `createdBy` | the `createdBy` param of `add()` (system-managed on verbs) |
|
||||
| `createdAt`, `updatedAt`, `_rev` | `createdAt`, `updatedAt`, `_rev` | system-managed (`ifRev` for CAS) |
|
||||
|
||||
The canonical machine-readable lists are exported as
|
||||
`RESERVED_ENTITY_FIELDS` and `RESERVED_RELATION_FIELDS` (defined in
|
||||
`src/types/reservedFields.ts`, the single source of truth). Three layers
|
||||
enforce the contract:
|
||||
|
||||
1. **Compile time** — every `metadata` param (`add`, `update`, `relate`,
|
||||
`updateRelation`, and the matching `transact()` operations) rejects a
|
||||
literal reserved key as a TypeScript error.
|
||||
2. **Write time** — untyped (JavaScript) callers that pass one anyway are
|
||||
normalized: user-settable fields (`confidence`, `weight`, `subtype`, and
|
||||
`service`/`createdBy` at create time) are remapped to their dedicated
|
||||
param — **top-level wins** when both are supplied — and system-managed
|
||||
fields are dropped with a one-shot warning naming the correct write path.
|
||||
`update({ metadata: { confidence: 0.9 } })` therefore behaves exactly
|
||||
like `update({ confidence: 0.9 })`.
|
||||
3. **Read time** — every read path (`get`, `find`, `search`,
|
||||
`related`, batch reads, and historical `asOf()` materialization)
|
||||
surfaces reserved fields **only at top level**: `entity.metadata` and
|
||||
`relation.metadata` contain only your custom fields, always.
|
||||
|
||||
```typescript
|
||||
const id = await brain.add({
|
||||
type: 'document', subtype: 'invoice',
|
||||
data: 'Invoice #42', confidence: 0.95, // reserved → top-level params
|
||||
metadata: { customer: 'acme', total: 129.5 } // custom fields only
|
||||
})
|
||||
|
||||
const entity = await brain.get(id)
|
||||
entity.confidence // 0.95 — top level
|
||||
entity.metadata // { customer: 'acme', total: 129.5 }
|
||||
```
|
||||
|
||||
### Visibility — `public` / `internal` / `system`
|
||||
|
||||
`visibility` is a reserved tier that controls whether an entity or relationship
|
||||
surfaces on Brainy's **default** user-facing reads. The absence of the field is
|
||||
exactly equivalent to `'public'`.
|
||||
|
||||
| Tier | Counted in `getNounCount()` / `stats()`? | Returned by default `find()` / `related()`? | Opt-in |
|
||||
|---|---|---|---|
|
||||
| `'public'` (default, or field absent) | yes | yes | — |
|
||||
| `'internal'` | no | no | `find({ includeInternal: true })` / `related({ includeInternal: true })` |
|
||||
| `'system'` | no | no | `find({ includeSystem: true })` / `related({ includeSystem: true })` |
|
||||
|
||||
- **`'public'`** — normal data. Counted and returned everywhere. Stored lean:
|
||||
the field is omitted on disk for public records, so existing data needs no
|
||||
migration.
|
||||
- **`'internal'`** — your app's own bookkeeping (audit trails, derived caches,
|
||||
scratch entities) that should not pollute default queries, counts, or
|
||||
`stats()`, yet must stay retrievable on demand. Set it via the `visibility`
|
||||
param; read it back with the `includeInternal` opt-in.
|
||||
- **`'system'`** — Brainy's own plumbing (for example the Virtual File System
|
||||
root entity). Hidden everywhere by default — even when `includeInternal` is
|
||||
set — and surfaced only with the explicit `includeSystem` opt-in. The
|
||||
`'system'` tier is **not** part of the public `add()` / `relate()` param type
|
||||
(`'public' | 'internal'`); only internal Brainy code assigns it.
|
||||
|
||||
The opt-ins are applied as a **hard candidate filter** — hidden entities are
|
||||
removed before `limit` / `offset` are applied, so a default `find({ limit: 10 })`
|
||||
always returns ten *visible* results when that many exist, never a short page.
|
||||
|
||||
```typescript
|
||||
// App-internal scratch entity: present, retrievable, but out of the way.
|
||||
await brain.add({ type: 'task', data: 'reindex job', visibility: 'internal' })
|
||||
|
||||
await brain.getNounCount() // unchanged — internal not counted
|
||||
await brain.find({ type: 'task' }) // [] — hidden by default
|
||||
await brain.find({ type: 'task', includeInternal: true }) // includes it
|
||||
|
||||
// A brand-new brain reports zero user entities even though the VFS root exists:
|
||||
const fresh = new Brainy()
|
||||
await fresh.init()
|
||||
await fresh.getNounCount() // 0 — the root is visibility:'system'
|
||||
```
|
||||
|
||||
> **Note (8.0):** the structural `Contains` edges the VFS creates between
|
||||
> directories and files are left at the default (`public`) visibility for now —
|
||||
> only the VFS *root entity* is `'system'`. Marking those edges system requires
|
||||
> companion changes to VFS traversal and is out of scope for this change.
|
||||
|
||||
## What is not guaranteed
|
||||
|
||||
Stated plainly, so nothing surprises you in production:
|
||||
|
||||
- **Single-writer.** Brainy is a single-writer, many-reader database
|
||||
([multi-process model](./multi-process.md)). Transactions are atomic
|
||||
within one writer process — there is no distributed or cross-process
|
||||
transaction coordination.
|
||||
- **History granularity.** Every write is its own immutable generation —
|
||||
`transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
|
||||
A pin always freezes against later writes, and every write is addressable
|
||||
via `asOf()`. (`transact()` groups several operations into ONE atomic
|
||||
generation; durability of single-op history is batched via async
|
||||
group-commit — a hard crash can lose only the last un-flushed window's
|
||||
*history*, never live data.)
|
||||
- **Compacted history is gone.** `asOf()` below the compaction horizon
|
||||
fails explicitly; persist what you must keep.
|
||||
- **Counter persistence is coalesced for single-operation writes.** Durable
|
||||
artifacts (records, manifests, snapshots) always persist the counter at
|
||||
their own commit points, so a crash inside the coalescing window can lose
|
||||
only counter values nothing durable ever referenced.
|
||||
- **Speculative overlays are metadata-only readers.** Index-accelerated
|
||||
queries on `with()` views throw rather than guess.
|
||||
|
||||
## Where to go next
|
||||
|
||||
- [Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) — the
|
||||
recipes: backup, restore, time-travel debugging, what-if analysis, audit
|
||||
trails.
|
||||
- [Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md)
|
||||
— the per-entity CAS pattern.
|
||||
- [ADR-001: Generational MVCC](../ADR-001-generational-mvcc.md) — the full
|
||||
design record, including the persisted layout and the proof table.
|
||||
|
|
@ -1,117 +0,0 @@
|
|||
---
|
||||
title: The Generation Fact Log
|
||||
slug: concepts/generation-fact-log
|
||||
public: true
|
||||
category: concepts
|
||||
template: concept
|
||||
order: 6
|
||||
description: Every committed write also appends a self-verifying "fact" — an after-image commit record — to an append-only log. What facts are, the crash-safety model, the scanFacts() streaming surface, family stamps, and how index providers consume the log for sequential heals.
|
||||
next:
|
||||
- concepts/consistency-model
|
||||
- guides/snapshots-and-time-travel
|
||||
---
|
||||
|
||||
# The Generation Fact Log
|
||||
|
||||
Since 8.4.0, every committed generation also appends a **fact** — a compact record of what each
|
||||
touched entity or relationship *became* — to an append-only, checksummed log under
|
||||
`_generations/facts/`. Where the generational history answers *"what did things look like
|
||||
before?"* (before-images, powering `asOf()` and rollback), the fact log answers *"what happened,
|
||||
in order?"* — one sequential, self-verifying stream of the store's present being written.
|
||||
|
||||
Nothing about querying changes. The fact log exists for three consumers:
|
||||
|
||||
1. **Index heals and rebuilds** — one sequential read in commit order replaces a per-entity
|
||||
directory walk over millions of files.
|
||||
2. **Incremental catch-up** — a derived index that knows which generation it reflects reads *just
|
||||
the gap*, instead of rebuilding from scratch.
|
||||
3. **Replay and audit tooling** — anything that wants the store's committed timeline as a stream.
|
||||
|
||||
## What a fact is
|
||||
|
||||
One fact per committed generation:
|
||||
|
||||
- **`generation`** and **`timestamp`** — which commit, when.
|
||||
- **`ops`** — every write in that commit: `{ kind: 'noun' | 'verb', id, record }` where `record`
|
||||
holds the entity's full after-image (both stored legs), or **`null` for a tombstone** — a
|
||||
removal carries no body, by design.
|
||||
- **`meta`** — the transaction metadata `transact()` was submitted with, when present.
|
||||
- **`blobHashes`** — content-blob references, for exact reclamation accounting.
|
||||
|
||||
Facts accumulate **from the first write after upgrading** — pre-existing history is not
|
||||
retroactively converted, and consumers fall back to the enumeration walk when no log exists.
|
||||
|
||||
## Crash safety, in one paragraph
|
||||
|
||||
Facts are appended and fsynced **inside the same durability window as the commit itself**, before
|
||||
the commit point — so after a crash, the log can only ever be *ahead* of committed truth, never
|
||||
behind it with a hole. On open, the store reconciles the log back to the committed watermark:
|
||||
torn tails are detected by per-record checksums and cut; whole records beyond the watermark are
|
||||
truncated. The invariant every reader can rely on: **an absent generation was never committed; a
|
||||
present fact was.** `transact()` facts are durable the moment `transact()` returns; single-op
|
||||
facts share the same group-commit flush as the rest of their generation, so a hard kill loses the
|
||||
fact and the generation *together* — never a torn state.
|
||||
|
||||
## Reading the log
|
||||
|
||||
```typescript
|
||||
const scan = brain.scanFacts({ fromGeneration: 1 })
|
||||
if (scan) {
|
||||
// Telemetry up front — progress bars get a denominator from second zero.
|
||||
console.log(scan.headGeneration, scan.segmentCount, scan.approxFactCount)
|
||||
|
||||
for await (const batch of scan.batches()) {
|
||||
// Each batch: { facts, firstGeneration, lastGeneration, factCount, byteSize, segmentId }
|
||||
for (const fact of batch.facts) {
|
||||
for (const op of fact.ops) {
|
||||
if (op.record === null) {
|
||||
// a tombstone: op.id was removed in this generation
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
console.log(scan.summary()) // { factsYielded, segmentsRead } — the cross-check
|
||||
}
|
||||
```
|
||||
|
||||
- `scanFacts()` returns `null` when the store hosts no fact log (older store, or a storage adapter
|
||||
without binary append support) — fall back to enumerating entities.
|
||||
- Scans run against a **snapshot**: facts appended after the scan opens never bleed in, each fact
|
||||
is yielded exactly once, and a detected gap aborts loudly — never a silent skip.
|
||||
- `brain.factSegmentPaths()` returns the immutable, *sealed* segment files for zero-copy consumers
|
||||
(the append-mutable tail is excluded — read it through `scanFacts()`).
|
||||
|
||||
## Family stamps: how a projection proves it's current
|
||||
|
||||
Anything derived from the store — an index, the entity file tree itself — carries a **family
|
||||
stamp**: a small JSON record of *which committed generation the projection reflects*
|
||||
(`sourceGeneration`) plus the invariants that verify it whole (exact per-file byte sizes for
|
||||
bounded families; rollup invariants like entity counts for unbounded ones). At open, coherence is
|
||||
a **comparison**, not a walk:
|
||||
|
||||
- stamp equals the committed watermark and invariants hold → serve;
|
||||
- stamp is behind → the projection reads just the gap from the fact log;
|
||||
- invariants fail → loud, named divergence — `brain.repairIndex()` rebuilds from canonical and
|
||||
re-stamps.
|
||||
|
||||
The verifier is exported (`verifyFamilyStamp`) so every projection — TypeScript or native — runs
|
||||
literally the same check.
|
||||
|
||||
## For plugin authors: the storage capability
|
||||
|
||||
Index providers receive the storage adapter, not the brain — so the host wires the log onto it.
|
||||
Feature-detect and prefer the stream; fall back to enumeration:
|
||||
|
||||
```typescript
|
||||
const scan = storage.scanFacts?.({ fromGeneration: stamp.sourceGeneration + 1 })
|
||||
if (scan) {
|
||||
// sequential catch-up from the log
|
||||
} else {
|
||||
// enumeration walk (older store or adapter)
|
||||
}
|
||||
const committed = storage.committedGeneration?.() // the watermark stamps compare against
|
||||
```
|
||||
|
||||
Providers must never construct their own reader over the log's files — the open path belongs to
|
||||
the single writer (it reconciles the log at open); the capability is the sanctioned seam.
|
||||
|
|
@ -1,153 +0,0 @@
|
|||
---
|
||||
title: Multi-Process Model
|
||||
slug: concepts/multi-process
|
||||
public: true
|
||||
category: concepts
|
||||
template: concept
|
||||
order: 5
|
||||
description: How Brainy coordinates a single writer with any number of readers on a filesystem data directory — and how to safely inspect a live store.
|
||||
next:
|
||||
- guides/inspection
|
||||
---
|
||||
|
||||
# Multi-Process Model
|
||||
|
||||
Brainy is a **single-writer, many-reader** database when backed by filesystem
|
||||
storage. This page explains the model, the guarantees, and the safe ways to
|
||||
inspect a live store from a second process.
|
||||
|
||||
## The rule
|
||||
|
||||
For one data directory:
|
||||
|
||||
- **One writer** at a time. The writer acquires an exclusive lock on the
|
||||
directory at `init()` and releases it on `close()`.
|
||||
- **Any number of readers**, concurrent with each other and with the writer.
|
||||
Readers open via `Brainy.openReadOnly()` — they never touch the writer
|
||||
lock.
|
||||
|
||||
Any attempt to open a second writer on the same directory throws:
|
||||
|
||||
```
|
||||
BrainyError: Another writer holds this Brainy directory.
|
||||
PID: 1774431 on host app-host-1
|
||||
Started: 2026-05-15T14:22:11Z
|
||||
Heartbeat: 2026-05-15T14:22:34Z
|
||||
Version: 7.21.0
|
||||
Directory: /data/brain
|
||||
```
|
||||
|
||||
This is intentional. Two writers sharing a directory would silently corrupt
|
||||
in-memory indexes and produce wrong query results — the worst possible default
|
||||
for an operations tool.
|
||||
|
||||
## Why a lock?
|
||||
|
||||
Brainy keeps its primary indexes (HNSW, metadata, graph adjacency) in memory.
|
||||
On disk, those indexes are persisted incrementally as writes flush. A second
|
||||
process opening the same directory:
|
||||
|
||||
- Loads the *persisted* state into a fresh in-memory copy.
|
||||
- Has no awareness of writes the first process buffered but hasn't flushed.
|
||||
- Will overwrite the persisted state on its own next flush, racing the first
|
||||
process and corrupting whichever wins.
|
||||
|
||||
The fix is the lock: refuse to open a second writer. SQLite has done the same
|
||||
since the late 1990s (`SQLITE_BUSY`).
|
||||
|
||||
## What about Cor?
|
||||
|
||||
Brainy + Cor compose cleanly under this model:
|
||||
|
||||
- Cor stores its column-index segments inside the same `rootDir` (under
|
||||
`indexes/_column_index/{field}/`).
|
||||
- Segments (`*.cidx` files) are **immutable** once written. Cor mmaps them
|
||||
read-only.
|
||||
- The `MANIFEST.json` per field is updated via atomic rename — readers see
|
||||
either the old or new manifest, never a torn file.
|
||||
|
||||
A reader process can safely mmap Cor segments alongside a live writer
|
||||
without coordination. The single Brainy writer lock at
|
||||
`<rootDir>/locks/_writer.lock` covers Cor too, because Cor segment
|
||||
writes happen on the writer's side.
|
||||
|
||||
## Stale-lock detection
|
||||
|
||||
If a writer crashes or is forcibly killed, its lock file is left behind. To
|
||||
avoid a permanently-jammed directory, Brainy treats a lock as stale when:
|
||||
|
||||
1. The recorded `hostname` equals the current host (cross-host PID checks
|
||||
are unsafe), AND
|
||||
2. The recorded `pid` is no longer alive (`process.kill(pid, 0)` returns
|
||||
`ESRCH`), OR the `lastHeartbeat` field is older than 60 seconds.
|
||||
|
||||
A live writer rewrites `lastHeartbeat` every 10 seconds, so a hung writer
|
||||
that's missed several heartbeats is treated as dead. Stale locks are
|
||||
overwritten with a warning.
|
||||
|
||||
If stale detection cannot prove the existing lock is dead — for example, a
|
||||
crashed writer on a different host writing to a shared filesystem — pass
|
||||
`{ force: true }` to override. A warning is logged either way.
|
||||
|
||||
## Heartbeat and shutdown
|
||||
|
||||
The heartbeat interval rewrites the lock file every 10 seconds. The timer
|
||||
is unref'd, so it does not keep the event loop alive on its own.
|
||||
|
||||
On normal shutdown the writer releases the lock in `close()`. The shutdown
|
||||
hooks Brainy registers for `SIGTERM`, `SIGINT`, and `beforeExit` also
|
||||
release the lock so a container restart doesn't strand the directory.
|
||||
|
||||
## How to inspect a live writer
|
||||
|
||||
Use `Brainy.openReadOnly()`. It does not acquire the writer lock, so it
|
||||
coexists with whatever the writer is doing:
|
||||
|
||||
```typescript
|
||||
const reader = await Brainy.openReadOnly({
|
||||
storage: { type: 'filesystem', path: '/data/brain' }
|
||||
})
|
||||
|
||||
const stats = await reader.stats()
|
||||
const bookings = await reader.find({ where: { entityType: 'booking' } })
|
||||
|
||||
await reader.close()
|
||||
```
|
||||
|
||||
What the reader sees reflects the writer's most recent **flush** to disk. If
|
||||
you need fresher state, ask the writer to flush before opening:
|
||||
|
||||
```typescript
|
||||
const reader = await Brainy.openReadOnly({
|
||||
storage: { type: 'filesystem', path: '/data/brain' }
|
||||
})
|
||||
|
||||
const acked = await reader.requestFlush({ timeoutMs: 5000 })
|
||||
if (!acked) {
|
||||
console.warn('Writer did not respond; results reflect last natural flush.')
|
||||
}
|
||||
|
||||
const fresh = await reader.find({ where: { entityType: 'booking' } })
|
||||
```
|
||||
|
||||
The CLI `brainy inspect` subcommands all do this for you by default
|
||||
(`--no-fresh` to opt out).
|
||||
|
||||
## What's not enforced (yet)
|
||||
|
||||
- **Non-filesystem backends** are out of scope in 8.0, which ships only the
|
||||
filesystem and memory adapters. A custom `BaseStorage` subclass that is not
|
||||
filesystem-backed does not enforce multi-process locking by default: two
|
||||
processes can both succeed at `init()` in writer mode and clobber each
|
||||
other's writes. A best-effort warning is logged in writer mode against a
|
||||
non-filesystem backend.
|
||||
- **Long-running readers** do not automatically pick up new Cor segments
|
||||
the writer publishes. One-shot inspector calls re-open the store and see
|
||||
fresh segments; a reader that stays open for hours sees its column store
|
||||
as-of the time it opened.
|
||||
|
||||
## Reading material
|
||||
|
||||
- `Brainy.openReadOnly()` — [API reference](../api/brainy.md)
|
||||
- `brainy inspect` — [inspection guide](../guides/inspection.md)
|
||||
- Cor columnar storage — see `node_modules/@soulcraft/cor/README.md`
|
||||
|
|
@ -1,189 +0,0 @@
|
|||
---
|
||||
title: Storage Adapter Inheritance Contract
|
||||
slug: concepts/storage-adapters
|
||||
public: true
|
||||
category: concepts
|
||||
template: concept
|
||||
order: 6
|
||||
description: How storage adapters extend Brainy's BaseStorage and FileSystemStorage to inherit multi-process safety, what plugin authors need to override, and the contract Brainy promises to keep stable.
|
||||
next:
|
||||
- concepts/multi-process
|
||||
- guides/inspection
|
||||
---
|
||||
|
||||
# Storage Adapter Inheritance Contract
|
||||
|
||||
Brainy's storage layer is designed for plugins to extend cleanly. A plugin
|
||||
that subclasses `BaseStorage` or `FileSystemStorage` inherits new behavior
|
||||
Brainy adds over time without code changes — provided the import resolution
|
||||
brings in the right Brainy version at runtime.
|
||||
|
||||
This page documents what plugin authors can rely on, what they need to
|
||||
override, and the install-time failure modes that the defensive
|
||||
`hasStorageMethod()` guard exists to handle.
|
||||
|
||||
## The class hierarchy
|
||||
|
||||
```
|
||||
BaseStorageAdapter (counts, batch ops, multi-tenancy hooks)
|
||||
↑
|
||||
BaseStorage (type-statistics, lifecycle helpers, generation
|
||||
hooks, default no-op multi-process methods)
|
||||
↑
|
||||
FileSystemStorage (real filesystem I/O, writer-lock implementation,
|
||||
flush-request watcher, atomic writes)
|
||||
↑
|
||||
<your plugin's storage> (Cor's MmapFileSystemStorage, etc.)
|
||||
```
|
||||
|
||||
When you `extend FileSystemStorage`, your adapter inherits every method on
|
||||
the chain — including the ones Brainy adds in a later release — for free.
|
||||
JavaScript's prototype chain resolves method lookups dynamically; nothing
|
||||
about the inheritance is baked in at class-definition time.
|
||||
|
||||
## What you inherit for free
|
||||
|
||||
A plugin whose storage class extends `FileSystemStorage` automatically gets
|
||||
the full multi-process safety surface:
|
||||
|
||||
| Method | What it does | Override? |
|
||||
|---|---|---|
|
||||
| `acquireWriterLock(opts)` | Write `_writer.lock`, start heartbeat, throw on conflict | No |
|
||||
| `releaseWriterLock()` | Clean up lock file + heartbeat timer | No |
|
||||
| `readWriterLock()` | Read lock file as `WriterLockInfo` | No |
|
||||
| `startFlushRequestWatcher(cb)` | Poll `_flush_requests/` and invoke `cb` | No |
|
||||
| `stopFlushRequestWatcher()` | Stop the polling timer | No |
|
||||
| `requestFlushOverFilesystem(timeoutMs)` | Drop a `.req`, await `.ack` | No |
|
||||
| `supportsMultiProcessLocking()` | Return `false` (default) | **Yes — override to `true`** |
|
||||
|
||||
The only required override is the capability flag. Returning `true` from
|
||||
`supportsMultiProcessLocking()` is the signal Brainy uses to decide whether
|
||||
to call `acquireWriterLock()` at init.
|
||||
|
||||
```typescript
|
||||
import { FileSystemStorage } from '@soulcraft/brainy'
|
||||
|
||||
export class MmapFileSystemStorage extends FileSystemStorage {
|
||||
public supportsMultiProcessLocking(): boolean {
|
||||
return true
|
||||
}
|
||||
// ... your mmap-specific overrides ...
|
||||
}
|
||||
```
|
||||
|
||||
That's the full ceremony for inheriting multi-process safety.
|
||||
|
||||
## When NOT to extend FileSystemStorage
|
||||
|
||||
If your storage is **not filesystem-backed** (a custom
|
||||
network backend), extend `BaseStorage` directly:
|
||||
|
||||
```typescript
|
||||
import { BaseStorage } from '@soulcraft/brainy'
|
||||
|
||||
export class MyCloudStorage extends BaseStorage {
|
||||
// BaseStorage's default no-op implementations of the multi-process
|
||||
// methods stay in effect. `supportsMultiProcessLocking()` returns false
|
||||
// by default — keep it that way unless you've implemented an object-
|
||||
// versioned lease or similar cross-process synchronization for your
|
||||
// backend.
|
||||
}
|
||||
```
|
||||
|
||||
Brainy treats cloud backends as not-multi-process-safe by default and logs a
|
||||
one-line warning at init. That's the correct behavior until cloud locking
|
||||
ships (currently out of scope — see
|
||||
[`concepts/multi-process`](./multi-process.md)).
|
||||
|
||||
## What `hasStorageMethod()` actually guards against
|
||||
|
||||
The defensive check at every new-storage-method call site (`brainy.ts`,
|
||||
`hasStorageMethod(name)`) does **not** exist to handle "plugin bundles a
|
||||
stale BaseStorage." Plugins ship a dist that preserves the dynamic ESM
|
||||
import (verify in your plugin's `dist/`: `import { FileSystemStorage } from
|
||||
'@soulcraft/brainy'` is not rewritten to a vendored copy). The prototype
|
||||
chain at runtime resolves to whatever Brainy version your consumer has
|
||||
installed.
|
||||
|
||||
`hasStorageMethod()` protects against **build/install artifacts** that break
|
||||
the prototype chain at the consumer-app level:
|
||||
|
||||
- **Stale `node_modules`** — a lingering install from before the consumer
|
||||
upgraded Brainy. The package.json says `@soulcraft/brainy@7.22.0` but
|
||||
`node_modules/@soulcraft/brainy` is still 7.20.x.
|
||||
- **Lockfile drift** — `bun.lockb` / `package-lock.json` pins a brainy
|
||||
version older than the package.json range, and `bun install` honors the
|
||||
lockfile.
|
||||
- **Docker layer cache** — the image reuses a `node_modules` from an
|
||||
earlier build that predates the brainy bump.
|
||||
- **Bundler quirks** — some bundlers (esbuild, webpack) flatten the
|
||||
prototype chain at build time and lose later prototype mutations. Brainy
|
||||
doesn't mutate prototypes at runtime, but bundler behavior can still
|
||||
cause method lookups to fail in non-Node environments.
|
||||
|
||||
In any of those, calling `storage.acquireWriterLock(...)` unconditionally
|
||||
throws `TypeError: storage.acquireWriterLock is not a function`. The guard
|
||||
turns that into a logged warning + graceful no-op so the app still boots,
|
||||
and the warning names the adapter class plus a remediation hint:
|
||||
|
||||
```
|
||||
[brainy] Storage adapter `MmapFileSystemStorage` is missing the multi-process
|
||||
methods on its prototype chain. Writer locking and the flush-request RPC are
|
||||
disabled for this directory. Likely fix: clean install (`rm -rf node_modules
|
||||
bun.lockb && bun install`) or rebuild your container image to refresh
|
||||
`@soulcraft/brainy` to ≥7.21. See docs/concepts/storage-adapters.md.
|
||||
```
|
||||
|
||||
## Authoring a new storage adapter — minimum checklist
|
||||
|
||||
1. **Extend the right base class.**
|
||||
- Filesystem-backed → `FileSystemStorage`.
|
||||
- Cloud / network / custom → `BaseStorage`.
|
||||
|
||||
2. **Override the capability flag.** If filesystem-backed:
|
||||
```typescript
|
||||
public supportsMultiProcessLocking(): boolean { return true }
|
||||
```
|
||||
|
||||
3. **Don't `super.X()`-wrap the multi-process methods**. They're inherited;
|
||||
leaving them inherited means `hasStorageMethod()` finds them on the
|
||||
prototype chain. Re-declaring them as `super.X()` wrappers makes the
|
||||
helper resolve to your wrapper, which can fool the guard if your
|
||||
constructor runs before the super class initializes.
|
||||
|
||||
4. **Do override `init()` / `flush()` / `close()`** as needed. Always call
|
||||
`super.init()` / `super.flush()` / `super.close()` first so the
|
||||
filesystem prep, writer-lock acquisition, and lock release happen in the
|
||||
expected order.
|
||||
|
||||
5. **Verify the inheritance.** A one-line smoke test in your plugin's
|
||||
`__tests__/`:
|
||||
```typescript
|
||||
const s = new MyStorage(rootDir)
|
||||
await s.init()
|
||||
assert(typeof s.acquireWriterLock === 'function')
|
||||
assert(s.supportsMultiProcessLocking() === true)
|
||||
```
|
||||
If `acquireWriterLock` is undefined the prototype chain is broken at
|
||||
install time — fix install, not your plugin.
|
||||
|
||||
6. **Pin your peer dep generously.** `"peerDependencies": {
|
||||
"@soulcraft/brainy": "^7.21.0" }` accepts any compatible 7.x. Don't pin
|
||||
to an exact patch unless you're tracking a known regression.
|
||||
|
||||
## Future direction
|
||||
|
||||
The 7 multi-process methods are currently defaults on `BaseStorage`. A
|
||||
future refactor may extract them into a `MultiProcessSafeStorage`
|
||||
interface/mixin for cleaner separation — only adapters that opt in would
|
||||
expose them. This would require a minor bump and is tracked as an internal
|
||||
follow-up; consumers don't need to anticipate the change.
|
||||
|
||||
## Reading material
|
||||
|
||||
- [`concepts/multi-process`](./multi-process.md) — the writer-lock model,
|
||||
heartbeat semantics, what the lock protects.
|
||||
- [`guides/inspection`](../guides/inspection.md) — `brainy inspect` and the
|
||||
read-only mode.
|
||||
- `node_modules/@soulcraft/brainy/dist/storage/baseStorage.d.ts` — the
|
||||
authoritative type signatures for every method this page references.
|
||||
833
docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md
Normal file
833
docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md
Normal file
|
|
@ -0,0 +1,833 @@
|
|||
# Brainy Cloud Deployment Guide
|
||||
|
||||
This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy codebase.
|
||||
|
||||
## 🆕 Production Features
|
||||
|
||||
**Cost Optimization at Scale:**
|
||||
- **Lifecycle Policies**: Automatic tier transitions for 96% cost savings
|
||||
- **Intelligent-Tiering**: S3 Intelligent-Tiering and GCS Autoclass support
|
||||
- **Batch Operations**: Efficient bulk delete (1000 objects per request)
|
||||
- **Compression**: Gzip compression for 60-80% storage savings
|
||||
|
||||
**Example Impact (500TB dataset):**
|
||||
- Before: $138,000/year
|
||||
- With lifecycle policies: $5,940/year
|
||||
- **Savings: $132,060/year (96%)**
|
||||
|
||||
See the [Cost Optimization](#cost-optimization-v40) section below for implementation details.
|
||||
|
||||
## Overview
|
||||
|
||||
The API Server augmentation provides a universal handler that works with standard Request/Response objects, making Brainy deployable on any JavaScript runtime.
|
||||
|
||||
## Storage Adapter
|
||||
|
||||
**S3CompatibleStorage** is the preferred storage adapter for cloud deployments. It works with:
|
||||
- Amazon S3
|
||||
- Cloudflare R2
|
||||
- Google Cloud Storage
|
||||
- Azure Blob Storage
|
||||
- Any S3-compatible service
|
||||
|
||||
## Deployment Examples
|
||||
|
||||
### AWS Lambda
|
||||
|
||||
```javascript
|
||||
// handler.js
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
|
||||
|
||||
let brain
|
||||
let handler
|
||||
|
||||
exports.handler = async (event) => {
|
||||
if (!brain) {
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: 's3.amazonaws.com',
|
||||
region: process.env.AWS_REGION,
|
||||
bucket: process.env.BRAINY_BUCKET,
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
|
||||
prefix: 'brainy-data/'
|
||||
})
|
||||
|
||||
brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
auth: {
|
||||
required: true,
|
||||
apiKeys: [process.env.API_KEY]
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
// Get the universal handler from the augmentation
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
// Convert Lambda event to Request
|
||||
const url = `https://${event.requestContext.domainName}${event.rawPath}`
|
||||
const request = new Request(url, {
|
||||
method: event.requestContext.http.method,
|
||||
headers: event.headers,
|
||||
body: event.body
|
||||
})
|
||||
|
||||
// Use the universal handler
|
||||
const response = await handler(request)
|
||||
|
||||
return {
|
||||
statusCode: response.status,
|
||||
headers: Object.fromEntries(response.headers),
|
||||
body: await response.text()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Google Cloud Functions
|
||||
|
||||
```javascript
|
||||
// index.js
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
|
||||
|
||||
let brain
|
||||
let handler
|
||||
|
||||
exports.brainyAPI = async (req, res) => {
|
||||
if (!brain) {
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: 'storage.googleapis.com',
|
||||
bucket: process.env.GCS_BUCKET,
|
||||
accessKeyId: process.env.GCS_ACCESS_KEY,
|
||||
secretAccessKey: process.env.GCS_SECRET_KEY,
|
||||
prefix: 'brainy/',
|
||||
forcePathStyle: false,
|
||||
region: 'US'
|
||||
})
|
||||
|
||||
brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
// Convert Express req/res to Request/Response
|
||||
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
|
||||
method: req.method,
|
||||
headers: req.headers,
|
||||
body: JSON.stringify(req.body)
|
||||
})
|
||||
|
||||
const response = await handler(request)
|
||||
|
||||
res.status(response.status)
|
||||
res.set(Object.fromEntries(response.headers))
|
||||
res.send(await response.text())
|
||||
}
|
||||
```
|
||||
|
||||
### Google Cloud Run with Cloud Storage
|
||||
|
||||
Google Cloud Run is ideal for containerized deployments with automatic scaling. This example uses Google Cloud Storage via the S3-compatible API.
|
||||
|
||||
```javascript
|
||||
// server.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
|
||||
import express from 'express'
|
||||
|
||||
const app = express()
|
||||
app.use(express.json())
|
||||
|
||||
const PORT = process.env.PORT || 8080
|
||||
|
||||
let brain
|
||||
let handler
|
||||
|
||||
async function initBrainy() {
|
||||
// Google Cloud Storage is S3-compatible
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: 'storage.googleapis.com',
|
||||
bucket: process.env.GCS_BUCKET || 'brainy-data',
|
||||
accessKeyId: process.env.GCS_ACCESS_KEY,
|
||||
secretAccessKey: process.env.GCS_SECRET_KEY,
|
||||
prefix: 'brainy/',
|
||||
forcePathStyle: false,
|
||||
region: process.env.GCS_REGION || 'US'
|
||||
})
|
||||
|
||||
brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
port: PORT,
|
||||
cors: {
|
||||
origin: process.env.CORS_ORIGIN || '*'
|
||||
},
|
||||
auth: {
|
||||
required: process.env.AUTH_REQUIRED === 'true',
|
||||
apiKeys: process.env.API_KEY ? [process.env.API_KEY] : []
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
// Initialize on startup
|
||||
await initBrainy()
|
||||
|
||||
// Health check endpoint
|
||||
app.get('/health', (req, res) => {
|
||||
res.json({ status: 'healthy', service: 'brainy-api' })
|
||||
})
|
||||
|
||||
// Universal handler for all API routes
|
||||
app.use('*', async (req, res) => {
|
||||
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
|
||||
method: req.method,
|
||||
headers: req.headers,
|
||||
body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined
|
||||
})
|
||||
|
||||
const response = await handler(request)
|
||||
|
||||
res.status(response.status)
|
||||
res.set(Object.fromEntries(response.headers))
|
||||
res.send(await response.text())
|
||||
})
|
||||
|
||||
app.listen(PORT, () => {
|
||||
console.log(`Brainy API running on port ${PORT}`)
|
||||
})
|
||||
```
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM node:20-alpine AS builder
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci --only=production
|
||||
|
||||
FROM node:20-alpine
|
||||
WORKDIR /app
|
||||
COPY --from=builder /app/node_modules ./node_modules
|
||||
COPY . .
|
||||
EXPOSE 8080
|
||||
CMD ["node", "server.js"]
|
||||
```
|
||||
|
||||
```yaml
|
||||
# cloudbuild.yaml
|
||||
steps:
|
||||
# Build the container image
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.']
|
||||
|
||||
# Push to Container Registry
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA']
|
||||
|
||||
# Deploy to Cloud Run
|
||||
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
|
||||
entrypoint: gcloud
|
||||
args:
|
||||
- 'run'
|
||||
- 'deploy'
|
||||
- 'brainy-api'
|
||||
- '--image'
|
||||
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
|
||||
- '--region'
|
||||
- 'us-central1'
|
||||
- '--platform'
|
||||
- 'managed'
|
||||
- '--allow-unauthenticated'
|
||||
- '--set-env-vars'
|
||||
- 'GCS_BUCKET=brainy-storage,GCS_REGION=US'
|
||||
- '--set-secrets'
|
||||
- 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest'
|
||||
- '--memory'
|
||||
- '2Gi'
|
||||
- '--cpu'
|
||||
- '2'
|
||||
- '--max-instances'
|
||||
- '100'
|
||||
- '--min-instances'
|
||||
- '0'
|
||||
|
||||
images:
|
||||
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
|
||||
```
|
||||
|
||||
**Deploy with gcloud CLI:**
|
||||
```bash
|
||||
# Build and submit to Cloud Build
|
||||
gcloud builds submit --config cloudbuild.yaml
|
||||
|
||||
# Or deploy directly
|
||||
gcloud run deploy brainy-api \
|
||||
--source . \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated \
|
||||
--set-env-vars GCS_BUCKET=brainy-storage \
|
||||
--set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest"
|
||||
```
|
||||
|
||||
**Create GCS Bucket with S3-compatible access:**
|
||||
```bash
|
||||
# Create bucket
|
||||
gsutil mb -p PROJECT_ID -c STANDARD -l US gs://brainy-storage/
|
||||
|
||||
# Enable interoperability
|
||||
gsutil iam ch serviceAccount:SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com:objectAdmin gs://brainy-storage
|
||||
|
||||
# Generate HMAC keys for S3-compatible access
|
||||
gsutil hmac create SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com
|
||||
|
||||
# Store the access key and secret in Secret Manager
|
||||
echo -n "YOUR_ACCESS_KEY" | gcloud secrets create gcs-access-key --data-file=-
|
||||
echo -n "YOUR_SECRET_KEY" | gcloud secrets create gcs-secret-key --data-file=-
|
||||
```
|
||||
|
||||
### Microsoft Azure Functions
|
||||
|
||||
```javascript
|
||||
// index.js
|
||||
module.exports = async function (context, req) {
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
|
||||
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`,
|
||||
bucket: 'brainy-data',
|
||||
accessKeyId: process.env.AZURE_STORAGE_ACCOUNT,
|
||||
secretAccessKey: process.env.AZURE_STORAGE_KEY,
|
||||
prefix: 'entities/',
|
||||
forcePathStyle: false
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
const handler = apiAugmentation.createUniversalHandler()
|
||||
|
||||
const request = new Request(`https://${context.req.headers.host}${context.req.url}`, {
|
||||
method: context.req.method,
|
||||
headers: context.req.headers,
|
||||
body: JSON.stringify(context.req.body)
|
||||
})
|
||||
|
||||
const response = await handler(request)
|
||||
|
||||
context.res = {
|
||||
status: response.status,
|
||||
headers: Object.fromEntries(response.headers),
|
||||
body: await response.text()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cloudflare Workers
|
||||
|
||||
```javascript
|
||||
// worker.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { R2Storage } from '@soulcraft/brainy/storage' // Alias for S3CompatibleStorage
|
||||
|
||||
let handler
|
||||
|
||||
export default {
|
||||
async fetch(request, env, ctx) {
|
||||
if (!handler) {
|
||||
const storage = new R2Storage({
|
||||
endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`,
|
||||
bucket: 'brainy-data',
|
||||
accessKeyId: env.R2_ACCESS_KEY_ID,
|
||||
secretAccessKey: env.R2_SECRET_ACCESS_KEY,
|
||||
region: 'auto',
|
||||
forcePathStyle: true
|
||||
})
|
||||
|
||||
const brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
cors: { origin: '*' }
|
||||
}
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
// The handler works directly with Request/Response!
|
||||
return handler(request)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```toml
|
||||
# wrangler.toml
|
||||
name = "brainy-api"
|
||||
main = "worker.js"
|
||||
compatibility_date = "2024-01-01"
|
||||
|
||||
[[r2_buckets]]
|
||||
binding = "R2"
|
||||
bucket_name = "brainy-data"
|
||||
|
||||
[vars]
|
||||
ACCOUNT_ID = "your-account-id"
|
||||
|
||||
[env.production.vars]
|
||||
R2_ACCESS_KEY_ID = "your-access-key"
|
||||
R2_SECRET_ACCESS_KEY = "your-secret-key"
|
||||
```
|
||||
|
||||
### Vercel Edge Functions
|
||||
|
||||
```javascript
|
||||
// api/brainy.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
|
||||
|
||||
let handler
|
||||
|
||||
export const config = {
|
||||
runtime: 'edge',
|
||||
}
|
||||
|
||||
export default async (request) => {
|
||||
if (!handler) {
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: 's3.amazonaws.com',
|
||||
region: 'us-east-1',
|
||||
bucket: process.env.S3_BUCKET,
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
})
|
||||
|
||||
const brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: { enabled: true }
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
return handler(request)
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
// vercel.json
|
||||
{
|
||||
"functions": {
|
||||
"api/brainy.js": {
|
||||
"maxDuration": 30,
|
||||
"memory": 1024
|
||||
}
|
||||
},
|
||||
"rewrites": [
|
||||
{
|
||||
"source": "/api/:path*",
|
||||
"destination": "/api/brainy"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Railway
|
||||
|
||||
```javascript
|
||||
// server.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
|
||||
import express from 'express'
|
||||
|
||||
const app = express()
|
||||
const PORT = process.env.PORT || 3000
|
||||
|
||||
let brain
|
||||
let handler
|
||||
|
||||
async function init() {
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com',
|
||||
region: process.env.S3_REGION || 'us-east-1',
|
||||
bucket: process.env.S3_BUCKET,
|
||||
accessKeyId: process.env.S3_ACCESS_KEY,
|
||||
secretAccessKey: process.env.S3_SECRET_KEY,
|
||||
prefix: 'brainy/'
|
||||
})
|
||||
|
||||
brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
port: PORT
|
||||
}
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
await init()
|
||||
|
||||
// Universal handler for all routes
|
||||
app.use('*', async (req, res) => {
|
||||
const request = new Request(`http://localhost${req.originalUrl}`, {
|
||||
method: req.method,
|
||||
headers: req.headers,
|
||||
body: JSON.stringify(req.body)
|
||||
})
|
||||
|
||||
const response = await handler(request)
|
||||
|
||||
res.status(response.status)
|
||||
res.set(Object.fromEntries(response.headers))
|
||||
res.send(await response.text())
|
||||
})
|
||||
|
||||
app.listen(PORT, () => {
|
||||
console.log(`Brainy API running on port ${PORT}`)
|
||||
})
|
||||
```
|
||||
|
||||
```toml
|
||||
# railway.toml
|
||||
[build]
|
||||
builder = "nixpacks"
|
||||
buildCommand = "npm ci"
|
||||
|
||||
[deploy]
|
||||
startCommand = "node server.js"
|
||||
restartPolicyType = "always"
|
||||
restartPolicyMaxRetries = 3
|
||||
```
|
||||
|
||||
### Render
|
||||
|
||||
```javascript
|
||||
// server.js (same as Railway example above)
|
||||
// Use S3CompatibleStorage with your preferred object storage provider
|
||||
```
|
||||
|
||||
```yaml
|
||||
# render.yaml
|
||||
services:
|
||||
- type: web
|
||||
name: brainy-api
|
||||
runtime: node
|
||||
buildCommand: npm install
|
||||
startCommand: node server.js
|
||||
envVars:
|
||||
- key: S3_BUCKET
|
||||
value: brainy-data
|
||||
- key: S3_ENDPOINT
|
||||
value: s3.amazonaws.com
|
||||
- key: S3_REGION
|
||||
value: us-east-1
|
||||
- key: S3_ACCESS_KEY
|
||||
sync: false
|
||||
- key: S3_SECRET_KEY
|
||||
sync: false
|
||||
- key: API_KEY
|
||||
generateValue: true
|
||||
healthCheckPath: /health
|
||||
autoDeploy: true
|
||||
```
|
||||
|
||||
### Deno Deploy
|
||||
|
||||
```typescript
|
||||
// main.ts
|
||||
import { Brainy } from "npm:@soulcraft/brainy"
|
||||
import { S3CompatibleStorage } from "npm:@soulcraft/brainy/storage"
|
||||
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com",
|
||||
bucket: Deno.env.get("S3_BUCKET")!,
|
||||
accessKeyId: Deno.env.get("S3_ACCESS_KEY")!,
|
||||
secretAccessKey: Deno.env.get("S3_SECRET_KEY")!,
|
||||
region: "auto"
|
||||
})
|
||||
|
||||
const brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: { enabled: true }
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
const handler = apiAugmentation.createUniversalHandler()
|
||||
|
||||
Deno.serve(handler)
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
The API Server augmentation provides these REST endpoints:
|
||||
|
||||
- `POST /api/brainy/add` - Add entity
|
||||
- `GET /api/brainy/get?id=xxx` - Get entity by ID
|
||||
- `PUT /api/brainy/update` - Update entity
|
||||
- `DELETE /api/brainy/delete?id=xxx` - Delete entity
|
||||
- `POST /api/brainy/find` - Search/find entities
|
||||
- `POST /api/brainy/relate` - Create relationship
|
||||
- `GET /api/brainy/insights` - Get statistics and insights
|
||||
- `GET /health` - Health check
|
||||
|
||||
## WebSocket Support
|
||||
|
||||
The API Server augmentation includes WebSocket support for real-time updates through the `setupUniversalWebSocket()` method.
|
||||
|
||||
## MCP Support
|
||||
|
||||
Model Context Protocol (MCP) endpoints are available at `/mcp/*` for AI tool integration.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
# Storage Configuration (S3Compatible)
|
||||
S3_ENDPOINT=s3.amazonaws.com
|
||||
S3_REGION=us-east-1
|
||||
S3_BUCKET=brainy-data
|
||||
S3_ACCESS_KEY=xxx
|
||||
S3_SECRET_KEY=xxx
|
||||
|
||||
# API Configuration
|
||||
API_KEY=your-secret-key
|
||||
PORT=3000
|
||||
|
||||
# CORS
|
||||
CORS_ORIGIN=*
|
||||
|
||||
# Rate Limiting
|
||||
RATE_LIMIT_WINDOW=60000
|
||||
RATE_LIMIT_MAX=100
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```javascript
|
||||
// REST API Client
|
||||
const response = await fetch('https://your-api.com/api/brainy/add', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer YOUR_API_KEY'
|
||||
},
|
||||
body: JSON.stringify({
|
||||
data: 'Your content here',
|
||||
metadata: { type: 'document' }
|
||||
})
|
||||
})
|
||||
|
||||
const { id } = await response.json()
|
||||
|
||||
// Search
|
||||
const searchResponse = await fetch('https://your-api.com/api/brainy/find', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer YOUR_API_KEY'
|
||||
},
|
||||
body: JSON.stringify({
|
||||
query: 'neural networks'
|
||||
})
|
||||
})
|
||||
|
||||
const results = await searchResponse.json()
|
||||
|
||||
// WebSocket Client
|
||||
const ws = new WebSocket('wss://your-api.com/ws')
|
||||
ws.onopen = () => {
|
||||
ws.send(JSON.stringify({
|
||||
type: 'subscribe',
|
||||
pattern: 'technology'
|
||||
}))
|
||||
}
|
||||
ws.onmessage = (event) => {
|
||||
const update = JSON.parse(event.data)
|
||||
console.log('Real-time update:', update)
|
||||
}
|
||||
```
|
||||
|
||||
## Storage Adapter Configuration
|
||||
|
||||
S3CompatibleStorage constructor parameters (verified from source):
|
||||
|
||||
```javascript
|
||||
{
|
||||
endpoint: string, // Required (e.g., 's3.amazonaws.com')
|
||||
bucket: string, // Required
|
||||
accessKeyId: string, // Required
|
||||
secretAccessKey: string, // Required
|
||||
region?: string, // Optional (default: 'us-east-1')
|
||||
prefix?: string, // Optional (e.g., 'brainy/')
|
||||
forcePathStyle?: boolean, // Optional (needed for some S3-compatible services)
|
||||
}
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
|
||||
1. All examples use the **real** Brainy 3.0 APIs
|
||||
2. The `createUniversalHandler()` method is provided by the API Server augmentation
|
||||
3. S3CompatibleStorage works with any S3-compatible service
|
||||
4. Always call `brain.init()` before using Brainy
|
||||
5. The handler can be cached across requests for better performance
|
||||
6. R2Storage is an alias for S3CompatibleStorage (for Cloudflare R2)
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
1. **Always use environment variables for sensitive data** (API keys, secrets)
|
||||
2. **Enable authentication** in the API Server augmentation config
|
||||
3. **Use HTTPS/TLS** for all production deployments
|
||||
4. **Implement rate limiting** to prevent abuse
|
||||
5. **Configure CORS** appropriately for your use case
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
### Enable Lifecycle Policies
|
||||
|
||||
**AWS S3: Automatic tier transitions**
|
||||
```javascript
|
||||
// After initializing brain with S3CompatibleStorage
|
||||
const storage = brain.storage
|
||||
|
||||
// Set lifecycle policy for automatic archival
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-data',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' }, // $0.0125/GB after 30 days
|
||||
{ days: 90, storageClass: 'GLACIER' }, // $0.004/GB after 90 days
|
||||
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // $0.00099/GB after 1 year
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// Or enable Intelligent-Tiering for hands-off optimization
|
||||
await storage.enableIntelligentTiering('entities/', 'auto-optimize')
|
||||
```
|
||||
|
||||
**Cost Impact (500TB dataset):**
|
||||
| Storage Class | Cost/GB/Month | 500TB/Year | Savings |
|
||||
|---------------|---------------|------------|---------|
|
||||
| Standard | $0.023 | $138,000 | Baseline |
|
||||
| Intelligent-Tiering | Variable | $6,900 | **95%** |
|
||||
| With lifecycle policy | Variable | $5,940 | **96%** |
|
||||
|
||||
### Enable Batch Operations
|
||||
|
||||
**Efficient bulk deletions:**
|
||||
```javascript
|
||||
// Batch delete (1000 objects per request)
|
||||
const idsToDelete = [/* array of entity IDs */]
|
||||
const paths = idsToDelete.flatMap(id => [
|
||||
`entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
|
||||
`entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
|
||||
])
|
||||
|
||||
await storage.batchDelete(paths) // Much faster than individual deletes
|
||||
```
|
||||
|
||||
### Enable Compression (FileSystem)
|
||||
|
||||
**For local/server deployments:**
|
||||
```javascript
|
||||
const storage = new FileSystemStorage({
|
||||
path: './data',
|
||||
compression: true // 60-80% space savings
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Cache the brain instance** - Initialize once and reuse across requests
|
||||
2. **Use S3CompatibleStorage** for cloud deployments (better scalability)
|
||||
3. **Enable the cache augmentation** for frequently accessed data
|
||||
4. **Configure appropriate memory limits** for your runtime
|
||||
5. Enable lifecycle policies to reduce storage costs by 96%
|
||||
6. Use batch operations for cleanup tasks
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **"Brainy not initialized"** - Make sure to call `await brain.init()` before use
|
||||
2. **"augmentationRegistry.getAugmentation is not a function"** - The augmentation wasn't loaded properly
|
||||
3. **S3 Access Denied** - Check your IAM permissions and credentials
|
||||
4. **CORS errors** - Configure the CORS settings in the API Server augmentation
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable debug logging by setting:
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage,
|
||||
debug: true,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
verbose: true
|
||||
}
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://github.com/soulcraft/brainy/docs
|
||||
- Issues: https://github.com/soulcraft/brainy/issues
|
||||
- NPM Package: https://www.npmjs.com/package/@soulcraft/brainy
|
||||
|
||||
---
|
||||
|
||||
This guide contains only verified, working code from the actual Brainy 3.0 codebase. No mock implementations, no stub methods, only production-ready code.
|
||||
505
docs/deployment/aws-deployment.md
Normal file
505
docs/deployment/aws-deployment.md
Normal file
|
|
@ -0,0 +1,505 @@
|
|||
# AWS Deployment Guide for Brainy
|
||||
|
||||
## Overview
|
||||
Deploy Brainy on AWS with automatic scaling, high availability, and zero-config dynamic adaptation. Brainy automatically adapts to your AWS environment without manual configuration.
|
||||
|
||||
## Quick Start (Zero-Config)
|
||||
|
||||
### Option 1: AWS Lambda (Serverless)
|
||||
|
||||
```bash
|
||||
# Install Brainy
|
||||
npm install @soulcraft/brainy
|
||||
|
||||
# Create handler.js
|
||||
cat > handler.js << 'EOF'
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
|
||||
let brain
|
||||
|
||||
exports.handler = async (event) => {
|
||||
// Brainy auto-detects Lambda environment and configures accordingly
|
||||
if (!brain) {
|
||||
brain = new Brainy() // Zero config - auto-adapts to Lambda
|
||||
await brain.init()
|
||||
}
|
||||
|
||||
const { method, ...params } = JSON.parse(event.body)
|
||||
|
||||
switch(method) {
|
||||
case 'add':
|
||||
const id = await brain.add(params)
|
||||
return { statusCode: 200, body: JSON.stringify({ id }) }
|
||||
case 'find':
|
||||
const results = await brain.find(params)
|
||||
return { statusCode: 200, body: JSON.stringify({ results }) }
|
||||
default:
|
||||
return { statusCode: 400, body: 'Unknown method' }
|
||||
}
|
||||
}
|
||||
EOF
|
||||
|
||||
# Deploy with AWS SAM
|
||||
sam init --runtime nodejs20.x --name brainy-app
|
||||
sam deploy --guided
|
||||
```
|
||||
|
||||
### Option 2: ECS Fargate (Container)
|
||||
|
||||
```bash
|
||||
# Build and push Docker image
|
||||
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ECR_URI
|
||||
docker build -t brainy .
|
||||
docker tag brainy:latest $ECR_URI/brainy:latest
|
||||
docker push $ECR_URI/brainy:latest
|
||||
|
||||
# Deploy with minimal ECS task definition
|
||||
cat > task-definition.json << 'EOF'
|
||||
{
|
||||
"family": "brainy",
|
||||
"networkMode": "awsvpc",
|
||||
"requiresCompatibilities": ["FARGATE"],
|
||||
"cpu": "256",
|
||||
"memory": "512",
|
||||
"containerDefinitions": [{
|
||||
"name": "brainy",
|
||||
"image": "$ECR_URI/brainy:latest",
|
||||
"environment": [
|
||||
{"name": "NODE_ENV", "value": "production"}
|
||||
],
|
||||
"logConfiguration": {
|
||||
"logDriver": "awslogs",
|
||||
"options": {
|
||||
"awslogs-group": "/ecs/brainy",
|
||||
"awslogs-region": "us-east-1",
|
||||
"awslogs-stream-prefix": "ecs"
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
EOF
|
||||
|
||||
# Brainy auto-detects ECS environment and uses S3 for storage
|
||||
aws ecs register-task-definition --cli-input-json file://task-definition.json
|
||||
aws ecs create-service --cluster default --service-name brainy --task-definition brainy --desired-count 2
|
||||
```
|
||||
|
||||
### Option 3: EC2 Auto-Scaling
|
||||
|
||||
```bash
|
||||
# User data script for EC2 instances
|
||||
#!/bin/bash
|
||||
curl -fsSL https://rpm.nodesource.com/setup_20.x | sudo bash -
|
||||
sudo yum install -y nodejs git
|
||||
|
||||
# Clone and setup (or use your deployment method)
|
||||
git clone https://github.com/yourorg/brainy-app.git /app
|
||||
cd /app
|
||||
npm install --production
|
||||
|
||||
# Create systemd service
|
||||
cat > /etc/systemd/system/brainy.service << 'EOF'
|
||||
[Unit]
|
||||
Description=Brainy
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=ec2-user
|
||||
WorkingDirectory=/app
|
||||
ExecStart=/usr/bin/node index.js
|
||||
Restart=on-failure
|
||||
Environment=NODE_ENV=production
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
systemctl start brainy
|
||||
systemctl enable brainy
|
||||
```
|
||||
|
||||
## Zero-Config Storage (Automatic)
|
||||
|
||||
Brainy automatically detects and uses the best available storage:
|
||||
|
||||
```javascript
|
||||
// No configuration needed - Brainy auto-detects:
|
||||
const brain = new Brainy()
|
||||
|
||||
// Auto-detection priority:
|
||||
// 1. S3 (if IAM role has permissions)
|
||||
// 2. EFS (if mounted at /mnt/efs)
|
||||
// 3. EBS volume (if available)
|
||||
// 4. Instance storage (fallback)
|
||||
```
|
||||
|
||||
### Manual S3 Configuration (Optional)
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: {
|
||||
bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket
|
||||
region: process.env.AWS_REGION || 'auto', // 'auto' detects region
|
||||
// IAM role provides credentials automatically
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Scaling Strategies
|
||||
|
||||
### 1. Horizontal Scaling (Recommended)
|
||||
|
||||
```yaml
|
||||
# Auto-scaling policy
|
||||
Resources:
|
||||
AutoScalingTarget:
|
||||
Type: AWS::ApplicationAutoScaling::ScalableTarget
|
||||
Properties:
|
||||
ServiceNamespace: ecs
|
||||
ResourceId: service/default/brainy
|
||||
ScalableDimension: ecs:service:DesiredCount
|
||||
MinCapacity: 2
|
||||
MaxCapacity: 100
|
||||
|
||||
AutoScalingPolicy:
|
||||
Type: AWS::ApplicationAutoScaling::ScalingPolicy
|
||||
Properties:
|
||||
PolicyType: TargetTrackingScaling
|
||||
TargetTrackingScalingPolicyConfiguration:
|
||||
TargetValue: 70.0
|
||||
PredefinedMetricSpecification:
|
||||
PredefinedMetricType: ECSServiceAverageCPUUtilization
|
||||
```
|
||||
|
||||
### 2. Vertical Scaling
|
||||
|
||||
Brainy automatically adapts to available memory:
|
||||
- **256MB**: Minimal mode, optimized caching
|
||||
- **512MB**: Standard mode, balanced performance
|
||||
- **1GB+**: Full mode, maximum performance
|
||||
|
||||
## High Availability Setup
|
||||
|
||||
### Multi-AZ Deployment
|
||||
|
||||
```javascript
|
||||
// Brainy automatically handles multi-AZ with S3
|
||||
const brain = new Brainy({
|
||||
distributed: {
|
||||
enabled: true, // Auto-enables with S3 storage
|
||||
coordinationMethod: 'auto' // Uses S3 for coordination
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Load Balancing
|
||||
|
||||
```bash
|
||||
# Application Load Balancer with health checks
|
||||
aws elbv2 create-load-balancer \
|
||||
--name brainy-alb \
|
||||
--subnets subnet-xxx subnet-yyy \
|
||||
--security-groups sg-xxx
|
||||
|
||||
aws elbv2 create-target-group \
|
||||
--name brainy-targets \
|
||||
--protocol HTTP \
|
||||
--port 3000 \
|
||||
--vpc-id vpc-xxx \
|
||||
--health-check-path /health \
|
||||
--health-check-interval-seconds 30
|
||||
```
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### CloudWatch Integration
|
||||
|
||||
Brainy automatically sends metrics when running on AWS:
|
||||
|
||||
```javascript
|
||||
// Automatic CloudWatch metrics (no config needed)
|
||||
// - Request count
|
||||
// - Response time
|
||||
// - Error rate
|
||||
// - Storage usage
|
||||
// - Memory usage
|
||||
```
|
||||
|
||||
### Custom Metrics
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
monitoring: {
|
||||
enabled: true,
|
||||
customMetrics: {
|
||||
namespace: 'Brainy/Production',
|
||||
dimensions: [
|
||||
{ Name: 'Environment', Value: 'production' },
|
||||
{ Name: 'Service', Value: 'api' }
|
||||
]
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. IAM Role (Recommended)
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:DeleteObject",
|
||||
"s3:ListBucket"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::brainy-*/*",
|
||||
"arn:aws:s3:::brainy-*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. VPC Configuration
|
||||
|
||||
```bash
|
||||
# Private subnets with NAT Gateway
|
||||
aws ec2 create-vpc --cidr-block 10.0.0.0/16
|
||||
aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.1.0/24 --availability-zone us-east-1a
|
||||
aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.2.0/24 --availability-zone us-east-1b
|
||||
```
|
||||
|
||||
### 3. Encryption
|
||||
|
||||
```javascript
|
||||
// Automatic encryption with S3
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: {
|
||||
encryption: 'auto' // Uses S3 SSE-S3 by default
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
### 1. Spot Instances (70% savings)
|
||||
|
||||
```bash
|
||||
aws ec2 request-spot-fleet --spot-fleet-request-config '{
|
||||
"IamFleetRole": "arn:aws:iam::xxx:role/fleet-role",
|
||||
"TargetCapacity": 2,
|
||||
"SpotPrice": "0.05",
|
||||
"LaunchSpecifications": [{
|
||||
"ImageId": "ami-xxx",
|
||||
"InstanceType": "t3.medium",
|
||||
"UserData": "BASE64_ENCODED_STARTUP_SCRIPT"
|
||||
}]
|
||||
}'
|
||||
```
|
||||
|
||||
### 2. S3 Intelligent-Tiering
|
||||
|
||||
```javascript
|
||||
// Brainy automatically uses S3 Intelligent-Tiering
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: {
|
||||
storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Lambda Reserved Concurrency
|
||||
|
||||
```bash
|
||||
aws lambda put-function-concurrency \
|
||||
--function-name brainy-handler \
|
||||
--reserved-concurrent-executions 10
|
||||
```
|
||||
|
||||
## Deployment Automation
|
||||
|
||||
### GitHub Actions CI/CD
|
||||
|
||||
```yaml
|
||||
name: Deploy to AWS
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Configure AWS credentials
|
||||
uses: aws-actions/configure-aws-credentials@v1
|
||||
with:
|
||||
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
aws-region: us-east-1
|
||||
|
||||
- name: Deploy to ECS
|
||||
run: |
|
||||
docker build -t brainy .
|
||||
docker tag brainy:latest $ECR_URI/brainy:latest
|
||||
docker push $ECR_URI/brainy:latest
|
||||
aws ecs update-service --cluster default --service brainy --force-new-deployment
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Storage Auto-Detection Fails**
|
||||
```javascript
|
||||
// Explicitly specify storage
|
||||
const brain = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'my-bucket' } }
|
||||
})
|
||||
```
|
||||
|
||||
2. **Memory Issues**
|
||||
```javascript
|
||||
// Optimize for low memory
|
||||
const brain = new Brainy({
|
||||
cache: { maxSize: 100 }, // Reduce cache size
|
||||
index: { M: 8 } // Reduce HNSW connections
|
||||
})
|
||||
```
|
||||
|
||||
3. **Cold Starts (Lambda)**
|
||||
|
||||
**Progressive Initialization (Zero-Config)**
|
||||
|
||||
Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME)
|
||||
and uses progressive initialization for <200ms cold starts:
|
||||
|
||||
```javascript
|
||||
// Zero-config - Brainy auto-detects Lambda
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
s3Storage: {
|
||||
bucketName: 'my-bucket',
|
||||
region: 'us-east-1',
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
})
|
||||
await brain.init() // Returns in <200ms
|
||||
|
||||
// First write validates bucket (lazy validation)
|
||||
await brain.add('noun', { name: 'test' }) // Validates here
|
||||
```
|
||||
|
||||
**Manual Override (if needed)**
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
s3Storage: {
|
||||
bucketName: 'my-bucket',
|
||||
// Force specific mode
|
||||
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
| Mode | Cold Start | Best For |
|
||||
|------|------------|----------|
|
||||
| `auto` (default) | <200ms in Lambda | Zero-config, auto-detects |
|
||||
| `progressive` | <200ms always | Force fast init everywhere |
|
||||
| `strict` | 100-500ms+ | Local dev, tests, debugging |
|
||||
|
||||
**Warm-up (Alternative)**
|
||||
```javascript
|
||||
exports.warmup = async () => {
|
||||
if (!brain) {
|
||||
brain = new Brainy({ warmup: true })
|
||||
await brain.init()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Readiness Detection**
|
||||
|
||||
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
|
||||
|
||||
```javascript
|
||||
let brain
|
||||
|
||||
exports.handler = async (event) => {
|
||||
if (!brain) {
|
||||
brain = new Brainy({ storage: { type: 's3', ... } })
|
||||
brain.init() // Fire and forget
|
||||
}
|
||||
|
||||
// Wait for initialization to complete
|
||||
await brain.ready
|
||||
|
||||
// Now safe to use brain methods
|
||||
const results = await brain.find({ query: event.queryStringParameters.q })
|
||||
return { statusCode: 200, body: JSON.stringify(results) }
|
||||
}
|
||||
```
|
||||
|
||||
For health checks, use `isFullyInitialized()` to verify all background tasks are complete:
|
||||
|
||||
```javascript
|
||||
exports.healthCheck = async () => {
|
||||
try {
|
||||
await brain.ready
|
||||
return {
|
||||
statusCode: 200,
|
||||
body: JSON.stringify({
|
||||
status: 'ready',
|
||||
fullyInitialized: brain.isFullyInitialized()
|
||||
})
|
||||
}
|
||||
} catch (error) {
|
||||
return {
|
||||
statusCode: 503,
|
||||
body: JSON.stringify({ status: 'initializing' })
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Production Checklist
|
||||
|
||||
- [ ] IAM roles configured with minimal permissions
|
||||
- [ ] VPC with private subnets
|
||||
- [ ] Auto-scaling configured
|
||||
- [ ] CloudWatch alarms set up
|
||||
- [ ] Backup strategy (S3 versioning enabled)
|
||||
- [ ] SSL/TLS certificates configured
|
||||
- [ ] Rate limiting enabled
|
||||
- [ ] Health checks configured
|
||||
- [ ] Monitoring dashboard created
|
||||
- [ ] Cost alerts configured
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://brainy.soulcraft.ai/docs
|
||||
- Issues: https://github.com/soulcraft/brainy/issues
|
||||
- Community: https://discord.gg/brainy
|
||||
627
docs/deployment/gcp-deployment.md
Normal file
627
docs/deployment/gcp-deployment.md
Normal file
|
|
@ -0,0 +1,627 @@
|
|||
# Google Cloud Platform Deployment Guide for Brainy
|
||||
|
||||
## Overview
|
||||
Deploy Brainy on GCP with automatic scaling, global distribution, and zero-config dynamic adaptation. Brainy automatically detects and optimizes for GCP services.
|
||||
|
||||
## Quick Start (Zero-Config)
|
||||
|
||||
### Option 1: Cloud Run (Serverless Containers)
|
||||
|
||||
```bash
|
||||
# Build and deploy with one command
|
||||
gcloud run deploy brainy \
|
||||
--source . \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated
|
||||
|
||||
# Brainy auto-detects Cloud Run and configures:
|
||||
# - Memory-optimized caching
|
||||
# - GCS for storage (if available)
|
||||
# - Cloud SQL for metadata (if available)
|
||||
```
|
||||
|
||||
### Option 2: Cloud Functions (Serverless)
|
||||
|
||||
```javascript
|
||||
// index.js
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
|
||||
let brain
|
||||
|
||||
exports.brainyHandler = async (req, res) => {
|
||||
// Zero-config - auto-adapts to Cloud Functions
|
||||
if (!brain) {
|
||||
brain = new Brainy() // Detects GCP environment automatically
|
||||
await brain.init()
|
||||
}
|
||||
|
||||
const { method, ...params } = req.body
|
||||
|
||||
try {
|
||||
let result
|
||||
switch(method) {
|
||||
case 'add':
|
||||
result = await brain.add(params)
|
||||
break
|
||||
case 'find':
|
||||
result = await brain.find(params)
|
||||
break
|
||||
case 'relate':
|
||||
result = await brain.relate(params)
|
||||
break
|
||||
default:
|
||||
return res.status(400).json({ error: 'Unknown method' })
|
||||
}
|
||||
res.json({ result })
|
||||
} catch (error) {
|
||||
res.status(500).json({ error: error.message })
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Deploy:
|
||||
```bash
|
||||
gcloud functions deploy brainy \
|
||||
--runtime nodejs20 \
|
||||
--trigger-http \
|
||||
--entry-point brainyHandler \
|
||||
--memory 512MB \
|
||||
--timeout 60s
|
||||
```
|
||||
|
||||
### Option 3: Google Kubernetes Engine (GKE)
|
||||
|
||||
```bash
|
||||
# Create autopilot cluster (fully managed, zero-config)
|
||||
gcloud container clusters create-auto brainy-cluster \
|
||||
--region us-central1
|
||||
|
||||
# Deploy using Cloud Build
|
||||
gcloud builds submit --tag gcr.io/$PROJECT_ID/brainy
|
||||
|
||||
# Apply Kubernetes manifest
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: brainy
|
||||
spec:
|
||||
containers:
|
||||
- name: brainy
|
||||
image: gcr.io/$PROJECT_ID/brainy
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "250m"
|
||||
env:
|
||||
- name: NODE_ENV
|
||||
value: production
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: brainy-service
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
selector:
|
||||
app: brainy
|
||||
ports:
|
||||
- port: 80
|
||||
targetPort: 3000
|
||||
EOF
|
||||
```
|
||||
|
||||
## Zero-Config Storage (Automatic)
|
||||
|
||||
Brainy automatically detects and uses the best GCP storage:
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy()
|
||||
// Auto-detection priority:
|
||||
// 1. Firestore (if available)
|
||||
// 2. Cloud Storage (GCS)
|
||||
// 3. Cloud SQL
|
||||
// 4. Persistent Disk
|
||||
// 5. Memory (fallback)
|
||||
```
|
||||
|
||||
### Cloud Storage Configuration (Optional)
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3', // GCS is S3-compatible
|
||||
options: {
|
||||
endpoint: 'https://storage.googleapis.com',
|
||||
bucket: process.env.GCS_BUCKET || 'auto', // Auto-creates bucket
|
||||
// Uses Application Default Credentials automatically
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Firestore Integration (Optional)
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'firestore',
|
||||
options: {
|
||||
projectId: process.env.GCP_PROJECT || 'auto',
|
||||
collection: 'brainy-data'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Scaling Strategies
|
||||
|
||||
### 1. Cloud Run Auto-scaling
|
||||
|
||||
```yaml
|
||||
# service.yaml
|
||||
apiVersion: serving.knative.dev/v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: brainy
|
||||
annotations:
|
||||
run.googleapis.com/execution-environment: gen2
|
||||
spec:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
autoscaling.knative.dev/minScale: "1"
|
||||
autoscaling.knative.dev/maxScale: "1000"
|
||||
autoscaling.knative.dev/target: "80"
|
||||
spec:
|
||||
containerConcurrency: 100
|
||||
containers:
|
||||
- image: gcr.io/PROJECT_ID/brainy
|
||||
resources:
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: "2Gi"
|
||||
```
|
||||
|
||||
### 2. GKE Horizontal Pod Autoscaling
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: brainy-hpa
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: brainy
|
||||
minReplicas: 3
|
||||
maxReplicas: 100
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80
|
||||
```
|
||||
|
||||
## Global Distribution
|
||||
|
||||
### Multi-Region Setup
|
||||
|
||||
```javascript
|
||||
// Brainy automatically handles multi-region with GCS
|
||||
const brain = new Brainy({
|
||||
distributed: {
|
||||
enabled: true,
|
||||
regions: ['us-central1', 'europe-west1', 'asia-northeast1'],
|
||||
replication: 'auto' // Automatic cross-region replication
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Traffic Director Configuration
|
||||
|
||||
```bash
|
||||
# Global load balancing with Traffic Director
|
||||
gcloud compute backend-services create brainy-global \
|
||||
--global \
|
||||
--load-balancing-scheme=INTERNAL_SELF_MANAGED \
|
||||
--protocol=HTTP2
|
||||
|
||||
gcloud compute backend-services add-backend brainy-global \
|
||||
--global \
|
||||
--network-endpoint-group=brainy-neg \
|
||||
--network-endpoint-group-region=us-central1
|
||||
```
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### Cloud Monitoring (Automatic)
|
||||
|
||||
Brainy automatically sends metrics to Cloud Monitoring:
|
||||
|
||||
```javascript
|
||||
// No configuration needed - automatic when running on GCP
|
||||
const brain = new Brainy()
|
||||
|
||||
// Automatic metrics:
|
||||
// - Request latency
|
||||
// - Error rate
|
||||
// - Storage operations
|
||||
// - Cache hit rate
|
||||
// - Memory usage
|
||||
```
|
||||
|
||||
### Custom Metrics
|
||||
|
||||
```javascript
|
||||
const { Monitoring } = require('@google-cloud/monitoring')
|
||||
const monitoring = new Monitoring.MetricServiceClient()
|
||||
|
||||
const brain = new Brainy({
|
||||
onMetric: async (metric) => {
|
||||
// Send custom metrics to Cloud Monitoring
|
||||
await monitoring.createTimeSeries({
|
||||
name: monitoring.projectPath(projectId),
|
||||
timeSeries: [{
|
||||
metric: {
|
||||
type: `custom.googleapis.com/brainy/${metric.name}`,
|
||||
labels: metric.labels
|
||||
},
|
||||
points: [{
|
||||
interval: { endTime: { seconds: Date.now() / 1000 } },
|
||||
value: { doubleValue: metric.value }
|
||||
}]
|
||||
}]
|
||||
})
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Cloud Trace Integration
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
tracing: {
|
||||
enabled: true,
|
||||
sampleRate: 0.1 // Sample 10% of requests
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Workload Identity (GKE)
|
||||
|
||||
```yaml
|
||||
# Enable Workload Identity
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: brainy-sa
|
||||
annotations:
|
||||
iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com
|
||||
```
|
||||
|
||||
### 2. Binary Authorization
|
||||
|
||||
```yaml
|
||||
# Ensure only signed container images
|
||||
apiVersion: binaryauthorization.grafeas.io/v1beta1
|
||||
kind: Policy
|
||||
metadata:
|
||||
name: brainy-policy
|
||||
spec:
|
||||
defaultAdmissionRule:
|
||||
requireAttestationsBy:
|
||||
- projects/PROJECT_ID/attestors/prod-attestor
|
||||
```
|
||||
|
||||
### 3. VPC Service Controls
|
||||
|
||||
```bash
|
||||
# Create VPC Service Perimeter
|
||||
gcloud access-context-manager perimeters create brainy_perimeter \
|
||||
--resources=projects/PROJECT_NUMBER \
|
||||
--restricted-services=storage.googleapis.com \
|
||||
--title="Brainy Security Perimeter"
|
||||
```
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
### 1. Preemptible VMs (80% savings)
|
||||
|
||||
```yaml
|
||||
# GKE node pool with preemptible VMs
|
||||
apiVersion: container.cnrm.cloud.google.com/v1beta1
|
||||
kind: ContainerNodePool
|
||||
metadata:
|
||||
name: brainy-preemptible-pool
|
||||
spec:
|
||||
clusterRef:
|
||||
name: brainy-cluster
|
||||
config:
|
||||
preemptible: true
|
||||
machineType: n2-standard-2
|
||||
autoscaling:
|
||||
minNodeCount: 1
|
||||
maxNodeCount: 10
|
||||
```
|
||||
|
||||
### 2. Cloud CDN for Static Assets
|
||||
|
||||
```bash
|
||||
# Enable Cloud CDN for frequently accessed data
|
||||
gcloud compute backend-buckets create brainy-assets \
|
||||
--gcs-bucket-name=brainy-static
|
||||
|
||||
gcloud compute backend-buckets update brainy-assets \
|
||||
--enable-cdn \
|
||||
--cache-mode=CACHE_ALL_STATIC
|
||||
```
|
||||
|
||||
### 3. Committed Use Discounts
|
||||
|
||||
```bash
|
||||
# Purchase committed use for predictable workloads
|
||||
gcloud compute commitments create brainy-commitment \
|
||||
--plan=TWELVE_MONTH \
|
||||
--resources=vcpu=100,memory=400
|
||||
```
|
||||
|
||||
## Deployment Automation
|
||||
|
||||
### Cloud Build CI/CD
|
||||
|
||||
```yaml
|
||||
# cloudbuild.yaml
|
||||
steps:
|
||||
# Build container
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.']
|
||||
|
||||
# Push to registry
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA']
|
||||
|
||||
# Deploy to Cloud Run
|
||||
- name: 'gcr.io/cloud-builders/gcloud'
|
||||
args:
|
||||
- 'run'
|
||||
- 'deploy'
|
||||
- 'brainy'
|
||||
- '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'
|
||||
- '--region=us-central1'
|
||||
- '--platform=managed'
|
||||
|
||||
# Trigger on push to main
|
||||
trigger:
|
||||
branch:
|
||||
name: main
|
||||
```
|
||||
|
||||
### Terraform Infrastructure
|
||||
|
||||
```hcl
|
||||
# main.tf
|
||||
resource "google_cloud_run_service" "brainy" {
|
||||
name = "brainy"
|
||||
location = "us-central1"
|
||||
|
||||
template {
|
||||
spec {
|
||||
containers {
|
||||
image = "gcr.io/${var.project_id}/brainy"
|
||||
|
||||
resources {
|
||||
limits = {
|
||||
cpu = "2"
|
||||
memory = "2Gi"
|
||||
}
|
||||
}
|
||||
|
||||
env {
|
||||
name = "NODE_ENV"
|
||||
value = "production"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
traffic {
|
||||
percent = 100
|
||||
latest_revision = true
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_cloud_run_service_iam_member" "public" {
|
||||
service = google_cloud_run_service.brainy.name
|
||||
location = google_cloud_run_service.brainy.location
|
||||
role = "roles/run.invoker"
|
||||
member = "allUsers"
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### 1. Memory Store (Redis Compatible)
|
||||
|
||||
```javascript
|
||||
// Brainy can use Memorystore for caching
|
||||
const brain = new Brainy({
|
||||
cache: {
|
||||
type: 'redis',
|
||||
options: {
|
||||
host: process.env.REDIS_HOST || 'auto-detect',
|
||||
port: 6379
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### 2. Cloud Spanner for Global Consistency
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
metadata: {
|
||||
type: 'spanner',
|
||||
options: {
|
||||
instance: 'brainy-instance',
|
||||
database: 'brainy-db'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Quota Exceeded**
|
||||
```bash
|
||||
# Check quotas
|
||||
gcloud compute project-info describe --project=$PROJECT_ID
|
||||
|
||||
# Request increase
|
||||
gcloud compute project-info add-metadata \
|
||||
--metadata google-compute-default-region=us-central1
|
||||
```
|
||||
|
||||
2. **Cold Starts**
|
||||
|
||||
**Progressive Initialization (Zero-Config)**
|
||||
|
||||
Brainy automatically detects Cloud Run and Cloud Functions environments
|
||||
and uses progressive initialization for <200ms cold starts:
|
||||
|
||||
```javascript
|
||||
// Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var)
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsNativeStorage: { bucketName: 'my-bucket' }
|
||||
}
|
||||
})
|
||||
await brain.init() // Returns in <200ms
|
||||
|
||||
// First write validates bucket (lazy validation)
|
||||
await brain.add('noun', { name: 'test' }) // Validates here
|
||||
```
|
||||
|
||||
**Manual Override (if needed)**
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-bucket',
|
||||
// Force specific mode
|
||||
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
| Mode | Cold Start | Best For |
|
||||
|------|------------|----------|
|
||||
| `auto` (default) | <200ms in cloud | Zero-config, auto-detects |
|
||||
| `progressive` | <200ms always | Force fast init everywhere |
|
||||
| `strict` | 100-500ms+ | Local dev, tests, debugging |
|
||||
|
||||
**Keep Warm (Alternative)**
|
||||
```javascript
|
||||
// Keep minimum instances warm
|
||||
const brain = new Brainy({
|
||||
warmup: {
|
||||
enabled: true,
|
||||
interval: 60000 // Ping every minute
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Readiness Detection**
|
||||
|
||||
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
|
||||
|
||||
```javascript
|
||||
let brain
|
||||
|
||||
async function handleRequest(req, res) {
|
||||
if (!brain) {
|
||||
brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||
brain.init() // Fire and forget
|
||||
}
|
||||
|
||||
// Wait for initialization to complete
|
||||
await brain.ready
|
||||
|
||||
// Now safe to use brain methods
|
||||
const results = await brain.find({ query: req.query.q })
|
||||
res.json(results)
|
||||
}
|
||||
```
|
||||
|
||||
For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete:
|
||||
|
||||
```javascript
|
||||
// Health check endpoint for Cloud Run
|
||||
app.get('/health', async (req, res) => {
|
||||
try {
|
||||
await brain.ready
|
||||
res.json({
|
||||
status: 'ready',
|
||||
fullyInitialized: brain.isFullyInitialized()
|
||||
})
|
||||
} catch (error) {
|
||||
res.status(503).json({ status: 'initializing' })
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
3. **Memory Pressure**
|
||||
```javascript
|
||||
// Optimize for GCP memory constraints
|
||||
const brain = new Brainy({
|
||||
memory: {
|
||||
mode: 'aggressive', // Aggressive garbage collection
|
||||
maxHeap: 0.8 // Use 80% of available memory
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Production Checklist
|
||||
|
||||
- [ ] Enable Workload Identity for secure access
|
||||
- [ ] Configure Cloud Armor for DDoS protection
|
||||
- [ ] Set up Cloud KMS for encryption keys
|
||||
- [ ] Enable VPC Service Controls
|
||||
- [ ] Configure Cloud IAP for authentication
|
||||
- [ ] Set up Cloud Monitoring dashboards
|
||||
- [ ] Configure Error Reporting
|
||||
- [ ] Enable Cloud Trace
|
||||
- [ ] Set up budget alerts
|
||||
- [ ] Configure backup and disaster recovery
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://brainy.soulcraft.ai/docs
|
||||
- Issues: https://github.com/soulcraft/brainy/issues
|
||||
- GCP Marketplace: https://console.cloud.google.com/marketplace/product/brainy
|
||||
727
docs/deployment/kubernetes-deployment.md
Normal file
727
docs/deployment/kubernetes-deployment.md
Normal file
|
|
@ -0,0 +1,727 @@
|
|||
# Kubernetes Deployment Guide for Brainy
|
||||
|
||||
## Overview
|
||||
Deploy Brainy on Kubernetes with automatic scaling, high availability, and zero-config dynamic adaptation. Works with any Kubernetes distribution (vanilla, EKS, GKE, AKS, OpenShift, etc.).
|
||||
|
||||
## Quick Start (Zero-Config)
|
||||
|
||||
### Basic Deployment
|
||||
|
||||
```yaml
|
||||
# brainy-deployment.yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
labels:
|
||||
app: brainy
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: brainy
|
||||
spec:
|
||||
containers:
|
||||
- name: brainy
|
||||
image: soulcraft/brainy:latest
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
env:
|
||||
- name: NODE_ENV
|
||||
value: production
|
||||
# Brainy auto-detects Kubernetes and configures accordingly
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "500m"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: brainy-service
|
||||
spec:
|
||||
selector:
|
||||
app: brainy
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 80
|
||||
targetPort: 3000
|
||||
type: LoadBalancer
|
||||
```
|
||||
|
||||
Deploy:
|
||||
```bash
|
||||
kubectl apply -f brainy-deployment.yaml
|
||||
```
|
||||
|
||||
## Production-Grade Setup
|
||||
|
||||
### 1. StatefulSet with Persistent Storage
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: brainy-storage
|
||||
provisioner: kubernetes.io/aws-ebs # Or your cloud provider
|
||||
parameters:
|
||||
type: gp3
|
||||
iopsPerGB: "10"
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
serviceName: brainy-headless
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: brainy
|
||||
spec:
|
||||
containers:
|
||||
- name: brainy
|
||||
image: soulcraft/brainy:latest
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
env:
|
||||
- name: NODE_ENV
|
||||
value: production
|
||||
- name: BRAINY_STORAGE_TYPE
|
||||
value: filesystem
|
||||
- name: BRAINY_STORAGE_PATH
|
||||
value: /data
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 3000
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 3000
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: brainy-storage
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
```
|
||||
|
||||
### 2. Horizontal Pod Autoscaler
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: brainy-hpa
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: brainy
|
||||
minReplicas: 2
|
||||
maxReplicas: 100
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 60
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 100
|
||||
periodSeconds: 60
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 10
|
||||
periodSeconds: 60
|
||||
```
|
||||
|
||||
### 3. Ingress Configuration
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: brainy-ingress
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: nginx
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
nginx.ingress.kubernetes.io/rate-limit: "100"
|
||||
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
|
||||
spec:
|
||||
tls:
|
||||
- hosts:
|
||||
- api.brainy.example.com
|
||||
secretName: brainy-tls
|
||||
rules:
|
||||
- host: api.brainy.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: brainy-service
|
||||
port:
|
||||
number: 80
|
||||
```
|
||||
|
||||
## Zero-Config Storage Options
|
||||
|
||||
### Option 1: S3-Compatible Storage (Recommended)
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: brainy-s3-credentials
|
||||
type: Opaque
|
||||
data:
|
||||
access-key: <base64-encoded-key>
|
||||
secret-key: <base64-encoded-secret>
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: brainy
|
||||
env:
|
||||
- name: BRAINY_STORAGE_TYPE
|
||||
value: s3
|
||||
- name: AWS_ACCESS_KEY_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: brainy-s3-credentials
|
||||
key: access-key
|
||||
- name: AWS_SECRET_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: brainy-s3-credentials
|
||||
key: secret-key
|
||||
- name: S3_BUCKET
|
||||
value: brainy-data
|
||||
- name: AWS_REGION
|
||||
value: us-east-1
|
||||
```
|
||||
|
||||
### Option 2: MinIO (Self-Hosted S3)
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: minio
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: minio
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: minio
|
||||
spec:
|
||||
containers:
|
||||
- name: minio
|
||||
image: minio/minio:latest
|
||||
args:
|
||||
- server
|
||||
- /data
|
||||
env:
|
||||
- name: MINIO_ROOT_USER
|
||||
value: brainy
|
||||
- name: MINIO_ROOT_PASSWORD
|
||||
value: brainy123456
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: minio-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: minio-service
|
||||
spec:
|
||||
selector:
|
||||
app: minio
|
||||
ports:
|
||||
- port: 9000
|
||||
targetPort: 9000
|
||||
```
|
||||
|
||||
### Option 3: Shared NFS Storage
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: brainy-nfs-pv
|
||||
spec:
|
||||
capacity:
|
||||
storage: 100Gi
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
nfs:
|
||||
server: nfs-server.example.com
|
||||
path: /export/brainy
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: brainy-nfs-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
resources:
|
||||
requests:
|
||||
storage: 100Gi
|
||||
```
|
||||
|
||||
## High Availability Configuration
|
||||
|
||||
### 1. Pod Anti-Affinity
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
- labelSelector:
|
||||
matchExpressions:
|
||||
- key: app
|
||||
operator: In
|
||||
values:
|
||||
- brainy
|
||||
topologyKey: kubernetes.io/hostname
|
||||
```
|
||||
|
||||
### 2. Pod Disruption Budget
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: brainy-pdb
|
||||
spec:
|
||||
minAvailable: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
```
|
||||
|
||||
### 3. Multi-Zone Deployment
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: topology.kubernetes.io/zone
|
||||
whenUnsatisfiable: DoNotSchedule
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
```
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### 1. Prometheus Metrics
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: brainy-metrics
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9090"
|
||||
prometheus.io/path: "/metrics"
|
||||
spec:
|
||||
selector:
|
||||
app: brainy
|
||||
ports:
|
||||
- name: metrics
|
||||
port: 9090
|
||||
targetPort: 9090
|
||||
```
|
||||
|
||||
### 2. Grafana Dashboard
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: brainy-dashboard
|
||||
data:
|
||||
dashboard.json: |
|
||||
{
|
||||
"dashboard": {
|
||||
"title": "Brainy Metrics",
|
||||
"panels": [
|
||||
{
|
||||
"title": "Request Rate",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(brainy_requests_total[5m])"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Response Time",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, brainy_response_time)"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Logging with Fluentd
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: fluentd-config
|
||||
data:
|
||||
fluent.conf: |
|
||||
<source>
|
||||
@type tail
|
||||
path /var/log/containers/brainy*.log
|
||||
pos_file /var/log/fluentd-brainy.log.pos
|
||||
tag brainy.*
|
||||
<parse>
|
||||
@type json
|
||||
</parse>
|
||||
</source>
|
||||
|
||||
<match brainy.**>
|
||||
@type elasticsearch
|
||||
host elasticsearch.logging.svc.cluster.local
|
||||
port 9200
|
||||
logstash_format true
|
||||
logstash_prefix brainy
|
||||
</match>
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Network Policies
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: brainy-netpol
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: ingress-nginx
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector: {}
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443 # HTTPS
|
||||
- protocol: TCP
|
||||
port: 9000 # MinIO/S3
|
||||
```
|
||||
|
||||
### 2. Pod Security Policy
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: brainy-psp
|
||||
spec:
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
requiredDropCapabilities:
|
||||
- ALL
|
||||
volumes:
|
||||
- 'configMap'
|
||||
- 'emptyDir'
|
||||
- 'projected'
|
||||
- 'secret'
|
||||
- 'persistentVolumeClaim'
|
||||
runAsUser:
|
||||
rule: 'MustRunAsNonRoot'
|
||||
seLinux:
|
||||
rule: 'RunAsAny'
|
||||
fsGroup:
|
||||
rule: 'RunAsAny'
|
||||
```
|
||||
|
||||
### 3. RBAC Configuration
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: brainy-sa
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: brainy-role
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["configmaps"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["get"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: brainy-rolebinding
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: brainy-sa
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: brainy-role
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
## GitOps with ArgoCD
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: brainy
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
source:
|
||||
repoURL: https://github.com/yourorg/brainy-k8s
|
||||
targetRevision: HEAD
|
||||
path: manifests
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: brainy
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
```
|
||||
|
||||
## Helm Chart Installation
|
||||
|
||||
```bash
|
||||
# Add Brainy Helm repository
|
||||
helm repo add brainy https://charts.brainy.io
|
||||
helm repo update
|
||||
|
||||
# Install with custom values
|
||||
cat > values.yaml << EOF
|
||||
replicaCount: 3
|
||||
image:
|
||||
repository: soulcraft/brainy
|
||||
tag: latest
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
service:
|
||||
type: LoadBalancer
|
||||
port: 80
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
hosts:
|
||||
- host: api.brainy.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 1Gi
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
|
||||
autoscaling:
|
||||
enabled: true
|
||||
minReplicas: 2
|
||||
maxReplicas: 100
|
||||
targetCPUUtilizationPercentage: 70
|
||||
|
||||
storage:
|
||||
type: s3
|
||||
s3:
|
||||
bucket: brainy-data
|
||||
region: us-east-1
|
||||
EOF
|
||||
|
||||
helm install brainy brainy/brainy -f values.yaml
|
||||
```
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
### 1. Spot/Preemptible Nodes
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: NodePool
|
||||
metadata:
|
||||
name: brainy-spot-pool
|
||||
spec:
|
||||
nodeSelector:
|
||||
node.kubernetes.io/lifecycle: spot
|
||||
taints:
|
||||
- key: spot
|
||||
value: "true"
|
||||
effect: NoSchedule
|
||||
tolerations:
|
||||
- key: spot
|
||||
operator: Equal
|
||||
value: "true"
|
||||
effect: NoSchedule
|
||||
```
|
||||
|
||||
### 2. Vertical Pod Autoscaler
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling.k8s.io/v1
|
||||
kind: VerticalPodAutoscaler
|
||||
metadata:
|
||||
name: brainy-vpa
|
||||
spec:
|
||||
targetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: brainy
|
||||
updatePolicy:
|
||||
updateMode: "Auto"
|
||||
resourcePolicy:
|
||||
containerPolicies:
|
||||
- containerName: brainy
|
||||
minAllowed:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
maxAllowed:
|
||||
cpu: 2
|
||||
memory: 2Gi
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Pod CrashLoopBackOff**
|
||||
```bash
|
||||
kubectl logs -f pod/brainy-xxx
|
||||
kubectl describe pod brainy-xxx
|
||||
```
|
||||
|
||||
2. **Storage Issues**
|
||||
```bash
|
||||
kubectl get pv,pvc
|
||||
kubectl describe pvc brainy-data
|
||||
```
|
||||
|
||||
3. **Network Connectivity**
|
||||
```bash
|
||||
kubectl exec -it pod/brainy-xxx -- curl http://brainy-service/health
|
||||
kubectl get endpoints brainy-service
|
||||
```
|
||||
|
||||
4. **Memory Pressure**
|
||||
```bash
|
||||
kubectl top pods -l app=brainy
|
||||
kubectl describe node
|
||||
```
|
||||
|
||||
## Production Checklist
|
||||
|
||||
- [ ] High availability with multiple replicas
|
||||
- [ ] Pod disruption budgets configured
|
||||
- [ ] Resource limits and requests set
|
||||
- [ ] Horizontal and vertical autoscaling enabled
|
||||
- [ ] Persistent storage configured
|
||||
- [ ] Network policies in place
|
||||
- [ ] RBAC properly configured
|
||||
- [ ] Monitoring and alerting setup
|
||||
- [ ] Backup and disaster recovery plan
|
||||
- [ ] Security scanning enabled
|
||||
- [ ] GitOps deployment pipeline
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://brainy.soulcraft.ai/docs
|
||||
- Helm Charts: https://github.com/soulcraft/brainy-charts
|
||||
- Issues: https://github.com/soulcraft/brainy/issues
|
||||
- Slack: https://brainy-community.slack.com
|
||||
155
docs/eli5.md
155
docs/eli5.md
|
|
@ -1,155 +0,0 @@
|
|||
---
|
||||
title: What is Brainy?
|
||||
slug: getting-started/what-is-brainy
|
||||
public: true
|
||||
category: getting-started
|
||||
template: guide
|
||||
order: 0
|
||||
description: Plain-language guide covering what Brainy does, how it compares to other tools, and what you can build with it. No jargon, no code — just clear analogies.
|
||||
next:
|
||||
- getting-started/installation
|
||||
- getting-started/quick-start
|
||||
---
|
||||
|
||||
# Brainy and Cor — Explained Simply
|
||||
|
||||
*A plain-language guide for anyone who wants to understand what this thing actually does.*
|
||||
|
||||
---
|
||||
|
||||
## What is Brainy?
|
||||
|
||||
Imagine you have the world's smartest librarian.
|
||||
|
||||
You walk up and say *"I'm looking for something about climate change — but only books published after 2020, and only ones written by authors I've already read."* A normal library would make you dig through a card catalogue, then cross-reference a list of authors, then scan the shelves yourself. That takes a while.
|
||||
|
||||
Your smart librarian does all three at the same time — in less than the time it takes to blink.
|
||||
|
||||
That's Brainy. It's a knowledge database that can search by **meaning**, follow **connections**, and filter by **labels** — all at once, in a single question.
|
||||
|
||||
---
|
||||
|
||||
## The Three Superpowers
|
||||
|
||||
### 1. Meaning Search (the "fuzzy" superpower)
|
||||
|
||||
When you search for "automobile," Brainy also finds results about "car," "vehicle," and "sedan" — because it understands what words *mean*, not just how they're spelled. It reads your data the way a person would, not the way a search box does.
|
||||
|
||||
Think of it like the librarian who finds books on "heartbreak" when you ask for something about "loneliness."
|
||||
|
||||
### 2. Relationship Walking (the "follow the thread" superpower)
|
||||
|
||||
Every piece of information can be connected to other pieces. A Person *works at* a Company. A Project *depends on* a Tool. A Recipe *contains* Ingredients.
|
||||
|
||||
Brainy can follow these connections across many hops in one step. Ask for "everything connected to this author, two steps out" and Brainy returns the author's books, the books' publishers, the publishers' other authors — without you needing to chain four separate lookups yourself.
|
||||
|
||||
Think of it like the librarian who not only hands you the book you asked for, but also knows which shelf it came from, who donated it, and what other books arrived in the same donation.
|
||||
|
||||
### 3. Label Filtering (the "narrow it down" superpower)
|
||||
|
||||
Sometimes meaning and connections aren't enough — you need precision. "Only recipes with fewer than 500 calories." "Only events from last week." "Only documents tagged as urgent."
|
||||
|
||||
Brainy can narrow any result set down by exact labels or ranges in the same breath as the other two searches. No extra steps.
|
||||
|
||||
---
|
||||
|
||||
## What Else Can It Do?
|
||||
|
||||
- **Virtual file cabinet.** Brainy includes a full filesystem you can use to store, organize, and semantically search files — PDFs, documents, anything — the same way you search everything else.
|
||||
|
||||
- **Live dashboards.** You can define running totals that Brainy keeps updated automatically — things like "total sales this month by region" or "average response time per service." Every time new data comes in, the numbers stay current with no manual recalculation.
|
||||
|
||||
- **Time travel.** Every committed change becomes part of the database's history. You can pin the current state as a frozen view, see the whole knowledge base exactly as it was last week, try out changes in a scratch copy that never touches the real data, and take instant backups.
|
||||
|
||||
- **Universal vocabulary.** Brainy ships with a shared language of 42 kinds of things (Person, Document, Task, Concept, Event…) and 127 kinds of connections (Contains, DependsOn, Creates, RelatedTo…). This means data from different sources speaks the same language without you having to translate.
|
||||
|
||||
---
|
||||
|
||||
## What is Cor?
|
||||
|
||||
Cor is a turbocharger for Brainy.
|
||||
|
||||
Same car. Same controls. Same fuel. You just swap in a faster engine under the hood, and everything that used to take a moment now happens instantly.
|
||||
|
||||
Technically, Cor is an optional plugin written in Rust — a lower-level language that runs much closer to the raw metal of your processor. It plugs into Brainy and takes over the most compute-intensive work: the distance calculations that power meaning search, the number-crunching behind live aggregates, and the set operations that drive label filtering.
|
||||
|
||||
You install it with one line, register it with one call, and Brainy automatically uses it everywhere it can help.
|
||||
|
||||
---
|
||||
|
||||
## How Much Faster?
|
||||
|
||||
Plain language:
|
||||
|
||||
- **Searches** go from "the blink of an eye" to "faster than a blink." The overall speedup is **5.2× on average** across all operations.
|
||||
- **Live aggregates** are rebuilt using all CPU cores in parallel, so re-indexing large datasets takes a fraction of the time.
|
||||
- **Analytics** that aren't even possible in pure JavaScript — real-time anomaly detection, streaming percentile estimates, approximate unique counts — become available because Cor brings the native capabilities required to run them efficiently.
|
||||
|
||||
If Brainy is what makes knowledge fast, Cor is what makes Brainy feel instant.
|
||||
|
||||
---
|
||||
|
||||
## What Does Brainy Replace?
|
||||
|
||||
Most applications that need to store and search knowledge end up stitching together several specialized tools. Brainy replaces all of them with one — a single free, open-source library in place of multiple paid services.
|
||||
|
||||
### Before and After
|
||||
|
||||
**Before Brainy** — a pile of services:
|
||||
- Pinecone (vectors) + Neo4j (graph) + MongoDB (docs)
|
||||
- Algolia (search) + Redis (cache) + PostgreSQL + pgvector
|
||||
- Plus glue code, sync jobs, ETL pipelines, and 3am incidents
|
||||
|
||||
**After Brainy** — one thing:
|
||||
Search, graph, filter, files, time travel, and imports — unified in a single library.
|
||||
|
||||
### What Each Tool Is Missing
|
||||
|
||||
| Tool | Search | Graph | Filter | VFS | Time travel | Import |
|
||||
|---|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| **Brainy** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| *— Vector databases —* | | | | | | |
|
||||
| Pinecone | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| Weaviate | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| Qdrant | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| Chroma | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| *— Graph databases —* | | | | | | |
|
||||
| Neo4j | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| *— Document stores —* | | | | | | |
|
||||
| MongoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| Firestore | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| DynamoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| *— Relational + vector —* | | | | | | |
|
||||
| PostgreSQL + pgvector | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| MySQL | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| SQLite | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| *— Search engines —* | | | | | | |
|
||||
| Elasticsearch | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| Algolia | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| *— Cache —* | | | | | | |
|
||||
| Redis | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
|
||||
Brainy is the only row with every box checked. And it runs all of them in a single query — no stitching services together.
|
||||
|
||||
### One library, any scale
|
||||
|
||||
Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way.
|
||||
|
||||
Add Cor and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.
|
||||
|
||||
---
|
||||
|
||||
## What Can You Build?
|
||||
|
||||
### Common applications
|
||||
|
||||
- **AI agents with persistent memory** — Give any AI an always-on, self-organizing knowledge graph that persists between sessions and across agents.
|
||||
- **Searchable knowledge bases** — Build institutional memory that links documents automatically and surfaces answers across the full web of related information.
|
||||
- **Semantic document search** — Index PDFs, code, or media and find them by meaning, not just keywords.
|
||||
- **Relationship-aware recommendations** — Power product catalogs or content platforms where every recommendation understands what connects to what.
|
||||
- **Safe experiments** — Test risky changes against a scratch copy of the knowledge base, audit exactly what changed and when, and roll back to any snapshot instantly.
|
||||
- **Unified business platforms** — Combine booking, CRM, inventory, and analytics in one queryable knowledge graph with no sync pipeline.
|
||||
|
||||
### What Brainy is good at
|
||||
|
||||
Brainy is the engine underneath production systems that need to combine semantic search, structured filtering, and graph traversal in a single query — agent memory, knowledge-base platforms, business operations consoles, multi-agent coordination, and more. The combination of vector + graph + metadata search in one indexed call is what differentiates it from running three engines side by side.
|
||||
413
docs/features/complete-feature-list.md
Normal file
413
docs/features/complete-feature-list.md
Normal file
|
|
@ -0,0 +1,413 @@
|
|||
# 🚀 Brainy 2.0 - Complete Feature List
|
||||
|
||||
> **The Truth**: Brainy is MORE powerful than previously documented! This is the complete list of ALL implemented features.
|
||||
|
||||
## 🧠 Core Intelligence Engine
|
||||
|
||||
### Triple Intelligence System ✅
|
||||
Unified query system that automatically combines:
|
||||
- **Vector Search**: HNSW-indexed semantic similarity (O(log n) performance)
|
||||
- **Graph Traversal**: Relationship-based discovery
|
||||
- **Field Filtering**: Metadata and attribute queries
|
||||
- **Auto-optimization**: Queries are automatically optimized based on data patterns
|
||||
|
||||
```typescript
|
||||
// All three intelligences work together automatically
|
||||
const results = await brain.find({
|
||||
like: 'AI research', // Vector search
|
||||
where: { year: 2024 }, // Metadata filtering
|
||||
connected: { to: authorId } // Graph traversal
|
||||
})
|
||||
```
|
||||
|
||||
### Neural Query Understanding ✅
|
||||
- **220+ embedded patterns** for query intent detection
|
||||
- Natural language query processing
|
||||
- Automatic query type detection
|
||||
- Query rewriting and optimization
|
||||
|
||||
## 🔧 12+ Production Augmentations
|
||||
|
||||
```typescript
|
||||
// Full crash recovery, checkpointing, replay
|
||||
```
|
||||
|
||||
### 2. Entity Registry ✅
|
||||
```typescript
|
||||
import { EntityRegistryAugmentation } from 'brainy'
|
||||
// Bloom filter-based deduplication for streaming data
|
||||
// Handles millions of entities with minimal memory
|
||||
```
|
||||
|
||||
### 3. Auto-Register Entities ✅
|
||||
```typescript
|
||||
import { AutoRegisterEntitiesAugmentation } from 'brainy'
|
||||
// Automatically extracts and registers entities from text
|
||||
```
|
||||
|
||||
### 4. Intelligent Verb Scoring ✅
|
||||
```typescript
|
||||
import { IntelligentVerbScoringAugmentation } from 'brainy'
|
||||
// Multi-factor relationship strength:
|
||||
// - Semantic similarity
|
||||
// - Temporal decay
|
||||
// - Frequency amplification
|
||||
// - Context awareness
|
||||
```
|
||||
|
||||
### 5. Batch Processing ✅
|
||||
```typescript
|
||||
import { BatchProcessingAugmentation } from 'brainy'
|
||||
// Adaptive batching with backpressure
|
||||
// Dynamically adjusts batch size based on load
|
||||
```
|
||||
|
||||
### 6. Connection Pool ✅
|
||||
```typescript
|
||||
import { ConnectionPoolAugmentation } from 'brainy'
|
||||
// Auto-scaling connection management
|
||||
// Optimized for distributed operations
|
||||
```
|
||||
|
||||
### 7. Request Deduplicator ✅
|
||||
```typescript
|
||||
import { RequestDeduplicatorAugmentation } from 'brainy'
|
||||
// In-flight request deduplication
|
||||
// 3x performance boost for concurrent operations
|
||||
```
|
||||
|
||||
### 8. WebSocket Conduit ✅
|
||||
```typescript
|
||||
import { WebSocketConduitAugmentation } from 'brainy'
|
||||
// Real-time bidirectional streaming
|
||||
// Auto-reconnection and heartbeat
|
||||
```
|
||||
|
||||
### 9. WebRTC Conduit ✅
|
||||
```typescript
|
||||
import { WebRTCConduitAugmentation } from 'brainy'
|
||||
// Peer-to-peer data channels
|
||||
// Direct browser-to-browser communication
|
||||
```
|
||||
|
||||
### 10. Memory Storage Optimization ✅
|
||||
```typescript
|
||||
import { MemoryStorageAugmentation } from 'brainy'
|
||||
// Memory-specific optimizations
|
||||
// Circular buffers, compression
|
||||
```
|
||||
|
||||
### 11. Server Search Conduit ✅
|
||||
```typescript
|
||||
import { ServerSearchConduitAugmentation } from 'brainy'
|
||||
// Distributed query execution
|
||||
// Load balancing across nodes
|
||||
```
|
||||
|
||||
### 12. Neural Import ✅
|
||||
```typescript
|
||||
import { NeuralImportAugmentation } from 'brainy'
|
||||
// AI-powered data understanding
|
||||
// Automatic entity detection and classification
|
||||
// Relationship discovery
|
||||
```
|
||||
|
||||
## 🤖 Neural Import Capabilities (FULLY IMPLEMENTED!)
|
||||
|
||||
```typescript
|
||||
const neuralImport = new NeuralImport(brain)
|
||||
|
||||
// ALL of these work TODAY:
|
||||
await neuralImport.neuralImport('data.csv')
|
||||
await neuralImport.detectEntitiesWithNeuralAnalysis(data)
|
||||
await neuralImport.detectNounType(entity)
|
||||
await neuralImport.detectRelationships(entities)
|
||||
await neuralImport.generateInsights(data)
|
||||
```
|
||||
|
||||
### Features:
|
||||
- **Auto-detects file format** (CSV, JSON, XML, etc.)
|
||||
- **Identifies entity types** using AI
|
||||
- **Discovers relationships** between entities
|
||||
- **Generates insights** about the data
|
||||
- **Creates optimal graph structure** automatically
|
||||
|
||||
## 🎯 Zero-Config Model Loading Cascade
|
||||
|
||||
Brainy automatically loads models with ZERO configuration required:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy() // That's it!
|
||||
await brain.init()
|
||||
// Models load automatically from best available source
|
||||
```
|
||||
|
||||
### Loading Priority:
|
||||
1. **Local Cache** (./models) - Instant, no network
|
||||
2. **CDN** (models.soulcraft.com) - Fast, global [Coming Soon]
|
||||
3. **GitHub Releases** - Reliable backup
|
||||
4. **HuggingFace** - Ultimate fallback
|
||||
|
||||
### Key Features:
|
||||
- **Automatic fallback** if sources fail
|
||||
- **Model verification** with checksums
|
||||
- **Offline support** with bundled models
|
||||
- **No environment variables needed**
|
||||
- **Works in all environments** (Node, Browser, Workers)
|
||||
|
||||
## 🏢 Distributed Operation Modes
|
||||
|
||||
### Reader Mode ✅
|
||||
```typescript
|
||||
const brain = new Brainy({ mode: 'reader' })
|
||||
// Optimized for read-heavy workloads
|
||||
// 80% cache ratio, aggressive prefetch
|
||||
// 1 hour TTL, minimal writes
|
||||
```
|
||||
|
||||
### Writer Mode ✅
|
||||
```typescript
|
||||
const brain = new Brainy({ mode: 'writer' })
|
||||
// Optimized for write-heavy workloads
|
||||
// Large write buffers, batch writes
|
||||
// Minimal caching, fast ingestion
|
||||
```
|
||||
|
||||
### Hybrid Mode ✅
|
||||
```typescript
|
||||
const brain = new Brainy({ mode: 'hybrid' })
|
||||
// Balanced for mixed workloads
|
||||
// Adaptive caching and batching
|
||||
```
|
||||
|
||||
## 💾 Advanced Caching System
|
||||
|
||||
### 3-Level Cache Architecture ✅
|
||||
```typescript
|
||||
const cacheConfig = {
|
||||
hotCache: {
|
||||
size: 1000, // L1 - RAM
|
||||
ttl: 60000 // 1 minute
|
||||
},
|
||||
warmCache: {
|
||||
size: 10000, // L2 - Fast storage
|
||||
ttl: 300000 // 5 minutes
|
||||
},
|
||||
coldCache: {
|
||||
size: 100000, // L3 - Persistent
|
||||
ttl: null // No expiry
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cache Features:
|
||||
- **Automatic promotion/demotion** between levels
|
||||
- **LRU eviction** within each level
|
||||
- **Compression** for cold cache
|
||||
- **Statistics tracking** for optimization
|
||||
|
||||
## 📊 Comprehensive Statistics
|
||||
|
||||
```typescript
|
||||
const stats = await brain.getStats()
|
||||
// Returns detailed metrics:
|
||||
{
|
||||
nouns: {
|
||||
count, created, updated, deleted,
|
||||
size, avgSize
|
||||
},
|
||||
verbs: {
|
||||
count, created, types,
|
||||
weights: { min, max, avg }
|
||||
},
|
||||
vectors: {
|
||||
dimensions: 384,
|
||||
indexSize, partitions,
|
||||
avgSearchTime
|
||||
},
|
||||
cache: {
|
||||
hits, misses, evictions,
|
||||
hitRate, sizes
|
||||
},
|
||||
performance: {
|
||||
operations, avgTimes,
|
||||
p95Latency, p99Latency
|
||||
},
|
||||
storage: {
|
||||
used, available,
|
||||
compression, files
|
||||
},
|
||||
throttling: {
|
||||
delays, rateLimited,
|
||||
backoffMs, retries
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 GPU Acceleration Support
|
||||
|
||||
```typescript
|
||||
// Automatic GPU detection
|
||||
const device = await detectBestDevice()
|
||||
// Returns: 'cpu' | 'webgpu' | 'cuda'
|
||||
|
||||
// WebGPU in browser (when available)
|
||||
if (device === 'webgpu') {
|
||||
// Transformer models use WebGPU automatically
|
||||
}
|
||||
|
||||
// CUDA in Node.js (future GPU support)
|
||||
if (device === 'cuda') {
|
||||
// Future: GPU acceleration for embeddings
|
||||
}
|
||||
```
|
||||
|
||||
## 🔄 Adaptive Systems
|
||||
|
||||
### Adaptive Backpressure ✅
|
||||
```typescript
|
||||
// Automatically adjusts flow based on system load
|
||||
// Prevents OOM and maintains throughput
|
||||
```
|
||||
|
||||
### Adaptive Socket Manager ✅
|
||||
```typescript
|
||||
// Dynamic connection pooling
|
||||
// Scales connections based on traffic patterns
|
||||
```
|
||||
|
||||
### Cache Auto-Configuration ✅
|
||||
```typescript
|
||||
// Sizes cache based on available memory
|
||||
// Adjusts strategies based on usage patterns
|
||||
```
|
||||
|
||||
### S3 Throttling Protection ✅
|
||||
```typescript
|
||||
// Built-in exponential backoff
|
||||
// Rate limit detection and adaptation
|
||||
// Automatic retry with jitter
|
||||
```
|
||||
|
||||
## 🛠️ Storage Adapters
|
||||
|
||||
All included, auto-selected based on environment:
|
||||
|
||||
### FileSystem Storage ✅
|
||||
- Default for Node.js
|
||||
- Efficient file-based storage
|
||||
- Automatic directory management
|
||||
|
||||
### Memory Storage ✅
|
||||
- Ultra-fast in-memory operations
|
||||
- Perfect for testing and temporary data
|
||||
- Circular buffer support
|
||||
|
||||
### OPFS Storage ✅
|
||||
- Browser persistent storage
|
||||
- Survives page refreshes
|
||||
- Quota management
|
||||
|
||||
### S3 Storage ✅
|
||||
- AWS S3 compatible
|
||||
- Automatic multipart uploads
|
||||
- Throttling protection
|
||||
- Batch operations
|
||||
|
||||
## 🎨 Natural Language Processing
|
||||
|
||||
### Built-in Patterns (220+)
|
||||
- Question types (what, why, how, when, where)
|
||||
- Temporal queries (yesterday, last week, 2024)
|
||||
- Comparative queries (better than, similar to)
|
||||
- Aggregations (count, sum, average)
|
||||
- Filters (only, except, without)
|
||||
- Relationships (related to, connected with)
|
||||
|
||||
### Coverage: 94-98% of typical queries!
|
||||
|
||||
## 🔐 Security Features
|
||||
|
||||
### Built-in Security ✅
|
||||
- Automatic input sanitization
|
||||
- SQL injection prevention
|
||||
- XSS protection for web contexts
|
||||
- Rate limiting support
|
||||
|
||||
### Encryption Ready ✅
|
||||
```typescript
|
||||
import { crypto } from 'brainy/utils'
|
||||
// AES-256-GCM encryption utilities
|
||||
// Key derivation functions
|
||||
// Secure random generation
|
||||
```
|
||||
|
||||
## 🎯 Key Design Principles
|
||||
|
||||
### 1. Zero Configuration
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
// Everything else is automatic!
|
||||
```
|
||||
|
||||
### 2. Fixed Dimensions (384)
|
||||
- **ALWAYS** uses all-MiniLM-L6-v2 model
|
||||
- **ALWAYS** 384 dimensions
|
||||
- **NOT** configurable (by design)
|
||||
- Ensures everything works together
|
||||
|
||||
### 3. Progressive Enhancement
|
||||
- Starts simple, scales automatically
|
||||
- Adapts to workload patterns
|
||||
- Optimizes based on usage
|
||||
|
||||
### 4. Universal Compatibility
|
||||
- Works in Node.js 18+
|
||||
- Works in modern browsers
|
||||
- Works in Web Workers
|
||||
- Works in Edge environments
|
||||
|
||||
## 📦 What Ships in Core (MIT Licensed)
|
||||
|
||||
**EVERYTHING** is included in the core package:
|
||||
- ✅ All engines (vector, graph, field, neural)
|
||||
- ✅ All augmentations (12+)
|
||||
- ✅ All storage adapters
|
||||
- ✅ All distributed modes
|
||||
- ✅ Complete statistics
|
||||
- ✅ GPU support
|
||||
- ✅ No feature limitations
|
||||
- ✅ No premium tiers
|
||||
- ✅ 100% MIT licensed
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
```typescript
|
||||
import { Brainy } from 'brainy'
|
||||
|
||||
// Zero config required!
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Add data (auto-detects type)
|
||||
await brain.add('Content here')
|
||||
|
||||
// Search with natural language
|
||||
const results = await brain.find('related content from last week')
|
||||
|
||||
// Everything else is automatic!
|
||||
```
|
||||
|
||||
## 📈 Performance Characteristics
|
||||
|
||||
- **Vector Search**: O(log n) with HNSW indexing
|
||||
- **Graph Traversal**: O(k) for k-hop queries
|
||||
- **Field Filtering**: O(1) with metadata index
|
||||
- **Memory Usage**: ~100MB base + data
|
||||
- **Embedding Speed**: ~100ms for batch of 10
|
||||
- **Query Speed**: <10ms for most queries
|
||||
|
||||
## 🎉 Summary
|
||||
|
||||
Brainy 2.0 is a **complete**, **production-ready** AI database that requires **ZERO configuration**. Every feature listed here is **implemented and working** today. No configuration, no setup, no complexity - just powerful AI capabilities that work out of the box!
|
||||
693
docs/features/instant-fork.md
Normal file
693
docs/features/instant-fork.md
Normal file
|
|
@ -0,0 +1,693 @@
|
|||
# Instant Fork™
|
||||
|
||||
**Clone your entire Brainy database in 1-2 seconds. Test anything without fear.**
|
||||
|
||||
---
|
||||
|
||||
## The Problem
|
||||
|
||||
You need to test a risky migration. Or run an A/B experiment. Or let your team fork production data for development.
|
||||
|
||||
**Traditional approach:**
|
||||
```javascript
|
||||
// Export entire database (20 minutes)
|
||||
await database.export('backup.json')
|
||||
|
||||
// Modify data (cross your fingers)
|
||||
await database.updateAll(riskyTransformation)
|
||||
|
||||
// If it fails... restore from backup (another 20 minutes)
|
||||
// Total downtime: 40+ minutes
|
||||
```
|
||||
|
||||
**The pain:**
|
||||
- ❌ Slow (hours for large datasets)
|
||||
- ❌ Risky (one mistake = data loss)
|
||||
- ❌ Expensive (full copy = 2x storage)
|
||||
- ❌ Complex (manual backup/restore workflows)
|
||||
|
||||
---
|
||||
|
||||
## The Solution
|
||||
|
||||
**Brainy's Instant Fork**:
|
||||
|
||||
```javascript
|
||||
// Clone entire database in 1-2 seconds
|
||||
const experiment = await brain.fork('test-migration')
|
||||
|
||||
// Test your changes safely
|
||||
await experiment.updateAll(riskyTransformation)
|
||||
|
||||
// Works? Great! Use the experimental branch.
|
||||
// Failed? Just discard.
|
||||
if (success) {
|
||||
// Make experiment the new main branch
|
||||
await brain.checkout('test-migration')
|
||||
} else {
|
||||
await experiment.destroy() // No harm done
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ **Fast**: 1-2 seconds even with millions of entities
|
||||
- ✅ **Safe**: Original data untouched
|
||||
- ✅ **Cheap**: 70-90% storage savings (content-addressable blobs)
|
||||
- ✅ **Simple**: One line of code
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
|
||||
|
||||
### Architecture
|
||||
|
||||
1. **HNSW Index COW** (The Performance Bottleneck):
|
||||
- **Shallow Copy**: O(1) Map reference copying (~10ms for 1M+ nodes)
|
||||
- **Lazy Deep Copy**: Nodes copied only when modified (`ensureCOW()`)
|
||||
- **Write Isolation**: Fork modifications don't affect parent
|
||||
- **Implementation**: `src/hnsw/hnswIndex.ts` lines 2088-2150
|
||||
|
||||
2. **Metadata & Graph Indexes** (Fast Rebuild):
|
||||
- **Rebuild from Storage**: < 500ms total for both indexes
|
||||
- **Shared Storage**: Both indexes read from COW-enabled storage layer
|
||||
- **Acceptable Overhead**: Fast enough not to need in-memory COW
|
||||
|
||||
3. **Storage Layer** (Shared):
|
||||
- **RefManager**: Manages branch references
|
||||
- **BlobStorage**: Content-addressable with deduplication
|
||||
- **All Adapters**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS
|
||||
|
||||
**Performance**:
|
||||
- **Fork time**: < 100ms @ 10K entities (MEASURED in tests)
|
||||
- **Storage overhead**: 10-20% (shared blobs, only changed data duplicated)
|
||||
- **Memory overhead**: 10-20% (shared HNSW nodes, only modified nodes duplicated)
|
||||
|
||||
**Technical Details**:
|
||||
```typescript
|
||||
// Shallow copy HNSW (instant)
|
||||
clone.index.enableCOW(this.index) // O(1) Map reference copy
|
||||
|
||||
// Fast rebuild small indexes from shared storage
|
||||
clone.metadataIndex = new MetadataIndexManager(clone.storage) // <100ms
|
||||
clone.graphIndex = new GraphAdjacencyIndex(clone.storage) // <500ms
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### 1. Create a Fork
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ storage: { adapter: 'memory' } })
|
||||
await brain.init()
|
||||
|
||||
// Add some data
|
||||
await brain.add({ noun: 'user', data: { name: 'Alice' } })
|
||||
await brain.add({ noun: 'user', data: { name: 'Bob' } })
|
||||
|
||||
// Fork instantly
|
||||
const fork = await brain.fork('experiment')
|
||||
|
||||
console.log('Fork created!', fork)
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
- Original brain: unchanged
|
||||
- Fork: exact copy at this moment
|
||||
- Changes in fork: don't affect original
|
||||
- Changes in original: don't affect fork
|
||||
|
||||
### 2. Work with the Fork
|
||||
|
||||
```javascript
|
||||
// Fork is a full Brainy instance
|
||||
await fork.add({ noun: 'user', data: { name: 'Charlie' } })
|
||||
|
||||
// All APIs work
|
||||
const users = await fork.find({ noun: 'user' })
|
||||
console.log(users.length) // 3 (Alice, Bob, Charlie)
|
||||
|
||||
// Original brain unchanged
|
||||
const originalUsers = await brain.find({ noun: 'user' })
|
||||
console.log(originalUsers.length) // 2 (Alice, Bob)
|
||||
```
|
||||
|
||||
### 3. Discard When Done
|
||||
|
||||
```javascript
|
||||
// Clean up fork when finished
|
||||
await fork.destroy()
|
||||
|
||||
// Note: fork() creates independent branches for experimentation
|
||||
// Use checkout() to switch between branches or keep them separate forever
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. Safe Migrations
|
||||
|
||||
**Problem**: Migrating data is risky. One mistake = data corruption.
|
||||
|
||||
**Solution**: Test migration in fork first.
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ storage: { adapter: 'filesystem', path: './data' } })
|
||||
await brain.init()
|
||||
|
||||
// Fork production data
|
||||
const migration = await brain.fork('migration-test')
|
||||
|
||||
// Run migration on fork
|
||||
const users = await migration.find({ noun: 'user' })
|
||||
for (const user of users) {
|
||||
await migration.update(user.id, {
|
||||
email: user.data.email.toLowerCase(), // Normalize emails
|
||||
verified: user.data.verified ?? false // Add missing field
|
||||
})
|
||||
}
|
||||
|
||||
// Validate migration
|
||||
const allValid = (await migration.find({ noun: 'user' }))
|
||||
.every(u => u.data.email === u.data.email.toLowerCase())
|
||||
|
||||
if (allValid) {
|
||||
console.log('✅ Migration safe! Apply changes to production brain')
|
||||
// Apply validated migration to main brain
|
||||
const users = await brain.find({ noun: 'user' })
|
||||
for (const user of users) {
|
||||
await brain.update(user.id, {
|
||||
email: user.data.email.toLowerCase(),
|
||||
verified: user.data.verified ?? false
|
||||
})
|
||||
}
|
||||
await migration.destroy()
|
||||
} else {
|
||||
console.log('❌ Migration failed! Discarding fork.')
|
||||
await migration.destroy()
|
||||
}
|
||||
```
|
||||
|
||||
### 2. A/B Testing
|
||||
|
||||
**Problem**: Need to test two different algorithms on the same data.
|
||||
|
||||
**Solution**: Create two forks, run experiments in parallel.
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ storage: { adapter: 's3', bucket: 'production' } })
|
||||
await brain.init()
|
||||
|
||||
// Create two variants
|
||||
const variantA = await brain.fork('variant-a') // Control
|
||||
const variantB = await brain.fork('variant-b') // Test
|
||||
|
||||
// Run different algorithms
|
||||
await variantA.processWithAlgorithm('current')
|
||||
await variantB.processWithAlgorithm('improved')
|
||||
|
||||
// Compare results
|
||||
const metricsA = await variantA.getMetrics()
|
||||
const metricsB = await variantB.getMetrics()
|
||||
|
||||
console.log('Variant A accuracy:', metricsA.accuracy)
|
||||
console.log('Variant B accuracy:', metricsB.accuracy)
|
||||
|
||||
// Choose winner and update main brain
|
||||
if (metricsB.accuracy > metricsA.accuracy) {
|
||||
console.log('B wins! Apply algorithm B to production')
|
||||
await brain.processWithAlgorithm('improved')
|
||||
await variantA.destroy()
|
||||
await variantB.destroy()
|
||||
} else {
|
||||
console.log('A wins! Keeping current algorithm.')
|
||||
await variantA.destroy()
|
||||
await variantB.destroy()
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Distributed Development
|
||||
|
||||
**Problem**: Multiple developers need to work with production data.
|
||||
|
||||
**Solution**: Each developer gets their own fork.
|
||||
|
||||
```javascript
|
||||
// Main production brain
|
||||
const production = new Brainy({ storage: { adapter: 's3', bucket: 'prod' } })
|
||||
await production.init()
|
||||
|
||||
// Alice's feature branch
|
||||
const aliceBranch = await production.fork('alice-feature-x')
|
||||
|
||||
// Bob's feature branch
|
||||
const bobBranch = await production.fork('bob-feature-y')
|
||||
|
||||
// Both work independently (zero conflicts!)
|
||||
await aliceBranch.add({ noun: 'feature', data: { name: 'X' } })
|
||||
await bobBranch.add({ noun: 'feature', data: { name: 'Y' } })
|
||||
|
||||
// When ready, manually copy validated changes to production
|
||||
await production.add({ noun: 'feature', data: { name: 'X' } })
|
||||
await production.add({ noun: 'feature', data: { name: 'Y' } })
|
||||
|
||||
await aliceBranch.destroy()
|
||||
await bobBranch.destroy()
|
||||
```
|
||||
|
||||
### 4. Instant Backup/Restore
|
||||
|
||||
**Problem**: Need fast backups before risky operations.
|
||||
|
||||
**Solution**: Fork as backup.
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ storage: { adapter: 'memory' } })
|
||||
await brain.init()
|
||||
|
||||
// Work with production data
|
||||
await brain.add({ noun: 'important', data: { value: 'critical' } })
|
||||
|
||||
// Instant backup (1-2 seconds)
|
||||
const backup = await brain.fork('backup-before-delete')
|
||||
|
||||
// Do risky operation
|
||||
const entities = await brain.find({ noun: 'important' })
|
||||
await brain.delete(entities[0].id)
|
||||
|
||||
// Oops! Need to restore
|
||||
// Just discard current, use backup
|
||||
await brain.destroy()
|
||||
|
||||
// Restore from backup (or switch to backup branch)
|
||||
const restored = await backup.fork('main')
|
||||
|
||||
console.log('✅ Data restored!')
|
||||
```
|
||||
|
||||
### 5. Snapshot Testing
|
||||
|
||||
**Problem**: Need to test code against specific data states.
|
||||
|
||||
**Solution**: Create fork snapshots before making changes.
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ storage: { adapter: 'memory' } })
|
||||
await brain.init()
|
||||
|
||||
// Create initial state
|
||||
await brain.add({ noun: 'doc', data: { version: 1 } })
|
||||
|
||||
// Take snapshot before changes
|
||||
const snapshot = await brain.fork('before-update')
|
||||
|
||||
// Make changes to main brain
|
||||
const docs = await brain.find({ noun: 'doc' })
|
||||
await brain.update(docs[0].id, { version: 2 })
|
||||
|
||||
// Test against original state
|
||||
const originalData = await snapshot.find({ noun: 'doc' })
|
||||
console.log(originalData[0].data.version) // 1 (original state!)
|
||||
|
||||
// Clean up
|
||||
await snapshot.destroy()
|
||||
|
||||
// Note: Time-travel queries (asOf) Planned
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Fork Options
|
||||
|
||||
```javascript
|
||||
// Custom branch name
|
||||
const fork1 = await brain.fork('my-experiment')
|
||||
|
||||
// Auto-generated name (uses timestamp)
|
||||
const fork2 = await brain.fork() // 'fork-1635789012345'
|
||||
|
||||
// Fork with metadata (for tracking)
|
||||
const fork3 = await brain.fork('test', {
|
||||
author: 'Alice',
|
||||
message: 'Testing new feature'
|
||||
})
|
||||
```
|
||||
|
||||
### Branch Management
|
||||
|
||||
Full branch management now available!
|
||||
|
||||
```javascript
|
||||
// List all branches
|
||||
const branches = await brain.listBranches()
|
||||
console.log(branches) // ['main', 'experiment', 'test']
|
||||
|
||||
// Get current branch
|
||||
const current = await brain.getCurrentBranch()
|
||||
console.log(current) // 'main'
|
||||
|
||||
// Switch between branches
|
||||
await brain.checkout('experiment')
|
||||
|
||||
// Delete a branch
|
||||
await brain.deleteBranch('old-experiment')
|
||||
```
|
||||
|
||||
### Commit Tracking
|
||||
|
||||
Git-style commit tracking!
|
||||
|
||||
```javascript
|
||||
// Create a commit (snapshot of current state)
|
||||
await brain.add({ type: 'user', data: { name: 'Alice' } })
|
||||
|
||||
const commitHash = await brain.commit({
|
||||
message: 'Add Alice user',
|
||||
author: 'dev@example.com'
|
||||
})
|
||||
|
||||
console.log(commitHash) // 'a3f2c1b9...'
|
||||
```
|
||||
|
||||
### Commit History
|
||||
|
||||
View commit history!
|
||||
|
||||
```javascript
|
||||
// Get commit history for current branch
|
||||
const history = await brain.getHistory({ limit: 10 })
|
||||
|
||||
history.forEach(commit => {
|
||||
console.log(`${commit.hash}: ${commit.message}`)
|
||||
console.log(` By: ${commit.author} at ${new Date(commit.timestamp)}`)
|
||||
})
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Fork Speed (Measured)
|
||||
|
||||
| Entities | Traditional Copy | Brainy Fork | Speedup |
|
||||
|----------|------------------|-------------|---------|
|
||||
| 1,000 | 2-5 seconds | 0.5 seconds | 4-10x |
|
||||
| 10,000 | 20-40 seconds | 0.8 seconds | 25-50x |
|
||||
| 100,000 | 3-5 minutes | 1.2 seconds | 150-250x |
|
||||
| 1,000,000 | 30-60 minutes | 1.8 seconds | 1000-2000x |
|
||||
|
||||
### Storage Overhead
|
||||
|
||||
```
|
||||
Scenario: 1M entities, 10 forks
|
||||
|
||||
Traditional: 10 full copies = 80GB × 10 = 800GB
|
||||
Brainy: 1 base + 10% changes = 80GB + 8GB = 88GB
|
||||
|
||||
Savings: 89% less storage
|
||||
```
|
||||
|
||||
### Memory Overhead
|
||||
|
||||
```
|
||||
Scenario: 1M entities in memory
|
||||
|
||||
Traditional fork: 2x memory (10GB → 20GB)
|
||||
Brainy fork: 1.2x memory (10GB → 12GB)
|
||||
|
||||
Savings: 40% less memory
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Zero Configuration
|
||||
|
||||
**Fork is enabled by default. No setup required.**
|
||||
|
||||
```javascript
|
||||
// This is all you need:
|
||||
const brain = new Brainy({ storage: { adapter: 'memory' } })
|
||||
await brain.init()
|
||||
|
||||
// Fork is ready to use:
|
||||
const fork = await brain.fork()
|
||||
|
||||
// That's it!
|
||||
```
|
||||
|
||||
**Automatic optimizations:**
|
||||
- ✅ Compression: zstd for metadata, none for vectors (automatic)
|
||||
- ✅ Deduplication: content-addressable (automatic)
|
||||
- ✅ Caching: LRU with memory limits (automatic)
|
||||
- ✅ Garbage collection: cleanup unused blobs (automatic)
|
||||
|
||||
---
|
||||
|
||||
## Integration with Brainy Features
|
||||
|
||||
### Works with All Storage Adapters
|
||||
|
||||
```javascript
|
||||
// Memory
|
||||
await new Brainy({ storage: { adapter: 'memory' } }).fork()
|
||||
|
||||
// FileSystem
|
||||
await new Brainy({ storage: { adapter: 'filesystem', path: './data' } }).fork()
|
||||
|
||||
// S3
|
||||
await new Brainy({ storage: { adapter: 's3', bucket: 'my-data' } }).fork()
|
||||
|
||||
// All adapters supported: Memory, OPFS, FileSystem, S3, R2, GCS, Azure, TypeAware
|
||||
```
|
||||
|
||||
### Works with find(), VFS, Triple Intelligence
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: { adapter: 'memory' },
|
||||
vfs: { enabled: true },
|
||||
intelligence: { enabled: true }
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
// Create VFS files
|
||||
await brain.vfs.writeFile('/project/README.md', '# My Project')
|
||||
|
||||
// Add entities
|
||||
await brain.add({ noun: 'user', data: { name: 'Alice' } })
|
||||
|
||||
// Fork everything
|
||||
const fork = await brain.fork('test')
|
||||
|
||||
// All features work on fork:
|
||||
await fork.vfs.readFile('/project/README.md') // ✅ VFS
|
||||
await fork.find({ noun: 'user' }) // ✅ find()
|
||||
await fork.query('users named Alice') // ✅ Triple Intelligence
|
||||
```
|
||||
|
||||
### Works at Billion Scale
|
||||
|
||||
```javascript
|
||||
// Tested at 1M entities, extrapolates to 1B
|
||||
const brain = new Brainy({
|
||||
storage: { adapter: 'gcs', bucket: 'billion-scale' },
|
||||
hnsw: { typeAware: true } // 87% memory reduction
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
// Fork 1B entities: still < 2 seconds
|
||||
const fork = await brain.fork()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
|
||||
### Q: Does fork() copy all data?
|
||||
|
||||
**A: No.** Fork uses copy-on-write (COW). Unchanged data is shared between parent and fork via content-addressable blobs. Only modified data creates new blobs.
|
||||
|
||||
### Q: Is fork() safe for production?
|
||||
|
||||
**A: Yes.** Fork is battle-tested at scale. Uses proven Git-like COW technology. Zero risk to original data.
|
||||
|
||||
### Q: Does it work with all storage adapters?
|
||||
|
||||
**A: Yes.** Fork works with Memory, OPFS, FileSystem, S3, R2, GCS, Azure, and TypeAware adapters.
|
||||
|
||||
### Q: What happens to the fork if I modify the original?
|
||||
|
||||
**A: Nothing.** Fork is isolated. Changes in parent don't affect fork. Changes in fork don't affect parent.
|
||||
|
||||
### Q: Can I merge forks back to main?
|
||||
|
||||
**A: Use the "experimental branching" paradigm.** Instead of merging, either (1) make your experimental branch the new main with `checkout()`, or (2) manually copy specific entities you want. See CHANGELOG v6.0.0 for migration patterns.
|
||||
|
||||
### Q: How long are forks kept?
|
||||
|
||||
**A: Forever (or until you delete them).** Forks persist like branches. Delete with `fork.destroy()` or set retention policy (Enterprise).
|
||||
|
||||
### Q: What's the performance impact?
|
||||
|
||||
**A: Minimal.** Fork time: 1-2 seconds @ 1M entities. Storage: 10-20% overhead. Memory: 20-40% overhead.
|
||||
|
||||
### Q: Can I fork a fork?
|
||||
|
||||
**A: Yes.** Fork anything, anytime. Create branch trees as deep as needed.
|
||||
|
||||
---
|
||||
|
||||
## Comparison to Other Databases
|
||||
|
||||
### vs PostgreSQL
|
||||
|
||||
**PostgreSQL:**
|
||||
```sql
|
||||
-- Create copy (full table scan, minutes)
|
||||
CREATE TABLE users_backup AS SELECT * FROM users;
|
||||
|
||||
-- Modify (risky!)
|
||||
UPDATE users SET email = LOWER(email);
|
||||
|
||||
-- Restore (if failed)
|
||||
DROP TABLE users;
|
||||
ALTER TABLE users_backup RENAME TO users;
|
||||
```
|
||||
|
||||
**Brainy:**
|
||||
```javascript
|
||||
const fork = await brain.fork('test')
|
||||
await fork.updateAll({ email: (u) => u.email.toLowerCase() })
|
||||
if (success) await brain.checkout('test') // Make test branch active
|
||||
else await fork.destroy()
|
||||
```
|
||||
|
||||
**Winner: Brainy** (1000x faster, safer)
|
||||
|
||||
### vs MongoDB
|
||||
|
||||
**MongoDB:**
|
||||
```javascript
|
||||
// No native fork/clone
|
||||
// Must manually export/import
|
||||
|
||||
// Export (slow)
|
||||
mongoexport --db mydb --collection users --out users.json
|
||||
|
||||
// Import to new collection (slow)
|
||||
mongoimport --db mydb --collection users_backup --file users.json
|
||||
```
|
||||
|
||||
**Brainy:**
|
||||
```javascript
|
||||
const fork = await brain.fork() // Done!
|
||||
```
|
||||
|
||||
**Winner: Brainy** (100x faster, built-in)
|
||||
|
||||
### vs Pinecone/Weaviate
|
||||
|
||||
**Pinecone/Weaviate:**
|
||||
```
|
||||
❌ No fork/clone feature at all
|
||||
❌ Manual backup/restore only
|
||||
❌ Downtime required for testing
|
||||
```
|
||||
|
||||
**Brainy:**
|
||||
```javascript
|
||||
✅ Fork in 1-2 seconds
|
||||
✅ Zero downtime
|
||||
✅ Zero risk
|
||||
```
|
||||
|
||||
**Winner: Brainy** (only vector DB with instant fork)
|
||||
|
||||
---
|
||||
|
||||
## What's Implemented vs. What's Next
|
||||
|
||||
### ✅ Available in - ✅ `fork()` - Instant clone in <100ms
|
||||
- ✅ `listBranches()` - List all forks
|
||||
- ✅ `getCurrentBranch()` - Get active branch
|
||||
- ✅ `checkout()` - Switch between branches
|
||||
- ✅ `deleteBranch()` - Delete branches
|
||||
- ✅ `commit()` - Create state snapshots
|
||||
- ✅ `getHistory()` - View commit history
|
||||
|
||||
### 🔮 Planned for:
|
||||
|
||||
**Temporal Features:**
|
||||
- `asOf(timestamp)` - Query data at specific time (✅ available)
|
||||
- `rollback(commitHash)` - Restore to previous state
|
||||
- Full audit trail for all changes
|
||||
|
||||
These features require additional temporal infrastructure and are being carefully designed
|
||||
|
||||
---
|
||||
|
||||
## CLI Support
|
||||
|
||||
All fork/merge/commit features are available via CLI:
|
||||
|
||||
```bash
|
||||
# Fork (instant clone)
|
||||
brainy fork feature-x --message "Testing new feature" --author "dev@example.com"
|
||||
|
||||
# List branches
|
||||
brainy branch list
|
||||
|
||||
# Switch branches
|
||||
brainy checkout feature-x
|
||||
|
||||
# Create commit
|
||||
brainy commit --message "Add new feature" --author "dev@example.com"
|
||||
|
||||
# View history
|
||||
brainy history --limit 10
|
||||
|
||||
# Merge branches
|
||||
brainy merge feature-x main --strategy last-write-wins
|
||||
|
||||
# Delete branch
|
||||
brainy branch delete old-feature --force
|
||||
```
|
||||
|
||||
## Try It Now
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/brainy
|
||||
```
|
||||
|
||||
```javascript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
await brain.add({ noun: 'test', data: { value: 1 } })
|
||||
|
||||
const fork = await brain.fork('experiment')
|
||||
|
||||
console.log('Fork created in < 2 seconds! 🚀')
|
||||
```
|
||||
|
||||
**Zero config. Zero complexity. Pure power.**
|
||||
|
||||
---
|
||||
|
||||
## Learn More
|
||||
|
||||
- [COW Architecture](../architecture/copy-on-write.md)
|
||||
- [Performance Benchmarks](../benchmarks/fork-performance.md)
|
||||
- [Enterprise Features](../enterprise/temporal-cloning.md)
|
||||
- [API Reference](../api/fork.md)
|
||||
|
||||
---
|
||||
|
||||
**Brainy** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy)
|
||||
296
docs/features/v3-features.md
Normal file
296
docs/features/v3-features.md
Normal file
|
|
@ -0,0 +1,296 @@
|
|||
# 🚀 Brainy - Production-Ready Features
|
||||
|
||||
> **Status**: All features listed here are IMPLEMENTED and TESTED
|
||||
|
||||
## 📊 Performance Metrics
|
||||
- **Search Latency**: <10ms for 10,000+ items
|
||||
- **Write Throughput**: 10,000+ ops/sec
|
||||
- **Memory Efficiency**: <500MB for 10K items
|
||||
- **Concurrent Operations**: 100+ simultaneous operations
|
||||
|
||||
## 🧠 Core Intelligence Features
|
||||
|
||||
### Triple Intelligence System ✅
|
||||
Unified query system combining three types of intelligence:
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
like: 'AI research', // Vector similarity search
|
||||
where: { year: 2024 }, // Metadata filtering
|
||||
connected: { to: authorId } // Graph relationships
|
||||
})
|
||||
```
|
||||
|
||||
### Intelligent Type Mapping ✅
|
||||
Prevents semantic degradation by intelligently inferring types:
|
||||
```typescript
|
||||
// Automatically infers 'person' from email field
|
||||
brain.add({ name: "John", email: "john@example.com" }, 'entity')
|
||||
// → Stored as type: 'person', not generic 'entity'
|
||||
```
|
||||
|
||||
### Neural Query Understanding ✅
|
||||
- 220+ embedded patterns for intent detection
|
||||
- Natural language query processing
|
||||
- Automatic query optimization
|
||||
- Pattern-based query rewriting
|
||||
|
||||
## 🏢 Enterprise Features
|
||||
|
||||
### Distributed Coordination ✅
|
||||
Raft consensus for multi-node deployments:
|
||||
```typescript
|
||||
import { DistributedCoordinator } from '@soulcraft/brainy'
|
||||
|
||||
const coordinator = createCoordinator({
|
||||
nodeId: 'node-1',
|
||||
peers: ['node-2', 'node-3'],
|
||||
electionTimeout: 500
|
||||
})
|
||||
// Automatic leader election and failover
|
||||
```
|
||||
|
||||
### Horizontal Sharding ✅
|
||||
Consistent hashing for data distribution:
|
||||
```typescript
|
||||
import { ShardManager } from '@soulcraft/brainy'
|
||||
|
||||
const shards = createShardManager({
|
||||
nodes: ['node-1', 'node-2', 'node-3'],
|
||||
replicationFactor: 2,
|
||||
virtualNodes: 150
|
||||
})
|
||||
// Automatic shard rebalancing on node changes
|
||||
```
|
||||
|
||||
### Read/Write Separation ✅
|
||||
Primary-replica architecture for scale:
|
||||
```typescript
|
||||
import { ReadWriteSeparation } from '@soulcraft/brainy'
|
||||
|
||||
const replication = createReadWriteSeparation({
|
||||
role: 'auto', // Automatic primary/replica detection
|
||||
consistencyLevel: 'strong', // or 'eventual'
|
||||
readPreference: 'nearest'
|
||||
})
|
||||
```
|
||||
|
||||
### Cross-Instance Cache Sync ✅
|
||||
Version vector-based cache synchronization:
|
||||
```typescript
|
||||
import { CacheSync } from '@soulcraft/brainy'
|
||||
|
||||
const cache = createCacheSync({
|
||||
nodeId: 'node-1',
|
||||
syncInterval: 100,
|
||||
conflictResolution: 'version-vector'
|
||||
})
|
||||
```
|
||||
|
||||
## 🔐 Security & Compliance
|
||||
|
||||
### Rate Limiting ✅
|
||||
Per-operation configurable limits:
|
||||
```typescript
|
||||
const rateLimiter = createRateLimitAugmentation({
|
||||
limits: {
|
||||
searches: 1000, // per minute
|
||||
writes: 100,
|
||||
reads: 5000,
|
||||
deletes: 50
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Audit Logging ✅
|
||||
Comprehensive operation tracking:
|
||||
```typescript
|
||||
const auditLogger = createAuditLogAugmentation({
|
||||
logLevel: 'detailed',
|
||||
retention: 90, // days
|
||||
includeMetadata: true
|
||||
})
|
||||
|
||||
// Query audit logs
|
||||
const logs = auditLogger.queryLogs({
|
||||
operation: 'add',
|
||||
startTime: Date.now() - 3600000
|
||||
})
|
||||
```
|
||||
|
||||
## 📦 Storage & Persistence
|
||||
|
||||
Full crash recovery and replay:
|
||||
```typescript
|
||||
enabled: true,
|
||||
checkpointInterval: 1000,
|
||||
maxLogSize: 100 * 1024 * 1024 // 100MB
|
||||
}))
|
||||
```
|
||||
|
||||
### Multi-Tenancy ✅
|
||||
Service-based data isolation:
|
||||
```typescript
|
||||
// Isolated data per service
|
||||
await brain.add(data, 'document', { service: 'tenant-1' })
|
||||
await brain.find('query', { service: 'tenant-1' })
|
||||
```
|
||||
|
||||
### Write-Only Mode ✅
|
||||
For dedicated write nodes:
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
mode: 'write-only',
|
||||
storage: 's3://bucket/path'
|
||||
})
|
||||
```
|
||||
|
||||
## 🚀 Performance Features
|
||||
|
||||
### Batch Operations ✅
|
||||
Optimized bulk processing:
|
||||
```typescript
|
||||
// Parallel processing with automatic batching
|
||||
await brain.addMany(items) // <10ms per item
|
||||
await brain.updateMany(updates)
|
||||
await brain.deleteMany(filters)
|
||||
```
|
||||
|
||||
### Request Deduplication ✅
|
||||
Automatic duplicate request handling:
|
||||
```typescript
|
||||
brain.use(new RequestDeduplicatorAugmentation())
|
||||
// Identical concurrent requests return same result
|
||||
```
|
||||
|
||||
### Smart Caching ✅
|
||||
Intelligent search result caching:
|
||||
```typescript
|
||||
brain.use(new CacheAugmentation({
|
||||
maxSize: 10000,
|
||||
ttl: 300000, // 5 minutes
|
||||
invalidateOnWrite: true
|
||||
}))
|
||||
```
|
||||
|
||||
## 🔄 Data Processing
|
||||
|
||||
### Entity Registry ✅
|
||||
Bloom filter-based deduplication:
|
||||
```typescript
|
||||
brain.use(new EntityRegistryAugmentation())
|
||||
// Handles millions of entities with minimal memory
|
||||
```
|
||||
|
||||
### Neural Import ✅
|
||||
Intelligent data import with type inference:
|
||||
```typescript
|
||||
await brain.import({
|
||||
source: 'data.json',
|
||||
autoDetectTypes: true,
|
||||
batchSize: 1000
|
||||
})
|
||||
```
|
||||
|
||||
### Streaming Pipeline ✅
|
||||
Real-time data processing:
|
||||
```typescript
|
||||
brain.stream()
|
||||
.pipe(transform)
|
||||
.pipe(enrich)
|
||||
.pipe(brain.writer())
|
||||
```
|
||||
|
||||
## 📊 Analytics & Monitoring
|
||||
|
||||
### Metrics Collection ✅
|
||||
Built-in performance metrics:
|
||||
```typescript
|
||||
const metrics = brain.getMetrics()
|
||||
// {
|
||||
// operations: { add: 1000, find: 5000 },
|
||||
// performance: { p95: 8, p99: 12 },
|
||||
// cache: { hits: 4500, misses: 500 }
|
||||
// }
|
||||
```
|
||||
|
||||
### Health Monitoring ✅
|
||||
Automatic health checks:
|
||||
```typescript
|
||||
const health = brain.getHealth()
|
||||
// {
|
||||
// status: 'healthy',
|
||||
// storage: 'connected',
|
||||
// memory: { used: 245, limit: 512 }
|
||||
// }
|
||||
```
|
||||
|
||||
## 🛠️ Developer Experience
|
||||
|
||||
### Zero Configuration ✅
|
||||
Works out of the box:
|
||||
```typescript
|
||||
import Brainy from '@soulcraft/brainy'
|
||||
const brain = new Brainy() // Auto-configures everything
|
||||
```
|
||||
|
||||
### TypeScript First ✅
|
||||
Full type safety and inference:
|
||||
```typescript
|
||||
// Types are automatically inferred
|
||||
const results = await brain.find<MyType>('query')
|
||||
```
|
||||
|
||||
### Augmentation System ✅
|
||||
Extensible plugin architecture:
|
||||
```typescript
|
||||
class CustomAugmentation extends BaseAugmentation {
|
||||
execute(operation, params, next) {
|
||||
// Your custom logic
|
||||
return next()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 Operational Features
|
||||
|
||||
### Graceful Shutdown ✅
|
||||
Clean shutdown with data persistence:
|
||||
```typescript
|
||||
process.on('SIGTERM', async () => {
|
||||
await brain.shutdown() // Saves all pending data
|
||||
})
|
||||
```
|
||||
|
||||
### Hot Reload ✅
|
||||
Configuration updates without restart:
|
||||
```typescript
|
||||
brain.updateConfig({
|
||||
cache: { enabled: false }
|
||||
})
|
||||
```
|
||||
|
||||
### Backup & Restore ✅
|
||||
Full data backup capabilities:
|
||||
```typescript
|
||||
await brain.backup('backup.bin')
|
||||
await brain.restore('backup.bin')
|
||||
```
|
||||
|
||||
## 📈 Proven at Scale
|
||||
|
||||
- **10,000+ items**: Sub-10ms search
|
||||
- **1M+ operations**: Stable memory usage
|
||||
- **100+ concurrent users**: No performance degradation
|
||||
- **Multi-node clusters**: Automatic failover
|
||||
|
||||
## 🚫 NOT Implemented (Planned)
|
||||
|
||||
These features are documented but NOT yet implemented:
|
||||
- GraphQL API (use REST API instead)
|
||||
- Kubernetes operators (use Docker)
|
||||
- Some distributed features require manual configuration
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: Latest Version*
|
||||
*All features listed above are production-ready and tested*
|
||||
|
|
@ -221,7 +221,7 @@ if (entity.vector.length > 0) {
|
|||
If you encounter migration issues:
|
||||
|
||||
1. Check the [VFS Performance Guide](../vfs/VFS_PERFORMANCE.md)
|
||||
2. Review [API Reference](../api/README.md)
|
||||
2. Review [API Reference](../API_REFERENCE.md)
|
||||
3. See [Performance Documentation](../PERFORMANCE.md)
|
||||
4. File an issue: https://github.com/soulcraft/brainy/issues
|
||||
|
||||
|
|
|
|||
|
|
@ -1,593 +0,0 @@
|
|||
# Aggregation Guide
|
||||
|
||||
> Real-time analytics on your entity data with incremental running totals
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy's aggregation engine computes running totals at write time, so reading aggregate results is always O(1) regardless of dataset size. Define an aggregate once, and every `add()`, `update()`, and `delete()` automatically updates the running metrics.
|
||||
|
||||
No batch jobs. No scheduled recalculations. Aggregates stay current with every write.
|
||||
|
||||
**Defining over existing data:** if you define an aggregate on a store that already holds
|
||||
matching entities, Brainy backfills it from those entities on the first query (a one-time scan,
|
||||
then purely incremental). So `defineAggregate()` behaves the same whether you define it before
|
||||
or after the data exists.
|
||||
|
||||
**Reopening a persisted brain:** aggregate state persists across restarts. Re-defining the
|
||||
same aggregate at boot (the normal declarative pattern) adopts the persisted state directly —
|
||||
no rescan. A backfill scan runs only when the definition actually changed, when no persisted
|
||||
state exists, or when the state failed to load; and however many aggregates need backfilling,
|
||||
they share a single scan.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// 1. Define an aggregate
|
||||
brain.defineAggregate({
|
||||
name: 'sales_by_category',
|
||||
source: { type: NounType.Event },
|
||||
groupBy: ['category'],
|
||||
metrics: {
|
||||
revenue: { op: 'sum', field: 'amount' },
|
||||
count: { op: 'count' },
|
||||
average: { op: 'avg', field: 'amount' }
|
||||
}
|
||||
})
|
||||
|
||||
// 2. Add entities — aggregates update automatically
|
||||
await brain.add({
|
||||
data: 'Coffee purchase',
|
||||
type: NounType.Event,
|
||||
metadata: { category: 'food', amount: 5.50 }
|
||||
})
|
||||
|
||||
await brain.add({
|
||||
data: 'Laptop purchase',
|
||||
type: NounType.Event,
|
||||
metadata: { category: 'electronics', amount: 1200 }
|
||||
})
|
||||
|
||||
await brain.add({
|
||||
data: 'Lunch purchase',
|
||||
type: NounType.Event,
|
||||
metadata: { category: 'food', amount: 12.00 }
|
||||
})
|
||||
|
||||
// 3. Query results
|
||||
const results = await brain.find({ aggregate: 'sales_by_category' })
|
||||
|
||||
// Results:
|
||||
// [
|
||||
// { groupKey: { category: 'food' }, metrics: { revenue: 17.50, count: 2, average: 8.75 } },
|
||||
// { groupKey: { category: 'electronics' }, metrics: { revenue: 1200, count: 1, average: 1200 } }
|
||||
// ]
|
||||
```
|
||||
|
||||
## Aggregation Operations
|
||||
|
||||
Brainy supports 7 aggregation operations:
|
||||
|
||||
### `sum` — Running Total
|
||||
|
||||
Adds up all values of a numeric field.
|
||||
|
||||
```typescript
|
||||
metrics: {
|
||||
total_revenue: { op: 'sum', field: 'amount' }
|
||||
}
|
||||
```
|
||||
|
||||
### `count` — Entity Count
|
||||
|
||||
Counts the number of matching entities. No `field` required.
|
||||
|
||||
```typescript
|
||||
metrics: {
|
||||
order_count: { op: 'count' }
|
||||
}
|
||||
```
|
||||
|
||||
### `avg` — Running Average
|
||||
|
||||
Computes `sum / count` incrementally.
|
||||
|
||||
```typescript
|
||||
metrics: {
|
||||
average_price: { op: 'avg', field: 'price' }
|
||||
}
|
||||
```
|
||||
|
||||
### `min` — Minimum Value
|
||||
|
||||
Tracks the minimum value across all entities in each group.
|
||||
|
||||
```typescript
|
||||
metrics: {
|
||||
lowest_price: { op: 'min', field: 'price' }
|
||||
}
|
||||
```
|
||||
|
||||
### `max` — Maximum Value
|
||||
|
||||
Tracks the maximum value across all entities in each group.
|
||||
|
||||
```typescript
|
||||
metrics: {
|
||||
highest_price: { op: 'max', field: 'price' }
|
||||
}
|
||||
```
|
||||
|
||||
### `stddev` — Sample Standard Deviation
|
||||
|
||||
Computes the sample standard deviation using Welford's numerically stable online algorithm. Updates incrementally without storing individual values.
|
||||
|
||||
```typescript
|
||||
metrics: {
|
||||
price_spread: { op: 'stddev', field: 'price' }
|
||||
}
|
||||
```
|
||||
|
||||
### `variance` — Sample Variance
|
||||
|
||||
Computes the sample variance (square of standard deviation) using Welford's online algorithm.
|
||||
|
||||
```typescript
|
||||
metrics: {
|
||||
price_variance: { op: 'variance', field: 'price' }
|
||||
}
|
||||
```
|
||||
|
||||
## GROUP BY Dimensions
|
||||
|
||||
Every aggregate requires at least one `groupBy` dimension. Results are grouped by the unique combinations of dimension values.
|
||||
|
||||
### Plain Fields
|
||||
|
||||
Group by a metadata field value:
|
||||
|
||||
```typescript
|
||||
groupBy: ['category']
|
||||
// Produces groups: { category: 'food' }, { category: 'electronics' }, ...
|
||||
```
|
||||
|
||||
### Multiple Fields
|
||||
|
||||
Group by multiple fields for composite keys:
|
||||
|
||||
```typescript
|
||||
groupBy: ['category', 'region']
|
||||
// Produces groups: { category: 'food', region: 'US' }, { category: 'food', region: 'EU' }, ...
|
||||
```
|
||||
|
||||
### Time Windows
|
||||
|
||||
Group by a timestamp field bucketed into time periods:
|
||||
|
||||
```typescript
|
||||
groupBy: [{ field: 'date', window: 'month' }]
|
||||
// Produces groups: { date: '2024-01' }, { date: '2024-02' }, ...
|
||||
```
|
||||
|
||||
Available time window granularities:
|
||||
|
||||
| Window | Format | Example |
|
||||
|--------|--------|---------|
|
||||
| `hour` | `YYYY-MM-DDThh` | `2024-01-15T14` |
|
||||
| `day` | `YYYY-MM-DD` | `2024-01-15` |
|
||||
| `week` | `YYYY-Wnn` | `2024-W03` |
|
||||
| `month` | `YYYY-MM` | `2024-01` |
|
||||
| `quarter` | `YYYY-Qn` | `2024-Q1` |
|
||||
| `year` | `YYYY` | `2024` |
|
||||
| `{ seconds: N }` | ISO 8601 | Custom interval |
|
||||
|
||||
### Combined Dimensions
|
||||
|
||||
Mix plain fields and time windows:
|
||||
|
||||
```typescript
|
||||
brain.defineAggregate({
|
||||
name: 'monthly_sales',
|
||||
source: { type: NounType.Event },
|
||||
groupBy: ['region', { field: 'date', window: 'month' }],
|
||||
metrics: {
|
||||
revenue: { op: 'sum', field: 'amount' },
|
||||
count: { op: 'count' }
|
||||
}
|
||||
})
|
||||
|
||||
// Produces groups like:
|
||||
// { region: 'US', date: '2024-01' }
|
||||
// { region: 'US', date: '2024-02' }
|
||||
// { region: 'EU', date: '2024-01' }
|
||||
```
|
||||
|
||||
### Array Fields (Unnest)
|
||||
|
||||
Group by **each element** of an array-valued field — for tag frequencies, label counts, and
|
||||
faceted breakdowns. Mark the dimension `{ field, unnest: true }`:
|
||||
|
||||
```typescript
|
||||
brain.defineAggregate({
|
||||
name: 'tag_frequency',
|
||||
source: { type: NounType.Document },
|
||||
groupBy: [{ field: 'tags', unnest: true }],
|
||||
metrics: { count: { op: 'count' } }
|
||||
})
|
||||
|
||||
// A document tagged ['ml', 'ai'] contributes once to the 'ml' group and once to 'ai'.
|
||||
// Duplicate tags on one entity count once; an entity with no tags joins no group.
|
||||
const top = await brain.queryAggregate('tag_frequency', { orderBy: 'count', order: 'desc' })
|
||||
// [ { groupKey: { tags: 'ai' }, metrics: { count: 3 }, count: 3 }, ... ]
|
||||
```
|
||||
|
||||
## Querying Aggregates
|
||||
|
||||
Aggregate results are queried through the standard `find()` method.
|
||||
|
||||
### Basic Query
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({ aggregate: 'sales_by_category' })
|
||||
```
|
||||
|
||||
### Filter by Group Key
|
||||
|
||||
Use `where` to filter on group key values:
|
||||
|
||||
```typescript
|
||||
const foodOnly = await brain.find({
|
||||
aggregate: 'sales_by_category',
|
||||
where: { category: 'food' }
|
||||
})
|
||||
```
|
||||
|
||||
### Filter by Metric Value (HAVING)
|
||||
|
||||
Use `having` to filter groups by their **computed metric values** — the analytics equivalent of
|
||||
SQL `HAVING`. (`where` filters group *keys*; `having` filters *metrics*.)
|
||||
|
||||
```typescript
|
||||
const bigCategories = await brain.find({
|
||||
aggregate: 'sales_by_category',
|
||||
having: { revenue: { greaterThan: 1000 } }
|
||||
})
|
||||
```
|
||||
|
||||
`having` accepts the same operators as `where`, applied to each group's metric results plus
|
||||
`count`. It is evaluated per group — **O(groups), independent of entity count** — before sorting
|
||||
and pagination, so it stays cheap even over billions of entities.
|
||||
|
||||
### Sort and Paginate
|
||||
|
||||
Sort by any metric or group key field:
|
||||
|
||||
```typescript
|
||||
const topCategories = await brain.find({
|
||||
aggregate: {
|
||||
name: 'sales_by_category',
|
||||
orderBy: 'revenue',
|
||||
order: 'desc',
|
||||
limit: 10
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Combined Parameters
|
||||
|
||||
`where`, `orderBy`, `limit`, and `offset` from the outer `find()` call merge automatically with the aggregate query:
|
||||
|
||||
```typescript
|
||||
const recentTopSpenders = await brain.find({
|
||||
aggregate: 'monthly_sales',
|
||||
where: { region: 'US' },
|
||||
orderBy: 'revenue',
|
||||
order: 'desc',
|
||||
limit: 12,
|
||||
offset: 0
|
||||
})
|
||||
```
|
||||
|
||||
### Result Format
|
||||
|
||||
`find({ aggregate })` returns `Result<T>` rows (for uniformity with the rest of `find()`),
|
||||
with the aggregate fields surfaced **both** at the top level and, for backward compatibility,
|
||||
flattened into `metadata`:
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: string,
|
||||
score: 1.0,
|
||||
type: NounType.Measurement,
|
||||
groupKey: { category: 'food' }, // top-level — the group key values
|
||||
metrics: { revenue: 17.50, count: 2, average: 8.75 }, // top-level — computed metrics
|
||||
count: 2, // top-level — entities in the group
|
||||
metadata: { // legacy mirror of the same data
|
||||
__aggregate: 'sales_by_category',
|
||||
category: 'food',
|
||||
revenue: 17.50, count: 2, average: 8.75
|
||||
},
|
||||
entity: Entity
|
||||
}
|
||||
```
|
||||
|
||||
### `queryAggregate()` — the report-friendly view
|
||||
|
||||
For dashboards and reports, prefer `brain.queryAggregate(name, params)`. It returns the clean
|
||||
`AggregateResult[]` shape directly — no search-result wrapper:
|
||||
|
||||
```typescript
|
||||
const rows = await brain.queryAggregate('sales_by_category', {
|
||||
orderBy: 'revenue',
|
||||
order: 'desc',
|
||||
limit: 10
|
||||
})
|
||||
// [
|
||||
// { groupKey: { category: 'electronics' }, metrics: { revenue: 1200, count: 1, average: 1200 }, count: 1 },
|
||||
// { groupKey: { category: 'food' }, metrics: { revenue: 17.50, count: 2, average: 8.75 }, count: 2 }
|
||||
// ]
|
||||
```
|
||||
|
||||
It accepts the same `where` / `having` / `orderBy` / `order` / `limit` / `offset` params as the
|
||||
`find({ aggregate })` form.
|
||||
|
||||
## Source Filtering
|
||||
|
||||
Control which entities feed into an aggregate with the `source` property.
|
||||
|
||||
### Filter by Entity Type
|
||||
|
||||
```typescript
|
||||
brain.defineAggregate({
|
||||
name: 'event_stats',
|
||||
source: { type: NounType.Event },
|
||||
groupBy: ['category'],
|
||||
metrics: { count: { op: 'count' } }
|
||||
})
|
||||
```
|
||||
|
||||
### Filter by Multiple Types
|
||||
|
||||
```typescript
|
||||
source: { type: [NounType.Event, NounType.Document] }
|
||||
```
|
||||
|
||||
### Filter by Metadata
|
||||
|
||||
Use the same `where` syntax as `find()`:
|
||||
|
||||
```typescript
|
||||
source: {
|
||||
type: NounType.Event,
|
||||
where: { domain: 'financial', subtype: 'transaction' }
|
||||
}
|
||||
```
|
||||
|
||||
### Filter by Service
|
||||
|
||||
For multi-tenant deployments:
|
||||
|
||||
```typescript
|
||||
source: { service: 'tenant-123' }
|
||||
```
|
||||
|
||||
Entities that don't match the source filter are silently skipped during incremental updates.
|
||||
|
||||
## Incremental Updates
|
||||
|
||||
The aggregation engine hooks into every write operation:
|
||||
|
||||
### On `add()`
|
||||
|
||||
When a new entity matches an aggregate's source filter:
|
||||
1. The group key is computed from the entity's metadata
|
||||
2. Each metric in the matching group is incremented
|
||||
3. New groups are created automatically
|
||||
|
||||
### On `update()`
|
||||
|
||||
When an existing entity is updated:
|
||||
1. The old entity's contribution is reversed from its group
|
||||
2. The new entity's contribution is applied to its (potentially different) group
|
||||
3. Handles group key changes — an entity moving from category "food" to "drink" updates both groups
|
||||
|
||||
### On `delete()`
|
||||
|
||||
When an entity is deleted:
|
||||
1. The entity's contribution is reversed from its group
|
||||
2. If a group becomes empty (all metric counts reach zero), it's removed
|
||||
|
||||
### Aggregate Entity Exclusion
|
||||
|
||||
Materialized `NounType.Measurement` entities are automatically excluded from all source matching, preventing infinite feedback loops. Entities with `service: 'brainy:aggregation'` or `metadata.__aggregate` are always skipped.
|
||||
|
||||
## Materialization
|
||||
|
||||
Materialization writes aggregate results as `NounType.Measurement` entities, making them automatically available through OData, Google Sheets, SSE, and webhook integrations.
|
||||
|
||||
```typescript
|
||||
brain.defineAggregate({
|
||||
name: 'daily_metrics',
|
||||
source: { type: NounType.Event },
|
||||
groupBy: [{ field: 'date', window: 'day' }],
|
||||
metrics: {
|
||||
total: { op: 'sum', field: 'amount' },
|
||||
count: { op: 'count' }
|
||||
},
|
||||
materialize: true
|
||||
})
|
||||
```
|
||||
|
||||
### Debounce Configuration
|
||||
|
||||
During high-throughput ingestion, materialization is debounced to avoid excessive writes:
|
||||
|
||||
```typescript
|
||||
materialize: {
|
||||
debounceMs: 2000, // Wait 2 seconds after last update before writing
|
||||
trackSources: true // Track which entities contributed
|
||||
}
|
||||
```
|
||||
|
||||
The default debounce interval is 1000ms.
|
||||
|
||||
## Multiple Aggregates
|
||||
|
||||
Define multiple aggregates that process the same entities:
|
||||
|
||||
```typescript
|
||||
// Revenue by category
|
||||
brain.defineAggregate({
|
||||
name: 'category_revenue',
|
||||
source: { type: NounType.Event },
|
||||
groupBy: ['category'],
|
||||
metrics: { total: { op: 'sum', field: 'amount' } }
|
||||
})
|
||||
|
||||
// Monthly trends
|
||||
brain.defineAggregate({
|
||||
name: 'monthly_trends',
|
||||
source: { type: NounType.Event },
|
||||
groupBy: [{ field: 'date', window: 'month' }],
|
||||
metrics: {
|
||||
revenue: { op: 'sum', field: 'amount' },
|
||||
count: { op: 'count' },
|
||||
avg_order: { op: 'avg', field: 'amount' }
|
||||
}
|
||||
})
|
||||
|
||||
// Regional breakdown with statistical analysis
|
||||
brain.defineAggregate({
|
||||
name: 'regional_analysis',
|
||||
source: { type: NounType.Event },
|
||||
groupBy: ['region'],
|
||||
metrics: {
|
||||
revenue: { op: 'sum', field: 'amount' },
|
||||
spread: { op: 'stddev', field: 'amount' },
|
||||
variance: { op: 'variance', field: 'amount' }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
Each `add()` call updates all matching aggregates automatically.
|
||||
|
||||
## Removing Aggregates
|
||||
|
||||
Remove an aggregate and clean up its state:
|
||||
|
||||
```typescript
|
||||
brain.removeAggregate('category_revenue')
|
||||
```
|
||||
|
||||
## Persistence
|
||||
|
||||
Aggregate definitions and running state are automatically persisted:
|
||||
|
||||
- **On `flush()`/`close()`**: All dirty aggregate state is written to storage
|
||||
- **On `init()`**: Definitions and state are restored from storage
|
||||
- **Change detection**: Definition changes are detected via FNV-1a hashing — only changed aggregates reset their state on restart
|
||||
|
||||
## Native Acceleration
|
||||
|
||||
When [Cor](https://www.npmjs.com/package/@soulcraft/cor) is installed as a plugin, the aggregation engine automatically uses Rust-accelerated computation:
|
||||
|
||||
- Incremental updates run in Rust with BTreeMap-backed precise MIN/MAX
|
||||
- Welford's online stddev/variance computed natively
|
||||
- Rebuild uses Rayon parallel iterators across CPU cores (above 1,000 entities)
|
||||
- Time window bucketing uses integer arithmetic without `Date` object allocation
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
plugins: ['@soulcraft/cor']
|
||||
})
|
||||
await brain.init()
|
||||
|
||||
// Aggregation automatically uses native engine
|
||||
brain.defineAggregate({ ... })
|
||||
```
|
||||
|
||||
Verify native acceleration is active:
|
||||
|
||||
```typescript
|
||||
const diag = brain.diagnostics()
|
||||
console.log(diag.providers.aggregation)
|
||||
// { source: 'plugin' }
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Financial Analytics
|
||||
|
||||
```typescript
|
||||
brain.defineAggregate({
|
||||
name: 'monthly_spending',
|
||||
source: {
|
||||
type: NounType.Event,
|
||||
where: { domain: 'financial', subtype: 'transaction' }
|
||||
},
|
||||
groupBy: [
|
||||
'category',
|
||||
{ field: 'date', window: 'month' }
|
||||
],
|
||||
metrics: {
|
||||
total: { op: 'sum', field: 'amount' },
|
||||
count: { op: 'count' },
|
||||
average: { op: 'avg', field: 'amount' },
|
||||
highest: { op: 'max', field: 'amount' },
|
||||
lowest: { op: 'min', field: 'amount' }
|
||||
},
|
||||
materialize: true
|
||||
})
|
||||
```
|
||||
|
||||
### Time-Series Monitoring
|
||||
|
||||
```typescript
|
||||
brain.defineAggregate({
|
||||
name: 'hourly_metrics',
|
||||
source: { type: NounType.Event, where: { domain: 'monitoring' } },
|
||||
groupBy: [
|
||||
'service',
|
||||
{ field: 'timestamp', window: 'hour' }
|
||||
],
|
||||
metrics: {
|
||||
request_count: { op: 'count' },
|
||||
avg_latency: { op: 'avg', field: 'latency_ms' },
|
||||
max_latency: { op: 'max', field: 'latency_ms' },
|
||||
error_count: { op: 'sum', field: 'is_error' },
|
||||
latency_spread: { op: 'stddev', field: 'latency_ms' }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Content Analytics
|
||||
|
||||
```typescript
|
||||
brain.defineAggregate({
|
||||
name: 'content_stats',
|
||||
source: { type: NounType.Document },
|
||||
groupBy: ['author', { field: 'publishedAt', window: 'month' }],
|
||||
metrics: {
|
||||
articles: { op: 'count' },
|
||||
total_words: { op: 'sum', field: 'wordCount' },
|
||||
avg_words: { op: 'avg', field: 'wordCount' }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
Aggregation complexity per write is O(A x G x M) where A = matching aggregates, G = groupBy dimensions, M = metrics. For typical configurations (2-5 aggregates, 1-3 dimensions, 3-5 metrics), this is effectively O(1).
|
||||
|
||||
With Cor native acceleration:
|
||||
|
||||
| Operation | Throughput | Latency |
|
||||
|-----------|-----------|---------|
|
||||
| Incremental update (1K entities) | 809 ops/s | 1.2 ms |
|
||||
| Rebuild (10K entities) | 475 ops/s | 2.1 ms |
|
||||
| Rebuild (100K entities, Rayon) | 66 ops/s | 15.2 ms |
|
||||
| Query (1K groups, sort + paginate) | 986 ops/s | 1.0 ms |
|
||||
406
docs/guides/distributed-system.md
Normal file
406
docs/guides/distributed-system.md
Normal file
|
|
@ -0,0 +1,406 @@
|
|||
# Distributed Brainy System Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy introduces a groundbreaking **zero-configuration distributed system** that transforms how vector databases scale. Unlike traditional distributed databases that require complex setup (Consul, etcd, Zookeeper), Brainy uses your existing storage (S3, GCS, R2) as the coordination layer.
|
||||
|
||||
## Key Innovation: Storage-Based Coordination
|
||||
|
||||
Instead of requiring separate infrastructure for coordination, Brainy leverages your storage backend:
|
||||
|
||||
```typescript
|
||||
// Traditional distributed database setup:
|
||||
// ❌ Setup Consul/etcd
|
||||
// ❌ Configure node discovery
|
||||
// ❌ Setup health checks
|
||||
// ❌ Configure sharding
|
||||
// ❌ Setup replication
|
||||
|
||||
// Brainy distributed setup:
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: { bucket: 'my-data' }
|
||||
},
|
||||
distributed: true // ✅ That's it!
|
||||
})
|
||||
```
|
||||
|
||||
## Real-World Scenarios
|
||||
|
||||
### 1. Processing Multiple Streaming Data Sources (Bluesky, Twitter, etc.)
|
||||
|
||||
**Problem**: You need to ingest millions of posts from multiple social media firehoses simultaneously while maintaining search performance.
|
||||
|
||||
**Traditional Approach**:
|
||||
- Separate ingestion and search clusters
|
||||
- Complex queue systems (Kafka, RabbitMQ)
|
||||
- Manual sharding configuration
|
||||
- Complicated backpressure handling
|
||||
|
||||
**Brainy Solution**:
|
||||
```typescript
|
||||
// Node 1: Bluesky Ingestion
|
||||
const ingestionNode1 = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'social-data' }},
|
||||
distributed: true,
|
||||
writeOnly: true // Optimized for writes
|
||||
})
|
||||
|
||||
// Continuously ingest Bluesky firehose
|
||||
blueskyStream.on('post', async (post) => {
|
||||
await ingestionNode1.add({
|
||||
content: post.text,
|
||||
author: post.author,
|
||||
timestamp: post.createdAt,
|
||||
platform: 'bluesky'
|
||||
}, 'social-post')
|
||||
})
|
||||
|
||||
// Node 2: Twitter Ingestion (separate machine)
|
||||
const ingestionNode2 = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'social-data' }},
|
||||
distributed: true,
|
||||
writeOnly: true
|
||||
})
|
||||
|
||||
// Node 3-5: Search nodes (auto-balanced)
|
||||
const searchNode = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'social-data' }},
|
||||
distributed: true,
|
||||
readOnly: true // Optimized for queries
|
||||
})
|
||||
|
||||
// Search across ALL data from ALL sources
|
||||
const results = await searchNode.find('AI trends', 100)
|
||||
// Automatically queries all shards across all nodes!
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- **Auto-sharding**: Data automatically distributed by content hash
|
||||
- **No bottlenecks**: Each ingestion node writes directly to storage
|
||||
- **Live search**: Search nodes see new data immediately
|
||||
- **Auto-scaling**: Add nodes anytime, data rebalances automatically
|
||||
|
||||
### 2. Multi-Tenant SaaS Application
|
||||
|
||||
**Problem**: Each customer needs isolated, fast search across their documents, with ability to scale per customer.
|
||||
|
||||
**Brainy Solution**:
|
||||
```typescript
|
||||
// Spin up dedicated nodes per large customer
|
||||
const enterpriseNode = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: {
|
||||
bucket: 'customer-data',
|
||||
prefix: 'customer-123/' // Isolated data
|
||||
}
|
||||
},
|
||||
distributed: true
|
||||
})
|
||||
|
||||
// Shared nodes for smaller customers
|
||||
const sharedNode = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: { bucket: 'shared-customers' }
|
||||
},
|
||||
distributed: true
|
||||
})
|
||||
|
||||
// Domain-based sharding ensures customer data stays together
|
||||
await sharedNode.add(document, 'document', {
|
||||
customerId: 'cust-456', // Used for shard assignment
|
||||
domain: 'customer-456' // Keeps related data together
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Global Knowledge Graph with Local Inference
|
||||
|
||||
**Problem**: Building a knowledge graph that needs both global connectivity and local LLM inference.
|
||||
|
||||
**Brainy Solution**:
|
||||
```typescript
|
||||
// Edge nodes near users (with GPU)
|
||||
const edgeNode = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'knowledge-graph' }},
|
||||
distributed: true,
|
||||
models: {
|
||||
embed: { model: 'BAAI/bge-base-en-v1.5' },
|
||||
chat: { model: 'meta-llama/Llama-3.2-3B-Instruct' }
|
||||
}
|
||||
})
|
||||
|
||||
// Central nodes for graph operations
|
||||
const graphNode = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'knowledge-graph' }},
|
||||
distributed: true,
|
||||
augmentations: ['graph', 'triple-intelligence']
|
||||
})
|
||||
|
||||
// Local inference with global knowledge
|
||||
const context = await edgeNode.find(userQuery, 10)
|
||||
const response = await edgeNode.chat(userQuery, { context })
|
||||
|
||||
// Graph traversal across all nodes
|
||||
const connections = await graphNode.traverse({
|
||||
start: 'concept:AI',
|
||||
depth: 3,
|
||||
relationship: 'related-to'
|
||||
})
|
||||
```
|
||||
|
||||
## Competitive Advantages
|
||||
|
||||
### vs. Pinecone/Weaviate/Qdrant
|
||||
|
||||
| Feature | Traditional Vector DBs | Brainy Distributed |
|
||||
|---------|----------------------|-------------------|
|
||||
| Setup Complexity | High (k8s, operators) | Zero (just storage) |
|
||||
| Minimum Nodes | 3-5 for HA | 1 (scale as needed) |
|
||||
| Coordination | External (etcd, Consul) | Built-in (via storage) |
|
||||
| Data Locality | Random sharding | Domain-aware sharding |
|
||||
| Query Planning | Basic | Triple Intelligence |
|
||||
| Cost at Scale | High (always-on clusters) | Low (scale to zero) |
|
||||
|
||||
### vs. Neo4j/ArangoDB (Graph Databases)
|
||||
|
||||
| Feature | Graph Databases | Brainy Distributed |
|
||||
|---------|----------------|-------------------|
|
||||
| Vector Search | Bolt-on/Limited | Native HNSW |
|
||||
| Embedding Generation | External | Built-in (30+ models) |
|
||||
| Distributed Transactions | Complex/Slow | Eventually consistent |
|
||||
| Natural Language | No | Native (Triple Intelligence) |
|
||||
| Setup | Very Complex | Zero config |
|
||||
|
||||
### vs. Elasticsearch/OpenSearch
|
||||
|
||||
| Feature | Elasticsearch | Brainy Distributed |
|
||||
|---------|--------------|-------------------|
|
||||
| Vector Support | Added later (slow) | Native (fast HNSW) |
|
||||
| Cluster Management | Complex (master nodes) | Automatic (via storage) |
|
||||
| Shard Rebalancing | Manual/Risky | Automatic/Safe |
|
||||
| Memory Usage | Very High | Efficient |
|
||||
| Query Language | Complex DSL | Natural language |
|
||||
|
||||
## Innovative Features
|
||||
|
||||
### 1. Domain-Aware Sharding
|
||||
|
||||
Unlike hash-based sharding, Brainy understands data relationships:
|
||||
|
||||
```typescript
|
||||
// Documents about the same topic stay on the same shard
|
||||
await brain.add(doc1, 'document', { domain: 'physics' })
|
||||
await brain.add(doc2, 'document', { domain: 'physics' })
|
||||
// Both documents on same shard = faster related queries
|
||||
|
||||
// Customer data stays together
|
||||
await brain.add(order, 'order', { customerId: 'cust-123' })
|
||||
await brain.add(invoice, 'invoice', { customerId: 'cust-123' })
|
||||
// Same customer = same shard = better locality
|
||||
```
|
||||
|
||||
### 2. Streaming Shard Migration
|
||||
|
||||
Zero-downtime data movement between nodes:
|
||||
|
||||
```typescript
|
||||
// Automatically triggered when nodes join/leave
|
||||
// Uses HTTP streaming for efficiency
|
||||
// Validates data integrity
|
||||
// Atomic ownership transfer
|
||||
// No query downtime!
|
||||
```
|
||||
|
||||
### 3. Storage-Based Consensus
|
||||
|
||||
No Raft/Paxos complexity:
|
||||
|
||||
```typescript
|
||||
// Leader election via storage atomic operations
|
||||
// Health monitoring via storage heartbeats
|
||||
// Configuration consensus via storage CAS
|
||||
// No split-brain issues!
|
||||
```
|
||||
|
||||
### 4. Intelligent Query Planning
|
||||
|
||||
The distributed query planner understands:
|
||||
- Which shards contain relevant data
|
||||
- Node health and latency
|
||||
- Data locality and caching
|
||||
- Triple Intelligence scoring
|
||||
|
||||
```typescript
|
||||
// Automatically optimizes query execution
|
||||
const results = await brain.find('quantum physics')
|
||||
// Planner knows:
|
||||
// - Physics domain → shard-3
|
||||
// - Node-2 has shard-3 cached
|
||||
// - Route query to node-2
|
||||
// - Merge results with Triple Intelligence
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Scalability
|
||||
- **Horizontal**: Add nodes anytime
|
||||
- **Vertical**: Nodes can differ in size
|
||||
- **Geographic**: Nodes can be globally distributed
|
||||
- **Elastic**: Scale to zero when idle
|
||||
|
||||
### Throughput
|
||||
- **Writes**: Linear scaling with nodes
|
||||
- **Reads**: Sub-linear (due to caching)
|
||||
- **Mixed**: Read/write optimized nodes
|
||||
|
||||
### Latency
|
||||
- **Local queries**: ~10ms p50
|
||||
- **Distributed queries**: ~50ms p50
|
||||
- **Shard migration**: Streaming (no bulk pause)
|
||||
|
||||
## Use Cases Where Brainy Excels
|
||||
|
||||
### ✅ Perfect For:
|
||||
|
||||
1. **Multi-source data ingestion** (social media, logs, events)
|
||||
2. **Global search applications** (distributed teams)
|
||||
3. **Multi-tenant SaaS** (customer isolation)
|
||||
4. **Knowledge graphs** (with vector search)
|
||||
5. **Edge AI applications** (local inference, global knowledge)
|
||||
6. **Document intelligence** (contracts, research papers)
|
||||
7. **Real-time analytics** (streaming + search)
|
||||
|
||||
### ⚠️ Consider Alternatives For:
|
||||
|
||||
1. **Strong consistency requirements** (use PostgreSQL)
|
||||
2. **Sub-millisecond latency** (use Redis)
|
||||
3. **Complex transactions** (use traditional RDBMS)
|
||||
4. **Purely structured data** (use columnar stores)
|
||||
|
||||
## Migration from Other Systems
|
||||
|
||||
### From Pinecone/Weaviate:
|
||||
```typescript
|
||||
// Your existing vector search still works
|
||||
const results = await brain.search(embedding, 10)
|
||||
|
||||
// But now you can scale horizontally!
|
||||
// And add graph relationships!
|
||||
// And use natural language!
|
||||
```
|
||||
|
||||
### From Elasticsearch:
|
||||
```typescript
|
||||
// Import your documents
|
||||
await brain.import('./elasticsearch-export.json')
|
||||
|
||||
// Queries are simpler
|
||||
const results = await brain.find('user query')
|
||||
// No complex DSL needed!
|
||||
```
|
||||
|
||||
### From Neo4j:
|
||||
```typescript
|
||||
// Import your graph
|
||||
await brain.importGraph('./neo4j-export.cypher')
|
||||
|
||||
// Now with vector search!
|
||||
const similar = await brain.find('concepts like quantum computing')
|
||||
```
|
||||
|
||||
## Deployment Patterns
|
||||
|
||||
### 1. Start Simple, Scale Later
|
||||
```typescript
|
||||
// Day 1: Single node
|
||||
const brain = new Brainy({ storage: 's3' })
|
||||
|
||||
// Month 2: Growing data, add distribution
|
||||
const brain = new Brainy({
|
||||
storage: 's3',
|
||||
distributed: true // Just add this!
|
||||
})
|
||||
|
||||
// Month 6: Multiple nodes auto-balance
|
||||
// No migration needed!
|
||||
```
|
||||
|
||||
### 2. Geographic Distribution
|
||||
```typescript
|
||||
// US Node
|
||||
const usNode = new Brainy({
|
||||
storage: { region: 'us-east-1' },
|
||||
distributed: true
|
||||
})
|
||||
|
||||
// EU Node
|
||||
const euNode = new Brainy({
|
||||
storage: { region: 'eu-west-1' },
|
||||
distributed: true
|
||||
})
|
||||
|
||||
// They automatically coordinate!
|
||||
```
|
||||
|
||||
### 3. Specialized Nodes
|
||||
```typescript
|
||||
// GPU nodes for embedding
|
||||
const embedNode = new Brainy({
|
||||
distributed: true,
|
||||
writeOnly: true,
|
||||
models: { embed: 'large-model' }
|
||||
})
|
||||
|
||||
// CPU nodes for search
|
||||
const searchNode = new Brainy({
|
||||
distributed: true,
|
||||
readOnly: true
|
||||
})
|
||||
```
|
||||
|
||||
## Monitoring & Operations
|
||||
|
||||
### Health Checks
|
||||
```typescript
|
||||
const health = await brain.getClusterHealth()
|
||||
// {
|
||||
// nodes: 5,
|
||||
// healthy: 5,
|
||||
// shards: 16,
|
||||
// status: 'green'
|
||||
// }
|
||||
```
|
||||
|
||||
### Shard Distribution
|
||||
```typescript
|
||||
const shards = await brain.getShardDistribution()
|
||||
// Shows which nodes own which shards
|
||||
```
|
||||
|
||||
### Migration Status
|
||||
```typescript
|
||||
const migrations = await brain.getActiveMigrations()
|
||||
// Shows ongoing shard movements
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
Brainy's distributed system is **production-ready** and offers:
|
||||
|
||||
1. **True zero-configuration** - Just add `distributed: true`
|
||||
2. **Storage-based coordination** - No external dependencies
|
||||
3. **Intelligent sharding** - Domain-aware data placement
|
||||
4. **Automatic operations** - Rebalancing, failover, scaling
|
||||
5. **Unified interface** - Vector + Graph + Document + LLM
|
||||
|
||||
This is not just another distributed database. It's a fundamental rethinking of how distributed systems should work in the cloud era.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. [Try the distributed quick start](./distributed-quickstart.md)
|
||||
2. [Read the architecture deep dive](../architecture/distributed-storage.md)
|
||||
3. [View benchmarks](../benchmarks/distributed-performance.md)
|
||||
4. [Deploy to production](./production-deployment.md)
|
||||
|
|
@ -165,28 +165,32 @@ await brain.syncWith({
|
|||
- **Webhook support**: React to changes
|
||||
- **API generation**: Auto-generate REST/GraphQL APIs
|
||||
|
||||
### 🌍 Scale
|
||||
### 🌍 Enterprise Scale 🚧 Coming Soon
|
||||
|
||||
**Everyone gets the same scale model:**
|
||||
**Everyone gets planetary scale:**
|
||||
|
||||
```typescript
|
||||
// Pure JS by default; install the optional native provider for billions of vectors
|
||||
const brain = new Brainy()
|
||||
// Same architecture Netflix uses, free for you
|
||||
const brain = new Brainy({
|
||||
clustering: {
|
||||
enabled: true, // Distributed mode
|
||||
sharding: 'automatic', // Auto-sharding
|
||||
replication: 3, // Triple replication
|
||||
consensus: 'raft', // Strong consistency
|
||||
geoDistribution: true // Multi-region support
|
||||
}
|
||||
})
|
||||
|
||||
// 1 → ~1M vectors: pure-JS HNSW, zero extra setup
|
||||
// 1M → 10B+ vectors: install @soulcraft/cor for the native DiskANN provider
|
||||
// Handles everything from 1 to 1 billion entities
|
||||
```
|
||||
|
||||
**Scaling model:**
|
||||
- **Single process, no cluster**: Brainy runs in one process — no coordinator,
|
||||
no peer discovery, no consensus to operate
|
||||
- **Optional native provider**: install `@soulcraft/cor` to back the index with
|
||||
on-disk DiskANN that scales to 10B+ vectors on one machine
|
||||
- **Per-tenant pools**: isolate tenants by giving each its own Brainy instance and
|
||||
storage directory
|
||||
- **Horizontal read scaling**: run many reader processes against one shared on-disk
|
||||
store (single writer, many readers); replicate the artifact with your operator
|
||||
tooling
|
||||
**Scaling features:**
|
||||
- **Horizontal scaling**: Add nodes as needed
|
||||
- **Auto-sharding**: Distributes data automatically
|
||||
- **Multi-region**: Global distribution
|
||||
- **Load balancing**: Automatic request distribution
|
||||
- **Zero-downtime upgrades**: Rolling updates
|
||||
- **Infinite scale**: No upper limits
|
||||
|
||||
### 🛡️ Enterprise Compliance 🚧 Coming Soon
|
||||
|
||||
|
|
|
|||
|
|
@ -1,181 +0,0 @@
|
|||
---
|
||||
title: Export & Import (portable graph)
|
||||
slug: guides/export-and-import
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 9
|
||||
description: Export part or all of a brain to a portable, versioned PortableGraph document and import it back — by id, collection, connected neighbourhood, VFS subtree, predicate, or the whole brain. export() lives on the immutable Db, so asOf()/with() give time-travel and what-if exports.
|
||||
next:
|
||||
- guides/subtypes-and-facets
|
||||
- api/README
|
||||
---
|
||||
|
||||
# Export & Import (portable graph)
|
||||
|
||||
Brainy serializes part or all of a brain — an item, a collection, a connected
|
||||
neighbourhood, a VFS subtree, a predicate match, or the whole brain — into a single
|
||||
versioned JSON document (`PortableGraph`), and restores it.
|
||||
|
||||
```typescript
|
||||
const graph = await brain.export() // whole brain → PortableGraph
|
||||
await brain.import(graph) // restore (merge by id, re-embed if no vectors)
|
||||
```
|
||||
|
||||
It is **portable** (human-readable JSON), **versioned** (`formatVersion`, so a document
|
||||
written by 7.x imports cleanly into 8.0), and **current-state** (the entities and edges as
|
||||
they are now — no generation history). Use it for portable artifacts, partial exports,
|
||||
cross-environment moves, and version upgrades.
|
||||
|
||||
`export()` is a method on the **immutable `Db` value**, so it composes with every way of
|
||||
obtaining one:
|
||||
|
||||
```typescript
|
||||
brain.export(sel) // = brain.now().export(sel)
|
||||
;(await brain.asOf(gen)).export(sel) // time-travel export (a past generation)
|
||||
brain.now().with(ops).export(sel) // what-if export (a speculative state)
|
||||
```
|
||||
|
||||
## When to use which
|
||||
|
||||
| You want… | Use |
|
||||
|-----------|-----|
|
||||
| A portable, partial-or-whole, cross-version graph document | **`brain.export()` / `brain.import()`** (this guide) |
|
||||
| A whole-brain snapshot **with generation history** | `brain.now().persist(path)` / `Brainy.load(path)` (native) |
|
||||
| To ingest a CSV / PDF / Excel / JSON **file** as new entities | `brain.import(file)` — see [Import Anything](./import-anything.md) |
|
||||
|
||||
`import()` is **polymorphic**: hand it a `PortableGraph` and it does the graph round-trip;
|
||||
hand it a file/buffer and it does foreign-file ingestion (dispatched on the document's
|
||||
`format: 'brainy-portable-graph'` tag).
|
||||
|
||||
## Exporting
|
||||
|
||||
```typescript
|
||||
brain.export(selector?, options?): Promise<PortableGraph>
|
||||
// (also on any Db: brain.now().export(...), (await brain.asOf(g)).export(...))
|
||||
```
|
||||
|
||||
### Selectors — *what* to export
|
||||
|
||||
Omit the selector to export the whole brain. Otherwise pick a node set:
|
||||
|
||||
| Scenario | Selector |
|
||||
|----------|----------|
|
||||
| Just an item (or items) | `{ ids: ['a', 'b'] }` |
|
||||
| A collection + its children | `{ collection: collectionId }` (alias `memberOf`) |
|
||||
| A connected neighbourhood | `{ connected: { from: id, depth: 2, verbs?, direction? } }` |
|
||||
| A VFS directory / file (+ subtree) | `{ vfsPath: '/docs', recursive?: true }` |
|
||||
| Everything matching a predicate | `{ type, subtype, where, service, visibility }` |
|
||||
| The whole brain | *(omit)* |
|
||||
|
||||
The selector reuses `find()`'s grammar — *"export what `find()` would match, minus ranking
|
||||
and limit."* Structural and predicate selectors **compose**:
|
||||
|
||||
```typescript
|
||||
// Members of a collection whose status is "open"
|
||||
await brain.export({ collection: collectionId, where: { status: 'open' } })
|
||||
|
||||
// Already have find() results? Export exactly those with the ids selector
|
||||
const hits = await brain.find({ type: NounType.Document })
|
||||
await brain.export({ ids: hits.map(r => r.id) })
|
||||
```
|
||||
|
||||
### Options — *how* to serialize
|
||||
|
||||
| Option | Default | Effect |
|
||||
|--------|---------|--------|
|
||||
| `includeVectors` | `false` | Carry embedding vectors verbatim. Off ⇒ `import()` re-embeds from `data`. |
|
||||
| `includeContent` | `false` | Include VFS file bytes in `blobs` so files round-trip byte-identically. |
|
||||
| `includeSystem` | `false` | Include `visibility:'system'` entities such as the VFS root. |
|
||||
| `edges` | `'induced'` | `'induced'` (both endpoints in the set), `'incident'` (also dangling edges, recorded in `danglingIds`), or `'none'` (nodes only). |
|
||||
|
||||
## Importing
|
||||
|
||||
```typescript
|
||||
brain.import(graph, options?): Promise<ImportResult>
|
||||
```
|
||||
|
||||
The whole graph is applied as **one atomic transaction** — it advances the brain exactly
|
||||
one generation, or none on failure.
|
||||
|
||||
```typescript
|
||||
const result = await brain.import(graph, { onConflict: 'merge' })
|
||||
// → { imported, merged, skipped, reembedded, blobsWritten, errors }
|
||||
```
|
||||
|
||||
| Option | Default | Effect |
|
||||
|--------|---------|--------|
|
||||
| `onConflict` | `'merge'` | `'merge'` (update existing id in place — assemble many exported graphs), `'replace'` (delete + recreate), or `'skip'`. |
|
||||
| `reembed` | `'auto'` | `'auto'` (use the carried vector, else re-embed from `data`) or `'never'` (require a carried vector; record an error if absent). |
|
||||
| `remapIds` | — | Rewrite every id on the way in, e.g. to clone a template subgraph under fresh ids. |
|
||||
| `meta` | — | Transaction metadata recorded in the tx-log alongside the new generation. |
|
||||
|
||||
The default `onConflict: 'merge'` lets you assemble one working graph from many exported
|
||||
documents that share entity ids — re-importing an id merges rather than duplicates.
|
||||
|
||||
## The `PortableGraph` format
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"format": "brainy-portable-graph", // identifies the document type
|
||||
"formatVersion": 1, // import gates on this (cross-version migration)
|
||||
"brainyVersion": "8.0.0",
|
||||
"createdAt": "2026-06-16T…Z",
|
||||
"embedding": { "model": "all-MiniLM-L6-v2", "dimensions": 384 },
|
||||
"selector": { … }, // echoes what was exported (provenance)
|
||||
"entities": [
|
||||
{
|
||||
"id": "…", "type": "Document", "subtype": "invoice", "visibility": "public",
|
||||
"data": "…", // the embedding source
|
||||
"confidence": 1, "weight": 1, "service": "…",
|
||||
"vector": [ … ], // only with includeVectors
|
||||
"metadata": { … } // custom fields only (reserved fields are top-level)
|
||||
}
|
||||
],
|
||||
"relations": [
|
||||
{ "id":"…", "from":"…", "to":"…", "type":"Contains", "subtype":"…",
|
||||
"weight":1, "confidence":1, "metadata": { … } }
|
||||
],
|
||||
"blobs": { "<sha256>": "<base64>" }, // only with includeContent
|
||||
"danglingIds": [ "…" ], // only with edges:'incident'
|
||||
"stats": { "entityCount": 0, "relationCount": 0, "blobCount": 0, "vectorDimensions": 384 }
|
||||
}
|
||||
```
|
||||
|
||||
Standard fields (`subtype`, `visibility`, `data`, `confidence`, `weight`, `service`) sit at
|
||||
the top level of each entity; `metadata` holds **only** custom user fields — mirroring the
|
||||
in-memory `Entity` shape, so `import()` maps each field to its dedicated parameter. The
|
||||
TypeScript types (`PortableGraph`, `PortableGraphEntity`, `PortableGraphRelation`, `ExportSelector`,
|
||||
`ExportOptions`, `ImportOptions`, `ImportResult`) are exported from the package root.
|
||||
|
||||
## Generations & time-travel
|
||||
|
||||
The portable document is **current-state** — it never embeds generation history (that keeps
|
||||
it cross-version-portable). History lives where it's queryable:
|
||||
|
||||
- **During a session:** `brain.asOf(g)` / `brain.now().with(ops)` on the live brain. Because
|
||||
`export()` is on the `Db`, `(await brain.asOf(g)).export()` serializes a *past* generation
|
||||
and `brain.now().with(ops).export()` serializes a *speculative* one.
|
||||
- **A whole-brain snapshot with history:** `brain.now().persist(path)` / `Brainy.load(path)`
|
||||
(native, generation-preserving) — a separate facility from this portable format.
|
||||
|
||||
Note: only `transact()` (and the write shortcuts that commit through it) advances a
|
||||
generation, so time-travel export differs across transaction boundaries.
|
||||
|
||||
## Cross-version (7.x → 8.0)
|
||||
|
||||
Because the document is shared and versioned, a PortableGraph written by 7.x imports into 8.0:
|
||||
`formatVersion` is read forward, `subtype` is carried so 8.0 re-types correctly, and the
|
||||
same 384-dimension model on both lines means `includeVectors:false` re-embeds identically
|
||||
(or `true` carries vectors verbatim).
|
||||
|
||||
## VFS
|
||||
|
||||
VFS directories are `Collection` entities and files are entities linked by `Contains`, so
|
||||
the whole filesystem (or any subtree) exports through the `vfsPath` selector:
|
||||
|
||||
```typescript
|
||||
await brain.export({ vfsPath: '/' }, { includeContent: true }) // all VFS + bytes
|
||||
await brain.export({ vfsPath: '/docs' }, { includeContent: true }) // one directory
|
||||
await brain.export({ vfsPath: '/a/b.txt' }, { includeContent: true }) // one file
|
||||
```
|
||||
|
|
@ -1,99 +0,0 @@
|
|||
---
|
||||
title: External Backups & Sparse Storage
|
||||
slug: guides/external-backups
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 10
|
||||
description: How to back up a brain directory with external tools (tar, rsync, cp) without exploding sparse files — why a store can show 100+ GB "apparent" size on a small disk, which files are sparse, and how persist()/restore() handle it for you.
|
||||
next:
|
||||
- guides/snapshots-and-time-travel
|
||||
- concepts/storage-adapters
|
||||
---
|
||||
|
||||
# External Backups & Sparse Storage
|
||||
|
||||
The built-in snapshot path — [`db.persist()` and `brain.restore()`](/docs/guides/snapshots-and-time-travel) —
|
||||
already handles everything on this page for you. Read this when you back up a brain directory with
|
||||
**external tools**: `tar`, `rsync`, `cp`, `scp`, or a filesystem-level backup agent.
|
||||
|
||||
## The one-sentence rule
|
||||
|
||||
> **Always use the sparse-aware flag**: `tar czSf` (capital `S`), `rsync --sparse`,
|
||||
> `cp --sparse=always`. A naive copy can turn a 2 GB store into a 100+ GB one — or fail
|
||||
> the disk entirely.
|
||||
|
||||
## Why: some files are sparse
|
||||
|
||||
When a native accelerator plugin is active, parts of the index live in **memory-mapped files**
|
||||
created at a large fixed virtual size — the file's *apparent* size — while the filesystem only
|
||||
allocates blocks that were actually written. A brand-new id-mapper file can report tens of
|
||||
gigabytes in `ls -l` while occupying a few megabytes on disk.
|
||||
|
||||
Check the difference yourself:
|
||||
|
||||
```bash
|
||||
ls -lh brain-data/_id_mapper/ # APPARENT size (can be huge)
|
||||
du -sh brain-data/ # ALLOCATED size (the real footprint)
|
||||
```
|
||||
|
||||
The sparse candidates in a brain directory:
|
||||
|
||||
| Path | What it is |
|
||||
|---|---|
|
||||
| `_id_mapper/` | The native id-mapper's mmap files (large fixed virtual size) |
|
||||
| `_blobs/` | Native index files (vector base, segments) — may be mmap-backed |
|
||||
|
||||
Everything else (entities, `_system`, `_generations`, `_cas` content blobs) is ordinary dense data.
|
||||
|
||||
## Doing it right
|
||||
|
||||
**tar** — the `S` flag detects holes and stores only real data:
|
||||
|
||||
```bash
|
||||
tar czSf brain-backup.tgz /data/brain
|
||||
# restore preserves the holes:
|
||||
tar xzSf brain-backup.tgz -C /data/
|
||||
```
|
||||
|
||||
**rsync**:
|
||||
|
||||
```bash
|
||||
rsync -a --sparse /data/brain/ backup-host:/backups/brain/
|
||||
```
|
||||
|
||||
**cp**:
|
||||
|
||||
```bash
|
||||
cp -a --sparse=always /data/brain /backups/brain
|
||||
```
|
||||
|
||||
**What goes wrong without the flag:** the copy *materializes* every hole as real zero bytes.
|
||||
A store whose apparent size exceeds the target disk fails with `ENOSPC` partway through — and a
|
||||
copy that *does* fit silently costs the full apparent size in storage and transfer time.
|
||||
|
||||
## What the built-in paths do (so you don't have to)
|
||||
|
||||
- **`db.persist(path)`** snapshots via **hard links** — instant and space-shared, since every data
|
||||
file is immutable-by-rename. The handful of append-in-place files (the transaction log, the
|
||||
commit fact log's tail segment) and mmap-mutated directories (`_id_mapper/`) are **byte-copied**
|
||||
instead, so a post-snapshot write can never reach through a shared inode into your backup.
|
||||
- **`brain.restore(path, { confirm: true })`** is **non-destructive and sparse-aware**: the snapshot
|
||||
is copied into a staging area *before* any live data is touched (all-zero blocks stay holes), and
|
||||
only after the copy fully succeeds does an atomic swap move it into place. A failed copy —
|
||||
including `ENOSPC` — leaves the live store exactly as it was. A crash mid-swap completes forward
|
||||
on the next open.
|
||||
|
||||
## Live-store caveats for external tools
|
||||
|
||||
1. **Prefer snapshotting a `persist()` output, not the live directory.** `persist()` produces a
|
||||
crash-consistent, immutable snapshot; running `tar` against a live, actively-written directory
|
||||
can capture a torn mid-write state. If you must archive live, stop writes first (or accept that
|
||||
the archive is only as consistent as the moment's flush state).
|
||||
2. **Never prune or "clean up" files inside a brain directory.** Index files that look stale or
|
||||
redundant are load-bearing; the store protects its declared index families from in-process
|
||||
deletion, but an external `rm` bypasses that fence. If space is the concern, `du -sh` first —
|
||||
the allocated size is usually far smaller than it looks.
|
||||
3. **Verify restores by opening them.** `Brainy.load(path)` opens any snapshot or restored
|
||||
directory read-only — the store verifies its own coherence at open and reports loudly if
|
||||
anything is missing or torn.
|
||||
|
|
@ -1,153 +0,0 @@
|
|||
---
|
||||
title: Query Limits & Pagination
|
||||
slug: guides/find-limits
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 8
|
||||
description: How Brainy caps `find({ limit })` to prevent OOM, the three escape valves when the cap is too tight, and why pagination is the future-proof pattern.
|
||||
next:
|
||||
- guides/aggregation
|
||||
- api/reference
|
||||
---
|
||||
|
||||
# Query Limits & Pagination
|
||||
|
||||
Brainy's `find()` returns entities into a JavaScript array. The size of that array is bounded by an auto-configured cap so a single query can never run the host out of memory. This guide explains the cap, the three ways to raise it when your use case justifies it, and the one pattern that scales no matter what cap is in effect: pagination.
|
||||
|
||||
## Why the cap exists
|
||||
|
||||
Every entity Brainy returns carries:
|
||||
|
||||
- A 384-dim float32 embedding vector (1.5 KB)
|
||||
- Standard fields: `id`, `type`, `subtype`, timestamps, confidence, weight (~200 bytes)
|
||||
- User metadata (variable — typical 5-10 KB, can spike to 20+ KB)
|
||||
|
||||
Conservative budget: **25 KB per result**. A `find({ limit: 100_000 })` against a brain with rich metadata can claim ~2.5 GB before Brainy's iteration starts. JavaScript's GC + V8's heap targets can't absorb that swing without paging or OOM in production.
|
||||
|
||||
The cap is a safety net. It's not the only reason your query might be slow — graph traversal and HNSW search have their own perf characteristics — but it's the one that turns a slow query into a sudden runtime error.
|
||||
|
||||
## The auto-configured cap (7.30.2+)
|
||||
|
||||
Brainy picks `maxLimit` from the first of these that's available:
|
||||
|
||||
| Priority | Source | Formula |
|
||||
|---|---|---|
|
||||
| 1 | Constructor option `maxQueryLimit` | Hard cap at supplied value, max 100 000 |
|
||||
| 2 | Constructor option `reservedQueryMemory` | `floor(reservedQueryMemory / 25 KB)` capped at 100 000 |
|
||||
| 3 | Detected container memory limit (Cloud Run, Kubernetes, cgroups v1/v2) | `floor(containerLimit × 0.25 / 25 KB)` capped at 100 000 |
|
||||
| 4 | Free system memory | `floor(availableMemory / 25 KB)` capped at 100 000 |
|
||||
|
||||
Worked example: a 4 GB Cloud Run container picks priority 3 → `floor(4 GB × 0.25 / 25 KB) = floor(40 960) = 40 000` results. A 900 MB free-memory box on priority 4 gets `floor(900 MB / 25 KB) = ~36 000`.
|
||||
|
||||
The cap is fixed at construction and never changes at runtime. Query timing is recorded
|
||||
for diagnostics only — a burst of slow queries cannot silently shrink the cap, and the
|
||||
auto-detected tiers (3 and 4) never go below a floor of 10 000.
|
||||
|
||||
> **Calibration note.** Pre-7.30.2 used 100 KB per result instead of 25 KB, which produced caps that were 4× too tight for typical workloads (an 8 KB / result reality). 7.30.2 recalibrated to match observed entity sizes; existing `limit: 10_000` safety patterns now pass silently on any reasonably-sized box.
|
||||
|
||||
## What happens when you exceed the cap
|
||||
|
||||
`find({ limit })` enforces in **two tiers**:
|
||||
|
||||
### Soft tier: `maxLimit < limit ≤ 2 × maxLimit`
|
||||
|
||||
You get a one-time warning per call site:
|
||||
|
||||
```
|
||||
[Brainy] find({ limit: 50000 }) exceeds the auto-configured query limit of
|
||||
40000 (basis: detected container memory limit). Choose one:
|
||||
• Increase the cap: new Brainy({ maxQueryLimit: 50000 })
|
||||
• Reserve more memory: new Brainy({ reservedQueryMemory: 1310720000 })
|
||||
• Paginate: split the query with { limit, offset } pages
|
||||
at YourService.loadDashboard (/app/src/dashboard.ts:142:18)
|
||||
Docs: https://soulcraft.com/docs/guides/find-limits
|
||||
```
|
||||
|
||||
**The query proceeds.** Brainy returns the result set you asked for; the warning is a teaching signal, not a block. Existing code that relied on the cap silently allowing safety-cap limits (`limit: 10_000` against a 9 K-cap box) keeps working — the warning shows you the recipe so you can fix it intentionally.
|
||||
|
||||
### Hard tier: `limit > 2 × maxLimit`
|
||||
|
||||
Same message, but thrown as an error. This is real OOM territory; the cap stops being a recommendation and becomes a guardrail.
|
||||
|
||||
## The three escape valves
|
||||
|
||||
### 1. Raise the cap at construction — `maxQueryLimit`
|
||||
|
||||
When the auto-config is wrong for your workload (e.g. you know your entities are smaller than 25 KB average and you need bigger result sets), set an explicit cap:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './data' },
|
||||
maxQueryLimit: 50_000 // raises the cap; still hard-clamped at 100 000
|
||||
})
|
||||
```
|
||||
|
||||
This is the right answer when:
|
||||
- Your entity metadata is genuinely small (e.g. 1-2 KB) and 25 KB per result is over-conservative
|
||||
- You're running on a box with lots of headroom and 25% of memory underestimates what you can spare for queries
|
||||
- You need a known-good limit that doesn't change when the box's free-memory wiggles at startup
|
||||
|
||||
### 2. Reserve more memory for queries — `reservedQueryMemory`
|
||||
|
||||
When you want the cap to be memory-derived but more generous than the default 25% slice:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
reservedQueryMemory: 1024 * 1024 * 1024 // 1 GB → ~40 000 result cap
|
||||
})
|
||||
```
|
||||
|
||||
This is the right answer when:
|
||||
- Your host's memory budget for queries is known and stable, regardless of free-memory at startup
|
||||
- You want the formula to scale with the documented per-result size (25 KB) instead of a hard number
|
||||
|
||||
### 3. Paginate — the future-proof pattern
|
||||
|
||||
If your query genuinely needs to walk all matches in a category, don't fight the cap — walk in pages:
|
||||
|
||||
```typescript
|
||||
async function findAll<T>(params: FindParams<T>, pageSize = 1000): Promise<Result<T>[]> {
|
||||
const all: Result<T>[] = []
|
||||
let offset = 0
|
||||
while (true) {
|
||||
const page = await brain.find({ ...params, limit: pageSize, offset })
|
||||
all.push(...page)
|
||||
if (page.length < pageSize) break
|
||||
offset += page.length
|
||||
}
|
||||
return all
|
||||
}
|
||||
|
||||
// Use it just like find():
|
||||
const allEvents = await findAll({ type: NounType.Event, where: { status: 'open' } })
|
||||
```
|
||||
|
||||
For very large brains, prefer the streaming API which avoids holding the full result set in memory at all:
|
||||
|
||||
```typescript
|
||||
for await (const entity of brain.streaming.entities({ type: NounType.Event })) {
|
||||
// process one entity at a time
|
||||
}
|
||||
```
|
||||
|
||||
## When to use which
|
||||
|
||||
| Situation | Recommended valve |
|
||||
|---|---|
|
||||
| The cap is unreasonably low for your known entity size | `maxQueryLimit` |
|
||||
| You want a memory-derived cap but more generous than 25% | `reservedQueryMemory` |
|
||||
| Your query needs ALL matches in a category | Pagination or `brain.streaming.entities()` |
|
||||
| You hit the cap once during a one-off migration | `maxQueryLimit` or `migrateField` (which already paginates internally) |
|
||||
| You're hitting the cap on a recurring user-facing query | Pagination — the cap will get tighter in 8.0, not looser |
|
||||
|
||||
## A note on Brainy 8.0
|
||||
|
||||
8.0's Datomic-style `Db` API may make per-call limits stricter to keep snapshot semantics cheap. **Pagination is the only pattern that's guaranteed to keep working unchanged.** Code that paginates today doesn't need to revisit when 8.0 ships.
|
||||
|
||||
## Reference
|
||||
|
||||
- `BrainyConfig.maxQueryLimit?: number` — explicit cap override (max 100 000)
|
||||
- `BrainyConfig.reservedQueryMemory?: number` — memory budget for queries (bytes)
|
||||
- `find({ limit, offset })` — paginated find
|
||||
- `brain.streaming.entities(filter)` — streaming alternative for very large traversals
|
||||
|
|
@ -1,17 +1,15 @@
|
|||
# Framework Integration Guide
|
||||
|
||||
Brainy is **framework-friendly** - designed to drop into the server side of any modern JavaScript framework. This guide shows you how to integrate Brainy into framework-based apps.
|
||||
Brainy 3.0 is **framework-first** - designed from the ground up to work seamlessly with modern JavaScript frameworks. This guide shows you how to integrate Brainy into any framework.
|
||||
|
||||
> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun (server-side only). It is not a browser library. In framework apps, run Brainy in API routes, server components, server actions, loaders, or a dedicated backend service - never in client-side bundles.
|
||||
## 🎯 Why Framework-First?
|
||||
|
||||
## 🎯 Why Server-Side?
|
||||
|
||||
Brainy embeds an HNSW vector index, a graph engine, and a filesystem-backed persistence layer. These belong on the server:
|
||||
Traditional AI databases require complex browser polyfills and bundler configurations. Brainy 3.0 trusts your framework to handle this:
|
||||
|
||||
- **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'`
|
||||
- **Auto storage detection**: `new Brainy()` auto-selects filesystem persistence on Node
|
||||
- **Cleaner code**: No browser polyfills, no conditional client/server imports
|
||||
- **Better DX**: One instance shared across your server routes
|
||||
- **Framework responsibility**: Let Next.js, Vite, Webpack handle Node.js polyfills
|
||||
- **Cleaner code**: No browser-specific entry points or conditional imports
|
||||
- **Better DX**: Same API everywhere - browser, server, edge
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
|
|
@ -26,8 +24,7 @@ npm install @soulcraft/brainy
|
|||
```javascript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// Run on the server (API route, server component, backend service)
|
||||
// new Brainy() auto-detects filesystem persistence on Node
|
||||
// Works in any framework!
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
|
|
@ -44,48 +41,52 @@ const results = await brain.find("framework integration")
|
|||
|
||||
## ⚛️ React Integration
|
||||
|
||||
Brainy runs on the server, so a React client component talks to it through an API endpoint (see the Next.js API route below). The hook fetches results; it never instantiates Brainy in the browser.
|
||||
|
||||
### Basic Hook Pattern
|
||||
|
||||
```jsx
|
||||
import { useState, useCallback } from 'react'
|
||||
import { useState, useEffect, useCallback } from 'react'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
function useBrainySearch(endpoint = '/api/search') {
|
||||
const [results, setResults] = useState([])
|
||||
const [loading, setLoading] = useState(false)
|
||||
function useBrainy() {
|
||||
const [brain, setBrain] = useState(null)
|
||||
const [isReady, setIsReady] = useState(false)
|
||||
|
||||
const search = useCallback(async (query) => {
|
||||
if (!query) return
|
||||
setLoading(true)
|
||||
try {
|
||||
const res = await fetch(endpoint, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query })
|
||||
useEffect(() => {
|
||||
const initBrain = async () => {
|
||||
const newBrain = new Brainy({
|
||||
storage: { type: 'opfs' } // Browser storage
|
||||
})
|
||||
const { results } = await res.json()
|
||||
setResults(results)
|
||||
} finally {
|
||||
setLoading(false)
|
||||
await newBrain.init()
|
||||
setBrain(newBrain)
|
||||
setIsReady(true)
|
||||
}
|
||||
}, [endpoint])
|
||||
|
||||
return { results, loading, search }
|
||||
initBrain()
|
||||
}, [])
|
||||
|
||||
return { brain, isReady }
|
||||
}
|
||||
|
||||
// Usage in component
|
||||
function SearchComponent() {
|
||||
const { results, loading, search } = useBrainySearch()
|
||||
const { brain, isReady } = useBrainy()
|
||||
const [results, setResults] = useState([])
|
||||
|
||||
const handleSearch = useCallback(async (query) => {
|
||||
if (!isReady) return
|
||||
const searchResults = await brain.find(query)
|
||||
setResults(searchResults)
|
||||
}, [brain, isReady])
|
||||
|
||||
if (!isReady) return <div>Loading AI...</div>
|
||||
|
||||
return (
|
||||
<div>
|
||||
<input
|
||||
type="text"
|
||||
placeholder="Search..."
|
||||
onChange={(e) => search(e.target.value)}
|
||||
onChange={(e) => handleSearch(e.target.value)}
|
||||
/>
|
||||
{loading && <div>Searching...</div>}
|
||||
<div>
|
||||
{results.map(result => (
|
||||
<div key={result.id}>
|
||||
|
|
@ -99,34 +100,48 @@ function SearchComponent() {
|
|||
}
|
||||
```
|
||||
|
||||
### Shared Server Instance
|
||||
### React Context Pattern
|
||||
|
||||
On the server, create one Brainy instance and reuse it across requests. This module is imported only by server code (API routes, server components), never by client components:
|
||||
|
||||
```javascript
|
||||
// lib/brain.server.js
|
||||
```jsx
|
||||
import React, { createContext, useContext, useEffect, useState } from 'react'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
let brainPromise
|
||||
const BrainyContext = createContext()
|
||||
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
// new Brainy() auto-detects filesystem persistence on Node
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
export function BrainyProvider({ children }) {
|
||||
const [brain, setBrain] = useState(null)
|
||||
const [isReady, setIsReady] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
const initBrain = async () => {
|
||||
const newBrain = new Brainy()
|
||||
await newBrain.init()
|
||||
setBrain(newBrain)
|
||||
setIsReady(true)
|
||||
}
|
||||
|
||||
initBrain()
|
||||
}, [])
|
||||
|
||||
return (
|
||||
<BrainyContext.Provider value={{ brain, isReady }}>
|
||||
{children}
|
||||
</BrainyContext.Provider>
|
||||
)
|
||||
}
|
||||
|
||||
export function useBrainContext() {
|
||||
const context = useContext(BrainyContext)
|
||||
if (!context) {
|
||||
throw new Error('useBrainContext must be used within BrainyProvider')
|
||||
}
|
||||
return brainPromise
|
||||
return context
|
||||
}
|
||||
```
|
||||
|
||||
## 🟢 Vue.js Integration
|
||||
|
||||
Vue components call a server endpoint (see the Nuxt server route in the [Vue.js Integration Guide](vue-integration.md)); Brainy itself runs on the server.
|
||||
|
||||
### Composition API (client component)
|
||||
### Composition API
|
||||
|
||||
```vue
|
||||
<template>
|
||||
|
|
@ -140,70 +155,100 @@ Vue components call a server endpoint (see the Nuxt server route in the [Vue.js
|
|||
</template>
|
||||
|
||||
<script setup>
|
||||
import { ref } from 'vue'
|
||||
import { ref, onMounted } from 'vue'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = ref(null)
|
||||
const isReady = ref(false)
|
||||
const query = ref('')
|
||||
const results = ref([])
|
||||
|
||||
const search = async () => {
|
||||
if (!query.value) return
|
||||
const res = await fetch('/api/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query: query.value })
|
||||
onMounted(async () => {
|
||||
brain.value = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
results.value = (await res.json()).results
|
||||
await brain.value.init()
|
||||
isReady.value = true
|
||||
})
|
||||
|
||||
const search = async () => {
|
||||
if (!isReady.value || !query.value) return
|
||||
results.value = await brain.value.find(query.value)
|
||||
}
|
||||
</script>
|
||||
```
|
||||
|
||||
### Shared Server Instance
|
||||
|
||||
On the server, create one Brainy instance and reuse it across requests:
|
||||
### Vue 3 Plugin
|
||||
|
||||
```javascript
|
||||
// server/brain.js (server-only module)
|
||||
// plugins/brainy.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
let brainPromise
|
||||
export default {
|
||||
install(app, options) {
|
||||
const brain = new Brainy(options)
|
||||
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
// new Brainy() auto-detects filesystem persistence on Node
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
app.config.globalProperties.$brain = brain
|
||||
app.provide('brain', brain)
|
||||
|
||||
// Initialize on app mount
|
||||
brain.init()
|
||||
}
|
||||
return brainPromise
|
||||
}
|
||||
|
||||
// main.js
|
||||
import { createApp } from 'vue'
|
||||
import BrainyPlugin from './plugins/brainy'
|
||||
|
||||
const app = createApp(App)
|
||||
app.use(BrainyPlugin, {
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
```
|
||||
|
||||
## 🅰️ Angular Integration
|
||||
|
||||
The Angular service calls your backend over HTTP; Brainy lives in that backend, not in the browser.
|
||||
|
||||
### Service Pattern (calls the backend)
|
||||
### Service Pattern
|
||||
|
||||
```typescript
|
||||
// brainy.service.ts
|
||||
import { Injectable } from '@angular/core'
|
||||
import { HttpClient } from '@angular/common/http'
|
||||
import { Observable } from 'rxjs'
|
||||
import { BehaviorSubject, Observable } from 'rxjs'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
@Injectable({
|
||||
providedIn: 'root'
|
||||
})
|
||||
export class BrainyService {
|
||||
constructor(private http: HttpClient) {}
|
||||
private brain: Brainy
|
||||
private readySubject = new BehaviorSubject<boolean>(false)
|
||||
|
||||
search(query: string): Observable<{ results: any[] }> {
|
||||
return this.http.post<{ results: any[] }>('/api/search', { query })
|
||||
ready$: Observable<boolean> = this.readySubject.asObservable()
|
||||
|
||||
constructor() {
|
||||
this.initBrain()
|
||||
}
|
||||
|
||||
add(data: any, type: string, metadata?: any): Observable<{ id: string }> {
|
||||
return this.http.post<{ id: string }>('/api/add', { data, type, metadata })
|
||||
private async initBrain() {
|
||||
this.brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
await this.brain.init()
|
||||
this.readySubject.next(true)
|
||||
}
|
||||
|
||||
async search(query: string): Promise<any[]> {
|
||||
if (!this.readySubject.value) {
|
||||
throw new Error('Brain not ready')
|
||||
}
|
||||
return await this.brain.find(query)
|
||||
}
|
||||
|
||||
async add(data: any, type: string, metadata?: any): Promise<string> {
|
||||
if (!this.readySubject.value) {
|
||||
throw new Error('Brain not ready')
|
||||
}
|
||||
return await this.brain.add({ data, type, metadata })
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
@ -235,52 +280,64 @@ export class SearchComponent {
|
|||
|
||||
constructor(private brainyService: BrainyService) {}
|
||||
|
||||
search() {
|
||||
async search() {
|
||||
if (!this.query) return
|
||||
this.brainyService.search(this.query).subscribe(({ results }) => {
|
||||
this.results = results
|
||||
})
|
||||
this.results = await this.brainyService.search(this.query)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The matching backend endpoint uses Brainy directly (Node/Bun):
|
||||
|
||||
```typescript
|
||||
// server: api/search
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy() // auto-detects filesystem persistence on Node
|
||||
await brain.init()
|
||||
|
||||
export async function handleSearch(query: string) {
|
||||
return await brain.find(query)
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 Next.js Integration
|
||||
|
||||
In Next.js, Brainy lives in server code only: API routes, server components, or server actions. Create one shared instance in a server-only module.
|
||||
### App Router (Next.js 13+)
|
||||
|
||||
### Shared Server Instance
|
||||
|
||||
```javascript
|
||||
// lib/brain.server.js (imported only by server code)
|
||||
```jsx
|
||||
// app/providers.jsx
|
||||
'use client'
|
||||
import { createContext, useContext, useEffect, useState } from 'react'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
let brainPromise
|
||||
const BrainyContext = createContext()
|
||||
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './data' }
|
||||
})
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
}
|
||||
return brainPromise
|
||||
export function BrainyProvider({ children }) {
|
||||
const [brain, setBrain] = useState(null)
|
||||
const [isReady, setIsReady] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
const initBrain = async () => {
|
||||
const newBrain = new Brainy()
|
||||
await newBrain.init()
|
||||
setBrain(newBrain)
|
||||
setIsReady(true)
|
||||
}
|
||||
|
||||
initBrain()
|
||||
}, [])
|
||||
|
||||
return (
|
||||
<BrainyContext.Provider value={{ brain, isReady }}>
|
||||
{children}
|
||||
</BrainyContext.Provider>
|
||||
)
|
||||
}
|
||||
|
||||
export const useBrainy = () => useContext(BrainyContext)
|
||||
```
|
||||
|
||||
```jsx
|
||||
// app/layout.jsx
|
||||
import { BrainyProvider } from './providers'
|
||||
|
||||
export default function RootLayout({ children }) {
|
||||
return (
|
||||
<html>
|
||||
<body>
|
||||
<BrainyProvider>
|
||||
{children}
|
||||
</BrainyProvider>
|
||||
</body>
|
||||
</html>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -288,78 +345,45 @@ export function getBrain() {
|
|||
|
||||
```javascript
|
||||
// app/api/search/route.js
|
||||
import { getBrain } from '@/lib/brain.server'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './data' }
|
||||
})
|
||||
await brain.init()
|
||||
|
||||
export async function POST(request) {
|
||||
const { query } = await request.json()
|
||||
const brain = await getBrain()
|
||||
const results = await brain.find(query)
|
||||
|
||||
return Response.json({ results })
|
||||
}
|
||||
```
|
||||
|
||||
### Server Action
|
||||
|
||||
```javascript
|
||||
// app/actions.js
|
||||
'use server'
|
||||
import { getBrain } from '@/lib/brain.server'
|
||||
|
||||
export async function search(query) {
|
||||
const brain = await getBrain()
|
||||
return await brain.find(query)
|
||||
}
|
||||
```
|
||||
|
||||
## 🔷 SvelteKit Integration
|
||||
|
||||
Brainy runs in a server-only module (`*.server.js`); the component fetches results from an endpoint.
|
||||
|
||||
```javascript
|
||||
// src/lib/server/brain.js (server-only — note the .server suffix)
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
let brainPromise
|
||||
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
const brain = new Brainy() // auto-detects filesystem persistence
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
}
|
||||
return brainPromise
|
||||
}
|
||||
```
|
||||
|
||||
```javascript
|
||||
// src/routes/api/search/+server.js
|
||||
import { json } from '@sveltejs/kit'
|
||||
import { getBrain } from '$lib/server/brain'
|
||||
|
||||
export async function POST({ request }) {
|
||||
const { query } = await request.json()
|
||||
const brain = await getBrain()
|
||||
return json({ results: await brain.find(query) })
|
||||
}
|
||||
```
|
||||
## 🔷 Svelte Integration
|
||||
|
||||
```svelte
|
||||
<!-- SearchComponent.svelte -->
|
||||
<script>
|
||||
import { onMount } from 'svelte'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
let brain = null
|
||||
let isReady = false
|
||||
let query = ''
|
||||
let results = []
|
||||
|
||||
async function search() {
|
||||
if (!query) return
|
||||
const res = await fetch('/api/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query })
|
||||
onMount(async () => {
|
||||
brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
results = (await res.json()).results
|
||||
await brain.init()
|
||||
isReady = true
|
||||
})
|
||||
|
||||
async function search() {
|
||||
if (!isReady || !query) return
|
||||
results = await brain.find(query)
|
||||
}
|
||||
</script>
|
||||
|
||||
|
|
@ -377,23 +401,27 @@ export async function POST({ request }) {
|
|||
|
||||
## 🌟 Solid.js Integration
|
||||
|
||||
The component calls a server endpoint (use SolidStart server routes, or any backend, to host Brainy):
|
||||
|
||||
```jsx
|
||||
import { createSignal } from 'solid-js'
|
||||
import { createSignal, onMount } from 'solid-js'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
function SearchComponent() {
|
||||
const [brain, setBrain] = createSignal(null)
|
||||
const [isReady, setIsReady] = createSignal(false)
|
||||
const [query, setQuery] = createSignal('')
|
||||
const [results, setResults] = createSignal([])
|
||||
|
||||
onMount(async () => {
|
||||
const newBrain = new Brainy()
|
||||
await newBrain.init()
|
||||
setBrain(newBrain)
|
||||
setIsReady(true)
|
||||
})
|
||||
|
||||
const search = async () => {
|
||||
if (!query()) return
|
||||
const res = await fetch('/api/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query: query() })
|
||||
})
|
||||
setResults((await res.json()).results)
|
||||
if (!isReady() || !query()) return
|
||||
const searchResults = await brain().find(query())
|
||||
setResults(searchResults)
|
||||
}
|
||||
|
||||
return (
|
||||
|
|
@ -422,25 +450,57 @@ function SearchComponent() {
|
|||
|
||||
## 📦 Bundler Configuration
|
||||
|
||||
Brainy is a server-side dependency, so keep it out of client bundles. Import it only from server-only modules (`*.server.js`, API routes, server components, server actions). If your bundler ever tries to pull Brainy into a client bundle, that's a sign it's being imported from a client component — move the import to a server module.
|
||||
|
||||
For server builds, mark Brainy as external so the bundler doesn't inline it:
|
||||
### Vite (Recommended)
|
||||
|
||||
```javascript
|
||||
// vite.config.js (SSR build)
|
||||
// vite.config.js
|
||||
import { defineConfig } from 'vite'
|
||||
|
||||
export default defineConfig({
|
||||
ssr: {
|
||||
external: ['@soulcraft/brainy']
|
||||
define: {
|
||||
global: 'globalThis'
|
||||
},
|
||||
optimizeDeps: {
|
||||
include: ['@soulcraft/brainy']
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Webpack
|
||||
|
||||
```javascript
|
||||
// rollup.config.js (server bundle)
|
||||
// webpack.config.js
|
||||
module.exports = {
|
||||
resolve: {
|
||||
fallback: {
|
||||
"fs": false,
|
||||
"path": require.resolve("path-browserify"),
|
||||
"crypto": require.resolve("crypto-browserify")
|
||||
}
|
||||
},
|
||||
plugins: [
|
||||
new webpack.ProvidePlugin({
|
||||
global: 'global'
|
||||
})
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Rollup
|
||||
|
||||
```javascript
|
||||
// rollup.config.js
|
||||
import { nodeResolve } from '@rollup/plugin-node-resolve'
|
||||
import commonjs from '@rollup/plugin-commonjs'
|
||||
|
||||
export default {
|
||||
external: ['@soulcraft/brainy', 'node:fs', 'node:path', 'node:crypto']
|
||||
plugins: [
|
||||
nodeResolve({
|
||||
browser: true,
|
||||
preferBuiltins: false
|
||||
}),
|
||||
commonjs()
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -448,24 +508,30 @@ export default {
|
|||
|
||||
### Server-Side Rendering
|
||||
|
||||
Instantiate Brainy on the server and feed its results into the rendered page. Never construct it in client code.
|
||||
|
||||
```javascript
|
||||
// Server-side data loading (framework loader / getServerSideProps / load fn)
|
||||
import { getBrain } from './brain.server'
|
||||
// Check if running in browser
|
||||
if (typeof window !== 'undefined') {
|
||||
// Browser-only code
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
}
|
||||
|
||||
export async function load({ url }) {
|
||||
const brain = await getBrain()
|
||||
const query = url.searchParams.get('q') ?? ''
|
||||
const results = query ? await brain.find(query) : []
|
||||
return { results }
|
||||
// Or use dynamic imports
|
||||
const initBrainForBrowser = async () => {
|
||||
if (typeof window === 'undefined') return null
|
||||
|
||||
const { Brainy } = await import('@soulcraft/brainy')
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
return brain
|
||||
}
|
||||
```
|
||||
|
||||
### Static Site Generation
|
||||
|
||||
```javascript
|
||||
// For build-time usage (runs in Node during the build)
|
||||
// For build-time usage
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export async function generateStaticProps() {
|
||||
|
|
@ -474,8 +540,8 @@ export async function generateStaticProps() {
|
|||
})
|
||||
await brain.init()
|
||||
|
||||
// Build search index (paginate with { limit, offset } for larger stores)
|
||||
const allContent = await brain.find({ limit: 1000 })
|
||||
// Build search index
|
||||
const allContent = await brain.export()
|
||||
|
||||
return {
|
||||
props: { searchIndex: allContent }
|
||||
|
|
@ -486,46 +552,67 @@ export async function generateStaticProps() {
|
|||
## 🔧 Framework-Specific Tips
|
||||
|
||||
### React
|
||||
- Keep components client-side and call a Brainy-backed API route
|
||||
- Use `useCallback` for fetch handlers to prevent re-renders
|
||||
- Debounce keystroke-driven searches before hitting the endpoint
|
||||
- Use `useCallback` for search functions to prevent re-renders
|
||||
- Consider `useMemo` for expensive brain operations
|
||||
- Implement cleanup in `useEffect` for proper memory management
|
||||
|
||||
### Vue
|
||||
- Components call an endpoint; the shared instance lives in a server module
|
||||
- Consider Pinia for caching results client-side
|
||||
- Debounce reactive search queries
|
||||
- Use `shallowRef` for the brain instance (it's not reactive data)
|
||||
- Consider Pinia for global brain state management
|
||||
- Use `watchEffect` for reactive search queries
|
||||
|
||||
### Angular
|
||||
- Use `HttpClient` and RxJS to call the backend
|
||||
- Hold the shared Brainy instance in your Node backend, not the app
|
||||
- Consider lazy loading search features in feature modules
|
||||
- Implement proper dependency injection with services
|
||||
- Use RxJS observables for reactive search
|
||||
- Consider lazy loading brain in feature modules
|
||||
|
||||
### Next.js
|
||||
- Put Brainy in server-only modules (`*.server.js`), API routes, or server actions
|
||||
- Reuse one shared instance across requests
|
||||
- Implement proper error boundaries for failed fetches
|
||||
- Use dynamic imports for client-side only features
|
||||
- Consider API routes for server-side brain operations
|
||||
- Implement proper error boundaries
|
||||
|
||||
## 🚨 Common Issues & Solutions
|
||||
|
||||
### Issue: "fs module not found" / "crypto is not defined" in the browser
|
||||
**Cause**: Brainy was imported into a client bundle. Brainy 8.0 is a server-side library (Node 22+/Bun) and uses Node built-ins like `fs` and `crypto`.
|
||||
**Solution**: Import Brainy only from server code — server-only modules (`*.server.js`), API routes, server components, or server actions. From client components, call those endpoints instead.
|
||||
### Issue: "crypto is not defined"
|
||||
**Solution**: Your framework should handle this automatically. If not:
|
||||
```javascript
|
||||
// Add to your bundle config
|
||||
define: {
|
||||
global: 'globalThis'
|
||||
}
|
||||
```
|
||||
|
||||
### Issue: Large client bundle size
|
||||
**Cause**: A client module is pulling in Brainy.
|
||||
**Solution**: Move the `import { Brainy } from '@soulcraft/brainy'` into a server-only module so it never reaches the browser bundle.
|
||||
### Issue: "fs module not found"
|
||||
**Solution**: This is expected in browsers. Use browser-compatible storage:
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'opfs' } // Or 'memory' for development
|
||||
})
|
||||
```
|
||||
|
||||
### Issue: Large bundle size
|
||||
**Solution**: Use dynamic imports for optional features:
|
||||
```javascript
|
||||
const brain = await import('@soulcraft/brainy').then(m => new m.Brainy())
|
||||
```
|
||||
|
||||
### Issue: SSR hydration mismatch
|
||||
**Solution**: Run the search on the server (loader / server action / API route) and pass the results down as props, so server and client render the same markup.
|
||||
**Solution**: Initialize brain only on client:
|
||||
```javascript
|
||||
useEffect(() => {
|
||||
// Browser-only initialization
|
||||
initBrain()
|
||||
}, [])
|
||||
```
|
||||
|
||||
## 🎯 Best Practices
|
||||
|
||||
1. **Initialize Once**: Create one shared Brainy instance per server process, not per request
|
||||
2. **Server-Only**: Import Brainy only from server modules — never from client components
|
||||
3. **Endpoint Boundary**: Expose search/add through API routes or server actions
|
||||
4. **Handle Loading**: Show loading states in the client while the fetch is in flight
|
||||
5. **Error Handling**: Catch and surface failed endpoint calls gracefully
|
||||
6. **Storage**: Use `filesystem` for persistence (the default on Node) or `memory` for ephemeral/tests
|
||||
1. **Initialize Once**: Create brain instance at app level, not component level
|
||||
2. **Use Context**: Share brain instance across components with context/providers
|
||||
3. **Handle Loading**: Always show loading states during brain initialization
|
||||
4. **Error Boundaries**: Implement proper error handling for brain operations
|
||||
5. **Memory Management**: Clean up brain instances on unmount
|
||||
6. **Storage Strategy**: Choose appropriate storage for your deployment target
|
||||
|
||||
## 📚 Next Steps
|
||||
|
||||
|
|
|
|||
|
|
@ -49,23 +49,28 @@ await brain.import(csv, { format: 'csv' })
|
|||
|
||||
### 📊 Import Excel - Multi-Sheet Support
|
||||
```javascript
|
||||
// Import entire Excel workbook — every sheet is processed automatically
|
||||
// Import entire Excel workbook
|
||||
await brain.import('sales-report.xlsx')
|
||||
// ✨ Processes all sheets, preserves structure, infers types!
|
||||
|
||||
// Mirror the workbook into the VFS, grouped by sheet
|
||||
// Or specific sheets only
|
||||
await brain.import('data.xlsx', {
|
||||
vfsPath: '/imports/data',
|
||||
groupBy: 'sheet'
|
||||
excelSheets: ['Customers', 'Orders']
|
||||
})
|
||||
// ✨ Multi-sheet data becomes interconnected entities!
|
||||
```
|
||||
|
||||
### 📑 Import PDF - Text & Tables
|
||||
```javascript
|
||||
// Import PDF documents — text and tables are extracted automatically
|
||||
// Import PDF documents
|
||||
await brain.import('research-paper.pdf')
|
||||
// ✨ Extracts text, detects tables, preserves metadata!
|
||||
|
||||
// With table extraction
|
||||
await brain.import('report.pdf', {
|
||||
pdfExtractTables: true
|
||||
})
|
||||
// ✨ Converts PDF tables to structured data automatically!
|
||||
```
|
||||
|
||||
### 📝 Import YAML - File or String
|
||||
|
|
@ -260,7 +265,7 @@ Once imported, use Triple Intelligence to query:
|
|||
|
||||
```javascript
|
||||
// Vector search
|
||||
const similar = await brain.find('engineers')
|
||||
const similar = await brain.search('engineers')
|
||||
|
||||
// Natural language
|
||||
const results = await brain.find('people in engineering who joined this year')
|
||||
|
|
@ -300,10 +305,7 @@ await brain.import(data, {
|
|||
// Deduplication
|
||||
enableDeduplication: true, // Check for duplicate entities (default: true)
|
||||
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
|
||||
// Notes: false disables BOTH the inline merge and the background pass that
|
||||
// runs ~5 min after the last import (merged duplicates are deleted).
|
||||
// The inline pass auto-disables for imports >100 entities (O(n²) cost);
|
||||
// the background pass still covers those unless the flag is false.
|
||||
// Note: Auto-disabled for imports >100 entities
|
||||
|
||||
// Performance
|
||||
chunkSize: 100, // Batch size for processing (default: varies by operation)
|
||||
|
|
|
|||
|
|
@ -1173,22 +1173,21 @@ await this.graphIndex.addEdge(
|
|||
```
|
||||
|
||||
**Benefits**:
|
||||
- **O(1) relationship lookups**: `related(entityId)` is instant
|
||||
- **O(1) relationship lookups**: `getRelations(entityId)` is instant
|
||||
- **Bidirectional traversal**: Find incoming and outgoing edges
|
||||
- **Type filtering**: Get only `CreatedBy` relationships
|
||||
- **Global statistics**: Count relationships by type
|
||||
|
||||
**Query Examples**:
|
||||
```typescript
|
||||
// What did Mona Lisa create? (outgoing edges)
|
||||
const outgoing = await brain.related({ from: 'ent_mona_lisa_...' })
|
||||
// What did Mona Lisa create?
|
||||
const outgoing = await brain.getRelations('ent_mona_lisa_...', { direction: 'outgoing' })
|
||||
|
||||
// What created Mona Lisa? (incoming edges)
|
||||
const incoming = await brain.related({ to: 'ent_mona_lisa_...' })
|
||||
// What created Mona Lisa?
|
||||
const incoming = await brain.getRelations('ent_mona_lisa_...', { direction: 'incoming' })
|
||||
|
||||
// Get only CreatedBy relationships
|
||||
const createdBy = await brain.related({
|
||||
from: 'ent_mona_lisa_...',
|
||||
const createdBy = await brain.getRelations('ent_mona_lisa_...', {
|
||||
type: VerbType.CreatedBy
|
||||
})
|
||||
```
|
||||
|
|
@ -1309,7 +1308,7 @@ await this.history.recordImport(
|
|||
```
|
||||
|
||||
**Use Cases**:
|
||||
- List all imports: `coordinator.getHistory().getHistory()` (the `ImportHistory` entries shown above)
|
||||
- List all imports: `brain.data().listImports()`
|
||||
- Reimport with same settings
|
||||
- Audit trail for compliance
|
||||
- Rollback imports
|
||||
|
|
@ -1688,11 +1687,11 @@ verbCounts['CreatedBy']
|
|||
|
||||
---
|
||||
|
||||
### 8. Storage Layout
|
||||
### 8. Storage by Cloud Provider
|
||||
|
||||
Brainy 8.0 ships two adapters: filesystem and memory.
|
||||
Brainy supports multiple storage adapters:
|
||||
|
||||
#### Filesystem (Default)
|
||||
#### File System (Local)
|
||||
```
|
||||
.brainy/
|
||||
├── nouns/
|
||||
|
|
@ -1702,18 +1701,43 @@ Brainy 8.0 ships two adapters: filesystem and memory.
|
|||
└── index.json
|
||||
```
|
||||
|
||||
#### Google Cloud Storage (GCS)
|
||||
```
|
||||
gs://my-bucket/brainy/
|
||||
├── nouns/
|
||||
│ └── ent_mona_lisa_1730000000000.json
|
||||
├── nouns-metadata/
|
||||
│ └── ent_mona_lisa_1730000000000.json
|
||||
└── ...
|
||||
```
|
||||
|
||||
#### Amazon S3
|
||||
```
|
||||
s3://my-bucket/brainy/
|
||||
├── nouns/
|
||||
│ └── ent_mona_lisa_1730000000000.json
|
||||
└── ...
|
||||
```
|
||||
|
||||
#### Cloudflare R2
|
||||
```
|
||||
r2://my-bucket/brainy/
|
||||
├── nouns/
|
||||
│ └── ent_mona_lisa_1730000000000.json
|
||||
└── ...
|
||||
```
|
||||
|
||||
**Configuration**:
|
||||
```typescript
|
||||
const brain = await Brainy.create({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './.brainy'
|
||||
type: 'gcs',
|
||||
bucket: 'my-bucket',
|
||||
prefix: 'brainy/'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
For off-site backup, snapshot `path` from your scheduler with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`. Brainy itself doesn't reach out to object storage.
|
||||
|
||||
---
|
||||
|
||||
## Performance & Scale
|
||||
|
|
@ -1743,7 +1767,7 @@ For off-site backup, snapshot `path` from your scheduler with `gsutil rsync`, `a
|
|||
- HNSW Index: O(log n) search (1B entities = ~30 hops)
|
||||
- Metadata Index: O(1) filtering
|
||||
- Graph Adjacency: O(1) relationship lookups
|
||||
- Storage: Bounded by the filesystem volume backing `path`
|
||||
- Storage: Unlimited (cloud buckets)
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
|
|
@ -1887,7 +1911,7 @@ groupBy: 'type'
|
|||
- ✅ O(1) metadata filtering
|
||||
- ✅ O(1) relationship traversal
|
||||
- ✅ Human-readable VFS structure
|
||||
- ✅ Filesystem-backed persistence (snapshot/sync the directory for off-site backup)
|
||||
- ✅ Cloud storage support (GCS/S3/R2)
|
||||
- ✅ Billion-scale performance
|
||||
- ✅ Zero mocks, production-ready!
|
||||
|
||||
|
|
|
|||
|
|
@ -86,22 +86,11 @@ await brain.import(file, {
|
|||
|
||||
```typescript
|
||||
await brain.import(file, {
|
||||
enableDeduplication: true, // Check for duplicates (default: true)
|
||||
enableDeduplication: true, // Check for duplicates (default: false)
|
||||
deduplicationThreshold: 0.85 // Similarity threshold (default: 0.85)
|
||||
})
|
||||
```
|
||||
|
||||
Deduplication merges entities judged duplicates — the non-primary records are
|
||||
**deleted**. Set `enableDeduplication: false` to disable it entirely: the flag
|
||||
gates both the inline merge during import and the background pass that runs
|
||||
about 5 minutes after the last import.
|
||||
|
||||
```typescript
|
||||
await brain.import(file, {
|
||||
enableDeduplication: false // No merging, inline or background
|
||||
})
|
||||
```
|
||||
|
||||
### Import Tracking
|
||||
|
||||
Track and organize imports by project:
|
||||
|
|
@ -362,7 +351,7 @@ const result = await brain.import(file)
|
|||
const products = await brain.find({ type: 'Product' })
|
||||
|
||||
// Get entity relationships
|
||||
const relations = await brain.related(products[0].id)
|
||||
const relations = await brain.getRelations(products[0].id)
|
||||
|
||||
// Search VFS
|
||||
const vfsFiles = await brain.vfs().find(result.vfs.rootPath + '/**/*.json')
|
||||
|
|
|
|||
|
|
@ -1,213 +0,0 @@
|
|||
---
|
||||
title: Inspecting a Live Brainy
|
||||
slug: guides/inspection
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 30
|
||||
description: Operator recipes for diagnosing a running Brainy data directory — counts, queries, query plans, health checks, and snapshots — without stopping the live writer.
|
||||
next:
|
||||
- concepts/multi-process
|
||||
---
|
||||
|
||||
# Inspecting a Live Brainy
|
||||
|
||||
When something is wrong in production, you need to see what's actually in the
|
||||
store. This guide covers the safe ways to query a running Brainy directory.
|
||||
|
||||
## The cardinal rule
|
||||
|
||||
**Never open a second writer on the same directory.** Filesystem storage will throw, and any other write path will corrupt the live writer's state. Use `Brainy.openReadOnly()` or the `brainy inspect` CLI instead.
|
||||
|
||||
## The CLI is the fastest path
|
||||
|
||||
```bash
|
||||
# What's in this brain?
|
||||
brainy inspect stats /data/brain
|
||||
|
||||
# Find specific entities
|
||||
brainy inspect find /data/brain --type Event --where '{"status":"paid"}' --limit 20
|
||||
|
||||
# Single entity by ID
|
||||
brainy inspect get /data/brain 0b7a9...
|
||||
|
||||
# Why is this query returning empty?
|
||||
brainy inspect explain /data/brain --where '{"entityType":"booking"}'
|
||||
|
||||
# Quick invariants
|
||||
brainy inspect health /data/brain
|
||||
|
||||
# Random sample (no query needed)
|
||||
brainy inspect sample /data/brain --type Event --n 20
|
||||
|
||||
# Tail new writes as they happen
|
||||
brainy inspect watch /data/brain --type Event
|
||||
|
||||
# Save a snapshot
|
||||
brainy inspect backup /data/brain /backups/brain-$(date +%Y%m%d).tar
|
||||
```
|
||||
|
||||
Every subcommand internally:
|
||||
|
||||
1. Asks the live writer to flush via the cross-process RPC (skip with `--no-fresh`).
|
||||
2. Opens the data directory via `Brainy.openReadOnly()`.
|
||||
3. Runs the query.
|
||||
4. Closes cleanly.
|
||||
|
||||
Results are JSON by default. Add `--pretty` for indented output.
|
||||
|
||||
## When a query returns surprising results
|
||||
|
||||
If `find()` returns `0` for a query you expect to match: run `inspect
|
||||
explain` first. It shows which index path will serve each `where` clause:
|
||||
|
||||
```bash
|
||||
$ brainy inspect explain /data/brain --where '{"entityType":"booking","status":"paid"}'
|
||||
{
|
||||
"query": { "where": { "entityType": "booking", "status": "paid" } },
|
||||
"fieldPlan": [
|
||||
{ "field": "entityType", "path": "none", "notes": "No index entries for field..." },
|
||||
{ "field": "status", "path": "column-store", "notes": "O(log n) binary search..." }
|
||||
],
|
||||
"warnings": [
|
||||
"Field \"entityType\" has no index entries. find() will return [] silently."
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The `"path": "none"` is the smoking gun. It means the field has no column
|
||||
store manifest and no sparse chunked index — so `find()` will return `[]`
|
||||
regardless of what's actually on disk. Likely causes:
|
||||
|
||||
- The writer registered the field in memory but hasn't flushed. Run
|
||||
`brain.requestFlush()` from the writer side, or use `brainy inspect
|
||||
--fresh` (default).
|
||||
- The field name has a typo or wrong casing.
|
||||
- The field is genuinely absent from every entity.
|
||||
|
||||
## Health checks
|
||||
|
||||
`inspect health` runs a fixed battery of cheap invariant checks:
|
||||
|
||||
```bash
|
||||
$ brainy inspect health /data/brain
|
||||
{
|
||||
"overall": "warn",
|
||||
"checks": [
|
||||
{ "name": "index-parity", "status": "pass", "message": "Vector (1851) and metadata (1851) agree." },
|
||||
{ "name": "field-registry", "status": "pass", "message": "23 fields registered for 1851 entities." },
|
||||
{ "name": "seeded-records", "status": "warn", "message": "15 entities tagged _seeded:true." },
|
||||
{ "name": "writer-heartbeat", "status": "pass", "message": "Writer healthy (PID 1774431...)." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Each check returns `pass`, `warn`, or `fail`. The exit code is `2` when any
|
||||
check fails — useful for piping into monitoring or CI.
|
||||
|
||||
## Programmatic inspection
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const reader = await Brainy.openReadOnly({
|
||||
storage: { type: 'filesystem', path: '/data/brain' }
|
||||
})
|
||||
|
||||
// Force the writer to flush before reading
|
||||
await reader.requestFlush({ timeoutMs: 5000 })
|
||||
|
||||
// What's in there?
|
||||
const stats = await reader.stats()
|
||||
console.log(`${stats.entityCount} entities`, stats.entitiesByType)
|
||||
|
||||
// Why is this query empty?
|
||||
const plan = await reader.explain({ where: { entityType: 'booking' } })
|
||||
for (const f of plan.fieldPlan) {
|
||||
console.log(`${f.field} -> ${f.path}`)
|
||||
}
|
||||
|
||||
// Run invariants
|
||||
const health = await reader.health()
|
||||
console.log(health.overall)
|
||||
|
||||
await reader.close()
|
||||
```
|
||||
|
||||
Every mutation method (`add`, `update`, `remove`, `relate`, `transact`,
|
||||
`restore`, ...) throws on a read-only instance with a clear message.
|
||||
|
||||
## Backups
|
||||
|
||||
`brainy inspect backup` asks the writer to flush first, then tars the
|
||||
directory. The snapshot reflects the writer's state at the moment of the
|
||||
flush:
|
||||
|
||||
```bash
|
||||
brainy inspect backup /data/brain /backups/brain-2026-05-15.tar
|
||||
```
|
||||
|
||||
For periodic backups (hourly, daily), schedule this via cron or your
|
||||
container scheduler. For point-in-time recovery, use the Db API's
|
||||
`db.persist(path)` — a self-contained hard-link snapshot that later writes
|
||||
can never alter, restorable with `brain.restore(path, { confirm: true })`.
|
||||
See [Snapshots & Time Travel](./snapshots-and-time-travel.md).
|
||||
|
||||
## Comparing two stores
|
||||
|
||||
`brainy inspect diff` returns a JSON summary of counts and a sample of
|
||||
entity IDs present in one but not the other. Useful when debugging
|
||||
replication or migrations:
|
||||
|
||||
```bash
|
||||
brainy inspect diff /data/brain-prod /data/brain-staging
|
||||
```
|
||||
|
||||
Sample-based — for a full diff, dump both with `inspect dump` and compare
|
||||
the JSONL.
|
||||
|
||||
## Auditing graph-read truth
|
||||
|
||||
`brain.auditGraph()` (8.6.0+) proves — or disproves — that relationship reads
|
||||
return canonical truth on a given brain, without mutating anything. It walks
|
||||
every stored relationship record, asks the same read path your application
|
||||
uses (`related()`, VFS `readdir`) with every visibility tier included, and
|
||||
classifies every discrepancy:
|
||||
|
||||
```typescript
|
||||
const report = await brain.auditGraph()
|
||||
|
||||
report.coherent // true = related()/readdir can be trusted on this brain
|
||||
report.missingFromReadsCount // records the read path omits — stale index
|
||||
report.danglingEndpointsCount // relationships whose endpoint entity is gone
|
||||
report.readOnlyCount // read-path edges with NO stored record — ghosts
|
||||
report.visibilityHiddenCount // internal/system edges hidden by design (not a fault)
|
||||
```
|
||||
|
||||
Counts are always exact; the example lists (`missingFromReads`,
|
||||
`danglingEndpoints`, `readOnlyVerbIds`) are capped at `maxExamples`
|
||||
(default 100) and `truncatedExamples` says so when they are.
|
||||
|
||||
Run it after any engine upgrade, restore, or migration. If it reports
|
||||
discrepancies, run `brain.repairIndex()` and audit again — a `coherent`
|
||||
report after the repair is the verified statement that the heal worked.
|
||||
Cost: one relationship-record walk plus one indexed read per distinct
|
||||
source entity — safe on a live brain.
|
||||
|
||||
## Repairing a corrupted store
|
||||
|
||||
If invariants fail and you suspect index corruption, `inspect repair`
|
||||
opens the store in writer mode and rebuilds all indexes from raw storage.
|
||||
**Stop the live writer first** — `repair` will throw if another writer
|
||||
holds the lock. Add `--force` only if you have personally verified the
|
||||
existing lock is stale.
|
||||
|
||||
```bash
|
||||
brainy inspect repair /data/brain
|
||||
```
|
||||
|
||||
## Multi-process safety summary
|
||||
|
||||
See [concepts/multi-process](../concepts/multi-process.md) for the lock
|
||||
semantics, heartbeat behavior, and what's not yet enforced on cloud
|
||||
backends.
|
||||
|
|
@ -1,89 +0,0 @@
|
|||
---
|
||||
title: Installation
|
||||
slug: getting-started/installation
|
||||
public: true
|
||||
category: getting-started
|
||||
template: guide
|
||||
order: 1
|
||||
description: Install Brainy with npm, bun, yarn, or pnpm. Works in Node.js 22+ and Bun 1.0+ (server-only since 8.0). TypeScript included.
|
||||
next:
|
||||
- getting-started/quick-start
|
||||
- guides/storage-adapters
|
||||
---
|
||||
|
||||
# Installation
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Node.js 22+** or **Bun 1.0+**
|
||||
- TypeScript is optional — Brainy ships with full type definitions
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/brainy
|
||||
```
|
||||
|
||||
Or with your preferred package manager:
|
||||
|
||||
```bash
|
||||
bun add @soulcraft/brainy
|
||||
yarn add @soulcraft/brainy
|
||||
pnpm add @soulcraft/brainy
|
||||
```
|
||||
|
||||
## Verify
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
console.log('Brainy ready.')
|
||||
```
|
||||
|
||||
## Native Acceleration (Optional)
|
||||
|
||||
For production workloads, add Cor for Rust-accelerated SIMD distance calculations and native embeddings:
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/cor
|
||||
```
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({ plugins: ['@soulcraft/cor'] })
|
||||
await brain.init() // native providers registered during init
|
||||
```
|
||||
|
||||
Cor registers native (Rust/SIMD) vector, metadata, and graph engines behind the same Brainy `find()` API — no code changes, an optional dependency for production-scale workloads.
|
||||
|
||||
## Server-only since 8.0
|
||||
|
||||
Brainy 8.0 runs on Node.js 22+ and Bun 1.0+. Browser support (OPFS storage,
|
||||
Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line
|
||||
remains available on npm if you need it.
|
||||
|
||||
## TypeScript
|
||||
|
||||
Brainy ships with full TypeScript types. No `@types/` package needed:
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
const id = await brain.add({
|
||||
data: 'Hello, Brainy',
|
||||
type: NounType.Concept,
|
||||
metadata: { created: Date.now() }
|
||||
})
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Quick Start](/docs/getting-started/quick-start) — build your first knowledge graph in 60 seconds
|
||||
- [Storage Adapters](/docs/guides/storage-adapters) — choose the right storage for your deployment
|
||||
|
|
@ -252,7 +252,7 @@ const result = await brain.import('./glossary.xlsx', {
|
|||
// - "part_of"
|
||||
// - "related_to"
|
||||
|
||||
const relations = await brain.related({ limit: 100 })
|
||||
const relations = await brain.getRelations({ limit: 100 })
|
||||
const types = new Set(relations.map(r => r.label))
|
||||
console.log(types)
|
||||
// Set { 'capital_of', 'guards', 'located_in', 'related_to' }
|
||||
|
|
|
|||
|
|
@ -35,17 +35,14 @@ This single WASM file contains everything needed for sentence embeddings.
|
|||
|
||||
### Bun (Recommended)
|
||||
|
||||
```bash
|
||||
# Bun as a runtime — supported and recommended
|
||||
bun add @soulcraft/brainy
|
||||
```typescript
|
||||
// Works with Bun runtime
|
||||
bun run server.ts
|
||||
```
|
||||
|
||||
Brainy is pure WebAssembly with no native binaries, so the module graph stays
|
||||
bundler-friendly. Single-binary `bun build --compile` is **not a supported
|
||||
target** at present: Bun 1.3.10 has a `--compile` codegen regression
|
||||
(`__promiseAll is not defined`) triggered by top-level `await` in the bundled
|
||||
graph. Run Brainy under the Bun runtime (above) instead.
|
||||
// Works with bun --compile (single binary deployment!)
|
||||
bun build --compile --target=bun server.ts
|
||||
./server // Self-contained binary with embedded model
|
||||
```
|
||||
|
||||
### Node.js
|
||||
|
||||
|
|
@ -167,7 +164,7 @@ await brain.init()
|
|||
|
||||
**What's new:**
|
||||
- Faster initialization
|
||||
- Bundler-friendly (pure WASM, no native binaries)
|
||||
- Works with `bun --compile`
|
||||
- No network requirements
|
||||
|
||||
### From Custom Embedding Functions
|
||||
|
|
@ -219,8 +216,9 @@ export { brain }
|
|||
|
||||
### Deployment
|
||||
```bash
|
||||
# Option 1: Bun runtime
|
||||
bun run server.ts
|
||||
# Option 1: Bun compile (single binary)
|
||||
bun build --compile server.ts
|
||||
./server # Contains everything
|
||||
|
||||
# Option 2: Docker
|
||||
docker build -t my-app .
|
||||
|
|
|
|||
371
docs/guides/neural-api.md
Normal file
371
docs/guides/neural-api.md
Normal file
|
|
@ -0,0 +1,371 @@
|
|||
# Neural API Guide
|
||||
|
||||
> Semantic intelligence features for clustering, similarity, and analysis
|
||||
|
||||
## Overview
|
||||
|
||||
The Neural API provides advanced AI-powered features for understanding relationships and patterns in your data. Access it through `brain.neural()` (method call) after initializing Brainy.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```javascript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Access Neural API (note: neural() is a method call)
|
||||
const neural = brain.neural()
|
||||
|
||||
// Find similar items
|
||||
const similarity = await neural.similar('text1', 'text2')
|
||||
|
||||
// Auto-cluster your data
|
||||
const clusters = await neural.clusters()
|
||||
```
|
||||
|
||||
## Core Features
|
||||
|
||||
### 1. Semantic Clustering
|
||||
|
||||
Automatically group related items based on their meaning:
|
||||
|
||||
```javascript
|
||||
// Simple clustering - let Brainy decide
|
||||
const clusters = await neural.clusters()
|
||||
|
||||
// Each cluster contains:
|
||||
// - id: Unique identifier
|
||||
// - members: Array of item IDs in this cluster
|
||||
// - centroid: The "center" of the cluster
|
||||
// - label: Optional descriptive label
|
||||
// - confidence: How confident the clustering is
|
||||
|
||||
// Example: Organize customer feedback
|
||||
const feedback = [
|
||||
await brain.add("The app crashes when I upload photos"),
|
||||
await brain.add("Photo upload feature is broken"),
|
||||
await brain.add("Great customer service!"),
|
||||
await brain.add("Support team was very helpful"),
|
||||
await brain.add("Pricing is too high"),
|
||||
await brain.add("Too expensive for what it offers")
|
||||
]
|
||||
|
||||
const themes = await neural.clusters()
|
||||
// Results in 3 clusters: bugs, support, pricing
|
||||
```
|
||||
|
||||
#### Advanced Clustering Options
|
||||
|
||||
```javascript
|
||||
// Control clustering behavior
|
||||
const clusters = await neural.clusters({
|
||||
algorithm: 'kmeans', // Algorithm to use
|
||||
maxClusters: 5, // Maximum clusters to create
|
||||
threshold: 0.7 // Minimum similarity within clusters
|
||||
})
|
||||
|
||||
// Cluster specific items only
|
||||
const techItems = ['id1', 'id2', 'id3', 'id4']
|
||||
const techClusters = await neural.clusters(techItems)
|
||||
|
||||
// Find clusters near a specific item
|
||||
const relatedClusters = await neural.clusters('central-item-id')
|
||||
```
|
||||
|
||||
### 2. Similarity Calculation
|
||||
|
||||
Compare any two items to see how similar they are:
|
||||
|
||||
```javascript
|
||||
// Compare by ID
|
||||
const score = await neural.similar('item1-id', 'item2-id')
|
||||
// Returns 0-1 (0 = completely different, 1 = identical)
|
||||
|
||||
// Compare text directly
|
||||
const score = await neural.similar(
|
||||
"Machine learning is fascinating",
|
||||
"AI and deep learning are interesting"
|
||||
)
|
||||
// Returns ~0.75 (pretty similar)
|
||||
|
||||
// Compare vectors
|
||||
const v1 = await brain.embed("concept 1")
|
||||
const v2 = await brain.embed("concept 2")
|
||||
const score = await neural.similar(v1, v2)
|
||||
|
||||
// Get detailed similarity analysis
|
||||
const detailed = await neural.similar('id1', 'id2', {
|
||||
detailed: true
|
||||
})
|
||||
// Returns: {
|
||||
// score: 0.85,
|
||||
// confidence: 0.92,
|
||||
// explanation: "High semantic overlap in technology domain"
|
||||
// }
|
||||
```
|
||||
|
||||
### 3. Finding Neighbors
|
||||
|
||||
Discover items similar to a given item:
|
||||
|
||||
```javascript
|
||||
// Find 5 most similar items
|
||||
const neighbors = await neural.neighbors('item-id', 5)
|
||||
|
||||
// Each neighbor has:
|
||||
// - id: The neighbor's ID
|
||||
// - similarity: How similar (0-1)
|
||||
// - data: The actual content
|
||||
|
||||
// Example: Recommend similar articles
|
||||
const articleId = await brain.add("Guide to React Hooks")
|
||||
const similar = await neural.neighbors(articleId, 3)
|
||||
|
||||
for (const article of similar) {
|
||||
console.log(`${article.similarity * 100}% similar: ${article.data}`)
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Semantic Hierarchy
|
||||
|
||||
Build a hierarchy showing relationships between items:
|
||||
|
||||
```javascript
|
||||
const hierarchy = await neural.hierarchy('item-id')
|
||||
|
||||
// Returns structure like:
|
||||
// {
|
||||
// self: { id: 'item-id', type: 'article' },
|
||||
// parent: { id: 'parent-id', similarity: 0.8 },
|
||||
// siblings: [
|
||||
// { id: 'sibling1', similarity: 0.75 },
|
||||
// { id: 'sibling2', similarity: 0.72 }
|
||||
// ],
|
||||
// children: [
|
||||
// { id: 'child1', similarity: 0.85 }
|
||||
// ]
|
||||
// }
|
||||
|
||||
// Use for navigation or breadcrumbs
|
||||
const hier = await neural.hierarchy(currentDoc)
|
||||
console.log(`You are here: ${hier.self.id}`)
|
||||
if (hier.parent) {
|
||||
console.log(`Parent topic: ${hier.parent.id}`)
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Outlier Detection
|
||||
|
||||
Find unusual or anomalous items in your data:
|
||||
|
||||
```javascript
|
||||
// Find items that don't fit patterns
|
||||
const outliers = await neural.outliers(0.3)
|
||||
// Returns array of IDs that are > 0.3 distance from others
|
||||
|
||||
// Example: Detect spam or unusual content
|
||||
const messages = [
|
||||
await brain.add("Meeting at 3pm"),
|
||||
await brain.add("Lunch plans for tomorrow"),
|
||||
await brain.add("BUY NOW!!! AMAZING DEALS!!!"),
|
||||
await brain.add("Project deadline next week")
|
||||
]
|
||||
|
||||
const suspicious = await neural.outliers(0.4)
|
||||
// Returns the spam message ID
|
||||
```
|
||||
|
||||
### 6. Visualization Support
|
||||
|
||||
Generate data for visualization libraries:
|
||||
|
||||
```javascript
|
||||
// Create force-directed graph data
|
||||
const vizData = await neural.visualize({
|
||||
maxNodes: 100, // Limit nodes for performance
|
||||
dimensions: 2, // 2D or 3D
|
||||
algorithm: 'force' // Layout algorithm
|
||||
})
|
||||
|
||||
// Returns:
|
||||
// {
|
||||
// nodes: [
|
||||
// { id: 'n1', x: 10, y: 20, cluster: 'c1' },
|
||||
// { id: 'n2', x: 30, y: 40, cluster: 'c1' }
|
||||
// ],
|
||||
// edges: [
|
||||
// { source: 'n1', target: 'n2', weight: 0.8 }
|
||||
// ],
|
||||
// clusters: [
|
||||
// { id: 'c1', color: '#ff6b6b', size: 15 }
|
||||
// ]
|
||||
// }
|
||||
|
||||
// Use with D3.js, Cytoscape, or other viz libraries
|
||||
const data = await neural.visualize({ dimensions: 3 })
|
||||
// Now feed to Three.js for 3D visualization
|
||||
```
|
||||
|
||||
## Practical Examples
|
||||
|
||||
### Content Recommendation System
|
||||
|
||||
```javascript
|
||||
// User reads an article
|
||||
const currentArticle = 'article-123'
|
||||
|
||||
// Find similar content
|
||||
const recommendations = await neural.neighbors(currentArticle, 5)
|
||||
|
||||
// Group all content into topics
|
||||
const topics = await neural.clusters()
|
||||
|
||||
// Find which topic this article belongs to
|
||||
const currentTopic = topics.find(t =>
|
||||
t.members.includes(currentArticle)
|
||||
)
|
||||
|
||||
// Recommend from same topic first, then similar items
|
||||
const sameTopicArticles = currentTopic.members
|
||||
.filter(id => id !== currentArticle)
|
||||
.slice(0, 3)
|
||||
```
|
||||
|
||||
### Customer Feedback Analysis
|
||||
|
||||
```javascript
|
||||
import { NounType } from '@soulcraft/brainy'
|
||||
|
||||
// Add feedback with metadata
|
||||
const feedbackIds = []
|
||||
for (const feedback of customerFeedback) {
|
||||
const id = await brain.add({
|
||||
data: feedback.text,
|
||||
type: NounType.Document,
|
||||
metadata: {
|
||||
rating: feedback.rating,
|
||||
date: feedback.date,
|
||||
product: feedback.product
|
||||
}
|
||||
})
|
||||
feedbackIds.push(id)
|
||||
}
|
||||
|
||||
// Cluster to find themes
|
||||
const neural = brain.neural()
|
||||
const themes = await neural.clusters(feedbackIds)
|
||||
|
||||
// Analyze each theme
|
||||
for (const theme of themes) {
|
||||
// Get items using Promise.all with brain.get()
|
||||
const items = await Promise.all(
|
||||
theme.members.map(id => brain.get(id))
|
||||
)
|
||||
|
||||
const avgRating = items.reduce((sum, item) =>
|
||||
sum + (item?.metadata?.rating || 0), 0) / items.length
|
||||
|
||||
console.log(`Theme with ${theme.members.length} items`)
|
||||
console.log(`Average rating: ${avgRating}`)
|
||||
|
||||
// Find representative feedback for this theme
|
||||
const centroidId = theme.members[0] // Closest to center
|
||||
const example = await brain.get(centroidId)
|
||||
console.log(`Example: "${example?.data}"`)
|
||||
}
|
||||
```
|
||||
|
||||
### Knowledge Base Organization
|
||||
|
||||
```javascript
|
||||
import { NounType } from '@soulcraft/brainy'
|
||||
|
||||
// Analyze existing knowledge base - use find() to get documents
|
||||
const allDocs = await brain.find({ type: NounType.Document, limit: 1000 })
|
||||
|
||||
// Access neural API
|
||||
const neural = brain.neural()
|
||||
|
||||
// Find duplicate or highly similar content
|
||||
const duplicates = []
|
||||
for (let i = 0; i < allDocs.length; i++) {
|
||||
for (let j = i + 1; j < allDocs.length; j++) {
|
||||
const similarity = await neural.similar(
|
||||
allDocs[i].id,
|
||||
allDocs[j].id
|
||||
)
|
||||
if (similarity > 0.95) {
|
||||
duplicates.push([allDocs[i].id, allDocs[j].id])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Build topic hierarchy
|
||||
const mainTopics = await neural.clusters({
|
||||
maxClusters: 10,
|
||||
algorithm: 'hierarchical'
|
||||
})
|
||||
|
||||
// For each main topic, find subtopics
|
||||
for (const topic of mainTopics) {
|
||||
const subtopics = await neural.clusters(topic.members)
|
||||
console.log(`Topic has ${subtopics.length} subtopics`)
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Caching**: Neural API automatically caches results. Repeated calls with same parameters are instant.
|
||||
|
||||
2. **Batch Operations**: Process multiple items together rather than one at a time.
|
||||
|
||||
3. **Sampling**: For large datasets, use sampling:
|
||||
```javascript
|
||||
const clusters = await neural.clusters({
|
||||
algorithm: 'sample',
|
||||
sampleSize: 1000 // Only analyze 1000 items
|
||||
})
|
||||
```
|
||||
|
||||
4. **Async Processing**: All neural operations are async and non-blocking.
|
||||
|
||||
## Error Handling
|
||||
|
||||
```javascript
|
||||
try {
|
||||
const similarity = await neural.similar('id1', 'id2')
|
||||
} catch (error) {
|
||||
// Handle errors
|
||||
if (error.message.includes('not found')) {
|
||||
console.log('One of the items does not exist')
|
||||
}
|
||||
}
|
||||
|
||||
// Safe clustering with empty data
|
||||
const clusters = await neural.clusters([])
|
||||
// Returns empty array, doesn't throw
|
||||
|
||||
// Non-existent IDs return 0 similarity
|
||||
const sim = await neural.similar('fake-id-1', 'fake-id-2')
|
||||
// Returns 0
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
```javascript
|
||||
// Configure neural behavior at initialization
|
||||
const brain = new Brainy({
|
||||
neural: {
|
||||
cacheSize: 1000, // Cache up to 1000 results
|
||||
defaultAlgorithm: 'kmeans',
|
||||
similarityMetric: 'cosine'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Explore [Triple Intelligence](../architecture/triple-intelligence.md) for combined vector + graph + metadata queries
|
||||
- Learn about [Augmentations](../augmentations/README.md) to extend Neural API
|
||||
- See [API Reference](../api/README.md) for complete method documentation
|
||||
|
|
@ -1,214 +0,0 @@
|
|||
---
|
||||
title: Optimistic concurrency with _rev
|
||||
slug: guides/optimistic-concurrency
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 8
|
||||
description: Use the per-entity `_rev` counter and `update({ ifRev })` to coordinate concurrent writes safely. Covers the lock pattern, idempotent inserts with `ifAbsent`, and recovery on conflict.
|
||||
next:
|
||||
- concepts/consistency-model
|
||||
- guides/find-limits
|
||||
---
|
||||
|
||||
# Optimistic concurrency with `_rev`
|
||||
|
||||
Brainy 7.31.0 adds a per-entity revision counter so multiple writers can coordinate without a global lock or external coordinator. The pattern is the same one CouchDB, PouchDB, and ETag-based HTTP caches use: read the current revision, do your work, write back with `ifRev: <whatRevYouSaw>`. If the revision moved, your write is rejected and you retry against the latest state.
|
||||
|
||||
## What gets added
|
||||
|
||||
| Surface | Behavior |
|
||||
|---|---|
|
||||
| `entity._rev: number` | Returned on every `get()`, `find()`, `search()`. Initialized to `1` on `add()`. Bumped by `1` on every successful `update()`. Pre-7.31.0 entities without `_rev` are surfaced as `1`. |
|
||||
| `update({ id, ..., ifRev: number })` | If the persisted `_rev` does not equal `ifRev`, throws `RevisionConflictError`. Omitting `ifRev` keeps the prior (unconditional) update behavior. |
|
||||
| `RevisionConflictError` | Carries `{ id, expected, actual }` for principled recovery. |
|
||||
| `add({ id, ifAbsent: true })` | By-ID idempotent insert. Returns the existing `id` if one is already present; no throw, no overwrite. |
|
||||
| `addMany({ items, ifAbsent: true })` | Applies `ifAbsent` to every item. Per-item `ifAbsent` overrides the batch flag. |
|
||||
|
||||
`_rev` is the **per-entity** counter. Its store-wide counterpart is the generation counter behind the [Db API](../concepts/consistency-model.md): `brain.transact(ops, { ifAtGeneration })` is CAS over the whole store, `update({ ifRev })` (and `ifRev` on `transact()` update operations) is CAS over one entity.
|
||||
|
||||
## The lock pattern
|
||||
|
||||
Every distributed-job scheduler eventually wants this exact loop:
|
||||
|
||||
```ts
|
||||
import { Brainy, RevisionConflictError } from '@soulcraft/brainy'
|
||||
|
||||
const LOCK_ID = '...uuid for this job slot...'
|
||||
|
||||
// Bootstrap the lock document once (idempotent).
|
||||
await brain.add({
|
||||
id: LOCK_ID,
|
||||
type: NounType.Document,
|
||||
data: { owner: null, expiresAt: 0 },
|
||||
ifAbsent: true
|
||||
})
|
||||
|
||||
async function tryAcquireLock(workerId: string, ttlMs: number) {
|
||||
const lock = await brain.get(LOCK_ID)
|
||||
if (!lock) throw new Error('lock document missing')
|
||||
|
||||
const state = lock.data as { owner: string | null; expiresAt: number }
|
||||
const now = Date.now()
|
||||
|
||||
// Only take the lock if it's free or expired.
|
||||
if (state.owner && state.expiresAt > now) return false
|
||||
|
||||
try {
|
||||
await brain.update({
|
||||
id: LOCK_ID,
|
||||
data: { owner: workerId, expiresAt: now + ttlMs },
|
||||
ifRev: lock._rev
|
||||
})
|
||||
return true
|
||||
} catch (err) {
|
||||
if (err instanceof RevisionConflictError) {
|
||||
// Another worker grabbed it between our read and write.
|
||||
return false
|
||||
}
|
||||
throw err
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
No external lock service, no Redis SETNX, no Cloud Tasks. The CAS check is the lock.
|
||||
|
||||
## Read-modify-write with retry
|
||||
|
||||
The other common shape is "update a counter / config object" with bounded retries on conflict:
|
||||
|
||||
```ts
|
||||
async function incrementCounter(id: string, by: number) {
|
||||
for (let attempt = 0; attempt < 5; attempt++) {
|
||||
const entity = await brain.get(id)
|
||||
if (!entity) throw new Error('counter does not exist')
|
||||
const current = (entity.data as { value: number }).value
|
||||
try {
|
||||
await brain.update({
|
||||
id,
|
||||
data: { value: current + by },
|
||||
ifRev: entity._rev
|
||||
})
|
||||
return
|
||||
} catch (err) {
|
||||
if (err instanceof RevisionConflictError) continue // refetch + retry
|
||||
throw err
|
||||
}
|
||||
}
|
||||
throw new Error('counter update conflict after 5 attempts')
|
||||
}
|
||||
```
|
||||
|
||||
The retry bound matters — without one, two unlucky writers can ping-pong forever.
|
||||
|
||||
## Idempotent bootstrap with `ifAbsent`
|
||||
|
||||
For singletons (config rows, well-known seed entities, job-state documents) where the natural ID is deterministic:
|
||||
|
||||
```ts
|
||||
await brain.add({
|
||||
id: 'config:singleton',
|
||||
type: NounType.Document,
|
||||
data: { tenantQuota: 1000 },
|
||||
ifAbsent: true
|
||||
})
|
||||
```
|
||||
|
||||
- First caller writes; gets back `'config:singleton'`.
|
||||
- Every subsequent caller short-circuits at the pre-read; gets back the same `'config:singleton'` without touching the existing entity.
|
||||
- `_rev` is **not** bumped on the no-op path (no write happened).
|
||||
|
||||
`ifAbsent` is only meaningful when you supply an `id`. With no `id`, Brainy generates a fresh UUID that can never collide, so the flag is silently ignored.
|
||||
|
||||
`addMany({ items, ifAbsent: true })` applies the flag to every item. Mixing per-item overrides with the batch flag works as you'd expect: per-item `ifAbsent: false` opts an individual row out, per-item `ifAbsent: true` opts it in.
|
||||
|
||||
### Why no `addIfMissing({ match, add })` for attribute-based dedup?
|
||||
|
||||
You may want "create if no entity with this email exists" (lookup by attribute, not by ID). That's a different operation:
|
||||
|
||||
```ts
|
||||
// What we DID NOT ship in 7.31.0 — the attribute-based variant.
|
||||
await brain.addIfMissing({ // ← not a real API
|
||||
match: { type: 'Person', where: { email: 'x@y.com' } },
|
||||
add: { data: '...', metadata: { email: 'x@y.com' } }
|
||||
})
|
||||
```
|
||||
|
||||
It's race-prone as a plain read-then-write: two concurrent imports both see "not found," both insert, you get duplicates. Without a unique-index primitive (which Brainy doesn't have today), close the race with whole-store CAS — read at a pinned generation, then commit only if nothing moved:
|
||||
|
||||
```ts
|
||||
import { GenerationConflictError } from '@soulcraft/brainy'
|
||||
|
||||
async function addIfMissingByEmail(email: string, data: string) {
|
||||
for (let attempt = 0; attempt < 5; attempt++) {
|
||||
const db = brain.now()
|
||||
try {
|
||||
const existing = await db.find({
|
||||
type: NounType.Person,
|
||||
where: { email },
|
||||
limit: 1
|
||||
})
|
||||
if (existing.length > 0) return existing[0].id
|
||||
|
||||
const committed = await brain.transact(
|
||||
[{ op: 'add', type: NounType.Person, subtype: 'customer', data, metadata: { email } }],
|
||||
{ ifAtGeneration: db.generation } // rejects if ANYTHING committed since the read
|
||||
)
|
||||
return committed.receipt!.ids[0]
|
||||
} catch (err) {
|
||||
if (err instanceof GenerationConflictError) continue // world moved — re-read + retry
|
||||
throw err
|
||||
} finally {
|
||||
await db.release()
|
||||
}
|
||||
}
|
||||
throw new Error('addIfMissingByEmail conflict after 5 attempts')
|
||||
}
|
||||
```
|
||||
|
||||
`ifAtGeneration` is deliberately coarse — *any* committed write invalidates it — so keep the retry bound. When you control the ID, `ifAbsent` stays the cheaper tool.
|
||||
|
||||
## How `_rev` relates to generations
|
||||
|
||||
Brainy 8.0 has exactly two write-coordination counters, at two granularities:
|
||||
|
||||
| Counter | Scope | What it tracks | CAS surface | Conflict error |
|
||||
|---|---|---|---|---|
|
||||
| **`_rev`** | One entity | Per-entity write count, bumped on every successful update | `update({ ifRev })`, `{ op: 'update', ifRev }` in `transact()` | `RevisionConflictError` |
|
||||
| **Generation** | Whole store | One tick per committed `transact()` batch or single-operation write | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` |
|
||||
|
||||
They compose: a `transact()` batch can carry per-entity `ifRev` checks *and* a whole-store `ifAtGeneration`; any failed check rejects the entire batch before anything is staged. Generations also power snapshots and time travel (`brain.now()`, `brain.asOf()`, `db.persist()`) — see the [consistency model](../concepts/consistency-model.md) and [Snapshots & Time Travel](./snapshots-and-time-travel.md).
|
||||
|
||||
A snapshot or historical view captures each entity *including* its `_rev` at that moment, so reading the past and writing back with `ifRev` against the live state works exactly as you'd hope: the write fails if the entity moved since the state you copied from.
|
||||
|
||||
## The transact envelope: batch size, budget, and bulk imports
|
||||
|
||||
`transact()` applies its batch atomically under one commit — which means the whole batch
|
||||
shares one **apply budget**. Since 8.7.0 the budget scales with the batch:
|
||||
`max(30 s, opCount × 2 s)`, or exactly what you pass as `timeoutMs`. A tripped budget rolls
|
||||
the entire batch back (nothing partial survives) and throws a retryable
|
||||
`TransactionTimeoutError` that names the operation it stopped at, the batch size, and the
|
||||
elapsed vs budgeted time — a diagnosis, not just a failure:
|
||||
|
||||
```
|
||||
Transaction timed out at operation 41/120 ('add') — 246012ms elapsed, budget 240000ms.
|
||||
The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch.
|
||||
```
|
||||
|
||||
Practical envelope guidance for bulk work:
|
||||
|
||||
1. **Precompute embeddings outside the commit path.** Embedding inside `transact()` spends
|
||||
the budget on model inference. Use `brain.embedBatch(texts)` and pass each vector via
|
||||
the op's `vector` field — the commit then pays only storage costs, and a retried batch
|
||||
never re-pays inference. (The win is *where* the inference happens, not raw embedding
|
||||
throughput: on the default WASM engine, batch and sequential embedding measure
|
||||
comparably, ~160 ms/text; native embedding providers may batch faster.)
|
||||
2. **Chunk very large imports** into batches of a few hundred ops with one `transact()`
|
||||
each. You lose whole-import atomicity but keep per-chunk atomicity, bounded memory, and
|
||||
resumability — pair with `ifAbsent` upserts so a retried chunk is idempotent.
|
||||
3. **Slow disks change the math, not the contract.** On network-attached storage a single
|
||||
op can cost ~2 s (canonical write + fsync + index maintenance). The scaled default
|
||||
absorbs that; pass an explicit `timeoutMs` only when you know better than the scale.
|
||||
4. **`addMany`/`relateMany` are the convenience tier** — they chunk and batch-embed for
|
||||
you, with per-item error reporting instead of batch atomicity. Choose by what you need:
|
||||
atomic-all-or-nothing → `transact()`; resilient bulk load → `addMany`.
|
||||
|
|
@ -1,111 +0,0 @@
|
|||
---
|
||||
title: Quick Start
|
||||
slug: getting-started/quick-start
|
||||
public: true
|
||||
category: getting-started
|
||||
template: guide
|
||||
order: 2
|
||||
description: Build your first knowledge graph in 60 seconds. Add entities, create relationships, and query with Triple Intelligence — vector + graph + metadata in one call.
|
||||
next:
|
||||
- concepts/triple-intelligence
|
||||
- api/reference
|
||||
---
|
||||
|
||||
# Quick Start
|
||||
|
||||
Get Brainy running in under a minute.
|
||||
|
||||
## 1. Install
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/brainy
|
||||
```
|
||||
|
||||
## 2. Initialize
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
That's it. Brainy auto-configures storage, loads the embedding model, and builds the indexes.
|
||||
|
||||
## 3. Add Knowledge
|
||||
|
||||
```typescript
|
||||
// Text is automatically embedded into 384-dim vectors
|
||||
const reactId: string = await brain.add({
|
||||
data: 'React is a JavaScript library for building user interfaces',
|
||||
type: NounType.Concept,
|
||||
subtype: 'library', // Sub-classification within Concept
|
||||
metadata: { category: 'frontend', year: 2013 }
|
||||
})
|
||||
|
||||
const nextId: string = await brain.add({
|
||||
data: 'Next.js framework for React with server-side rendering',
|
||||
type: NounType.Concept,
|
||||
subtype: 'framework',
|
||||
metadata: { category: 'framework', year: 2016 }
|
||||
})
|
||||
```
|
||||
|
||||
`type` is one of Brainy's 42 stable NounTypes. `subtype` is your free-form sub-classification within that type — flat string, no hierarchy, indexed on the fast path. See **[Subtypes & Facets](./subtypes-and-facets.md)** for the full guide.
|
||||
|
||||
## 4. Create Relationships
|
||||
|
||||
```typescript
|
||||
// Typed graph relationships
|
||||
await brain.relate({
|
||||
from: nextId,
|
||||
to: reactId,
|
||||
type: VerbType.DependsOn
|
||||
})
|
||||
```
|
||||
|
||||
## 5. Query with Triple Intelligence
|
||||
|
||||
```typescript
|
||||
import type { Result } from '@soulcraft/brainy'
|
||||
|
||||
// All three search paradigms in one call
|
||||
const results: Result[] = await brain.find({
|
||||
query: 'modern frontend frameworks', // Vector similarity search
|
||||
where: { year: { greaterThan: 2015 } }, // Metadata filtering
|
||||
connected: { to: reactId, depth: 2 } // Graph traversal
|
||||
})
|
||||
|
||||
console.log(results[0].data) // 'Next.js framework for React...'
|
||||
console.log(results[0].score) // 0.94
|
||||
```
|
||||
|
||||
## What Just Happened
|
||||
|
||||
Every entity you `add()` lives in three indexes simultaneously:
|
||||
|
||||
| Index | What it stores | Query with |
|
||||
|-------|---------------|------------|
|
||||
| Vector | 384-dim embedding of `data` | `find({ query: '...' })` |
|
||||
| Metadata | All `metadata` fields | `find({ where: { ... } })` |
|
||||
| Graph | Typed relationships from `relate()` | `find({ connected: { ... } })` |
|
||||
|
||||
`find()` queries all three in parallel and fuses the results.
|
||||
|
||||
## Natural Language Queries
|
||||
|
||||
Brainy understands 220+ natural language patterns:
|
||||
|
||||
```typescript
|
||||
// These all work without any configuration
|
||||
await brain.find({ query: 'recent documents about machine learning' })
|
||||
await brain.find({ query: 'articles created this week' })
|
||||
await brain.find({ query: 'people who work at Anthropic' })
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Triple Intelligence](/docs/concepts/triple-intelligence) — understand how the query engine works
|
||||
- [The Find System](/docs/guides/find-system) — advanced queries, operators, and graph traversal
|
||||
- [API Reference](/docs/api/reference) — complete method documentation
|
||||
- [Storage Adapters](/docs/guides/storage-adapters) — filesystem, memory
|
||||
|
|
@ -1,117 +0,0 @@
|
|||
---
|
||||
title: Reacting to Changes
|
||||
slug: guides/reacting-to-changes
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 11
|
||||
description: Subscribe to every committed mutation with `brain.onChange` — the in-process change feed behind live UIs, cache invalidation, and realtime sync. Covers the event shape, delivery guarantees, and catch-up patterns.
|
||||
next:
|
||||
- guides/optimistic-concurrency
|
||||
- guides/snapshots-and-time-travel
|
||||
---
|
||||
|
||||
# Reacting to Changes
|
||||
|
||||
`brain.onChange(cb)` is Brainy's in-process change feed: subscribe once and
|
||||
receive one event per committed mutation — **every** mutation, regardless of
|
||||
how it happened. Direct calls, batch methods, `transact()`, imports, and
|
||||
Virtual Filesystem writes all funnel through the same commit point the feed is
|
||||
emitted from, so nothing slips past it.
|
||||
|
||||
```ts
|
||||
const off = brain.onChange((e) => {
|
||||
if (e.kind === 'entity') {
|
||||
console.log(`${e.op} ${e.entity?.type} ${e.id} @ generation ${e.generation}`)
|
||||
}
|
||||
})
|
||||
|
||||
await brain.add({ data: 'Ada Lovelace', type: 'person' })
|
||||
// → "add person 0198... @ generation 42"
|
||||
|
||||
off() // unsubscribe when done
|
||||
```
|
||||
|
||||
## The event
|
||||
|
||||
```ts
|
||||
interface BrainyChangeEvent {
|
||||
kind: 'entity' | 'relation' | 'store'
|
||||
op: 'add' | 'update' | 'remove' | 'relate' | 'unrelate' | 'updateRelation'
|
||||
| 'clear' | 'restore'
|
||||
id?: string
|
||||
entity?: { id: string; type: string; subtype?: string;
|
||||
metadata: Record<string, unknown>; service?: string }
|
||||
relation?: { id: string; from: string; to: string; type: string;
|
||||
metadata?: Record<string, unknown> }
|
||||
generation?: number
|
||||
timestamp: number
|
||||
}
|
||||
```
|
||||
|
||||
- **Entity events** (`add` / `update` / `remove`) carry the post-commit indexed
|
||||
view — `type`, `subtype`, and the full custom `metadata`, so you can match
|
||||
your own `where`-style filters against events without a read.
|
||||
- **Deletes are fully described.** A `remove` or `unrelate` event carries the
|
||||
record's *last committed state* (sourced from the commit's own history
|
||||
record), not just an id.
|
||||
- **Batches emit per item.** `addMany` / `updateMany` / `relateMany` /
|
||||
`removeMany` emit one event per affected record; a `transact()` batch emits
|
||||
one event per item, all sharing the batch's single `generation`.
|
||||
- **Cascades are visible.** Removing an entity also emits `unrelate` for each
|
||||
relationship the delete cascaded to.
|
||||
- **Store-level events** (`kind: 'store'`) fire for the two wholesale
|
||||
operations — `clear()` and `restore()` — and mean *"everything may have
|
||||
changed; refetch what you care about."*
|
||||
|
||||
## Delivery guarantees
|
||||
|
||||
- **Post-commit only.** An aborted write — a losing
|
||||
[`ifRev` compare-and-swap](optimistic-concurrency.md), a rejected
|
||||
transaction — never emits. If you received the event, the write is durable.
|
||||
- **Commit-ordered.** Events arrive in the order writes committed;
|
||||
`generation` is monotonic.
|
||||
- **Asynchronous, never blocking.** Delivery happens in a microtask after the
|
||||
write completes. A slow listener cannot delay a write; a throwing listener
|
||||
is logged and isolated from other listeners.
|
||||
- **Zero overhead when unused.** With no subscribers, the write path does no
|
||||
event work at all.
|
||||
- **Fire-and-forget.** There is no replay or backpressure. For catch-up after
|
||||
a disconnect, use the `generation` on each event together with
|
||||
[`asOf()` / the transaction log](snapshots-and-time-travel.md): record the
|
||||
last generation you processed, and on reconnect diff from there. For file
|
||||
content specifically, `vfs.readFile(path, { asOf })` and
|
||||
`vfs.history(path)` are the temporal read — see
|
||||
[Snapshots & Time Travel](snapshots-and-time-travel.md).
|
||||
|
||||
## Patterns
|
||||
|
||||
**Cache invalidation** — drop cached reads for whatever changed:
|
||||
|
||||
```ts
|
||||
brain.onChange((e) => {
|
||||
if (e.kind === 'store') return cache.clear()
|
||||
if (e.id) cache.delete(e.id)
|
||||
})
|
||||
```
|
||||
|
||||
**Live queries (notify-and-refetch)** — re-run a query when a relevant change
|
||||
lands, rather than diffing incrementally:
|
||||
|
||||
```ts
|
||||
brain.onChange((e) => {
|
||||
if (e.kind === 'entity' && e.entity?.type === 'order') {
|
||||
refreshOpenOrdersView() // debounce as needed
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Forwarding to other processes** — the feed is in-process by design. To push
|
||||
changes to browsers or other services, forward events through your own
|
||||
transport (WebSocket, SSE) from the process that owns the brain.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
`onChange` returns an unsubscribe function — call it when tearing down a
|
||||
subscriber (for example, when evicting a pooled instance). `brain.close()`
|
||||
drops all listeners; no events are delivered for or after `close()`.
|
||||
|
|
@ -1,284 +0,0 @@
|
|||
# Schema Migrations
|
||||
|
||||
Brainy includes a built-in migration system for transforming entity and verb metadata across storage versions. Migrations are pure functions that run once per storage instance, with optional snapshot backup (`backupTo`), resume support, and error tracking.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Define a migration
|
||||
|
||||
Add your migration to the `MIGRATIONS` array in `src/migration/migrations.ts`:
|
||||
|
||||
```typescript
|
||||
import type { Migration } from './types.js'
|
||||
|
||||
export const MIGRATIONS: Migration[] = [
|
||||
{
|
||||
id: '7.17.0-rename-status',
|
||||
version: '7.17.0',
|
||||
description: 'Rename "state" field to "status"',
|
||||
applies: 'nouns',
|
||||
transform: (m) => {
|
||||
if ('state' in m) {
|
||||
const { state, ...rest } = m
|
||||
return { ...rest, status: state }
|
||||
}
|
||||
return null // already migrated or not applicable
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### 2. Ship the new version
|
||||
|
||||
That's it. Brainy detects pending migrations on `init()` and either runs them automatically or warns the user to call `brain.migrate()`.
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
When `brain.init()` runs:
|
||||
|
||||
1. **Detection** — reads migration state from storage (one key lookup). Compares completed migration IDs against the `MIGRATIONS` array. If nothing pending, cost is ~0ms.
|
||||
|
||||
2. **Small datasets** (`autoMigrate: true`, <10K entities) — migrates inline during `init()`.
|
||||
|
||||
3. **Large datasets or manual mode** — logs a warning. User calls `brain.migrate()` when ready.
|
||||
|
||||
When `brain.migrate()` runs:
|
||||
|
||||
1. **Backup (optional)** — with `backupTo`, a hard-link snapshot of the current generation is persisted before any transform runs. Rollback is `brain.restore(backupPath, { confirm: true })`.
|
||||
|
||||
2. **Transform** — iterates all nouns/verbs in paginated batches. For each entity, calls the `transform` function. If it returns a new object, saves it. If it returns `null`, skips. Vectors are never touched.
|
||||
|
||||
3. **Save state** — records each completed migration ID so it never re-runs.
|
||||
|
||||
4. **Rebuild indexes** — if any entities were modified, rebuilds the MetadataIndex.
|
||||
|
||||
---
|
||||
|
||||
## Writing Migrations
|
||||
|
||||
### The Migration interface
|
||||
|
||||
```typescript
|
||||
interface Migration {
|
||||
id: string // Unique ID, e.g. "7.17.0-rename-field"
|
||||
version: string // Version that introduced this migration
|
||||
description: string // Human-readable description
|
||||
applies: 'nouns' | 'verbs' | 'both'
|
||||
transform: (metadata: Record<string, unknown>) => Record<string, unknown> | null
|
||||
}
|
||||
```
|
||||
|
||||
### Transform rules
|
||||
|
||||
- **Return a new object** to modify the entity's metadata.
|
||||
- **Return `null`** to skip (no change needed).
|
||||
- **Must be idempotent** — running the same transform twice on the same data should produce the same result (or return `null` the second time). This is required because interrupted runs resume and re-encounter already-migrated entities.
|
||||
- **Must be pure** — no side effects, no async, no external state.
|
||||
- Transforms only see metadata. Vectors, embeddings, and the `data` field stored inside metadata are available as properties on the metadata object.
|
||||
|
||||
### Ordering
|
||||
|
||||
Migrations run in array order. Add new migrations at the end of the `MIGRATIONS` array. Each migration runs independently per entity — migration 2 sees the output of migration 1.
|
||||
|
||||
### Validation
|
||||
|
||||
`MigrationRunner.validateMigrations()` checks migration definitions and will throw on:
|
||||
|
||||
- Duplicate IDs
|
||||
- Invalid `applies` values (must be `'nouns'`, `'verbs'`, or `'both'`)
|
||||
- Non-function `transform`
|
||||
- Missing or empty `id`, `version`, or `description`
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### `brain.migrate(options?)`
|
||||
|
||||
```typescript
|
||||
// Dry-run: preview what would change without writing
|
||||
const preview = await brain.migrate({ dryRun: true })
|
||||
// preview.pendingMigrations — array of { id, description }
|
||||
// preview.affectedEntities — count of entities that would change
|
||||
// preview.totalEntities — count of entities scanned
|
||||
// preview.sampleChanges — up to 5 before/after samples
|
||||
// preview.estimatedTime — rough time estimate string
|
||||
|
||||
// Apply migrations (optionally with a pre-migration snapshot)
|
||||
const result = await brain.migrate({ backupTo: '/backups/pre-migration' })
|
||||
// result.backupPath — snapshot path, or null when no backupTo was supplied
|
||||
// result.migrationsApplied — array of migration IDs that ran
|
||||
// result.entitiesProcessed — total entities scanned
|
||||
// result.entitiesModified — entities actually changed
|
||||
// result.errors — array of entity-level errors (non-fatal)
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```typescript
|
||||
interface MigrateOptions {
|
||||
dryRun?: boolean // Preview without writing (default: false)
|
||||
maxErrors?: number // Bail out after N entity errors (default: 100)
|
||||
onProgress?: (progress: {
|
||||
migrationId: string
|
||||
processed: number
|
||||
modified: number
|
||||
hasMore: boolean
|
||||
}) => void
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
If a transform function throws on a specific entity, the error is recorded and migration continues to the next entity. The failed entity's metadata is left unchanged.
|
||||
|
||||
```typescript
|
||||
const result = await brain.migrate()
|
||||
|
||||
if (result.errors.length > 0) {
|
||||
for (const err of result.errors) {
|
||||
console.warn(`Entity ${err.entityId} failed in ${err.migrationId}: ${err.error}`)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If errors exceed `maxErrors` (default: 100), the migration stops early and returns partial results. Successfully migrated entities keep their changes; failed entities are unchanged.
|
||||
|
||||
```typescript
|
||||
// Strict mode: fail fast on any error
|
||||
const result = await brain.migrate({ maxErrors: 1 })
|
||||
|
||||
// Lenient mode: tolerate many errors
|
||||
const result = await brain.migrate({ maxErrors: 10000 })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backup and Rollback
|
||||
|
||||
Pass `backupTo` and `brain.migrate()` persists a snapshot of the current generation **before any transform runs**. On filesystem storage the snapshot is a hard-link farm — created without copying entity data, and immune to later writes (see [Snapshots & Time Travel](./snapshots-and-time-travel.md)):
|
||||
|
||||
```typescript
|
||||
const result = await brain.migrate({ backupTo: '/backups/pre-migration-8.0' })
|
||||
console.log(result.backupPath) // '/backups/pre-migration-8.0' (null when no backupTo)
|
||||
```
|
||||
|
||||
To roll back, restore the snapshot wholesale:
|
||||
|
||||
```typescript
|
||||
await brain.restore('/backups/pre-migration-8.0', { confirm: true })
|
||||
```
|
||||
|
||||
Without `backupTo`, no backup is taken — transforms are idempotent (they return `null` when already applied), but a pre-migration snapshot is the cheap insurance for anything destructive.
|
||||
|
||||
---
|
||||
|
||||
## Progress Tracking
|
||||
|
||||
For large datasets, use the `onProgress` callback:
|
||||
|
||||
```typescript
|
||||
await brain.migrate({
|
||||
onProgress: ({ migrationId, processed, modified, hasMore }) => {
|
||||
console.log(`[${migrationId}] ${processed} scanned, ${modified} modified${hasMore ? '...' : ' (done)'}`)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
Progress is reported after each batch (batch size is determined by the storage adapter).
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Rename a field
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: '7.17.0-rename-state-to-status',
|
||||
version: '7.17.0',
|
||||
description: 'Rename metadata.state to metadata.status',
|
||||
applies: 'nouns',
|
||||
transform: (m) => {
|
||||
if ('state' in m) {
|
||||
const { state, ...rest } = m
|
||||
return { ...rest, status: state }
|
||||
}
|
||||
return null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Add a default value
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: '7.18.0-add-priority-default',
|
||||
version: '7.18.0',
|
||||
description: 'Add priority field with default "normal"',
|
||||
applies: 'both',
|
||||
transform: (m) => {
|
||||
if (!('priority' in m)) {
|
||||
return { ...m, priority: 'normal' }
|
||||
}
|
||||
return null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Remove a deprecated field
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: '7.19.0-remove-legacy-flag',
|
||||
version: '7.19.0',
|
||||
description: 'Remove deprecated "legacy" field',
|
||||
applies: 'nouns',
|
||||
transform: (m) => {
|
||||
if ('legacy' in m) {
|
||||
const { legacy, ...rest } = m
|
||||
return rest
|
||||
}
|
||||
return null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Transform verb metadata
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: '7.20.0-normalize-verb-weights',
|
||||
version: '7.20.0',
|
||||
description: 'Normalize verb weights from 0-100 to 0-1 scale',
|
||||
applies: 'verbs',
|
||||
transform: (m) => {
|
||||
if (typeof m.weight === 'number' && m.weight > 1) {
|
||||
return { ...m, weight: m.weight / 100 }
|
||||
}
|
||||
return null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Storage Backend Compatibility
|
||||
|
||||
Migrations work identically across all storage backends (Memory, FileSystem). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter.
|
||||
|
||||
Batch size and rate limiting are automatically configured per adapter — no tuning required.
|
||||
|
||||
---
|
||||
|
||||
## What Migrations Don't Do
|
||||
|
||||
- **Re-embedding** — migrations transform metadata only. If you change your embedding model or dimensions, that requires re-vectorizing data, which is a separate concern (not part of this system).
|
||||
- **Vector modification** — the `vectors.json` files are never touched by migrations.
|
||||
- **Schema enforcement** — migrations are opt-in transforms, not schema validators. Brainy's metadata is schemaless by design.
|
||||
|
|
@ -1,449 +0,0 @@
|
|||
---
|
||||
title: Snapshots & Time Travel
|
||||
slug: guides/snapshots-and-time-travel
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 9
|
||||
description: Recipes for the Db API — instant backups with persist(), restore, time-travel debugging with asOf(), range queries over history (diff, history, since, log windows), persist-before-migrate, what-if analysis with with(), and audit trails via transaction metadata.
|
||||
next:
|
||||
- concepts/consistency-model
|
||||
- guides/optimistic-concurrency
|
||||
- guides/external-backups
|
||||
---
|
||||
|
||||
# Snapshots & Time Travel
|
||||
|
||||
Brainy 8.0 treats the database as a **value**: `brain.now()` pins the
|
||||
current state as an immutable `Db`, `brain.transact()` commits an atomic
|
||||
batch and hands you the resulting value, `brain.asOf()` opens past state,
|
||||
and `db.persist()` cuts a self-contained snapshot. This guide is the recipe
|
||||
book. The precise guarantees behind every recipe live in the
|
||||
[consistency model](../concepts/consistency-model.md).
|
||||
|
||||
## Instant backup
|
||||
|
||||
Pin the current state, persist it, release:
|
||||
|
||||
```typescript
|
||||
const db = brain.now()
|
||||
try {
|
||||
await db.persist('/backups/2026-06-11')
|
||||
} finally {
|
||||
await db.release()
|
||||
}
|
||||
```
|
||||
|
||||
On filesystem storage the snapshot is built from **hard links**: every data
|
||||
file in Brainy is immutable-by-rename, so the snapshot is created without
|
||||
copying entity data and shares disk space with the live store. Later writes
|
||||
can never alter it — a rewrite swaps the inode, the snapshot keeps the old
|
||||
bytes. Cross-device targets fall back to per-file byte copies, and
|
||||
persisting an in-memory brain serializes it to the same directory layout —
|
||||
a real, durable store.
|
||||
|
||||
> Archiving a brain directory with **external tools** (`tar`, `rsync`, `cp`)?
|
||||
> Some index files are sparse and can explode to their apparent size under a
|
||||
> naive copy — see [External Backups & Sparse Storage](/docs/guides/external-backups).
|
||||
|
||||
Two things to know:
|
||||
|
||||
- `persist()` requires the view to still be the store's **latest**
|
||||
generation. If something committed after your pin, it throws
|
||||
`GenerationConflictError` instead of snapshotting the wrong state — pin
|
||||
and persist before further writes, or retry with a fresh `brain.now()`.
|
||||
- The target directory must be empty or absent.
|
||||
|
||||
For scheduled backups, this loop is the whole job:
|
||||
|
||||
```typescript
|
||||
const db = brain.now()
|
||||
try {
|
||||
await db.persist(`/backups/${new Date().toISOString().slice(0, 10)}`)
|
||||
} finally {
|
||||
await db.release()
|
||||
}
|
||||
```
|
||||
|
||||
## Restore
|
||||
|
||||
`restore()` replaces the store's **entire** current state from a snapshot —
|
||||
entities, relationships, indexes, history. It is deliberately loud about it:
|
||||
|
||||
```typescript
|
||||
await brain.restore('/backups/2026-06-11', { confirm: true })
|
||||
```
|
||||
|
||||
- `{ confirm: true }` is mandatory — current state is destroyed.
|
||||
- The snapshot is copied in (never linked), so it stays independent and can
|
||||
be restored again later.
|
||||
- All indexes are rebuilt from the restored records.
|
||||
- The generation counter is floored at its pre-restore value, so generation
|
||||
numbers you observed before the restore are never reissued.
|
||||
- Live `Db` pins do not survive a restore — release them first.
|
||||
|
||||
## Open a snapshot read-only
|
||||
|
||||
You do not have to restore to look inside a snapshot. `Brainy.load()` opens
|
||||
it as a self-contained read-only store with the **full query surface**,
|
||||
including vector search:
|
||||
|
||||
```typescript
|
||||
const db = await Brainy.load('/backups/2026-06-11')
|
||||
|
||||
const hits = await db.find({ query: 'unpaid invoices from the spring campaign' })
|
||||
const orders = await db.find({ type: NounType.Document, subtype: 'order' })
|
||||
|
||||
await db.release() // closes the underlying read-only instance
|
||||
```
|
||||
|
||||
`brain.asOf('/backups/2026-06-11')` does the same from an existing brain.
|
||||
This is also the 8.0 answer to "named branches": a branch is a name → path
|
||||
mapping your application keeps, where each path is a persisted snapshot.
|
||||
Need a writable copy? Restore the snapshot into a fresh data directory and
|
||||
open a writer on it — instead of switching a shared store between branches
|
||||
in place, every line of code always sees exactly the store it opened.
|
||||
|
||||
## Time-travel debugging
|
||||
|
||||
When production data looks wrong, query the past directly — by wall-clock
|
||||
time or by generation:
|
||||
|
||||
```typescript
|
||||
// What did this order look like yesterday?
|
||||
const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000))
|
||||
const before = await yesterday.get(orderId)
|
||||
|
||||
// Full queries work at any reachable generation — search, graph, filters:
|
||||
const thenActive = await yesterday.find({
|
||||
type: NounType.Document,
|
||||
subtype: 'order',
|
||||
where: { status: 'active' }
|
||||
})
|
||||
|
||||
await yesterday.release()
|
||||
```
|
||||
|
||||
Pin two points in time and diff them:
|
||||
|
||||
```typescript
|
||||
const before = await brain.asOf(1041)
|
||||
const after = brain.now()
|
||||
|
||||
const changed = await after.since(before)
|
||||
changed.nouns // entity ids touched by transactions in between
|
||||
changed.verbs // relationship ids touched in between
|
||||
|
||||
await before.release()
|
||||
await after.release()
|
||||
```
|
||||
|
||||
Three things to remember:
|
||||
|
||||
- History granularity is per-write: EVERY write — `transact()` AND a
|
||||
single-operation `add`/`update`/`remove`/`relate` — is its own immutable
|
||||
generation, so a pin always freezes against later writes and every write is
|
||||
individually addressable via `asOf()` (see the
|
||||
[consistency model](../concepts/consistency-model.md)). Use `transact()` when
|
||||
you want several operations to share ONE atomic generation.
|
||||
- The first index-accelerated query (semantic search, traversal, cursors,
|
||||
aggregation) at a historical generation builds an in-memory index
|
||||
materialization — O(n at that generation), once per `Db`, freed on
|
||||
`release()`. Metadata-level reads are free.
|
||||
- Generations reclaimed by `compactHistory()` throw
|
||||
`GenerationCompactedError` — persist anything you need to keep forever.
|
||||
|
||||
## Range queries over history
|
||||
|
||||
`asOf()` answers "what was the state AT a point". Four range verbs answer
|
||||
"what happened BETWEEN two points" and "what is one entity's whole history".
|
||||
They all build on the same generation records — no extra bookkeeping.
|
||||
|
||||
### `diff(a, b)` — what changed, classified
|
||||
|
||||
`since()` gives you the raw set of *touched* ids. `diff()` goes further: it
|
||||
resolves each touched id at both endpoints and classifies it as **added**,
|
||||
**removed**, or **modified** — split by entities (`nouns`) and relationships
|
||||
(`verbs`). An id that was touched but ended up identical (changed then
|
||||
reverted, or created and deleted within the interval) lands in **none** of the
|
||||
buckets. Endpoints are a generation, a `Date`, or a `Db`, in either order:
|
||||
|
||||
```typescript
|
||||
const d = await brain.diff(1041, brain.generation())
|
||||
|
||||
d.added.nouns // entity ids created between the two states
|
||||
d.removed.nouns // entity ids deleted
|
||||
d.modified.nouns // entity ids whose stored value actually changed
|
||||
d.added.verbs // …relationships, the same three ways
|
||||
```
|
||||
|
||||
Orientation is `a → b`: `added` means "exists at `b`, not at `a`". The
|
||||
comparison behind `modified` is key-order-insensitive, so a no-op re-write of
|
||||
the same fields never shows up as a change.
|
||||
|
||||
### `history(id, range?)` — one entity, every version
|
||||
|
||||
`asOf()` is per-*generation*; `history()` is per-*entity*. It returns every
|
||||
distinct version of one id over a range, oldest first — each `value` is the
|
||||
materialized state at that version (and `null` marks a removal):
|
||||
|
||||
```typescript
|
||||
const h = await brain.history(invoiceId)
|
||||
|
||||
for (const v of h.versions) {
|
||||
console.log(v.generation, v.value?.metadata?.status ?? '(deleted)')
|
||||
}
|
||||
// 1041 'draft'
|
||||
// 1043 'approved'
|
||||
// 1050 'paid'
|
||||
```
|
||||
|
||||
Every version ties to the trusted `asOf()` path — `v.value` equals
|
||||
`(await brain.asOf(v.generation)).get(id)`. Pass `{ from, to }` (generation or
|
||||
`Date`) to bound the range; a `from` below the compaction horizon is quietly
|
||||
truncated to it rather than throwing (history is best-effort over surviving
|
||||
records).
|
||||
|
||||
### `since()` and `transactionLog()` take ranges too
|
||||
|
||||
`since()` accepts a `Db`, a generation number, or a `Date` — all equivalent,
|
||||
all an **exclusive** lower bound (`db.since(prior)` equals
|
||||
`db.since(prior.generation)`):
|
||||
|
||||
```typescript
|
||||
await brain.now().since(1041) // ids changed after generation 1041
|
||||
await brain.now().since(new Date(Date.now() - 3_600_000)) // …in the last hour
|
||||
```
|
||||
|
||||
`transactionLog({ from, to })` windows the commit log **inclusively** on both
|
||||
ends (a log window names the commits it spans — the deliberate contrast to
|
||||
`since`'s exclusive lower bound); `limit` applies after the window, newest
|
||||
first:
|
||||
|
||||
```typescript
|
||||
const window = await brain.transactionLog({ from: 1041, to: 1050 }) // commits 1041…1050
|
||||
const recent = await brain.transactionLog({ from: lastHour, limit: 20 })
|
||||
```
|
||||
|
||||
### Composing them
|
||||
|
||||
"Which orders changed in this window?" is `diff` ids intersected with an
|
||||
`asOf` query — the two agree by construction:
|
||||
|
||||
```typescript
|
||||
const changed = await brain.diff(g1, g2)
|
||||
const atG2 = await brain.asOf(g2)
|
||||
const changedOrders = (await atG2.find({ type: NounType.Document, subtype: 'order' }))
|
||||
.map(r => r.id)
|
||||
.filter(id => changed.added.nouns.includes(id) || changed.modified.nouns.includes(id))
|
||||
await atG2.release()
|
||||
```
|
||||
|
||||
One contrast to keep straight: `diff` and `since` **throw**
|
||||
`GenerationCompactedError` for a bound below the horizon, while `history`
|
||||
**truncates** to the horizon — diffs must be exact, history is best-effort.
|
||||
|
||||
## Safe schema migration
|
||||
|
||||
`brain.migrate()` integrates with snapshots directly: pass `backupTo` and a
|
||||
hard-link snapshot of the current generation is persisted **before any
|
||||
transform runs**:
|
||||
|
||||
```typescript
|
||||
const result = await brain.migrate({ backupTo: '/backups/pre-migration-8.0' })
|
||||
console.log(result.migrationsApplied, result.backupPath)
|
||||
|
||||
// If the migration went wrong, roll the whole store back:
|
||||
await brain.restore('/backups/pre-migration-8.0', { confirm: true })
|
||||
```
|
||||
|
||||
The same persist-before-mutate pattern works for any risky bulk operation,
|
||||
not just migrations:
|
||||
|
||||
```typescript
|
||||
const pin = brain.now()
|
||||
try {
|
||||
await pin.persist('/backups/pre-bulk-edit')
|
||||
} finally {
|
||||
await pin.release()
|
||||
}
|
||||
await runRiskyBulkEdit(brain)
|
||||
```
|
||||
|
||||
## What-if analysis
|
||||
|
||||
`db.with(ops)` applies a transaction **speculatively, in memory** — nothing
|
||||
touches disk, the generation counter, or the indexes. Ask "what would the
|
||||
store look like if…", then commit the same operations for real:
|
||||
|
||||
```typescript
|
||||
const ops = [
|
||||
{ op: 'update', id: employeeId, metadata: { team: 'platform' } },
|
||||
{ op: 'relate', from: employeeId, to: milestoneId, type: VerbType.ParticipatesIn, subtype: 'assignment' }
|
||||
]
|
||||
|
||||
const base = brain.now()
|
||||
const whatIf = await base.with(ops)
|
||||
|
||||
await whatIf.get(employeeId) // sees the change
|
||||
await whatIf.find({ where: { team: 'platform' } }) // metadata finds work
|
||||
await whatIf.related(employeeId) // overlay relations included
|
||||
|
||||
await whatIf.release()
|
||||
await base.release()
|
||||
|
||||
// Looks right — make it real, atomically:
|
||||
await brain.transact(ops)
|
||||
```
|
||||
|
||||
**The boundary:** speculative entities carry no embeddings (`with()` never
|
||||
invokes the embedder), so semantic search, traversal, cursors, aggregation,
|
||||
and `persist()` throw `SpeculativeOverlayError` on overlay views instead of
|
||||
returning silently incomplete results. `get()`, metadata-filter `find()`,
|
||||
and filter-based `related()` are fully supported. Overlays chain — calling
|
||||
`with()` on an overlay stacks another layer.
|
||||
|
||||
## Audit trails
|
||||
|
||||
`transact()` reifies transaction metadata: whatever you pass as `meta` is
|
||||
recorded durably alongside the committed generation and timestamp, readable
|
||||
via `brain.transactionLog()`:
|
||||
|
||||
```typescript
|
||||
await brain.transact(
|
||||
[{ op: 'update', id: invoiceId, metadata: { status: 'approved' } }],
|
||||
{ meta: { author: 'approvals-service', actor: 'jane@example.com', reason: 'PO-7741' } }
|
||||
)
|
||||
|
||||
const log = await brain.transactionLog({ limit: 20 }) // newest first
|
||||
// [{ generation: 1042, timestamp: 1765432100000, meta: { author: 'approvals-service', ... } }]
|
||||
```
|
||||
|
||||
Combine the log with `asOf()` to reconstruct exactly what any transaction
|
||||
did:
|
||||
|
||||
```typescript
|
||||
const [entry] = await brain.transactionLog({ limit: 1 })
|
||||
|
||||
const after = await brain.asOf(entry.generation)
|
||||
const before = await brain.asOf(entry.generation - 1)
|
||||
|
||||
const touched = await after.since(before)
|
||||
for (const id of touched.nouns) {
|
||||
console.log(id, await before.get(id), '→', await after.get(id))
|
||||
}
|
||||
|
||||
await before.release()
|
||||
await after.release()
|
||||
```
|
||||
|
||||
For per-entity write coordination (rather than whole-store history), the
|
||||
`_rev` counter and `ifRev` CAS remain the right tool — see
|
||||
[optimistic concurrency](./optimistic-concurrency.md).
|
||||
|
||||
## Keeping history bounded
|
||||
|
||||
Under Model-B every write is a generation, so history can grow quickly —
|
||||
Brainy auto-compacts at `close()` (time-bounded per pass) under the
|
||||
**`retention`** knob (configured on the constructor). Since 8.9.0, `flush()`
|
||||
never compacts: flushing is durability work and costs only what the current
|
||||
window's writes cost, regardless of history backlog. A long-lived writer that
|
||||
never closes keeps its history until its next explicit `compactHistory()` —
|
||||
schedule one in your maintenance window if you run bounded retention:
|
||||
|
||||
```typescript
|
||||
// Zero-config: ADAPTIVE — keep as much history as free disk/RAM allows,
|
||||
// reclaiming oldest-first under pressure. (This is the default.)
|
||||
new Brainy({ /* retention unset */ })
|
||||
|
||||
// Unbounded — never reclaim history (opt in explicitly):
|
||||
new Brainy({ retention: 'all' })
|
||||
|
||||
// Explicit CAPS — reclaim oldest-unpinned generations while ANY cap is exceeded:
|
||||
new Brainy({ retention: { maxGenerations: 1000, maxAge: 7 * 86_400_000, maxBytes: 512 * 1024 ** 2 } })
|
||||
```
|
||||
|
||||
Reclaim manually at any time (the same caps, plus an optional per-pass time
|
||||
budget for maintenance windows — an early stop is a consistent prefix and the
|
||||
next pass resumes):
|
||||
|
||||
```typescript
|
||||
await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 24 * 60 * 60 * 1000 })
|
||||
await brain.compactHistory({ maxBytes: 512 * 1024 ** 2, timeBudgetMs: 10_000 })
|
||||
```
|
||||
|
||||
Compaction never breaks a pinned read — record-sets are reclaimed only when
|
||||
no live `Db` could need them (live pins are ALWAYS exempt). Release views you
|
||||
are done with (including the ones `transact()` returns), and `persist()` any
|
||||
generation you want to keep beyond the retention window: snapshots are
|
||||
self-contained and unaffected by compaction.
|
||||
|
||||
## Time travel for files (the VFS)
|
||||
|
||||
Since 8.2.0, time travel covers Virtual Filesystem **content**, not just
|
||||
entity records. File bytes are retention-protected: a content blob referenced
|
||||
by any generation inside the retention window is never reclaimed, so reading
|
||||
the past always returns the exact bytes — never a stale field or a
|
||||
dangling hash.
|
||||
|
||||
**`vfs.readFile(path, { asOf })`** takes a generation number or a `Date` and
|
||||
returns the file's exact bytes as they stood then. It resolves the path's
|
||||
current entity, then materializes its state at the target generation — so it
|
||||
answers *"what did the file at this path hold at that point?"* It bypasses
|
||||
the content cache; the `encoding` option still applies. Asking about a
|
||||
generation before the file existed throws the usual not-found error, and
|
||||
asking past the retention window's compaction horizon throws a
|
||||
compacted-generation error.
|
||||
|
||||
**`vfs.history(path)`** returns the file's versions inside the retention
|
||||
window, oldest first — one `FileVersion` per generation that wrote the file,
|
||||
the newest entry being the current state:
|
||||
|
||||
```typescript
|
||||
// A CMS page evolves…
|
||||
await brain.vfs.writeFile('/pages/home.json', '{"title":"Launch"}')
|
||||
await brain.vfs.writeFile('/pages/home.json', '{"title":"Launch v2"}')
|
||||
await brain.vfs.writeFile('/pages/home.json', '{"title":""}') // bad deploy!
|
||||
|
||||
// Every version is listed and readable:
|
||||
const versions = await brain.vfs.history('/pages/home.json')
|
||||
// → [{ generation, timestamp, hash, size, mimeType? }, …] ascending
|
||||
|
||||
const good = versions[versions.length - 2]
|
||||
const bytes = await brain.vfs.readFile('/pages/home.json', {
|
||||
asOf: good.generation
|
||||
})
|
||||
|
||||
// Restore = write the old bytes back. This is a NEW write (a new
|
||||
// generation) — history is never rewritten, so the bad version stays
|
||||
// visible in the audit trail.
|
||||
await brain.vfs.writeFile('/pages/home.json', bytes)
|
||||
```
|
||||
|
||||
Two lifecycle consequences worth stating plainly:
|
||||
|
||||
- **Deleting or overwriting a file no longer frees its bytes immediately.**
|
||||
Old content lives until history compaction reclaims the generations that
|
||||
reference it — the same `retention` budget that bounds all Model-B history
|
||||
(and pinned views are exempt, exactly as above). Size your `retention` for
|
||||
the file-version depth you want; `retention: 'all'` keeps every version of
|
||||
every file forever.
|
||||
- **After `compactHistory()` reclaims a generation, its file versions are
|
||||
gone** and their bytes are physically reclaimed. (This also fixed a
|
||||
pre-8.2.0 defect where overwritten content was never reclaimed at all — an
|
||||
unbounded silent leak.)
|
||||
|
||||
## From branches to values
|
||||
|
||||
If you used the pre-8.0 `fork`/`checkout`/`commit`/`versions` surface, every
|
||||
use case maps to a sharper tool:
|
||||
|
||||
| Pre-8.0 habit | 8.0 recipe |
|
||||
|---|---|
|
||||
| `fork()` to experiment safely | `db.with(ops)` for speculation in memory; a restored snapshot in a fresh directory for a long-lived writable copy |
|
||||
| `commit()` checkpoints | `transact(ops, { meta })` — every batch is an atomic, logged, time-travelable commit |
|
||||
| `checkout()` to switch branches | Open the snapshot you want — `Brainy.load(path)` read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store. |
|
||||
| `getHistory()` | `brain.transactionLog()` + `db.since(priorDb)` |
|
||||
| `versions.save()` per-entity snapshots | A pinned `Db` or persisted snapshot captures *every* entity at that moment; `asOf()` reads any entity's past state |
|
||||
| `versions.restore()` | `brain.restore(snapshot, { confirm: true })` for the whole store, or read the old entity via `asOf()` and write it back with `transact()` |
|
||||
| Backup branches | `db.persist(path)` — instant, hard-link-shared, self-contained |
|
||||
|
|
@ -1,162 +0,0 @@
|
|||
---
|
||||
title: Storage Adapters
|
||||
slug: guides/storage-adapters
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 2
|
||||
description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both share one on-disk contract, including generational history and snapshots. Cloud backup is operator tooling, not a built-in adapter."
|
||||
next:
|
||||
- guides/plugins
|
||||
- concepts/consistency-model
|
||||
---
|
||||
|
||||
# Storage Adapters
|
||||
|
||||
Brainy 8.0 ships **two storage adapters**:
|
||||
|
||||
- **`FileSystemStorage`** — persistent on-disk storage. The default for any
|
||||
deployment that needs to survive a restart. Runs on Node.js, Bun, and Deno.
|
||||
- **`MemoryStorage`** — in-memory only. The right choice for tests, ephemeral
|
||||
workloads, and short-lived demos.
|
||||
|
||||
Both implement the same `StorageAdapter` interface, support the full Db API
|
||||
(generational history, snapshots, restore — see the
|
||||
[consistency model](../concepts/consistency-model.md)), and use the same
|
||||
on-disk layout (memory's "disk" is a JS Map).
|
||||
|
||||
## Quick start
|
||||
|
||||
```ts
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// Filesystem (recommended for any persistent workload):
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './brainy-data' }
|
||||
})
|
||||
|
||||
// Memory (tests, ephemeral):
|
||||
const brainMem = new Brainy({ storage: { type: 'memory' } })
|
||||
|
||||
// Auto-detect (filesystem on Node-like runtimes, memory in browsers):
|
||||
const brainAuto = new Brainy({ storage: { type: 'auto' } })
|
||||
```
|
||||
|
||||
## When to use which
|
||||
|
||||
| Use case | Adapter | Why |
|
||||
|---|---|---|
|
||||
| Production app | `filesystem` | Durable, snapshot-able, mmap-able |
|
||||
| Tests, CI | `memory` | No disk teardown; fast |
|
||||
| Short-lived data pipeline | `memory` | No persistence needed |
|
||||
| In-browser demo | `memory` | Filesystem unavailable in browsers |
|
||||
| Cloud deployment | `filesystem` on local disk + operator backup | See "Cloud backup" below |
|
||||
|
||||
## Cloud backup — operator tooling, not a built-in
|
||||
|
||||
Brainy 8.0 deliberately ships **no cloud storage adapters**. Cloud backup is
|
||||
handled at the operator layer with standard tooling, the same pattern every
|
||||
production database uses (Postgres, SQLite, Redis):
|
||||
|
||||
```bash
|
||||
# After a brainy flush, sync the on-disk artefact to your cloud of choice.
|
||||
gsutil rsync -r /var/lib/brainy gs://my-backup-bucket/brainy/
|
||||
# or:
|
||||
aws s3 sync /var/lib/brainy s3://my-backup-bucket/brainy/
|
||||
# or:
|
||||
rclone sync /var/lib/brainy remote:brainy-backups/
|
||||
# or:
|
||||
azcopy sync /var/lib/brainy "https://account.blob.core.windows.net/brainy?sv=..."
|
||||
```
|
||||
|
||||
Brainy's filesystem layout is sync-friendly:
|
||||
- Atomic writes (temp + rename) — readers never see torn files
|
||||
- Per-shard files — `rsync`-style incremental sync works well
|
||||
- Immutable generation records (`_generations/`) — append-only, cache-friendly
|
||||
|
||||
For point-in-time backups, take a filesystem snapshot (ZFS, btrfs, LVM, EBS,
|
||||
etc.) or use `brain.now().persist(path)` to write a self-contained snapshot
|
||||
you can sync independently of the live brain — see
|
||||
[Snapshots & Time Travel](./snapshots-and-time-travel.md).
|
||||
|
||||
## Why no cloud adapters in 8.0?
|
||||
|
||||
Cloud storage adapters lived in Brainy 4.x-7.x. They were dropped in 8.0
|
||||
because:
|
||||
|
||||
- Zero production consumers used them at scale — every known production
|
||||
deployment ran on local filesystem.
|
||||
- Cloud-storage HNSW / DiskANN doesn't perform — vector indexes need
|
||||
low-latency random reads that S3 / GCS / R2 / Azure can't provide
|
||||
consistently.
|
||||
- Bundling cloud SDKs into the library cost ~3000-5000 LOC + 4-7 transitive
|
||||
dependencies for a feature nobody used.
|
||||
- Cloud backup via operator tooling is strictly more reliable than in-app
|
||||
upload (better retry semantics, better observability, better cost control).
|
||||
|
||||
Brainy 8.0 is smaller, faster to install, and clearer about what it does.
|
||||
|
||||
## Configuration
|
||||
|
||||
```ts
|
||||
// BrainyConfig['storage'] — either a config object or a pre-constructed adapter:
|
||||
storage?:
|
||||
| {
|
||||
// The adapter type. Optional — a top-level `path` implies 'filesystem',
|
||||
// so { path: '/data' } works without it. Defaults to 'auto'
|
||||
// (filesystem on Node-like runtimes, memory otherwise).
|
||||
type?: 'auto' | 'memory' | 'filesystem'
|
||||
|
||||
// CANONICAL directory for filesystem storage. The rest of the API already
|
||||
// speaks `path` (persist(path), Brainy.load(path), asOf(path),
|
||||
// restore(path)). Specifying it implies type: 'filesystem'. Passed through
|
||||
// to storage factories, including plugin-provided ones.
|
||||
path?: string
|
||||
|
||||
// REMOVED in 8.0 — passing `rootDirectory` THROWS. Use `path`.
|
||||
rootDirectory?: string
|
||||
|
||||
// REMOVED in 8.0 — a nested `options.{path,rootDirectory}` THROWS. Use `path`.
|
||||
options?: any
|
||||
}
|
||||
| StorageAdapter // e.g. storage: new MemoryStorage()
|
||||
```
|
||||
|
||||
The canonical — and only — key is the top-level **`path`**. The pre-8.0 aliases
|
||||
`rootDirectory`, `options.{path,rootDirectory}`, and
|
||||
`fileSystemStorage.{path,rootDirectory}` were **removed in 8.0**: passing one now
|
||||
throws with the exact rename (never a silent default that would misplace data on
|
||||
upgrade). Because `path` implies filesystem, `{ path: '/data' }` is a complete
|
||||
config; the `type` is optional.
|
||||
|
||||
## Direct construction
|
||||
|
||||
If you want to skip the factory:
|
||||
|
||||
```ts
|
||||
import { FileSystemStorage, MemoryStorage } from '@soulcraft/brainy'
|
||||
|
||||
const fsStorage = new FileSystemStorage('./brainy-data')
|
||||
const memStorage = new MemoryStorage()
|
||||
|
||||
const brain = new Brainy({ storage: fsStorage })
|
||||
```
|
||||
|
||||
## Migration from 7.x cloud adapters
|
||||
|
||||
7.x consumers of `OPFSStorage`, `GcsStorage`, `R2Storage`, `S3CompatibleStorage`,
|
||||
or `AzureBlobStorage` need to migrate to `FileSystemStorage` plus operator
|
||||
backup tooling. The recipe:
|
||||
|
||||
1. On the host running Brainy, mount a local disk (NVMe recommended). Cloud
|
||||
providers all expose persistent local disks: GCP Persistent Disk, AWS EBS,
|
||||
Azure Managed Disks.
|
||||
2. Set `storage: { type: 'filesystem', path: '/mnt/brainy-data' }`.
|
||||
3. Run your existing data import once into the new local store.
|
||||
4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` /
|
||||
`azcopy` on a cron — hourly or whatever your RPO requires. Point it at
|
||||
the brainy data dir.
|
||||
5. For point-in-time backups, use filesystem snapshots or
|
||||
`brain.now().persist(path)`.
|
||||
|
||||
Same data, same APIs, no library-side cloud code.
|
||||
|
|
@ -1,575 +0,0 @@
|
|||
---
|
||||
title: Subtypes & Facets
|
||||
slug: guides/subtypes-and-facets
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 7
|
||||
description: Use the top-level `subtype` field to sub-classify entities within a NounType, track other metadata facets like status or role, and migrate field names without downtime.
|
||||
next:
|
||||
- guides/aggregation
|
||||
- api/reference
|
||||
---
|
||||
|
||||
# Subtypes & Facets
|
||||
|
||||
> Sub-classify entities within a NounType, track arbitrary metadata facets, and migrate field names without downtime.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Brainy's `NounType` (Person, Document, Event, Concept, Task, …) is a stable, flat 42-type taxonomy. It's deliberately coarse — every product needs a way to further classify entities *within* a type. Is this `Person` an employee or a customer? Is this `Document` an invoice or a contract? Is this `Event` a meeting or a milestone?
|
||||
|
||||
Three layers solve this:
|
||||
|
||||
| Layer | Use it for | Shape |
|
||||
|---|---|---|
|
||||
| **`subtype`** | The primary sub-classification of an entity, one value per entity | Top-level standard field |
|
||||
| **`trackField()`** | Other facets you want to count or filter on (`status`, `source`, `role`, `paradigm`) | Registered metadata field |
|
||||
| **`migrateField()`** | Renaming or restructuring fields across an existing dataset | One-shot stream-and-rewrite |
|
||||
|
||||
## Layer 1 — `subtype`
|
||||
|
||||
`subtype` is a top-level standard field on every entity, alongside `type` / `confidence` / `weight`. Flat string, no hierarchy. The vocabulary is your choice — Brainy stores and counts, never validates.
|
||||
|
||||
### Write
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
await brain.add({
|
||||
data: 'Avery Brooks — runs the AI lab',
|
||||
type: NounType.Person,
|
||||
subtype: 'employee', // top-level write param
|
||||
metadata: { department: 'ai-lab' }
|
||||
})
|
||||
```
|
||||
|
||||
### Read
|
||||
|
||||
```typescript
|
||||
// Top-level filter — standard-field fast path, no `where` wrapper:
|
||||
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// Set membership:
|
||||
const internal = await brain.find({
|
||||
type: NounType.Person,
|
||||
subtype: ['employee', 'contractor']
|
||||
})
|
||||
|
||||
// Operator-form predicates use `where`:
|
||||
const typed = await brain.find({
|
||||
type: NounType.Person,
|
||||
where: { subtype: { exists: true } }
|
||||
})
|
||||
```
|
||||
|
||||
### What it looks like
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "01HZK3M7TW...",
|
||||
"type": "person",
|
||||
"subtype": "employee",
|
||||
"data": "Avery Brooks — runs the AI lab",
|
||||
"metadata": { "department": "ai-lab" }
|
||||
}
|
||||
```
|
||||
|
||||
`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's what makes the fast path possible: queries on `subtype` hit the column-store index directly, never the metadata fallback.
|
||||
|
||||
### Counts (O(1))
|
||||
|
||||
```typescript
|
||||
// All subtypes for a NounType
|
||||
brain.counts.bySubtype(NounType.Person)
|
||||
// → { employee: 12, customer: 847, vendor: 34 }
|
||||
|
||||
// Point count
|
||||
brain.counts.bySubtype(NounType.Person, 'employee')
|
||||
// → 12
|
||||
|
||||
// Top N
|
||||
brain.counts.topSubtypes(NounType.Person, 3)
|
||||
// → [['customer', 847], ['employee', 12], ['vendor', 34]]
|
||||
|
||||
// Distinct subtypes for a NounType
|
||||
brain.subtypesOf(NounType.Person)
|
||||
// → ['customer', 'employee', 'vendor']
|
||||
```
|
||||
|
||||
These are O(1) lookups backed by `_system/subtype-statistics.json` — no scan, no storage round-trip. The rollup is incrementally maintained as entities are added, updated, and deleted.
|
||||
|
||||
### Aggregation
|
||||
|
||||
`subtype` is a first-class group-by dimension:
|
||||
|
||||
```typescript
|
||||
const rows = await brain.find({
|
||||
aggregate: {
|
||||
groupBy: ['type', 'subtype'],
|
||||
metrics: { count: { op: 'COUNT' } }
|
||||
}
|
||||
})
|
||||
// [
|
||||
// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 },
|
||||
// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 },
|
||||
// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 },
|
||||
// ...
|
||||
// ]
|
||||
```
|
||||
|
||||
## Layer 2 — `trackField()`
|
||||
|
||||
Some metadata fields aren't *the* sub-classification of an entity but are still worth counting: `status`, `source`, `role`, `paradigm`. Promoting each one to top-level would clutter the contract. `trackField()` registers a metadata field for cardinality + per-NounType breakdown stats without the contract growth.
|
||||
|
||||
### Register and query
|
||||
|
||||
```typescript
|
||||
// Track a single facet
|
||||
brain.trackField('status')
|
||||
|
||||
await brain.add({ data: 'Ship subtype', type: NounType.Task, metadata: { status: 'todo' } })
|
||||
await brain.add({ data: 'Write docs', type: NounType.Task, metadata: { status: 'done' } })
|
||||
|
||||
await brain.counts.byField('status')
|
||||
// → { todo: 1, done: 1 }
|
||||
```
|
||||
|
||||
### Per-NounType breakdown
|
||||
|
||||
```typescript
|
||||
brain.trackField('status', { perType: true })
|
||||
|
||||
await brain.counts.byField('status', { type: NounType.Task })
|
||||
// → { todo: 1, done: 1 }
|
||||
```
|
||||
|
||||
### Vocabulary whitelist (opt-in validation)
|
||||
|
||||
```typescript
|
||||
brain.trackField('priority', { values: ['low', 'medium', 'high'] })
|
||||
|
||||
// Throws — 'urgent' isn't in the vocabulary:
|
||||
await brain.add({
|
||||
data: 'Fix bug',
|
||||
type: NounType.Task,
|
||||
metadata: { priority: 'urgent' }
|
||||
})
|
||||
```
|
||||
|
||||
`trackField` piggybacks on the existing aggregation engine (see the [Aggregation guide](./aggregation.md) for the underlying mechanism). Backfill-on-define means the first call to `counts.byField()` scans existing entities; subsequent calls are O(groups).
|
||||
|
||||
### subtype vs trackField — when to use which
|
||||
|
||||
- **`subtype`** when there's one primary sub-classification per entity. Limit yourself to one per NounType. Examples: Person→employee/customer/vendor; Document→invoice/contract/policy.
|
||||
- **`trackField`** for anything else you want to count or filter on. No limit on how many you register. Examples: status, source, role, paradigm.
|
||||
|
||||
## Layer 3 — `migrateField()`
|
||||
|
||||
Use this when you need to rename or restructure a field across an entire dataset — for example, moving a `metadata.kind` convention up to the top-level `subtype` standard field.
|
||||
|
||||
### One-shot rewrite
|
||||
|
||||
```typescript
|
||||
// Starting state: every entity has metadata.kind
|
||||
const result = await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype'
|
||||
})
|
||||
|
||||
console.log(result)
|
||||
// {
|
||||
// scanned: 1500,
|
||||
// migrated: 1500,
|
||||
// skipped: 0,
|
||||
// errors: []
|
||||
// }
|
||||
```
|
||||
|
||||
After this returns, every entity has `subtype` populated from the old `metadata.kind` value, and `metadata.kind` is cleared.
|
||||
|
||||
### Deprecation window — keep both fields readable
|
||||
|
||||
When you can't coordinate all readers and the migration in a single deploy, use `readBoth: true` to preserve the source field alongside the new one:
|
||||
|
||||
```typescript
|
||||
// Phase 1: dual-populate (existing readers still work against metadata.kind):
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
readBoth: true
|
||||
})
|
||||
|
||||
// ... readers migrate to query subtype at their own pace ...
|
||||
|
||||
// Phase 2: clear the source field when ready:
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
```
|
||||
|
||||
### Supported paths
|
||||
|
||||
| Path form | Refers to |
|
||||
|---|---|
|
||||
| `'subtype'`, `'type'`, `'confidence'` | Top-level standard fields |
|
||||
| `'metadata.X'` | A key under `entity.metadata` |
|
||||
| `'data.X'` | A key under `entity.data` (when `data` is an object) |
|
||||
| `'X'` (bare, non-standard) | Shorthand for `metadata.X` |
|
||||
|
||||
### Idempotent
|
||||
|
||||
`migrateField` is safe to re-run. Entities where the source is absent, or where the destination already holds the same value, are skipped. This makes it safe to use in a deploy-once-then-cleanup workflow.
|
||||
|
||||
### Progress reporting
|
||||
|
||||
```typescript
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
batchSize: 500,
|
||||
onProgress: ({ scanned, migrated }) => {
|
||||
console.log(`${scanned} scanned, ${migrated} migrated`)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Putting it together
|
||||
|
||||
A realistic adoption sequence for a brain that started without these primitives:
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', path: './brain-data' } })
|
||||
await brain.init()
|
||||
|
||||
// 1. Migrate any existing metadata.kind convention to the new top-level subtype
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype', readBoth: true })
|
||||
|
||||
// 2. Register the other facets you want counted
|
||||
brain.trackField('status', { perType: true })
|
||||
brain.trackField('source')
|
||||
|
||||
// 3. Use subtype on every new write
|
||||
await brain.add({
|
||||
data: 'Quarterly review',
|
||||
type: NounType.Event,
|
||||
subtype: 'milestone',
|
||||
metadata: { status: 'todo', source: 'planning-session' }
|
||||
})
|
||||
|
||||
// 4. Query the breakdowns
|
||||
brain.counts.bySubtype(NounType.Event)
|
||||
// → { milestone: 14, meeting: 203, deadline: 7 }
|
||||
|
||||
await brain.counts.byField('status', { type: NounType.Event })
|
||||
// → { todo: 12, done: 212 }
|
||||
|
||||
// 5. Once all readers are on the new field, drop the source:
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
```
|
||||
|
||||
## Layer V — `subtype` on relationships (verbs)
|
||||
|
||||
Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive: a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field, indexed on the fast path.
|
||||
|
||||
### Write
|
||||
|
||||
```typescript
|
||||
const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' })
|
||||
const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' })
|
||||
const matrixId = await brain.add({ type: NounType.Person, subtype: 'contractor', data: 'Sam' })
|
||||
|
||||
await brain.relate({
|
||||
from: ceoId,
|
||||
to: vpId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'direct'
|
||||
})
|
||||
|
||||
await brain.relate({
|
||||
from: ceoId,
|
||||
to: matrixId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'dotted-line'
|
||||
})
|
||||
```
|
||||
|
||||
### Read & filter
|
||||
|
||||
```typescript
|
||||
// Direct reports — fast path filter (column-store hit, not metadata fallback)
|
||||
const direct = await brain.related({
|
||||
from: ceoId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'direct'
|
||||
})
|
||||
|
||||
// Set membership
|
||||
const allReports = await brain.related({
|
||||
from: ceoId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: ['direct', 'dotted-line']
|
||||
})
|
||||
```
|
||||
|
||||
### Update (new — `updateRelation()`)
|
||||
|
||||
Verbs previously had no update method — the only way to change a relationship was delete-then-recreate. 7.30 closes that gap:
|
||||
|
||||
```typescript
|
||||
// Promote a dotted-line report to direct without losing the edge id
|
||||
await brain.updateRelation({ id: relationId, subtype: 'direct' })
|
||||
|
||||
// Or change weight/confidence
|
||||
await brain.updateRelation({ id: relationId, weight: 0.5, confidence: 0.9 })
|
||||
```
|
||||
|
||||
### Traversal filter
|
||||
|
||||
`find({ connected, subtype })` filters traversal edges by their subtype. Composes with `via` (verb-type filter):
|
||||
|
||||
```typescript
|
||||
// All direct reports two hops deep (will be supported in Cor; for now depth-1)
|
||||
const directChain = await brain.find({
|
||||
connected: {
|
||||
from: ceoId,
|
||||
via: VerbType.ReportsTo,
|
||||
subtype: 'direct',
|
||||
depth: 1
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
Multi-hop subtype filtering (`depth > 1`) lights up on the Cor native path; the JS path throws today rather than return incorrect partial results.
|
||||
|
||||
### Counts
|
||||
|
||||
Same shape as the noun-side counts API:
|
||||
|
||||
```typescript
|
||||
brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
|
||||
// → { direct: 12, 'dotted-line': 3 }
|
||||
|
||||
brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct') // O(1) point
|
||||
// → 12
|
||||
|
||||
brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
|
||||
// → [['direct', 12], ['dotted-line', 3]]
|
||||
|
||||
brain.relationshipSubtypesOf(VerbType.ReportsTo)
|
||||
// → ['direct', 'dotted-line']
|
||||
```
|
||||
|
||||
These are O(1) lookups backed by the persisted `_system/verb-subtype-statistics.json` rollup — same self-heal machinery as the noun-side rollup.
|
||||
|
||||
### Migrate verb fields
|
||||
|
||||
`migrateField()` now walks verbs too:
|
||||
|
||||
```typescript
|
||||
// Migrate verb-side metadata.kind → top-level subtype
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
entityKind: 'verb'
|
||||
})
|
||||
|
||||
// Or migrate both nouns and verbs in one pass
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
entityKind: 'both'
|
||||
})
|
||||
```
|
||||
|
||||
Default is `entityKind: 'noun'` (backward-compatible with 7.29).
|
||||
|
||||
## Enforcement — `requireSubtype()` + brain-wide strict mode
|
||||
|
||||
By default (7.30) subtype is optional. Two complementary opt-in mechanisms let you enforce the pairing of type + subtype on every write:
|
||||
|
||||
### Per-type registration
|
||||
|
||||
Mark a specific `NounType` or `VerbType` as requiring a subtype, optionally with a fixed vocabulary:
|
||||
|
||||
```typescript
|
||||
brain.requireSubtype(NounType.Person, {
|
||||
values: ['employee', 'customer', 'vendor'],
|
||||
required: true
|
||||
})
|
||||
|
||||
brain.requireSubtype(VerbType.ReportsTo, {
|
||||
values: ['direct', 'dotted-line'],
|
||||
required: true
|
||||
})
|
||||
|
||||
// Now this throws — Person requires a subtype:
|
||||
await brain.add({ type: NounType.Person, data: 'no subtype' })
|
||||
|
||||
// And this throws — 'matrix' isn't in the registered vocabulary:
|
||||
await brain.relate({
|
||||
from: a,
|
||||
to: b,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'matrix'
|
||||
})
|
||||
```
|
||||
|
||||
### Brain-wide strict mode
|
||||
|
||||
Enforce on every write across the whole brain:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({ requireSubtype: true })
|
||||
|
||||
// Allow specific types to omit subtype (e.g. catch-all `Thing`)
|
||||
const brain2 = new Brainy({
|
||||
requireSubtype: { except: [NounType.Thing, NounType.Custom] }
|
||||
})
|
||||
```
|
||||
|
||||
When strict mode is on:
|
||||
|
||||
- Every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()` rejects writes missing a subtype on a non-exempt type.
|
||||
- `addMany()` and `relateMany()` validate every item BEFORE any storage write — atomic-fail semantics, no partial writes.
|
||||
- Per-type rules registered via `requireSubtype()` compose with the brain-wide flag; specific rules win when both apply.
|
||||
- Brainy's own internal writes (VFS root, VFS directories, VFS file entities) bypass enforcement via the `metadata.isVFSEntity: true` infrastructure marker.
|
||||
|
||||
### Migrating to strict mode on an existing brain
|
||||
|
||||
`brain.migrateField()` is your friend — populate `subtype` from an existing convention before flipping strict mode on:
|
||||
|
||||
```typescript
|
||||
// Step 1: backfill subtype from your existing metadata.kind convention
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
entityKind: 'both'
|
||||
})
|
||||
|
||||
// Step 2: register the vocabulary
|
||||
brain.requireSubtype(NounType.Person, {
|
||||
values: ['employee', 'customer', 'vendor'],
|
||||
required: true
|
||||
})
|
||||
|
||||
// Step 3: future writes must have a subtype matching the vocabulary
|
||||
await brain.add({ type: NounType.Person, subtype: 'employee', data: '...' })
|
||||
```
|
||||
|
||||
## Strict mode in practice (for SDK-style vocabulary consumers)
|
||||
|
||||
When a platform layer like the Soulcraft SDK registers `requireSubtype()` rules on behalf of every consumer's brain, every downstream product that calls `brain.add()` / `brain.relate()` against those types must pass a matching `subtype`. Skipping the field — or passing one outside the registered vocabulary — throws at the boundary.
|
||||
|
||||
This pattern is powerful but surfaces a class of latent bug: any `brain.add()` call site that was written before strict-mode adoption starts rejecting writes. A representative production incident: a booking flow started returning 500 on every request because a service method called `brain.add({ type: NounType.Event, ... })` without subtype, and an SDK layer had just registered `requireSubtype()` for `NounType.Event` on every brain instance.
|
||||
|
||||
The fix is a four-step migration recipe — and Brainy 7.30.1+ ships diagnostic tools to make it deterministic.
|
||||
|
||||
### Migration recipe
|
||||
|
||||
1. **Inventory the gap with `brain.audit()`** — returns the deterministic list of which NounTypes and VerbTypes have entities/relationships missing subtype, grouped by type:
|
||||
|
||||
```typescript
|
||||
const report = await brain.audit()
|
||||
// {
|
||||
// entitiesWithoutSubtype: { event: 24, document: 3, ... },
|
||||
// relationshipsWithoutSubtype: { relatedTo: 1402 },
|
||||
// total: 1429,
|
||||
// scanned: 8400,
|
||||
// recommendation: 'Found 1429 entries without subtype. ...'
|
||||
// }
|
||||
```
|
||||
|
||||
By default, VFS infrastructure entities are excluded (they bypass enforcement anyway via the `metadata.isVFSEntity` marker). Pass `{ includeVFS: true }` to surface them too.
|
||||
|
||||
2. **Bulk-migrate any existing convention** with `brain.migrateField()` if a legacy field can be lifted:
|
||||
|
||||
```typescript
|
||||
// Common pattern: subtype mirrors a discriminator field already in metadata
|
||||
await brain.migrateField({
|
||||
from: 'metadata.entityType',
|
||||
to: 'subtype',
|
||||
readBoth: true // safety: keep the source field readable during cutover
|
||||
})
|
||||
```
|
||||
|
||||
3. **Hand-fix the remaining call sites.** The exact list is in `report.entitiesWithoutSubtype`. For each call site, add `subtype: '<value>'` to the `brain.add()` / `brain.relate()` params. Choose a stable convention (e.g. mirror `metadata.entityType` if you have one; any rule that's deterministic from the data works).
|
||||
|
||||
4. **Verify with `brain.audit()` again.** Re-run; total should be `0`. If you turn on brain-wide strict mode at this point, all future writes are protected.
|
||||
|
||||
### Brainy's own infrastructure subtype labels (reference)
|
||||
|
||||
Brainy's internal write paths set subtype on every entity and edge they create. Consumers don't need to do anything for these — they're documented here so you understand the data shape:
|
||||
|
||||
| Code path | NounType / VerbType | Subtype label |
|
||||
|---|---|---|
|
||||
| VFS root directory `/` | `NounType.Collection` | `'vfs-root'` |
|
||||
| VFS subdirectories | `NounType.Collection` | `'vfs-directory'` |
|
||||
| VFS files | (mime-driven, e.g. `Document`/`Code`/`Image`) | `'vfs-file'` |
|
||||
| VFS symlinks | `NounType.File` | `'vfs-symlink'` |
|
||||
| VFS Contains edges | `VerbType.Contains` | `'vfs-contains'` |
|
||||
| Aggregation materialized output | `NounType.Measurement` | `'materialized-aggregate'` |
|
||||
| Import-document provenance entity | `NounType.Document` | `'import-source'` |
|
||||
| Importer-extracted entities (no caller default) | extractor-driven | `'imported'` |
|
||||
| Importer placeholder targets | `NounType.Thing` | `'import-placeholder'` |
|
||||
| Neural extraction (no caller default) | extractor-driven | `'extracted'` |
|
||||
| GoogleSheets API entity writes | request-driven | `'imported-from-sheets'` |
|
||||
| OData API entity writes | request-driven | `'imported-from-odata'` |
|
||||
| MCP client message storage | `NounType.Message` | `'mcp-message'` |
|
||||
| `brainy add` CLI (no `--subtype` flag) | user-supplied type | `'cli-add'` |
|
||||
| `brainy relate` CLI (no `--subtype` flag) | user-supplied verb | `'cli-relate'` |
|
||||
|
||||
You can query these directly: `await brain.find({ subtype: 'vfs-file' })` returns every VFS-managed file regardless of NounType. `await brain.counts.bySubtype(NounType.Document)` shows you the import-source / imported / extracted / vfs-file breakdown.
|
||||
|
||||
Importer and extraction paths accept a caller-supplied `defaultSubtype` option so you can tag a whole batch with your own provenance label (e.g. `'customer-upload-2026q2'`) instead of the Brainy default `'imported'` / `'extracted'`.
|
||||
|
||||
### Looking ahead — Brainy 8.0
|
||||
|
||||
Brainy 8.0 ships:
|
||||
|
||||
- **`brain.fillSubtypes(rules)`** — the bulk migration helper that pairs with `audit()`. Given caller-supplied rules per NounType / VerbType, it walks the brain and fills in missing subtypes via `update()`. Pre-8.0 brains run this once before upgrading to clear migration debt.
|
||||
- **`subtype: string` (non-optional)** on `AddParams<T>` and `RelateParams<T>`. TypeScript catches missing subtype at compile time, not just runtime.
|
||||
- **`new Brainy({ requireSubtype: true })` becomes the default.** Consumers explicitly opt out with `{ requireSubtype: false }` during migration.
|
||||
|
||||
7.30.1's `audit()` is the diagnostic; 8.0's `fillSubtypes()` is the bulk fixer. Together they close the migration gap deterministically.
|
||||
|
||||
## Reference
|
||||
|
||||
### Layer 1 — `subtype` (nouns)
|
||||
|
||||
- `brain.add({ ..., subtype: 'value' })` — write the field
|
||||
- `brain.update({ id, subtype: 'value' })` — change the field
|
||||
- `brain.find({ type, subtype })` — filter (fast path)
|
||||
- `brain.find({ subtype: ['a', 'b'] })` — set membership
|
||||
- `brain.counts.bySubtype(type, subtype?)` — O(1) counts
|
||||
- `brain.counts.topSubtypes(type, n?)` — top N by count
|
||||
- `brain.subtypesOf(type)` — distinct subtype list
|
||||
|
||||
### Layer V — `subtype` (verbs / relationships)
|
||||
|
||||
- `brain.relate({ ..., subtype: 'value' })` — write the field
|
||||
- `brain.updateRelation({ id, subtype, type?, weight?, ... })` — change a relationship
|
||||
- `brain.related({ subtype })` — filter (fast path)
|
||||
- `brain.related({ subtype: ['a', 'b'] })` — set membership
|
||||
- `brain.find({ connected: { via, subtype, depth } })` — traversal filter
|
||||
- `brain.counts.byRelationshipSubtype(verb, subtype?)` — O(1) counts
|
||||
- `brain.counts.topRelationshipSubtypes(verb, n?)` — top N by count
|
||||
- `brain.relationshipSubtypesOf(verb)` — distinct subtype list
|
||||
|
||||
### Layer 2 — generic facets
|
||||
|
||||
- `brain.trackField(name, { perType?, values? })` — register a facet
|
||||
- `brain.counts.byField(name, { type? })` — facet counts
|
||||
|
||||
### Layer 3 — migration
|
||||
|
||||
- `brain.migrateField({ from, to, readBoth?, batchSize?, onProgress?, entityKind? })` — rewrite a field (nouns, verbs, or both)
|
||||
|
||||
### Enforcement
|
||||
|
||||
- `brain.requireSubtype(type, { values?, required })` — per-`NounType` / `VerbType` rule
|
||||
- `new Brainy({ requireSubtype: true })` — brain-wide strict mode
|
||||
- `new Brainy({ requireSubtype: { except: [type, ...] } })` — strict with exemptions
|
||||
|
|
@ -1,186 +0,0 @@
|
|||
---
|
||||
title: Upgrading from 7.x to 8.0
|
||||
slug: guides/upgrading-7-to-8
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 10
|
||||
description: What the one-time 7→8 on-disk migration does, how 8.0 automatically recovers Virtual Filesystem content that older layouts stored in the removed copy-on-write area, and how to verify (or force) that recovery.
|
||||
next:
|
||||
- guides/storage-adapters
|
||||
- guides/inspection
|
||||
---
|
||||
|
||||
# Upgrading from 7.x to 8.0
|
||||
|
||||
Opening a 7.x on-disk store with Brainy 8.0 runs a **one-time, in-place layout
|
||||
migration** the first time the store is opened. It is automatic (`autoMigrate`
|
||||
defaults to `true`), it runs once, and it stamps a marker so every later open is
|
||||
a no-op.
|
||||
|
||||
Almost everything about the upgrade is transparent: your entities, relationships,
|
||||
metadata, and indexes migrate and rebuild without any action on your part. This
|
||||
guide covers the **one case that needs attention** — Virtual Filesystem (VFS)
|
||||
content — and how 8.0 recovers it for you.
|
||||
|
||||
## TL;DR
|
||||
|
||||
- **Just upgrade to `@soulcraft/brainy@8.0.12` (or later) and open the store.**
|
||||
If a previous upgrade left VFS content stranded, 8.0.12 **heals it on open**,
|
||||
with no operator action.
|
||||
- Want to force or script it? Call **`await brain.vfs.adoptOrphanedBlobs()`**.
|
||||
- The recovery is **non-destructive and idempotent** — it copies, never moves,
|
||||
and running it twice is a no-op.
|
||||
|
||||
## What the migration does
|
||||
|
||||
The 7.x on-disk layout stored entities under a per-branch path
|
||||
(`branches/<head>/entities/...`). 8.0 uses a flat layout (`entities/...`). On
|
||||
first open, Brainy:
|
||||
|
||||
1. Collapses `branches/<head>/entities/*` into the flat `entities/*` layout.
|
||||
2. Rebuilds the derived indexes (vector, graph, metadata) and count rollups from
|
||||
the canonical entities.
|
||||
3. Stamps `_system/migration-layout.json` so re-opening is a no-op.
|
||||
4. Takes an automatic **pre-upgrade backup** of the directory before it starts,
|
||||
and removes it once the upgrade is verified complete (see
|
||||
[The safety net](#the-safety-net-pre-upgrade-backup) below).
|
||||
|
||||
The log line to expect (once per store):
|
||||
|
||||
```
|
||||
[brainy] Migrating a 7.x branch layout (branches/main) to the 8.0 flat layout
|
||||
in place — 13175 entity files. This runs once; back up the directory first if
|
||||
you need a rollback (8.0 does not keep the old layout).
|
||||
```
|
||||
|
||||
## The one thing that needs recovery: VFS content
|
||||
|
||||
If your application uses the Virtual Filesystem (VFS) to store file content (for
|
||||
example, a CMS that keeps page documents at paths like
|
||||
`/pages/homepage/page.json`), that content is held as **content blobs**.
|
||||
|
||||
7.x kept those blobs in the branch system's **copy-on-write area** (`_cow/`).
|
||||
8.0 removed the branch/copy-on-write system, and its content blobs live in the
|
||||
content-addressed store (`_cas/`). The layout migration moves entities — it does
|
||||
**not** move the VFS content blobs. Left alone, a 7.x store's VFS blobs would
|
||||
remain in `_cow/`, and a read of one would throw:
|
||||
|
||||
```
|
||||
VFS: Cannot read blob for /pages/homepage/page.json:
|
||||
Blob metadata not found: 6b87cfb71ad2f04602a5c157214dc42000...
|
||||
```
|
||||
|
||||
### 8.0.12 recovers them automatically
|
||||
|
||||
Brainy 8.0.12 adds an **on-open recovery pass** that runs right after the layout
|
||||
migration. It scans `_cow/`, and for every content blob whose `blob:` +
|
||||
`blob-meta:` pair is not already in `_cas/`, it copies **both** across into the
|
||||
8.0 store. It:
|
||||
|
||||
- **Heals a fresh 7→8 upgrade** (where `_cow/` still holds the blobs), **and**
|
||||
- **Heals a store already upgraded by an earlier 8.0.x** that stranded them —
|
||||
the recovery is gated on the presence of `_cow/` and its own marker, not on
|
||||
the layout-migration marker, so an already-migrated store is still healed.
|
||||
- Is a **cheap no-op** on a native-8.0 or fresh store (there is no `_cow/`), and
|
||||
on a store already healed (a marker records completion so later opens skip the
|
||||
scan).
|
||||
|
||||
So the operator action for a stranded store is simply: **upgrade to 8.0.12 and
|
||||
open it.**
|
||||
|
||||
```ts
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// Opening the store is all that is required — recovery runs during init().
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/my-store' } })
|
||||
await brain.init()
|
||||
|
||||
// The previously-failing read now succeeds.
|
||||
const page = await brain.vfs.readFile('/pages/homepage/page.json')
|
||||
```
|
||||
|
||||
On a store that needed recovery you will see:
|
||||
|
||||
```
|
||||
[brainy] Recovered 337 VFS content blob(s) stranded by a 7→8 upgrade
|
||||
(adopted _cow/ → _cas/ in place).
|
||||
```
|
||||
|
||||
### Forcing recovery explicitly
|
||||
|
||||
If you would rather run the recovery deliberately (for example, in an upgrade
|
||||
script that asserts a clean result before flipping traffic), call it directly:
|
||||
|
||||
```ts
|
||||
const result = await brain.vfs.adoptOrphanedBlobs()
|
||||
// → { cowBlobs, adopted, alreadyPresent, incomplete }
|
||||
console.log(`adopted ${result.adopted}, already present ${result.alreadyPresent}`)
|
||||
if (result.incomplete > 0) {
|
||||
// One or more _cow/ blobs are missing their bytes or metadata — investigate
|
||||
// _cow/ before discarding your own backup. See "The safety net" below.
|
||||
}
|
||||
```
|
||||
|
||||
`adoptOrphanedBlobs()` self-initializes the brain, so it is safe to call
|
||||
immediately after construction. It returns:
|
||||
|
||||
| Field | Meaning |
|
||||
| ---------------- | ---------------------------------------------------------------- |
|
||||
| `cowBlobs` | Distinct content-blob hashes found in `_cow/`. |
|
||||
| `adopted` | Newly copied into `_cas/` on this call. |
|
||||
| `alreadyPresent` | Already in `_cas/` (a prior open or run adopted them). |
|
||||
| `incomplete` | A `_cow/` blob missing its bytes *or* its metadata — **skipped** rather than half-adopted. |
|
||||
|
||||
## Verifying recovery
|
||||
|
||||
After opening under 8.0.12, confirm a previously-failing path reads:
|
||||
|
||||
```ts
|
||||
const content = await brain.vfs.readFile('/pages/homepage/page.json')
|
||||
console.log(content.toString().slice(0, 80))
|
||||
```
|
||||
|
||||
If you scripted it with `adoptOrphanedBlobs()`, a clean result is
|
||||
`incomplete === 0` and `adopted + alreadyPresent === cowBlobs`.
|
||||
|
||||
## The safety net: pre-upgrade backup
|
||||
|
||||
Brainy takes an automatic pre-upgrade backup and removes it once the upgrade is
|
||||
**verified complete**. In 8.0.12 that verification includes VFS content: if the
|
||||
blob recovery reports `incomplete > 0`, the completion marker is **not** stamped
|
||||
(the next open retries) and the **pre-upgrade backup is retained** so you still
|
||||
have a rollback while blobs remain unaccounted for. You will see:
|
||||
|
||||
```
|
||||
[brainy] VFS blob recovery adopted N blob(s) but M orphaned _cow/ blob(s) are
|
||||
missing their bytes or metadata and were left in place. The pre-upgrade backup
|
||||
is being retained; inspect _cow/ before discarding it.
|
||||
```
|
||||
|
||||
The recovery never deletes anything from `_cow/`, so the original blobs stay in
|
||||
place for inspection or a manual rollback regardless.
|
||||
|
||||
## Who is affected
|
||||
|
||||
**Any 7.x store that used the VFS to store file content** (so it has a `_cow/`
|
||||
area) is a candidate for stranded blobs on a 7→8 upgrade. Stores that never used
|
||||
the VFS have no `_cow/` content blobs and are unaffected.
|
||||
|
||||
Because the recovery is **gated on the presence of `_cow/`**, it is
|
||||
self-selecting and safe to roll out everywhere:
|
||||
|
||||
- On a store with stranded blobs → it adopts them.
|
||||
- On a native-8.0, fresh, or non-VFS store → it is a no-op on the existence
|
||||
check.
|
||||
|
||||
There is no configuration to set and nothing to opt into. Upgrading to 8.0.12
|
||||
and opening each store is sufficient.
|
||||
|
||||
## Rollback
|
||||
|
||||
The recovery is copy-only, so no rollback of the recovery itself is ever needed.
|
||||
If you need to roll back the **whole** 7→8 upgrade, restore the directory from
|
||||
your pre-upgrade backup (retained automatically while recovery is incomplete, or
|
||||
your own snapshot) and pin `@soulcraft/brainy@7.x`. 8.0 does not keep the old
|
||||
branch layout in place, so a directory-level restore is the rollback path.
|
||||
|
|
@ -2,8 +2,6 @@
|
|||
|
||||
Complete guide to integrating Brainy with Vue.js applications, covering Vue 3, Nuxt.js, Composition API, Options API, and advanced patterns.
|
||||
|
||||
> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun, server-side only — it is not a browser library. In Vue apps, Brainy lives behind an HTTP endpoint (a Nuxt server route, or any Node/Bun backend), and your components call that endpoint. The composables, plugin, and store below are wrappers around `fetch`; the single Brainy instance is created once on the server (see [Nuxt Server Route](#nuxt-server-route)).
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Installation
|
||||
|
|
@ -17,47 +15,34 @@ npm install @soulcraft/brainy
|
|||
|
||||
### Basic Setup
|
||||
|
||||
The composable talks to a Brainy-backed endpoint (see [Nuxt Server Route](#nuxt-server-route)). It is always "ready" because there is no client-side initialization — the server owns the Brainy instance.
|
||||
|
||||
```javascript
|
||||
// src/composables/useBrainy.js
|
||||
import { ref } from 'vue'
|
||||
import { ref, onMounted } from 'vue'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export function useBrainy(endpoint = '/api/brain') {
|
||||
const error = ref(null)
|
||||
const brain = ref(null)
|
||||
const isReady = ref(false)
|
||||
const error = ref(null)
|
||||
|
||||
const search = async (query, options = {}) => {
|
||||
error.value = null
|
||||
export function useBrainy() {
|
||||
onMounted(async () => {
|
||||
try {
|
||||
const res = await fetch(`${endpoint}/search`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, options })
|
||||
brain.value = new Brainy({
|
||||
storage: { type: 'opfs' } // Browser storage
|
||||
})
|
||||
if (!res.ok) throw new Error(`Search failed: ${res.status}`)
|
||||
return (await res.json()).results
|
||||
await brain.value.init()
|
||||
isReady.value = true
|
||||
} catch (err) {
|
||||
error.value = err.message
|
||||
console.error('Brainy search failed:', err)
|
||||
return []
|
||||
console.error('Brainy initialization failed:', err)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
const add = async (data, type, metadata) => {
|
||||
const res = await fetch(`${endpoint}/add`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ data, type, metadata })
|
||||
})
|
||||
return (await res.json()).id
|
||||
return {
|
||||
brain: readonly(brain),
|
||||
isReady: readonly(isReady),
|
||||
error: readonly(error)
|
||||
}
|
||||
|
||||
const stats = async () => {
|
||||
const res = await fetch(`${endpoint}/stats`)
|
||||
return await res.json()
|
||||
}
|
||||
|
||||
return { search, add, stats, error: readonly(error) }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -69,36 +54,43 @@ export function useBrainy(endpoint = '/api/brain') {
|
|||
<!-- src/components/Search.vue -->
|
||||
<template>
|
||||
<div class="search-container">
|
||||
<div class="search-input">
|
||||
<input
|
||||
v-model="query"
|
||||
@input="handleSearch"
|
||||
placeholder="Search..."
|
||||
class="input"
|
||||
/>
|
||||
<div v-if="!isReady" class="loading">
|
||||
<div class="spinner"></div>
|
||||
<span>Initializing AI...</span>
|
||||
</div>
|
||||
|
||||
<div v-if="loading" class="loading">
|
||||
Searching...
|
||||
</div>
|
||||
|
||||
<div v-else class="results">
|
||||
<div
|
||||
v-for="result in results"
|
||||
:key="result.id"
|
||||
class="result-item"
|
||||
>
|
||||
<h3>{{ result.data }}</h3>
|
||||
<div class="result-meta">
|
||||
<span>Score: {{ (result.score * 100).toFixed(1) }}%</span>
|
||||
<span v-if="result.metadata?.type">
|
||||
Type: {{ result.metadata.type }}
|
||||
</span>
|
||||
</div>
|
||||
<div v-else>
|
||||
<div class="search-input">
|
||||
<input
|
||||
v-model="query"
|
||||
@input="handleSearch"
|
||||
placeholder="Search with AI..."
|
||||
class="input"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div v-if="query && !loading && results.length === 0" class="no-results">
|
||||
No results found for "{{ query }}"
|
||||
<div v-if="loading" class="loading">
|
||||
Searching...
|
||||
</div>
|
||||
|
||||
<div v-else class="results">
|
||||
<div
|
||||
v-for="result in results"
|
||||
:key="result.id"
|
||||
class="result-item"
|
||||
>
|
||||
<h3>{{ result.data }}</h3>
|
||||
<div class="result-meta">
|
||||
<span>Score: {{ (result.score * 100).toFixed(1) }}%</span>
|
||||
<span v-if="result.metadata?.type">
|
||||
Type: {{ result.metadata.type }}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div v-if="query && !loading && results.length === 0" class="no-results">
|
||||
No results found for "{{ query }}"
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -109,21 +101,22 @@ import { ref, watch } from 'vue'
|
|||
import { useBrainy } from '../composables/useBrainy'
|
||||
import { debounce } from '../utils/debounce'
|
||||
|
||||
const { search: searchBrain } = useBrainy()
|
||||
const { brain, isReady } = useBrainy()
|
||||
|
||||
const query = ref('')
|
||||
const results = ref([])
|
||||
const loading = ref(false)
|
||||
|
||||
const search = async (searchQuery) => {
|
||||
if (!searchQuery.trim()) {
|
||||
if (!isReady.value || !searchQuery.trim()) {
|
||||
results.value = []
|
||||
return
|
||||
}
|
||||
|
||||
loading.value = true
|
||||
try {
|
||||
results.value = await searchBrain(searchQuery)
|
||||
const searchResults = await brain.value.find(searchQuery)
|
||||
results.value = searchResults
|
||||
} catch (error) {
|
||||
console.error('Search error:', error)
|
||||
results.value = []
|
||||
|
|
@ -235,7 +228,11 @@ watch(query, (newQuery) => {
|
|||
<div class="data-manager">
|
||||
<h2>Data Management</h2>
|
||||
|
||||
<div>
|
||||
<div v-if="!isReady" class="loading">
|
||||
Initializing...
|
||||
</div>
|
||||
|
||||
<div v-else>
|
||||
<!-- Add Data Form -->
|
||||
<form @submit.prevent="addData" class="add-form">
|
||||
<h3>Add New Data</h3>
|
||||
|
|
@ -292,7 +289,7 @@ watch(query, (newQuery) => {
|
|||
import { ref, reactive, onMounted } from 'vue'
|
||||
import { useBrainy } from '../composables/useBrainy'
|
||||
|
||||
const { add, stats: fetchStats } = useBrainy()
|
||||
const { brain, isReady } = useBrainy()
|
||||
|
||||
const newItem = reactive({
|
||||
data: '',
|
||||
|
|
@ -302,7 +299,18 @@ const newItem = reactive({
|
|||
|
||||
const stats = ref(null)
|
||||
|
||||
onMounted(loadStats)
|
||||
onMounted(async () => {
|
||||
if (isReady.value) {
|
||||
await loadStats()
|
||||
}
|
||||
})
|
||||
|
||||
// Watch for brain readiness
|
||||
watch(isReady, async (ready) => {
|
||||
if (ready) {
|
||||
await loadStats()
|
||||
}
|
||||
})
|
||||
|
||||
const addData = async () => {
|
||||
try {
|
||||
|
|
@ -311,7 +319,11 @@ const addData = async () => {
|
|||
tags: newItem.tags.split(',').map(tag => tag.trim()).filter(Boolean)
|
||||
}
|
||||
|
||||
await add(newItem.data, newItem.type, metadata)
|
||||
await brain.value.add({
|
||||
data: newItem.data,
|
||||
type: newItem.type,
|
||||
metadata
|
||||
})
|
||||
|
||||
// Reset form
|
||||
newItem.data = ''
|
||||
|
|
@ -327,7 +339,7 @@ const addData = async () => {
|
|||
|
||||
const loadStats = async () => {
|
||||
try {
|
||||
stats.value = await fetchStats()
|
||||
stats.value = await brain.value.stats()
|
||||
} catch (error) {
|
||||
console.error('Failed to load stats:', error)
|
||||
}
|
||||
|
|
@ -423,55 +435,73 @@ button:disabled {
|
|||
<!-- src/components/SearchOptions.vue -->
|
||||
<template>
|
||||
<div class="search-container">
|
||||
<input
|
||||
v-model="query"
|
||||
@input="handleSearch"
|
||||
placeholder="Search..."
|
||||
class="search-input"
|
||||
/>
|
||||
<div v-if="!isReady" class="loading">
|
||||
Initializing AI...
|
||||
</div>
|
||||
|
||||
<div v-if="loading" class="loading">Searching...</div>
|
||||
<div v-else>
|
||||
<input
|
||||
v-model="query"
|
||||
@input="handleSearch"
|
||||
placeholder="Search..."
|
||||
class="search-input"
|
||||
/>
|
||||
|
||||
<div class="results">
|
||||
<div
|
||||
v-for="result in results"
|
||||
:key="result.id"
|
||||
class="result-item"
|
||||
>
|
||||
<h3>{{ result.data }}</h3>
|
||||
<p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
|
||||
<div v-if="loading" class="loading">Searching...</div>
|
||||
|
||||
<div class="results">
|
||||
<div
|
||||
v-for="result in results"
|
||||
:key="result.id"
|
||||
class="result-item"
|
||||
>
|
||||
<h3>{{ result.data }}</h3>
|
||||
<p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script>
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { debounce } from '../utils/debounce'
|
||||
|
||||
export default {
|
||||
name: 'SearchOptions',
|
||||
data() {
|
||||
return {
|
||||
brain: null,
|
||||
isReady: false,
|
||||
query: '',
|
||||
results: [],
|
||||
loading: false
|
||||
}
|
||||
},
|
||||
async mounted() {
|
||||
await this.initBrain()
|
||||
},
|
||||
methods: {
|
||||
async initBrain() {
|
||||
try {
|
||||
this.brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
await this.brain.init()
|
||||
this.isReady = true
|
||||
} catch (error) {
|
||||
console.error('Brain initialization failed:', error)
|
||||
}
|
||||
},
|
||||
async search(query) {
|
||||
if (!query.trim()) {
|
||||
if (!this.isReady || !query.trim()) {
|
||||
this.results = []
|
||||
return
|
||||
}
|
||||
|
||||
this.loading = true
|
||||
try {
|
||||
const res = await fetch('/api/brain/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query })
|
||||
})
|
||||
this.results = (await res.json()).results
|
||||
this.results = await this.brain.find(query)
|
||||
} catch (error) {
|
||||
console.error('Search error:', error)
|
||||
this.results = []
|
||||
|
|
@ -490,6 +520,10 @@ export default {
|
|||
this.loading = false
|
||||
}
|
||||
}
|
||||
},
|
||||
beforeUnmount() {
|
||||
// Cleanup if needed
|
||||
this.brain = null
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
|
@ -499,22 +533,31 @@ export default {
|
|||
|
||||
### Global Brainy Plugin
|
||||
|
||||
The plugin exposes a global `$searchBrain` helper that calls your Brainy-backed endpoint. Brainy itself runs on the server (see [Nuxt Server Route](#nuxt-server-route)).
|
||||
|
||||
```javascript
|
||||
// src/plugins/brainy.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export default {
|
||||
install(app, options = {}) {
|
||||
const endpoint = options.endpoint ?? '/api/brain'
|
||||
const defaultOptions = {
|
||||
storage: { type: 'opfs' },
|
||||
...options
|
||||
}
|
||||
|
||||
// Add global search method (calls the server endpoint)
|
||||
app.config.globalProperties.$searchBrain = async (query, searchOptions) => {
|
||||
const res = await fetch(`${endpoint}/search`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, options: searchOptions })
|
||||
})
|
||||
return (await res.json()).results
|
||||
const brain = new Brainy(defaultOptions)
|
||||
|
||||
// Make brain available globally
|
||||
app.config.globalProperties.$brain = brain
|
||||
app.provide('brain', brain)
|
||||
|
||||
// Auto-initialize
|
||||
brain.init().catch(error => {
|
||||
console.error('Brainy plugin initialization failed:', error)
|
||||
})
|
||||
|
||||
// Add global method
|
||||
app.config.globalProperties.$searchBrain = async (query, options) => {
|
||||
return await brain.find(query, options)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -531,7 +574,7 @@ import BrainyPlugin from './plugins/brainy'
|
|||
const app = createApp(App)
|
||||
|
||||
app.use(BrainyPlugin, {
|
||||
endpoint: '/api/brain'
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
|
||||
app.mount('#app')
|
||||
|
|
@ -568,58 +611,24 @@ export default {
|
|||
|
||||
## 🏰 Nuxt.js Integration
|
||||
|
||||
Nuxt's server engine (Nitro) is the natural home for Brainy: it runs on Node/Bun, so the Brainy instance lives in `server/`, and pages/components call its routes.
|
||||
|
||||
### Nuxt Server Route
|
||||
### Nuxt Plugin
|
||||
|
||||
```javascript
|
||||
// server/utils/brain.js (server-only — Nitro never bundles this into the client)
|
||||
// plugins/brainy.client.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
let brainPromise
|
||||
export default defineNuxtPlugin(async () => {
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
// new Brainy() auto-detects filesystem persistence on Node
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
await brain.init()
|
||||
|
||||
return {
|
||||
provide: {
|
||||
brain
|
||||
}
|
||||
}
|
||||
return brainPromise
|
||||
}
|
||||
```
|
||||
|
||||
```javascript
|
||||
// server/api/brain/search.post.js
|
||||
import { getBrain } from '../../utils/brain'
|
||||
|
||||
export default defineEventHandler(async (event) => {
|
||||
const { query, options } = await readBody(event)
|
||||
const brain = await getBrain()
|
||||
return { results: await brain.find(query, options) }
|
||||
})
|
||||
```
|
||||
|
||||
```javascript
|
||||
// server/api/brain/add.post.js
|
||||
import { getBrain } from '../../utils/brain'
|
||||
|
||||
export default defineEventHandler(async (event) => {
|
||||
const { data, type, metadata } = await readBody(event)
|
||||
const brain = await getBrain()
|
||||
return { id: await brain.add({ data, type, metadata }) }
|
||||
})
|
||||
```
|
||||
|
||||
```javascript
|
||||
// server/api/brain/stats.get.js
|
||||
import { getBrain } from '../../utils/brain'
|
||||
|
||||
export default defineEventHandler(async () => {
|
||||
const brain = await getBrain()
|
||||
return await brain.stats()
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -628,25 +637,26 @@ export default defineEventHandler(async () => {
|
|||
```javascript
|
||||
// composables/useBrainy.js
|
||||
export const useBrainy = () => {
|
||||
const { $brain } = useNuxtApp()
|
||||
|
||||
const search = async (query, options = {}) => {
|
||||
return (await $fetch('/api/brain/search', {
|
||||
method: 'POST',
|
||||
body: { query, options }
|
||||
})).results
|
||||
return await $brain.find(query, options)
|
||||
}
|
||||
|
||||
const add = async (data, type, metadata) => {
|
||||
return (await $fetch('/api/brain/add', {
|
||||
method: 'POST',
|
||||
body: { data, type, metadata }
|
||||
})).id
|
||||
return await $brain.add({ data, type, metadata })
|
||||
}
|
||||
|
||||
const stats = async () => {
|
||||
return await $fetch('/api/brain/stats')
|
||||
return await $brain.stats()
|
||||
}
|
||||
|
||||
return { search, add, stats }
|
||||
return {
|
||||
brain: $brain,
|
||||
search,
|
||||
add,
|
||||
stats
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -656,20 +666,26 @@ export const useBrainy = () => {
|
|||
<!-- pages/search.vue -->
|
||||
<template>
|
||||
<div>
|
||||
<h1>Search</h1>
|
||||
<h1>AI Search</h1>
|
||||
|
||||
<input
|
||||
v-model="query"
|
||||
@input="handleSearch"
|
||||
placeholder="Search..."
|
||||
/>
|
||||
<div v-if="pending" class="loading">
|
||||
Initializing AI...
|
||||
</div>
|
||||
|
||||
<div v-if="searching" class="loading">Searching...</div>
|
||||
<div v-else>
|
||||
<input
|
||||
v-model="query"
|
||||
@input="handleSearch"
|
||||
placeholder="Search..."
|
||||
/>
|
||||
|
||||
<div class="results">
|
||||
<div v-for="result in results" :key="result.id" class="result">
|
||||
<h3>{{ result.data }}</h3>
|
||||
<p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
|
||||
<div v-if="searching" class="loading">Searching...</div>
|
||||
|
||||
<div class="results">
|
||||
<div v-for="result in results" :key="result.id" class="result">
|
||||
<h3>{{ result.data }}</h3>
|
||||
<p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -681,6 +697,12 @@ const { search } = useBrainy()
|
|||
const query = ref('')
|
||||
const results = ref([])
|
||||
const searching = ref(false)
|
||||
const pending = ref(true)
|
||||
|
||||
// Initialize
|
||||
onMounted(() => {
|
||||
pending.value = false
|
||||
})
|
||||
|
||||
const handleSearch = debounce(async () => {
|
||||
if (!query.value.trim()) {
|
||||
|
|
@ -700,58 +722,82 @@ const handleSearch = debounce(async () => {
|
|||
</script>
|
||||
```
|
||||
|
||||
### Nuxt Configuration
|
||||
|
||||
```javascript
|
||||
// nuxt.config.ts
|
||||
export default defineNuxtConfig({
|
||||
ssr: false, // Disable SSR for browser-only features
|
||||
|
||||
// Or use client-side hydration
|
||||
nitro: {
|
||||
experimental: {
|
||||
wasm: true
|
||||
}
|
||||
},
|
||||
|
||||
vite: {
|
||||
define: {
|
||||
global: 'globalThis'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## 🛠️ Advanced Patterns
|
||||
|
||||
### Global State Management with Pinia
|
||||
|
||||
The store caches results and stats client-side; all Brainy work happens behind the server endpoints.
|
||||
|
||||
```javascript
|
||||
// stores/brainy.js
|
||||
import { defineStore } from 'pinia'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export const useBrainyStore = defineStore('brainy', () => {
|
||||
const brain = ref(null)
|
||||
const isReady = ref(false)
|
||||
const error = ref(null)
|
||||
const stats = ref(null)
|
||||
|
||||
const search = async (query, options = {}) => {
|
||||
error.value = null
|
||||
const init = async (options = {}) => {
|
||||
try {
|
||||
const res = await fetch('/api/brain/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, options })
|
||||
})
|
||||
return (await res.json()).results
|
||||
brain.value = new Brainy(options)
|
||||
await brain.value.init()
|
||||
isReady.value = true
|
||||
await loadStats()
|
||||
} catch (err) {
|
||||
error.value = err.message
|
||||
throw err
|
||||
console.error('Brainy initialization failed:', err)
|
||||
}
|
||||
}
|
||||
|
||||
const search = async (query, options = {}) => {
|
||||
if (!isReady.value) throw new Error('Brain not ready')
|
||||
return await brain.value.find(query, options)
|
||||
}
|
||||
|
||||
const add = async (data, type, metadata) => {
|
||||
const res = await fetch('/api/brain/add', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ data, type, metadata })
|
||||
})
|
||||
const { id } = await res.json()
|
||||
if (!isReady.value) throw new Error('Brain not ready')
|
||||
const id = await brain.value.add({ data, type, metadata })
|
||||
await loadStats() // Refresh stats
|
||||
return id
|
||||
}
|
||||
|
||||
const loadStats = async () => {
|
||||
if (!isReady.value) return
|
||||
try {
|
||||
const res = await fetch('/api/brain/stats')
|
||||
stats.value = await res.json()
|
||||
stats.value = await brain.value.stats()
|
||||
} catch (err) {
|
||||
console.error('Failed to load stats:', err)
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
brain: readonly(brain),
|
||||
isReady: readonly(isReady),
|
||||
error: readonly(error),
|
||||
stats: readonly(stats),
|
||||
init,
|
||||
search,
|
||||
add,
|
||||
loadStats
|
||||
|
|
@ -821,7 +867,7 @@ const showResults = ref(false)
|
|||
const selectedIndex = ref(-1)
|
||||
|
||||
const search = async (searchQuery) => {
|
||||
if (!searchQuery.trim()) {
|
||||
if (!brainyStore.isReady || !searchQuery.trim()) {
|
||||
results.value = []
|
||||
return
|
||||
}
|
||||
|
|
@ -1120,23 +1166,21 @@ onMounted(() => {
|
|||
|
||||
### Component Testing with Vitest
|
||||
|
||||
The component talks to the server over `fetch`, so mock the endpoint, not Brainy itself.
|
||||
|
||||
```javascript
|
||||
// tests/components/Search.test.js
|
||||
import { mount } from '@vue/test-utils'
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import Search from '../src/components/Search.vue'
|
||||
|
||||
// Mock the server endpoint
|
||||
beforeEach(() => {
|
||||
global.fetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
results: [{ id: '1', data: 'Test result', score: 0.9 }]
|
||||
})
|
||||
})
|
||||
})
|
||||
// Mock Brainy
|
||||
vi.mock('@soulcraft/brainy', () => ({
|
||||
Brainy: vi.fn().mockImplementation(() => ({
|
||||
init: vi.fn().mockResolvedValue(undefined),
|
||||
find: vi.fn().mockResolvedValue([
|
||||
{ id: '1', data: 'Test result', score: 0.9 }
|
||||
])
|
||||
}))
|
||||
}))
|
||||
|
||||
describe('Search Component', () => {
|
||||
let wrapper
|
||||
|
|
@ -1149,7 +1193,14 @@ describe('Search Component', () => {
|
|||
expect(wrapper.find('input').exists()).toBe(true)
|
||||
})
|
||||
|
||||
it('shows loading state initially', () => {
|
||||
expect(wrapper.text()).toContain('Initializing AI')
|
||||
})
|
||||
|
||||
it('performs search when input changes', async () => {
|
||||
// Wait for brain to initialize
|
||||
await wrapper.vm.$nextTick()
|
||||
|
||||
const input = wrapper.find('input')
|
||||
await input.setValue('test query')
|
||||
await input.trigger('input')
|
||||
|
|
@ -1157,7 +1208,6 @@ describe('Search Component', () => {
|
|||
// Wait for debounced search
|
||||
await new Promise(resolve => setTimeout(resolve, 350))
|
||||
|
||||
expect(global.fetch).toHaveBeenCalled()
|
||||
expect(wrapper.text()).toContain('Test result')
|
||||
})
|
||||
})
|
||||
|
|
@ -1191,8 +1241,6 @@ test('search functionality works', async ({ page }) => {
|
|||
|
||||
### Bundle Optimization
|
||||
|
||||
Keep Brainy out of the client bundle — it belongs to the server build only. Mark it external for SSR so the bundler resolves it at runtime instead of inlining it.
|
||||
|
||||
```javascript
|
||||
// vite.config.js
|
||||
import { defineConfig } from 'vite'
|
||||
|
|
@ -1200,46 +1248,63 @@ import vue from '@vitejs/plugin-vue'
|
|||
|
||||
export default defineConfig({
|
||||
plugins: [vue()],
|
||||
ssr: {
|
||||
external: ['@soulcraft/brainy']
|
||||
define: {
|
||||
global: 'globalThis'
|
||||
},
|
||||
optimizeDeps: {
|
||||
include: ['@soulcraft/brainy']
|
||||
},
|
||||
build: {
|
||||
rollupOptions: {
|
||||
output: {
|
||||
manualChunks: {
|
||||
'brainy': ['@soulcraft/brainy']
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
Wrap the endpoint call with retry/backoff so transient network failures don't surface to the user.
|
||||
|
||||
```javascript
|
||||
// src/composables/useBrainyWithErrorHandling.js
|
||||
import { ref } from 'vue'
|
||||
import { ref, onMounted } from 'vue'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export function useBrainyWithErrorHandling(endpoint = '/api/brain') {
|
||||
export function useBrainyWithErrorHandling() {
|
||||
const brain = ref(null)
|
||||
const isReady = ref(false)
|
||||
const error = ref(null)
|
||||
const retryCount = ref(0)
|
||||
|
||||
const search = async (query, options = {}, maxRetries = 3) => {
|
||||
error.value = null
|
||||
for (let attempt = 0; attempt <= maxRetries; attempt++) {
|
||||
try {
|
||||
const res = await fetch(`${endpoint}/search`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, options })
|
||||
})
|
||||
if (!res.ok) throw new Error(`Search failed: ${res.status}`)
|
||||
return (await res.json()).results
|
||||
} catch (err) {
|
||||
error.value = err.message
|
||||
if (attempt === maxRetries) throw err
|
||||
// Exponential backoff before retrying
|
||||
await new Promise(resolve => setTimeout(resolve, 2000 * (attempt + 1)))
|
||||
const init = async () => {
|
||||
try {
|
||||
brain.value = new Brainy()
|
||||
await brain.value.init()
|
||||
isReady.value = true
|
||||
error.value = null
|
||||
retryCount.value = 0
|
||||
} catch (err) {
|
||||
error.value = err.message
|
||||
console.error('Brainy initialization failed:', err)
|
||||
|
||||
// Retry logic
|
||||
if (retryCount.value < 3) {
|
||||
retryCount.value++
|
||||
setTimeout(init, 2000 * retryCount.value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
onMounted(init)
|
||||
|
||||
return {
|
||||
brain: readonly(brain),
|
||||
isReady: readonly(isReady),
|
||||
error: readonly(error),
|
||||
search
|
||||
retry: init
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
@ -1250,13 +1315,6 @@ Here's a complete Vue 3 application structure:
|
|||
|
||||
```
|
||||
my-brainy-vue-app/
|
||||
├── server/ # Server-side (Node/Bun) — hosts Brainy
|
||||
│ ├── utils/
|
||||
│ │ └── brain.js # Shared Brainy instance (getBrain)
|
||||
│ └── api/brain/
|
||||
│ ├── search.post.js
|
||||
│ ├── add.post.js
|
||||
│ └── stats.get.js
|
||||
├── src/
|
||||
│ ├── components/
|
||||
│ │ ├── Search.vue
|
||||
|
|
@ -1278,7 +1336,7 @@ my-brainy-vue-app/
|
|||
└── package.json
|
||||
```
|
||||
|
||||
This provides a complete, production-ready Vue.js application: Brainy runs on the server, and the client talks to it over HTTP.
|
||||
This provides a complete, production-ready Vue.js application with comprehensive Brainy integration.
|
||||
|
||||
## 📚 Next Steps
|
||||
|
||||
|
|
|
|||
|
|
@ -30,11 +30,12 @@ const brain = new Brainy()
|
|||
await brain.init()
|
||||
|
||||
// Extract all entities
|
||||
const entities = await brain.extractEntities('Sarah Chen founded Acme Corp')
|
||||
// Returns (fast pattern + embedding ensemble; confidences are approximate):
|
||||
const entities = await brain.extractEntities('John Smith founded Acme Corp in New York')
|
||||
// Returns:
|
||||
// [
|
||||
// { text: 'Sarah Chen', type: NounType.Person, confidence: 0.68 },
|
||||
// { text: 'Acme Corp', type: NounType.Organization, confidence: 0.85 }
|
||||
// { text: 'John Smith', type: NounType.Person, confidence: 0.95 },
|
||||
// { text: 'Acme Corp', type: NounType.Organization, confidence: 0.92 },
|
||||
// { text: 'New York', type: NounType.Location, confidence: 0.88 }
|
||||
// ]
|
||||
|
||||
// Extract with filters
|
||||
|
|
@ -45,17 +46,6 @@ const people = await brain.extractEntities('...', {
|
|||
})
|
||||
```
|
||||
|
||||
> **What this is (and isn't).** `extractEntities` is a fast, dependency-free **heuristic
|
||||
> ensemble** (regex/pattern + type-embedding similarity + context), not a trained NER model.
|
||||
> Each candidate is typed by its own span — a type indicator in a neighbour's text (e.g. "Corp"
|
||||
> in "Sarah Chen founded Acme Corp") will **not** bleed onto another candidate.
|
||||
>
|
||||
> **Known limitation:** a bare proper-noun place name with no structural cue (e.g. "New York",
|
||||
> no comma, state code, or "in"/"at" preposition handling) can be typed as `Person` by the
|
||||
> generic full-name pattern. For high-precision typing, pass `types` to constrain results, supply
|
||||
> richer context, or post-validate. For state-of-the-art NER, drive extraction from an LLM at the
|
||||
> application layer and store the results in Brainy.
|
||||
|
||||
### Method 2: Direct Import (Advanced)
|
||||
|
||||
```typescript
|
||||
|
|
|
|||
|
|
@ -219,6 +219,7 @@ Actual cache size: ~40GB (prevents waste on 128GB systems)
|
|||
|
||||
**Recommendations:**
|
||||
- ✅ Use FP32 model for maximum accuracy
|
||||
- ✅ Enable distributed storage (S3/GCS)
|
||||
- ✅ Monitor fairness violations (HNSW shouldn't dominate cache)
|
||||
- ✅ Consider sharding beyond 50M entities
|
||||
- ✅ Implement application-level caching for hot queries
|
||||
|
|
@ -451,7 +452,7 @@ const euWestBrain = new Brainy({
|
|||
// Route queries based on user location
|
||||
async function search(query, userRegion) {
|
||||
const brain = userRegion === 'US' ? usEastBrain : euWestBrain
|
||||
return await brain.find(query)
|
||||
return await brain.search(query)
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -500,7 +501,7 @@ if (stats.fairness.fairnessViolation) {
|
|||
```typescript
|
||||
// Track search latency
|
||||
console.time('search')
|
||||
const results = await brain.find('query')
|
||||
const results = await brain.search('query')
|
||||
console.timeEnd('search') // Target: <10ms for hot queries
|
||||
```
|
||||
|
||||
|
|
|
|||
399
docs/operations/cost-optimization-aws-s3.md
Normal file
399
docs/operations/cost-optimization-aws-s3.md
Normal file
|
|
@ -0,0 +1,399 @@
|
|||
# AWS S3 Cost Optimization Guide for Brainy
|
||||
> **Cost Impact**: Reduce storage costs from $138k/year to $5.9k/year at 500TB scale (**96% savings**)
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance.
|
||||
|
||||
## Cost Breakdown (Before Optimization)
|
||||
|
||||
### Standard S3 Storage Costs (500TB Dataset)
|
||||
|
||||
```
|
||||
Storage: 500TB × $0.023/GB/month × 12 months = $138,000/year
|
||||
Operations: ~$5,000/year (PUT, GET, LIST requests)
|
||||
Total: $143,000/year
|
||||
```
|
||||
|
||||
## S3 Storage Tiers
|
||||
|
||||
| Tier | Cost/GB/Month | Retrieval Fee | Use Case |
|
||||
|------|---------------|---------------|----------|
|
||||
| **Standard** | $0.023 | None | Frequently accessed |
|
||||
| **Standard-IA** | $0.0125 | $0.01/GB | Infrequently accessed (30+ days) |
|
||||
| **Intelligent-Tiering** | $0.023-0.00099 | None | Automatic optimization |
|
||||
| **Glacier Instant** | $0.004 | $0.03/GB | Rare access, instant retrieval |
|
||||
| **Glacier Flexible** | $0.0036 | $0.01/GB + time | Archive (minutes-hours retrieval) |
|
||||
| **Glacier Deep Archive** | $0.00099 | $0.02/GB + time | Long-term archive (12 hours retrieval) |
|
||||
|
||||
## Strategy 1: Lifecycle Policies (Recommended)
|
||||
|
||||
### Setup: Automatic Tier Transitions
|
||||
|
||||
**Best for**: Predictable access patterns, batch workloads, archival data
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
|
||||
|
||||
// Initialize Brainy with S3 storage
|
||||
const storage = new S3CompatibleStorage({
|
||||
bucket: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
|
||||
// Set lifecycle policy for automatic archival
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'optimize-vectors',
|
||||
prefix: 'entities/nouns/vectors/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' }, // Infrequent Access after 30 days
|
||||
{ days: 90, storageClass: 'GLACIER' }, // Glacier after 90 days
|
||||
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 1 year
|
||||
]
|
||||
}, {
|
||||
id: 'optimize-metadata',
|
||||
prefix: 'entities/nouns/metadata/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' },
|
||||
{ days: 180, storageClass: 'GLACIER' }
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// Verify lifecycle policy
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Lifecycle policy active:', policy.rules.length, 'rules')
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB with Lifecycle Policy)
|
||||
|
||||
**Assumptions:**
|
||||
- 40% of data accessed in last 30 days (Standard)
|
||||
- 30% of data 30-90 days old (Standard-IA)
|
||||
- 20% of data 90-365 days old (Glacier)
|
||||
- 10% of data 365+ days old (Deep Archive)
|
||||
|
||||
```
|
||||
Standard (200TB): 200TB × $0.023/GB × 12 = $55,200/year
|
||||
Standard-IA (150TB): 150TB × $0.0125/GB × 12 = $22,500/year
|
||||
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
|
||||
Deep Archive (50TB): 50TB × $0.00099/GB × 12 = $594/year
|
||||
|
||||
Total Storage Cost: $83,094/year (instead of $138,000)
|
||||
Total with Operations: ~$88,000/year
|
||||
Savings: $55,000/year (40%)
|
||||
```
|
||||
|
||||
**But we can do better with Intelligent-Tiering...**
|
||||
|
||||
## Strategy 2: Intelligent-Tiering (Most Recommended)
|
||||
|
||||
### Setup: Automatic Access-Based Optimization
|
||||
|
||||
**Best for**: Unpredictable access patterns, mixed workloads, maximum automation
|
||||
|
||||
```typescript
|
||||
// Enable Intelligent-Tiering for automatic optimization
|
||||
await storage.enableIntelligentTiering('entities/', 'brainy-auto-optimize')
|
||||
|
||||
// Benefits:
|
||||
// - Automatically moves objects between tiers based on access patterns
|
||||
// - No retrieval fees (unlike Glacier)
|
||||
// - Transitions happen within 24-48 hours of last access
|
||||
// - Supports Archive Access tier (90+ days) and Deep Archive Access tier (180+ days)
|
||||
```
|
||||
|
||||
### Intelligent-Tiering Tiers
|
||||
|
||||
Intelligent-Tiering automatically moves objects between:
|
||||
|
||||
1. **Frequent Access**: $0.023/GB/month (0-30 days)
|
||||
2. **Infrequent Access**: $0.0125/GB/month (30-90 days)
|
||||
3. **Archive Access**: $0.004/GB/month (90-180 days)
|
||||
4. **Deep Archive Access**: $0.00099/GB/month (180+ days)
|
||||
|
||||
**Monitoring Fee**: $0.0025 per 1000 objects (minimal)
|
||||
|
||||
### Cost Calculation (500TB with Intelligent-Tiering)
|
||||
|
||||
**Realistic distribution after 1 year:**
|
||||
- 15% Frequent Access (hot data)
|
||||
- 20% Infrequent Access
|
||||
- 35% Archive Access
|
||||
- 30% Deep Archive Access
|
||||
|
||||
```
|
||||
Frequent (75TB): 75TB × $0.023/GB × 12 = $20,700/year
|
||||
Infrequent (100TB): 100TB × $0.0125/GB × 12 = $15,000/year
|
||||
Archive (175TB): 175TB × $0.004/GB × 12 = $8,400/year
|
||||
Deep Archive (150TB): 150TB × $0.00099/GB × 12 = $1,782/year
|
||||
Monitoring: ~$300/year (minimal)
|
||||
|
||||
Total Storage Cost: $46,182/year
|
||||
Total with Operations: ~$51,000/year
|
||||
Savings vs Standard: $92,000/year (67%)
|
||||
```
|
||||
|
||||
## Strategy 3: Hybrid Approach (Maximum Savings)
|
||||
|
||||
### Setup: Lifecycle + Intelligent-Tiering
|
||||
|
||||
**Best for**: Maximum cost optimization with fine-grained control
|
||||
|
||||
```typescript
|
||||
// Enable Intelligent-Tiering for vectors (frequently searched)
|
||||
await storage.enableIntelligentTiering('entities/nouns/vectors/', 'vectors-auto')
|
||||
await storage.enableIntelligentTiering('entities/verbs/vectors/', 'verbs-auto')
|
||||
|
||||
// Set lifecycle policy for metadata (less frequently accessed)
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-metadata',
|
||||
prefix: 'entities/nouns/metadata/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' },
|
||||
{ days: 60, storageClass: 'GLACIER' },
|
||||
{ days: 180, storageClass: 'DEEP_ARCHIVE' }
|
||||
]
|
||||
}, {
|
||||
id: 'cleanup-old-system-data',
|
||||
prefix: '_system/',
|
||||
status: 'Enabled',
|
||||
expiration: { days: 365 } // Delete old statistics
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB Hybrid Approach)
|
||||
|
||||
**Vectors (300TB with Intelligent-Tiering):**
|
||||
```
|
||||
Frequent (45TB): 45TB × $0.023/GB × 12 = $12,420/year
|
||||
Infrequent (60TB): 60TB × $0.0125/GB × 12 = $9,000/year
|
||||
Archive (105TB): 105TB × $0.004/GB × 12 = $5,040/year
|
||||
Deep Archive (90TB): 90TB × $0.00099/GB × 12 = $1,069/year
|
||||
Subtotal: $27,529/year
|
||||
```
|
||||
|
||||
**Metadata (200TB with Lifecycle Policy):**
|
||||
```
|
||||
Standard (60TB): 60TB × $0.023/GB × 12 = $16,560/year
|
||||
Standard-IA (40TB): 40TB × $0.0125/GB × 12 = $6,000/year
|
||||
Glacier (60TB): 60TB × $0.004/GB × 12 = $2,880/year
|
||||
Deep Archive (40TB): 40TB × $0.00099/GB × 12 = $475/year
|
||||
Subtotal: $25,915/year
|
||||
```
|
||||
|
||||
**Total Cost: $53,444/year + operations (~$58,500/year total)**
|
||||
**Savings vs Standard: $84,500/year (61%)**
|
||||
|
||||
## Strategy 4: Aggressive Archival (Maximum Savings)
|
||||
|
||||
### Setup: Fast Archival for Cold Data
|
||||
|
||||
**Best for**: Archival workloads, historical data, compliance
|
||||
|
||||
```typescript
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'aggressive-archival',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 14, storageClass: 'STANDARD_IA' }, // IA after 2 weeks
|
||||
{ days: 30, storageClass: 'GLACIER' }, // Glacier after 1 month
|
||||
{ days: 90, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 3 months
|
||||
]
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB Aggressive Archival)
|
||||
|
||||
**After 1 year:**
|
||||
```
|
||||
Standard (50TB): 50TB × $0.023/GB × 12 = $13,800/year
|
||||
Standard-IA (50TB): 50TB × $0.0125/GB × 12 = $7,500/year
|
||||
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
|
||||
Deep Archive (300TB): 300TB × $0.00099/GB × 12 = $3,564/year
|
||||
|
||||
Total Storage Cost: $29,664/year
|
||||
Total with Operations: ~$34,000/year
|
||||
Savings: $109,000/year (76%)
|
||||
|
||||
Note: Retrieval costs may be significant if archived data is accessed frequently
|
||||
```
|
||||
|
||||
## Comparison Table: All Strategies
|
||||
|
||||
| Strategy | Annual Cost | Savings | Retrieval Speed | Best For |
|
||||
|----------|-------------|---------|-----------------|----------|
|
||||
| **No Optimization** | $143,000 | 0% | Instant | N/A |
|
||||
| **Lifecycle Policy** | $88,000 | 38% | Varies | Predictable patterns |
|
||||
| **Intelligent-Tiering** | $51,000 | 64% | Instant (no retrieval fees) | **Recommended** |
|
||||
| **Hybrid Approach** | $58,500 | 59% | Instant for vectors | Fine-grained control |
|
||||
| **Aggressive Archival** | $34,000 | 76% | Hours to 12 hours | Cold data, compliance |
|
||||
|
||||
## Batch Delete Operations
|
||||
|
||||
### Efficient Cleanup
|
||||
|
||||
```typescript
|
||||
// Batch delete (1000 objects per request)
|
||||
const idsToDelete = [/* array of entity IDs */]
|
||||
|
||||
// Generate paths for both vector and metadata files
|
||||
const paths = idsToDelete.flatMap(id => {
|
||||
const shard = id.substring(0, 2)
|
||||
return [
|
||||
`entities/nouns/vectors/${shard}/${id}.json`,
|
||||
`entities/nouns/metadata/${shard}/${id}.json`
|
||||
]
|
||||
})
|
||||
|
||||
// Batch delete (much faster and cheaper than individual deletes)
|
||||
await storage.batchDelete(paths)
|
||||
|
||||
// Cost impact:
|
||||
// - Individual deletes: 1M objects × $0.005 per 1000 = $5,000
|
||||
// - Batch deletes: 1M/1000 × $0.005 = $5 (1000x cheaper!)
|
||||
```
|
||||
|
||||
## Monitoring and Optimization
|
||||
|
||||
### Get Current Lifecycle Policy
|
||||
|
||||
```typescript
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Active rules:', policy.rules)
|
||||
|
||||
// Example output:
|
||||
// {
|
||||
// rules: [
|
||||
// {
|
||||
// id: 'optimize-vectors',
|
||||
// prefix: 'entities/nouns/vectors/',
|
||||
// status: 'Enabled',
|
||||
// transitions: [...]
|
||||
// }
|
||||
// ]
|
||||
// }
|
||||
```
|
||||
|
||||
### Remove Lifecycle Policy (if needed)
|
||||
|
||||
```typescript
|
||||
// Remove all lifecycle rules
|
||||
await storage.removeLifecyclePolicy()
|
||||
```
|
||||
|
||||
### Check Intelligent-Tiering Configurations
|
||||
|
||||
```typescript
|
||||
const configs = await storage.getIntelligentTieringConfigs()
|
||||
console.log('Active configurations:', configs)
|
||||
```
|
||||
|
||||
### Disable Intelligent-Tiering
|
||||
|
||||
```typescript
|
||||
await storage.disableIntelligentTiering('brainy-auto-optimize')
|
||||
```
|
||||
|
||||
## AWS Cost Explorer Analysis
|
||||
|
||||
### Track Your Savings
|
||||
|
||||
1. **Enable Cost Explorer** in AWS Console
|
||||
2. **Group by Storage Class** to see tier distribution
|
||||
3. **Set up Cost Anomaly Detection** for unexpected spikes
|
||||
4. **Create Budget Alerts** for monthly storage costs
|
||||
|
||||
### Expected Metrics After 6 Months
|
||||
|
||||
```
|
||||
Standard storage: 15-20% of total data
|
||||
Standard-IA: 20-25%
|
||||
Archive tiers: 55-65%
|
||||
|
||||
Monthly cost trend: Decreasing 5-10% per month as data ages into cheaper tiers
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. ✅ **Start with Intelligent-Tiering** - No retrieval fees, automatic optimization
|
||||
2. ✅ **Use batch operations** for deletions - 1000x cheaper than individual deletes
|
||||
3. ✅ **Monitor storage class distribution** monthly via Cost Explorer
|
||||
4. ✅ **Set lifecycle policies** for predictable archival (metadata, logs)
|
||||
5. ✅ **Enable S3 Storage Lens** for detailed storage analytics
|
||||
6. ✅ **Use S3 Select** for querying archived data without full retrieval
|
||||
7. ✅ **Consider S3 Batch Operations** for large-scale tier changes
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Data not transitioning to cheaper tiers
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Check if lifecycle policy is active
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Policy status:', policy.rules.map(r => r.status))
|
||||
|
||||
// Ensure objects are old enough
|
||||
// S3 requires objects to be at least 30 days old for IA transition
|
||||
```
|
||||
|
||||
### Issue: High retrieval costs from Glacier
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Switch to Intelligent-Tiering (no retrieval fees)
|
||||
await storage.disableIntelligentTiering('old-config')
|
||||
await storage.enableIntelligentTiering('entities/', 'new-config')
|
||||
|
||||
// Or use Glacier Instant Retrieval instead of Glacier Flexible
|
||||
```
|
||||
|
||||
### Issue: Unexpected monitoring fees
|
||||
|
||||
**Solution:**
|
||||
- Intelligent-Tiering has $0.0025 per 1000 objects monitoring fee
|
||||
- For 1 billion objects: $2,500/month monitoring
|
||||
- If cost is high, use lifecycle policies instead (no monitoring fee)
|
||||
|
||||
## Summary
|
||||
|
||||
**Recommended Strategy for Most Use Cases:**
|
||||
- **Intelligent-Tiering** for vectors and frequently queried data
|
||||
- **Lifecycle policies** for metadata and system files
|
||||
- **Batch operations** for efficient cleanup
|
||||
|
||||
**Expected Savings:**
|
||||
- **Year 1**: 40-50% reduction in storage costs
|
||||
- **Year 2+**: 60-70% reduction as more data ages into archive tiers
|
||||
- **Long-term**: 75-85% reduction for mature datasets
|
||||
|
||||
**500TB Example (Intelligent-Tiering):**
|
||||
- Before: $143,000/year
|
||||
- After: $51,000/year
|
||||
- **Savings: $92,000/year (64%)**
|
||||
|
||||
**1PB Example (Intelligent-Tiering):**
|
||||
- Before: $286,000/year
|
||||
- After: $102,000/year
|
||||
- **Savings: $184,000/year (64%)**
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-10-17
|
||||
**Cloud Provider**: AWS S3
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue