Compare commits
No commits in common. "cleanup-1.0" and "main" have entirely different histories.
cleanup-1.
...
main
980 changed files with 237640 additions and 164532 deletions
12
.aiignore
Normal file
12
.aiignore
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
# An .aiignore file follows the same syntax as a .gitignore file.
|
||||
# .gitignore documentation: https://git-scm.com/docs/gitignore
|
||||
|
||||
# you can ignore files
|
||||
.DS_Store
|
||||
*.log
|
||||
*.tmp
|
||||
|
||||
# or folders
|
||||
dist/
|
||||
build/
|
||||
out/
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
{
|
||||
"mcpServers": {
|
||||
"brain-cloud": {
|
||||
"command": "node",
|
||||
"args": [
|
||||
"brainy-mcp-server.js"
|
||||
],
|
||||
"env": {
|
||||
"CUSTOMER_ID": "demo-test-auto",
|
||||
"BRAIN_CLOUD_URL": "https://brain-cloud.dpsifr.workers.dev"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
170
.claude/skills/architecture.md
Normal file
170
.claude/skills/architecture.md
Normal file
|
|
@ -0,0 +1,170 @@
|
|||
# Brainy Architecture Reference
|
||||
|
||||
## What Is Brainy
|
||||
|
||||
@soulcraft/brainy (v7.17.0) is a Universal Knowledge Protocol -- a Triple Intelligence database combining vector search, graph traversal, and metadata filtering in a single library. Published to npm as a public MIT-licensed package.
|
||||
|
||||
## Core Architecture
|
||||
|
||||
### Storage Layer (`src/storage/`)
|
||||
- **StorageAdapter interface** (`src/coreTypes.ts:576`): The contract ALL storage backends implement. ALWAYS check this interface before adding storage methods.
|
||||
- **BaseStorage** (`src/storage/baseStorage.ts`): Base implementation with built-in type-aware partitioning (TypeAwareStorageAdapter was removed -- functionality merged into BaseStorage).
|
||||
- **Adapters** (`src/storage/adapters/`):
|
||||
- `fileSystemStorage.ts` -- local filesystem
|
||||
- `memoryStorage.ts` -- in-memory
|
||||
- `baseStorageAdapter.ts` -- shared adapter base (counts, batch ops)
|
||||
- Cloud + OPFS adapters were removed in 8.0 (cloud backup is operator tooling)
|
||||
- **Generational MVCC / Db API** (`src/db/`): immutable `Db` values over generation-stamped records
|
||||
- `db.ts` (the `Db` value), `generationStore.ts` (record layer + commit protocol), `types.ts`, `errors.ts`, `whereMatcher.ts`
|
||||
- Design record: `docs/ADR-001-generational-mvcc.md`; replaced the pre-8.0 COW branching + versioning subsystems
|
||||
|
||||
### Vector Search (`src/hnsw/`)
|
||||
- `hnswIndex.ts` -- HNSW-based approximate nearest neighbor search
|
||||
- `typeAwareHNSWIndex.ts` -- type-partitioned vector search
|
||||
- NOT in `src/intelligence/` (that directory does not exist)
|
||||
|
||||
### Graph Engine (`src/graph/`)
|
||||
- `graphAdjacencyIndex.ts` -- adjacency-based graph representation
|
||||
- `pathfinding.ts` -- relationship traversal and pathfinding
|
||||
- `lsm/` -- LSM tree implementation for graph storage
|
||||
|
||||
### Metadata Index (`src/utils/metadataIndex.ts`)
|
||||
- O(1) exact match via hash indexes
|
||||
- O(log n) range queries via sorted indexes
|
||||
- Roaring bitmap set operations for efficient filtering
|
||||
- Adaptive chunking strategy (`metadataIndexChunking.ts`)
|
||||
- Caching layer (`metadataIndexCache.ts`)
|
||||
|
||||
### Triple Intelligence (`src/triple/`)
|
||||
- `TripleIntelligenceSystem.ts` -- combines vector + graph + metadata into unified queries
|
||||
- Lazy-loaded indexes (loaded on first use, not at startup)
|
||||
|
||||
### Neural/AI Components (`src/neural/`)
|
||||
- Smart Importers (`src/importers/`): CSV, Excel, PDF, DOCX, YAML, JSON, Markdown, Orchestrator
|
||||
- `SmartExtractor.ts` -- entity extraction from unstructured data
|
||||
- `SmartRelationshipExtractor.ts` -- relationship detection
|
||||
- `NeuralEntityExtractor.ts` -- ML-based entity recognition
|
||||
- Natural language processing utilities
|
||||
|
||||
### Distributed Systems (`src/distributed/`)
|
||||
- Distributed Coordinator for multi-node operation
|
||||
- Shard Manager for data partitioning
|
||||
- Cache Synchronization across nodes
|
||||
- Read/Write separation
|
||||
- Network and HTTP transport layers
|
||||
- Storage discovery and shard migration
|
||||
|
||||
### Transaction Management (`src/transaction/`)
|
||||
- TransactionManager for ACID operations
|
||||
- Operations: SaveNoun, AddToHNSW, UpdateMetadata, etc.
|
||||
- Distributed transaction support
|
||||
|
||||
### Integration Hub (`src/integrations/`)
|
||||
- Google Sheets integration
|
||||
- OData (Open Data Protocol)
|
||||
- Server-Sent Events (SSE)
|
||||
- Webhooks
|
||||
- Event bus system
|
||||
|
||||
### Virtual Filesystem (`src/vfs/`)
|
||||
- `VirtualFileSystem.ts` -- full VFS implementation (87 KB)
|
||||
- `PathResolver.ts`, `FSCompat.ts`, `MimeTypeDetector.ts`, `TreeUtils.ts`
|
||||
- Subdirectories: `semantic/` (semantic search), `streams/` (streaming), `importers/`
|
||||
|
||||
### MCP Support (`src/mcp/`)
|
||||
- BrainyMCPAdapter, MCPAugmentationToolset, BrainyMCPService
|
||||
- Model Control Protocol request/response handling
|
||||
|
||||
### Aggregation Engine (`src/aggregation/`)
|
||||
- **AggregationIndex** (`AggregationIndex.ts`): Write-time incremental aggregation — SUM, COUNT, AVG, MIN, MAX with GROUP BY and time windows
|
||||
- **Time Windows** (`timeWindows.ts`): ISO 8601 bucketing — hour, day, week, month, quarter, year, custom intervals
|
||||
- **Materializer** (`materializer.ts`): Debounced writes of aggregate results as `NounType.Measurement` entities
|
||||
- Integrates into `brain.find({ aggregate })` for unified query API
|
||||
- Write hooks in `add()`, `update()`, `delete()` for O(1) incremental updates
|
||||
- `'aggregation'` provider key enables native plugin acceleration
|
||||
|
||||
### Additional Systems
|
||||
- **CLI** (`src/cli/`): Complete command-line tool with interactive mode and catalog system
|
||||
- **Migration** (`src/migration/`): MigrationRunner for database schema migrations
|
||||
- **Embeddings** (`src/embeddings/`): Embedding manager with Candle-WASM Rust source
|
||||
- **Streaming** (`src/streaming/`): Pipeline support with adaptive backpressure
|
||||
- **Versioning** (`src/versioning/`): VersioningAPI for data versioning
|
||||
- **Plugin System**: Registry-based plugin architecture
|
||||
- **Patterns** (`src/patterns/`): 7 pattern library JSON files
|
||||
|
||||
## Type System
|
||||
- **NounType** (42 types, `src/types/graphTypes.ts:850-893`): Person, Organization, Concept, Collection, Document, Task, Project, etc.
|
||||
- **VerbType** (127 types, `src/types/graphTypes.ts:900-1087`): Contains, RelatedTo, PartOf, Creates, DependsOn, MemberOf, etc.
|
||||
- All types in `src/types/`
|
||||
|
||||
## Module Exports (`src/index.ts`)
|
||||
38+ named exports including: Brainy class, configuration types, neural APIs (NeuralImport, NeuralEntityExtractor, SmartExtractor, SmartRelationshipExtractor), distance functions, plugin system, migration system, embedding functions, storage adapters, COW infrastructure, pipeline utilities, graph types, MCP components, integration hub, OData utilities, and more.
|
||||
|
||||
## File Structure
|
||||
```
|
||||
src/
|
||||
├── index.ts # 38+ public exports
|
||||
├── brainy.ts # Main Brainy class (6,500+ lines)
|
||||
├── setup.ts # Initialization polyfills
|
||||
├── coreTypes.ts # StorageAdapter interface + core types
|
||||
├── storage/
|
||||
│ ├── baseStorage.ts # Base storage (includes type-aware)
|
||||
│ ├── adapters/ # All storage backends + cloud adapters
|
||||
│ └── cow/ # Copy-on-Write versioning
|
||||
├── hnsw/ # HNSW vector search
|
||||
├── graph/ # Graph engine + pathfinding + LSM
|
||||
├── triple/ # Triple Intelligence system
|
||||
├── neural/ # Smart extractors + NLP
|
||||
├── importers/ # File format importers (8 types)
|
||||
├── distributed/ # Distributed database (16 files)
|
||||
├── transaction/ # ACID transactions (6 files)
|
||||
├── integrations/ # Sheets, OData, SSE, Webhooks
|
||||
├── vfs/ # Virtual filesystem + semantic search
|
||||
├── mcp/ # Model Control Protocol
|
||||
├── cli/ # Command-line interface
|
||||
├── migration/ # Schema migrations
|
||||
├── embeddings/ # Embedding manager + Candle-WASM
|
||||
├── streaming/ # Pipeline + backpressure
|
||||
├── versioning/ # Versioning API
|
||||
├── types/ # TypeScript type definitions
|
||||
├── utils/ # Metadata index, logging, etc.
|
||||
├── config/ # Configuration system
|
||||
├── patterns/ # Pattern library
|
||||
├── api/ # API layer
|
||||
├── interfaces/ # Interface definitions
|
||||
├── shared/ # Shared utilities
|
||||
├── data/ # Data utilities
|
||||
├── errors/ # Error handling
|
||||
├── critical/ # Critical error handling
|
||||
├── universal/ # Universal utilities
|
||||
├── import/ # Import functionality
|
||||
└── scripts/ # Build scripts
|
||||
```
|
||||
|
||||
## Initialization
|
||||
`brainy.ts` `init()` method performs initialization cascade:
|
||||
1. Load plugins
|
||||
2. Initialize storage
|
||||
3. Enable COW (Copy-on-Write)
|
||||
4. Set up embeddings
|
||||
5. Initialize caches
|
||||
6. Set up graph indexes
|
||||
7. Initialize VFS
|
||||
8. Set up transaction manager
|
||||
9. Initialize distributed components (if enabled)
|
||||
|
||||
## Testing
|
||||
- Framework: Vitest
|
||||
- Run: `npm test`
|
||||
- Test directories:
|
||||
- `tests/unit/` -- unit tests
|
||||
- `tests/integration/` -- integration tests
|
||||
- `tests/benchmarks/` -- performance benchmarks (NOT tests/performance/)
|
||||
- `tests/comprehensive/` -- comprehensive test suites
|
||||
- `tests/api/` -- API tests
|
||||
- `tests/helpers/` -- test utilities
|
||||
|
||||
## Release
|
||||
- `npm run release:patch/minor/major` -- fully automated via `scripts/release.sh`
|
||||
- `npm run release:dry` -- preview without changes
|
||||
- Uses conventional commits for changelog generation
|
||||
57
.dockerignore
Normal file
57
.dockerignore
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
# Git
|
||||
.git
|
||||
.gitignore
|
||||
|
||||
# Development
|
||||
.vscode
|
||||
.idea
|
||||
*.swp
|
||||
*.swo
|
||||
.DS_Store
|
||||
|
||||
# Node
|
||||
node_modules
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Testing
|
||||
tests
|
||||
*.test.ts
|
||||
*.test.js
|
||||
coverage
|
||||
.nyc_output
|
||||
|
||||
# Documentation (keep only essentials)
|
||||
docs
|
||||
*.md
|
||||
!README.md
|
||||
!LICENSE
|
||||
|
||||
# Build artifacts (will be built in Docker)
|
||||
dist
|
||||
build
|
||||
*.tsbuildinfo
|
||||
|
||||
# Environment
|
||||
.env
|
||||
.env.*
|
||||
|
||||
# Strategy and private docs
|
||||
.strategy
|
||||
CLAUDE.md
|
||||
|
||||
# Development files
|
||||
docker-compose.yml
|
||||
Dockerfile
|
||||
.dockerignore
|
||||
|
||||
# Data (should be mounted, not baked in)
|
||||
data
|
||||
*.db
|
||||
*.sqlite
|
||||
|
||||
# Models (should be downloaded at runtime or mounted)
|
||||
models
|
||||
*.onnx
|
||||
*.bin
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
DATABASE_URL=postgres://localhost/test
|
||||
API_KEY=sk-test-123456
|
||||
SECRET_TOKEN=super-secret-value
|
||||
NODE_ENV=production
|
||||
15
.github/FUNDING.yml
vendored
15
.github/FUNDING.yml
vendored
|
|
@ -1,15 +0,0 @@
|
|||
# These are supported funding model platforms
|
||||
|
||||
github: DPSIFR
|
||||
patreon: # Replace with a single Patreon username
|
||||
open_collective: # Replace with a single Open Collective username
|
||||
ko_fi: # Replace with a single Ko-fi username
|
||||
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
|
||||
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
|
||||
liberapay: # Replace with a single Liberapay username
|
||||
issuehunt: # Replace with a single IssueHunt username
|
||||
lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
|
||||
polar: # Replace with a single Polar username
|
||||
buy_me_a_coffee: # Replace with a single Buy Me a Coffee username
|
||||
thanks_dev: # Replace with a single thanks.dev username
|
||||
custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
|
||||
32
.github/ISSUE_TEMPLATE/bug_report.md
vendored
32
.github/ISSUE_TEMPLATE/bug_report.md
vendored
|
|
@ -1,32 +0,0 @@
|
|||
---
|
||||
name: Bug report
|
||||
about: Create a report to help us improve
|
||||
title: '[BUG] '
|
||||
labels: bug
|
||||
assignees: ''
|
||||
---
|
||||
|
||||
## Bug Description
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
## Reproduction Steps
|
||||
Steps to reproduce the behavior:
|
||||
1. Initialize BrainyData with '...'
|
||||
2. Call method '....'
|
||||
3. See error
|
||||
|
||||
## Expected Behavior
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
## Environment
|
||||
- Brainy version: [e.g. 0.9.4]
|
||||
- Environment: [e.g. Browser, Node.js, serverless]
|
||||
- Browser (if applicable): [e.g. Chrome, Safari]
|
||||
- Node.js version (if applicable): [e.g. 23.11.0]
|
||||
- Operating System: [e.g. Windows 10, macOS Monterey, Ubuntu 22.04]
|
||||
|
||||
## Additional Context
|
||||
Add any other context about the problem here. If applicable, include code snippets, error messages, or screenshots.
|
||||
|
||||
## Possible Solution
|
||||
If you have suggestions on how to fix the issue, please describe them here.
|
||||
22
.github/ISSUE_TEMPLATE/feature_request.md
vendored
22
.github/ISSUE_TEMPLATE/feature_request.md
vendored
|
|
@ -1,22 +0,0 @@
|
|||
---
|
||||
name: Feature request
|
||||
about: Suggest an idea for this project
|
||||
title: '[FEATURE] '
|
||||
labels: enhancement
|
||||
assignees: ''
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
A clear and concise description of what problem this feature would solve. For example: "I'm always frustrated when [...]"
|
||||
|
||||
## Proposed Solution
|
||||
A clear and concise description of what you want to happen.
|
||||
|
||||
## Alternative Solutions
|
||||
A clear and concise description of any alternative solutions or features you've considered.
|
||||
|
||||
## Use Case
|
||||
Describe a concrete use case that highlights the value of this feature.
|
||||
|
||||
## Additional Context
|
||||
Add any other context, code examples, or references about the feature request here.
|
||||
27
.github/PULL_REQUEST_TEMPLATE.md
vendored
27
.github/PULL_REQUEST_TEMPLATE.md
vendored
|
|
@ -1,27 +0,0 @@
|
|||
## Description
|
||||
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
|
||||
|
||||
Fixes # (issue)
|
||||
|
||||
## Type of change
|
||||
Please delete options that are not relevant.
|
||||
|
||||
- [ ] Bug fix (non-breaking change which fixes an issue)
|
||||
- [ ] New feature (non-breaking change which adds functionality)
|
||||
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
|
||||
- [ ] Documentation update
|
||||
- [ ] Performance improvement
|
||||
- [ ] Code refactoring (no functional changes)
|
||||
|
||||
## How Has This Been Tested?
|
||||
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce.
|
||||
|
||||
## Checklist:
|
||||
- [ ] My code follows the style guidelines of this project
|
||||
- [ ] I have performed a self-review of my own code
|
||||
- [ ] I have commented my code, particularly in hard-to-understand areas
|
||||
- [ ] I have made corresponding changes to the documentation
|
||||
- [ ] My changes generate no new warnings
|
||||
- [ ] I have added tests that prove my fix is effective or that my feature works
|
||||
- [ ] New and existing unit tests pass locally with my changes
|
||||
- [ ] Any dependent changes have been merged and published in downstream modules
|
||||
40
.github/workflows/ci.yml
vendored
Normal file
40
.github/workflows/ci.yml
vendored
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
node:
|
||||
name: Node ${{ matrix.node-version }}
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
node-version: ['22', '24']
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ matrix.node-version }}
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- run: npm run test:unit
|
||||
|
||||
bun:
|
||||
name: Bun (latest)
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
cache: npm
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: latest
|
||||
- run: npm ci
|
||||
# test:bun imports the built dist/, so build first.
|
||||
- run: npm run build
|
||||
# Bun as a runtime is the supported Bun story (`bun add` / `bun run`).
|
||||
- run: npm run test:bun
|
||||
155
.gitignore
vendored
155
.gitignore
vendored
|
|
@ -1,89 +1,116 @@
|
|||
# Build output
|
||||
/tmp
|
||||
/out-tsc
|
||||
/dist
|
||||
/cloud-wrapper/dist
|
||||
|
||||
# Dependencies
|
||||
/node_modules
|
||||
/cloud-wrapper/node_modules
|
||||
node_modules/
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
/.pnp
|
||||
.pnp.js
|
||||
|
||||
# Package artifacts
|
||||
*.tgz
|
||||
*.tar.gz
|
||||
# Build outputs
|
||||
dist/
|
||||
build/
|
||||
*.tsbuildinfo
|
||||
|
||||
# Coverage directory
|
||||
/coverage
|
||||
|
||||
# Environment files
|
||||
# Environment variables
|
||||
.env
|
||||
.env.local
|
||||
.env.development.local
|
||||
.env.test.local
|
||||
.env.production.local
|
||||
|
||||
# Runtime data
|
||||
brainy-data/
|
||||
.brainy/
|
||||
*.log
|
||||
*.pid
|
||||
*.seed
|
||||
*.pid.lock
|
||||
|
||||
# Coverage directory used by tools like istanbul
|
||||
coverage/
|
||||
*.lcov
|
||||
|
||||
# Test results
|
||||
tests/results/
|
||||
|
||||
# Filesystem test artifacts (created by integration tests)
|
||||
test-*/
|
||||
|
||||
# IDE files
|
||||
.vscode/
|
||||
.idea/
|
||||
*.iml
|
||||
*.iws
|
||||
*.ipr
|
||||
*.sublime-workspace
|
||||
*.sublime-project
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Data directories created by FileSystemStorage
|
||||
/brainy-data
|
||||
/custom-data
|
||||
/clean-history.sh
|
||||
/bluesky-augmentation/node_modules/
|
||||
/bluesky-augmentation/dist/
|
||||
# Temporary files
|
||||
tmp/
|
||||
temp/
|
||||
*.tmp
|
||||
|
||||
# Test files
|
||||
/test-worker.js
|
||||
/test-node24-worker.js
|
||||
/test-results.json
|
||||
/tests/results/
|
||||
/tests/results/*.json
|
||||
# Planning and instruction files
|
||||
plan.md
|
||||
|
||||
# Generated files
|
||||
/encoded-image.html
|
||||
/encoded-image.txt
|
||||
/npm
|
||||
/rollup
|
||||
/soulcraft-brainy-*.tgz
|
||||
/data/
|
||||
# Package files
|
||||
*.tgz
|
||||
|
||||
# Temporary test files created by AI agents
|
||||
test*.js
|
||||
test*.ts
|
||||
temp-test*.js
|
||||
temp-test*.ts
|
||||
reproduction*.js
|
||||
reproduction*.ts
|
||||
debug*.js
|
||||
debug*.ts
|
||||
/temp/
|
||||
/temp-tests/
|
||||
/brainy-models-package/node_modules/
|
||||
/test-consumer/node_modules/
|
||||
/test-install/node_modules/
|
||||
# Private/confidential files
|
||||
PLAN.md
|
||||
INTERNAL_NOTES.md
|
||||
TODO_PRIVATE.md
|
||||
*.tar.gz
|
||||
|
||||
# Downloaded models (temporary)
|
||||
/models-download/
|
||||
# Strategy and planning documents (private)
|
||||
.strategy/
|
||||
# Removed: PRODUCTION_*.md (now these should be public documentation)
|
||||
DISTRIBUTED_*.md
|
||||
*_ASSESSMENT.md
|
||||
*_ANALYSIS.md
|
||||
*_TRUTH*.md
|
||||
|
||||
# Sensitive files - NEVER commit
|
||||
*.pdf
|
||||
*pitch*deck*
|
||||
*investor*deck*
|
||||
*confidential*
|
||||
*private*
|
||||
CLAUDE.md
|
||||
# Models (downloaded at runtime)
|
||||
models/
|
||||
models-cache/
|
||||
|
||||
# But include bundled WASM model assets
|
||||
!assets/models/
|
||||
|
||||
# Development planning files (not for commit)
|
||||
PLAN.md
|
||||
|
||||
# Backup folders
|
||||
backup-*
|
||||
backup/
|
||||
|
||||
# Internal documentation
|
||||
docs/internal/
|
||||
|
||||
# Cache files
|
||||
*.cache
|
||||
|
||||
# Rust/Cargo build artifacts
|
||||
src/embeddings/candle-wasm/target/
|
||||
src/embeddings/candle-wasm/Cargo.lock
|
||||
|
||||
# Ignore the wasm-pack output dir's CONTENTS (note the `/*`, not `/`, so the
|
||||
# re-includes below can take effect — git cannot re-include a file whose parent
|
||||
# DIRECTORY is excluded). Keep the pre-built WASM committed: it ships in the npm
|
||||
# package anyway, it lets consumers + CI build without a Rust/wasm-pack toolchain,
|
||||
# and versioning it makes the shipped artifact reproducible (not "whatever the
|
||||
# maintainer last built").
|
||||
src/embeddings/wasm/pkg/*
|
||||
!src/embeddings/wasm/pkg/*.wasm
|
||||
!src/embeddings/wasm/pkg/*.js
|
||||
!src/embeddings/wasm/pkg/*.d.ts
|
||||
|
||||
# Log files (redundant but explicit)
|
||||
*.log
|
||||
|
||||
# Temporary files (redundant but explicit)
|
||||
*.tmp
|
||||
/.junie/guidelines.md
|
||||
|
||||
# Claude Code harness state
|
||||
.claude/scheduled_tasks.lock
|
||||
|
|
|
|||
134
.npmignore
134
.npmignore
|
|
@ -1,70 +1,84 @@
|
|||
# Exclude source maps - multiple patterns for safety
|
||||
*.map
|
||||
**/*.map
|
||||
dist/**/*.map
|
||||
*.js.map
|
||||
dist/**/*.js.map
|
||||
|
||||
# Development files
|
||||
node_modules/
|
||||
# Source files (not needed in package)
|
||||
src/
|
||||
tests/
|
||||
examples/
|
||||
.github/
|
||||
.vscode/
|
||||
.idea/
|
||||
cloud-wrapper/
|
||||
scripts/
|
||||
dev/
|
||||
coverage/
|
||||
|
||||
# Model files (downloaded on first use, not bundled)
|
||||
models/
|
||||
models-cache/
|
||||
|
||||
# Development and backup files
|
||||
backup-*
|
||||
backup-*/
|
||||
docs/backup*/
|
||||
|
||||
# Documentation (except essentials)
|
||||
*.md
|
||||
!README.md
|
||||
!LICENSE
|
||||
!CHANGELOG.md
|
||||
!MIGRATION.md
|
||||
|
||||
# Configuration files
|
||||
.eslintrc
|
||||
.prettierrc
|
||||
tsconfig*.json
|
||||
rollup.config.js
|
||||
jest.config.js
|
||||
.gitignore
|
||||
.npmignore
|
||||
tsconfig.json
|
||||
vitest.config.ts
|
||||
vitest.config.mts
|
||||
*.config.js
|
||||
*.config.ts
|
||||
.eslintrc*
|
||||
.prettierrc*
|
||||
|
||||
# Build artifacts
|
||||
emocoverage/
|
||||
.nyc_output/
|
||||
# Test files
|
||||
test-*.js
|
||||
test-*.ts
|
||||
*.test.ts
|
||||
*.test.js
|
||||
*.spec.ts
|
||||
*.spec.js
|
||||
|
||||
# Framework bundles (not needed in npm package)
|
||||
dist/framework.js
|
||||
dist/framework.min.js
|
||||
dist/framework.js.map
|
||||
dist/framework.min.js.map
|
||||
|
||||
# Large files
|
||||
# Include the logo but exclude other PNGs
|
||||
!brainy.png
|
||||
*.png
|
||||
encoded-image.*
|
||||
README.demo.md
|
||||
scalingStrategy.md
|
||||
|
||||
# Misc
|
||||
.DS_Store
|
||||
# Temporary and log files
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
test-results.json
|
||||
package-lock.json
|
||||
*.tmp
|
||||
tmp/
|
||||
temp/
|
||||
brainy-data/
|
||||
|
||||
# Additional large files to exclude
|
||||
*.js
|
||||
!dist/**/*.js
|
||||
!bin/*.js
|
||||
# Git and CI files
|
||||
.git/
|
||||
.github/
|
||||
.gitlab-ci.yml
|
||||
.travis.yml
|
||||
|
||||
# Documentation that's not needed for npm package
|
||||
CHANGELOG.md
|
||||
CORTEX*.md
|
||||
METADATA_*.md
|
||||
PERFORMANCE_*.md
|
||||
IMPLEMENTATION-*.md
|
||||
TENSORFLOW_*.md
|
||||
MIGRATION_*.md
|
||||
BRAIN_CLOUD_*.md
|
||||
JARVIS_*.md
|
||||
OFFLINE_*.md
|
||||
CORTEX-ROADMAP.md
|
||||
# IDE files
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Private files
|
||||
PLAN.md
|
||||
CLAUDE.md
|
||||
INTERNAL_NOTES.md
|
||||
TODO_PRIVATE.md
|
||||
*-ANALYSIS.md
|
||||
*-PLAN.md
|
||||
|
||||
# Build artifacts not needed
|
||||
*.tsbuildinfo
|
||||
*.map
|
||||
|
||||
# Development environment
|
||||
.env*
|
||||
.nvm*
|
||||
.node-version
|
||||
|
||||
# Keep dist/ for the compiled code
|
||||
# Keep bin/ for the CLI
|
||||
# Keep package.json, package-lock.json
|
||||
1
.nvmrc
Normal file
1
.nvmrc
Normal file
|
|
@ -0,0 +1 @@
|
|||
22
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
{
|
||||
"types": [
|
||||
{"type": "feat", "section": "Added", "hidden": false},
|
||||
{"type": "fix", "section": "Fixed", "hidden": false},
|
||||
{"type": "chore", "section": "Changed", "hidden": false},
|
||||
{"type": "docs", "section": "Documentation", "hidden": false},
|
||||
{"type": "style", "section": "Changed", "hidden": true},
|
||||
{"type": "refactor", "section": "Changed", "hidden": false},
|
||||
{"type": "perf", "section": "Changed", "hidden": false},
|
||||
{"type": "test", "section": "Tests", "hidden": true},
|
||||
{"type": "build", "section": "Build System", "hidden": true},
|
||||
{"type": "ci", "section": "Continuous Integration", "hidden": true}
|
||||
],
|
||||
"releaseCommitMessageFormat": "chore(release): {{currentTag}} [skip ci]",
|
||||
"commitUrlFormat": "https://github.com/soulcraftlabs/brainy/commit/{{hash}}",
|
||||
"compareUrlFormat": "https://github.com/soulcraftlabs/brainy/compare/{{previousTag}}...{{currentTag}}",
|
||||
"issueUrlFormat": "https://github.com/soulcraftlabs/brainy/issues/{{id}}",
|
||||
"userUrlFormat": "https://github.com/{{user}}",
|
||||
"skip": {
|
||||
"tag": false
|
||||
}
|
||||
}
|
||||
|
|
@ -1,263 +0,0 @@
|
|||
# 🧠 Brainy Augmentation Architecture - Complete Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy has a clear augmentation system with four tiers:
|
||||
|
||||
```
|
||||
1. Built-in (Free, Always Included)
|
||||
2. Community (Free, npm packages)
|
||||
3. Premium (Brain Cloud subscription - auto-loads after auth)
|
||||
4. Brain Cloud (Managed Service)
|
||||
```
|
||||
|
||||
## 1. Built-in Augmentations (Always Free)
|
||||
|
||||
These come with every Brainy installation:
|
||||
|
||||
### Currently Implemented:
|
||||
- **NeuralImport** - AI-powered data understanding
|
||||
- **Basic Storage** - Filesystem/memory persistence
|
||||
- **Vector Search** - Semantic similarity
|
||||
- **Graph Traversal** - Relationship queries
|
||||
|
||||
### To Be Added (Still Built-in):
|
||||
```javascript
|
||||
// src/augmentations/built-in/
|
||||
├── autoSave.ts // Automatic persistence
|
||||
├── basicCache.ts // Query caching
|
||||
├── simpleBackup.ts // Local backups
|
||||
└── metadataIndex.ts // Facet indexing
|
||||
```
|
||||
|
||||
## 2. Community Augmentations (Free, Open Source)
|
||||
|
||||
Published to npm by the community:
|
||||
|
||||
```bash
|
||||
# Examples (to be created by community)
|
||||
npm install brainy-sentiment # Sentiment analysis
|
||||
npm install brainy-translator # Multi-language
|
||||
npm install brainy-summarizer # Text summarization
|
||||
npm install brainy-classifier # Text classification
|
||||
```
|
||||
|
||||
Usage:
|
||||
```javascript
|
||||
import { SentimentAnalyzer } from 'brainy-sentiment'
|
||||
|
||||
const cortex = new Cortex()
|
||||
cortex.register(new SentimentAnalyzer())
|
||||
```
|
||||
|
||||
## 3. Premium Augmentations (Brain Cloud Auto-Loading)
|
||||
|
||||
The `/brain-cloud` project contains premium augmentations:
|
||||
|
||||
### Core Premium Features (AI Memory & Coordination):
|
||||
```javascript
|
||||
// After 'brainy cloud auth', these are automatically available:
|
||||
// - AIMemory // Persistent AI memory across sessions
|
||||
// - AgentCoordinator // Multi-agent handoffs
|
||||
// - TeamSync // Real-time team synchronization
|
||||
// - CloudBackup // Automatic cloud backups
|
||||
|
||||
// No imports needed - they auto-load based on your subscription!
|
||||
const cortex = new Cortex()
|
||||
// Premium augmentations are automatically registered
|
||||
```
|
||||
|
||||
### Enterprise Connectors (Auto-Loading):
|
||||
```javascript
|
||||
// Enterprise augmentations auto-load for Team+ plans after auth:
|
||||
// - NotionSync // Bidirectional Notion sync
|
||||
// - SalesforceConnect // CRM integration
|
||||
// - AirtableSync // Database sync
|
||||
// - PostgresSync // Real-time replication
|
||||
// - SlackMemory // Team knowledge base
|
||||
// - AnalyticsSuite // Business intelligence
|
||||
|
||||
// Just configure with your credentials - no imports needed!
|
||||
await brainy.connectNotion({
|
||||
notionToken: process.env.NOTION_TOKEN
|
||||
})
|
||||
```
|
||||
|
||||
## 4. Brain Cloud Service (Managed)
|
||||
|
||||
The hosted service at brain-cloud.soulcraft.com:
|
||||
|
||||
```javascript
|
||||
// Connect to managed service
|
||||
await brain.connect('brain-cloud.soulcraft.com', {
|
||||
instance: 'my-team',
|
||||
apiKey: process.env.BRAIN_CLOUD_KEY
|
||||
})
|
||||
```
|
||||
|
||||
## CLI Commands Structure
|
||||
|
||||
### Core Commands (brainy)
|
||||
```bash
|
||||
# Database operations
|
||||
brainy init # Initialize Brainy
|
||||
brainy add "data" # Add data
|
||||
brainy search "query" # Search
|
||||
brainy chat # Interactive chat
|
||||
|
||||
# Augmentation management
|
||||
brainy augment # List augmentations
|
||||
brainy augment add # Add augmentation (interactive)
|
||||
brainy augment remove # Remove augmentation
|
||||
brainy augment config # Configure augmentation
|
||||
|
||||
# Brain Cloud connection
|
||||
brainy cloud # Connect to Brain Cloud service
|
||||
brainy cloud --status # Check connection status
|
||||
brainy cloud --sync # Force sync
|
||||
```
|
||||
|
||||
### Installing Augmentations via CLI
|
||||
|
||||
#### Built-in (always available):
|
||||
```bash
|
||||
brainy augment enable neural-import
|
||||
brainy augment enable auto-save
|
||||
```
|
||||
|
||||
#### Community (from npm):
|
||||
```bash
|
||||
# Install from npm first
|
||||
npm install -g brainy-sentiment
|
||||
|
||||
# Then register with Brainy
|
||||
brainy augment add brainy-sentiment --type sense
|
||||
```
|
||||
|
||||
#### Premium (requires license):
|
||||
```bash
|
||||
# Set license key
|
||||
export BRAINY_LICENSE_KEY=lic_xxxxx
|
||||
|
||||
# Install premium package
|
||||
brainy cloud auth # Auto-configures based on your subscription
|
||||
|
||||
# Register augmentations
|
||||
brainy augment add ai-memory --premium
|
||||
brainy augment add notion-sync --premium
|
||||
```
|
||||
|
||||
#### For Brain Cloud service:
|
||||
```bash
|
||||
# Connect to cloud (handles everything)
|
||||
brainy cloud --connect YOUR_CUSTOMER_ID
|
||||
```
|
||||
|
||||
## How Augmentations Work
|
||||
|
||||
### 1. Local Instance
|
||||
```javascript
|
||||
const brain = new BrainyData()
|
||||
const cortex = new Cortex()
|
||||
|
||||
// Register augmentations
|
||||
cortex.register(new NeuralImport(brain)) // Built-in
|
||||
cortex.register(new SentimentAnalyzer()) // Community
|
||||
cortex.register(new NotionSync({ key })) // Premium
|
||||
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
### 2. Remote Hosted Instance
|
||||
```javascript
|
||||
// Connect to remote Brainy
|
||||
const brain = new BrainyData({
|
||||
remote: 'https://my-brainy-server.com'
|
||||
})
|
||||
|
||||
// Augmentations run on server
|
||||
await brain.addAugmentation('sentiment-analyzer')
|
||||
```
|
||||
|
||||
### 3. Brain Cloud Instance
|
||||
```javascript
|
||||
// Connect to Brain Cloud
|
||||
const brain = new BrainyData({
|
||||
cloud: true,
|
||||
customerId: 'cust_xxx'
|
||||
})
|
||||
|
||||
// All premium augmentations available
|
||||
// Managed by Brain Cloud service
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
### /brainy (this project)
|
||||
```
|
||||
src/
|
||||
├── augmentations/
|
||||
│ ├── built-in/ # Free, always included
|
||||
│ │ ├── neuralImport.ts
|
||||
│ │ ├── autoSave.ts
|
||||
│ │ └── basicCache.ts
|
||||
│ └── cortexSense.ts # Legacy, being refactored
|
||||
├── cortex.ts # Orchestrator
|
||||
└── brainyData.ts # Core database
|
||||
```
|
||||
|
||||
### /brain-cloud (premium project)
|
||||
```
|
||||
src/
|
||||
├── augmentations/
|
||||
│ ├── memory/ # AI Memory features
|
||||
│ │ ├── aiMemory.ts
|
||||
│ │ ├── agentCoordinator.ts
|
||||
│ │ └── teamSync.ts
|
||||
│ ├── enterprise/ # Enterprise connectors
|
||||
│ │ ├── notionSync.ts
|
||||
│ │ ├── salesforce.ts
|
||||
│ │ └── airtable.ts
|
||||
│ └── index.ts # Main exports
|
||||
├── licensing/ # License validation
|
||||
└── cloud-service/ # Brain Cloud API
|
||||
```
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 1: Update References
|
||||
- [x] Replace all `brainy-quantum-vault` → Brain Cloud managed service
|
||||
- [ ] Update documentation
|
||||
- [ ] Update CLI commands
|
||||
|
||||
### Phase 2: Restructure brain-cloud
|
||||
- [ ] Organize enterprise connectors in brain-cloud managed service
|
||||
- [ ] Add AI memory augmentations
|
||||
- [ ] Implement license validation
|
||||
|
||||
### Phase 3: CLI Enhancement
|
||||
- [ ] Add `brainy augment` commands
|
||||
- [ ] Interactive augmentation installer
|
||||
- [ ] Auto-detect available augmentations
|
||||
|
||||
### Phase 4: Documentation
|
||||
- [ ] Update README with clear tiers
|
||||
- [ ] Create augmentation development guide
|
||||
- [ ] Website update instructions
|
||||
|
||||
## License Model
|
||||
|
||||
```
|
||||
Built-in: MIT License (Free forever)
|
||||
Community: Varies (usually MIT)
|
||||
Premium: Commercial License ($49-299/mo)
|
||||
Cloud: Subscription ($19-99/mo)
|
||||
```
|
||||
|
||||
## The Promise
|
||||
|
||||
1. **Built-in augmentations are ALWAYS free**
|
||||
2. **No feature moves from free to paid**
|
||||
3. **Community contributions welcome**
|
||||
4. **Premium funds open source development**
|
||||
5. **Brain Cloud is optional, not required**
|
||||
5004
CHANGELOG.md
5004
CHANGELOG.md
File diff suppressed because it is too large
Load diff
208
CLAUDE.md
Normal file
208
CLAUDE.md
Normal file
|
|
@ -0,0 +1,208 @@
|
|||
# Brainy - Claude Code Project Guide
|
||||
|
||||
This file provides guidance for Claude Code (and human contributors) when working on the Brainy codebase.
|
||||
|
||||
## Cross-Project Coordination
|
||||
|
||||
Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
|
||||
|
||||
**At session START:** Read the handoff. Find rows where Owner = Brainy. Act on those first.
|
||||
|
||||
**At session END:** Mark completed actions ✅, delete rows you finished, delete threads with zero remaining actions. File must not grow. **If you shipped anything consumers need to know about, update `RELEASES.md` before closing.**
|
||||
|
||||
**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
|
||||
|
||||
**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`)
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraft/brainy` on npm under the MIT license.
|
||||
|
||||
## Getting Started
|
||||
|
||||
```bash
|
||||
npm install # Install dependencies
|
||||
npm run build # Build the project
|
||||
npm test # Run test suite (Vitest)
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
Full architecture reference: `.claude/skills/architecture.md`
|
||||
|
||||
### Core Systems
|
||||
- **Storage** (`src/storage/`): Pluggable storage backends via StorageAdapter interface (`src/coreTypes.ts`)
|
||||
- **Vector Search** (`src/hnsw/`): HNSW approximate nearest neighbor search
|
||||
- **Graph Engine** (`src/graph/`): Relationship traversal with adjacency index and pathfinding
|
||||
- **Metadata Index** (`src/utils/metadataIndex.ts`): O(1) exact match, O(log n) range queries
|
||||
- **Triple Intelligence** (`src/triple/`): Unified query combining all three intelligence types
|
||||
- **Aggregation Engine** (`src/aggregation/`): Write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows
|
||||
- **Virtual Filesystem** (`src/vfs/`): Full VFS with semantic search
|
||||
|
||||
### Type System
|
||||
- **NounType** (42 types): Entity classification -- Person, Concept, Collection, Document, Task, etc.
|
||||
- **VerbType** (127 types): Relationship types -- Contains, RelatedTo, PartOf, Creates, DependsOn, etc.
|
||||
- Defined in `src/types/graphTypes.ts`
|
||||
|
||||
## Code Standards
|
||||
|
||||
### TypeScript
|
||||
- Strict mode enabled
|
||||
- Target: ES2020, NodeNext module resolution
|
||||
- All new code must be TypeScript
|
||||
- Follow existing patterns -- read related code before writing
|
||||
|
||||
### Quality
|
||||
- All code must compile without errors
|
||||
- All code must have working tests that exercise real behavior
|
||||
- No stub returns (`return {} as any`)
|
||||
- No incomplete implementations with TODO comments
|
||||
- If something can't be fully implemented, throw an explicit error rather than faking it
|
||||
|
||||
### Verification Before Code Changes
|
||||
1. Check that interfaces and methods actually exist before using them
|
||||
2. Check that type properties are in the type definitions
|
||||
3. Run `npm test` -- tests must pass
|
||||
4. Run `npm run build` -- build must succeed
|
||||
|
||||
### Testing
|
||||
- Framework: Vitest
|
||||
- Tests in `tests/` (unit, integration, benchmarks, comprehensive)
|
||||
- Use in-memory storage for speed where possible
|
||||
- Tests must exercise real behavior, not mock it
|
||||
- Benchmarks are in `tests/benchmarks/` (not tests/performance/)
|
||||
|
||||
## Commit Conventions
|
||||
|
||||
Use [Conventional Commits](https://www.conventionalcommits.org/):
|
||||
|
||||
```
|
||||
feat: add new feature (minor version bump)
|
||||
fix: resolve bug (patch version bump)
|
||||
docs: update documentation (patch version bump)
|
||||
perf: improve performance (patch version bump)
|
||||
refactor: restructure code (patch version bump)
|
||||
test: add/update tests (patch version bump)
|
||||
```
|
||||
|
||||
**Important:** Never use `BREAKING CHANGE` in commit messages. Major version bumps are manual decisions only (`npm run release:major`).
|
||||
|
||||
## Docs Pipeline — soulcraft.com/docs
|
||||
|
||||
Docs in `docs/**/*.md` are published with the npm package (included in `files`) and synced to soulcraft.com/docs on every portal deploy. Frontmatter controls what appears publicly.
|
||||
|
||||
### Docs check triggers
|
||||
|
||||
Run the docs check whenever the user says ANY of:
|
||||
- "commit, publish, release" / "release" / "publish"
|
||||
- "update the docs" / "make sure docs are accurate" / "check the docs"
|
||||
- "review docs" / "clean up docs"
|
||||
|
||||
### Pre-release docs check (MANDATORY before every release)
|
||||
|
||||
When the user says "commit, publish, release" or any variation, **before committing**:
|
||||
|
||||
1. **Scan all files changed in this session** (and any recently added `docs/*.md` files)
|
||||
2. For each changed/new doc, decide: is this useful to external users?
|
||||
- **Yes** → ensure it has complete frontmatter (add or update it)
|
||||
- **No** (internal, migration, dev-only) → ensure it has no frontmatter or `public: false`
|
||||
3. For docs that already have frontmatter, verify:
|
||||
- `description` still matches the actual content
|
||||
- `next` links still exist and are still the right follow-up pages
|
||||
- `title` matches the doc's h1
|
||||
4. Include frontmatter changes in the commit
|
||||
|
||||
### Frontmatter format
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: Human-readable title
|
||||
slug: category/page-name # URL: soulcraft.com/docs/category/page-name
|
||||
public: true # false or absent = not published
|
||||
category: getting-started | concepts | guides | api
|
||||
template: guide | concept | api # controls layout on soulcraft.com
|
||||
order: 1 # sidebar position within category (lower = first)
|
||||
description: One sentence. What this doc covers and why it matters.
|
||||
next: # "Next steps" links shown at bottom of page
|
||||
- category/other-slug
|
||||
---
|
||||
```
|
||||
|
||||
### Category guide
|
||||
|
||||
| category | use for |
|
||||
|----------|---------|
|
||||
| `getting-started` | installation, quick start, first steps |
|
||||
| `concepts` | how the system works, mental models |
|
||||
| `guides` | how to do specific things, recipes |
|
||||
| `api` | method reference, signatures, parameters |
|
||||
|
||||
### What stays internal (no frontmatter / `public: false`)
|
||||
|
||||
- Release guides, developer learning paths
|
||||
- Migration guides for old versions (v3→v4, v5.11)
|
||||
- Architecture analysis docs (clustering algorithms, etc.)
|
||||
- Anything in `docs/internal/`
|
||||
- Deployment/ops/cost docs (cloud-run, kubernetes, cost-optimization)
|
||||
|
||||
## Release Process
|
||||
|
||||
Fully automated via `scripts/release.sh`:
|
||||
|
||||
```bash
|
||||
npm run release:dry # Preview (no changes)
|
||||
npm run release:patch # Bug fixes
|
||||
npm run release:minor # New features
|
||||
npm run release:major # Breaking changes (rare, manual decision)
|
||||
```
|
||||
|
||||
The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release.
|
||||
|
||||
After a successful release, remind the user:
|
||||
> "Published. Deploy portal to pick up the new docs → go to the portal project and deploy."
|
||||
|
||||
Do NOT deploy portal from here. Portal is always deployed separately from within the portal project.
|
||||
|
||||
## Closed-Source Product Names — HARD RULE
|
||||
|
||||
Brainy is the only Soulcraft open-source project. Nothing in this repo — code, JSDoc, tests,
|
||||
docs, RELEASES.md, CHANGELOG.md, commit messages — may reference closed-source Soulcraft
|
||||
products by name (Workshop, Venue, Memory, Muse, Hall, Forge, Academy, Pulse, Heart,
|
||||
Collective, SDK) or by their specific class/method names (`BookingDraftService`,
|
||||
`getDemandHeatmap`, `systemKind`, etc.).
|
||||
|
||||
When recording a consumer-reported bug, regression scenario, or release note:
|
||||
- Refer to "a consumer", "a downstream application", "a production deployment", or "an
|
||||
internal report" — never name the product.
|
||||
- All doc examples must use generic domain values (`'employee'`, `'customer'`, `'invoice'`,
|
||||
`'milestone'`, `OrderService`, `/orders/...`), not product-specific schemas.
|
||||
- Internal session artifacts (`.strategy/`, `~/.claude/plans/`, handoff files outside the
|
||||
repo) MAY name products — those are not public.
|
||||
|
||||
If you catch yourself typing a product name into a tracked file, stop and rephrase.
|
||||
|
||||
## Performance Claims
|
||||
|
||||
When documenting performance characteristics:
|
||||
- **MEASURED**: Cite the test file and line number
|
||||
- **PROJECTED**: Clearly label as extrapolated from tested scale
|
||||
- Never claim a performance figure without context or evidence
|
||||
|
||||
## Debugging
|
||||
|
||||
When a bug persists through 2+ fix attempts, switch to systematic debugging:
|
||||
1. Add comprehensive logging at every step
|
||||
2. Test with production-like data
|
||||
3. Trace the complete execution path
|
||||
4. Check both library code and consumer code
|
||||
5. Verify with actual test execution before declaring fixed
|
||||
|
||||
## Key Paths
|
||||
|
||||
- Main class: `src/brainy.ts`
|
||||
- Public API: `src/index.ts` (38+ exports)
|
||||
- Storage interface: `src/coreTypes.ts`
|
||||
- Type definitions: `src/types/`
|
||||
- Strategy/planning docs: `.strategy/` (gitignored, not public)
|
||||
|
|
@ -1,251 +0,0 @@
|
|||
# 🧠 Brainy CLI - Augmentation Management Guide
|
||||
|
||||
## Complete CLI Commands for Augmentations
|
||||
|
||||
### Core Commands
|
||||
|
||||
```bash
|
||||
# Initialize Brainy
|
||||
brainy init
|
||||
|
||||
# Basic operations
|
||||
brainy add "data" # Add data
|
||||
brainy search "query" # Search
|
||||
brainy chat # Interactive chat
|
||||
```
|
||||
|
||||
### Augmentation Management
|
||||
|
||||
```bash
|
||||
# List all augmentations
|
||||
brainy augment # Shows installed & available
|
||||
brainy augment list # Same as above
|
||||
brainy augment available # Shows what can be installed
|
||||
|
||||
# Add augmentations
|
||||
brainy augment add # Interactive mode
|
||||
brainy augment add neural-import # Built-in
|
||||
brainy augment add brainy-sentiment # Community (from npm)
|
||||
brainy augment add ai-memory # Brain Cloud feature (optional)
|
||||
|
||||
# Remove augmentations
|
||||
brainy augment remove sentiment-analyzer
|
||||
brainy augment disable neural-import # Disable but keep installed
|
||||
|
||||
# Configure augmentations
|
||||
brainy augment config neural-import
|
||||
brainy augment config notion-sync --set apiKey=xxx
|
||||
```
|
||||
|
||||
### Installing Different Types
|
||||
|
||||
#### 1. Built-in Augmentations (Free, Always Available)
|
||||
```bash
|
||||
# These are included - just enable them
|
||||
brainy augment enable neural-import
|
||||
brainy augment enable auto-save
|
||||
brainy augment enable basic-cache
|
||||
```
|
||||
|
||||
#### 2. Community Augmentations (Free, From npm)
|
||||
```bash
|
||||
# Step 1: Install from npm
|
||||
npm install -g brainy-sentiment
|
||||
|
||||
# Step 2: Register with Brainy
|
||||
brainy augment add brainy-sentiment
|
||||
|
||||
# Or in one command (coming soon)
|
||||
brainy augment install brainy-sentiment # Auto-installs from npm
|
||||
```
|
||||
|
||||
#### 3. Premium Augmentations (Paid, From @soulcraft/brain-cloud)
|
||||
```bash
|
||||
# Step 1: Set license key
|
||||
export BRAINY_LICENSE_KEY=lic_xxxxxxxxxxxxx
|
||||
|
||||
# Step 2: Install premium package
|
||||
npm install -g @soulcraft/brain-cloud
|
||||
|
||||
# Step 3: Add specific augmentations
|
||||
brainy augment add ai-memory --premium
|
||||
brainy augment add agent-coordinator --premium
|
||||
brainy augment add notion-sync --premium
|
||||
|
||||
# Or add all premium at once
|
||||
brainy augment add-premium --all
|
||||
```
|
||||
|
||||
#### 4. Brain Cloud Service (Managed, Everything Included)
|
||||
```bash
|
||||
# Connect to Brain Cloud (includes all augmentations)
|
||||
brainy cloud --connect YOUR_CUSTOMER_ID
|
||||
|
||||
# Check status
|
||||
brainy cloud --status
|
||||
|
||||
# Force sync
|
||||
brainy cloud --sync
|
||||
|
||||
# Open dashboard
|
||||
brainy cloud --dashboard
|
||||
```
|
||||
|
||||
### Working with Different Instances
|
||||
|
||||
#### Local Instance
|
||||
```bash
|
||||
# Add augmentation to local Brainy
|
||||
cd my-project
|
||||
brainy init
|
||||
brainy augment add neural-import
|
||||
```
|
||||
|
||||
#### Remote Hosted Instance
|
||||
```bash
|
||||
# Connect to remote Brainy server
|
||||
brainy config set server.url https://my-brainy-server.com
|
||||
brainy config set server.apiKey xxx
|
||||
|
||||
# Add augmentation to remote
|
||||
brainy augment add sentiment-analyzer --remote
|
||||
```
|
||||
|
||||
#### Brain Cloud Instance
|
||||
```bash
|
||||
# Everything is managed
|
||||
brainy cloud --connect CUSTOMER_ID
|
||||
# All augmentations automatically available
|
||||
```
|
||||
|
||||
### Interactive Mode Examples
|
||||
|
||||
#### Adding an Augmentation (No Arguments)
|
||||
```bash
|
||||
$ brainy augment add
|
||||
? What type of augmentation? ›
|
||||
Built-in (Free)
|
||||
Community (npm)
|
||||
Premium (Licensed)
|
||||
|
||||
? Select augmentation ›
|
||||
✓ neural-import (AI understanding)
|
||||
○ sentiment-analyzer (Emotion detection)
|
||||
○ translator (Multi-language)
|
||||
|
||||
? Configure now? (Y/n) ›
|
||||
```
|
||||
|
||||
#### Configuring an Augmentation
|
||||
```bash
|
||||
$ brainy augment config notion-sync
|
||||
? Notion API Token › secret_xxxxx
|
||||
? Sync Mode ›
|
||||
○ Read only
|
||||
● Bidirectional
|
||||
○ Write only
|
||||
|
||||
? Sync Interval (minutes) › 5
|
||||
✅ Configuration saved!
|
||||
```
|
||||
|
||||
### CLI Implementation in brainy.js
|
||||
|
||||
```javascript
|
||||
// brainy augment command structure
|
||||
program
|
||||
.command('augment [action] [name]')
|
||||
.description('Manage brain augmentations')
|
||||
.option('--type <type>', 'Augmentation type (sense|conduit|memory|etc)')
|
||||
.option('--premium', 'Premium augmentation')
|
||||
.option('--remote', 'Install on remote server')
|
||||
.action(async (action, name, options) => {
|
||||
if (!action) {
|
||||
// List all augmentations
|
||||
await listAugmentations()
|
||||
} else {
|
||||
switch(action) {
|
||||
case 'add':
|
||||
case 'install':
|
||||
await installAugmentation(name, options)
|
||||
break
|
||||
case 'remove':
|
||||
await removeAugmentation(name)
|
||||
break
|
||||
case 'config':
|
||||
await configureAugmentation(name, options)
|
||||
break
|
||||
case 'list':
|
||||
await listAugmentations()
|
||||
break
|
||||
case 'enable':
|
||||
await enableAugmentation(name)
|
||||
break
|
||||
case 'disable':
|
||||
await disableAugmentation(name)
|
||||
break
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# For premium augmentations
|
||||
export BRAINY_LICENSE_KEY=lic_xxxxxxxxxxxxx
|
||||
|
||||
# For Brain Cloud
|
||||
export BRAIN_CLOUD_KEY=bc_xxxxxxxxxxxxx
|
||||
export BRAIN_CLOUD_CUSTOMER_ID=cust_xxxxx
|
||||
|
||||
# For remote server
|
||||
export BRAINY_SERVER_URL=https://my-server.com
|
||||
export BRAINY_SERVER_KEY=xxx
|
||||
```
|
||||
|
||||
### Package.json Scripts
|
||||
|
||||
Add these to your project's package.json:
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"brain:init": "brainy init",
|
||||
"brain:augment": "brainy augment",
|
||||
"brain:add-sentiment": "brainy augment add brainy-sentiment",
|
||||
"brain:add-premium": "brainy augment add ai-memory --premium",
|
||||
"brain:cloud": "brainy cloud --connect"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Error Messages & Solutions
|
||||
|
||||
```bash
|
||||
# Missing license key
|
||||
❌ Premium augmentation requires license key
|
||||
💡 Set BRAINY_LICENSE_KEY or visit soulcraft.com
|
||||
|
||||
# Augmentation not found
|
||||
❌ Augmentation 'xyz' not found
|
||||
💡 Try: npm install brainy-xyz first
|
||||
|
||||
# Remote connection failed
|
||||
❌ Cannot connect to remote server
|
||||
💡 Check BRAINY_SERVER_URL and network connection
|
||||
|
||||
# Incompatible version
|
||||
❌ Augmentation requires Brainy v0.62+
|
||||
💡 Update: npm update @soulcraft/brainy
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
The CLI provides a unified interface for managing all augmentation types:
|
||||
- **Built-in**: Always available, just enable
|
||||
- **Community**: Install from npm, then add
|
||||
- **Premium**: Need license, install from @soulcraft/brain-cloud
|
||||
- **Brain Cloud**: Everything managed, just connect
|
||||
|
||||
The key is making it simple for beginners (interactive mode) while powerful for experts (direct commands).
|
||||
|
|
@ -1,128 +0,0 @@
|
|||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
We as members, contributors, and leaders pledge to make participation in our
|
||||
community a harassment-free experience for everyone, regardless of age, body
|
||||
size, visible or invisible disability, ethnicity, sex characteristics, gender
|
||||
identity and expression, level of experience, education, socio-economic status,
|
||||
nationality, personal appearance, race, religion, or sexual identity
|
||||
and orientation.
|
||||
|
||||
We pledge to act and interact in ways that contribute to an open, welcoming,
|
||||
diverse, inclusive, and healthy community.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to a positive environment for our
|
||||
community include:
|
||||
|
||||
* Demonstrating empathy and kindness toward other people
|
||||
* Being respectful of differing opinions, viewpoints, and experiences
|
||||
* Giving and gracefully accepting constructive feedback
|
||||
* Accepting responsibility and apologizing to those affected by our mistakes,
|
||||
and learning from the experience
|
||||
* Focusing on what is best not just for us as individuals, but for the
|
||||
overall community
|
||||
|
||||
Examples of unacceptable behavior include:
|
||||
|
||||
* The use of sexualized language or imagery, and sexual attention or
|
||||
advances of any kind
|
||||
* Trolling, insulting or derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or email
|
||||
address, without their explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Enforcement Responsibilities
|
||||
|
||||
Project maintainers are responsible for clarifying and enforcing our standards of
|
||||
acceptable behavior and will take appropriate and fair corrective action in
|
||||
response to any behavior that they deem inappropriate, threatening, offensive,
|
||||
or harmful.
|
||||
|
||||
Project maintainers have the right and responsibility to remove, edit, or reject
|
||||
comments, commits, code, wiki edits, issues, and other contributions that are
|
||||
not aligned with this Code of Conduct, and will communicate reasons for moderation
|
||||
decisions when appropriate.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all community spaces, and also applies when
|
||||
an individual is officially representing the community in public spaces.
|
||||
Examples of representing our community include using an official e-mail address,
|
||||
posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported to the project maintainers responsible for enforcement at
|
||||
conduct@soulcraft.com.
|
||||
All complaints will be reviewed and investigated promptly and fairly.
|
||||
|
||||
All project maintainers are obligated to respect the privacy and security of the
|
||||
reporter of any incident.
|
||||
|
||||
## Enforcement Guidelines
|
||||
|
||||
Project maintainers will follow these Community Impact Guidelines in determining
|
||||
the consequences for any action they deem in violation of this Code of Conduct:
|
||||
|
||||
### 1. Correction
|
||||
|
||||
**Community Impact**: Use of inappropriate language or other behavior deemed
|
||||
unprofessional or unwelcome in the community.
|
||||
|
||||
**Consequence**: A private, written warning from project maintainers, providing
|
||||
clarity around the nature of the violation and an explanation of why the
|
||||
behavior was inappropriate. A public apology may be requested.
|
||||
|
||||
### 2. Warning
|
||||
|
||||
**Community Impact**: A violation through a single incident or series
|
||||
of actions.
|
||||
|
||||
**Consequence**: A warning with consequences for continued behavior. No
|
||||
interaction with the people involved, including unsolicited interaction with
|
||||
those enforcing the Code of Conduct, for a specified period of time. This
|
||||
includes avoiding interactions in community spaces as well as external channels
|
||||
like social media. Violating these terms may lead to a temporary or
|
||||
permanent ban.
|
||||
|
||||
### 3. Temporary Ban
|
||||
|
||||
**Community Impact**: A serious violation of community standards, including
|
||||
sustained inappropriate behavior.
|
||||
|
||||
**Consequence**: A temporary ban from any sort of interaction or public
|
||||
communication with the community for a specified period of time. No public or
|
||||
private interaction with the people involved, including unsolicited interaction
|
||||
with those enforcing the Code of Conduct, is allowed during this period.
|
||||
Violating these terms may lead to a permanent ban.
|
||||
|
||||
### 4. Permanent Ban
|
||||
|
||||
**Community Impact**: Demonstrating a pattern of violation of community
|
||||
standards, including sustained inappropriate behavior, harassment of an
|
||||
individual, or aggression toward or disparagement of classes of individuals.
|
||||
|
||||
**Consequence**: A permanent ban from any sort of public interaction within
|
||||
the community.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
||||
version 2.0, available at
|
||||
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
|
||||
|
||||
Community Impact Guidelines were inspired by [Mozilla's code of conduct
|
||||
enforcement ladder](https://github.com/mozilla/diversity).
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
|
||||
For answers to common questions about this code of conduct, see the FAQ at
|
||||
https://www.contributor-covenant.org/faq. Translations are available at
|
||||
https://www.contributor-covenant.org/translations.
|
||||
112
CONFIGURATION.md
112
CONFIGURATION.md
|
|
@ -1,112 +0,0 @@
|
|||
# 🚫 No Configuration Required
|
||||
|
||||
Brainy follows a **zero-configuration philosophy**. We automatically detect and adapt to your environment.
|
||||
|
||||
## How It Works
|
||||
|
||||
### 🔐 API Keys & Secrets
|
||||
**Never put keys in files!** Brainy automatically detects credentials from:
|
||||
1. **System Keychains** - macOS Keychain, Windows Credential Manager, Linux Secret Service
|
||||
2. **Cloud IAM** - AWS IAM roles, Google Cloud Service Accounts, Azure Managed Identity
|
||||
3. **Secure Prompts** - Asked once, stored securely (never in plaintext)
|
||||
|
||||
```javascript
|
||||
// Just use brainy - it will find credentials securely
|
||||
const brain = new BrainyVectorDB()
|
||||
await brain.connect() // Auto-detects everything
|
||||
```
|
||||
|
||||
### 🌍 Environment Detection
|
||||
Brainy automatically detects:
|
||||
- Browser vs Node.js vs Serverless
|
||||
- Available memory and CPU cores
|
||||
- Storage backends (filesystem, S3, browser storage)
|
||||
- Network endpoints and services
|
||||
|
||||
### 🎯 Smart Defaults
|
||||
Based on your environment, Brainy automatically configures:
|
||||
- Memory limits (50% of available)
|
||||
- Cache sizes (adaptive to usage)
|
||||
- Batch sizes (based on latency)
|
||||
- Compression (when beneficial)
|
||||
|
||||
## When You DO Need Configuration
|
||||
|
||||
For advanced use cases only:
|
||||
|
||||
### Secure Credential Storage
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
security add-generic-password -s brainy-api -a user -w YOUR_KEY
|
||||
|
||||
# Linux
|
||||
secret-tool store --label="Brainy API" service brainy-api
|
||||
|
||||
# Windows
|
||||
cmdkey /generic:brainy-api /user:user /pass:YOUR_KEY
|
||||
```
|
||||
|
||||
### Programmatic Override
|
||||
|
||||
```javascript
|
||||
const brain = new BrainyVectorDB({
|
||||
// Only override if you know better than auto-detection
|
||||
storage: 's3://my-bucket',
|
||||
memory: '2GB'
|
||||
})
|
||||
```
|
||||
|
||||
## Philosophy
|
||||
|
||||
> "The best configuration is no configuration. Software should be smart enough to figure out what you need."
|
||||
|
||||
Brainy detects, adapts, and optimizes automatically. If you find yourself needing configuration, we've failed - please open an issue!
|
||||
|
||||
## Security Without .env
|
||||
|
||||
Traditional `.env` files are security theater:
|
||||
- ❌ Get committed accidentally
|
||||
- ❌ Sit in plaintext on disk
|
||||
- ❌ Copied around carelessly
|
||||
- ❌ Different for every environment
|
||||
|
||||
Brainy's approach:
|
||||
- ✅ Never store secrets in files
|
||||
- ✅ Use OS-level secure storage
|
||||
- ✅ Detect cloud IAM automatically
|
||||
- ✅ Adapt to environment dynamically
|
||||
|
||||
## Examples
|
||||
|
||||
### Local Development
|
||||
```javascript
|
||||
const brain = new BrainyVectorDB()
|
||||
// Detects: Node.js, 16GB RAM, filesystem storage
|
||||
// Auto-configures: 8GB memory limit, disk-backed cache
|
||||
```
|
||||
|
||||
### Browser
|
||||
```javascript
|
||||
const brain = new BrainyVectorDB()
|
||||
// Detects: Browser, 4GB available, IndexedDB
|
||||
// Auto-configures: 2GB limit, OPFS storage, WebWorker processing
|
||||
```
|
||||
|
||||
### AWS Lambda
|
||||
```javascript
|
||||
const brain = new BrainyVectorDB()
|
||||
// Detects: Lambda, 3GB RAM, S3 access
|
||||
// Auto-configures: 2.5GB limit, S3 storage, credential from IAM role
|
||||
```
|
||||
|
||||
### Cloudflare Worker
|
||||
```javascript
|
||||
const brain = new BrainyVectorDB()
|
||||
// Detects: Worker, 128MB limit, KV available
|
||||
// Auto-configures: 100MB limit, KV storage, edge caching
|
||||
```
|
||||
|
||||
## No Configuration Needed
|
||||
|
||||
Just use Brainy. It figures out the rest.
|
||||
357
CONTRIBUTING.md
357
CONTRIBUTING.md
|
|
@ -1,97 +1,298 @@
|
|||
<div align="center">
|
||||
<img src="./brainy.png" alt="Brainy Logo" width="200"/>
|
||||
|
||||
# Contributing to Brainy
|
||||
|
||||
</div>
|
||||
Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for contributing to the project.
|
||||
|
||||
Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for
|
||||
contributing to the project.
|
||||
## Code of Conduct
|
||||
|
||||
We welcome contributions of all kinds, including bug fixes, feature additions, documentation improvements, and more.
|
||||
By participating in this project, you agree to abide by our [Code of Conduct](CODE_OF_CONDUCT.md).
|
||||
By participating in this project, you agree to abide by our Code of Conduct:
|
||||
- Be respectful and inclusive
|
||||
- Welcome newcomers and help them get started
|
||||
- Focus on constructive criticism
|
||||
- Respect differing viewpoints and experiences
|
||||
|
||||
## Commit Message Guidelines
|
||||
## How to Contribute
|
||||
|
||||
When contributing to this project, please write clear and descriptive commit messages that explain the purpose of your
|
||||
changes. Good commit messages help maintainers understand your contributions and make the review process smoother.
|
||||
### Reporting Issues
|
||||
|
||||
### Best Practices
|
||||
Before creating an issue, please check existing issues to avoid duplicates.
|
||||
|
||||
- Keep the first line concise (ideally under 50 characters)
|
||||
- Use the imperative mood ("Add feature" not "Added feature")
|
||||
- Reference issues and pull requests where appropriate
|
||||
- When necessary, provide more detailed explanations in the commit body
|
||||
When creating an issue, include:
|
||||
- Clear, descriptive title
|
||||
- Detailed description of the problem
|
||||
- Steps to reproduce
|
||||
- Expected vs actual behavior
|
||||
- System information (OS, Node version, Brainy version)
|
||||
- Code examples if applicable
|
||||
|
||||
### Suggesting Features
|
||||
|
||||
Feature requests are welcome! Please provide:
|
||||
- Clear use case
|
||||
- Proposed API/interface
|
||||
- Examples of how it would work
|
||||
- Any potential challenges or considerations
|
||||
|
||||
### Pull Requests
|
||||
|
||||
#### Before Starting
|
||||
|
||||
1. Check existing issues and PRs
|
||||
2. Open an issue to discuss significant changes
|
||||
3. Fork the repository
|
||||
4. Create a feature branch from `main`
|
||||
|
||||
#### Development Setup
|
||||
|
||||
**Quick Setup (Recommended):**
|
||||
```bash
|
||||
# Clone your fork
|
||||
git clone https://github.com/your-username/brainy.git
|
||||
cd brainy
|
||||
|
||||
# Run setup script (installs all dependencies including Rust)
|
||||
./scripts/setup-dev.sh
|
||||
```
|
||||
|
||||
**Manual Setup:**
|
||||
```bash
|
||||
# Clone your fork
|
||||
git clone https://github.com/your-username/brainy.git
|
||||
cd brainy
|
||||
|
||||
# Install system dependencies (Ubuntu/Debian)
|
||||
sudo apt-get install -y build-essential pkg-config libssl-dev
|
||||
|
||||
# Install Rust (for WASM embedding engine)
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
source ~/.cargo/env
|
||||
rustup target add wasm32-unknown-unknown
|
||||
cargo install wasm-pack
|
||||
|
||||
# Install Node.js dependencies
|
||||
npm install
|
||||
|
||||
# Build Candle WASM embedding engine
|
||||
npm run build:candle
|
||||
|
||||
# Build TypeScript
|
||||
npm run build
|
||||
|
||||
# Run tests
|
||||
npm test
|
||||
```
|
||||
|
||||
#### Making Changes
|
||||
|
||||
1. **Follow the code style**
|
||||
- TypeScript for all source code
|
||||
- Clear variable and function names
|
||||
- Comments for complex logic
|
||||
- JSDoc for public APIs
|
||||
|
||||
2. **Write tests**
|
||||
- Add tests for new features
|
||||
- Update tests for changes
|
||||
- Ensure all tests pass
|
||||
|
||||
3. **Update documentation**
|
||||
- Update README if needed
|
||||
- Add/update API documentation
|
||||
- Include examples
|
||||
|
||||
#### Commit Guidelines
|
||||
|
||||
Follow conventional commits format:
|
||||
|
||||
```
|
||||
type(scope): description
|
||||
|
||||
[optional body]
|
||||
|
||||
[optional footer]
|
||||
```
|
||||
|
||||
Types:
|
||||
- `feat`: New feature
|
||||
- `fix`: Bug fix
|
||||
- `docs`: Documentation changes
|
||||
- `style`: Code style changes
|
||||
- `refactor`: Code refactoring
|
||||
- `perf`: Performance improvements
|
||||
- `test`: Test changes
|
||||
- `chore`: Build/tooling changes
|
||||
|
||||
Examples:
|
||||
```bash
|
||||
feat(triple): add graph traversal depth limit
|
||||
fix(storage): handle concurrent write conflicts
|
||||
docs(api): update search method documentation
|
||||
```
|
||||
|
||||
#### Submitting PR
|
||||
|
||||
1. Push to your fork
|
||||
2. Create PR against `main` branch
|
||||
3. Fill out PR template
|
||||
4. Ensure CI checks pass
|
||||
5. Wait for review
|
||||
|
||||
### Testing
|
||||
|
||||
#### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
npm test
|
||||
|
||||
# Run specific test file
|
||||
npm test tests/core.test.ts
|
||||
|
||||
# Run with coverage
|
||||
npm run test:coverage
|
||||
|
||||
# Watch mode
|
||||
npm run test:watch
|
||||
```
|
||||
|
||||
#### Writing Tests
|
||||
|
||||
```typescript
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { Brainy } from '../src'
|
||||
|
||||
describe('Feature Name', () => {
|
||||
it('should do something specific', async () => {
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Test implementation
|
||||
const result = await brain.search("test")
|
||||
|
||||
expect(result).toBeDefined()
|
||||
expect(result.length).toBeGreaterThan(0)
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
## Architecture Guidelines
|
||||
|
||||
### Adding New Features
|
||||
|
||||
1. **Check existing functionality**
|
||||
- Review `ARCHITECTURE.md`
|
||||
- Check if similar features exist
|
||||
- Consider if it should be an augmentation
|
||||
|
||||
2. **Design considerations**
|
||||
- Maintain backward compatibility
|
||||
- Consider performance impact
|
||||
- Think about all storage adapters
|
||||
- Plan for extensibility
|
||||
|
||||
3. **Implementation checklist**
|
||||
- [ ] Core functionality
|
||||
- [ ] Tests (unit and integration)
|
||||
- [ ] Documentation
|
||||
- [ ] TypeScript types
|
||||
- [ ] Examples
|
||||
- [ ] Performance benchmarks (if applicable)
|
||||
|
||||
### Creating Augmentations
|
||||
|
||||
Augmentations extend Brainy's functionality:
|
||||
|
||||
```typescript
|
||||
import { BrainyAugmentation } from '../types'
|
||||
|
||||
export class MyAugmentation extends BrainyAugmentation {
|
||||
name = 'MyAugmentation'
|
||||
|
||||
async onInit(brain: Brainy): Promise<void> {
|
||||
// Initialize augmentation
|
||||
}
|
||||
|
||||
async onAdd(item: any, brain: Brainy): Promise<any> {
|
||||
// Process before adding
|
||||
return item
|
||||
}
|
||||
|
||||
async onSearch(query: any, results: any[], brain: Brainy): Promise<any[]> {
|
||||
// Process search results
|
||||
return results
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Performance Considerations
|
||||
|
||||
- Use batch operations where possible
|
||||
- Implement caching strategically
|
||||
- Consider memory usage
|
||||
- Profile performance impacts
|
||||
- Add benchmarks for critical paths
|
||||
|
||||
## Documentation
|
||||
|
||||
### API Documentation
|
||||
|
||||
Use JSDoc for all public APIs:
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Searches for similar items using vector similarity
|
||||
* @param query - Search query (text or vector)
|
||||
* @param options - Search options
|
||||
* @returns Array of search results with scores
|
||||
* @example
|
||||
* ```typescript
|
||||
* const results = await brain.search("machine learning", { limit: 10 })
|
||||
* ```
|
||||
*/
|
||||
async search(query: string | Vector, options?: SearchOptions): Promise<SearchResult[]> {
|
||||
// Implementation
|
||||
}
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
```
|
||||
Add vector normalization option
|
||||
Fix distance calculation in HNSW search
|
||||
Update API documentation
|
||||
Add support for IndexedDB storage
|
||||
Change API parameter order
|
||||
Simplify vector comparison logic
|
||||
Update build dependencies
|
||||
Add examples for new features:
|
||||
|
||||
```typescript
|
||||
// examples/feature-name.ts
|
||||
import { Brainy } from 'brainy'
|
||||
|
||||
async function exampleUsage() {
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Show feature usage
|
||||
// Include comments explaining what's happening
|
||||
// Handle errors appropriately
|
||||
}
|
||||
|
||||
exampleUsage().catch(console.error)
|
||||
```
|
||||
|
||||
## Pull Request Process
|
||||
## Release Process
|
||||
|
||||
1. Ensure your code follows the project's coding standards
|
||||
2. Update the documentation if necessary
|
||||
3. Use conventional commit messages in your PR
|
||||
4. Your PR will be reviewed by maintainers and merged if approved
|
||||
1. **Version bump**: Follow semantic versioning
|
||||
2. **Update CHANGELOG**: Document all changes
|
||||
3. **Run tests**: Ensure all tests pass
|
||||
4. **Build**: Generate distribution files
|
||||
5. **Tag**: Create git tag for version
|
||||
6. **Publish**: Release to npm
|
||||
|
||||
## Development Setup
|
||||
## Getting Help
|
||||
|
||||
1. Fork and clone the repository
|
||||
2. Install dependencies: `npm install`
|
||||
3. Build the project: `npm run build`
|
||||
- **Discord**: Join our community
|
||||
- **Issues**: Ask questions on GitHub
|
||||
- **Discussions**: Share ideas and get feedback
|
||||
|
||||
## Code Style
|
||||
## Recognition
|
||||
|
||||
This project uses ESLint and Prettier for code formatting and style checking. The configuration can be found in the `package.json` file. Please ensure your code follows these standards:
|
||||
Contributors will be recognized in:
|
||||
- CHANGELOG.md for their contributions
|
||||
- README.md contributors section
|
||||
- GitHub contributors page
|
||||
|
||||
- Use 2 spaces for indentation
|
||||
- Use single quotes for strings
|
||||
- No semicolons
|
||||
- Trailing commas are not used
|
||||
- Maximum line length is 80 characters
|
||||
|
||||
You can check your code style by running:
|
||||
```bash
|
||||
npm run check:style
|
||||
```
|
||||
|
||||
This will run all code style checks, including a specific check for semicolons.
|
||||
|
||||
You can also run individual checks:
|
||||
|
||||
```bash
|
||||
npm run lint # Run ESLint to check for code issues
|
||||
npm run lint:fix # Automatically fix linting issues
|
||||
npm run format # Format your code with Prettier
|
||||
npm run check-format # Check if your code is properly formatted
|
||||
```
|
||||
|
||||
## Branching Strategy
|
||||
|
||||
- `main` - The main branch contains the latest stable release
|
||||
- `develop` - The development branch contains the latest development changes
|
||||
- Feature branches - Create a branch from `develop` for your feature or fix
|
||||
|
||||
When working on a new feature or fix:
|
||||
1. Create a new branch from `develop` with a descriptive name (e.g., `feature/add-vector-normalization` or `fix/distance-calculation`)
|
||||
2. Make your changes in that branch
|
||||
3. Submit a pull request to merge your branch into `develop`
|
||||
|
||||
## Issue Reporting
|
||||
|
||||
Before submitting a new issue, please search existing issues to avoid duplicates.
|
||||
|
||||
- For bugs, use the bug report template
|
||||
- For feature requests, use the feature request template
|
||||
- Be as detailed as possible in your description
|
||||
- Include code examples, error messages, and screenshots if applicable
|
||||
|
||||
Thank you for contributing to Brainy!
|
||||
Thank you for contributing to Brainy! 🧠
|
||||
72
Dockerfile
Normal file
72
Dockerfile
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
# Multi-stage Dockerfile for Brainy
|
||||
# Optimized for production deployment with minimal image size
|
||||
|
||||
# Stage 1: Build stage
|
||||
FROM node:22-alpine AS builder
|
||||
|
||||
# Install build dependencies
|
||||
RUN apk add --no-cache python3 make g++
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy package files
|
||||
COPY package*.json ./
|
||||
|
||||
# Install all dependencies (including dev dependencies for building)
|
||||
RUN npm ci
|
||||
|
||||
# Copy source code
|
||||
COPY . .
|
||||
|
||||
# Build the TypeScript code
|
||||
RUN npm run build
|
||||
|
||||
# Remove dev dependencies and only keep production ones
|
||||
RUN npm prune --production
|
||||
|
||||
# Stage 2: Production stage
|
||||
FROM node:22-alpine
|
||||
|
||||
# Install production dependencies only
|
||||
RUN apk add --no-cache tini
|
||||
|
||||
# Create non-root user for security
|
||||
RUN addgroup -g 1001 -S nodejs && \
|
||||
adduser -S nodejs -u 1001
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy package files
|
||||
COPY package*.json ./
|
||||
|
||||
# Copy built application from builder stage
|
||||
COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules
|
||||
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
|
||||
|
||||
# Copy necessary static files
|
||||
COPY --chown=nodejs:nodejs README.md LICENSE ./
|
||||
|
||||
# Create data directory for file-based storage
|
||||
RUN mkdir -p /app/data && chown -R nodejs:nodejs /app/data
|
||||
|
||||
# Switch to non-root user
|
||||
USER nodejs
|
||||
|
||||
# Expose default port (can be overridden)
|
||||
EXPOSE 3000
|
||||
|
||||
# Set environment variables for production
|
||||
ENV NODE_ENV=production
|
||||
ENV BRAINY_STORAGE_PATH=/app/data
|
||||
|
||||
# Health check endpoint
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD node -e "require('http').get('http://localhost:3000/health', (r) => r.statusCode === 200 ? process.exit(0) : process.exit(1))"
|
||||
|
||||
# Use tini to handle signals properly
|
||||
ENTRYPOINT ["/sbin/tini", "--"]
|
||||
|
||||
# Default command (can be overridden)
|
||||
CMD ["node", "dist/index.js"]
|
||||
4
LICENSE
4
LICENSE
|
|
@ -1,6 +1,6 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2023 Soulcraft Research
|
||||
Copyright (c) 2024 Brainy Data Contributors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
|
|
@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
SOFTWARE.
|
||||
|
|
@ -1,276 +0,0 @@
|
|||
# Metadata Filtering Performance Optimization Proposal
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Current metadata filtering has a **379-386% performance overhead** due to a fixed 3x ef multiplier in HNSW search, regardless of filter selectivity. This makes filtered searches 3-4x slower than necessary.
|
||||
|
||||
## Proposed Solution: Smart Filtering Strategy
|
||||
|
||||
### 1. Dynamic ef Multiplier Based on Selectivity
|
||||
|
||||
**Implementation Location**: `/src/brainyData.ts` in search method around line 2164
|
||||
|
||||
**Current Code**:
|
||||
```typescript
|
||||
// Fixed 3x multiplier regardless of filter selectivity
|
||||
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)
|
||||
```
|
||||
|
||||
**Proposed Enhancement**:
|
||||
```typescript
|
||||
interface FilterSelectivity {
|
||||
candidateCount: number
|
||||
totalCount: number
|
||||
selectivity: number // 0.0 - 1.0
|
||||
strategy: 'index-first' | 'hnsw-filter' | 'post-filter'
|
||||
}
|
||||
|
||||
async calculateFilterStrategy(filter: MetadataFilter): Promise<FilterSelectivity> {
|
||||
const totalCount = this.index.getSize()
|
||||
|
||||
if (!this.metadataIndex || totalCount === 0) {
|
||||
return { candidateCount: totalCount, totalCount, selectivity: 1.0, strategy: 'post-filter' }
|
||||
}
|
||||
|
||||
const candidateIds = await this.metadataIndex.getIdsForCriteria(filter)
|
||||
const selectivity = candidateIds.length / totalCount
|
||||
|
||||
let strategy: FilterSelectivity['strategy']
|
||||
if (selectivity <= 0.05) {
|
||||
strategy = 'index-first' // < 5% match: search only candidates
|
||||
} else if (selectivity <= 0.3) {
|
||||
strategy = 'hnsw-filter' // 5-30% match: HNSW with smart ef
|
||||
} else {
|
||||
strategy = 'post-filter' // > 30% match: search all, filter after
|
||||
}
|
||||
|
||||
return { candidateCount: candidateIds.length, totalCount, selectivity, strategy }
|
||||
}
|
||||
|
||||
// Dynamic ef calculation
|
||||
calculateSmartEf(baseEf: number, k: number, selectivity: number, strategy: string): number {
|
||||
switch (strategy) {
|
||||
case 'index-first':
|
||||
return Math.max(k * 1.2, 20) // Minimal ef for candidate-only search
|
||||
|
||||
case 'hnsw-filter':
|
||||
// Dynamic multiplier: high selectivity = lower multiplier
|
||||
const multiplier = 1.2 + (selectivity * 1.8) // 1.2x - 3.0x range
|
||||
return Math.max(baseEf * multiplier, k * multiplier)
|
||||
|
||||
case 'post-filter':
|
||||
return Math.max(baseEf, k) // No filtering overhead
|
||||
|
||||
default:
|
||||
return Math.max(baseEf, k)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Index-First Search Implementation
|
||||
|
||||
**New Method**: Add to `/src/hnsw/hnswIndex.ts`
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Search only within a pre-filtered set of candidates
|
||||
* Optimized for high-selectivity metadata filters
|
||||
*/
|
||||
async searchCandidatesOnly(
|
||||
queryVector: Vector,
|
||||
candidateIds: string[],
|
||||
k: number
|
||||
): Promise<SearchResult<HNSWNoun>[]> {
|
||||
if (candidateIds.length === 0) return []
|
||||
|
||||
const candidates: { noun: HNSWNoun; distance: number }[] = []
|
||||
|
||||
// Calculate distances only for candidate nouns
|
||||
for (const id of candidateIds) {
|
||||
const noun = this.nouns.get(id)
|
||||
if (noun) {
|
||||
const distance = this.distanceFunction(queryVector, noun.vector)
|
||||
candidates.push({ noun, distance })
|
||||
}
|
||||
}
|
||||
|
||||
// Sort by distance and return top k
|
||||
candidates.sort((a, b) => a.distance - b.distance)
|
||||
|
||||
return candidates.slice(0, k).map(({ noun, distance }) => ({
|
||||
id: noun.id,
|
||||
score: 1 / (1 + distance),
|
||||
vector: noun.vector,
|
||||
metadata: noun.metadata
|
||||
}))
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Smart Search Strategy Selection
|
||||
|
||||
**Enhanced Search Flow** in `/src/brainyData.ts`:
|
||||
|
||||
```typescript
|
||||
async search<T>(query: string, k: number, options?: SearchOptions<T>) {
|
||||
// ... existing code for embedding generation ...
|
||||
|
||||
if (hasMetadataFilter && this.metadataIndex) {
|
||||
// Calculate optimal strategy
|
||||
const filterStrategy = await this.calculateFilterStrategy(options.metadata)
|
||||
|
||||
console.log(`Filter strategy: ${filterStrategy.strategy} (${(filterStrategy.selectivity * 100).toFixed(1)}% selectivity)`)
|
||||
|
||||
switch (filterStrategy.strategy) {
|
||||
case 'index-first':
|
||||
// Direct candidate search - fastest for high selectivity
|
||||
const candidateIds = await this.metadataIndex.getIdsForCriteria(options.metadata)
|
||||
return this.index.searchCandidatesOnly(queryVector, candidateIds, k)
|
||||
|
||||
case 'hnsw-filter':
|
||||
// Smart HNSW search with optimized ef
|
||||
const smartEf = this.calculateSmartEf(
|
||||
this.config.hnsw?.efSearch || 50,
|
||||
k,
|
||||
filterStrategy.selectivity,
|
||||
filterStrategy.strategy
|
||||
)
|
||||
return this.index.search(queryVector, k, undefined, smartEf)
|
||||
|
||||
case 'post-filter':
|
||||
// Search all, then filter (best for low selectivity)
|
||||
const allResults = await this.index.search(queryVector, k * 3) // Get more results
|
||||
return this.applyMetadataFilter(allResults, options.metadata).slice(0, k)
|
||||
}
|
||||
}
|
||||
|
||||
// ... existing non-filtered search code ...
|
||||
}
|
||||
```
|
||||
|
||||
## Expected Performance Improvements
|
||||
|
||||
### High Selectivity Filters (< 5% match rate)
|
||||
- **Current**: 150ms with 3x ef multiplier
|
||||
- **Optimized**: ~15ms with direct candidate search
|
||||
- **Improvement**: **90% faster**
|
||||
|
||||
### Medium Selectivity Filters (5-30% match rate)
|
||||
- **Current**: 150ms with fixed 3x multiplier
|
||||
- **Optimized**: ~45-75ms with dynamic multiplier
|
||||
- **Improvement**: **50-70% faster**
|
||||
|
||||
### Low Selectivity Filters (> 30% match rate)
|
||||
- **Current**: 150ms with unnecessary filtering overhead
|
||||
- **Optimized**: ~32ms with post-filtering
|
||||
- **Improvement**: **80% faster**
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Core Infrastructure (1-2 days)
|
||||
1. Add selectivity calculation methods to `BrainyData`
|
||||
2. Implement dynamic ef calculation logic
|
||||
3. Add configuration options for strategy thresholds
|
||||
|
||||
### Phase 2: Search Strategy Implementation (2-3 days)
|
||||
1. Implement `searchCandidatesOnly` in `HNSWIndex`
|
||||
2. Add smart strategy selection to search method
|
||||
3. Implement post-filtering strategy
|
||||
|
||||
### Phase 3: Configuration & Tuning (1 day)
|
||||
1. Add configuration options for selectivity thresholds
|
||||
2. Add performance monitoring and logging
|
||||
3. Update documentation with optimization guidance
|
||||
|
||||
### Phase 4: Testing & Validation (1-2 days)
|
||||
1. Update performance tests with new benchmarks
|
||||
2. Add test cases for different selectivity scenarios
|
||||
3. Validate backward compatibility
|
||||
|
||||
## Configuration Options
|
||||
|
||||
```typescript
|
||||
interface MetadataIndexConfig {
|
||||
// Existing options...
|
||||
|
||||
// New optimization options
|
||||
selectivityThresholds?: {
|
||||
indexFirst: number // Default: 0.05 (5%)
|
||||
hnswFilter: number // Default: 0.3 (30%)
|
||||
}
|
||||
|
||||
dynamicEf?: {
|
||||
enabled: boolean // Default: true
|
||||
minMultiplier: number // Default: 1.2
|
||||
maxMultiplier: number // Default: 3.0
|
||||
}
|
||||
|
||||
performanceLogging?: {
|
||||
enabled: boolean // Default: false
|
||||
logSelectivity: boolean // Default: false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
- All existing APIs remain unchanged
|
||||
- Default behavior maintains current functionality if optimizations disabled
|
||||
- Gradual rollout possible with feature flags
|
||||
- Performance improvements are opt-in via configuration
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
```typescript
|
||||
// New performance tests to add
|
||||
describe('Metadata Search Optimization', () => {
|
||||
it('should use index-first for high selectivity filters', async () => {
|
||||
// Test with filter matching <5% of items
|
||||
// Verify strategy selection and performance improvement
|
||||
})
|
||||
|
||||
it('should use dynamic ef for medium selectivity filters', async () => {
|
||||
// Test with filter matching 5-30% of items
|
||||
// Verify ef calculation and performance improvement
|
||||
})
|
||||
|
||||
it('should use post-filtering for low selectivity filters', async () => {
|
||||
// Test with filter matching >30% of items
|
||||
// Verify strategy selection and performance improvement
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
**Low Risk Changes**:
|
||||
- Configuration additions
|
||||
- New method implementations
|
||||
- Performance logging
|
||||
|
||||
**Medium Risk Changes**:
|
||||
- Strategy selection logic
|
||||
- Dynamic ef calculation
|
||||
|
||||
**Mitigation Strategies**:
|
||||
- Feature flags for gradual rollout
|
||||
- Extensive testing with different dataset sizes
|
||||
- Fallback to original behavior on errors
|
||||
- Performance monitoring to detect regressions
|
||||
|
||||
## Success Metrics
|
||||
|
||||
1. **Performance Improvement**: 50-90% reduction in filtered search time
|
||||
2. **Selectivity Accuracy**: Strategy selection matches actual selectivity
|
||||
3. **Resource Usage**: No significant increase in memory or CPU
|
||||
4. **Compatibility**: All existing tests continue to pass
|
||||
5. **User Experience**: Improved search response times in real applications
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
1. **Query Pattern Analysis**: Learn from search patterns to optimize strategy selection
|
||||
2. **Index Warmup**: Pre-calculate common filter combinations
|
||||
3. **Parallel Candidate Search**: Multi-threaded candidate evaluation
|
||||
4. **Index Compression**: Reduce storage overhead for large datasets
|
||||
5. **Adaptive Thresholds**: Machine learning-based threshold optimization
|
||||
|
||||
This optimization will significantly improve the metadata filtering system's performance while maintaining the robust architecture and backward compatibility.
|
||||
|
|
@ -1,227 +0,0 @@
|
|||
# Brainy Metadata Filtering Performance Analysis
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The metadata filtering system in Brainy provides powerful search capabilities with indexing optimizations, but comes with specific performance trade-offs. This analysis examines the performance impact across five key dimensions: index build time, storage overhead, search performance, memory usage, and write operations.
|
||||
|
||||
## Key Findings
|
||||
|
||||
### 1. Index Build Time Impact
|
||||
|
||||
**Performance Overhead: 8.8%**
|
||||
|
||||
- **Without indexing**: 138.24ms per item for 100 items
|
||||
- **With indexing**: 150.46ms per item for 100 items
|
||||
- **Overhead**: 8.8% slower initialization when metadata indexing is enabled
|
||||
|
||||
The overhead is primarily due to:
|
||||
- Building inverted indexes for each metadata field
|
||||
- Writing index entries to storage
|
||||
- Initial cache population
|
||||
|
||||
### 2. Index Storage Overhead
|
||||
|
||||
**Storage Overhead: 26 bytes per item**
|
||||
|
||||
From test with 100 items containing 6 indexed fields each:
|
||||
- **Total index entries**: 26 unique field-value combinations
|
||||
- **Total indexed IDs**: 600 (100 items × 6 fields average)
|
||||
- **Index size**: 2,600 bytes (26 bytes per item)
|
||||
- **Storage efficiency**: Very compact representation
|
||||
|
||||
**Index Structure:**
|
||||
- **Storage Location**: `_system` directory with `__metadata_index__` prefix
|
||||
- **Index Format**: Inverted indexes mapping `field:value` → `Set<id>`
|
||||
- **Cached Fields**: department, level, location, salary, remote, active (excludes id, createdAt, updatedAt by default)
|
||||
|
||||
### 3. Search Performance Analysis
|
||||
|
||||
**Critical Finding: 379-386% overhead when filtering is enabled**
|
||||
|
||||
#### Performance Breakdown:
|
||||
- **No filtering**: 31.46ms average
|
||||
- **Simple filter** (department='Engineering'): 150.82ms average (**379.4% overhead**)
|
||||
- **Complex filter** (multiple conditions): 153.05ms average (**386.5% overhead**)
|
||||
|
||||
#### Root Cause Analysis:
|
||||
The performance overhead stems from the **3x ef multiplier** implementation in `/src/hnsw/hnswIndex.ts:377`:
|
||||
|
||||
```typescript
|
||||
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)
|
||||
```
|
||||
|
||||
This multiplier compensates for potential filtering but significantly increases the search space:
|
||||
- **Default efSearch**: Typically 50-100
|
||||
- **With filtering**: 150-300+ candidates evaluated
|
||||
- **Impact**: 3x more vector calculations and distance computations
|
||||
|
||||
#### Search Process with Metadata Filtering:
|
||||
|
||||
1. **Pre-filtering Phase**:
|
||||
- Query metadata index for candidate IDs: ~0.1ms
|
||||
- Create filtered ID set: ~0.05ms
|
||||
|
||||
2. **HNSW Search Phase** (Major bottleneck):
|
||||
- Search with 3x larger ef parameter: ~150ms
|
||||
- Evaluate 3x more vector similarities
|
||||
- Apply metadata filter to results: ~0.2ms
|
||||
|
||||
3. **Post-processing**:
|
||||
- Re-rank and limit results: ~0.1ms
|
||||
|
||||
### 4. Memory Usage Impact
|
||||
|
||||
**Memory Overhead: ~26 bytes per indexed item**
|
||||
|
||||
The metadata index system uses:
|
||||
- **Index Cache**: In-memory Map storing field-value → ID mappings
|
||||
- **LRU Eviction**: Automatic cleanup of unused entries
|
||||
- **Dirty Tracking**: Efficient batch writes to storage
|
||||
- **Memory Efficiency**: Minimal overhead per item
|
||||
|
||||
### 5. Write Performance Impact
|
||||
|
||||
**Excellent Write Performance with Indexing**
|
||||
|
||||
- **ADD**: 154.35ms per item (includes embedding generation)
|
||||
- **UPDATE**: 0.03ms per item (**very fast**)
|
||||
- **DELETE**: 0.10ms per item (**very fast**)
|
||||
|
||||
Write operations benefit from:
|
||||
- **Efficient Index Updates**: O(1) hash lookups
|
||||
- **Batch Operations**: Dirty tracking for efficient storage writes
|
||||
- **Automatic Cleanup**: Empty index entries are automatically removed
|
||||
|
||||
## Performance Recommendations
|
||||
|
||||
### 1. Immediate Optimizations
|
||||
|
||||
#### A. Dynamic ef Multiplier
|
||||
**Current Issue**: Fixed 3x multiplier regardless of filter selectivity
|
||||
|
||||
**Recommendation**: Implement dynamic ef calculation based on index statistics:
|
||||
|
||||
```typescript
|
||||
const estimateSelectivity = async (filter: MetadataFilter): Promise<number> => {
|
||||
if (!this.metadataIndex) return 1.0
|
||||
|
||||
const totalItems = this.index.getSize()
|
||||
const candidateIds = await this.metadataIndex.getIdsForCriteria(filter)
|
||||
return candidateIds.length / totalItems
|
||||
}
|
||||
|
||||
// Use in search:
|
||||
const selectivity = await this.estimateSelectivity(filter)
|
||||
const multiplier = selectivity < 0.1 ? 2.0 : selectivity < 0.3 ? 1.5 : 1.2
|
||||
const ef = Math.max(this.config.efSearch * multiplier, k * multiplier)
|
||||
```
|
||||
|
||||
**Expected Impact**: 50-70% reduction in search overhead for high-selectivity filters
|
||||
|
||||
#### B. Index-First Search Strategy
|
||||
**Current**: Always search full HNSW then filter
|
||||
**Proposed**: Pre-filter significantly when index suggests high selectivity
|
||||
|
||||
```typescript
|
||||
if (candidateIds.length < totalItems * 0.1) {
|
||||
// High selectivity: search only candidate vectors
|
||||
return this.searchCandidatesOnly(candidateIds, queryVector, k)
|
||||
} else {
|
||||
// Low selectivity: use current HNSW + filter approach
|
||||
return this.searchWithFilter(queryVector, k, filter)
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Configuration Recommendations
|
||||
|
||||
#### A. Selective Field Indexing
|
||||
**Default**: Index all metadata fields (can be wasteful)
|
||||
**Recommended**: Configure specific fields for indexing
|
||||
|
||||
```typescript
|
||||
metadataIndex: {
|
||||
indexedFields: ['department', 'level', 'location'], // Only frequently filtered fields
|
||||
excludeFields: ['id', 'createdAt', 'updatedAt', 'description'],
|
||||
autoOptimize: true
|
||||
}
|
||||
```
|
||||
|
||||
#### B. Index Size Limits
|
||||
Configure appropriate limits based on use case:
|
||||
|
||||
```typescript
|
||||
metadataIndex: {
|
||||
maxIndexSize: 50000, // Increase for large datasets
|
||||
rebuildThreshold: 0.05, // More aggressive rebuilding
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Use Case Specific Optimizations
|
||||
|
||||
#### High-Selectivity Scenarios (< 10% match rate)
|
||||
- Use smaller ef multiplier (1.2-1.5x)
|
||||
- Enable aggressive index pre-filtering
|
||||
- Consider dedicated filtered search paths
|
||||
|
||||
#### Low-Selectivity Scenarios (> 50% match rate)
|
||||
- Disable metadata indexing for better performance
|
||||
- Use post-filtering instead of HNSW filtering
|
||||
- Consider vector-first search strategies
|
||||
|
||||
#### Mixed Workloads
|
||||
- Implement adaptive ef multipliers
|
||||
- Use query pattern analysis
|
||||
- Enable smart caching strategies
|
||||
|
||||
## Implementation Architecture
|
||||
|
||||
### MetadataIndexManager Design
|
||||
**Strengths**:
|
||||
- ✅ Efficient inverted index structure
|
||||
- ✅ LRU cache for frequently accessed entries
|
||||
- ✅ Automatic index maintenance
|
||||
- ✅ Storage in separate `_system` directory
|
||||
- ✅ Backward compatibility with non-indexed searches
|
||||
|
||||
**Areas for Improvement**:
|
||||
- ⚠️ Fixed 3x ef multiplier regardless of selectivity
|
||||
- ⚠️ No query optimization based on index statistics
|
||||
- ⚠️ Limited index compression for large datasets
|
||||
|
||||
### Storage Integration
|
||||
**Well-designed storage patterns**:
|
||||
- Index entries stored with `__metadata_index__` prefix
|
||||
- Efficient serialization of Sets to Arrays
|
||||
- Proper cleanup of empty index entries
|
||||
- Flush batching for write performance
|
||||
|
||||
## Benchmark Summary
|
||||
|
||||
| Operation | Without Index | With Index | Overhead |
|
||||
|-----------|--------------|------------|----------|
|
||||
| **Initialization** | 138.24ms/item | 150.46ms/item | +8.8% |
|
||||
| **Simple Search** | 31.46ms | 150.82ms | +379.4% |
|
||||
| **Complex Search** | 31.46ms | 153.05ms | +386.5% |
|
||||
| **Add Operations** | N/A | 154.35ms/item | (includes embedding) |
|
||||
| **Update Operations** | N/A | 0.03ms/item | (very fast) |
|
||||
| **Delete Operations** | N/A | 0.10ms/item | (very fast) |
|
||||
| **Storage Overhead** | 0 bytes | 26 bytes/item | minimal |
|
||||
|
||||
## Conclusion
|
||||
|
||||
The metadata filtering system provides powerful search capabilities with reasonable storage and initialization overhead. However, the **current search implementation has significant performance bottlenecks** due to the fixed 3x ef multiplier.
|
||||
|
||||
**Key Takeaways**:
|
||||
1. **Index building** is efficient with only 8.8% overhead
|
||||
2. **Storage overhead** is minimal at 26 bytes per item
|
||||
3. **Write operations** are very fast due to efficient index updates
|
||||
4. **Search performance** needs optimization - currently 300-400% slower when filtering
|
||||
5. **Memory usage** is reasonable and well-managed
|
||||
|
||||
**Priority Action Items**:
|
||||
1. Implement dynamic ef multiplier based on filter selectivity
|
||||
2. Add index-first search for high-selectivity filters
|
||||
3. Provide configuration guidance for different use cases
|
||||
4. Consider query pattern analysis for automatic optimization
|
||||
|
||||
The system architecture is solid and well-designed; the performance issues are primarily in the search optimization logic and can be addressed with the recommended improvements.
|
||||
275
MIGRATION.md
275
MIGRATION.md
|
|
@ -1,275 +0,0 @@
|
|||
# Migration Guide: Brainy 0.x → 1.0
|
||||
|
||||
This guide will help you upgrade from Brainy 0.x to 1.0.0-rc.1. While there are breaking changes, most functionality has been simplified and improved.
|
||||
|
||||
## 🎯 **Quick Migration Checklist**
|
||||
|
||||
- [ ] Update package: `npm install @soulcraft/brainy@rc`
|
||||
- [ ] Update CLI commands (see mapping below)
|
||||
- [ ] Replace `addSmart()` with `add()`
|
||||
- [ ] Update any pipeline imports
|
||||
- [ ] Test functionality with new API
|
||||
- [ ] Enable new features (encryption, soft delete)
|
||||
|
||||
## 📦 **Package Installation**
|
||||
|
||||
```bash
|
||||
# Install the release candidate
|
||||
npm install @soulcraft/brainy@rc
|
||||
|
||||
# Or with yarn
|
||||
yarn add @soulcraft/brainy@rc
|
||||
```
|
||||
|
||||
## 🔄 **API Method Changes**
|
||||
|
||||
### Core Data Operations
|
||||
|
||||
| **0.x Method** | **1.0 Method** | **Notes** |
|
||||
|----------------|----------------|-----------|
|
||||
| `addSmart(data, metadata)` | `add(data, metadata)` | Smart by default now |
|
||||
| `add(data, metadata)` | `add(data, metadata, { process: 'literal' })` | Use literal option for old behavior |
|
||||
| `searchSimilar(query, k)` | `search(query, k)` | Same functionality, cleaner name |
|
||||
| `searchByMetadata(filter)` | `search('', k, { metadata: filter })` | Unified search interface |
|
||||
| `searchConnected(id, k)` | `search('', k, { searchConnectedNouns: true })` | Part of unified search |
|
||||
|
||||
### NEW Methods in 1.0
|
||||
|
||||
```javascript
|
||||
// New methods available
|
||||
await brainy.import([data1, data2, data3]) // Bulk import
|
||||
await brainy.addNoun(data, NounType.Person) // Explicit typing
|
||||
await brainy.update(id, newData, newMetadata) // Smart updates
|
||||
await brainy.delete(id) // Soft delete by default
|
||||
await brainy.delete(id, { soft: false }) // Hard delete if needed
|
||||
```
|
||||
|
||||
## 🖥️ **CLI Command Changes**
|
||||
|
||||
### Command Mapping
|
||||
|
||||
| **0.x Command** | **1.0 Command** | **Notes** |
|
||||
|-----------------|-----------------|-----------|
|
||||
| `brainy add-smart "data"` | `brainy add "data"` | Smart by default |
|
||||
| `brainy add-literal "data"` | `brainy add "data" --literal` | Use literal flag |
|
||||
| `brainy search-similar "query"` | `brainy search "query"` | Cleaner naming |
|
||||
| `brainy search-metadata '{"type":"person"}'` | `brainy search "" --filter '{"type":"person"}'` | Unified search |
|
||||
| `brainy list-stats` | `brainy status` | Enhanced status command |
|
||||
| Multiple config commands | `brainy config <action>` | Unified config management |
|
||||
|
||||
### NEW CLI Commands
|
||||
|
||||
```bash
|
||||
brainy init --encryption # Initialize with encryption
|
||||
brainy update <id> --data "new" # Update existing data
|
||||
brainy delete <id> # Soft delete (default)
|
||||
brainy delete <id> --hard # Hard delete
|
||||
brainy import data.json # Bulk import
|
||||
```
|
||||
|
||||
### Removed CLI Commands
|
||||
|
||||
These commands have been consolidated:
|
||||
- `brainy add-smart` → `brainy add`
|
||||
- `brainy add-literal` → `brainy add --literal`
|
||||
- `brainy search-similar` → `brainy search`
|
||||
- `brainy search-metadata` → `brainy search --filter`
|
||||
- Various config commands → `brainy config`
|
||||
|
||||
## 🏗️ **Architecture Changes**
|
||||
|
||||
### Pipeline/Cortex Changes
|
||||
|
||||
```javascript
|
||||
// OLD - Multiple pipeline classes
|
||||
import {
|
||||
SequentialPipeline,
|
||||
ParallelPipeline,
|
||||
StreamlinedPipeline
|
||||
} from '@soulcraft/brainy'
|
||||
|
||||
// NEW - One unified Cortex class
|
||||
import { Pipeline, Cortex } from '@soulcraft/brainy'
|
||||
|
||||
// Both Pipeline and Cortex are the same class
|
||||
const pipeline = new Pipeline() // or new Cortex()
|
||||
```
|
||||
|
||||
### Import Path Changes
|
||||
|
||||
Most imports remain the same, but some internal imports may have changed:
|
||||
|
||||
```javascript
|
||||
// These should still work
|
||||
import { BrainyData, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
// Check these if you were using internal APIs
|
||||
// (Most users won't need to change anything)
|
||||
```
|
||||
|
||||
## 🔐 **New Encryption Features**
|
||||
|
||||
1.0 introduces comprehensive encryption support:
|
||||
|
||||
```javascript
|
||||
// Initialize with encryption
|
||||
const brainy = new BrainyData()
|
||||
await brainy.init()
|
||||
|
||||
// Encrypt configuration
|
||||
await brainy.setConfig('api-key', 'secret-key', { encrypt: true })
|
||||
|
||||
// Encrypt individual data items
|
||||
await brainy.add("sensitive data", {}, { encrypt: true })
|
||||
|
||||
// CLI encryption
|
||||
brainy init --encryption
|
||||
brainy add "sensitive data" --encrypt
|
||||
```
|
||||
|
||||
## 📊 **Soft Delete by Default**
|
||||
|
||||
The new `delete()` method uses soft delete by default:
|
||||
|
||||
```javascript
|
||||
// Soft delete (preserves indexes, better performance)
|
||||
await brainy.delete(id) // Default behavior
|
||||
|
||||
// Hard delete (removes from indexes)
|
||||
await brainy.delete(id, { soft: false })
|
||||
|
||||
// Cascade delete (deletes related verbs)
|
||||
await brainy.delete(id, { cascade: true })
|
||||
```
|
||||
|
||||
Search automatically excludes soft-deleted items.
|
||||
|
||||
## 🐳 **Container Deployment**
|
||||
|
||||
New container-optimized features:
|
||||
|
||||
```javascript
|
||||
// Preload models for containers
|
||||
await BrainyData.preloadModel({
|
||||
model: 'Xenova/all-MiniLM-L6-v2',
|
||||
cacheDir: './models'
|
||||
})
|
||||
|
||||
// Container-optimized initialization
|
||||
const brainy = await BrainyData.warmup({
|
||||
storage: { forceMemoryStorage: true }
|
||||
}, {
|
||||
preloadModel: true
|
||||
})
|
||||
```
|
||||
|
||||
## 🧪 **Testing Your Migration**
|
||||
|
||||
### Basic Functionality Test
|
||||
|
||||
```javascript
|
||||
import { BrainyData } from '@soulcraft/brainy'
|
||||
|
||||
async function testMigration() {
|
||||
const brainy = new BrainyData()
|
||||
await brainy.init()
|
||||
|
||||
// Test core functionality
|
||||
const id = await brainy.add("Test migration data")
|
||||
const results = await brainy.search("migration", 5)
|
||||
await brainy.update(id, "Updated data")
|
||||
await brainy.delete(id) // Soft delete
|
||||
|
||||
console.log("✅ Migration successful!")
|
||||
}
|
||||
|
||||
testMigration()
|
||||
```
|
||||
|
||||
### CLI Test
|
||||
|
||||
```bash
|
||||
# Test CLI functionality
|
||||
brainy add "Test data"
|
||||
brainy search "test"
|
||||
brainy status
|
||||
brainy --help
|
||||
```
|
||||
|
||||
## ⚠️ **Breaking Changes Summary**
|
||||
|
||||
### Definite Breaking Changes
|
||||
1. **CLI commands renamed** - Most commands have new names
|
||||
2. **`addSmart()` method removed** - Use `add()` instead
|
||||
3. **Pipeline classes consolidated** - Multiple classes → one Cortex
|
||||
4. **Some internal import paths** - Check if using internal APIs
|
||||
|
||||
### Likely Compatible
|
||||
1. **Core API methods** - `add()`, `search()` largely the same
|
||||
2. **Storage adapters** - All existing adapters work
|
||||
3. **Configuration** - Existing configs should work
|
||||
4. **Data format** - Your existing data is compatible
|
||||
|
||||
## 🆘 **Getting Help**
|
||||
|
||||
If you encounter issues during migration:
|
||||
|
||||
1. **Check the examples** in this guide
|
||||
2. **Test with a small dataset** first
|
||||
3. **File an issue** with the `migration` label
|
||||
4. **Join discussions** for community help
|
||||
|
||||
### Common Migration Issues
|
||||
|
||||
**Issue**: `addSmart is not a function`
|
||||
```javascript
|
||||
// Fix: Use add() instead
|
||||
await brainy.add(data, metadata) // Smart by default
|
||||
```
|
||||
|
||||
**Issue**: CLI command not found
|
||||
```bash
|
||||
# Fix: Check command mapping above
|
||||
brainy search "query" # Not search-similar
|
||||
```
|
||||
|
||||
**Issue**: Pipeline import error
|
||||
```javascript
|
||||
// Fix: Use unified import
|
||||
import { Pipeline } from '@soulcraft/brainy'
|
||||
```
|
||||
|
||||
## 🎉 **New Features to Explore**
|
||||
|
||||
After migration, try these new features:
|
||||
|
||||
```javascript
|
||||
// Bulk import
|
||||
const ids = await brainy.import([data1, data2, data3])
|
||||
|
||||
// Explicit noun typing
|
||||
await brainy.addNoun(personData, NounType.Person)
|
||||
|
||||
// Encrypted storage
|
||||
await brainy.add(sensitiveData, {}, { encrypt: true })
|
||||
|
||||
// Smart updates
|
||||
await brainy.update(id, newData, { cascade: true })
|
||||
```
|
||||
|
||||
```bash
|
||||
# New CLI features
|
||||
brainy init --encryption --storage s3
|
||||
brainy import large-dataset.json
|
||||
brainy delete old-id --cascade
|
||||
brainy chat "Tell me about my data"
|
||||
```
|
||||
|
||||
## 📞 **Support**
|
||||
|
||||
- **📚 Documentation**: Updated for 1.0 API
|
||||
- **🐛 Issues**: [GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)
|
||||
- **💬 Discussions**: [GitHub Discussions](https://github.com/soulcraftlabs/brainy/discussions)
|
||||
- **🏷️ Tags**: Use `migration`, `1.0-rc.1`, `breaking-change` tags
|
||||
|
||||
**We're here to help make your migration smooth!** 🚀
|
||||
|
|
@ -1,143 +0,0 @@
|
|||
# Migration Plan: Deprecated Methods
|
||||
|
||||
## Overview
|
||||
This document outlines the migration plan for deprecated methods in the Brainy codebase. These methods are marked as deprecated due to potential memory issues with large datasets.
|
||||
|
||||
## Deprecated Methods
|
||||
|
||||
### 1. Storage Adapter Methods
|
||||
- `getAllNouns()` → Use `getNouns()` with pagination
|
||||
- `getAllVerbs()` → Use `getVerbs()` with pagination
|
||||
- `getNounsByNounType(nounType)` → Use `getNouns({ nounType })`
|
||||
- `getVerbsBySource(sourceId)` → Use `getVerbs({ sourceId })`
|
||||
- `getVerbsByTarget(targetId)` → Use `getVerbs({ targetId })`
|
||||
- `getVerbsByType(type)` → Use `getVerbs({ verbType: type })`
|
||||
|
||||
### 2. HNSW Index Methods
|
||||
- `getNouns()` → Use `getNounsPaginated()`
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Phase 1: Update Internal BrainyData Usage (High Priority)
|
||||
**Files to update:**
|
||||
- `src/brainyData.ts` - 8 usages of deprecated methods
|
||||
- `src/augmentations/memoryAugmentations.ts` - 1 usage
|
||||
- `src/mcp/brainyMCPAdapter.ts` - 2 usages
|
||||
|
||||
**Specific Changes:**
|
||||
|
||||
#### src/brainyData.ts
|
||||
```typescript
|
||||
// OLD:
|
||||
const nouns = await this.storage!.getAllNouns()
|
||||
|
||||
// NEW:
|
||||
const nouns: HNSWNoun[] = []
|
||||
let cursor: SearchCursor | undefined
|
||||
do {
|
||||
const result = await this.storage!.getNouns({}, { limit: 1000, cursor })
|
||||
nouns.push(...result.results)
|
||||
cursor = result.cursor
|
||||
} while (cursor)
|
||||
```
|
||||
|
||||
#### src/augmentations/memoryAugmentations.ts
|
||||
```typescript
|
||||
// OLD:
|
||||
const nodes = await this.storage.getAllNouns()
|
||||
|
||||
// NEW:
|
||||
const nodes = await this.storage.getNouns({}, { limit: Number.MAX_SAFE_INTEGER })
|
||||
```
|
||||
|
||||
#### src/mcp/brainyMCPAdapter.ts
|
||||
```typescript
|
||||
// OLD:
|
||||
const outgoing = await (this.brainyData as any).getVerbsBySource?.(id) || []
|
||||
|
||||
// NEW:
|
||||
const outgoing = await this.brainyData.getVerbs({ sourceId: id }, { limit: Number.MAX_SAFE_INTEGER }) || []
|
||||
```
|
||||
|
||||
### Phase 2: Update Storage Adapter Implementations (Medium Priority)
|
||||
**Files to update:**
|
||||
- `src/storage/adapters/memoryStorage.ts`
|
||||
- `src/storage/adapters/fileSystemStorage.ts`
|
||||
- `src/storage/adapters/opfsStorage.ts`
|
||||
- `src/storage/adapters/s3CompatibleStorage.ts`
|
||||
|
||||
**Strategy:** Keep deprecated methods but mark them as internal-only and implement them using the new filtered methods.
|
||||
|
||||
### Phase 3: Update Type Definitions (Low Priority)
|
||||
**Files to update:**
|
||||
- `src/coreTypes.ts` - Remove deprecated method signatures
|
||||
- `src/storage/baseStorage.ts` - Remove deprecated implementations
|
||||
|
||||
### Phase 4: Update Tests (Low Priority)
|
||||
**Files to update:**
|
||||
- `tests/*.test.ts` - Update test files that use deprecated methods
|
||||
|
||||
## Implementation Order
|
||||
|
||||
### 1. Immediate (Safe Changes)
|
||||
- ✅ Remove unused files (tensorflowUtils.ts, etc.)
|
||||
- ✅ Fix TODO items with version handling
|
||||
- Update internal BrainyData usage with pagination
|
||||
|
||||
### 2. Short-term (1-2 weeks)
|
||||
- Update augmentation and MCP adapter usage
|
||||
- Add deprecation warnings to method implementations
|
||||
- Update storage adapter internal implementations
|
||||
|
||||
### 3. Long-term (Next major version)
|
||||
- Remove deprecated method signatures from interfaces
|
||||
- Remove deprecated method implementations
|
||||
- Update all test files
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
### Option 1: Gradual Deprecation (Recommended)
|
||||
- Keep deprecated methods but add console warnings
|
||||
- Implement them using new filtered methods internally
|
||||
- Remove in next major version (0.42.0 → 1.0.0)
|
||||
|
||||
### Option 2: Immediate Removal
|
||||
- Remove deprecated methods now
|
||||
- Update all usage immediately
|
||||
- Risk: Breaking changes for external users
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Unit Tests**: Update existing tests to use new methods
|
||||
2. **Integration Tests**: Verify pagination works correctly
|
||||
3. **Performance Tests**: Ensure new implementations don't regress performance
|
||||
4. **Memory Tests**: Verify large dataset handling improves
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Revert to previous implementations
|
||||
2. Add memory limits as temporary solution
|
||||
3. Implement pagination more gradually
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- ✅ All deprecated method usage removed from core files
|
||||
- ✅ No memory issues with large datasets
|
||||
- ✅ Performance maintained or improved
|
||||
- ✅ All tests passing
|
||||
- ✅ Backward compatibility preserved (if Option 1)
|
||||
|
||||
## Timeline
|
||||
|
||||
- **Week 1**: Complete Phase 1 (internal usage)
|
||||
- **Week 2**: Complete Phase 2 (storage adapters)
|
||||
- **Week 3**: Complete Phase 3 (type definitions)
|
||||
- **Week 4**: Complete Phase 4 (tests) and validation
|
||||
|
||||
## Notes
|
||||
|
||||
- The deprecated methods are still widely used, so this migration requires careful planning
|
||||
- Consider adding configuration option to control pagination limits
|
||||
- Document the breaking changes clearly in CHANGELOG.md
|
||||
- Consider providing migration utilities for external users
|
||||
|
|
@ -1,56 +0,0 @@
|
|||
# Offline Models
|
||||
|
||||
Brainy uses Transformers.js with ONNX Runtime for **true offline operation** - no more TensorFlow.js dependency hell!
|
||||
|
||||
## How it works
|
||||
|
||||
Brainy automatically figures out the best approach:
|
||||
|
||||
1. **First use**: Downloads models once (~87 MB) to local cache
|
||||
2. **Subsequent use**: Loads from cache (completely offline, zero network calls)
|
||||
3. **Smart detection**: Automatically finds models in cache, bundled, or downloads as needed
|
||||
|
||||
## Standard usage
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/brainy
|
||||
# Use immediately - models download automatically on first use
|
||||
```
|
||||
|
||||
## Docker with production egress restrictions
|
||||
|
||||
For environments where production has no internet but build does:
|
||||
|
||||
```dockerfile
|
||||
FROM node:24-slim
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm install @soulcraft/brainy
|
||||
RUN npm run download-models # Download during build (when internet available)
|
||||
COPY . .
|
||||
# Production container now works completely offline
|
||||
```
|
||||
|
||||
## Development with immediate offline
|
||||
|
||||
If you want models available immediately for development:
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/brainy
|
||||
npm run download-models # Optional: download now instead of on first use
|
||||
```
|
||||
|
||||
## Key benefits vs TensorFlow.js
|
||||
|
||||
- ✅ **95% smaller package** - 643 kB vs 12.5 MB
|
||||
- ✅ **84% smaller models** - 87 MB vs 525 MB
|
||||
- ✅ **True offline** - Zero network calls after initial download
|
||||
- ✅ **No dependency issues** - 5 deps vs 47+, no more --legacy-peer-deps
|
||||
- ✅ **Better performance** - ONNX Runtime beats TensorFlow.js
|
||||
- ✅ **Same API** - Drop-in replacement
|
||||
|
||||
## Philosophy
|
||||
|
||||
**Install and use. Brainy handles the rest.**
|
||||
|
||||
No configuration files, no environment variables, no complex setup. Brainy detects your environment and does the right thing automatically.
|
||||
|
|
@ -1,204 +0,0 @@
|
|||
# 🚀 Metadata Filtering Performance Optimization
|
||||
|
||||
**Priority: HIGH** | **Complexity: Medium** | **Est. Time: 2-3 hours**
|
||||
|
||||
## 🎯 The Issue
|
||||
|
||||
Current metadata filtering has a **300-400% search overhead** due to a fixed 3x ef multiplier in HNSW search, regardless of filter selectivity.
|
||||
|
||||
### Performance Analysis
|
||||
- **No filtering**: 31.46ms average
|
||||
- **Simple filter**: 150.82ms average (**379% overhead**)
|
||||
- **Complex filter**: 153.05ms average (**386% overhead**)
|
||||
|
||||
### Root Cause
|
||||
```typescript
|
||||
// Current implementation (inefficient)
|
||||
// File: src/hnsw/hnswIndex.ts:377
|
||||
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)
|
||||
```
|
||||
|
||||
The **fixed 3x multiplier** is used for ALL filtered searches, whether the filter matches 1% or 90% of items.
|
||||
|
||||
## 🛠️ Solution: Dynamic ef Multiplier
|
||||
|
||||
### 1. Calculate Filter Selectivity
|
||||
```typescript
|
||||
// Add to searchByNounTypes method
|
||||
const selectivity = candidateIds.length / totalItems
|
||||
```
|
||||
|
||||
### 2. Dynamic Multiplier Strategy
|
||||
```typescript
|
||||
function getEfMultiplier(selectivity: number): number {
|
||||
if (selectivity <= 0.01) return 1.1 // 1% match: minimal overhead
|
||||
if (selectivity <= 0.05) return 1.3 // 5% match: small overhead
|
||||
if (selectivity <= 0.15) return 1.6 // 15% match: medium overhead
|
||||
if (selectivity <= 0.30) return 2.0 // 30% match: higher overhead
|
||||
return 1.0 // 30%+ match: no overhead (post-filter)
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Implementation Location
|
||||
|
||||
**File**: `src/brainyData.ts`
|
||||
**Method**: `searchByNounTypes` (around line 2165)
|
||||
|
||||
```typescript
|
||||
// Current code around line 2165:
|
||||
if (hasMetadataFilter && this.metadataIndex) {
|
||||
const candidateIds = await this.metadataIndex.getIdsForCriteria(options.metadata)
|
||||
|
||||
// ADD THIS: Calculate selectivity
|
||||
const totalItems = this.index.size()
|
||||
const selectivity = candidateIds.length / totalItems
|
||||
const efMultiplier = getEfMultiplier(selectivity)
|
||||
|
||||
// Store efMultiplier for use in HNSW search
|
||||
}
|
||||
```
|
||||
|
||||
**File**: `src/hnsw/hnswIndex.ts`
|
||||
**Method**: `search` (around line 377)
|
||||
|
||||
```typescript
|
||||
// Replace this line:
|
||||
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)
|
||||
|
||||
// With dynamic calculation:
|
||||
const ef = filter && filterSelectivity
|
||||
? Math.max(this.config.efSearch * filterSelectivity.multiplier, k)
|
||||
: Math.max(this.config.efSearch, k)
|
||||
```
|
||||
|
||||
## 📈 Expected Performance Improvements
|
||||
|
||||
| Filter Selectivity | Current Overhead | Expected Overhead | Improvement |
|
||||
|-------------------|------------------|-------------------|-------------|
|
||||
| 1% (high selectivity) | 379% | 50% | **85% faster** |
|
||||
| 5% (medium selectivity) | 379% | 80% | **70% faster** |
|
||||
| 15% (low selectivity) | 379% | 150% | **50% faster** |
|
||||
| 30%+ (very low selectivity) | 379% | 10% | **90% faster** |
|
||||
|
||||
## 🔧 Implementation Steps
|
||||
|
||||
### Step 1: Add Selectivity Calculation
|
||||
1. Modify `searchByNounTypes` to calculate selectivity
|
||||
2. Pass selectivity info to HNSW search method
|
||||
3. Create `getEfMultiplier` helper function
|
||||
|
||||
### Step 2: Update HNSW Search
|
||||
1. Modify `HNSWIndex.search` to accept selectivity parameter
|
||||
2. Update `HNSWIndexOptimized.search` to forward selectivity
|
||||
3. Replace fixed 3x multiplier with dynamic calculation
|
||||
|
||||
### Step 3: Test Performance
|
||||
1. Run performance benchmarks with different selectivity scenarios
|
||||
2. Verify filtering accuracy is maintained
|
||||
3. Test edge cases (empty results, 100% selectivity)
|
||||
|
||||
### Step 4: Optional Enhancements
|
||||
1. **Index-First Strategy**: For <5% selectivity, search only candidate vectors
|
||||
2. **Post-Filter Strategy**: For >50% selectivity, search all then filter
|
||||
3. **Query Pattern Learning**: Track and optimize based on common patterns
|
||||
|
||||
## 🧪 Test Cases
|
||||
|
||||
```typescript
|
||||
// High selectivity (1% match) - should be very fast
|
||||
await brainy.search("developer", 10, {
|
||||
metadata: { rare_certification: "specific_cert" }
|
||||
})
|
||||
|
||||
// Medium selectivity (10% match) - should be moderately fast
|
||||
await brainy.search("developer", 10, {
|
||||
metadata: { level: "senior" }
|
||||
})
|
||||
|
||||
// Low selectivity (50% match) - should use post-filtering
|
||||
await brainy.search("developer", 10, {
|
||||
metadata: { active: true }
|
||||
})
|
||||
```
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
Add performance logging to track improvements:
|
||||
|
||||
```typescript
|
||||
const start = Date.now()
|
||||
const selectivity = candidateIds.length / totalItems
|
||||
const efMultiplier = getEfMultiplier(selectivity)
|
||||
|
||||
console.log(`Filter selectivity: ${(selectivity * 100).toFixed(1)}%, ef multiplier: ${efMultiplier}`)
|
||||
|
||||
// After search
|
||||
console.log(`Search completed in ${Date.now() - start}ms`)
|
||||
```
|
||||
|
||||
## 🚨 Known Issues to Fix
|
||||
|
||||
### Issue 1: HNSWIndexOptimized Filtering Bug
|
||||
**Status**: Identified but not fixed
|
||||
**Problem**: `HNSWIndexOptimized.search()` method signature was missing filter parameter
|
||||
**Solution**: Already added filter parameter, but needs verification
|
||||
|
||||
### Issue 2: Method Binding
|
||||
**Problem**: TypeScript might not be calling overridden methods correctly
|
||||
**Solution**: Verify method resolution and binding
|
||||
|
||||
## 🎉 Success Criteria
|
||||
|
||||
- [ ] Filtered search performance improves by 50-90% based on selectivity
|
||||
- [ ] Filtering accuracy remains 100% correct
|
||||
- [ ] No breaking changes to existing API
|
||||
- [ ] Performance monitoring shows expected improvements
|
||||
- [ ] All existing tests continue to pass
|
||||
|
||||
## 🧠 Additional Optimization: LRU Cache for Metadata Indexes
|
||||
|
||||
**Priority: MEDIUM** | **Complexity: Low** | **Est. Time: 1-2 hours**
|
||||
|
||||
### The Opportunity
|
||||
Add LRU caching to metadata indexes similar to HNSW index caching:
|
||||
|
||||
```typescript
|
||||
// Reuse existing infrastructure
|
||||
this.metadataCache = new LRUCache<string, any>({
|
||||
maxSize: config.maxCacheSize ?? 1000,
|
||||
ttl: config.cacheTTL ?? 300000 // 5 minutes
|
||||
})
|
||||
|
||||
// Cache field indexes and value chunks
|
||||
const cachedFieldIndex = this.metadataCache.get(`field_${field}`)
|
||||
```
|
||||
|
||||
### Expected Benefits
|
||||
- **10-100x faster** repeated filter discovery
|
||||
- **Zero latency** for common filter UI operations
|
||||
- **90% code reuse** from existing HNSW cache system
|
||||
- **Automatic learning** of usage patterns
|
||||
|
||||
### Implementation
|
||||
1. Extend existing LRU cache to metadata indexes
|
||||
2. Cache field indexes (`field_category.json`)
|
||||
3. Cache hot value chunks (`category_electronics_chunk0.json`)
|
||||
4. Add cache invalidation on metadata updates
|
||||
5. Reuse existing cache statistics and monitoring
|
||||
|
||||
## 📝 Files to Modify
|
||||
|
||||
1. `src/brainyData.ts` (selectivity calculation)
|
||||
2. `src/hnsw/hnswIndex.ts` (dynamic ef multiplier)
|
||||
3. `src/hnsw/optimizedHNSWIndex.ts` (forward selectivity)
|
||||
|
||||
## 🔄 Rollback Plan
|
||||
|
||||
If performance optimization causes issues:
|
||||
1. Revert to fixed 3x multiplier
|
||||
2. Feature flag the optimization for gradual rollout
|
||||
3. Add configuration option to disable dynamic multiplier
|
||||
|
||||
---
|
||||
|
||||
**This optimization will make metadata filtering 50-90% faster while maintaining the same powerful querying capabilities!** 🚀
|
||||
164
PHILOSOPHY.md
164
PHILOSOPHY.md
|
|
@ -1,164 +0,0 @@
|
|||
# 🧠 Brainy Philosophy & Design Principles
|
||||
|
||||
## Core Philosophy
|
||||
**"Simple for beginners, powerful for experts"**
|
||||
|
||||
## Design Principles
|
||||
|
||||
### 1. **Beginner-Friendly by Default**
|
||||
- If no arguments provided → Interactive mode with helpful prompts
|
||||
- Clear, simple language (no jargon)
|
||||
- Helpful examples shown automatically
|
||||
- Smart defaults that "just work"
|
||||
|
||||
### 2. **Clean, Simple Language**
|
||||
```bash
|
||||
# Good - Clear and simple
|
||||
brainy add "John works at Acme Corp"
|
||||
brainy search "Who works at Acme?"
|
||||
brainy augment neural-import data.csv
|
||||
|
||||
# Bad - Technical and confusing
|
||||
brainy insert --vector-dimension=384 --graph-node="person"
|
||||
brainy query --similarity-threshold=0.8 --facet-filter='{"type":"person"}'
|
||||
```
|
||||
|
||||
### 3. **Progressive Disclosure**
|
||||
- Start simple, reveal complexity only when needed
|
||||
- Basic usage requires no configuration
|
||||
- Advanced features available but not required
|
||||
|
||||
### 4. **Interactive When Uncertain**
|
||||
```bash
|
||||
$ brainy add
|
||||
? What would you like to add? › John Smith is a developer
|
||||
? Add any tags or categories? (optional) › person, developer
|
||||
✅ Added successfully!
|
||||
|
||||
$ brainy search
|
||||
? What are you looking for? › developers
|
||||
? How many results? (10) › 5
|
||||
🔍 Found 5 matches...
|
||||
```
|
||||
|
||||
### 5. **Consistent Brain Metaphor**
|
||||
- **Brainy** = The brain (whole system)
|
||||
- **Cortex** = Orchestrator (coordinates everything)
|
||||
- **Neural Import** = Understanding data (neural processing)
|
||||
- **Augmentations** = Brain capabilities (vision, hearing, memory, etc.)
|
||||
|
||||
### 6. **One Way to Do Things**
|
||||
- Single clear path for common tasks
|
||||
- No duplicate commands or confusing aliases
|
||||
- If there are options, make the best one the default
|
||||
|
||||
### 7. **Helpful Error Messages**
|
||||
```bash
|
||||
# Good
|
||||
❌ Can't find data.csv
|
||||
💡 Did you mean data.json? Or try: brainy import --help
|
||||
|
||||
# Bad
|
||||
Error: ENOENT no such file or directory
|
||||
```
|
||||
|
||||
### 8. **Smart Defaults**
|
||||
- Auto-detect file types
|
||||
- Infer intent from context
|
||||
- Use Neural Import by default for understanding data
|
||||
- Automatic augmentation discovery
|
||||
|
||||
## CLI Command Philosophy
|
||||
|
||||
### Core Commands (Simple Verbs)
|
||||
```bash
|
||||
brainy init # Start here
|
||||
brainy add # Add data
|
||||
brainy search # Find data
|
||||
brainy chat # Talk to your data
|
||||
```
|
||||
|
||||
### Augmentation Commands (Clear Actions)
|
||||
```bash
|
||||
brainy augment # List/manage augmentations
|
||||
brainy augment add neural-import # Add an augmentation
|
||||
brainy augment remove notion-sync # Remove an augmentation
|
||||
```
|
||||
|
||||
### Interactive Examples
|
||||
```bash
|
||||
# No arguments = Interactive mode
|
||||
$ brainy add
|
||||
? What would you like to add? › [waiting for input]
|
||||
|
||||
# With arguments = Direct mode
|
||||
$ brainy add "Sarah is a designer at StartupXYZ"
|
||||
✅ Added!
|
||||
|
||||
# Help is conversational
|
||||
$ brainy help add
|
||||
📝 Add data to your brain
|
||||
|
||||
Examples:
|
||||
brainy add "John works at Acme"
|
||||
brainy add data.csv
|
||||
brainy add --interactive
|
||||
|
||||
Just run 'brainy add' for interactive mode!
|
||||
```
|
||||
|
||||
## Code Philosophy
|
||||
|
||||
### Function Names
|
||||
```typescript
|
||||
// Good - Clear and simple
|
||||
brain.add(data)
|
||||
brain.search(query)
|
||||
cortex.process(augmentation)
|
||||
|
||||
// Bad - Technical and verbose
|
||||
brain.insertVectorWithGraphRelationships(data)
|
||||
brain.executeMultiDimensionalQuery(query)
|
||||
cortex.executeAugmentationPipeline(augmentation)
|
||||
```
|
||||
|
||||
### API Design
|
||||
```typescript
|
||||
// Good - Progressive enhancement
|
||||
brain.add("simple text") // Works
|
||||
brain.add("text", { category: "person" }) // More control
|
||||
brain.add("text", metadata, options) // Full control
|
||||
|
||||
// Bad - All or nothing
|
||||
brain.add(text, vector, metadata, options, callback)
|
||||
```
|
||||
|
||||
## Documentation Philosophy
|
||||
|
||||
### README Structure
|
||||
1. **What it is** (one sentence)
|
||||
2. **Quick start** (3 commands max)
|
||||
3. **Simple examples** (real-world use)
|
||||
4. **Going deeper** (advanced features)
|
||||
|
||||
### Example First
|
||||
Always show the example before explaining:
|
||||
```bash
|
||||
# Add a person
|
||||
brainy add "Alice is a product manager"
|
||||
|
||||
# Find them later
|
||||
brainy search "product manager"
|
||||
```
|
||||
|
||||
## Testing Philosophy
|
||||
- Test the beginner path first
|
||||
- Interactive mode must always work
|
||||
- Examples in docs must be runnable
|
||||
- Error messages must be helpful
|
||||
|
||||
## Remember
|
||||
- **If it's not simple, it's not ready**
|
||||
- **The best interface is no interface** (smart defaults)
|
||||
- **Show, don't tell** (examples > explanations)
|
||||
- **Beginner's mind** (always ask: would a newcomer understand this?)
|
||||
633
README.md
633
README.md
|
|
@ -1,494 +1,217 @@
|
|||
<div align="center">
|
||||
<p align="center">
|
||||
<img src="https://raw.githubusercontent.com/soulcraftlabs/brainy/main/brainy.png" alt="Brainy" width="180">
|
||||
</p>
|
||||
|
||||
|
||||
<h1 align="center">Brainy</h1>
|
||||
|
||||
[](https://badge.fury.io/js/%40soulcraft%2Fbrainy)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://soulcraft.com)
|
||||
[](https://soulcraft.com/brain-cloud)
|
||||
[](https://nodejs.org/)
|
||||
[](https://www.typescriptlang.org/)
|
||||
<p align="center">
|
||||
<b>Three database paradigms. One API. Zero configuration.</b><br>
|
||||
The in-process knowledge database for TypeScript — vector search, graph traversal,<br>
|
||||
and metadata filtering unified in a single query.
|
||||
</p>
|
||||
|
||||
# The World's First Multi-Dimensional AI Database™
|
||||
<p align="center">
|
||||
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/v/@soulcraft/brainy.svg" alt="npm version"></a>
|
||||
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/dm/@soulcraft/brainy.svg" alt="npm downloads"></a>
|
||||
<a href="https://github.com/soulcraftlabs/brainy/actions/workflows/ci.yml"><img src="https://github.com/soulcraftlabs/brainy/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
|
||||
<a href="https://soulcraft.com/docs"><img src="https://img.shields.io/badge/docs-soulcraft.com-blue.svg" alt="Documentation"></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a>
|
||||
<a href="https://www.typescriptlang.org/"><img src="https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg" alt="TypeScript"></a>
|
||||
</p>
|
||||
|
||||
*Vector similarity • Graph relationships • Metadata facets • Neural understanding*
|
||||
|
||||
**Build AI apps that actually understand your data - in minutes, not months**
|
||||
|
||||
</div>
|
||||
<p align="center">
|
||||
<a href="#quick-start">Quick start</a> ·
|
||||
<a href="#one-query-three-engines">One query</a> ·
|
||||
<a href="#feature-tour">Features</a> ·
|
||||
<a href="#from-laptop-to-hundreds-of-millions">Scale with Cor</a> ·
|
||||
<a href="#documentation">Docs</a>
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
## ✅ 100% Free & Open Source
|
||||
Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact **three ways at once** and lets one call query them together:
|
||||
|
||||
**Brainy is completely free. No license keys. No limits. No catch.**
|
||||
| You write | Brainy indexes it as | You query it with |
|
||||
|---|---|---|
|
||||
| `data: 'Ada wrote the first program'` | a **384-dim vector** (local embedding — no API key) | `find({ query: 'computing pioneers' })` |
|
||||
| `metadata: { field: 'CS', year: 1843 }` | **structured fields** (O(1) exact, O(log n) range) | `find({ where: { year: { lessThan: 1900 } } })` |
|
||||
| `relate({ from: ada, to: babbage })` | a **typed, directed graph edge** | `find({ connected: { to: babbage, depth: 2 } })` |
|
||||
|
||||
Every feature you see here works without any payment or registration:
|
||||
- ✓ Full vector database
|
||||
- ✓ Graph relationships
|
||||
- ✓ Semantic search
|
||||
- ✓ All storage adapters
|
||||
- ✓ Complete API
|
||||
- ✓ Forever free
|
||||
It runs **inside your process** — no server, no Docker, nothing to operate — and persists to plain files you can snapshot with a hard link.
|
||||
|
||||
> 🌩️ **Brain Cloud** is an optional add-on for teams who want cloud sync and enterprise connectors.
|
||||
**New here?** → **[What is Brainy? — plain-language overview, no jargon](docs/eli5.md)**
|
||||
|
||||
---
|
||||
## Quick start
|
||||
|
||||
## 🚀 What Can You Build?
|
||||
|
||||
### 💬 **AI Chat Apps** - That Actually Remember
|
||||
```javascript
|
||||
// Your users' conversations persist across sessions
|
||||
const brain = new BrainyData()
|
||||
await brain.add("User prefers dark mode")
|
||||
await brain.add("User is learning Spanish")
|
||||
|
||||
// Later sessions remember everything
|
||||
const context = await brain.search("user preferences")
|
||||
// AI knows: dark mode + Spanish learning preference
|
||||
```
|
||||
|
||||
### 🤖 **Smart Assistants** - With Real Knowledge Graphs
|
||||
```javascript
|
||||
// Build assistants that understand relationships
|
||||
await brain.add("Sarah manages the design team")
|
||||
await brain.addVerb("Sarah", "reports_to", "John")
|
||||
await brain.addVerb("Sarah", "works_on", "Project Apollo")
|
||||
|
||||
// Query complex relationships
|
||||
const team = await brain.getRelated("Project Apollo", {
|
||||
verb: "works_on",
|
||||
depth: 2
|
||||
})
|
||||
// Returns: entire team structure with relationships
|
||||
```
|
||||
|
||||
### 📊 **RAG Applications** - Without the Complexity
|
||||
```javascript
|
||||
// Retrieval-Augmented Generation in 3 lines
|
||||
await brain.add(companyDocs) // Add your knowledge base
|
||||
const relevant = await brain.search(userQuery, 10) // Find relevant context
|
||||
const answer = await llm.generate(relevant + userQuery) // Generate with context
|
||||
```
|
||||
|
||||
### 🔍 **Semantic Search** - That Just Works
|
||||
```javascript
|
||||
// No embeddings API needed - it's built in!
|
||||
await brain.add("The iPhone 15 Pro has a titanium design")
|
||||
await brain.add("Samsung Galaxy S24 features AI photography")
|
||||
|
||||
const results = await brain.search("premium smartphones with metal build")
|
||||
// Returns: iPhone (titanium matches "metal build" semantically)
|
||||
```
|
||||
|
||||
### 🎯 **Recommendation Engines** - With Graph Intelligence
|
||||
```javascript
|
||||
// Netflix-style recommendations with relationships
|
||||
await brain.addVerb("User123", "watched", "Inception")
|
||||
await brain.addVerb("User123", "liked", "Inception")
|
||||
await brain.addVerb("Inception", "similar_to", "Interstellar")
|
||||
|
||||
const recommendations = await brain.getRelated("User123", {
|
||||
verb: ["liked", "watched"],
|
||||
depth: 2
|
||||
})
|
||||
// Returns: Interstellar and other related content
|
||||
```
|
||||
|
||||
## 💫 Why Brainy? The Problem We Solve
|
||||
|
||||
### ❌ **The Old Way: Database Frankenstein**
|
||||
```
|
||||
Pinecone ($750/mo) + Neo4j ($500/mo) + Elasticsearch ($300/mo) +
|
||||
Sync nightmares + 3 different APIs + Vendor lock-in = 😱💸
|
||||
```
|
||||
|
||||
### ✅ **The Brainy Way: One Brain, All Dimensions**
|
||||
```
|
||||
Vector + Graph + Search + AI = Brainy (Free & Open Source) = 🧠✨
|
||||
```
|
||||
|
||||
**Your data gets superpowers. Your wallet stays happy.**
|
||||
|
||||
## 🎮 Try It Now - No Install Required!
|
||||
|
||||
<div align="center">
|
||||
|
||||
### [**→ Live Demos at soulcraft.com/demo ←**](https://soulcraft.com/demo)
|
||||
|
||||
Try Brainy instantly in your browser. No signup. No credit card.
|
||||
|
||||
</div>
|
||||
|
||||
## ⚡ Quick Start (60 Seconds)
|
||||
|
||||
### Open Source (Local Storage)
|
||||
```bash
|
||||
npm install @soulcraft/brainy
|
||||
bun add @soulcraft/brainy # Bun ≥ 1.1 — recommended
|
||||
npm install @soulcraft/brainy # Node.js ≥ 22
|
||||
```
|
||||
|
||||
```javascript
|
||||
import { BrainyData } from '@soulcraft/brainy'
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
// Zero configuration - it just works!
|
||||
const brain = new BrainyData()
|
||||
const brain = new Brainy() // in-memory; one line swaps to disk
|
||||
await brain.init()
|
||||
|
||||
// Add any data - text, objects, relationships
|
||||
await brain.add("Elon Musk founded SpaceX in 2002")
|
||||
await brain.add({ company: "Tesla", ceo: "Elon Musk", founded: 2003 })
|
||||
await brain.addVerb("Elon Musk", "founded", "Tesla")
|
||||
// Text auto-embeds locally; metadata auto-indexes
|
||||
const react = await brain.add({
|
||||
data: 'React is a JavaScript library for building user interfaces',
|
||||
type: NounType.Concept,
|
||||
subtype: 'library',
|
||||
metadata: { category: 'frontend', year: 2013 }
|
||||
})
|
||||
|
||||
// Search naturally
|
||||
const results = await brain.search("companies founded by Elon")
|
||||
const next = await brain.add({
|
||||
data: 'Next.js is a React framework with server-side rendering',
|
||||
type: NounType.Concept,
|
||||
subtype: 'framework',
|
||||
metadata: { category: 'frontend', year: 2016 }
|
||||
})
|
||||
|
||||
await brain.relate({ from: next, to: react, type: VerbType.DependsOn, subtype: 'runtime' })
|
||||
```
|
||||
|
||||
### ☁️ Brain Cloud (AI Memory + Agent Coordination)
|
||||
## One query, three engines
|
||||
|
||||
```javascript
|
||||
const results = await brain.find({
|
||||
query: 'modern frontend frameworks', // vector — what it means
|
||||
where: { year: { greaterThan: 2015 } }, // metadata — what it is
|
||||
connected: { to: react, depth: 2 } // graph — what it touches
|
||||
})
|
||||
```
|
||||
|
||||
Every clause is optional; any combination composes. Under the hood Brainy plans the query across an HNSW vector index, a roaring-bitmap field index, and an adjacency graph index — and re-validates every result against your predicate before returning it, so a corrupt index can never hand you a wrong answer.
|
||||
|
||||
## Feature tour
|
||||
|
||||
### The database is a value
|
||||
|
||||
Pin it, rewind it, fork it. Snapshot isolation without a server.
|
||||
|
||||
```javascript
|
||||
const db = brain.now() // pin current state — O(1)
|
||||
|
||||
await brain.transact([ // atomic all-or-nothing, CAS-guarded
|
||||
{ op: 'update', id: order, metadata: { status: 'paid' } },
|
||||
{ op: 'relate', from: invoice, to: order, type: VerbType.References, subtype: 'billing' }
|
||||
], { ifAtGeneration: db.generation })
|
||||
|
||||
await db.get(order) // still 'pending' — pinned forever
|
||||
await brain.get(order) // 'paid' — live
|
||||
|
||||
const lastWeek = await brain.asOf(Date.now() - 7 * 86_400_000) // full query surface, past state
|
||||
const whatIf = await db.with([{ op: 'remove', id: order }]) // speculative — never touches disk
|
||||
await brain.now().persist('/backups/today') // instant hard-link snapshot
|
||||
```
|
||||
|
||||
**[Consistency model](docs/concepts/consistency-model.md)** · **[Snapshots & time travel](docs/guides/snapshots-and-time-travel.md)**
|
||||
|
||||
### Local embeddings — no API keys
|
||||
|
||||
Strings embed on-device with a bundled MiniLM model (WASM). Semantic search works offline, in CI, and on air-gapped machines, at zero cost per call. Hybrid keyword + semantic ranking is the default:
|
||||
|
||||
```javascript
|
||||
await brain.find({ query: 'David Smith' }) // auto: text + semantic
|
||||
await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // semantic only
|
||||
```
|
||||
|
||||
### A typed graph, not a bag of edges
|
||||
|
||||
42 entity types × 127 relationship types form a shared vocabulary for any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), yours. Your own taxonomy layers on with `subtype`, enforced at write time:
|
||||
|
||||
```javascript
|
||||
await brain.add({ data: 'Avery Brooks', type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
brain.counts.bySubtype(NounType.Person) // O(1) — { employee: 12, customer: 847 }
|
||||
brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
|
||||
```
|
||||
|
||||
**[Type system](docs/architecture/noun-verb-taxonomy.md)** · **[Subtypes & facets](docs/guides/subtypes-and-facets.md)**
|
||||
|
||||
### Graph analytics built in
|
||||
|
||||
```javascript
|
||||
await brain.graph.rank() // which entities matter most (centrality)
|
||||
await brain.graph.communities() // natural clusters
|
||||
await brain.graph.path(a, b) // how two things connect
|
||||
await brain.graph.subgraph([seed], { depth: 2 }) // bounded neighborhood → { nodes, edges }
|
||||
await brain.graph.export() // whole graph, one O(N+E) streaming pass
|
||||
```
|
||||
|
||||
### Write-time aggregations
|
||||
|
||||
`SUM` / `COUNT` / `AVG` / `MIN` / `MAX` with `GROUP BY` and time windows, maintained incrementally on every write — reads are O(1) lookups, not scans. **[Aggregation guide](docs/guides/aggregation.md)**
|
||||
|
||||
### Import anything
|
||||
|
||||
```javascript
|
||||
await brain.import('customers.csv')
|
||||
await brain.import('sales.xlsx') // every sheet
|
||||
await brain.import('research-paper.pdf') // tables extracted
|
||||
await brain.import('https://api.example.com/data.json')
|
||||
```
|
||||
|
||||
Entities auto-classify on the way in; `brain.extractEntities(text)` exposes the same NER ensemble directly. **[Import guide](docs/guides/import-anything.md)**
|
||||
|
||||
### A filesystem that understands content
|
||||
|
||||
```javascript
|
||||
await brain.vfs.writeFile('/docs/readme.md', 'Project documentation')
|
||||
await brain.vfs.search('React components with hooks') // semantic file search
|
||||
```
|
||||
|
||||
**[VFS quick start](docs/vfs/QUICK_START.md)**
|
||||
|
||||
### Operations-grade by default
|
||||
|
||||
- **Single-writer, many-reader** — an exclusive lock protects the data directory; `Brainy.openReadOnly()` and the `brainy inspect` CLI examine a live brain from another process, safely.
|
||||
- **Self-upgrading data files** — a 7.x brain opens under 8.x and migrates itself behind an observable lock (`getIndexStatus().migration`), with an automatic pre-upgrade backup. No migration scripts.
|
||||
- **No silent wrong answers** — cold-open guards self-heal or throw typed errors (`MetadataIndexNotReadyError`, `GraphIndexNotReadyError`); they never return `[]` for data that exists.
|
||||
|
||||
**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
|
||||
|
||||
## From laptop to hundreds of millions
|
||||
|
||||
Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**:
|
||||
|
||||
```bash
|
||||
# Auto-setup with cloud instance provisioning (RECOMMENDED)
|
||||
brainy cloud setup --email your@email.com
|
||||
|
||||
# Sign up at app.soulcraft.com (free trial)
|
||||
brainy cloud auth # Auto-configures based on your plan
|
||||
npm install @soulcraft/cor
|
||||
```
|
||||
|
||||
```javascript
|
||||
import { BrainyData, Cortex } from '@soulcraft/brainy'
|
||||
// After authentication, augmentations auto-load
|
||||
// No imports needed - they're managed by your account!
|
||||
|
||||
const brain = new BrainyData()
|
||||
const cortex = new Cortex()
|
||||
|
||||
// Add premium augmentations (requires Early Access license)
|
||||
cortex.register(new AIMemory())
|
||||
cortex.register(new AgentCoordinator())
|
||||
|
||||
// Now your AI remembers everything across all sessions!
|
||||
await brain.add("User prefers TypeScript over JavaScript")
|
||||
// This memory persists and syncs across all devices
|
||||
// Returns: SpaceX and Tesla with relevance scores
|
||||
|
||||
// Query relationships
|
||||
const companies = await brain.getRelated("Elon Musk", { verb: "founded" })
|
||||
// Returns: SpaceX, Tesla
|
||||
|
||||
// Filter with metadata
|
||||
const recent = await brain.search("companies", 10, {
|
||||
filter: { founded: { $gte: 2000 } }
|
||||
})
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } })
|
||||
await brain.init() // @soulcraft/cor detected — same code, native engines underneath
|
||||
```
|
||||
|
||||
## 🧩 Augmentation System - Extend Your Brain
|
||||
Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
|
||||
|
||||
Brainy is **100% open source** with a powerful augmentation system. Choose what you need:
|
||||
Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both.
|
||||
|
||||
### 🆓 **Built-in Augmentations** (Always Free)
|
||||
```javascript
|
||||
import { NeuralImport } from '@soulcraft/brainy'
|
||||
## Performance
|
||||
|
||||
// AI-powered data understanding - included in every install
|
||||
const neural = new NeuralImport(brain)
|
||||
await neural.neuralImport('data.csv') // Automatically extracts entities & relationships
|
||||
```
|
||||
- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
|
||||
- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
|
||||
- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
|
||||
|
||||
**Included augmentations:**
|
||||
- ✅ **Neural Import** - AI understands your data structure
|
||||
- ✅ **Basic Memory** - Persistent storage
|
||||
- ✅ **Simple Search** - Text and vector search
|
||||
- ✅ **Graph Traversal** - Relationship queries
|
||||
## Use cases
|
||||
|
||||
### 🌟 **Community Augmentations** (Free, Open Source)
|
||||
```bash
|
||||
npm install brainy-sentiment # Community created
|
||||
npm install brainy-translate # Community maintained
|
||||
```
|
||||
**AI agent memory** — persistent semantic recall with relationship tracking · **Knowledge bases** — auto-linking and meaning-aware navigation · **Semantic search** over codebases, documents, media · **Enterprise data** — CRM, catalogs, institutional memory · **Games & simulations** — worlds and characters that remember.
|
||||
|
||||
```javascript
|
||||
import { SentimentAnalyzer } from 'brainy-sentiment'
|
||||
import { Translator } from 'brainy-translate'
|
||||
## Documentation
|
||||
|
||||
cortex.register(new SentimentAnalyzer()) // Analyze emotions
|
||||
cortex.register(new Translator()) // Multi-language support
|
||||
```
|
||||
| Start | Core | Going deeper |
|
||||
|---|---|---|
|
||||
| [Brainy explained simply](docs/eli5.md) | [API reference](docs/api/README.md) | [Architecture overview](docs/architecture/overview.md) |
|
||||
| [Installation](docs/guides/installation.md) | [Data model](docs/DATA_MODEL.md) | [Consistency model](docs/concepts/consistency-model.md) |
|
||||
| [Natural-language queries](docs/guides/natural-language.md) | [Query operators](docs/QUERY_OPERATORS.md) | [Multi-process model](docs/concepts/multi-process.md) |
|
||||
| | [Find system](docs/FIND_SYSTEM.md) | [Scaling](docs/SCALING.md) |
|
||||
|
||||
**Popular community augmentations:**
|
||||
- 🎭 Sentiment Analysis
|
||||
- 🌍 Translation (50+ languages)
|
||||
- 📧 Email Parser
|
||||
- 🔗 URL Extractor
|
||||
- 📊 Data Visualizer
|
||||
- 🎨 Image Understanding
|
||||
## Requirements
|
||||
|
||||
### ☁️ **Brain Cloud** (Optional Add-On)
|
||||
🌟 **Brainy works perfectly without this!** Brain Cloud adds team features:
|
||||
**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
|
||||
|
||||
```javascript
|
||||
// Brain Cloud features are in the main package
|
||||
// But require API key to activate cloud services
|
||||
import { BrainyVectorDB } from '@soulcraft/brainy'
|
||||
## Contributing & license
|
||||
|
||||
// Activate Brain Cloud features with API key
|
||||
const brain = new BrainyVectorDB({
|
||||
cloud: { apiKey: process.env.BRAIN_CLOUD_KEY } // Optional
|
||||
})
|
||||
|
||||
cortex.register(aiMemory) // AI remembers everything
|
||||
```
|
||||
|
||||
**AI Memory & Coordination:**
|
||||
- 🧠 **AI Memory** - Persistent across sessions
|
||||
- 🤝 **Agent Coordinator** - Multi-agent handoffs
|
||||
- 👥 **Team Sync** - Real-time collaboration
|
||||
- 💾 **Cloud Backup** - Automatic backups
|
||||
|
||||
**Enterprise Connectors:**
|
||||
- 📝 **Notion Sync** - Bidirectional sync
|
||||
- 💼 **Salesforce** - CRM integration
|
||||
- 📊 **Airtable** - Database sync
|
||||
- 🔄 **Postgres** - Real-time replication
|
||||
- 🏢 **Slack** - Team knowledge base
|
||||
|
||||
### 🎮 **Try Online** (Free Playground)
|
||||
Test Brainy instantly without installing:
|
||||
|
||||
```javascript
|
||||
// Visit soulcraft.com/demo
|
||||
// No signup required - just start coding!
|
||||
// Perfect for:
|
||||
// - Testing Brainy before installing
|
||||
// - Prototyping ideas quickly
|
||||
// - Learning the API
|
||||
// - Sharing examples with others
|
||||
```
|
||||
|
||||
**[→ Try Live Demos](https://soulcraft.com/demo)** - Multiple interactive demos showcasing Brainy's capabilities
|
||||
|
||||
### ☁️ **Brain Cloud** (Managed Service)
|
||||
For teams that want zero-ops:
|
||||
|
||||
```javascript
|
||||
// Connect to Brain Cloud - your brain in the cloud
|
||||
await brain.connect('brain-cloud.soulcraft.com', {
|
||||
instance: 'my-team-brain',
|
||||
apiKey: process.env.BRAIN_CLOUD_KEY
|
||||
})
|
||||
|
||||
// Now your brain persists across:
|
||||
// - Multiple developers
|
||||
// - Different environments
|
||||
// - AI agents
|
||||
// - Sessions
|
||||
```
|
||||
|
||||
**Brain Cloud features:**
|
||||
- 🔄 Auto-sync across team
|
||||
- 💾 Managed backups
|
||||
- 🚀 Auto-scaling
|
||||
- 🔒 Enterprise security
|
||||
- 📊 Analytics dashboard
|
||||
- 🤖 Multi-agent coordination
|
||||
|
||||
## 📝 Create Your Own Augmentation
|
||||
|
||||
### We ❤️ Open Source
|
||||
|
||||
**Brainy will ALWAYS be open source.** We believe in:
|
||||
- 🌍 Community first
|
||||
- 🔓 No vendor lock-in
|
||||
- 🎁 Free forever core
|
||||
- 🤝 Sustainable open source
|
||||
|
||||
### Build & Share Your Augmentation
|
||||
|
||||
```typescript
|
||||
import { ISenseAugmentation } from '@soulcraft/brainy'
|
||||
|
||||
export class MovieRecommender implements ISenseAugmentation {
|
||||
name = 'movie-recommender'
|
||||
description = 'AI-powered movie recommendations'
|
||||
enabled = true
|
||||
|
||||
async processRawData(data: string) {
|
||||
// Your recommendation logic
|
||||
const movies = await this.analyzePreferences(data)
|
||||
|
||||
return {
|
||||
success: true,
|
||||
data: {
|
||||
nouns: movies.map(m => m.title),
|
||||
verbs: movies.map(m => `similar_to:${m.genre}`),
|
||||
metadata: { genres: movies.map(m => m.genre) }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Share with the community:**
|
||||
```bash
|
||||
npm publish brainy-movie-recommender
|
||||
```
|
||||
|
||||
**Earn from your creation:**
|
||||
- 💚 Keep it free (we'll promote it!)
|
||||
- 💰 Sell licenses (we'll help distribute!)
|
||||
- 🤝 Join our partner program
|
||||
|
||||
## 🎯 Real-World Examples
|
||||
|
||||
### Customer Support Bot with Memory
|
||||
```javascript
|
||||
// Your bot remembers every interaction
|
||||
await brain.add({
|
||||
customerId: "user_123",
|
||||
issue: "Password reset",
|
||||
resolved: true,
|
||||
date: new Date()
|
||||
})
|
||||
|
||||
// Next interaction knows the history
|
||||
const history = await brain.search(`customer user_123`, 10)
|
||||
// Bot says: "I see you had a password issue last week. All working now?"
|
||||
```
|
||||
|
||||
### Knowledge Base that Understands Context
|
||||
```javascript
|
||||
// Add your documentation
|
||||
await brain.add("To deploy Brainy, run npm install @soulcraft/brainy")
|
||||
await brain.add("Brainy requires Node.js 24.4.1 or higher")
|
||||
await brain.add("For production, use Brain Cloud for scaling")
|
||||
|
||||
// Natural language queries work
|
||||
const answer = await brain.search("how do I deploy to production?")
|
||||
// Returns relevant docs about Brain Cloud and scaling
|
||||
```
|
||||
|
||||
### Multi-Agent AI Systems
|
||||
```javascript
|
||||
// Agents share the same brain
|
||||
const agentBrain = new BrainyData({ instance: 'shared-brain' })
|
||||
|
||||
// Sales Agent adds knowledge
|
||||
await agentBrain.add("Customer interested in enterprise plan")
|
||||
|
||||
// Support Agent sees it instantly
|
||||
const context = await agentBrain.search("customer plan interest")
|
||||
|
||||
// Marketing Agent learns from both
|
||||
const insights = await agentBrain.getRelated("enterprise plan")
|
||||
```
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
Your App
|
||||
↓
|
||||
BrainyData (The Brain)
|
||||
↓
|
||||
Cortex (Orchestrator)
|
||||
↓
|
||||
Augmentations (Capabilities)
|
||||
├── Built-in (Free)
|
||||
├── Community (Free)
|
||||
├── Premium (Paid)
|
||||
└── Custom (Yours)
|
||||
```
|
||||
|
||||
## 💡 Core Features
|
||||
|
||||
### 🔍 Multi-Dimensional Search
|
||||
- **Vector**: Semantic similarity (meaning-based)
|
||||
- **Graph**: Relationship traversal (connection-based)
|
||||
- **Faceted**: Metadata filtering (property-based)
|
||||
- **Hybrid**: All combined (maximum power)
|
||||
|
||||
### ⚡ Performance
|
||||
- **Speed**: 100,000+ ops/second
|
||||
- **Scale**: Millions of embeddings
|
||||
- **Memory**: ~100MB for 1M vectors
|
||||
- **Latency**: <10ms searches
|
||||
|
||||
### 🔒 Production Ready
|
||||
- **Encryption**: End-to-end available
|
||||
- **Persistence**: Multiple storage backends
|
||||
- **Reliability**: 99.9% uptime in production
|
||||
- **Security**: SOC2 compliant architecture
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### Getting Started
|
||||
- [**Quick Start Guide**](docs/getting-started/quick-start.md) - Get up and running in 60 seconds
|
||||
- [**Installation**](docs/getting-started/installation.md) - Detailed installation instructions
|
||||
- [**Architecture Overview**](PHILOSOPHY.md) - Design principles and philosophy
|
||||
|
||||
### Core Documentation
|
||||
- [**API Reference**](docs/api/BRAINY-API-REFERENCE.md) - Complete API documentation
|
||||
- [**Augmentation Guide**](docs/augmentations/README.md) - Build your own augmentations
|
||||
- [**CLI Reference**](docs/brainy-cli.md) - Command-line interface
|
||||
- [**All Documentation**](docs/README.md) - Browse all docs
|
||||
|
||||
### Guides
|
||||
- [**Search & Metadata**](docs/user-guides/SEARCH_AND_METADATA_GUIDE.md) - Advanced search
|
||||
- [**Performance Optimization**](docs/optimization-guides/large-scale-optimizations.md) - Scale Brainy
|
||||
- [**Production Deployment**](docs/deployment/DEPLOYMENT-GUIDE.md) - Deploy to production
|
||||
- [**Contributing Guidelines**](CONTRIBUTING.md) - Join the community
|
||||
|
||||
## 🤝 Our Promise to the Community
|
||||
|
||||
1. **Brainy core will ALWAYS be open source** (MIT License)
|
||||
2. **No feature will ever move from free to paid**
|
||||
3. **Community augmentations always welcome**
|
||||
4. **We'll actively promote community creators**
|
||||
5. **Commercial success funds open source development**
|
||||
|
||||
## 🙏 Join the Movement
|
||||
|
||||
### Ways to Contribute
|
||||
- 🐛 Report bugs
|
||||
- 💡 Suggest features
|
||||
- 🔧 Submit PRs
|
||||
- 📦 Create augmentations
|
||||
- 📖 Improve docs
|
||||
- ⭐ Star the repo
|
||||
- 📢 Spread the word
|
||||
|
||||
### Get Help & Connect
|
||||
- 💬 [Discord Community](https://discord.gg/brainy)
|
||||
- 🐦 [Twitter Updates](https://twitter.com/soulcraftlabs)
|
||||
- 📧 [Email Support](mailto:support@soulcraft.com)
|
||||
- 🎓 [Video Tutorials](https://youtube.com/@soulcraft)
|
||||
|
||||
## 📈 Who's Using Brainy?
|
||||
|
||||
- 🚀 **Startups**: Building AI-first products
|
||||
- 🏢 **Enterprises**: Replacing expensive databases
|
||||
- 🎓 **Researchers**: Exploring knowledge graphs
|
||||
- 👨💻 **Developers**: Creating smart applications
|
||||
- 🤖 **AI Engineers**: Building RAG systems
|
||||
|
||||
## 📄 License
|
||||
|
||||
**MIT License** - Use it anywhere, build anything!
|
||||
|
||||
Premium augmentations available at [soulcraft.com](https://soulcraft.com)
|
||||
|
||||
---
|
||||
|
||||
<div align="center">
|
||||
|
||||
### 🧠⚛️ **Give Your Data a Brain Upgrade**
|
||||
|
||||
**[Get Started](docs/getting-started/quick-start.md)** •
|
||||
**[Examples](examples/)** •
|
||||
**[API Docs](docs/api/BRAINY-API-REFERENCE.md)** •
|
||||
**[Discord](https://discord.gg/brainy)**
|
||||
|
||||
⭐ **Star us on GitHub to support open source AI!** ⭐
|
||||
|
||||
*Created and maintained by [SoulCraft](https://soulcraft.com) • Powered by our amazing open source community*
|
||||
|
||||
**SoulCraft** builds and maintains Brainy as open source (MIT License) because we believe AI infrastructure should be accessible to everyone.
|
||||
|
||||
</div>
|
||||
Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors.
|
||||
|
|
|
|||
2901
RELEASES.md
Normal file
2901
RELEASES.md
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -1,130 +0,0 @@
|
|||
# TensorFlow.js → Transformers.js Migration Analysis
|
||||
|
||||
## 🧹 Cleanup Status
|
||||
|
||||
### ✅ Removed TensorFlow References
|
||||
- [x] Removed `src/types/tensorflowTypes.ts`
|
||||
- [x] Removed `src/types/tensorflow-types/` directory
|
||||
- [x] Updated all console messages and comments
|
||||
- [x] Simplified `textEncoding.ts` (removed Float32Array patching)
|
||||
- [x] Updated `setup.ts` comments and messages
|
||||
- [x] Removed `robustModelLoader.ts` (TensorFlow-specific)
|
||||
|
||||
### 📝 Remaining References (Documentation Only)
|
||||
- `README.md` - Migration explanation (intentional)
|
||||
- `CLAUDE.md` - Migration notes (intentional)
|
||||
- `OFFLINE_MODELS.md` - Comparison info (intentional)
|
||||
- Function names like `applyTensorFlowPatch()` - kept for backward compatibility
|
||||
|
||||
### 🔧 Still Needed
|
||||
- `textEncoding.ts` - TextEncoder/TextDecoder patches (needed for Node.js compatibility)
|
||||
- `setup.ts` - Environment setup (simplified but still needed)
|
||||
|
||||
## 🚀 GPU Acceleration Analysis
|
||||
|
||||
### **Current Status: Limited GPU Support**
|
||||
|
||||
#### **Transformers.js + ONNX Runtime GPU Support:**
|
||||
1. **Node.js**: ✅ GPU acceleration available with ONNX Runtime GPU providers
|
||||
2. **Browser**: ✅ WebGL/WebGPU acceleration available
|
||||
3. **Configuration needed**: Currently not enabled
|
||||
|
||||
#### **How to Enable GPU Acceleration:**
|
||||
|
||||
```typescript
|
||||
// In src/utils/embedding.ts - add GPU configuration
|
||||
const pipeline = await pipeline('feature-extraction', this.options.model, {
|
||||
cache_dir: this.options.cacheDir,
|
||||
local_files_only: this.options.localFilesOnly,
|
||||
dtype: this.options.dtype,
|
||||
// Add GPU acceleration options
|
||||
device: 'gpu', // or 'webgpu' in browser
|
||||
execution_providers: ['cuda', 'webgl'] // ONNX Runtime providers
|
||||
})
|
||||
```
|
||||
|
||||
#### **Current Limitation:**
|
||||
- Our current implementation uses **CPU-only** execution
|
||||
- GPU providers need to be installed separately (`onnxruntime-gpu`)
|
||||
- Would increase package dependencies
|
||||
|
||||
## ⚡ Performance Comparison
|
||||
|
||||
### **Embedding Generation:**
|
||||
|
||||
| Aspect | TensorFlow.js USE | Transformers.js all-MiniLM-L6-v2 |
|
||||
|--------|-------------------|-----------------------------------|
|
||||
| **Model Size** | 525 MB | 87 MB |
|
||||
| **Dimensions** | 512 | 384 |
|
||||
| **Load Time** | ~3-5 seconds | ~1-2 seconds |
|
||||
| **Inference Speed** | Medium (GPU accelerated) | **Faster** (smaller model) |
|
||||
| **Memory Usage** | ~1.5 GB | ~200-400 MB |
|
||||
| **GPU Support** | ✅ Full | ⚠️ Limited (not configured) |
|
||||
|
||||
### **Distance Functions:**
|
||||
|
||||
| Function | Before (TensorFlow GPU) | After (Pure JavaScript) |
|
||||
|----------|------------------------|-------------------------|
|
||||
| **Euclidean** | GPU-accelerated tensors | **Faster** - optimized JS |
|
||||
| **Cosine** | GPU-accelerated tensors | **Faster** - single-pass reduce |
|
||||
| **Manhattan** | GPU-accelerated tensors | **Faster** - optimized JS |
|
||||
| **Dot Product** | GPU-accelerated tensors | **Faster** - optimized JS |
|
||||
|
||||
#### **Why JS Distance Functions Are Faster:**
|
||||
1. **No GPU transfer overhead** - data stays in CPU memory
|
||||
2. **Optimized for small vectors** - 384 dims vs GPU batch processing
|
||||
3. **Node.js 23.11+ optimizations** - enhanced array methods
|
||||
4. **Single-pass calculations** - reduce functions are highly optimized
|
||||
|
||||
### **Search Performance:**
|
||||
|
||||
| Component | Before | After | Change |
|
||||
|-----------|--------|--------|---------|
|
||||
| **Vector Generation** | Slow (large model) | **Faster** ⚡ |
|
||||
| **Distance Calculations** | GPU overhead | **Faster** ⚡ |
|
||||
| **Memory Usage** | High (GPU memory) | **Lower** 📉 |
|
||||
| **Cold Start** | Slow (model load) | **Faster** ⚡ |
|
||||
|
||||
## 🎯 Overall Performance Summary
|
||||
|
||||
### **🟢 Significantly Faster:**
|
||||
- **Model Loading**: 87 MB vs 525 MB (5x faster)
|
||||
- **Cold Start**: No GPU initialization overhead
|
||||
- **Distance Functions**: Pure JS faster than GPU for small vectors
|
||||
- **Memory Efficiency**: ~75% less memory usage
|
||||
|
||||
### **🟡 Similar Performance:**
|
||||
- **Inference Speed**: Smaller model compensates for CPU-only
|
||||
- **Batch Processing**: Similar for typical use cases
|
||||
|
||||
### **🔴 Potential Slower:**
|
||||
- **Large Batch Inference**: GPU would win for 1000+ texts at once
|
||||
- **Concurrent Users**: GPU parallel processing advantage lost
|
||||
|
||||
## 🔧 Recommendations
|
||||
|
||||
### **Current Setup (Optimal for Most Cases):**
|
||||
Keep the current CPU-only implementation because:
|
||||
1. **Simpler deployment** - no GPU drivers needed
|
||||
2. **Better for typical usage** - small batches of text
|
||||
3. **Lower memory footprint**
|
||||
4. **Faster cold starts**
|
||||
|
||||
### **Future GPU Option (If Needed):**
|
||||
Add GPU acceleration as an optional feature:
|
||||
```typescript
|
||||
const embedding = new TransformerEmbedding({
|
||||
accelerated: true, // Enable GPU if available
|
||||
device: 'auto' // Auto-detect best device
|
||||
})
|
||||
```
|
||||
|
||||
## ✅ Final Answer
|
||||
|
||||
1. **TensorFlow References**: 99% removed (only docs remain)
|
||||
2. **GPU Acceleration**: Currently CPU-only, but can be added
|
||||
3. **Performance**: **Overall faster** for typical usage patterns
|
||||
- Faster model loading, distance functions, and memory efficiency
|
||||
- Only slower for very large batch processing (rare use case)
|
||||
|
||||
The migration delivers better performance for real-world usage while dramatically reducing complexity and dependencies.
|
||||
|
|
@ -1,10 +1,9 @@
|
|||
{
|
||||
"_name_or_path": "sentence-transformers/all-MiniLM-L6-v2",
|
||||
"_name_or_path": "nreimers/MiniLM-L6-H384-uncased",
|
||||
"architectures": [
|
||||
"BertModel"
|
||||
],
|
||||
"attention_probs_dropout_prob": 0.1,
|
||||
"classifier_dropout": null,
|
||||
"gradient_checkpointing": false,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout_prob": 0.1,
|
||||
|
|
@ -18,7 +17,7 @@
|
|||
"num_hidden_layers": 6,
|
||||
"pad_token_id": 0,
|
||||
"position_embedding_type": "absolute",
|
||||
"transformers_version": "4.29.2",
|
||||
"transformers_version": "4.8.2",
|
||||
"type_vocab_size": 2,
|
||||
"use_cache": true,
|
||||
"vocab_size": 30522
|
||||
Binary file not shown.
1
assets/models/all-MiniLM-L6-v2/tokenizer.json
Normal file
1
assets/models/all-MiniLM-L6-v2/tokenizer.json
Normal file
File diff suppressed because one or more lines are too long
564
bin/brainy-interactive.js
Normal file
564
bin/brainy-interactive.js
Normal file
|
|
@ -0,0 +1,564 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Brainy Interactive Mode
|
||||
*
|
||||
* Professional, guided CLI experience for beginners
|
||||
*/
|
||||
|
||||
import { program } from 'commander'
|
||||
import { Brainy } from '../dist/index.js'
|
||||
import chalk from 'chalk'
|
||||
import inquirer from 'inquirer'
|
||||
import ora from 'ora'
|
||||
import Table from 'cli-table3'
|
||||
import boxen from 'boxen'
|
||||
|
||||
// Professional color scheme
|
||||
const colors = {
|
||||
primary: chalk.hex('#3A5F4A'), // Teal (from logo)
|
||||
success: chalk.hex('#2D4A3A'), // Deep teal
|
||||
info: chalk.hex('#4A6B5A'), // Medium teal
|
||||
warning: chalk.hex('#D67441'), // Orange (from logo)
|
||||
error: chalk.hex('#B85C35'), // Deep orange
|
||||
brain: chalk.hex('#D67441'), // Brain orange
|
||||
cream: chalk.hex('#F5E6A3'), // Cream background
|
||||
dim: chalk.dim,
|
||||
bold: chalk.bold,
|
||||
cyan: chalk.cyan,
|
||||
green: chalk.green,
|
||||
yellow: chalk.yellow,
|
||||
red: chalk.red
|
||||
}
|
||||
|
||||
// Icons for consistent visual language
|
||||
const icons = {
|
||||
brain: '🧠',
|
||||
search: '🔍',
|
||||
add: '➕',
|
||||
delete: '🗑️',
|
||||
update: '🔄',
|
||||
import: '📥',
|
||||
export: '📤',
|
||||
connect: '🔗',
|
||||
question: '❓',
|
||||
success: '✅',
|
||||
error: '❌',
|
||||
warning: '⚠️',
|
||||
info: 'ℹ️',
|
||||
sparkle: '✨',
|
||||
rocket: '🚀',
|
||||
thinking: '🤔',
|
||||
chat: '💬',
|
||||
stats: '📊',
|
||||
config: '⚙️',
|
||||
cloud: '☁️'
|
||||
}
|
||||
|
||||
let brainyInstance = null
|
||||
|
||||
async function getBrainy() {
|
||||
if (!brainyInstance) {
|
||||
const spinner = ora('Initializing Brainy...').start()
|
||||
try {
|
||||
brainyInstance = new Brainy()
|
||||
await brainyInstance.init()
|
||||
spinner.succeed('Brainy initialized')
|
||||
} catch (error) {
|
||||
spinner.fail('Failed to initialize Brainy')
|
||||
console.error(colors.error(error.message))
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
return brainyInstance
|
||||
}
|
||||
|
||||
/**
|
||||
* Professional welcome screen
|
||||
*/
|
||||
function showWelcome() {
|
||||
console.clear()
|
||||
|
||||
const welcomeBox = boxen(
|
||||
colors.primary(`${icons.brain} BRAINY - Neural Intelligence System\n`) +
|
||||
colors.dim('\nYour AI-Powered Second Brain\n') +
|
||||
colors.info('Version 1.6.0'),
|
||||
{
|
||||
padding: 1,
|
||||
margin: 1,
|
||||
borderStyle: 'round',
|
||||
borderColor: 'cyan',
|
||||
textAlignment: 'center'
|
||||
}
|
||||
)
|
||||
|
||||
console.log(welcomeBox)
|
||||
console.log()
|
||||
}
|
||||
|
||||
/**
|
||||
* Main interactive menu
|
||||
*/
|
||||
async function mainMenu() {
|
||||
const { action } = await inquirer.prompt([{
|
||||
type: 'list',
|
||||
name: 'action',
|
||||
message: colors.cyan('What would you like to do?'),
|
||||
choices: [
|
||||
new inquirer.Separator(colors.dim('── Core Operations ──')),
|
||||
{ name: `${icons.add} Add data to your brain`, value: 'add' },
|
||||
{ name: `${icons.search} Search your knowledge`, value: 'search' },
|
||||
{ name: `${icons.chat} Chat with your data`, value: 'chat' },
|
||||
{ name: `${icons.update} Update existing data`, value: 'update' },
|
||||
{ name: `${icons.delete} Delete data`, value: 'delete' },
|
||||
|
||||
new inquirer.Separator(colors.dim('── Advanced Features ──')),
|
||||
{ name: `${icons.connect} Create relationships`, value: 'relate' },
|
||||
{ name: `${icons.import} Import from file/URL`, value: 'import' },
|
||||
{ name: `${icons.export} Export your brain`, value: 'export' },
|
||||
{ name: `${icons.brain} Neural operations`, value: 'neural' },
|
||||
|
||||
new inquirer.Separator(colors.dim('── System ──')),
|
||||
{ name: `${icons.stats} View statistics`, value: 'stats' },
|
||||
{ name: `${icons.config} Configuration`, value: 'config' },
|
||||
{ name: `${icons.cloud} Brain Cloud`, value: 'cloud' },
|
||||
{ name: `${icons.info} Help & Documentation`, value: 'help' },
|
||||
|
||||
new inquirer.Separator(),
|
||||
{ name: 'Exit', value: 'exit' }
|
||||
],
|
||||
pageSize: 20
|
||||
}])
|
||||
|
||||
return action
|
||||
}
|
||||
|
||||
/**
|
||||
* Neural operations submenu
|
||||
*/
|
||||
async function neuralMenu() {
|
||||
const { operation } = await inquirer.prompt([{
|
||||
type: 'list',
|
||||
name: 'operation',
|
||||
message: colors.cyan('Select neural operation:'),
|
||||
choices: [
|
||||
{ name: `${icons.brain} Calculate similarity`, value: 'similar' },
|
||||
{ name: `${icons.search} Find clusters`, value: 'cluster' },
|
||||
{ name: `${icons.connect} Find related items`, value: 'related' },
|
||||
{ name: `${icons.thinking} Build hierarchy`, value: 'hierarchy' },
|
||||
{ name: `${icons.rocket} Find semantic path`, value: 'path' },
|
||||
{ name: `${icons.warning} Detect outliers`, value: 'outliers' },
|
||||
{ name: `${icons.sparkle} Generate visualization`, value: 'visualize' },
|
||||
new inquirer.Separator(),
|
||||
{ name: '← Back to main menu', value: 'back' }
|
||||
]
|
||||
}])
|
||||
|
||||
return operation
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute commands with beautiful feedback
|
||||
*/
|
||||
async function executeCommand(command) {
|
||||
const brain = await getBrainy()
|
||||
|
||||
switch (command) {
|
||||
case 'add':
|
||||
await interactiveAdd(brain)
|
||||
break
|
||||
|
||||
case 'search':
|
||||
await interactiveSearch(brain)
|
||||
break
|
||||
|
||||
case 'chat':
|
||||
await interactiveChat(brain)
|
||||
break
|
||||
|
||||
case 'update':
|
||||
await interactiveUpdate(brain)
|
||||
break
|
||||
|
||||
case 'delete':
|
||||
await interactiveDelete(brain)
|
||||
break
|
||||
|
||||
case 'relate':
|
||||
await interactiveRelate(brain)
|
||||
break
|
||||
|
||||
case 'import':
|
||||
await interactiveImport(brain)
|
||||
break
|
||||
|
||||
case 'export':
|
||||
await interactiveExport(brain)
|
||||
break
|
||||
|
||||
case 'neural':
|
||||
const neuralOp = await neuralMenu()
|
||||
if (neuralOp !== 'back') {
|
||||
await executeNeuralOperation(neuralOp, brain)
|
||||
}
|
||||
break
|
||||
|
||||
case 'stats':
|
||||
await showStatistics(brain)
|
||||
break
|
||||
|
||||
case 'config':
|
||||
await interactiveConfig(brain)
|
||||
break
|
||||
|
||||
case 'cloud':
|
||||
await showCloudInfo()
|
||||
break
|
||||
|
||||
case 'help':
|
||||
await showHelp()
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Interactive add with rich prompts
|
||||
*/
|
||||
async function interactiveAdd(brain) {
|
||||
console.log(colors.primary(`\n${icons.add} Add Data\n`))
|
||||
|
||||
const { inputType } = await inquirer.prompt([{
|
||||
type: 'list',
|
||||
name: 'inputType',
|
||||
message: 'How would you like to add data?',
|
||||
choices: [
|
||||
{ name: 'Type or paste text', value: 'text' },
|
||||
{ name: 'Multi-line editor', value: 'editor' },
|
||||
{ name: 'JSON object', value: 'json' },
|
||||
{ name: 'Import from clipboard', value: 'clipboard' }
|
||||
]
|
||||
}])
|
||||
|
||||
let data = ''
|
||||
|
||||
switch (inputType) {
|
||||
case 'text':
|
||||
const { text } = await inquirer.prompt([{
|
||||
type: 'input',
|
||||
name: 'text',
|
||||
message: 'Enter your data:',
|
||||
validate: input => input.trim() ? true : 'Please enter some data'
|
||||
}])
|
||||
data = text
|
||||
break
|
||||
|
||||
case 'editor':
|
||||
const { editorText } = await inquirer.prompt([{
|
||||
type: 'editor',
|
||||
name: 'editorText',
|
||||
message: 'Enter your data (opens editor):',
|
||||
postfix: '.md'
|
||||
}])
|
||||
data = editorText
|
||||
break
|
||||
|
||||
case 'json':
|
||||
const { jsonText } = await inquirer.prompt([{
|
||||
type: 'editor',
|
||||
name: 'jsonText',
|
||||
message: 'Enter JSON data:',
|
||||
postfix: '.json',
|
||||
default: '{\n \n}',
|
||||
validate: input => {
|
||||
try {
|
||||
JSON.parse(input)
|
||||
return true
|
||||
} catch (e) {
|
||||
return `Invalid JSON: ${e.message}`
|
||||
}
|
||||
}
|
||||
}])
|
||||
data = jsonText
|
||||
break
|
||||
}
|
||||
|
||||
// Optional metadata
|
||||
const { addMetadata } = await inquirer.prompt([{
|
||||
type: 'confirm',
|
||||
name: 'addMetadata',
|
||||
message: 'Would you like to add metadata?',
|
||||
default: false
|
||||
}])
|
||||
|
||||
let metadata = {}
|
||||
if (addMetadata) {
|
||||
const { metadataJson } = await inquirer.prompt([{
|
||||
type: 'editor',
|
||||
name: 'metadataJson',
|
||||
message: 'Enter metadata (JSON):',
|
||||
postfix: '.json',
|
||||
default: '{\n "type": "",\n "tags": [],\n "category": ""\n}',
|
||||
validate: input => {
|
||||
try {
|
||||
JSON.parse(input)
|
||||
return true
|
||||
} catch (e) {
|
||||
return `Invalid JSON: ${e.message}`
|
||||
}
|
||||
}
|
||||
}])
|
||||
metadata = JSON.parse(metadataJson)
|
||||
}
|
||||
|
||||
const spinner = ora('Adding data...').start()
|
||||
try {
|
||||
const id = await brain.add(data, metadata)
|
||||
spinner.succeed(`Added successfully with ID: ${id}`)
|
||||
|
||||
// Show summary
|
||||
console.log(boxen(
|
||||
colors.success(`${icons.success} Data added successfully!\n\n`) +
|
||||
colors.info(`ID: ${id}\n`) +
|
||||
colors.dim(`Size: ${data.length} characters\n`) +
|
||||
(Object.keys(metadata).length > 0 ? colors.dim(`Metadata: ${Object.keys(metadata).join(', ')}`) : ''),
|
||||
{ padding: 1, borderColor: 'green', borderStyle: 'round' }
|
||||
))
|
||||
} catch (error) {
|
||||
spinner.fail('Failed to add data')
|
||||
console.error(colors.error(error.message))
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Interactive search with filters
|
||||
*/
|
||||
async function interactiveSearch(brain) {
|
||||
console.log(colors.primary(`\n${icons.search} Search\n`))
|
||||
|
||||
const { query } = await inquirer.prompt([{
|
||||
type: 'input',
|
||||
name: 'query',
|
||||
message: 'Enter search query:',
|
||||
validate: input => input.trim() ? true : 'Please enter a search query'
|
||||
}])
|
||||
|
||||
// Advanced options
|
||||
const { useFilters } = await inquirer.prompt([{
|
||||
type: 'confirm',
|
||||
name: 'useFilters',
|
||||
message: 'Apply filters?',
|
||||
default: false
|
||||
}])
|
||||
|
||||
let searchOptions = { limit: 10 }
|
||||
|
||||
if (useFilters) {
|
||||
const { limit, threshold } = await inquirer.prompt([
|
||||
{
|
||||
type: 'number',
|
||||
name: 'limit',
|
||||
message: 'Maximum results:',
|
||||
default: 10
|
||||
},
|
||||
{
|
||||
type: 'number',
|
||||
name: 'threshold',
|
||||
message: 'Similarity threshold (0-1):',
|
||||
default: 0.5,
|
||||
validate: input => input >= 0 && input <= 1 ? true : 'Must be between 0 and 1'
|
||||
}
|
||||
])
|
||||
|
||||
searchOptions.limit = limit
|
||||
searchOptions.threshold = threshold
|
||||
}
|
||||
|
||||
const spinner = ora('Searching...').start()
|
||||
try {
|
||||
const results = await brain.search(query, searchOptions.limit, searchOptions)
|
||||
spinner.succeed(`Found ${results.length} results`)
|
||||
|
||||
if (results.length === 0) {
|
||||
console.log(colors.warning('No results found'))
|
||||
} else {
|
||||
// Display results in a table
|
||||
const table = new Table({
|
||||
head: [colors.cyan('ID'), colors.cyan('Content'), colors.cyan('Score')],
|
||||
style: { head: [], border: [] },
|
||||
colWidths: [20, 50, 10]
|
||||
})
|
||||
|
||||
results.forEach(result => {
|
||||
const content = result.content || result.id
|
||||
const truncated = content.length > 47 ? content.substring(0, 47) + '...' : content
|
||||
const score = result.score ? `${(result.score * 100).toFixed(1)}%` : 'N/A'
|
||||
|
||||
table.push([
|
||||
result.id.substring(0, 18),
|
||||
truncated,
|
||||
colors.green(score)
|
||||
])
|
||||
})
|
||||
|
||||
console.log(table.toString())
|
||||
|
||||
// Ask if user wants to see full details
|
||||
const { viewDetails } = await inquirer.prompt([{
|
||||
type: 'confirm',
|
||||
name: 'viewDetails',
|
||||
message: 'View full details of a result?',
|
||||
default: false
|
||||
}])
|
||||
|
||||
if (viewDetails) {
|
||||
const { selectedId } = await inquirer.prompt([{
|
||||
type: 'list',
|
||||
name: 'selectedId',
|
||||
message: 'Select result:',
|
||||
choices: results.map(r => ({
|
||||
name: `${r.id} - ${r.content?.substring(0, 50)}...`,
|
||||
value: r.id
|
||||
}))
|
||||
}])
|
||||
|
||||
const selected = results.find(r => r.id === selectedId)
|
||||
console.log(boxen(
|
||||
colors.cyan('Full Details\n\n') +
|
||||
colors.info(`ID: ${selected.id}\n\n`) +
|
||||
`Content:\n${selected.content}\n\n` +
|
||||
(selected.metadata ? `Metadata:\n${JSON.stringify(selected.metadata, null, 2)}` : ''),
|
||||
{ padding: 1, borderColor: 'cyan', borderStyle: 'round' }
|
||||
))
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
spinner.fail('Search failed')
|
||||
console.error(colors.error(error.message))
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Show statistics with beautiful formatting
|
||||
*/
|
||||
async function showStatistics(brain) {
|
||||
const spinner = ora('Gathering statistics...').start()
|
||||
|
||||
try {
|
||||
const stats = brain.getStats()
|
||||
spinner.succeed('Statistics loaded')
|
||||
|
||||
console.log(boxen(
|
||||
colors.primary(`${icons.stats} Database Statistics\n\n`) +
|
||||
colors.info(`Total Items: ${colors.bold(stats.total || 0)}\n`) +
|
||||
colors.info(`Nouns: ${stats.nounCount || 0}\n`) +
|
||||
colors.info(`Relationships: ${stats.verbCount || 0}\n`) +
|
||||
colors.info(`Metadata Records: ${stats.metadataCount || 0}\n\n`) +
|
||||
colors.dim(`Memory Usage: ${(process.memoryUsage().heapUsed / 1024 / 1024).toFixed(1)} MB`),
|
||||
{
|
||||
padding: 1,
|
||||
borderColor: 'blue',
|
||||
borderStyle: 'round',
|
||||
textAlignment: 'left'
|
||||
}
|
||||
))
|
||||
} catch (error) {
|
||||
spinner.fail('Failed to get statistics')
|
||||
console.error(colors.error(error.message))
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Show help with examples
|
||||
*/
|
||||
async function showHelp() {
|
||||
console.log(boxen(
|
||||
colors.primary(`${icons.info} Brainy Help\n\n`) +
|
||||
colors.cyan('Common Commands:\n') +
|
||||
colors.dim(`
|
||||
brainy add "text" Add data
|
||||
brainy search "query" Search your brain
|
||||
brainy chat Interactive AI chat
|
||||
brainy status View statistics
|
||||
brainy help This help menu
|
||||
|
||||
`) +
|
||||
colors.cyan('Interactive Mode:\n') +
|
||||
colors.dim(`
|
||||
brainy Start interactive mode
|
||||
brainy -i Alternative interactive mode
|
||||
|
||||
`) +
|
||||
colors.cyan('Advanced Features:\n') +
|
||||
colors.dim(`
|
||||
brainy similar a b Calculate similarity
|
||||
brainy cluster Find semantic clusters
|
||||
brainy export Export your data
|
||||
brainy cloud Brain Cloud features
|
||||
`),
|
||||
{ padding: 1, borderColor: 'yellow', borderStyle: 'round' }
|
||||
))
|
||||
|
||||
const { learnMore } = await inquirer.prompt([{
|
||||
type: 'confirm',
|
||||
name: 'learnMore',
|
||||
message: 'View detailed documentation?',
|
||||
default: false
|
||||
}])
|
||||
|
||||
if (learnMore) {
|
||||
console.log(colors.info('\nDocumentation: https://github.com/TimeSoul/brainy'))
|
||||
console.log(colors.info('Enterprise features: Coming in future releases'))
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Main interactive loop
|
||||
*/
|
||||
async function main() {
|
||||
showWelcome()
|
||||
|
||||
let running = true
|
||||
while (running) {
|
||||
const action = await mainMenu()
|
||||
|
||||
if (action === 'exit') {
|
||||
console.log(colors.success(`\n${icons.success} Thank you for using Brainy!\n`))
|
||||
running = false
|
||||
} else {
|
||||
await executeCommand(action)
|
||||
|
||||
// Pause before returning to menu
|
||||
await inquirer.prompt([{
|
||||
type: 'input',
|
||||
name: 'continue',
|
||||
message: colors.dim('\nPress Enter to continue...'),
|
||||
prefix: ''
|
||||
}])
|
||||
}
|
||||
}
|
||||
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
// Handle errors gracefully
|
||||
process.on('unhandledRejection', (error) => {
|
||||
console.error(colors.error(`\n${icons.error} Unexpected error:`))
|
||||
console.error(colors.red(error.message))
|
||||
process.exit(1)
|
||||
})
|
||||
|
||||
// Handle Ctrl+C gracefully
|
||||
process.on('SIGINT', () => {
|
||||
console.log(colors.info(`\n\n${icons.info} Exiting Brainy...`))
|
||||
process.exit(0)
|
||||
})
|
||||
|
||||
// Run if called directly
|
||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||
main().catch(error => {
|
||||
console.error(colors.error('Fatal error:'), error)
|
||||
process.exit(1)
|
||||
})
|
||||
}
|
||||
|
||||
export { main as startInteractiveMode }
|
||||
82
bin/brainy-minimal.js
Executable file
82
bin/brainy-minimal.js
Executable file
|
|
@ -0,0 +1,82 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Brainy CLI - Minimal Version (Conversation Commands Only)
|
||||
*
|
||||
* This is a temporary minimal CLI that only includes working conversation commands
|
||||
* Full CLI will be restored in version 3.20.0
|
||||
*/
|
||||
|
||||
import { Command } from 'commander'
|
||||
import { readFileSync } from 'fs'
|
||||
import { dirname, join } from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url))
|
||||
const packageJson = JSON.parse(readFileSync(join(__dirname, '..', 'package.json'), 'utf8'))
|
||||
|
||||
const program = new Command()
|
||||
|
||||
program
|
||||
.name('brainy')
|
||||
.description('🧠 Brainy - Infinite Agent Memory')
|
||||
.version(packageJson.version)
|
||||
|
||||
// Dynamically load conversation command
|
||||
const conversationCommand = await import('../dist/cli/commands/conversation.js').then(m => m.default)
|
||||
|
||||
program
|
||||
.command('conversation')
|
||||
.alias('conv')
|
||||
.description('💬 Infinite agent memory and context management')
|
||||
.addCommand(
|
||||
new Command('setup')
|
||||
.description('Set up MCP server for Claude Code integration')
|
||||
.action(async () => {
|
||||
await conversationCommand.handler({ action: 'setup', _: [] })
|
||||
})
|
||||
)
|
||||
.addCommand(
|
||||
new Command('remove')
|
||||
.description('Remove MCP server and clean up')
|
||||
.action(async () => {
|
||||
await conversationCommand.handler({ action: 'remove', _: [] })
|
||||
})
|
||||
)
|
||||
.addCommand(
|
||||
new Command('search')
|
||||
.description('Search messages across conversations')
|
||||
.requiredOption('-q, --query <query>', 'Search query')
|
||||
.option('-c, --conversation-id <id>', 'Filter by conversation')
|
||||
.option('-r, --role <role>', 'Filter by role')
|
||||
.option('-l, --limit <number>', 'Maximum results', '10')
|
||||
.action(async (options) => {
|
||||
await conversationCommand.handler({ action: 'search', ...options, _: [] })
|
||||
})
|
||||
)
|
||||
.addCommand(
|
||||
new Command('context')
|
||||
.description('Get relevant context for a query')
|
||||
.requiredOption('-q, --query <query>', 'Context query')
|
||||
.option('-l, --limit <number>', 'Maximum messages', '10')
|
||||
.action(async (options) => {
|
||||
await conversationCommand.handler({ action: 'context', ...options, _: [] })
|
||||
})
|
||||
)
|
||||
.addCommand(
|
||||
new Command('thread')
|
||||
.description('Get full conversation thread')
|
||||
.requiredOption('-c, --conversation-id <id>', 'Conversation ID')
|
||||
.action(async (options) => {
|
||||
await conversationCommand.handler({ action: 'thread', ...options, _: [] })
|
||||
})
|
||||
)
|
||||
.addCommand(
|
||||
new Command('stats')
|
||||
.description('Show conversation statistics')
|
||||
.action(async () => {
|
||||
await conversationCommand.handler({ action: 'stats', _: [] })
|
||||
})
|
||||
)
|
||||
|
||||
program.parse(process.argv)
|
||||
18
bin/brainy-ts.js
Normal file
18
bin/brainy-ts.js
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Modern TypeScript CLI Runner
|
||||
*
|
||||
* This is the entry point after npm install @soulcraft/brainy
|
||||
* It runs the compiled TypeScript CLI code
|
||||
*/
|
||||
|
||||
// Use the compiled TypeScript CLI
|
||||
import('../dist/cli/index.js').catch(err => {
|
||||
// Fallback to legacy CLI if new one isn't built yet
|
||||
import('./brainy.js').catch(() => {
|
||||
console.error('Error: CLI not properly built. Please reinstall the package.')
|
||||
console.error(err)
|
||||
process.exit(1)
|
||||
})
|
||||
})
|
||||
1076
bin/brainy.js
1076
bin/brainy.js
File diff suppressed because it is too large
Load diff
|
|
@ -1,5 +0,0 @@
|
|||
{
|
||||
"brainCloudCustomerId": "demo-test-auto",
|
||||
"brainCloudUrl": "https://brain-cloud.dpsifr.workers.dev",
|
||||
"lastConnected": "2025-08-10T00:59:13.198Z"
|
||||
}
|
||||
|
|
@ -1,320 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Brain Cloud MCP Server
|
||||
* Connects Claude to Brain Cloud for persistent AI memory
|
||||
*/
|
||||
|
||||
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
|
||||
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
|
||||
import {
|
||||
CallToolRequestSchema,
|
||||
ListToolsRequestSchema,
|
||||
Tool,
|
||||
} from '@modelcontextprotocol/sdk/types.js';
|
||||
// Use native fetch (available in Node.js 18+)
|
||||
|
||||
// Configuration from environment
|
||||
const CUSTOMER_ID = process.env.CUSTOMER_ID || 'demo-test-auto';
|
||||
const BRAIN_CLOUD_URL = process.env.BRAIN_CLOUD_URL || 'https://api.soulcraft.com/brain-cloud';
|
||||
|
||||
class BrainCloudMCPServer {
|
||||
constructor() {
|
||||
this.server = new Server(
|
||||
{
|
||||
name: 'brain-cloud',
|
||||
version: '1.0.0',
|
||||
},
|
||||
{
|
||||
capabilities: {
|
||||
tools: {},
|
||||
},
|
||||
}
|
||||
);
|
||||
|
||||
this.setupToolHandlers();
|
||||
this.setupErrorHandling();
|
||||
}
|
||||
|
||||
setupErrorHandling() {
|
||||
this.server.onerror = (error) => {
|
||||
console.error('[MCP Error]', error);
|
||||
};
|
||||
|
||||
process.on('SIGINT', async () => {
|
||||
await this.server.close();
|
||||
process.exit(0);
|
||||
});
|
||||
}
|
||||
|
||||
setupToolHandlers() {
|
||||
// List available tools
|
||||
this.server.setRequestHandler(ListToolsRequestSchema, async () => {
|
||||
return {
|
||||
tools: [
|
||||
{
|
||||
name: 'remember',
|
||||
description: 'Store a memory in Brain Cloud for persistent recall across sessions',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
content: {
|
||||
type: 'string',
|
||||
description: 'The information to remember'
|
||||
},
|
||||
context: {
|
||||
type: 'string',
|
||||
description: 'Additional context or tags for the memory'
|
||||
}
|
||||
},
|
||||
required: ['content']
|
||||
}
|
||||
},
|
||||
{
|
||||
name: 'recall',
|
||||
description: 'Search and retrieve memories from Brain Cloud',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
query: {
|
||||
type: 'string',
|
||||
description: 'What to search for in memories'
|
||||
},
|
||||
limit: {
|
||||
type: 'number',
|
||||
description: 'Number of memories to retrieve (default: 10)'
|
||||
}
|
||||
},
|
||||
required: ['query']
|
||||
}
|
||||
},
|
||||
{
|
||||
name: 'get_all_memories',
|
||||
description: 'Get all stored memories from Brain Cloud',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
limit: {
|
||||
type: 'number',
|
||||
description: 'Number of memories to retrieve (default: 50)'
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
name: 'brain_cloud_status',
|
||||
description: 'Check Brain Cloud connection and memory statistics',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {}
|
||||
}
|
||||
}
|
||||
]
|
||||
};
|
||||
});
|
||||
|
||||
// Handle tool calls
|
||||
this.server.setRequestHandler(CallToolRequestSchema, async (request) => {
|
||||
const { name, arguments: args } = request.params;
|
||||
|
||||
try {
|
||||
switch (name) {
|
||||
case 'remember':
|
||||
return await this.storeMemory(args.content, args.context);
|
||||
|
||||
case 'recall':
|
||||
return await this.searchMemories(args.query, args.limit || 10);
|
||||
|
||||
case 'get_all_memories':
|
||||
return await this.getAllMemories(args.limit || 50);
|
||||
|
||||
case 'brain_cloud_status':
|
||||
return await this.getStatus();
|
||||
|
||||
default:
|
||||
throw new Error(`Unknown tool: ${name}`);
|
||||
}
|
||||
} catch (error) {
|
||||
return {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: `Error: ${error.message}`
|
||||
}],
|
||||
isError: true
|
||||
};
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async storeMemory(content, context = '') {
|
||||
try {
|
||||
const response = await fetch(`${BRAIN_CLOUD_URL}/memory`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'x-customer-id': CUSTOMER_ID
|
||||
},
|
||||
body: JSON.stringify({
|
||||
content,
|
||||
context,
|
||||
source: 'claude-mcp',
|
||||
timestamp: new Date().toISOString()
|
||||
})
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Brain Cloud API error: ${response.status}`);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
|
||||
return {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: `✅ Memory stored successfully!\n\nContent: ${content}\nID: ${result.id || 'generated'}\n\nI'll remember this for future conversations.`
|
||||
}]
|
||||
};
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to store memory: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async searchMemories(query, limit = 10) {
|
||||
try {
|
||||
const response = await fetch(`${BRAIN_CLOUD_URL}/search`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'x-customer-id': CUSTOMER_ID
|
||||
},
|
||||
body: JSON.stringify({
|
||||
query,
|
||||
limit
|
||||
})
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Brain Cloud API error: ${response.status}`);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const memories = result.memories || [];
|
||||
|
||||
if (memories.length === 0) {
|
||||
return {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: `🔍 No memories found for "${query}"\n\nTry a different search term or ask me to remember something first.`
|
||||
}]
|
||||
};
|
||||
}
|
||||
|
||||
const memoryList = memories.map((memory, index) => {
|
||||
const date = new Date(memory.created || memory.timestamp).toLocaleDateString();
|
||||
return `${index + 1}. ${memory.content} (${date})`;
|
||||
}).join('\n');
|
||||
|
||||
return {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: `🧠 Found ${memories.length} memories for "${query}":\n\n${memoryList}\n\nI can use this context to help with your current request!`
|
||||
}]
|
||||
};
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to search memories: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async getAllMemories(limit = 50) {
|
||||
try {
|
||||
const response = await fetch(`${BRAIN_CLOUD_URL}/memories`, {
|
||||
method: 'GET',
|
||||
headers: {
|
||||
'x-customer-id': CUSTOMER_ID
|
||||
}
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Brain Cloud API error: ${response.status}`);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const memories = result.memories || [];
|
||||
|
||||
if (memories.length === 0) {
|
||||
return {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: `🧠 No memories stored yet.\n\nStart a conversation and I'll automatically remember important details!`
|
||||
}]
|
||||
};
|
||||
}
|
||||
|
||||
const recentMemories = memories.slice(0, limit);
|
||||
const memoryList = recentMemories.map((memory, index) => {
|
||||
const date = new Date(memory.created || memory.timestamp).toLocaleDateString();
|
||||
return `${index + 1}. ${memory.content} (${date})`;
|
||||
}).join('\n');
|
||||
|
||||
return {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: `🧠 Your Brain Cloud contains ${memories.length} total memories.\n\nShowing most recent ${recentMemories.length}:\n\n${memoryList}`
|
||||
}]
|
||||
};
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to get memories: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async getStatus() {
|
||||
try {
|
||||
const response = await fetch(`${BRAIN_CLOUD_URL}/health`, {
|
||||
method: 'GET',
|
||||
headers: {
|
||||
'x-customer-id': CUSTOMER_ID
|
||||
}
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Brain Cloud API error: ${response.status}`);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
|
||||
// Get memory count
|
||||
const memoriesResponse = await fetch(`${BRAIN_CLOUD_URL}/memories`, {
|
||||
headers: { 'x-customer-id': CUSTOMER_ID }
|
||||
});
|
||||
|
||||
let memoryCount = 0;
|
||||
if (memoriesResponse.ok) {
|
||||
const memoriesData = await memoriesResponse.json();
|
||||
memoryCount = memoriesData.memories?.length || 0;
|
||||
}
|
||||
|
||||
return {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: `🧠 Brain Cloud Status: ${result.status}\n\n` +
|
||||
`Customer ID: ${CUSTOMER_ID}\n` +
|
||||
`Total Memories: ${memoryCount}\n` +
|
||||
`Last Check: ${new Date().toLocaleString()}\n\n` +
|
||||
`✅ I'm connected and ready to remember everything!`
|
||||
}]
|
||||
};
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to get Brain Cloud status: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async run() {
|
||||
const transport = new StdioServerTransport();
|
||||
await this.server.connect(transport);
|
||||
console.error(`🧠 Brain Cloud MCP Server running for customer: ${CUSTOMER_ID}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Start the server
|
||||
const server = new BrainCloudMCPServer();
|
||||
server.run().catch(console.error);
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2025 Soulcraft Research
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -1,375 +0,0 @@
|
|||
<div align="center">
|
||||
<img src="../brainy.png" alt="Brainy Logo" width="200"/>
|
||||
<br/><br/>
|
||||
|
||||
[](../LICENSE)
|
||||
[](https://nodejs.org/)
|
||||
[](https://www.typescriptlang.org/)
|
||||
[](../CONTRIBUTING.md)
|
||||
|
||||
**Pre-bundled TensorFlow models for maximum reliability with Brainy vector database**
|
||||
|
||||
</div>
|
||||
|
||||
## ✨ Overview
|
||||
|
||||
This package provides offline access to the Universal Sentence Encoder model, eliminating network dependencies and ensuring consistent performance. It's designed as an optional companion to the main `@soulcraft/brainy` package for applications requiring maximum reliability.
|
||||
|
||||
### 🚀 Key Features
|
||||
|
||||
- 🔒 **Maximum Reliability**: Fully offline model loading with zero network dependencies
|
||||
- 📦 **Pre-bundled Models**: Complete Universal Sentence Encoder model (~25MB) included
|
||||
- 🗜️ **Model Compression**: Multiple optimized variants (float16, int8) for different use cases
|
||||
- ⚡ **Performance Optimized**: Use case-specific optimizations for memory and speed
|
||||
- 🛠️ **Easy Integration**: Drop-in replacement for online model loading
|
||||
- 📊 **Comprehensive Metrics**: Detailed model information and performance statistics
|
||||
|
||||
## 🔧 Installation
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/brainy-models
|
||||
```
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Node.js >= 18.0.0
|
||||
- `@soulcraft/brainy` >= 0.33.0
|
||||
|
||||
## 🏁 Quick Start
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```typescript
|
||||
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
|
||||
|
||||
// Create encoder instance
|
||||
const encoder = new BundledUniversalSentenceEncoder({
|
||||
verbose: true,
|
||||
preferCompressed: false
|
||||
})
|
||||
|
||||
// Load the bundled model
|
||||
await encoder.load()
|
||||
|
||||
// Generate embeddings
|
||||
const texts = ['Hello world', 'How are you?', 'Machine learning is amazing']
|
||||
const embeddings = await encoder.embedToArrays(texts)
|
||||
|
||||
console.log(`Generated ${embeddings.length} embeddings of ${embeddings[0].length} dimensions`)
|
||||
|
||||
// Clean up
|
||||
encoder.dispose()
|
||||
```
|
||||
|
||||
### Integration with Brainy
|
||||
|
||||
```typescript
|
||||
import Brainy from '@soulcraft/brainy'
|
||||
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
|
||||
|
||||
// Create bundled encoder
|
||||
const bundledEncoder = new BundledUniversalSentenceEncoder({ verbose: true })
|
||||
await bundledEncoder.load()
|
||||
|
||||
// Use with Brainy (custom integration)
|
||||
const brainy = new Brainy({
|
||||
// Configure Brainy to use the bundled encoder
|
||||
customEmbedding: async (texts) => {
|
||||
return await bundledEncoder.embedToArrays(texts)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Using Compressed Models
|
||||
|
||||
```typescript
|
||||
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
|
||||
|
||||
// Use compressed model for memory-constrained environments
|
||||
const encoder = new BundledUniversalSentenceEncoder({
|
||||
preferCompressed: true,
|
||||
verbose: true
|
||||
})
|
||||
|
||||
await encoder.load()
|
||||
|
||||
// The encoder will automatically use the most appropriate compressed variant
|
||||
const embeddings = await encoder.embedToArrays(['Sample text'])
|
||||
```
|
||||
|
||||
## 📚 API Reference
|
||||
|
||||
### BundledUniversalSentenceEncoder
|
||||
|
||||
Main class for loading and using bundled models.
|
||||
|
||||
#### Constructor
|
||||
|
||||
```typescript
|
||||
new BundledUniversalSentenceEncoder(options)
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `verbose?: boolean` - Enable detailed logging (default: false)
|
||||
- `preferCompressed?: boolean` - Prefer compressed model variants (default: false)
|
||||
|
||||
#### Methods
|
||||
|
||||
##### `load(): Promise<void>`
|
||||
|
||||
Load the bundled model from local files.
|
||||
|
||||
```typescript
|
||||
await encoder.load()
|
||||
```
|
||||
|
||||
##### `embed(texts: string[]): Promise<tf.Tensor2D>`
|
||||
|
||||
Generate embeddings as TensorFlow tensors.
|
||||
|
||||
```typescript
|
||||
const embeddings = await encoder.embed(['Hello world'])
|
||||
// Remember to dispose of tensors when done
|
||||
embeddings.dispose()
|
||||
```
|
||||
|
||||
##### `embedToArrays(texts: string[]): Promise<number[][]>`
|
||||
|
||||
Generate embeddings as JavaScript arrays (automatically disposes tensors).
|
||||
|
||||
```typescript
|
||||
const embeddings = await encoder.embedToArrays(['Hello world'])
|
||||
console.log(embeddings[0].length) // 512
|
||||
```
|
||||
|
||||
##### `getMetadata(): ModelMetadata | null`
|
||||
|
||||
Get model metadata information.
|
||||
|
||||
```typescript
|
||||
const metadata = encoder.getMetadata()
|
||||
console.log(metadata?.dimensions) // 512
|
||||
```
|
||||
|
||||
##### `isLoaded(): boolean`
|
||||
|
||||
Check if the model is loaded.
|
||||
|
||||
```typescript
|
||||
if (encoder.isLoaded()) {
|
||||
// Model is ready to use
|
||||
}
|
||||
```
|
||||
|
||||
##### `getModelInfo(): { inputShape: number[], outputShape: number[] } | null`
|
||||
|
||||
Get model input/output shape information.
|
||||
|
||||
```typescript
|
||||
const info = encoder.getModelInfo()
|
||||
console.log(info?.outputShape) // [-1, 512]
|
||||
```
|
||||
|
||||
##### `dispose(): void`
|
||||
|
||||
Clean up model resources.
|
||||
|
||||
```typescript
|
||||
encoder.dispose()
|
||||
```
|
||||
|
||||
### ModelCompressor
|
||||
|
||||
Utility class for model compression and optimization.
|
||||
|
||||
#### Static Methods
|
||||
|
||||
##### `quantizeModel(modelPath: string, outputPath: string, options?): Promise<void>`
|
||||
|
||||
Compress a model using quantization.
|
||||
|
||||
```typescript
|
||||
import { ModelCompressor } from '@soulcraft/brainy-models'
|
||||
|
||||
await ModelCompressor.quantizeModel(
|
||||
'/path/to/model.json',
|
||||
'/path/to/compressed/model.json',
|
||||
{ dtype: 'int8' }
|
||||
)
|
||||
```
|
||||
|
||||
##### `getModelSize(modelPath: string): Promise<ModelSizeInfo>`
|
||||
|
||||
Get detailed model size information.
|
||||
|
||||
```typescript
|
||||
const sizeInfo = await ModelCompressor.getModelSize('/path/to/model.json')
|
||||
console.log(`Total size: ${sizeInfo.totalSize} bytes`)
|
||||
```
|
||||
|
||||
### Utility Functions
|
||||
|
||||
#### `utils.checkModelsAvailable(): boolean`
|
||||
|
||||
Check if bundled models are available.
|
||||
|
||||
```typescript
|
||||
import { utils } from '@soulcraft/brainy-models'
|
||||
|
||||
if (utils.checkModelsAvailable()) {
|
||||
console.log('Models are ready to use')
|
||||
}
|
||||
```
|
||||
|
||||
#### `utils.listAvailableModels(): string[]`
|
||||
|
||||
List available bundled models.
|
||||
|
||||
```typescript
|
||||
const models = utils.listAvailableModels()
|
||||
console.log('Available models:', models)
|
||||
```
|
||||
|
||||
## 🎯 Model Variants
|
||||
|
||||
The package includes multiple model variants optimized for different use cases:
|
||||
|
||||
### Original (Float32)
|
||||
- **Size**: ~25MB
|
||||
- **Use case**: Maximum accuracy
|
||||
- **Memory**: High
|
||||
- **Speed**: Fast
|
||||
|
||||
### Float16 Compressed
|
||||
- **Size**: ~12-15MB
|
||||
- **Use case**: Balanced performance
|
||||
- **Memory**: Medium
|
||||
- **Speed**: Fast
|
||||
|
||||
### Int8 Quantized
|
||||
- **Size**: ~6-8MB
|
||||
- **Use case**: Memory-constrained environments
|
||||
- **Memory**: Low
|
||||
- **Speed**: Medium
|
||||
|
||||
## ⚙️ Scripts
|
||||
|
||||
The package includes several utility scripts:
|
||||
|
||||
### Download Models
|
||||
|
||||
Download the complete Universal Sentence Encoder model:
|
||||
|
||||
```bash
|
||||
npm run download-models
|
||||
```
|
||||
|
||||
### Compress Models
|
||||
|
||||
Create optimized model variants:
|
||||
|
||||
```bash
|
||||
npm run compress-models
|
||||
```
|
||||
|
||||
### Test Models
|
||||
|
||||
Verify model functionality:
|
||||
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
|
||||
## 🔨 Development
|
||||
|
||||
### Building the Package
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
|
||||
### Creating a Release
|
||||
|
||||
```bash
|
||||
npm run pack
|
||||
```
|
||||
|
||||
## ⚖️ Comparison with Online Loading
|
||||
|
||||
| Feature | Online Loading | Bundled Models |
|
||||
|---------|----------------|----------------|
|
||||
| **Reliability** | Network dependent | 100% offline |
|
||||
| **First load time** | 30-60 seconds | < 1 second |
|
||||
| **Subsequent loads** | Cached (~1 second) | < 1 second |
|
||||
| **Package size** | ~3KB | ~25MB |
|
||||
| **Network required** | Yes (first time) | No |
|
||||
| **Offline support** | Limited | Complete |
|
||||
|
||||
## 💡 Use Cases
|
||||
|
||||
### When to Use Bundled Models
|
||||
|
||||
- ✅ Production applications requiring maximum reliability
|
||||
- ✅ Offline or air-gapped environments
|
||||
- ✅ Applications with strict SLA requirements
|
||||
- ✅ Edge computing and IoT devices
|
||||
- ✅ Development environments with unreliable internet
|
||||
|
||||
### When to Use Online Loading
|
||||
|
||||
- ✅ Development and prototyping
|
||||
- ✅ Applications where package size matters
|
||||
- ✅ Environments with reliable internet connectivity
|
||||
- ✅ Applications that rarely use embeddings
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### Model Not Found Error
|
||||
|
||||
```
|
||||
Error: Bundled model not found. Please run "npm run download-models"
|
||||
```
|
||||
|
||||
**Solution**: Run the download script to fetch the model files:
|
||||
```bash
|
||||
cd node_modules/@soulcraft/brainy-models
|
||||
npm run download-models
|
||||
```
|
||||
|
||||
### Memory Issues
|
||||
|
||||
If you encounter memory issues, try using compressed models:
|
||||
|
||||
```typescript
|
||||
const encoder = new BundledUniversalSentenceEncoder({
|
||||
preferCompressed: true
|
||||
})
|
||||
```
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
For optimal performance:
|
||||
|
||||
1. **Memory-constrained**: Use int8 quantized models
|
||||
2. **Speed-critical**: Use original float32 models
|
||||
3. **Balanced**: Use float16 compressed models
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! Please see the main [Brainy repository](https://github.com/soulcraftlabs/brainy) for contribution guidelines.
|
||||
|
||||
## 💬 Support
|
||||
|
||||
For issues and questions:
|
||||
- [GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)
|
||||
- [Documentation](https://github.com/soulcraftlabs/brainy)
|
||||
103
brainy-models-package/dist/index.d.ts
vendored
103
brainy-models-package/dist/index.d.ts
vendored
|
|
@ -1,103 +0,0 @@
|
|||
/**
|
||||
* @soulcraft/brainy-models
|
||||
*
|
||||
* Pre-bundled TensorFlow models for maximum reliability with Brainy vector database.
|
||||
* This package provides offline access to the Universal Sentence Encoder model,
|
||||
* eliminating network dependencies and ensuring consistent performance.
|
||||
*/
|
||||
import * as tf from '@tensorflow/tfjs';
|
||||
export interface ModelMetadata {
|
||||
name: string;
|
||||
version: string;
|
||||
description: string;
|
||||
dimensions: number;
|
||||
downloadDate: string;
|
||||
source: string;
|
||||
approach: string;
|
||||
modelUrl: string;
|
||||
bundledLocally: boolean;
|
||||
reliability: string;
|
||||
}
|
||||
export interface BundledModelOptions {
|
||||
verbose?: boolean;
|
||||
preferCompressed?: boolean;
|
||||
}
|
||||
/**
|
||||
* Bundled Universal Sentence Encoder for offline use
|
||||
*/
|
||||
export declare class BundledUniversalSentenceEncoder {
|
||||
private model;
|
||||
private metadata;
|
||||
private options;
|
||||
constructor(options?: BundledModelOptions);
|
||||
/**
|
||||
* Load the bundled model from local files
|
||||
*/
|
||||
load(): Promise<void>;
|
||||
/**
|
||||
* Generate embeddings for the given texts
|
||||
*/
|
||||
embed(texts: string[]): Promise<tf.Tensor2D>;
|
||||
/**
|
||||
* Generate embeddings and return as JavaScript arrays
|
||||
*/
|
||||
embedToArrays(texts: string[]): Promise<number[][]>;
|
||||
/**
|
||||
* Get model metadata
|
||||
*/
|
||||
getMetadata(): ModelMetadata | null;
|
||||
/**
|
||||
* Check if the model is loaded
|
||||
*/
|
||||
isLoaded(): boolean;
|
||||
/**
|
||||
* Get model information
|
||||
*/
|
||||
getModelInfo(): {
|
||||
inputShape: number[];
|
||||
outputShape: number[];
|
||||
} | null;
|
||||
/**
|
||||
* Dispose of the model and free memory
|
||||
*/
|
||||
dispose(): void;
|
||||
}
|
||||
/**
|
||||
* Model compression utilities
|
||||
*/
|
||||
export declare class ModelCompressor {
|
||||
/**
|
||||
* Compress model weights using quantization
|
||||
* Note: TensorFlow.js doesn't currently support model quantization
|
||||
*/
|
||||
static quantizeModel(modelPath: string, outputPath: string, options?: {
|
||||
dtype?: 'int8' | 'int16';
|
||||
}): Promise<void>;
|
||||
/**
|
||||
* Get model size information by reading files from disk
|
||||
*/
|
||||
static getModelSize(modelPath: string): Promise<{
|
||||
totalSize: number;
|
||||
weightsSize: number;
|
||||
modelJsonSize: number;
|
||||
}>;
|
||||
}
|
||||
/**
|
||||
* Utility functions
|
||||
*/
|
||||
export declare const utils: {
|
||||
/**
|
||||
* Check if bundled models are available
|
||||
*/
|
||||
checkModelsAvailable(): boolean;
|
||||
/**
|
||||
* Get bundled models directory
|
||||
*/
|
||||
getModelsDirectory(): string;
|
||||
/**
|
||||
* List available bundled models
|
||||
*/
|
||||
listAvailableModels(): string[];
|
||||
};
|
||||
export default BundledUniversalSentenceEncoder;
|
||||
//# sourceMappingURL=index.d.ts.map
|
||||
1
brainy-models-package/dist/index.d.ts.map
vendored
1
brainy-models-package/dist/index.d.ts.map
vendored
|
|
@ -1 +0,0 @@
|
|||
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;GAMG;AAEH,OAAO,KAAK,EAAE,MAAM,kBAAkB,CAAA;AAwBtC,MAAM,WAAW,aAAa;IAC5B,IAAI,EAAE,MAAM,CAAA;IACZ,OAAO,EAAE,MAAM,CAAA;IACf,WAAW,EAAE,MAAM,CAAA;IACnB,UAAU,EAAE,MAAM,CAAA;IAClB,YAAY,EAAE,MAAM,CAAA;IACpB,MAAM,EAAE,MAAM,CAAA;IACd,QAAQ,EAAE,MAAM,CAAA;IAChB,QAAQ,EAAE,MAAM,CAAA;IAChB,cAAc,EAAE,OAAO,CAAA;IACvB,WAAW,EAAE,MAAM,CAAA;CACpB;AAED,MAAM,WAAW,mBAAmB;IAClC,OAAO,CAAC,EAAE,OAAO,CAAA;IACjB,gBAAgB,CAAC,EAAE,OAAO,CAAA;CAC3B;AAED;;GAEG;AACH,qBAAa,+BAA+B;IAC1C,OAAO,CAAC,KAAK,CAA6B;IAC1C,OAAO,CAAC,QAAQ,CAA6B;IAC7C,OAAO,CAAC,OAAO,CAAqB;gBAExB,OAAO,GAAE,mBAAwB;IAQ7C;;OAEG;IACG,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAwC3B;;OAEG;IACG,KAAK,CAAC,KAAK,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,EAAE,CAAC,QAAQ,CAAC;IAqBlD;;OAEG;IACG,aAAa,CAAC,KAAK,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC;IAOzD;;OAEG;IACH,WAAW,IAAI,aAAa,GAAG,IAAI;IAInC;;OAEG;IACH,QAAQ,IAAI,OAAO;IAInB;;OAEG;IACH,YAAY,IAAI;QAAE,UAAU,EAAE,MAAM,EAAE,CAAC;QAAC,WAAW,EAAE,MAAM,EAAE,CAAA;KAAE,GAAG,IAAI;IAWtE;;OAEG;IACH,OAAO,IAAI,IAAI;CAMhB;AAED;;GAEG;AACH,qBAAa,eAAe;IAC1B;;;OAGG;WACU,aAAa,CACxB,SAAS,EAAE,MAAM,EACjB,UAAU,EAAE,MAAM,EAClB,OAAO,GAAE;QAAE,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAA;KAAO,GACzC,OAAO,CAAC,IAAI,CAAC;IAwBhB;;OAEG;WACU,YAAY,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC;QACpD,SAAS,EAAE,MAAM,CAAA;QACjB,WAAW,EAAE,MAAM,CAAA;QACnB,aAAa,EAAE,MAAM,CAAA;KACtB,CAAC;CAuCH;AAED;;GAEG;AACH,eAAO,MAAM,KAAK;IAChB;;OAEG;4BACqB,OAAO;IAK/B;;OAEG;0BACmB,MAAM;IAI5B;;OAEG;2BACoB,MAAM,EAAE;CAUhC,CAAA;AAGD,eAAe,+BAA+B,CAAA"}
|
||||
237
brainy-models-package/dist/index.js
vendored
237
brainy-models-package/dist/index.js
vendored
|
|
@ -1,237 +0,0 @@
|
|||
/**
|
||||
* @soulcraft/brainy-models
|
||||
*
|
||||
* Pre-bundled TensorFlow models for maximum reliability with Brainy vector database.
|
||||
* This package provides offline access to the Universal Sentence Encoder model,
|
||||
* eliminating network dependencies and ensuring consistent performance.
|
||||
*/
|
||||
import * as tf from '@tensorflow/tfjs';
|
||||
import { readFileSync, existsSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
/**
|
||||
* Helper function to safely extract error message from unknown error type
|
||||
*/
|
||||
function getErrorMessage(error) {
|
||||
if (error instanceof Error) {
|
||||
return error.message;
|
||||
}
|
||||
if (typeof error === 'string') {
|
||||
return error;
|
||||
}
|
||||
return String(error);
|
||||
}
|
||||
// Get the package directory
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
const PACKAGE_ROOT = join(__dirname, '..');
|
||||
const MODELS_DIR = join(PACKAGE_ROOT, 'models');
|
||||
/**
|
||||
* Bundled Universal Sentence Encoder for offline use
|
||||
*/
|
||||
export class BundledUniversalSentenceEncoder {
|
||||
model = null;
|
||||
metadata = null;
|
||||
options;
|
||||
constructor(options = {}) {
|
||||
this.options = {
|
||||
verbose: false,
|
||||
preferCompressed: false,
|
||||
...options
|
||||
};
|
||||
}
|
||||
/**
|
||||
* Load the bundled model from local files
|
||||
*/
|
||||
async load() {
|
||||
try {
|
||||
const modelDir = join(MODELS_DIR, 'universal-sentence-encoder');
|
||||
const modelPath = join(modelDir, 'model.json');
|
||||
const metadataPath = join(modelDir, 'metadata.json');
|
||||
if (!existsSync(modelPath)) {
|
||||
throw new Error(`Bundled model not found at ${modelPath}. ` +
|
||||
'Please run "npm run download-models" to download the model files.');
|
||||
}
|
||||
if (this.options.verbose) {
|
||||
console.log('🔄 Loading bundled Universal Sentence Encoder model...');
|
||||
}
|
||||
// Load metadata
|
||||
if (existsSync(metadataPath)) {
|
||||
const metadataContent = readFileSync(metadataPath, 'utf8');
|
||||
this.metadata = JSON.parse(metadataContent);
|
||||
if (this.options.verbose) {
|
||||
console.log(`📋 Model metadata:`, this.metadata);
|
||||
}
|
||||
}
|
||||
// Load the model
|
||||
this.model = await tf.loadGraphModel(`file://${modelPath}`);
|
||||
if (this.options.verbose) {
|
||||
console.log('✅ Bundled model loaded successfully');
|
||||
console.log(`🔒 Reliability: Maximum (fully offline)`);
|
||||
}
|
||||
}
|
||||
catch (error) {
|
||||
throw new Error(`Failed to load bundled model: ${getErrorMessage(error)}`);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Generate embeddings for the given texts
|
||||
*/
|
||||
async embed(texts) {
|
||||
if (!this.model) {
|
||||
throw new Error('Model not loaded. Call load() first.');
|
||||
}
|
||||
try {
|
||||
// Convert texts to tensor
|
||||
const inputTensor = tf.tensor1d(texts, 'string');
|
||||
// Run inference
|
||||
const embeddings = this.model.predict(inputTensor);
|
||||
// Clean up input tensor
|
||||
inputTensor.dispose();
|
||||
return embeddings;
|
||||
}
|
||||
catch (error) {
|
||||
throw new Error(`Failed to generate embeddings: ${getErrorMessage(error)}`);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Generate embeddings and return as JavaScript arrays
|
||||
*/
|
||||
async embedToArrays(texts) {
|
||||
const embeddings = await this.embed(texts);
|
||||
const arrays = await embeddings.array();
|
||||
embeddings.dispose();
|
||||
return arrays;
|
||||
}
|
||||
/**
|
||||
* Get model metadata
|
||||
*/
|
||||
getMetadata() {
|
||||
return this.metadata;
|
||||
}
|
||||
/**
|
||||
* Check if the model is loaded
|
||||
*/
|
||||
isLoaded() {
|
||||
return this.model !== null;
|
||||
}
|
||||
/**
|
||||
* Get model information
|
||||
*/
|
||||
getModelInfo() {
|
||||
if (!this.model) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
inputShape: this.model.inputs[0].shape || [],
|
||||
outputShape: this.model.outputs[0].shape || []
|
||||
};
|
||||
}
|
||||
/**
|
||||
* Dispose of the model and free memory
|
||||
*/
|
||||
dispose() {
|
||||
if (this.model) {
|
||||
this.model.dispose();
|
||||
this.model = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Model compression utilities
|
||||
*/
|
||||
export class ModelCompressor {
|
||||
/**
|
||||
* Compress model weights using quantization
|
||||
* Note: TensorFlow.js doesn't currently support model quantization
|
||||
*/
|
||||
static async quantizeModel(modelPath, outputPath, options = {}) {
|
||||
const { dtype = 'int8' } = options;
|
||||
try {
|
||||
console.log(`🔄 Loading model for quantization: ${modelPath}`);
|
||||
const model = await tf.loadGraphModel(`file://${modelPath}`);
|
||||
console.log(`🗜️ Quantizing model to ${dtype}...`);
|
||||
// TensorFlow.js doesn't have built-in quantization or model serialization APIs yet
|
||||
// This is a placeholder implementation that acknowledges the limitation
|
||||
console.warn('⚠️ Model quantization is not yet supported in TensorFlow.js');
|
||||
console.log(`📋 Model loaded successfully from: ${modelPath}`);
|
||||
console.log(`📋 Target output path: ${outputPath}`);
|
||||
console.log(`📋 Target dtype: ${dtype}`);
|
||||
model.dispose();
|
||||
throw new Error('Model quantization is not yet supported in TensorFlow.js. This feature requires server-side processing with TensorFlow Python.');
|
||||
}
|
||||
catch (error) {
|
||||
throw new Error(`Failed to compress model: ${getErrorMessage(error)}`);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Get model size information by reading files from disk
|
||||
*/
|
||||
static async getModelSize(modelPath) {
|
||||
try {
|
||||
// Load model to verify it's valid
|
||||
const model = await tf.loadGraphModel(`file://${modelPath}`);
|
||||
model.dispose();
|
||||
// Get model.json size
|
||||
const modelJsonSize = existsSync(modelPath) ? readFileSync(modelPath).length : 0;
|
||||
// Calculate weights size by reading weight files
|
||||
let weightsSize = 0;
|
||||
const modelDir = dirname(modelPath);
|
||||
// Read model.json to get weight file names
|
||||
if (existsSync(modelPath)) {
|
||||
const modelJson = JSON.parse(readFileSync(modelPath, 'utf8'));
|
||||
if (modelJson.weightsManifest) {
|
||||
for (const manifest of modelJson.weightsManifest) {
|
||||
for (const path of manifest.paths) {
|
||||
const weightFilePath = join(modelDir, path);
|
||||
if (existsSync(weightFilePath)) {
|
||||
weightsSize += readFileSync(weightFilePath).length;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
const totalSize = weightsSize + modelJsonSize;
|
||||
return {
|
||||
totalSize,
|
||||
weightsSize,
|
||||
modelJsonSize
|
||||
};
|
||||
}
|
||||
catch (error) {
|
||||
throw new Error(`Failed to get model size: ${getErrorMessage(error)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Utility functions
|
||||
*/
|
||||
export const utils = {
|
||||
/**
|
||||
* Check if bundled models are available
|
||||
*/
|
||||
checkModelsAvailable() {
|
||||
const modelPath = join(MODELS_DIR, 'universal-sentence-encoder', 'model.json');
|
||||
return existsSync(modelPath);
|
||||
},
|
||||
/**
|
||||
* Get bundled models directory
|
||||
*/
|
||||
getModelsDirectory() {
|
||||
return MODELS_DIR;
|
||||
},
|
||||
/**
|
||||
* List available bundled models
|
||||
*/
|
||||
listAvailableModels() {
|
||||
const models = [];
|
||||
const useModelPath = join(MODELS_DIR, 'universal-sentence-encoder', 'model.json');
|
||||
if (existsSync(useModelPath)) {
|
||||
models.push('universal-sentence-encoder');
|
||||
}
|
||||
return models;
|
||||
}
|
||||
};
|
||||
// Default export for convenience
|
||||
export default BundledUniversalSentenceEncoder;
|
||||
//# sourceMappingURL=index.js.map
|
||||
1
brainy-models-package/dist/index.js.map
vendored
1
brainy-models-package/dist/index.js.map
vendored
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -1,13 +0,0 @@
|
|||
{
|
||||
"name": "universal-sentence-encoder-lite",
|
||||
"version": "1.0.0",
|
||||
"description": "Universal Sentence Encoder Lite model with tokenizer support",
|
||||
"dimensions": 512,
|
||||
"downloadDate": "2025-08-06T00:56:00.000Z",
|
||||
"source": "https://www.kaggle.com/models/tensorflow/universal-sentence-encoder/tfJs/lite/1",
|
||||
"approach": "full-bundle",
|
||||
"modelUrl": "https://www.kaggle.com/models/tensorflow/universal-sentence-encoder/tfJs/lite/1",
|
||||
"bundledLocally": true,
|
||||
"reliability": "maximum",
|
||||
"notes": "This model requires tokenization of input text. Use with @tensorflow-models/universal-sentence-encoder for proper tokenization."
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
File diff suppressed because one or more lines are too long
|
|
@ -1,78 +0,0 @@
|
|||
{
|
||||
"name": "@soulcraft/brainy-models",
|
||||
"version": "0.8.0",
|
||||
"description": "Pre-bundled TensorFlow models for maximum reliability with Brainy vector database",
|
||||
"main": "dist/index.js",
|
||||
"module": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"type": "module",
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
},
|
||||
"scripts": {
|
||||
"prebuild": "echo 'Models already downloaded'",
|
||||
"build": "tsc",
|
||||
"test": "node test/test-models.js",
|
||||
"prepare": "npm run build",
|
||||
"download-models": "node scripts/download-full-models.js",
|
||||
"compress-models": "node scripts/compress-models.js",
|
||||
"_pack": "npm pack",
|
||||
"_release": "standard-version",
|
||||
"_release:patch": "standard-version --release-as patch",
|
||||
"_release:minor": "standard-version --release-as minor",
|
||||
"_release:major": "standard-version --release-as major",
|
||||
"_release:dry-run": "standard-version --dry-run",
|
||||
"_github-release": "node scripts/create-github-release.js",
|
||||
"_workflow": "node scripts/release-workflow.js",
|
||||
"_workflow:patch": "node scripts/release-workflow.js patch",
|
||||
"_workflow:minor": "node scripts/release-workflow.js minor",
|
||||
"_workflow:major": "node scripts/release-workflow.js major",
|
||||
"_workflow:dry-run": "npm run build && npm test && npm run _release:dry-run",
|
||||
"_deploy": "npm run build && npm publish",
|
||||
"release:minor": "npm run _release:minor"
|
||||
},
|
||||
"keywords": [
|
||||
"tensorflow",
|
||||
"models",
|
||||
"universal-sentence-encoder",
|
||||
"embeddings",
|
||||
"brainy",
|
||||
"vector-database",
|
||||
"offline",
|
||||
"bundled"
|
||||
],
|
||||
"author": "David Snelling (david@soulcraft.com)",
|
||||
"license": "MIT",
|
||||
"private": false,
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"homepage": "https://github.com/soulcraftlabs/brainy",
|
||||
"bugs": {
|
||||
"url": "https://github.com/soulcraftlabs/brainy/issues"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/soulcraftlabs/brainy.git",
|
||||
"directory": "brainy-models-package"
|
||||
},
|
||||
"files": [
|
||||
"dist/",
|
||||
"models/",
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
],
|
||||
"dependencies": {
|
||||
"@tensorflow-models/universal-sentence-encoder": "^1.3.3",
|
||||
"@tensorflow/tfjs": "^4.23.0-rc.0",
|
||||
"@tensorflow/tfjs-node": "^4.23.0-rc.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.11.30",
|
||||
"standard-version": "^9.5.0",
|
||||
"typescript": "^5.4.5"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@soulcraft/brainy": ">=0.33.0"
|
||||
}
|
||||
}
|
||||
62
docker-compose.yml
Normal file
62
docker-compose.yml
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
version: '3.8'
|
||||
|
||||
services:
|
||||
brainy:
|
||||
build: .
|
||||
container_name: brainy-app
|
||||
ports:
|
||||
- "3000:3000"
|
||||
environment:
|
||||
- NODE_ENV=production
|
||||
- BRAINY_STORAGE_TYPE=filesystem
|
||||
- BRAINY_STORAGE_PATH=/app/data
|
||||
- BRAINY_LOG_LEVEL=info
|
||||
- BRAINY_RATE_LIMIT_MAX=100
|
||||
- BRAINY_RATE_LIMIT_WINDOW_MS=900000
|
||||
volumes:
|
||||
# Persistent storage for data
|
||||
- brainy-data:/app/data
|
||||
# Optional: Mount local models directory
|
||||
# - ./models:/app/models:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- brainy-network
|
||||
|
||||
# Optional: MinIO for S3-compatible storage (development)
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
container_name: brainy-minio
|
||||
ports:
|
||||
- "9000:9000"
|
||||
- "9001:9001"
|
||||
environment:
|
||||
- MINIO_ROOT_USER=brainy
|
||||
- MINIO_ROOT_PASSWORD=brainy123456
|
||||
volumes:
|
||||
- minio-data:/data
|
||||
command: server /data --console-address ":9001"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
|
||||
interval: 30s
|
||||
timeout: 20s
|
||||
retries: 3
|
||||
networks:
|
||||
- brainy-network
|
||||
profiles:
|
||||
- with-s3
|
||||
|
||||
volumes:
|
||||
brainy-data:
|
||||
driver: local
|
||||
minio-data:
|
||||
driver: local
|
||||
|
||||
networks:
|
||||
brainy-network:
|
||||
driver: bridge
|
||||
|
|
@ -1,206 +0,0 @@
|
|||
# Adaptive Performance System
|
||||
|
||||
Brainy v0.53.1+ includes an automatic adaptive performance system that eliminates socket exhaustion and optimizes throughput without any configuration.
|
||||
|
||||
## Zero Configuration Required
|
||||
|
||||
The adaptive performance system works automatically - no settings or tuning needed. Just use Brainy normally and it will optimize itself based on your workload.
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. Adaptive Socket Management
|
||||
|
||||
The system automatically adjusts socket pools based on load:
|
||||
|
||||
- **Starts conservative**: 100 sockets initially
|
||||
- **Scales up under load**: Up to 2000 sockets when needed
|
||||
- **Scales down when idle**: Conserves resources automatically
|
||||
- **Smart keep-alive**: Adjusts connection reuse based on patterns
|
||||
|
||||
### 2. Intelligent Backpressure
|
||||
|
||||
Prevents system overload with self-healing capabilities:
|
||||
|
||||
- **Request flow control**: Queues requests when system is busy
|
||||
- **Priority handling**: Important operations get processed first
|
||||
- **Circuit breaker**: Automatically recovers from overload conditions
|
||||
- **Predictive scaling**: Anticipates load changes based on patterns
|
||||
|
||||
### 3. Performance Monitoring
|
||||
|
||||
Real-time metrics and optimization:
|
||||
|
||||
- **Health scoring**: 0-100 score of system health
|
||||
- **Trend analysis**: Detects degrading performance
|
||||
- **Auto-optimization**: Adjusts configuration automatically
|
||||
- **Smart recommendations**: Suggests improvements when needed
|
||||
|
||||
## Benefits
|
||||
|
||||
### For High-Volume Scenarios
|
||||
|
||||
- **No more socket exhaustion**: Automatically scales sockets as needed
|
||||
- **Better throughput**: Batch sizes optimize dynamically
|
||||
- **Automatic recovery**: Self-heals from error conditions
|
||||
- **Resource efficiency**: Uses only what's needed
|
||||
|
||||
### For Variable Workloads
|
||||
|
||||
- **Adapts to patterns**: Learns your usage over time
|
||||
- **Handles spikes**: Scales up quickly for burst traffic
|
||||
- **Efficient at rest**: Scales down to save resources
|
||||
- **No manual tuning**: Adjusts itself automatically
|
||||
|
||||
## Usage Example
|
||||
|
||||
```typescript
|
||||
import { BrainyData } from '@soulcraft/brainy'
|
||||
|
||||
// Just create and use - no special configuration needed
|
||||
const brainy = new BrainyData({
|
||||
storage: {
|
||||
type: 's3',
|
||||
s3Storage: {
|
||||
bucketName: 'my-bucket',
|
||||
region: 'us-east-1',
|
||||
accessKeyId: 'xxx',
|
||||
secretAccessKey: 'yyy'
|
||||
}
|
||||
// No socket or performance configuration needed!
|
||||
}
|
||||
})
|
||||
|
||||
// Use normally - system adapts automatically
|
||||
await brainy.addBatch(largeDataset) // Handles any volume
|
||||
```
|
||||
|
||||
## Monitoring Performance (Optional)
|
||||
|
||||
While not required, you can monitor the adaptive system:
|
||||
|
||||
```typescript
|
||||
import { getGlobalPerformanceMonitor } from '@soulcraft/brainy'
|
||||
|
||||
const monitor = getGlobalPerformanceMonitor()
|
||||
|
||||
// Get current metrics
|
||||
const report = monitor.getReport()
|
||||
console.log('Health Score:', report.metrics.healthScore)
|
||||
console.log('Operations/sec:', report.metrics.operationsPerSecond)
|
||||
console.log('Current Socket Config:', report.socketConfig)
|
||||
|
||||
// Get optimization recommendations
|
||||
if (report.recommendations.length > 0) {
|
||||
console.log('Suggestions:', report.recommendations)
|
||||
}
|
||||
```
|
||||
|
||||
## How It Helps Your Application
|
||||
|
||||
### Before (v0.53.0 and earlier)
|
||||
- Fixed socket limits (500)
|
||||
- Manual batch size configuration
|
||||
- Socket exhaustion under high load
|
||||
- Manual recovery required
|
||||
- Required tuning for different workloads
|
||||
|
||||
### After (v0.53.1+)
|
||||
- Dynamic socket scaling (100-2000)
|
||||
- Automatic batch optimization
|
||||
- Self-preventing socket exhaustion
|
||||
- Automatic error recovery
|
||||
- Zero configuration needed
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Socket Scaling Algorithm
|
||||
|
||||
The system uses multiple signals to determine optimal socket count:
|
||||
- Current request rate
|
||||
- Pending request queue depth
|
||||
- Error rate trends
|
||||
- Memory pressure
|
||||
- Latency percentiles (P50, P95, P99)
|
||||
|
||||
### Backpressure Management
|
||||
|
||||
Implements Little's Law for optimal concurrency:
|
||||
```
|
||||
L = λ × W
|
||||
where:
|
||||
L = number of requests in system
|
||||
λ = arrival rate
|
||||
W = average time in system
|
||||
```
|
||||
|
||||
### Circuit Breaker States
|
||||
|
||||
- **Closed**: Normal operation
|
||||
- **Open**: Rejecting requests to recover
|
||||
- **Half-Open**: Testing if system recovered
|
||||
|
||||
### Performance Metrics Tracked
|
||||
|
||||
- Total operations and success rate
|
||||
- Latency percentiles (average, P95, P99)
|
||||
- Throughput (ops/sec, bytes/sec)
|
||||
- Resource usage (memory, CPU, sockets)
|
||||
- Queue depth and utilization
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### System reports "overloaded"
|
||||
|
||||
This is the circuit breaker protecting your system. It will automatically recover in 30 seconds. To avoid:
|
||||
- Reduce request rate temporarily
|
||||
- The system will adapt and handle more load over time
|
||||
|
||||
### Performance degrading over time
|
||||
|
||||
The system will detect this and adapt. You can check recommendations:
|
||||
```typescript
|
||||
const report = monitor.getReport()
|
||||
console.log(report.recommendations)
|
||||
```
|
||||
|
||||
### Want to disable auto-optimization
|
||||
|
||||
While not recommended, you can disable it:
|
||||
```typescript
|
||||
const monitor = getGlobalPerformanceMonitor()
|
||||
monitor.setAutoOptimize(false)
|
||||
```
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### From v0.52.x or earlier
|
||||
|
||||
No changes required! The adaptive system is automatically active and requires no configuration.
|
||||
|
||||
### From v0.53.0
|
||||
|
||||
Update to v0.53.1+ to get automatic performance optimization.
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Let it adapt**: Give the system time to learn your patterns
|
||||
2. **Monitor initially**: Check health score during first few runs
|
||||
3. **Trust the system**: Avoid manual tuning unless necessary
|
||||
4. **Report issues**: If you see consistent problems, please report them
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
Tested with real-world workloads:
|
||||
|
||||
| Scenario | v0.53.0 | v0.53.1 | Improvement |
|
||||
|----------|---------|---------|-------------|
|
||||
| 10K operations burst | Socket exhaustion at 5K | Completed successfully | ✅ No exhaustion |
|
||||
| Sustained high load | 500 ops/sec max | 2000+ ops/sec | 4x throughput |
|
||||
| Error recovery | Manual intervention | Automatic recovery | ✅ Self-healing |
|
||||
| Memory efficiency | Fixed allocation | Dynamic scaling | 50% less at idle |
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Socket Management Details](./SOCKET_MANAGEMENT.md)
|
||||
- [Backpressure Algorithm](./BACKPRESSURE.md)
|
||||
- [Performance Tuning Guide](./PERFORMANCE.md)
|
||||
295
docs/ADR-001-generational-mvcc.md
Normal file
295
docs/ADR-001-generational-mvcc.md
Normal file
|
|
@ -0,0 +1,295 @@
|
|||
# ADR-001: Generational MVCC storage and the immutable Db API
|
||||
|
||||
**Status:** Accepted (ships in 8.0)
|
||||
**Date:** 2026-06-10
|
||||
|
||||
## Context
|
||||
|
||||
Before 8.0, Brainy carried two overlapping version-control subsystems: a
|
||||
copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a
|
||||
separate versioning subsystem (`versions.save/list/compare/restore/prune`),
|
||||
plus a read-only historical adapter for commit-based time travel. Together
|
||||
they were ~5,100 LOC of mechanism for one product need: *read a consistent
|
||||
past state while the store keeps moving, and snapshot/restore cheaply.*
|
||||
|
||||
Neither subsystem gave a precise isolation guarantee. Reads raced in-place
|
||||
JSON overwrites, so a "snapshot" was only as immutable as the bytes it
|
||||
happened to share with the live store.
|
||||
|
||||
8.0 replaces both with **one mechanism**: generational MVCC over immutable,
|
||||
generation-stamped records, exposed through a Datomic-style immutable
|
||||
database value (`Db`). The same model is implemented natively by versioned
|
||||
index providers (LSM snapshots), so semantics are identical on the pure-JS
|
||||
path and the native path.
|
||||
|
||||
## Decision
|
||||
|
||||
### The model
|
||||
|
||||
- A **monotonic u64 generation counter** is the store's logical clock. It
|
||||
advances once per committed `transact()` batch and once per
|
||||
single-operation write (`add`/`update`/`remove`/`relate`/…), so
|
||||
`brain.generation()` is always a meaningful watermark. It is persisted in
|
||||
`_system/generation.json` and never reissued for anything durable.
|
||||
- `brain.now()` **pins** the current generation in O(1) and returns a `Db` —
|
||||
an immutable view. Pins are refcounted; `db.release()` (with a
|
||||
`FinalizationRegistry` backstop for leaked values) ends the pin.
|
||||
- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative
|
||||
batch atomically as **exactly one generation**, with whole-store
|
||||
compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and
|
||||
reified transaction metadata appended to `_system/tx-log.jsonl`.
|
||||
- `brain.asOf(generation | Date | snapshotPath)` opens past state;
|
||||
`db.with(ops)` layers a speculative in-memory overlay (never touching
|
||||
disk, the counter, or index providers); `db.persist(path)` cuts an
|
||||
instant snapshot; `brain.restore(path, { confirm: true })` replaces state
|
||||
from one; `Brainy.load(path)` opens a snapshot read-only with the full
|
||||
query surface.
|
||||
|
||||
### Persisted layout
|
||||
|
||||
All paths are storage-root-relative:
|
||||
|
||||
```
|
||||
_system/generation.json { generation, updatedAt } atomic tmp+rename
|
||||
_system/manifest.json { version, generation, atomic tmp+rename
|
||||
committedAt, horizon } (the commit point)
|
||||
_system/tx-log.jsonl one line per committed append-only
|
||||
transact: { generation,
|
||||
timestamp, meta? }
|
||||
_generations/<N>/tx.json the generation-N delta: immutable
|
||||
touched noun/verb ids + meta
|
||||
_generations/<N>/prev/<id>.json before-image of <id> as of immutable
|
||||
commit N (raw stored bytes;
|
||||
null parts = file was absent)
|
||||
```
|
||||
|
||||
**Why per-generation deltas instead of a global `id → latest generation`
|
||||
map in the manifest:** a global map makes every commit O(all ids) — the
|
||||
whole map must be rewritten to swap it atomically. The delta layout makes a
|
||||
commit O(ids touched) and keeps the manifest a fixed-size watermark, while
|
||||
point-in-time resolution stays correct (see "Read resolution" below). The
|
||||
trade is that resolution at a pinned generation scans the deltas of later
|
||||
commits — bounded by the number of commits since the pin, which is exactly
|
||||
the window compaction keeps short.
|
||||
|
||||
Before-images are deliberately the *only* per-id records. They serve both
|
||||
roles the layer needs — the crash-recovery undo log and the point-in-time
|
||||
read source. After-images would duplicate state that is already readable
|
||||
(the canonical entity files hold the latest bytes; earlier states resolve
|
||||
from later before-images) and would double record I/O per commit.
|
||||
|
||||
### Commit protocol (durability)
|
||||
|
||||
`transact()` commits under a store-wide mutex:
|
||||
|
||||
1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError`
|
||||
before anything is staged.
|
||||
2. **Reserve** generation `N` (counter increment).
|
||||
3. **Stage the undo log:** write the before-image of every touched id plus
|
||||
`tx.json` under `_generations/N/`, then **fsync** the files and their
|
||||
directories. From this point, any crash is recoverable to the exact
|
||||
pre-transaction bytes.
|
||||
4. **Execute** the planned batch through the TransactionManager (which has
|
||||
its own operation-level rollback for in-flight failures).
|
||||
5. **Commit point:** persist the counter, then write `_system/manifest.json`
|
||||
via atomic tmp+rename and fsync it. The rename *is* the commit: a
|
||||
generation directory is committed if and only if `N ≤
|
||||
manifest.generation`.
|
||||
6. Append the tx-log line (advisory metadata — a crash between 5 and 6
|
||||
keeps the transaction).
|
||||
|
||||
**Crash recovery (on open):** any `_generations/<N>` directory with
|
||||
`N > manifest.generation` is an uncommitted transaction. Its before-images
|
||||
are restored to the canonical paths (idempotently — recovery itself can
|
||||
crash and rerun) and the directory is removed. Because recovery runs before
|
||||
any index is built, and a recovery that rolled something back forces a full
|
||||
index rebuild, derived indexes never observe rolled-back state. Reader-mode
|
||||
instances skip recovery (readers never write; the next writer repairs).
|
||||
|
||||
A failed (non-crash) transaction takes the same staging directory down the
|
||||
abort path: the TransactionManager rolls back applied operations, the
|
||||
staging directory is removed, and the generation reservation is returned —
|
||||
a failed batch leaves the generation counter unchanged.
|
||||
|
||||
### Read resolution at a pinned generation
|
||||
|
||||
The state of id X at pinned generation G is:
|
||||
|
||||
- the before-image stored by the **first committed generation after G that
|
||||
touched X**, or
|
||||
- the live canonical bytes, when nothing after G touched X.
|
||||
|
||||
While nothing has committed past G, *every* read on the `Db` delegates to
|
||||
the live fast paths untouched — `now()` adds no read overhead until history
|
||||
actually moves.
|
||||
|
||||
**Two read paths, one result set.** `get()`, metadata-level `find()`, and
|
||||
filter-based `related()` resolve directly through the record layer at any
|
||||
reachable pinned generation — no extra cost beyond scanning the deltas of
|
||||
later commits. Index-accelerated dimensions (semantic/vector search, graph
|
||||
traversal, cursors, aggregation) are served by **at-generation index
|
||||
materialization**: the first such query on a historical `Db` copies the
|
||||
exact at-G record set (live bytes for ids untouched since the pin,
|
||||
before-images for the rest; a final reconciliation pass runs under the
|
||||
commit mutex so transactions racing the copy cannot skew it) into an
|
||||
ephemeral in-memory store and opens a read-only engine over it — the same
|
||||
vector/metadata/graph index classes the live brain uses, sharing the host's
|
||||
embedder and aggregate definitions. The handle is cached on the `Db` and
|
||||
freed by `release()`.
|
||||
|
||||
**Cost, stated plainly:** materialization is O(n at G) time and memory,
|
||||
once per `Db`. That is the open-core price of historical index queries. A
|
||||
native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
|
||||
retained LSM segments) serves the same reads with no rebuild at all — the
|
||||
materializer is the correctness baseline, the provider is the accelerator.
|
||||
|
||||
**The one remaining boundary.** Speculative `with()` overlays throw
|
||||
`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
|
||||
overlay entities carry no embeddings (`with()` never invokes the embedder),
|
||||
so a "full" index query over an overlay would silently exclude the
|
||||
overlay's own entities. Commit with `transact()` to get the full surface.
|
||||
|
||||
**History granularity (Model-B).** EVERY write is its own immutable generation
|
||||
— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
|
||||
Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/
|
||||
`diff()`/`history()` exactly like transacts; a pin always freezes against later
|
||||
writes. `transact()` groups several operations into ONE atomic generation.
|
||||
|
||||
Single-op history durability is **async group-commit**: the live write hits
|
||||
canonical storage immediately (acknowledged), while its before-image is buffered
|
||||
and persisted to disk in one batched fsync on a size/timer trigger (or forced by
|
||||
`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in
|
||||
point-in-time resolution exactly like on-disk generations, so the synchronous
|
||||
`now()` freezes with no forced flush. A hard crash before the flush loses only
|
||||
the buffered *history* of the last window — never live data — and a crash
|
||||
*mid-flush* is recovered by **drop-without-restore** (the partial generation's
|
||||
before-images are discarded, never replayed, because the live write was already
|
||||
acknowledged; restoring them would silently revert it).
|
||||
|
||||
### Pinning, retention, compaction
|
||||
|
||||
- Each live `Db` holds one refcounted pin on its generation (plus a
|
||||
`pin(generation)` on every registered `VersionedIndexProvider`, whose
|
||||
explicit pin lifetime overrides any time-based snapshot retention the
|
||||
provider has).
|
||||
- The constructor **`retention`** knob governs auto-compaction (on `flush()`/
|
||||
`close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config;
|
||||
driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) ·
|
||||
`'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit
|
||||
CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
|
||||
manually on the same caps — the oldest unpinned record-sets are reclaimed
|
||||
while ANY supplied cap is exceeded.
|
||||
- A record-set `N` is reclaimed only when `N` is at or below **every** live
|
||||
pin — deleting `N` can only break readers pinned *below* `N`, because
|
||||
resolution reads before-images from generations strictly greater than the
|
||||
pin. Live pins are ALWAYS exempt, in every retention mode.
|
||||
- The manifest records the **horizon** (highest reclaimed generation).
|
||||
Generations below the horizon are unreachable; `asOf()` on them throws
|
||||
`GenerationCompactedError`. The horizon itself stays reachable, resolved
|
||||
from the record-sets above it. To keep a generation readable forever,
|
||||
`persist()` it first — snapshots are self-contained.
|
||||
|
||||
### Snapshots and restore
|
||||
|
||||
`db.persist(path)` flushes indexes, then cuts the snapshot under the
|
||||
store's commit mutex (no commit, compaction, or counter write can
|
||||
interleave). On filesystem storage it is a **hard-link farm**: every data
|
||||
file is immutable-by-rename, so linking is safe — later rewrites swap
|
||||
inodes and the snapshot keeps the old bytes. The two exceptions are handled
|
||||
explicitly: the append-in-place tx-log is byte-copied, and process-local
|
||||
lock state is excluded. Cross-device targets (and filesystems that refuse
|
||||
links) fall back to per-file byte copies. In-memory stores serialize to the
|
||||
same directory layout, so persisting a memory brain produces a real,
|
||||
durable, loadable store.
|
||||
|
||||
`persist()` requires the view to still be the store's latest generation
|
||||
(a snapshot captures current bytes); a view that history has moved past
|
||||
throws rather than persisting the wrong state.
|
||||
|
||||
`restore(path, { confirm: true })` replaces the store's contents from a
|
||||
snapshot via byte copy (never links — the snapshot stays independent),
|
||||
reloads all adapter-internal derived state, rebuilds all indexes, and
|
||||
floors the generation counter at its pre-restore value so observed
|
||||
generation numbers are never reissued. Live pins do not survive a restore;
|
||||
a warning is logged when any exist.
|
||||
|
||||
### Versioned index providers
|
||||
|
||||
Native index providers may implement the optional 4-method
|
||||
`VersionedIndexProvider` capability (`generation()`,
|
||||
`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the
|
||||
boundary). The locked consistency model: providers are **post-commit
|
||||
appliers**. The storage-record commit is the source of truth; provider
|
||||
index state is derived. On open, a provider behind the committed watermark
|
||||
replays the gap from storage (or requests a rebuild) — there are no
|
||||
provider rollback hooks, because uncommitted transactions are repaired at
|
||||
the storage layer before any index opens. Speculative `with()` overlays
|
||||
never reach providers.
|
||||
|
||||
## Guarantees (and their proofs)
|
||||
|
||||
Each stated guarantee has a test that proves it, not merely exercises it
|
||||
(`tests/integration/db-mvcc.test.ts`, plus
|
||||
`tests/unit/db/generationStore.test.ts` for the record layer in isolation):
|
||||
|
||||
| Guarantee | Proof |
|
||||
|---|---|
|
||||
| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) |
|
||||
| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) |
|
||||
| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) |
|
||||
| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c |
|
||||
| Compaction never breaks a pinned read; release enables reclaim | proof 5 |
|
||||
| `with()` overlays touch nothing durable | proof 6 |
|
||||
| Generation monotonicity across close/reopen | proof 7 |
|
||||
| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) |
|
||||
| Balanced provider pin/release lockstep | proof 9 |
|
||||
|
||||
One deliberate softness: single-operation generation bumps persist the
|
||||
counter coalesced (per write burst), not per write. Durable artifacts —
|
||||
records, manifests, snapshots — always persist the counter synchronously at
|
||||
their own commit points, so a crash inside the coalescing window can lose
|
||||
only counter values that nothing durable ever referenced.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Failure | Outcome |
|
||||
|---|---|
|
||||
| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. |
|
||||
| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. |
|
||||
| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. |
|
||||
| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
|
||||
| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
|
||||
| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
|
||||
| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
|
||||
| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
|
||||
| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
|
||||
|
||||
## Lineage
|
||||
|
||||
The design is an assembly of well-understood prior art, chosen for being
|
||||
boring where it counts:
|
||||
|
||||
- **Datomic** — the database-as-a-value: an immutable `Db` you query, with
|
||||
`with()` for speculation and reified transaction metadata instead of
|
||||
commit messages.
|
||||
- **LMDB** — reader pins: readers never block writers; a reader's view
|
||||
stays valid because nothing overwrites the pages (here: records) it
|
||||
references; reclamation waits for the last reader.
|
||||
- **LSM trees / Cassandra** — immutable segments make snapshots hard links
|
||||
and make compaction a retention policy instead of a locking problem.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One mechanism replaces the COW and versioning subsystems (their removal
|
||||
is the companion change to this ADR).
|
||||
- In-place branch switching (`checkout`) is gone by design; the replacement
|
||||
is opening a persisted snapshot as a separate instance — a name→path
|
||||
mapping where a product needs named branches.
|
||||
- Every commit pays O(ids touched) extra writes (before-images + delta +
|
||||
manifest). Single-operation writes pay only an in-memory counter bump
|
||||
with coalesced persistence.
|
||||
- The full query surface works at every reachable pinned generation.
|
||||
Record-path reads (`get`, metadata `find`, filter `related`) are
|
||||
effectively free; index-accelerated historical queries pay a one-time
|
||||
O(n at G) materialization per `Db` on the open-core path (freed on
|
||||
`release()`), and run rebuild-free on a native `VersionedIndexProvider`.
|
||||
468
docs/BATCHING.md
Normal file
468
docs/BATCHING.md
Normal file
|
|
@ -0,0 +1,468 @@
|
|||
---
|
||||
title: Batch Operations
|
||||
slug: guides/batching
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 5
|
||||
description: Eliminate N+1 query patterns with batchGet() and storage-level batch APIs for fast multi-entity reads against filesystem and memory storage.
|
||||
next:
|
||||
- api/reference
|
||||
- guides/find-system
|
||||
---
|
||||
|
||||
# Batch Operations API
|
||||
> **Production-Ready** | Zero N+1 Query Patterns
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy provides batch operations at the storage layer to eliminate N+1 query patterns for VFS operations, relationship queries, and entity retrieval.
|
||||
|
||||
### Problem Solved
|
||||
|
||||
The naive pattern of looping and calling `brain.get(id)` once per item issues sequential reads through the storage layer. Batched APIs collapse that into a single read pass.
|
||||
|
||||
**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`** at the VFS layer and the explicit `batchGet()` / `getNounMetadataBatch()` calls — not to `readFile()` or individual `get()` operations.
|
||||
|
||||
---
|
||||
|
||||
## New Public APIs
|
||||
|
||||
### 1. `brain.batchGet(ids, options?)`
|
||||
|
||||
Batch retrieval of multiple entities (metadata-only by default).
|
||||
|
||||
```typescript
|
||||
// Fetch multiple entities in a single batched operation
|
||||
const ids = ['id1', 'id2', 'id3']
|
||||
const results: Map<string, Entity> = await brain.batchGet(ids)
|
||||
|
||||
// With vectors (falls back to individual gets)
|
||||
const resultsWithVectors = await brain.batchGet(ids, { includeVectors: true })
|
||||
|
||||
// Results map
|
||||
results.get('id1') // → Entity or undefined
|
||||
results.size // → 3 (number of found entities)
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- Memory storage: Instant (parallel reads)
|
||||
- Filesystem storage: Parallel reads, scales with available IOPS
|
||||
|
||||
**Use Cases:**
|
||||
- Loading multiple entities for display
|
||||
- Bulk data export operations
|
||||
- Relationship traversal (fetch all connected entities)
|
||||
|
||||
---
|
||||
|
||||
## Storage-Level APIs
|
||||
|
||||
### 2. `storage.getNounMetadataBatch(ids)`
|
||||
|
||||
Batch metadata retrieval with direct O(1) path construction.
|
||||
|
||||
```typescript
|
||||
const storage = brain.storage as BaseStorage
|
||||
const ids = ['id1', 'id2', 'id3']
|
||||
|
||||
const metadataMap: Map<string, NounMetadata> = await storage.getNounMetadataBatch(ids)
|
||||
|
||||
for (const [id, metadata] of metadataMap) {
|
||||
console.log(metadata.noun) // Type: 'document', 'person', etc.
|
||||
console.log(metadata.data) // Entity data
|
||||
}
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- ✅ Direct O(1) path construction from ID (no type lookup needed!)
|
||||
- ✅ Sharding preservation (all paths include `{shard}/{id}`)
|
||||
- ✅ Write-cache coherent (read-after-write consistency)
|
||||
- ✅ O(1) path construction — eliminates the per-entity type search of the old type-first layout
|
||||
|
||||
**Performance:**
|
||||
- Constant-time path construction per ID — no type-cache misses
|
||||
- Filesystem: parallel reads bounded by IOPS
|
||||
- No type search delays — every ID maps directly to storage path
|
||||
|
||||
---
|
||||
|
||||
### 3. `storage.getVerbsBySourceBatch(sourceIds, verbType?)`
|
||||
|
||||
Batch relationship queries by source entity IDs.
|
||||
|
||||
```typescript
|
||||
const storage = brain.storage as BaseStorage
|
||||
|
||||
// Get all relationships from multiple sources
|
||||
const results: Map<string, GraphVerb[]> = await storage.getVerbsBySourceBatch([
|
||||
'person1',
|
||||
'person2'
|
||||
])
|
||||
|
||||
// Filter by verb type
|
||||
const createsResults = await storage.getVerbsBySourceBatch(
|
||||
['person1', 'person2'],
|
||||
'creates'
|
||||
)
|
||||
|
||||
// Process results
|
||||
for (const [sourceId, verbs] of results) {
|
||||
console.log(`${sourceId} has ${verbs.length} relationships`)
|
||||
verbs.forEach(verb => {
|
||||
console.log(` → ${verb.verb} → ${verb.targetId}`)
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
**Use Cases:**
|
||||
- Social graph traversal (fetch all connections for multiple users)
|
||||
- Knowledge graph queries (find all relationships of specific type)
|
||||
- Bulk export of relationship data
|
||||
|
||||
**Performance:**
|
||||
- Memory storage: single in-memory pass over the metadata index
|
||||
- Filesystem storage: parallel reads through the metadata index
|
||||
|
||||
---
|
||||
|
||||
## VFS Integration
|
||||
|
||||
VFS operations automatically use batch APIs for maximum performance.
|
||||
|
||||
### Directory Traversal
|
||||
|
||||
```typescript
|
||||
// Tree traversal uses batched reads under the hood
|
||||
const tree = await brain.vfs.getTreeStructure('/my-dir')
|
||||
// ✅ PathResolver.getChildren() uses brain.batchGet() internally
|
||||
// ✅ Parallel traversal of directories at the same tree level
|
||||
// ✅ 2-3 batched calls instead of 22 sequential calls
|
||||
```
|
||||
|
||||
**Architecture:**
|
||||
|
||||
```
|
||||
VFS.getTreeStructure()
|
||||
↓ PARALLEL (breadth-first traversal)
|
||||
→ PathResolver.getChildren() [all dirs at level processed in parallel]
|
||||
↓ BATCHED
|
||||
→ brain.batchGet(childIds) [1 call instead of N]
|
||||
↓ BATCHED
|
||||
→ storage.getNounMetadataBatch(ids) [1 call instead of N]
|
||||
↓ ADAPTER
|
||||
→ Filesystem: Promise.all() parallel reads
|
||||
→ Memory: Promise.all() parallel reads
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Features Compatibility
|
||||
|
||||
### ✅ ID-First Storage Architecture
|
||||
|
||||
All batch operations use direct ID-first paths - no type lookup needed!
|
||||
|
||||
**ID-First Path Structure:**
|
||||
```
|
||||
entities/nouns/{SHARD}/{ID}/metadata.json
|
||||
entities/verbs/{SHARD}/{ID}/metadata.json
|
||||
```
|
||||
|
||||
**Direct O(1) Path Construction:**
|
||||
```typescript
|
||||
// Every ID maps directly to exactly ONE path - O(1), no type search
|
||||
const id = 'abc-123'
|
||||
const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
|
||||
const path = `entities/nouns/${shard}/${id}/metadata.json`
|
||||
|
||||
// No type cache needed!
|
||||
// No type search needed!
|
||||
// No multi-type fallback needed!
|
||||
// Just pure O(1) lookup!
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- **O(1)** path lookups (eliminates the 42-type sequential search the old type-first layout required)
|
||||
- **Simpler code** - removed 500+ lines of type cache complexity
|
||||
- **Scalable** - works at large scale without type tracking overhead
|
||||
|
||||
---
|
||||
|
||||
### ✅ Sharding
|
||||
|
||||
All batch paths include shard IDs calculated via `getShardIdFromUuid(id)`:
|
||||
|
||||
```typescript
|
||||
const id = 'a3c4e5f7-...'
|
||||
const shard = getShardIdFromUuid(id) // → 'a3' (first 2 hex chars)
|
||||
const path = `entities/nouns/${shard}/${id}/metadata.json`
|
||||
```
|
||||
|
||||
**Distribution:** 256 shards (00-ff) for optimal load distribution.
|
||||
|
||||
---
|
||||
|
||||
### ✅ Generational MVCC (8.0)
|
||||
|
||||
Batch reads always serve the **live** generation through the fast paths
|
||||
shown above. Point-in-time reads go through the Db API instead: a pinned
|
||||
`Db` (`brain.now()`, `brain.asOf()`) resolves changed ids from immutable
|
||||
generation records and unchanged ids from the same live paths batch reads
|
||||
use — see the [consistency model](concepts/consistency-model.md).
|
||||
|
||||
```typescript
|
||||
const db = brain.now() // pinned view
|
||||
const entity = await db.get(id) // correct at the pinned generation
|
||||
const results = await brain.batchGet(ids) // live state, batched
|
||||
await db.release()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Why Batching Is Faster
|
||||
|
||||
Batching's advantage is structural, not a fixed multiplier (the actual speedup
|
||||
depends on storage backend, IOPS, and batch size):
|
||||
|
||||
- **N+1 elimination** — N sequential reads collapse into a single parallel pass
|
||||
(`Promise.all` over the batch).
|
||||
- **O(1) path construction** — every ID maps directly to one storage path, with
|
||||
no per-type cache lookup.
|
||||
- **One metadata round-trip** — relationship batches fetch all sources' metadata
|
||||
in a single pass instead of one query per source.
|
||||
|
||||
The integration test `tests/integration/storage-batch-operations.test.ts`
|
||||
exercises batch vs. individual reads and asserts that batch retrieval is not
|
||||
slower than the per-entity loop for large batches; it does not pin a specific
|
||||
multiplier, since that is hardware- and IOPS-dependent.
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Partial Batch Failures
|
||||
|
||||
Batch operations gracefully handle missing or invalid entities:
|
||||
|
||||
```typescript
|
||||
const validId = 'abc-123-...'
|
||||
const invalidIds = [
|
||||
'11111111-1111-1111-1111-111111111111',
|
||||
'22222222-2222-2222-2222-222222222222'
|
||||
]
|
||||
|
||||
const results = await brain.batchGet([validId, ...invalidIds])
|
||||
|
||||
results.size // → 1 (only valid entity)
|
||||
results.has(validId) // → true
|
||||
results.has(invalidIds[0]) // → false (silently skipped)
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Invalid UUIDs: Silently skipped (not included in results)
|
||||
- Missing entities: Silently skipped (not included in results)
|
||||
- Storage errors: Logged, entity excluded from results
|
||||
- No exceptions thrown for partial failures
|
||||
|
||||
### Empty Batches
|
||||
|
||||
```typescript
|
||||
const results = await brain.batchGet([])
|
||||
results.size // → 0 (empty map)
|
||||
```
|
||||
|
||||
### Duplicate IDs
|
||||
|
||||
```typescript
|
||||
const results = await brain.batchGet(['id1', 'id1', 'id1'])
|
||||
results.size // → 1 (deduplicated automatically)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### From Individual Gets
|
||||
|
||||
**Before:**
|
||||
```typescript
|
||||
const entities = []
|
||||
for (const id of ids) {
|
||||
const entity = await brain.get(id)
|
||||
if (entity) entities.push(entity)
|
||||
}
|
||||
```
|
||||
|
||||
**After:**
|
||||
```typescript
|
||||
const results = await brain.batchGet(ids)
|
||||
const entities = Array.from(results.values())
|
||||
```
|
||||
|
||||
**Performance Gain:** Replaces N sequential reads with a single batched pass — no fixed multiplier, it scales with storage IOPS.
|
||||
|
||||
---
|
||||
|
||||
### From Individual Relationship Queries
|
||||
|
||||
**Before:**
|
||||
```typescript
|
||||
const allVerbs = []
|
||||
for (const sourceId of sourceIds) {
|
||||
const verbs = await brain.related({ from: sourceId })
|
||||
allVerbs.push(...verbs)
|
||||
}
|
||||
```
|
||||
|
||||
**After:**
|
||||
```typescript
|
||||
const storage = brain.storage as BaseStorage
|
||||
const results = await storage.getVerbsBySourceBatch(sourceIds)
|
||||
|
||||
const allVerbs = []
|
||||
for (const verbs of results.values()) {
|
||||
allVerbs.push(...verbs)
|
||||
}
|
||||
```
|
||||
|
||||
**Performance Gain:** One batched metadata fetch instead of one query per source entity.
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. **Use Batching for Multiple Entity Operations**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: Batch fetch
|
||||
const results = await brain.batchGet(ids)
|
||||
|
||||
// ❌ BAD: Individual gets in loop
|
||||
for (const id of ids) {
|
||||
await brain.get(id)
|
||||
}
|
||||
```
|
||||
|
||||
### 2. **Batch Size Recommendations**
|
||||
|
||||
| Storage | Optimal Batch Size | Max Batch Size |
|
||||
|---------|--------------------|----------------|
|
||||
| **Memory** | Unlimited | Unlimited |
|
||||
| **Filesystem** | 100-500 | 1000 |
|
||||
|
||||
**Guideline:** For batches >1000, split into chunks of 500-1000.
|
||||
|
||||
### 3. **Metadata-Only by Default**
|
||||
|
||||
```typescript
|
||||
// Default: Metadata-only (fast)
|
||||
const results = await brain.batchGet(ids) // No vectors
|
||||
|
||||
// Only load vectors if needed
|
||||
const withVectors = await brain.batchGet(ids, { includeVectors: true })
|
||||
```
|
||||
|
||||
### 4. **Error Handling**
|
||||
|
||||
```typescript
|
||||
// Batch operations never throw for missing entities
|
||||
const results = await brain.batchGet(ids)
|
||||
|
||||
// Check results
|
||||
for (const id of ids) {
|
||||
if (results.has(id)) {
|
||||
// Entity exists
|
||||
const entity = results.get(id)
|
||||
} else {
|
||||
// Entity missing (not an error)
|
||||
console.log(`Entity ${id} not found`)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Comprehensive test coverage in `tests/integration/storage-batch-operations.test.ts`:
|
||||
|
||||
```bash
|
||||
npx vitest run tests/integration/storage-batch-operations.test.ts
|
||||
```
|
||||
|
||||
**Test Coverage:**
|
||||
- ✅ brain.batchGet() high-level API
|
||||
- ✅ storage.getNounMetadataBatch() with ID-first paths
|
||||
- ✅ COW integration (branch isolation, inheritance)
|
||||
- ✅ storage.getVerbsBySourceBatch() relationship queries
|
||||
- ✅ VFS integration (PathResolver.getChildren())
|
||||
- ✅ Performance benchmarks (N+1 elimination)
|
||||
- ✅ Error handling (partial failures, empty batches, duplicates)
|
||||
- ✅ ID-first storage verification
|
||||
- ✅ Sharding preservation
|
||||
|
||||
**Results:** 23 tests passing ✅
|
||||
|
||||
---
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Architecture Layers
|
||||
|
||||
```
|
||||
User Code (brain.batchGet)
|
||||
↓
|
||||
High-Level API (src/brainy.ts)
|
||||
↓
|
||||
Storage Layer (src/storage/baseStorage.ts)
|
||||
↓
|
||||
Adapter Layer (readBatchFromAdapter)
|
||||
↓
|
||||
Storage Adapter (FileSystemStorage / MemoryStorage)
|
||||
```
|
||||
|
||||
### Parallel Reads
|
||||
|
||||
Both shipped adapters fall back to `Promise.all` over individual reads:
|
||||
|
||||
```typescript
|
||||
// BaseStorage.readBatchFromAdapter()
|
||||
return await Promise.all(resolvedPaths.map(path => this.read(path)))
|
||||
```
|
||||
|
||||
**Shipped Adapters:**
|
||||
- MemoryStorage
|
||||
- FileSystemStorage
|
||||
|
||||
---
|
||||
|
||||
## API Summary
|
||||
|
||||
- `brain.batchGet(ids, options?)` - High-level batch entity retrieval
|
||||
- `storage.getNounMetadataBatch(ids)` - Storage-level metadata batch
|
||||
- `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries
|
||||
|
||||
**Performance Improvements:**
|
||||
- VFS operations: single batched pass instead of N sequential reads
|
||||
- Entity retrieval: N+1 reads collapsed into one batched pass
|
||||
- Zero N+1 query patterns
|
||||
|
||||
**Compatibility:**
|
||||
- ✅ ID-first storage
|
||||
- ✅ Sharding (256 shards)
|
||||
- ✅ Generational MVCC — batch reads serve the live generation; pinned `Db` views serve the past
|
||||
- ✅ All indexes respected (vector, metadata, graph adjacency)
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
- **Documentation:** `/docs/BATCHING.md`, `/docs/PERFORMANCE.md`
|
||||
- **Tests:** `/tests/integration/storage-batch-operations.test.ts`
|
||||
- **Issues:** https://github.com/soulcraft/brainy/issues
|
||||
- **Discussions:** https://github.com/soulcraft/brainy/discussions
|
||||
|
||||
---
|
||||
|
||||
**Built with ❤️ for enterprise-scale knowledge graphs**
|
||||
271
docs/DATA_MODEL.md
Normal file
271
docs/DATA_MODEL.md
Normal file
|
|
@ -0,0 +1,271 @@
|
|||
# Data Model
|
||||
|
||||
> How Brainy stores entities and relationships, and the critical distinction between `data` and `metadata`.
|
||||
|
||||
---
|
||||
|
||||
## Entity (Noun)
|
||||
|
||||
An entity is the fundamental data unit in Brainy. Every entity has:
|
||||
|
||||
| Field | Type | Indexed | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `id` | `string` | Primary key | UUID v4 (auto-generated or custom) |
|
||||
| `data` | `any` | **HNSW vector index** | Content used for semantic/hybrid search. Strings auto-embed. |
|
||||
| `metadata` | `object` | **MetadataIndex** | Structured queryable fields (tags, dates, flags, etc.) |
|
||||
| `type` | `NounType` | MetadataIndex (as `noun`) | Entity type classification |
|
||||
| `vector` | `number[]` | HNSW | 384-dim embedding (auto-computed from `data` or user-provided) |
|
||||
| `confidence` | `number` | MetadataIndex | Type classification confidence (0-1) |
|
||||
| `weight` | `number` | MetadataIndex | Entity importance/salience (0-1) |
|
||||
| `service` | `string` | MetadataIndex | Multi-tenancy identifier |
|
||||
| `createdAt` | `number` | MetadataIndex | Creation timestamp (ms since epoch) |
|
||||
| `updatedAt` | `number` | MetadataIndex | Last update timestamp (ms since epoch) |
|
||||
| `createdBy` | `object` | MetadataIndex | Source augmentation info |
|
||||
|
||||
### Example
|
||||
|
||||
```typescript
|
||||
const id = await brain.add({
|
||||
data: 'John Smith is a software engineer at Acme Corp', // → embedded into vector
|
||||
type: NounType.Person,
|
||||
metadata: { // → indexed, queryable via where filters
|
||||
role: 'engineer',
|
||||
department: 'backend',
|
||||
yearsExperience: 8
|
||||
},
|
||||
confidence: 0.95,
|
||||
weight: 0.7
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Relationship (Verb)
|
||||
|
||||
A relationship is a typed, directed edge connecting two entities.
|
||||
|
||||
| Field | Type | Indexed | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `id` | `string` | Primary key | UUID v4 (auto-generated) |
|
||||
| `from` | `string` | **GraphAdjacencyIndex** | Source entity ID |
|
||||
| `to` | `string` | **GraphAdjacencyIndex** | Target entity ID |
|
||||
| `type` | `VerbType` | GraphAdjacencyIndex (as `verb`) | Relationship type classification |
|
||||
| `data` | `any` | — | Opaque content (overrides auto-computed vector if provided) |
|
||||
| `metadata` | `object` | — | Structured fields on the edge |
|
||||
| `weight` | `number` | — | Connection strength (0-1, default: 1.0) |
|
||||
| `confidence` | `number` | — | Relationship certainty (0-1) |
|
||||
| `evidence` | `RelationEvidence` | — | Why this relationship was detected |
|
||||
| `createdAt` | `number` | — | Creation timestamp (ms since epoch) |
|
||||
| `updatedAt` | `number` | — | Last update timestamp (ms since epoch) |
|
||||
| `service` | `string` | — | Multi-tenancy identifier |
|
||||
|
||||
### Example
|
||||
|
||||
```typescript
|
||||
const relId = await brain.relate({
|
||||
from: personId,
|
||||
to: projectId,
|
||||
type: VerbType.WorksOn,
|
||||
data: 'Lead engineer on the AI module', // Optional: content for this edge
|
||||
metadata: { // Optional: queryable edge fields
|
||||
role: 'lead',
|
||||
startDate: '2024-01-15'
|
||||
},
|
||||
weight: 0.9
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data vs Metadata
|
||||
|
||||
This is the most important concept in Brainy's storage model:
|
||||
|
||||
### `data` — Content for Semantic Search
|
||||
|
||||
- Embedded into a 384-dimensional vector via the WASM embedding engine
|
||||
- Searchable via **semantic similarity** (HNSW vector index) and **hybrid text+semantic** search
|
||||
- Queried by passing `query` to `find()`:
|
||||
```typescript
|
||||
brain.find({ query: 'machine learning algorithms' })
|
||||
```
|
||||
- **NOT** indexed by MetadataIndex — you cannot use `where` filters on `data`
|
||||
- Stored opaquely: strings, objects, numbers — anything goes
|
||||
|
||||
### `metadata` — Structured Queryable Fields
|
||||
|
||||
- Indexed by MetadataIndex with O(1) lookups per field
|
||||
- Queryable via `where` filters using [BFO operators](./QUERY_OPERATORS.md):
|
||||
```typescript
|
||||
brain.find({
|
||||
where: {
|
||||
department: 'engineering',
|
||||
yearsExperience: { greaterThan: 5 },
|
||||
tags: { contains: 'senior' }
|
||||
}
|
||||
})
|
||||
```
|
||||
- **NOT** used for vector/semantic search
|
||||
- Must be a flat or lightly nested object
|
||||
|
||||
### Quick Reference
|
||||
|
||||
| | `data` | `metadata` |
|
||||
|---|---|---|
|
||||
| **Purpose** | Content for embedding / semantic search | Structured fields for filtering |
|
||||
| **Searched by** | `find({ query })` — vector similarity, hybrid text+semantic | `find({ where })` — exact, range, set operators |
|
||||
| **Indexed by** | HNSW vector index | MetadataIndex |
|
||||
| **Queryable with operators?** | No | Yes (`equals`, `greaterThan`, `oneOf`, etc.) |
|
||||
| **Auto-embedded?** | Yes (strings → 384-dim vectors) | No |
|
||||
| **Typical content** | Text descriptions, document content | Tags, dates, status flags, categories, numeric fields |
|
||||
|
||||
### Common Pattern
|
||||
|
||||
```typescript
|
||||
// Add an article
|
||||
await brain.add({
|
||||
data: 'A deep dive into transformer architectures and attention mechanisms',
|
||||
type: NounType.Document,
|
||||
metadata: {
|
||||
title: 'Transformer Deep Dive',
|
||||
author: 'Dr. Chen',
|
||||
publishedYear: 2024,
|
||||
tags: ['AI', 'transformers', 'NLP'],
|
||||
status: 'published'
|
||||
}
|
||||
})
|
||||
|
||||
// Search by content (semantic — searches data)
|
||||
const results = await brain.find({ query: 'neural network attention' })
|
||||
|
||||
// Filter by fields (exact — queries metadata)
|
||||
const recent = await brain.find({
|
||||
where: {
|
||||
publishedYear: { greaterThan: 2023 },
|
||||
status: 'published'
|
||||
}
|
||||
})
|
||||
|
||||
// Combine both (Triple Intelligence)
|
||||
const precise = await brain.find({
|
||||
query: 'attention mechanisms', // Semantic search on data
|
||||
where: { author: 'Dr. Chen' }, // Metadata filter
|
||||
connected: { from: authorId, depth: 1 } // Graph traversal
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Storage Field Naming
|
||||
|
||||
Internally, Brainy uses different field names in storage vs the public API:
|
||||
|
||||
| Public API (Entity/Relation) | Storage (metadata object) | Notes |
|
||||
|------------------------------|--------------------------|-------|
|
||||
| `type` | `noun` | Entity type stored as `noun` |
|
||||
| `from` | `sourceId` | Relationship source |
|
||||
| `to` | `targetId` | Relationship target |
|
||||
| `type` (on Relation) | `verb` | Relationship type stored as `verb` |
|
||||
|
||||
When querying with `find()`, you can use:
|
||||
- `type` parameter (convenience alias, equivalent to `where.noun`)
|
||||
- `where.noun` directly
|
||||
|
||||
```typescript
|
||||
// These are equivalent:
|
||||
brain.find({ type: NounType.Person })
|
||||
brain.find({ where: { noun: NounType.Person } })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Standard Metadata Fields
|
||||
|
||||
When you add an entity, Brainy stores these standard fields in the metadata object alongside your custom fields:
|
||||
|
||||
| Field | Set By | Description |
|
||||
|-------|--------|-------------|
|
||||
| `noun` | System | Entity type (NounType enum value) |
|
||||
| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. |
|
||||
| `data` | System | The raw `data` value (stored opaquely) |
|
||||
| `createdAt` | System | Creation timestamp |
|
||||
| `updatedAt` | System | Last update timestamp |
|
||||
| `confidence` | User | Type classification confidence |
|
||||
| `weight` | User | Entity importance |
|
||||
| `service` | User | Multi-tenancy identifier |
|
||||
| `createdBy` | User/System | Source augmentation |
|
||||
|
||||
On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**.
|
||||
|
||||
### Subtype — sub-classification within a NounType
|
||||
|
||||
`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type:
|
||||
|
||||
```typescript
|
||||
// A Person who is an employee:
|
||||
await brain.add({
|
||||
data: 'Avery Brooks — runs the AI lab',
|
||||
type: NounType.Person,
|
||||
subtype: 'employee',
|
||||
metadata: { department: 'ai-lab' }
|
||||
})
|
||||
|
||||
// A Document that is an invoice:
|
||||
await brain.add({
|
||||
data: 'INV-2026-001',
|
||||
type: NounType.Document,
|
||||
subtype: 'invoice',
|
||||
metadata: { amount: 1500 }
|
||||
})
|
||||
```
|
||||
|
||||
`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`.
|
||||
|
||||
### Subtype — sub-classification within a VerbType (7.30+)
|
||||
|
||||
Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive — a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field on `HNSWVerbWithMetadata` and on the public `Relation<T>`:
|
||||
|
||||
```typescript
|
||||
await brain.relate({
|
||||
from: ceoId,
|
||||
to: vpId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'direct', // top-level standard field
|
||||
metadata: { since: '2025-Q1' } // user-custom fields stay in metadata
|
||||
})
|
||||
```
|
||||
|
||||
Fast-path filter on the verb side:
|
||||
|
||||
```typescript
|
||||
const direct = await brain.related({
|
||||
from: ceoId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'direct'
|
||||
})
|
||||
```
|
||||
|
||||
The verb-side rollup at `_system/verb-subtype-statistics.json` mirrors the noun-side `_system/subtype-statistics.json` — same shape, same self-heal machinery. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`.
|
||||
|
||||
Verbs and nouns now have full capability parity — every API on the noun side has a verb-side mirror, including the new `brain.updateRelation()` (which closed a pre-7.30 gap where relationships had no update path).
|
||||
|
||||
### Standard verb fields
|
||||
|
||||
The verb-side equivalent of `STANDARD_ENTITY_FIELDS` is `STANDARD_VERB_FIELDS`, exported from `src/coreTypes.ts`. Verb-specific standard fields:
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| `verb` | The VerbType enum value |
|
||||
| `sourceId` / `targetId` | The two endpoints of the relationship |
|
||||
| `subtype` | Sub-classification within the VerbType (7.30+) |
|
||||
| `confidence`, `weight`, `createdAt`, `updatedAt`, `service`, `createdBy`, `data` | Same semantics as the noun-side standard fields |
|
||||
|
||||
The companion `resolveVerbField(verb, field)` helper resolves field paths the same way `resolveEntityField` does for nouns: standard fields first, metadata fallback for everything else.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [API Reference](./api/README.md) — Complete API documentation
|
||||
- [Query Operators](./QUERY_OPERATORS.md) — All BFO operators with examples
|
||||
- [Find System](./FIND_SYSTEM.md) — Natural language find() details
|
||||
1166
docs/DEVELOPER_LEARNING_PATH.md
Normal file
1166
docs/DEVELOPER_LEARNING_PATH.md
Normal file
File diff suppressed because it is too large
Load diff
1423
docs/FIND_SYSTEM.md
Normal file
1423
docs/FIND_SYSTEM.md
Normal file
File diff suppressed because it is too large
Load diff
569
docs/MIGRATION-V3-TO-V4.md
Normal file
569
docs/MIGRATION-V3-TO-V4.md
Normal file
|
|
@ -0,0 +1,569 @@
|
|||
# Brainy v3 → v4.0.0 Migration Guide
|
||||
|
||||
> **Migration Complexity**: Low
|
||||
> **Breaking Changes**: None (fully backward compatible)
|
||||
> **New Features**: Lifecycle management, batch operations, compression, quota monitoring
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy v4.0.0 is a **backward-compatible** release focused on production-ready cost optimization features. Your existing v3 code will continue to work without modifications, but you'll want to enable the new v4.0.0 features for significant cost savings.
|
||||
|
||||
**Key Benefits of Upgrading:**
|
||||
- 💰 **96% cost savings** with lifecycle policies
|
||||
- 🚀 **1000x faster** bulk deletions with batch operations
|
||||
- 📦 **60-80% space savings** with gzip compression
|
||||
- 📊 **Real-time quota monitoring** for OPFS
|
||||
- 🎯 **Zero downtime** migration
|
||||
|
||||
## What's New in v4.0.0
|
||||
|
||||
### 1. Lifecycle Management (Cloud Storage)
|
||||
|
||||
**Automatic tier transitions for massive cost savings:**
|
||||
|
||||
```typescript
|
||||
// NEW in v4.0.0
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-data',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' },
|
||||
{ days: 90, storageClass: 'GLACIER' }
|
||||
]
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
**Supported on:**
|
||||
- ✅ AWS S3 (Lifecycle + Intelligent-Tiering)
|
||||
- ✅ Google Cloud Storage (Lifecycle + Autoclass)
|
||||
- ✅ Azure Blob Storage (Lifecycle policies)
|
||||
|
||||
### 2. Batch Operations
|
||||
|
||||
**1000x faster bulk deletions:**
|
||||
|
||||
```typescript
|
||||
// v3: Delete one at a time (slow, expensive)
|
||||
for (const id of idsToDelete) {
|
||||
await brain.remove(id) // 1000 API calls for 1000 entities
|
||||
}
|
||||
|
||||
// v4.0.0: Batch delete (fast, cheap)
|
||||
const paths = idsToDelete.flatMap(id => [
|
||||
`entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
|
||||
`entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
|
||||
])
|
||||
await storage.batchDelete(paths) // 1 API call for 1000 objects (S3)
|
||||
```
|
||||
|
||||
**Efficiency gains:**
|
||||
- S3: 1000 objects per batch
|
||||
- GCS: 100 objects per batch
|
||||
- Azure: 256 objects per batch
|
||||
|
||||
### 3. Compression (FileSystem)
|
||||
|
||||
**60-80% space savings for local storage:**
|
||||
|
||||
```typescript
|
||||
// NEW in v4.0.0
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // Enable gzip compression
|
||||
}
|
||||
})
|
||||
|
||||
// Automatic compression/decompression on all reads/writes
|
||||
```
|
||||
|
||||
### 4. Quota Monitoring (OPFS)
|
||||
|
||||
**Prevent quota exceeded errors in browsers:**
|
||||
|
||||
```typescript
|
||||
// NEW in v4.0.0
|
||||
const status = await storage.getStorageStatus()
|
||||
|
||||
if (status.details.usagePercent > 80) {
|
||||
console.warn('Approaching quota limit:', status.details)
|
||||
// Take action: cleanup old data, notify user, etc.
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Tier Management (Azure)
|
||||
|
||||
**Manual or automatic tier transitions:**
|
||||
|
||||
```typescript
|
||||
// NEW in v4.0.0
|
||||
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
|
||||
await storage.batchChangeTier([blob1, blob2], 'Archive') // 99% savings
|
||||
|
||||
// Rehydrate from Archive when needed
|
||||
await storage.rehydrateBlob(blobPath, 'High') // 1-hour rehydration
|
||||
```
|
||||
|
||||
## Storage Architecture Changes
|
||||
|
||||
### v3.x Storage Structure
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
├── nouns/
|
||||
│ └── {uuid}.json # Single file per entity
|
||||
├── verbs/
|
||||
│ └── {uuid}.json # Single file per relationship
|
||||
├── metadata/
|
||||
│ └── __metadata_*.json # Indexes
|
||||
└── _system/
|
||||
└── statistics.json
|
||||
```
|
||||
|
||||
### v4.0.0 Storage Structure (Automatic Migration)
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
├── entities/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # Vector + HNSW graph (NEW)
|
||||
│ │ │ ├── 00/ ... ff/ # 256 UUID shards (NEW)
|
||||
│ │ └── metadata/ # Business data (NEW)
|
||||
│ │ ├── 00/ ... ff/ # 256 UUID shards (NEW)
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (NEW)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ └── metadata/ # Relationship data (NEW)
|
||||
│ ├── 00/ ... ff/
|
||||
└── _system/ # Unchanged
|
||||
└── __metadata_*.json
|
||||
```
|
||||
|
||||
**Key Changes:**
|
||||
1. **Metadata/Vector Separation**: Entities split into 2 files for optimal I/O
|
||||
2. **UUID-Based Sharding**: 256 shards for cloud storage optimization
|
||||
3. **Automatic Migration**: Brainy handles migration transparently on first run
|
||||
|
||||
## Migration Steps
|
||||
|
||||
### Step 1: Update Brainy Package
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/brainy@latest
|
||||
```
|
||||
|
||||
**Check your version:**
|
||||
```bash
|
||||
npm list @soulcraft/brainy
|
||||
# Should show: @soulcraft/brainy@4.0.0
|
||||
```
|
||||
|
||||
### Step 2: No Code Changes Required! ✅
|
||||
|
||||
Your existing v3 code will work without modifications:
|
||||
|
||||
```typescript
|
||||
// This v3 code works perfectly in v4.0.0
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './data' }
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
await brain.add("content", { type: "entity" })
|
||||
const results = await brain.search("query")
|
||||
```
|
||||
|
||||
### Step 3: First Run (Automatic Migration)
|
||||
|
||||
On first initialization with v4.0.0:
|
||||
|
||||
1. **Brainy detects v3 storage structure**
|
||||
2. **Transparently migrates to v4.0.0 structure**:
|
||||
- Creates `entities/` directory
|
||||
- Migrates `nouns/` → `entities/nouns/vectors/` + `entities/nouns/metadata/`
|
||||
- Migrates `verbs/` → `entities/verbs/vectors/` + `entities/verbs/metadata/`
|
||||
- Applies UUID-based sharding
|
||||
3. **Old structure preserved** (optional cleanup later)
|
||||
|
||||
**Migration time:**
|
||||
- 10K entities: ~1 minute
|
||||
- 100K entities: ~10 minutes
|
||||
- 1M entities: ~2 hours
|
||||
|
||||
**Zero downtime:**
|
||||
- Migration happens during init()
|
||||
- No data loss
|
||||
- Automatic rollback on error
|
||||
|
||||
### Step 4: Enable v4.0.0 Features (Optional but Recommended)
|
||||
|
||||
#### Enable Lifecycle Policies (Cloud Storage)
|
||||
|
||||
**AWS S3:**
|
||||
```typescript
|
||||
// After init()
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'optimize-storage',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' },
|
||||
{ days: 90, storageClass: 'GLACIER' }
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// Or use Intelligent-Tiering (recommended)
|
||||
await storage.enableIntelligentTiering('entities/', 'auto-optimize')
|
||||
```
|
||||
|
||||
**Google Cloud Storage:**
|
||||
```typescript
|
||||
await storage.enableAutoclass({
|
||||
terminalStorageClass: 'ARCHIVE'
|
||||
})
|
||||
```
|
||||
|
||||
**Azure Blob Storage:**
|
||||
```typescript
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
name: 'optimize-blobs',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: { blobTypes: ['blockBlob'] },
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 30 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 90 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
#### Enable Compression (FileSystem)
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // NEW: 60-80% space savings
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
#### Use Batch Operations
|
||||
|
||||
```typescript
|
||||
// Replace individual deletes with batch delete
|
||||
const idsToDelete = [/* ... */]
|
||||
const paths = idsToDelete.flatMap(id => {
|
||||
const shard = id.substring(0, 2)
|
||||
return [
|
||||
`entities/nouns/vectors/${shard}/${id}.json`,
|
||||
`entities/nouns/metadata/${shard}/${id}.json`
|
||||
]
|
||||
})
|
||||
|
||||
await storage.batchDelete(paths) // Much faster!
|
||||
```
|
||||
|
||||
#### Monitor Quota (OPFS)
|
||||
|
||||
```typescript
|
||||
// Periodically check quota in browser apps
|
||||
setInterval(async () => {
|
||||
const status = await storage.getStorageStatus()
|
||||
if (status.details.usagePercent > 80) {
|
||||
notifyUser('Storage approaching limit')
|
||||
}
|
||||
}, 60000) // Check every minute
|
||||
```
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
### Guaranteed to Work (No Changes Needed)
|
||||
|
||||
✅ All v3 APIs remain unchanged
|
||||
✅ Storage adapters backward compatible
|
||||
✅ Metadata structure unchanged
|
||||
✅ Query APIs unchanged
|
||||
✅ Configuration options unchanged
|
||||
|
||||
### New Optional APIs (Add When Ready)
|
||||
|
||||
- `storage.setLifecyclePolicy()` - NEW in v4.0.0
|
||||
- `storage.getLifecyclePolicy()` - NEW in v4.0.0
|
||||
- `storage.removeLifecyclePolicy()` - NEW in v4.0.0
|
||||
- `storage.enableIntelligentTiering()` - NEW in v4.0.0 (S3)
|
||||
- `storage.enableAutoclass()` - NEW in v4.0.0 (GCS)
|
||||
- `storage.batchDelete()` - NEW in v4.0.0
|
||||
- `storage.changeBlobTier()` - NEW in v4.0.0 (Azure)
|
||||
- `storage.getStorageStatus()` - Enhanced in v4.0.0
|
||||
|
||||
## Testing Your Migration
|
||||
|
||||
### 1. Test in Development First
|
||||
|
||||
```typescript
|
||||
// Create test brain with v4.0.0
|
||||
const testBrain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './test-data' }
|
||||
})
|
||||
|
||||
await testBrain.init()
|
||||
|
||||
// Verify migration
|
||||
console.log('Initialization complete')
|
||||
|
||||
// Test basic operations
|
||||
const id = await testBrain.add("test content", { type: "test" })
|
||||
const results = await testBrain.search("test")
|
||||
console.log('Basic operations working:', results.length > 0)
|
||||
```
|
||||
|
||||
### 2. Verify Storage Structure
|
||||
|
||||
```bash
|
||||
# Check new directory structure
|
||||
ls -la ./test-data/entities/nouns/vectors/
|
||||
# Should see: 00/ 01/ 02/ ... ff/ (256 shards)
|
||||
|
||||
ls -la ./test-data/entities/nouns/metadata/
|
||||
# Should see: 00/ 01/ 02/ ... ff/ (256 shards)
|
||||
```
|
||||
|
||||
### 3. Verify Data Integrity
|
||||
|
||||
```typescript
|
||||
// Query all entities
|
||||
const allEntities = await testBrain.find({})
|
||||
console.log('Total entities:', allEntities.length)
|
||||
|
||||
// Verify specific entities
|
||||
const entity = await testBrain.get(knownEntityId)
|
||||
console.log('Entity retrieved:', entity !== null)
|
||||
```
|
||||
|
||||
### 4. Test Performance
|
||||
|
||||
```typescript
|
||||
// Benchmark search
|
||||
const start = Date.now()
|
||||
const results = await testBrain.search("query")
|
||||
const duration = Date.now() - start
|
||||
console.log('Search time:', duration, 'ms')
|
||||
|
||||
// Should be similar or faster than v3
|
||||
```
|
||||
|
||||
## Rollback Procedure (If Needed)
|
||||
|
||||
If you encounter issues, you can rollback:
|
||||
|
||||
### Option 1: Rollback Package
|
||||
|
||||
```bash
|
||||
# Reinstall v3
|
||||
npm install @soulcraft/brainy@^3.50.0
|
||||
|
||||
# Restart application
|
||||
```
|
||||
|
||||
**Important:** v3 can still read v3-structured data (preserved during migration)
|
||||
|
||||
### Option 2: Restore from Backup
|
||||
|
||||
```bash
|
||||
# If you backed up data before migration
|
||||
rm -rf ./data
|
||||
cp -r ./data-backup ./data
|
||||
|
||||
# Reinstall v3
|
||||
npm install @soulcraft/brainy@^3.50.0
|
||||
```
|
||||
|
||||
## Common Migration Scenarios
|
||||
|
||||
### Scenario 1: Small Application (<10K Entities)
|
||||
|
||||
**Migration time:** 1 minute
|
||||
**Recommended approach:**
|
||||
1. Update npm package
|
||||
2. Restart application (automatic migration)
|
||||
3. Enable lifecycle policies immediately
|
||||
|
||||
### Scenario 2: Medium Application (10K-1M Entities)
|
||||
|
||||
**Migration time:** 10 minutes - 2 hours
|
||||
**Recommended approach:**
|
||||
1. Backup data
|
||||
2. Update npm package
|
||||
3. Schedule maintenance window
|
||||
4. Restart application (automatic migration)
|
||||
5. Verify data integrity
|
||||
6. Enable lifecycle policies
|
||||
|
||||
### Scenario 3: Large Application (1M+ Entities)
|
||||
|
||||
**Migration time:** 2-24 hours
|
||||
**Recommended approach:**
|
||||
1. **Backup data** (critical!)
|
||||
2. Test migration on staging environment
|
||||
3. Schedule extended maintenance window
|
||||
4. Update npm package on production
|
||||
5. Restart application (automatic migration)
|
||||
6. Monitor migration progress
|
||||
7. Verify data integrity thoroughly
|
||||
8. Enable lifecycle policies gradually
|
||||
|
||||
## Cost Savings After Migration
|
||||
|
||||
### Enable All v4.0.0 Features
|
||||
|
||||
**500TB Dataset Example:**
|
||||
|
||||
**Before v4.0.0 (v3 with AWS S3 Standard):**
|
||||
```
|
||||
Storage: $138,000/year
|
||||
Operations: $5,000/year
|
||||
Total: $143,000/year
|
||||
```
|
||||
|
||||
**After v4.0.0 (with Intelligent-Tiering):**
|
||||
```
|
||||
Storage: $51,000/year (64% savings)
|
||||
Operations: $5,000/year
|
||||
Total: $56,000/year
|
||||
```
|
||||
|
||||
**After v4.0.0 (with Lifecycle Policies):**
|
||||
```
|
||||
Storage: $5,940/year (96% savings!)
|
||||
Operations: $5,000/year
|
||||
Total: $10,940/year
|
||||
```
|
||||
|
||||
**Annual Savings: $132,060 (96% reduction)**
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Migration takes too long
|
||||
|
||||
**Solution:**
|
||||
- Migration is I/O bound
|
||||
- For 1M+ entities, consider:
|
||||
- Running during off-peak hours
|
||||
- Using faster storage (SSD vs HDD)
|
||||
- Increasing available memory
|
||||
- Running on more powerful instance
|
||||
|
||||
### Issue: "Storage structure not recognized"
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Manually trigger migration
|
||||
await brain.storage.migrateToV4() // If automatic migration fails
|
||||
|
||||
// Or start fresh (data loss warning!)
|
||||
await brain.storage.clear()
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
### Issue: Lifecycle policy not working
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Verify policy is set
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Active rules:', policy.rules)
|
||||
|
||||
// Cloud providers may take 24-48 hours to start transitions
|
||||
// Check again after 2 days
|
||||
|
||||
// Verify in cloud console:
|
||||
// - AWS: S3 → Bucket → Management → Lifecycle
|
||||
// - GCS: Storage → Bucket → Lifecycle
|
||||
// - Azure: Storage Account → Lifecycle management
|
||||
```
|
||||
|
||||
### Issue: Batch delete not working
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Ensure storage adapter supports batch delete
|
||||
const status = await storage.getStorageStatus()
|
||||
console.log('Storage type:', status.type)
|
||||
|
||||
// Batch delete requires:
|
||||
// - S3CompatibleStorage ✅
|
||||
// - GcsStorage ✅
|
||||
// - AzureBlobStorage ✅
|
||||
// - FileSystemStorage ✅
|
||||
// - OPFSStorage ✅
|
||||
// - MemoryStorage ✅
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. ✅ **Backup before upgrading** (especially for large datasets)
|
||||
2. ✅ **Test on staging first** (verify migration works)
|
||||
3. ✅ **Monitor during migration** (watch logs for errors)
|
||||
4. ✅ **Enable lifecycle policies immediately** (start saving costs)
|
||||
5. ✅ **Use batch operations** (for any bulk cleanup)
|
||||
6. ✅ **Monitor quota** (OPFS browser apps)
|
||||
7. ✅ **Enable compression** (FileSystem storage)
|
||||
|
||||
## Getting Help
|
||||
|
||||
**Documentation:**
|
||||
- [AWS S3 Cost Optimization Guide](./operations/cost-optimization-aws-s3.md)
|
||||
- [GCS Cost Optimization Guide](./operations/cost-optimization-gcs.md)
|
||||
- [Azure Cost Optimization Guide](./operations/cost-optimization-azure.md)
|
||||
- [Cloudflare R2 Cost Optimization Guide](./operations/cost-optimization-cloudflare-r2.md)
|
||||
|
||||
**Support:**
|
||||
- GitHub Issues: [https://github.com/soulcraft/brainy/issues](https://github.com/soulcraft/brainy/issues)
|
||||
- GitHub Discussions: [https://github.com/soulcraft/brainy/discussions](https://github.com/soulcraft/brainy/discussions)
|
||||
|
||||
## Summary
|
||||
|
||||
**Migration Checklist:**
|
||||
- ✅ Backup data
|
||||
- ✅ Update npm package (`npm install @soulcraft/brainy@latest`)
|
||||
- ✅ Restart application (automatic migration)
|
||||
- ✅ Verify data integrity
|
||||
- ✅ Enable lifecycle policies
|
||||
- ✅ Enable compression (FileSystem)
|
||||
- ✅ Use batch operations
|
||||
- ✅ Monitor cost savings
|
||||
|
||||
**Expected Results:**
|
||||
- ✅ Zero downtime migration
|
||||
- ✅ Full backward compatibility
|
||||
- ✅ 60-96% cost savings
|
||||
- ✅ 1000x faster bulk operations
|
||||
- ✅ 60-80% space savings (with compression)
|
||||
|
||||
**Timeline:**
|
||||
- Small app (<10K): 1 minute migration
|
||||
- Medium app (10K-1M): 10 minutes - 2 hours
|
||||
- Large app (1M+): 2-24 hours
|
||||
|
||||
**Welcome to Brainy v4.0.0! 🎉**
|
||||
|
||||
---
|
||||
|
||||
**Version**: v4.0.0
|
||||
**Migration Difficulty**: Low
|
||||
**Breaking Changes**: None
|
||||
**Recommended Upgrade**: Yes (significant cost savings)
|
||||
|
|
@ -1,225 +0,0 @@
|
|||
# 🚀 Brainy Performance Impact Analysis
|
||||
|
||||
## Executive Summary: ZERO Performance Degradation
|
||||
|
||||
**The new features (augmentations, premium connectors, monitoring) have ZERO impact on core Brainy performance.**
|
||||
|
||||
---
|
||||
|
||||
## 📊 Performance Metrics Comparison
|
||||
|
||||
### Core Operations (Unchanged)
|
||||
| Operation | v0.45 (Before) | v0.56 (After) | Impact |
|
||||
|-----------|---------------|---------------|---------|
|
||||
| Vector Search (1M) | 2-8ms | 2-8ms | **0%** |
|
||||
| Graph Traversal | 1-3ms | 1-3ms | **0%** |
|
||||
| Combined Query | 5-15ms | 5-15ms | **0%** |
|
||||
| Add Operation | <1ms | <1ms | **0%** |
|
||||
| Relate Operation | <1ms | <1ms | **0%** |
|
||||
| Init Time | 150ms | 150ms* | **0%** |
|
||||
|
||||
*Augmentations only load if explicitly used
|
||||
|
||||
### Memory Footprint
|
||||
| Component | Size | When Loaded | Impact |
|
||||
|-----------|------|------------|---------|
|
||||
| Core Brainy | 643KB | Always | Baseline |
|
||||
| Cortex | +12KB | On demand | Optional |
|
||||
| Premium Connectors | +8KB each | Never (external) | **0%** |
|
||||
| Monitoring | +5KB | On demand | Optional |
|
||||
| Chat Interface | +7KB | On demand | Optional |
|
||||
|
||||
**Total core size unchanged: 643KB**
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Why Zero Impact?
|
||||
|
||||
### 1. Lazy Loading Architecture
|
||||
```javascript
|
||||
// Augmentations ONLY load when explicitly called
|
||||
const brainy = new BrainyData() // No augmentations loaded
|
||||
await brainy.init() // Still no augmentations
|
||||
|
||||
// This is when augmentation loads (if at all)
|
||||
await brainy.augment('neural-import', data) // NOW it loads
|
||||
```
|
||||
|
||||
### 2. External Premium Features
|
||||
```javascript
|
||||
// Premium features live in separate package
|
||||
import { NotionConnector } from '@soulcraft/brainy-quantum-vault'
|
||||
// ↑ This is a SEPARATE npm package, not in core
|
||||
```
|
||||
|
||||
### 3. Optional Monitoring
|
||||
```javascript
|
||||
// Monitoring is 100% opt-in
|
||||
const brainy = new BrainyData({
|
||||
monitoring: false // Default - no overhead
|
||||
})
|
||||
|
||||
// Even when enabled, uses efficient counters
|
||||
const brainy = new BrainyData({
|
||||
monitoring: true // Adds ~0.1ms per operation
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 Actually IMPROVES Performance
|
||||
|
||||
### 1. Smarter Caching
|
||||
- Cortex pre-processes data for faster searches
|
||||
- Augmentation pipeline can cache intermediate results
|
||||
- 95%+ cache hit rates on repeated operations
|
||||
|
||||
### 2. Better Resource Utilization
|
||||
- Monitoring helps identify bottlenecks
|
||||
- Auto-optimization based on usage patterns
|
||||
- Proactive memory management
|
||||
|
||||
### 3. Reduced Network Calls
|
||||
- Transformers.js migration eliminated TensorFlow network calls
|
||||
- Models cached locally after first download
|
||||
- Offline-first architecture
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Benchmark Results
|
||||
|
||||
### Test Environment
|
||||
- **Dataset**: 1M vectors, 10M relationships
|
||||
- **Hardware**: M2 MacBook Pro, 16GB RAM
|
||||
- **Node Version**: 24.4.1
|
||||
|
||||
### Results
|
||||
```
|
||||
Operation: Vector Search (1000 queries)
|
||||
v0.45: 2,134ms total (2.13ms avg)
|
||||
v0.56: 2,089ms total (2.09ms avg)
|
||||
Improvement: 2.1% FASTER
|
||||
|
||||
Operation: Graph Traversal (1000 queries)
|
||||
v0.45: 1,523ms total (1.52ms avg)
|
||||
v0.56: 1,498ms total (1.50ms avg)
|
||||
Improvement: 1.6% FASTER
|
||||
|
||||
Operation: Combined Query (1000 queries)
|
||||
v0.45: 8,234ms total (8.23ms avg)
|
||||
v0.56: 7,988ms total (7.99ms avg)
|
||||
Improvement: 3.0% FASTER
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Production Considerations
|
||||
|
||||
### What DOESN'T Impact Performance
|
||||
✅ Augmentation system (lazy loaded)
|
||||
✅ Premium connectors (external package)
|
||||
✅ Monitoring (opt-in, minimal overhead)
|
||||
✅ Chat interface (loaded on demand)
|
||||
✅ Webhook system (separate process)
|
||||
✅ Backup/restore (offline operations)
|
||||
|
||||
### What COULD Impact Performance (If Misused)
|
||||
⚠️ Running ALL augmentations on EVERY operation
|
||||
⚠️ Enabling verbose monitoring in production
|
||||
⚠️ Not configuring cache limits for large datasets
|
||||
⚠️ Using synchronous augmentations in hot paths
|
||||
|
||||
### Best Practices
|
||||
```javascript
|
||||
// ✅ GOOD: Selective augmentation
|
||||
const result = await brainy.add(data, {
|
||||
augment: ['neural-import'] // Only what you need
|
||||
})
|
||||
|
||||
// ❌ BAD: Unnecessary augmentation
|
||||
const result = await brainy.add(data, {
|
||||
augment: ['*'] // Don't do this in production
|
||||
})
|
||||
|
||||
// ✅ GOOD: Production config
|
||||
const brainy = new BrainyData({
|
||||
monitoring: false, // Or true with sampling
|
||||
cache: {
|
||||
maxSize: '1GB',
|
||||
ttl: 3600
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💡 Architecture Decisions That Preserve Performance
|
||||
|
||||
### 1. Plugin Architecture
|
||||
- Augmentations are plugins, not core modifications
|
||||
- Clean separation of concerns
|
||||
- No coupling between features
|
||||
|
||||
### 2. Event-Driven Design
|
||||
- Augmentations use events, not inline processing
|
||||
- Async by default
|
||||
- Non-blocking operations
|
||||
|
||||
### 3. Progressive Enhancement
|
||||
- Core works without any additions
|
||||
- Features enhance, don't replace
|
||||
- Graceful degradation
|
||||
|
||||
---
|
||||
|
||||
## 📊 Real-World Impact
|
||||
|
||||
### Customer A: E-commerce Search
|
||||
- **Dataset**: 2.5M products
|
||||
- **Usage**: 100K searches/day
|
||||
- **Impact**: 0% slower, 15% less memory (better caching)
|
||||
|
||||
### Customer B: Knowledge Graph
|
||||
- **Dataset**: 500K entities, 5M relationships
|
||||
- **Usage**: Real-time queries
|
||||
- **Impact**: 2% faster (optimized traversal)
|
||||
|
||||
### Customer C: AI Chat Platform
|
||||
- **Dataset**: 100K documents
|
||||
- **Usage**: RAG with chat interface
|
||||
- **Impact**: 30% faster responses (Cortex preprocessing)
|
||||
|
||||
---
|
||||
|
||||
## 🔬 Testing Methodology
|
||||
|
||||
```bash
|
||||
# Run performance benchmarks
|
||||
npm run test:performance
|
||||
|
||||
# Compare versions
|
||||
npm run benchmark:compare v0.45 v0.56
|
||||
|
||||
# Memory profiling
|
||||
npm run profile:memory
|
||||
|
||||
# Load testing
|
||||
npm run test:load -- --concurrent=1000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Conclusion
|
||||
|
||||
**Brainy v0.56 with all new features is:**
|
||||
- ✅ **Same speed or faster** for all operations
|
||||
- ✅ **Same memory footprint** for core functionality
|
||||
- ✅ **More efficient** with smart caching
|
||||
- ✅ **100% backward compatible**
|
||||
- ✅ **Zero impact** unless features explicitly used
|
||||
|
||||
**The augmentation system and premium features are architectural enhancements that maintain Brainy's blazing-fast performance while adding powerful capabilities for those who need them.**
|
||||
|
||||
---
|
||||
|
||||
*Last benchmarked: December 2024*
|
||||
492
docs/PERFORMANCE.md
Normal file
492
docs/PERFORMANCE.md
Normal file
|
|
@ -0,0 +1,492 @@
|
|||
# Brainy Performance & Architecture
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
Brainy achieves high performance through carefully optimized data structures and algorithms. The tables below describe each component by its **algorithmic complexity** — the durable, defensible guarantee. The example latencies are figures from a single 100-item run on one machine (see [Benchmarks](#benchmarks)); they are illustrative, not a committed benchmark, and vary with hardware, embedding model, and storage backend. The one component with a committed scale assertion is the graph adjacency index (`tests/performance/graph-scale-performance.test.ts:238`).
|
||||
|
||||
### Core Performance Summary
|
||||
|
||||
| Component | Operation | Time Complexity | Example latency (100-item run)\* | Data Structure |
|
||||
|-----------|-----------|-----------------|---------------------|----------------|
|
||||
| **Metadata Index** | Exact match | **O(1)** | 0.8ms | `Map<string, Set<string>>` |
|
||||
| **Metadata Index** | Range query | **O(log n) + O(k)** | 0.6ms | Sorted array + binary search |
|
||||
| **Graph Index** | Get neighbors | **O(1)** | 0.09ms | `Map<string, Set<string>>` |
|
||||
| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | Hierarchical graph |
|
||||
| **NLP Parser** | Query parsing | **O(m)** | 8.9ms | 220 pre-computed patterns |
|
||||
| **Type-Field Affinity** | Field matching | **O(f)** | 0.1ms | Type-specific field cache |
|
||||
| **Type Detection** | Noun/Verb matching | **O(t)** | 0.3ms | Pre-embedded type vectors |
|
||||
| **Triple Intelligence** | Combined query | **O(1) to O(log n)** | 1.8ms | Parallel execution |
|
||||
|
||||
\* Illustrative single-run figures at 100 items on one machine — not a committed benchmark. Only the graph index carries an asserted scale bound (measured <1 ms per neighbor lookup up to 1M relationships, `tests/performance/graph-scale-performance.test.ts:238`).
|
||||
|
||||
Where:
|
||||
- `n` = number of items in index
|
||||
- `k` = number of results returned
|
||||
- `m` = number of patterns to check
|
||||
- `f` = number of fields for entity type
|
||||
- `t` = number of types (42 nouns, 127 verbs)
|
||||
|
||||
### brain.get() Metadata-Only Optimization
|
||||
|
||||
`brain.get()` returns **metadata only by default**, skipping the 384-dimensional
|
||||
embedding — the bulk of an entity's payload. Callers that need the vector opt in
|
||||
with `{ includeVectors: true }`.
|
||||
|
||||
| Operation | Default (metadata-only) | With `includeVectors: true` | Use Case |
|
||||
|-----------|-------------------------|-----------------------------|----------|
|
||||
| **brain.get()** | Skips vector load | Loads full vector | VFS, existence checks, metadata |
|
||||
| **VFS readFile() / readdir()** | Inherits metadata-only path | n/a | File operations, directory listings |
|
||||
|
||||
**Key Innovation**: Lazy vector loading — only load the 384-dimensional embedding when explicitly needed.
|
||||
|
||||
The integration test `tests/integration/metadata-only-comprehensive.test.ts:306`
|
||||
asserts metadata-only `get()` is faster than the full-entity `get()`
|
||||
(`metadataTime < fullTime`). The *magnitude* of the speedup is
|
||||
environment-dependent (the percentage assertion in
|
||||
`tests/integration/vfs-performance-v5.11.1.test.ts` is intentionally skipped on
|
||||
CI for that reason), so no fixed percentage is quoted here.
|
||||
|
||||
**Why this matters**:
|
||||
- Most `brain.get()` calls don't need vectors (VFS, admin tools, import utilities, data APIs)
|
||||
- The embedding dominates an entity's serialized size, so skipping it is the largest win
|
||||
- **Zero code changes** for most applications — automatic by default
|
||||
|
||||
**When to use what**:
|
||||
```typescript
|
||||
// DEFAULT: Metadata-only (skips the vector load) - use for:
|
||||
const entity = await brain.get(id)
|
||||
// - VFS operations (readFile, stat, readdir)
|
||||
// - Existence checks: if (await brain.get(id)) ...
|
||||
// - Metadata access: entity.data, entity.type, entity.metadata
|
||||
// - Relationship traversal
|
||||
|
||||
// EXPLICIT: Full entity (same as before) - use ONLY for:
|
||||
const entity = await brain.get(id, { includeVectors: true })
|
||||
// - Computing similarity on THIS entity
|
||||
// - Manual vector operations
|
||||
// - Vector index graph traversal
|
||||
```
|
||||
|
||||
## Architecture Deep Dive
|
||||
|
||||
### 1. Metadata Index - O(1) Lookups
|
||||
|
||||
The `MetadataIndexManager` uses inverted indexes for lightning-fast metadata filtering.
|
||||
|
||||
**UPDATED**: Sorted indices for range queries are now built **incrementally during CRUD operations**. No lazy loading delays - range queries are consistently fast. Binary search insertions maintain O(log n) performance during updates.
|
||||
|
||||
```typescript
|
||||
class MetadataIndexManager {
|
||||
// O(1) exact match via HashMap
|
||||
private indexCache = new Map<string, MetadataIndexEntry>()
|
||||
|
||||
// O(log n) range queries via sorted arrays (incremental updates)
|
||||
private sortedIndices = new Map<string, SortedFieldIndex>()
|
||||
|
||||
// Type-field affinity for intelligent NLP
|
||||
private typeFieldAffinity = new Map<string, Map<string, number>>()
|
||||
|
||||
interface MetadataIndexEntry {
|
||||
field: string
|
||||
value: string | number | boolean
|
||||
ids: Set<string> // O(1) add/remove/has
|
||||
}
|
||||
|
||||
interface SortedFieldIndex {
|
||||
values: Array<[value: any, ids: Set<string>]> // Sorted for O(log n) ranges
|
||||
fieldType: 'number' | 'string' | 'date'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
1. Each field+value combination gets a unique key: `"category:tech"`
|
||||
2. Map lookup is O(1) average case
|
||||
3. Returns a Set of matching IDs instantly
|
||||
|
||||
**Example Query:**
|
||||
```javascript
|
||||
// Query: { where: { category: 'tech' } }
|
||||
// Internally: indexCache.get('category:tech') → O(1)
|
||||
```
|
||||
|
||||
### 2. Range Queries - O(log n)
|
||||
|
||||
For numeric/date fields, Brainy maintains sorted indices:
|
||||
|
||||
```typescript
|
||||
interface SortedFieldIndex {
|
||||
values: Array<[value: any, ids: Set<string>]> // Sorted by value
|
||||
fieldType: 'number' | 'string' | 'date'
|
||||
}
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
1. Binary search to find range start: O(log n)
|
||||
2. Binary search to find range end: O(log n)
|
||||
3. Collect all IDs in range: O(k) where k = items in range
|
||||
|
||||
**Example Query:**
|
||||
```javascript
|
||||
// Query: { where: { age: { greaterThan: 25, lessThan: 40 } } }
|
||||
// Internally: binarySearch(25) + binarySearch(40) + collect
|
||||
```
|
||||
|
||||
### 3. Graph Adjacency Index - O(1) Traversal
|
||||
|
||||
The `GraphAdjacencyIndex` provides instant graph traversal:
|
||||
|
||||
```typescript
|
||||
class GraphAdjacencyIndex {
|
||||
// Bidirectional adjacency lists
|
||||
private sourceIndex = new Map<string, Set<string>>() // id → outgoing
|
||||
private targetIndex = new Map<string, Set<string>>() // id → incoming
|
||||
|
||||
// O(1) neighbor lookup
|
||||
async getNeighbors(id: string, direction: 'in' | 'out' | 'both') {
|
||||
const outgoing = this.sourceIndex.get(id) // O(1)
|
||||
const incoming = this.targetIndex.get(id) // O(1)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Key Innovation:** Pure Map/Set operations - no database queries, no loops, just direct memory access.
|
||||
|
||||
### 4. Vector Index - O(log n)
|
||||
|
||||
The default vector index (`JsHnswVectorIndex`) provides logarithmic approximate nearest neighbor search through a hierarchical graph:
|
||||
|
||||
```typescript
|
||||
class JsHnswVectorIndex {
|
||||
private nouns: Map<string, HNSWNoun> = new Map()
|
||||
|
||||
interface HNSWNoun {
|
||||
id: string
|
||||
vector: number[]
|
||||
connections: Map<number, Set<string>> // layer → neighbors
|
||||
level: number
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
1. Start at entry point (top layer)
|
||||
2. Greedy search to find nearest neighbor at each layer
|
||||
3. Move down layers for progressively finer search
|
||||
4. Each layer has M connections (typically 16)
|
||||
|
||||
**Performance:** O(log n) due to hierarchical structure
|
||||
|
||||
### 5. Type-Aware NLP with Dynamic Field Discovery
|
||||
|
||||
The NLP processor uses **zero hardcoded fields** - everything is discovered dynamically from actual data:
|
||||
|
||||
```typescript
|
||||
class NaturalLanguageProcessor {
|
||||
// Pre-embedded NounTypes (42) and VerbTypes (127) - ONLY hardcoded vocabularies
|
||||
private nounTypeEmbeddings = new Map<string, Vector>()
|
||||
private verbTypeEmbeddings = new Map<string, Vector>()
|
||||
|
||||
// Dynamic field embeddings from actual indexed data
|
||||
private fieldEmbeddings = new Map<string, Vector>()
|
||||
|
||||
// Type-field affinity for intelligent prioritization
|
||||
async getFieldsForType(nounType: NounType) {
|
||||
return this.brain.getFieldsForType(nounType) // Real data patterns
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Type-Aware Intelligence Flow:**
|
||||
1. **Type Detection**: "documents" → `NounType.Document` (semantic similarity)
|
||||
2. **Field Prioritization**: Get fields common to Document type from real data
|
||||
3. **Semantic Field Matching**: "by" → "author" (with type affinity boost)
|
||||
4. **Validation**: Ensure "author" field actually appears with Document entities
|
||||
5. **Query Optimization**: Process low-cardinality type-specific fields first
|
||||
|
||||
**Performance Characteristics:**
|
||||
- Type detection: O(t) where t = 169 total types (42 noun + 127 verb)
|
||||
- Field matching: O(f) where f = fields for detected type (typically 5-15)
|
||||
- Validation: O(1) lookup in type-field affinity map
|
||||
- No hardcoded assumptions - learns from actual data patterns
|
||||
|
||||
### 6. NLP with 220 Pre-computed Patterns
|
||||
|
||||
Pattern matching with embedded templates for instant semantic understanding:
|
||||
|
||||
```typescript
|
||||
// 394KB of embedded patterns compiled into the source
|
||||
export const EMBEDDED_PATTERNS: Pattern[] = [/* 220 patterns */]
|
||||
export const PATTERN_EMBEDDINGS: Float32Array = /* 220 × 384 dimensions */
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
1. Query embedding computed once: O(1) with cached model
|
||||
2. Cosine similarity with 220 patterns: O(m) where m = 220
|
||||
3. Pattern templates enhanced with type context
|
||||
4. No network calls, no external dependencies, no hardcoded fields
|
||||
|
||||
## Parallel Execution
|
||||
|
||||
Triple Intelligence queries execute searches in parallel:
|
||||
|
||||
```javascript
|
||||
// Vector and proximity searches run simultaneously
|
||||
const searchPromises = [
|
||||
this.executeVectorSearch(params), // Runs in parallel
|
||||
this.executeProximitySearch(params) // Runs in parallel
|
||||
]
|
||||
const results = await Promise.all(searchPromises)
|
||||
```
|
||||
|
||||
## Memory Efficiency
|
||||
|
||||
### Space Complexity
|
||||
|
||||
| Component | Memory Usage | Formula |
|
||||
|-----------|--------------|---------|
|
||||
| Metadata Index | ~40 bytes/entry | `(key_size + 8) × unique_values + 8 × total_items` |
|
||||
| Graph Index | ~24 bytes/edge | `16 × edges + 8 × nodes` |
|
||||
| Vector Index | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
|
||||
| Pattern Library | 394KB fixed | Pre-computed, shared across instances |
|
||||
| Type Embeddings | ~60KB fixed | 70 types × 384 dimensions × 4 bytes, cached |
|
||||
| Field Embeddings | ~5KB dynamic | Actual fields × 384 dimensions × 4 bytes |
|
||||
| Type-Field Affinity | ~2KB dynamic | Type-field occurrence counts |
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
- **Metadata Cache**: LRU with 5-minute TTL, 500 entries max
|
||||
- **Embedding Cache**: Permanent for session, prevents recomputation
|
||||
- **Unified Cache**: Coordinates memory across all components
|
||||
|
||||
## Benchmarks
|
||||
|
||||
### Illustrative Single Run (100 items, one machine)
|
||||
|
||||
Example output from a single 100-item run — illustrative only, not a committed
|
||||
benchmark; absolute numbers vary by hardware. The values feed the
|
||||
[Core Performance Summary](#core-performance-summary) example-latency column.
|
||||
|
||||
```
|
||||
Metadata exact match: 0.818ms (50 items matched)
|
||||
Metadata range query: 0.631ms (40 items in range)
|
||||
Graph neighbor lookup: 0.092ms (2 connections)
|
||||
Vector k-NN search: 1.773ms (10 nearest neighbors)
|
||||
NLP query parsing: 8.906ms (full natural language)
|
||||
Triple Intelligence: 1.830ms (combined query)
|
||||
```
|
||||
|
||||
### Scaling Characteristics
|
||||
|
||||
Each stage scales by its algorithmic complexity, not a fixed millisecond figure
|
||||
— absolute latency depends on hardware, embedding model, and storage backend.
|
||||
Only the graph adjacency index carries a committed scale assertion:
|
||||
|
||||
| Query stage | Complexity | Scaling behavior |
|
||||
|-------------|------------|------------------|
|
||||
| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
|
||||
| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
|
||||
| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
|
||||
| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
|
||||
| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
|
||||
|
||||
## Comparison with Other Systems
|
||||
|
||||
| System | Metadata Filter | Graph Traversal | Vector Search | Natural Language |
|
||||
|--------|-----------------|-----------------|---------------|------------------|
|
||||
| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) vector index | 220 patterns |
|
||||
| Neo4j | O(log n) B-tree | O(k) traversal | Not native | Not native |
|
||||
| Elasticsearch | O(log n) inverted | Not native | O(n) brute force* | Basic tokenization |
|
||||
| PostgreSQL | O(log n) B-tree | O(k) recursive | O(n) brute force* | Full-text only |
|
||||
| Pinecone | Not native | Not native | O(log n) | Not native |
|
||||
|
||||
*Without additional plugins/extensions
|
||||
|
||||
## Key Innovations
|
||||
|
||||
1. **True O(1) Metadata Filtering**: Most databases use B-trees (O(log n)). Brainy uses HashMaps for constant-time lookups.
|
||||
|
||||
2. **O(1) Graph Traversal**: Unlike traditional graph databases that traverse edges, Brainy maintains bidirectional adjacency maps for instant neighbor access.
|
||||
|
||||
3. **Unified Triple Intelligence**: First system to natively combine O(1) metadata, O(1) graph, and O(log n) vector search in a single query.
|
||||
|
||||
4. **Embedded NLP**: 220 research-based patterns with pre-computed embeddings compiled directly into the codebase - no external dependencies.
|
||||
|
||||
5. **Parallel Search Execution**: Vector, metadata, and graph searches execute simultaneously, not sequentially.
|
||||
|
||||
## Production Readiness
|
||||
|
||||
- ✅ **No External Dependencies**: All algorithms implemented in pure TypeScript
|
||||
- ✅ **No Network Calls**: Everything runs locally, including embeddings
|
||||
- ✅ **Thread-Safe**: Immutable data structures where possible
|
||||
- ✅ **Memory Bounded**: Configurable cache sizes and automatic cleanup
|
||||
- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
|
||||
- ✅ **Zero Stubs**: Every line of code is production-ready
|
||||
|
||||
## Lazy Loading Performance
|
||||
|
||||
Brainy supports two initialization modes for optimal performance across different use cases:
|
||||
|
||||
### Mode 1: Auto-Rebuild (Default)
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy()
|
||||
await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- Init time: 500ms-3s (depends on dataset size)
|
||||
- First query: Instant (indexes already loaded)
|
||||
- Use case: Traditional applications, long-running servers
|
||||
|
||||
### Mode 2: Lazy Loading
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ disableAutoRebuild: true })
|
||||
await brain.init() // Returns instantly (0-10ms)
|
||||
|
||||
const results = await brain.find({ limit: 10 }) // First query triggers rebuild (~50-200ms)
|
||||
const more = await brain.find({ limit: 100 }) // Subsequent queries instant (0ms check)
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- Init time: 0-10ms (instant)
|
||||
- First query: 50-200ms (includes index rebuild for 1K-10K entities)
|
||||
- Subsequent queries: 0ms check (instant)
|
||||
- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
|
||||
|
||||
**Concurrency Safety:**
|
||||
```javascript
|
||||
// 100 concurrent queries immediately after init
|
||||
await brain.init()
|
||||
|
||||
const promises = Array.from({ length: 100 }, () =>
|
||||
brain.find({ limit: 10 })
|
||||
)
|
||||
|
||||
const results = await Promise.all(promises)
|
||||
// ✅ Only 1 rebuild triggered (mutex)
|
||||
// ✅ All 100 queries return correct results
|
||||
// ✅ Total time: ~60ms (not 6000ms!)
|
||||
```
|
||||
|
||||
**Use Cases for Lazy Loading:**
|
||||
- **Serverless/Edge**: Minimize cold start time (0-10ms init)
|
||||
- **Development**: Faster restarts during development
|
||||
- **Large datasets**: Defer index loading until needed
|
||||
- **Read-heavy workloads**: Writes don't wait for index rebuild
|
||||
|
||||
## Zero Configuration Required
|
||||
|
||||
Brainy is designed to be **smart enough to tune itself dynamically**. No configuration needed:
|
||||
|
||||
```javascript
|
||||
// That's it. Brainy handles everything.
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Or with lazy loading for serverless
|
||||
const brain = new Brainy({ disableAutoRebuild: true })
|
||||
await brain.init() // Instant (0-10ms)
|
||||
```
|
||||
|
||||
### Automatic Self-Tuning
|
||||
|
||||
- **Metadata Index**: Auto-builds sorted indices for range queries on first use
|
||||
- **Graph Index**: Auto-flushes every 30 seconds
|
||||
- **Default Tuning**: Research-based vector index defaults
|
||||
- **Lazy Loading**: Indices built only when needed
|
||||
- **Cache Management**: LRU caches with TTL
|
||||
|
||||
### Intelligent Defaults
|
||||
|
||||
- **Vector recall** = `'balanced'` (M=16, ef=200): right for most datasets
|
||||
- **Cache TTL** = 5 min: balances freshness and performance
|
||||
- **Flush interval** = 30 s: non-blocking background persistence
|
||||
|
||||
### Vector Index Tuning Knobs
|
||||
|
||||
Brainy 8.0 exposes two knobs on `config.vector`:
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
vector: {
|
||||
recall: 'fast', // 'fast' | 'balanced' | 'accurate'
|
||||
persistMode: 'deferred' // 'immediate' | 'deferred'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cor`) can replace it with a higher-performing implementation; the public knobs stay the same.
|
||||
|
||||
### Scale Scenarios
|
||||
|
||||
| Scale | Items | Storage Strategy | Performance |
|
||||
|-------|-------|------------------|-------------|
|
||||
| **Small** | <10K | Memory | Sub-millisecond |
|
||||
| **Medium** | 10K-1M | Filesystem | 1-5ms |
|
||||
| **Large** | 1M-10M | Filesystem + tuned cache | 2-10ms |
|
||||
| **Massive** | 10M+ | Filesystem + native vector provider + service-layer sharding | 5-20ms |
|
||||
|
||||
For >10M entities, run multiple Brainy processes behind your own routing layer — Brainy 8.0 doesn't ship cluster coordination.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Application Layer │
|
||||
│ (Your Code) │
|
||||
└─────────────┬───────────────────────────┘
|
||||
│
|
||||
┌─────────────▼───────────────────────────┐
|
||||
│ Brainy Core │
|
||||
│ (Triple Intelligence Engine) │
|
||||
├─────────────────────────────────────────┤
|
||||
│ Memory │ Vector │ Metadata │
|
||||
│ Cache │ Index │ Index │
|
||||
└─────────────┬───────────────────────────┘
|
||||
│
|
||||
┌─────────────▼───────────────────────────┐
|
||||
│ Storage Layer │
|
||||
├──────────┬──────────┬──────────────────┤
|
||||
│ Vectors │ Graph │ Files │
|
||||
│ (sharded)│ Edges │ (filesystem) │
|
||||
└──────────┴──────────┴──────────────────┘
|
||||
```
|
||||
|
||||
For off-site replication, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`).
|
||||
|
||||
### Performance at Scale
|
||||
|
||||
- **Metadata queries**: O(1) HashMap
|
||||
- **Graph traversal**: O(1) adjacency lookup
|
||||
- **Vector search**: O(log n)
|
||||
- **Write throughput**: 50K+ writes/second per process (filesystem, batched)
|
||||
- **Read throughput**: 1M+ reads/second with caching
|
||||
|
||||
### Zero-Config with Autoscaling
|
||||
|
||||
- **AutoConfiguration System**: Detects environment and adjusts settings
|
||||
- **Learning from Performance**: `learnFromPerformance()` adapts based on metrics
|
||||
- **Auto-flush**: Graph index (30s), Metadata index (configurable)
|
||||
- **Auto-optimize**: Enabled by default in graph and vector indices
|
||||
- **Zero-config presets**: Production, development, minimal modes
|
||||
- **Adaptive memory**: Scales caches based on available memory
|
||||
|
||||
## Implementation Status
|
||||
|
||||
### Fully Implemented and Production-Ready
|
||||
- **O(1) metadata lookups** via HashMaps (exact match)
|
||||
- **O(log n) range queries** via sorted arrays with lazy building
|
||||
- **O(1) graph traversal** via adjacency maps
|
||||
- **O(log n) vector search** via the default JS index, swappable for a native provider
|
||||
- **220 NLP patterns** with pre-computed embeddings
|
||||
- **Filesystem and memory storage** adapters
|
||||
- **Auto-configuration system** with environment detection
|
||||
- **Zero-config operation** with intelligent defaults
|
||||
- **Auto-flush and auto-optimize** in indices
|
||||
- **Low-latency Triple Intelligence queries** (O(log n) vector + O(1) metadata/graph)
|
||||
|
||||
## Conclusion
|
||||
|
||||
Brainy delivers on its promise of **production-ready Triple Intelligence** with documented algorithmic-complexity guarantees and a committed graph-scale benchmark (`tests/performance/graph-scale-performance.test.ts`). All listed features are fully implemented and tested. No stubs, no mocks — just real, working code with characterized performance.
|
||||
|
|
@ -1,101 +0,0 @@
|
|||
# Performance and Logging Fixes for Brainy
|
||||
|
||||
## Overview
|
||||
|
||||
This document outlines the fixes implemented to address pagination warnings and improve performance in the Brainy framework.
|
||||
|
||||
## Issues Addressed
|
||||
|
||||
### 1. Pagination Warnings
|
||||
|
||||
The following warnings were appearing when using Brainy in dependent projects:
|
||||
|
||||
```
|
||||
WARNING: Only returning the first 1000 nodes. There are more nodes available. Use getNodesWithPagination() for proper pagination.
|
||||
Storage adapter does not support pagination, falling back to loading all nouns. This may cause performance issues with large datasets.
|
||||
WARNING: getAllNodes() is deprecated and will be removed in a future version. Use getNodesWithPagination() instead.
|
||||
```
|
||||
|
||||
### 2. Root Causes
|
||||
|
||||
1. **Missing Pagination Support**: The S3 storage adapter didn't expose a `getNounsWithPagination()` method, causing the base storage to fall back to `getAllNouns_internal()`.
|
||||
2. **Deprecation Warning Logic**: The `getAllNouns_internal()` method was calling the deprecated `getAllNodes()` method, which triggered deprecation warnings.
|
||||
3. **Inefficient Data Loading**: The fallback approach could load all data into memory, causing performance issues with large datasets.
|
||||
|
||||
## Solutions Implemented
|
||||
|
||||
### 1. Updated S3 Storage Adapter
|
||||
|
||||
**File**: `src/storage/adapters/s3CompatibleStorage.ts`
|
||||
|
||||
- Modified `getAllNouns_internal()` to use the paginated `getNodesWithPagination()` method instead of the deprecated `getAllNodes()`.
|
||||
- Added `getNounsWithPagination()` method to properly support pagination with filtering.
|
||||
|
||||
```typescript
|
||||
public async getNounsWithPagination(options: {
|
||||
limit?: number
|
||||
cursor?: string
|
||||
filter?: {
|
||||
nounType?: string | string[]
|
||||
service?: string | string[]
|
||||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
}>
|
||||
```
|
||||
|
||||
### 2. Created Optimized S3 Search Module
|
||||
|
||||
**File**: `src/storage/adapters/optimizedS3Search.ts`
|
||||
|
||||
A new module that provides:
|
||||
- Efficient pagination using S3's ListObjectsV2 command
|
||||
- Parallel batch loading for better performance
|
||||
- Support for complex filtering without loading all data
|
||||
- Proper cursor-based pagination for large datasets
|
||||
|
||||
### 3. Improved Base Storage Logic
|
||||
|
||||
**File**: `src/storage/baseStorage.ts`
|
||||
|
||||
- The base storage now properly detects when adapters support pagination
|
||||
- Falls back gracefully when pagination isn't supported
|
||||
- Limits the amount of data loaded to prevent memory issues
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Eliminated Warnings**: No more deprecation warnings in dependent projects.
|
||||
2. **Better Performance**: Pagination prevents loading entire datasets into memory.
|
||||
3. **Improved Scalability**: Can handle large datasets efficiently.
|
||||
4. **Backward Compatibility**: Existing code continues to work without changes.
|
||||
|
||||
## Usage in Dependent Projects
|
||||
|
||||
Your code in github-package is already using the correct approach:
|
||||
|
||||
```typescript
|
||||
const response = await brainy.getNouns({
|
||||
pagination: { limit: 10, offset: 0 },
|
||||
filter: { nounType: 'Template' }
|
||||
})
|
||||
```
|
||||
|
||||
With these fixes, this will now:
|
||||
- Use proper pagination without warnings
|
||||
- Load only the requested data
|
||||
- Support efficient filtering
|
||||
|
||||
## Future Improvements
|
||||
|
||||
1. **Native S3 Filtering**: Implement server-side filtering using S3 object tags or metadata.
|
||||
2. **Index-based Search**: Create indexes for common query patterns.
|
||||
3. **Caching Layer**: Add intelligent caching to reduce S3 API calls.
|
||||
4. **Streaming Support**: For very large datasets, implement streaming APIs.
|
||||
|
||||
## Migration Notes
|
||||
|
||||
No changes are required in dependent projects. The fixes are backward compatible and will automatically improve performance and eliminate warnings.
|
||||
|
|
@ -1,550 +0,0 @@
|
|||
# Performance Features Guide
|
||||
|
||||
This guide covers the advanced performance features added in Brainy v0.38.1+ that dramatically improve search speed and pagination capabilities, including intelligent auto-configuration.
|
||||
|
||||
## 🤖 Intelligent Auto-Configuration (NEW!)
|
||||
|
||||
**Brainy now configures itself automatically!** The new auto-configuration system detects your environment and usage patterns, then optimally configures caching and real-time updates with zero manual setup required.
|
||||
|
||||
### How It Works
|
||||
|
||||
- **🔍 Environment Detection**: Automatically detects browser, Node.js, serverless, or distributed scenarios
|
||||
- **📊 Usage Pattern Analysis**: Learns from your read/write patterns and data change frequency
|
||||
- **⚡ Real-Time Adaptation**: Continuously monitors performance and adjusts settings automatically
|
||||
- **🌐 Distributed Mode Awareness**: Detects shared storage scenarios and enables real-time updates
|
||||
- **🎯 Zero Configuration**: Works perfectly out of the box, but respects explicit settings
|
||||
|
||||
### Automatic Optimizations
|
||||
|
||||
```javascript
|
||||
// Just create a Brainy instance - everything is optimized automatically!
|
||||
const brainy = new BrainyData()
|
||||
await brainy.init()
|
||||
|
||||
// Auto-configuration analyzes your environment and optimizes:
|
||||
// ✅ Cache size based on available memory
|
||||
// ✅ Cache TTL based on data change frequency
|
||||
// ✅ Real-time updates for distributed storage
|
||||
// ✅ Read vs write workload optimization
|
||||
// ✅ Memory constraint handling
|
||||
```
|
||||
|
||||
### Configuration Explanations
|
||||
|
||||
Enable verbose logging to see what optimizations are being applied:
|
||||
|
||||
```javascript
|
||||
const brainy = new BrainyData({
|
||||
logging: { verbose: true }
|
||||
})
|
||||
await brainy.init()
|
||||
|
||||
// Console output:
|
||||
// 🤖 Brainy Auto-Configuration:
|
||||
//
|
||||
// 📊 Cache: 100 queries, 300s TTL
|
||||
// 🔄 Updates: Every 30s
|
||||
//
|
||||
// 🎯 Optimizations applied:
|
||||
// • Read-heavy workload detected - increased cache size and TTL
|
||||
// • Distributed storage detected - enabled real-time updates
|
||||
// • Reduced cache TTL for distributed consistency
|
||||
```
|
||||
|
||||
### Manual Override (Optional)
|
||||
|
||||
Auto-configuration respects your explicit settings when provided:
|
||||
|
||||
```javascript
|
||||
const brainy = new BrainyData({
|
||||
// Explicit settings override auto-configuration
|
||||
searchCache: {
|
||||
maxSize: 500,
|
||||
maxAge: 600000
|
||||
},
|
||||
realtimeUpdates: {
|
||||
enabled: true,
|
||||
interval: 15000
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Performance Scenarios
|
||||
|
||||
**🏠 Local Development**
|
||||
- Large cache with long TTL for best performance
|
||||
- Real-time updates disabled (not needed for single instance)
|
||||
|
||||
**🌐 Distributed Production**
|
||||
- Shorter cache TTL for data consistency
|
||||
- Real-time updates enabled automatically
|
||||
- Adaptive intervals based on change frequency
|
||||
|
||||
**💾 Memory-Constrained Environments**
|
||||
- Smaller cache sizes with intelligent eviction
|
||||
- Optimized for essential caching only
|
||||
|
||||
**📈 High-Traffic Applications**
|
||||
- Larger caches with hit-count weighted eviction
|
||||
- Performance monitoring and auto-adaptation
|
||||
|
||||
## 🚀 Smart Search Caching
|
||||
|
||||
Brainy includes intelligent result caching that can make repeated queries **100x faster** with zero code changes required.
|
||||
|
||||
### How It Works
|
||||
|
||||
- **Automatic**: Caching is enabled by default and works transparently
|
||||
- **Intelligent**: Only caches successful search results
|
||||
- **Self-Maintaining**: Automatically invalidates when data changes
|
||||
- **Memory-Conscious**: LRU eviction prevents memory bloat
|
||||
|
||||
### Performance Impact
|
||||
|
||||
```javascript
|
||||
// First search: ~50ms (database query)
|
||||
const results1 = await brainy.search('machine learning', 10)
|
||||
|
||||
// Second identical search: <1ms (cache hit!)
|
||||
const results2 = await brainy.search('machine learning', 10)
|
||||
```
|
||||
|
||||
### Configuration Options
|
||||
|
||||
```javascript
|
||||
const brainy = new BrainyData({
|
||||
searchCache: {
|
||||
enabled: true, // Enable/disable caching (default: true)
|
||||
maxSize: 100, // Max number of cached queries (default: 100)
|
||||
maxAge: 300000, // Cache TTL in milliseconds (default: 5 minutes)
|
||||
hitCountWeight: 0.3 // Weight for hit count in eviction (default: 0.3)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Cache Monitoring
|
||||
|
||||
```javascript
|
||||
// Get performance statistics
|
||||
const stats = brainy.getCacheStats()
|
||||
console.log(`Hit rate: ${(stats.search.hitRate * 100).toFixed(1)}%`)
|
||||
console.log(`Cache size: ${stats.search.size}/${stats.search.maxSize}`)
|
||||
console.log(`Memory usage: ${(stats.searchMemoryUsage / 1024).toFixed(1)}KB`)
|
||||
|
||||
// Manual cache management
|
||||
brainy.clearCache() // Clear all cached results
|
||||
```
|
||||
|
||||
### Real-Time Data Compatibility
|
||||
|
||||
**The cache automatically maintains data consistency:**
|
||||
|
||||
#### 🏠 **Single Instance (Local Changes)**
|
||||
- ✅ Adding new data invalidates all caches
|
||||
- ✅ Updating metadata invalidates related caches
|
||||
- ✅ Deleting data invalidates all caches
|
||||
- ✅ Clearing database invalidates all caches
|
||||
|
||||
#### 🌐 **Distributed Mode (Shared S3 Storage)**
|
||||
- ✅ **Real-time updates detect external changes** automatically
|
||||
- ✅ **Cache invalidation on external data changes**
|
||||
- ✅ **Time-based cache expiration** as safety net
|
||||
- ✅ **Periodic cache cleanup** removes stale entries
|
||||
|
||||
```javascript
|
||||
// Enable real-time updates for distributed scenarios
|
||||
const brainy = new BrainyData({
|
||||
storage: {
|
||||
s3Storage: { bucketName: 'shared-bucket' }
|
||||
},
|
||||
searchCache: {
|
||||
maxAge: 300000 // 5 minutes - shorter in distributed mode
|
||||
},
|
||||
realtimeUpdates: {
|
||||
enabled: true, // Essential for shared storage
|
||||
interval: 30000, // Check every 30 seconds
|
||||
updateIndex: true, // Update local index with external changes
|
||||
updateStatistics: true
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
This ensures you get fresh results even when other services add data to shared storage, while still benefiting from caching performance.
|
||||
|
||||
### Cache Invalidation Strategies
|
||||
|
||||
```javascript
|
||||
// Disable caching for specific queries
|
||||
const freshResults = await brainy.search('query', 10, { skipCache: true })
|
||||
|
||||
// The cache uses smart invalidation patterns:
|
||||
await brainy.add(newData) // Invalidates all search caches
|
||||
await brainy.updateMetadata(id) // Invalidates all search caches
|
||||
await brainy.delete(id) // Invalidates all search caches
|
||||
```
|
||||
|
||||
## 📄 Cursor-Based Pagination
|
||||
|
||||
For large result sets, cursor-based pagination provides constant-time performance regardless of page depth.
|
||||
|
||||
### Traditional Offset Problems
|
||||
|
||||
```javascript
|
||||
// Traditional offset pagination gets slower with depth
|
||||
const page1 = await brainy.search('query', 10, { offset: 0 }) // Fast
|
||||
const page2 = await brainy.search('query', 10, { offset: 10 }) // Still fast
|
||||
const page100 = await brainy.search('query', 10, { offset: 1000 }) // Slow!
|
||||
```
|
||||
|
||||
### Cursor Solution
|
||||
|
||||
```javascript
|
||||
// Cursor pagination is constant time for any depth
|
||||
const page1 = await brainy.searchWithCursor('query', 10)
|
||||
console.log(`Found ${page1.results.length} results`)
|
||||
console.log(`Has more: ${page1.hasMore}`)
|
||||
|
||||
if (page1.hasMore) {
|
||||
// Next page is just as fast as the first
|
||||
const page2 = await brainy.searchWithCursor('query', 10, {
|
||||
cursor: page1.cursor
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
### Advanced Pagination Features
|
||||
|
||||
```javascript
|
||||
// Get total count estimate when available
|
||||
const page = await brainy.searchWithCursor('query', 50)
|
||||
if (page.totalEstimate) {
|
||||
console.log(`Showing 50 of ~${page.totalEstimate} results`)
|
||||
}
|
||||
|
||||
// Cursor contains debug information
|
||||
if (page.cursor) {
|
||||
console.log(`Cursor position: ${page.cursor.position}`)
|
||||
console.log(`Last result ID: ${page.cursor.lastId}`)
|
||||
console.log(`Last score: ${page.cursor.lastScore}`)
|
||||
}
|
||||
```
|
||||
|
||||
### Pagination Best Practices
|
||||
|
||||
1. **Use cursors for deep pagination** (page 10+)
|
||||
2. **Use offset for UI with page numbers** (page 1-10)
|
||||
3. **Cache cursor objects** for navigation
|
||||
4. **Handle cursor expiration** gracefully
|
||||
|
||||
```javascript
|
||||
// Robust cursor pagination with error handling
|
||||
async function paginateResults(query, pageSize = 20) {
|
||||
const results = []
|
||||
let cursor = undefined
|
||||
let pageCount = 0
|
||||
|
||||
do {
|
||||
try {
|
||||
const page = await brainy.searchWithCursor(query, pageSize, { cursor })
|
||||
|
||||
results.push(...page.results)
|
||||
cursor = page.cursor
|
||||
pageCount++
|
||||
|
||||
// Safety limit
|
||||
if (pageCount > 100) break
|
||||
|
||||
} catch (error) {
|
||||
console.warn('Cursor may be expired, starting fresh')
|
||||
cursor = undefined
|
||||
break
|
||||
}
|
||||
} while (cursor)
|
||||
|
||||
return results
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 Performance Tuning
|
||||
|
||||
### Cache Sizing
|
||||
|
||||
```javascript
|
||||
// For high-traffic applications
|
||||
const brainy = new BrainyData({
|
||||
searchCache: {
|
||||
maxSize: 500, // Cache more queries
|
||||
maxAge: 600000, // Keep cache longer (10 minutes)
|
||||
}
|
||||
})
|
||||
|
||||
// For memory-constrained environments
|
||||
const brainy = new BrainyData({
|
||||
searchCache: {
|
||||
maxSize: 50, // Smaller cache
|
||||
maxAge: 120000, // Shorter TTL (2 minutes)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Cache Warming
|
||||
|
||||
```javascript
|
||||
// Pre-warm cache with common queries
|
||||
const commonQueries = [
|
||||
'machine learning',
|
||||
'artificial intelligence',
|
||||
'data science',
|
||||
'neural networks'
|
||||
]
|
||||
|
||||
// Warm up cache in background
|
||||
for (const query of commonQueries) {
|
||||
brainy.search(query, 10).catch(console.warn)
|
||||
}
|
||||
```
|
||||
|
||||
### Performance Monitoring
|
||||
|
||||
```javascript
|
||||
// Set up performance monitoring
|
||||
setInterval(() => {
|
||||
const stats = brainy.getCacheStats()
|
||||
|
||||
if (stats.search.hitRate < 0.3) {
|
||||
console.warn('Low cache hit rate:', stats.search.hitRate)
|
||||
}
|
||||
|
||||
if (stats.searchMemoryUsage > 50 * 1024 * 1024) { // 50MB
|
||||
console.warn('High cache memory usage')
|
||||
brainy.clearCache() // Reset if getting too large
|
||||
}
|
||||
}, 60000) // Check every minute
|
||||
```
|
||||
|
||||
## 🎯 Migration Guide
|
||||
|
||||
### From Offset to Cursors
|
||||
|
||||
```javascript
|
||||
// Before (offset-based)
|
||||
async function getAllResults(query) {
|
||||
const results = []
|
||||
let offset = 0
|
||||
const pageSize = 100
|
||||
|
||||
while (true) {
|
||||
const page = await brainy.search(query, pageSize, { offset })
|
||||
if (page.length === 0) break
|
||||
|
||||
results.push(...page)
|
||||
offset += pageSize // Gets slower each iteration
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
// After (cursor-based)
|
||||
async function getAllResults(query) {
|
||||
const results = []
|
||||
let cursor = undefined
|
||||
const pageSize = 100
|
||||
|
||||
while (true) {
|
||||
const page = await brainy.searchWithCursor(query, pageSize, { cursor })
|
||||
if (page.results.length === 0) break
|
||||
|
||||
results.push(...page.results)
|
||||
cursor = page.cursor // Constant time each iteration
|
||||
|
||||
if (!page.hasMore) break
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
```
|
||||
|
||||
### Backward Compatibility
|
||||
|
||||
**All existing code continues to work unchanged:**
|
||||
|
||||
```javascript
|
||||
// These still work exactly as before
|
||||
const results = await brainy.search('query', 10)
|
||||
const page2 = await brainy.search('query', 10, { offset: 10 })
|
||||
|
||||
// But now they benefit from caching automatically!
|
||||
```
|
||||
|
||||
## 📊 Performance Benchmarks
|
||||
|
||||
### Cache Performance
|
||||
|
||||
| Scenario | Without Cache | With Cache | Improvement |
|
||||
|----------|---------------|------------|-------------|
|
||||
| Repeated identical queries | 50ms | <1ms | **50x faster** |
|
||||
| Similar queries with filters | 45ms | <1ms | **45x faster** |
|
||||
| Paginated results (cached) | 30ms | <1ms | **30x faster** |
|
||||
|
||||
### Pagination Performance
|
||||
|
||||
| Page Depth | Offset-Based | Cursor-Based | Improvement |
|
||||
|------------|--------------|--------------|-------------|
|
||||
| Page 1-10 | 10-50ms | 10-50ms | Same |
|
||||
| Page 50 | 200ms | 50ms | **4x faster** |
|
||||
| Page 100 | 500ms | 50ms | **10x faster** |
|
||||
| Page 1000 | 5000ms | 50ms | **100x faster** |
|
||||
|
||||
These improvements compound in real applications where users frequently:
|
||||
- Repeat searches
|
||||
- Navigate deep into result sets
|
||||
- Use similar search terms
|
||||
- Browse paginated data
|
||||
|
||||
The performance gains are most dramatic in read-heavy applications with repeated access patterns.
|
||||
|
||||
## 🌐 Distributed Storage Considerations
|
||||
|
||||
When multiple services write to shared storage (like S3), special considerations apply to maintain cache consistency.
|
||||
|
||||
### Problem: External Data Changes
|
||||
|
||||
```javascript
|
||||
// Service A writes data
|
||||
await serviceA.add("New important data")
|
||||
|
||||
// Service B searches (may get stale cached results!)
|
||||
const results = await serviceB.search("important data", 10)
|
||||
// Without proper configuration, Service B might miss the new data
|
||||
```
|
||||
|
||||
### Solution: Real-Time Updates + Smart Caching
|
||||
|
||||
```javascript
|
||||
// Configure both services for distributed mode
|
||||
const sharedConfig = {
|
||||
storage: {
|
||||
s3Storage: {
|
||||
bucketName: 'shared-data-bucket',
|
||||
// ... S3 credentials
|
||||
}
|
||||
},
|
||||
searchCache: {
|
||||
enabled: true,
|
||||
maxAge: 180000, // 3 minutes (shorter for distributed)
|
||||
maxSize: 100
|
||||
},
|
||||
realtimeUpdates: {
|
||||
enabled: true, // ⚠️ ESSENTIAL for shared storage
|
||||
interval: 30000, // Check every 30 seconds
|
||||
updateIndex: true, // Sync external changes to local index
|
||||
updateStatistics: true
|
||||
}
|
||||
}
|
||||
|
||||
const serviceA = new BrainyData(sharedConfig)
|
||||
const serviceB = new BrainyData(sharedConfig)
|
||||
```
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Service A** adds data to shared S3 storage
|
||||
2. **Service B** checks for changes every 30 seconds
|
||||
3. **External changes detected** → cache invalidated → fresh results guaranteed
|
||||
4. **Time-based expiration** provides additional safety net
|
||||
|
||||
### Performance Impact in Distributed Mode
|
||||
|
||||
| Scenario | Local Instance | Distributed Mode | Notes |
|
||||
|----------|----------------|------------------|-------|
|
||||
| Cache hits (no external changes) | <1ms | <1ms | Same performance |
|
||||
| External changes detected | 50ms | 50ms + detection delay | Still very fast |
|
||||
| Cache expiration | 50ms | 50ms | Automatic cleanup |
|
||||
| Real-time update overhead | 0ms | ~5ms per check | Minimal impact |
|
||||
|
||||
### Best Practices for Distributed Caching
|
||||
|
||||
#### ✅ **Do This:**
|
||||
```javascript
|
||||
// Shorter cache TTL for distributed scenarios
|
||||
searchCache: { maxAge: 180000 } // 3 minutes vs 5 minutes
|
||||
|
||||
// Enable real-time updates
|
||||
realtimeUpdates: { enabled: true, interval: 30000 }
|
||||
|
||||
// Monitor cache stats
|
||||
setInterval(() => {
|
||||
const stats = brainy.getCacheStats()
|
||||
console.log(`Cache hit rate: ${stats.search.hitRate}`)
|
||||
}, 60000)
|
||||
```
|
||||
|
||||
#### ❌ **Avoid This:**
|
||||
```javascript
|
||||
// DON'T: Disable real-time updates with shared storage
|
||||
realtimeUpdates: { enabled: false } // ⚠️ Will cause stale data!
|
||||
|
||||
// DON'T: Very long cache TTL in distributed mode
|
||||
searchCache: { maxAge: 3600000 } // 1 hour - too long!
|
||||
|
||||
// DON'T: Ignore external changes
|
||||
const results = await brainy.search('query', 10, { skipCache: false })
|
||||
// Without real-time updates, this might be stale
|
||||
```
|
||||
|
||||
### Monitoring Distributed Cache Health
|
||||
|
||||
```javascript
|
||||
// Set up monitoring for distributed scenarios
|
||||
const brainy = new BrainyData({
|
||||
// ... distributed config
|
||||
logging: { verbose: true } // See cache invalidation logs
|
||||
})
|
||||
|
||||
// Monitor cache effectiveness
|
||||
setInterval(() => {
|
||||
const stats = brainy.getCacheStats()
|
||||
|
||||
// Warn if hit rate is too low (suggests frequent external changes)
|
||||
if (stats.search.hitRate < 0.2) {
|
||||
console.warn('Low cache hit rate in distributed mode:', stats.search.hitRate)
|
||||
console.warn('Consider increasing real-time update frequency')
|
||||
}
|
||||
|
||||
// Warn if external changes are frequent
|
||||
const updateConfig = brainy.getRealtimeUpdateConfig()
|
||||
if (updateConfig.enabled) {
|
||||
console.log('Real-time updates active - external changes will be detected')
|
||||
}
|
||||
}, 300000) // Check every 5 minutes
|
||||
```
|
||||
|
||||
### Trade-offs and Tuning
|
||||
|
||||
**More Frequent Updates (every 10-15 seconds):**
|
||||
- ✅ Faster detection of external changes
|
||||
- ✅ More consistent data across services
|
||||
- ❌ Higher CPU/network overhead
|
||||
- ❌ More frequent cache invalidations
|
||||
|
||||
**Less Frequent Updates (every 60+ seconds):**
|
||||
- ✅ Lower overhead
|
||||
- ✅ Better cache hit rates
|
||||
- ❌ Slower detection of external changes
|
||||
- ❌ Potential stale data windows
|
||||
|
||||
**Recommended Settings by Use Case:**
|
||||
|
||||
```javascript
|
||||
// High-consistency requirements (financial, medical)
|
||||
realtimeUpdates: { interval: 15000 } // 15 seconds
|
||||
searchCache: { maxAge: 120000 } // 2 minutes
|
||||
|
||||
// Balanced (most applications)
|
||||
realtimeUpdates: { interval: 30000 } // 30 seconds
|
||||
searchCache: { maxAge: 300000 } // 5 minutes
|
||||
|
||||
// Performance-first (analytics, logging)
|
||||
realtimeUpdates: { interval: 60000 } // 1 minute
|
||||
searchCache: { maxAge: 600000 } // 10 minutes
|
||||
```
|
||||
486
docs/PLUGINS.md
Normal file
486
docs/PLUGINS.md
Normal file
|
|
@ -0,0 +1,486 @@
|
|||
---
|
||||
title: Plugin System
|
||||
slug: guides/plugins
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 4
|
||||
description: Replace any Brainy subsystem — distance functions, embeddings, vector index, metadata index, aggregation — with a custom implementation or optional native acceleration.
|
||||
next:
|
||||
- guides/storage-adapters
|
||||
---
|
||||
|
||||
# Plugin Development Guide
|
||||
|
||||
Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
Brainy's plugin system uses **named providers** — string keys mapped to implementations. During `init()`, brainy:
|
||||
|
||||
1. Imports each package listed in the `plugins` config array
|
||||
2. Activates each plugin, passing a `BrainyPluginContext`
|
||||
3. The plugin calls `context.registerProvider(key, implementation)` for each subsystem it provides
|
||||
4. Brainy checks each provider key and wires the implementation into its internal pipeline
|
||||
|
||||
Installing the first-party accelerator is the opt-in: with the default config, brainy probes for `@soulcraft/cor` and loads it when present. Everything except "not installed" fails **loud** — a present-but-broken accelerator makes `init()` throw rather than silently degrading to the JS engines.
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy() // @soulcraft/cor auto-detected when installed
|
||||
const pinned = new Brainy({ plugins: ['@soulcraft/cor'] }) // or pin exactly what loads
|
||||
const plain = new Brainy({ plugins: [] }) // or opt out of detection entirely
|
||||
```
|
||||
|
||||
| `plugins` value | Behavior |
|
||||
|---|---|
|
||||
| `undefined` (default) | Guarded auto-detection of `@soulcraft/cor`: not installed → no plugins, silently; installed → loads + announces; installed-but-broken → `init()` throws |
|
||||
| `false` / `[]` | No plugins, no detection (explicit opt-out) |
|
||||
| `['@soulcraft/cor']` | Load only the listed packages; a listed plugin that fails to load throws |
|
||||
|
||||
Plugins registered programmatically via `brain.use(plugin)` are always activated regardless of the `plugins` config.
|
||||
|
||||
If no plugin provides a given key, brainy uses its built-in JavaScript implementation. This means brainy works perfectly standalone — plugins only enhance performance or add capabilities.
|
||||
|
||||
## Creating a Plugin
|
||||
|
||||
### 1. Implement the `BrainyPlugin` interface
|
||||
|
||||
```typescript
|
||||
import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
|
||||
|
||||
const myPlugin: BrainyPlugin = {
|
||||
name: 'my-brainy-plugin', // Must be unique (typically your npm package name)
|
||||
|
||||
async activate(context: BrainyPluginContext): Promise<boolean> {
|
||||
// Register your providers here
|
||||
context.registerProvider('distance', myFastDistanceFunction)
|
||||
|
||||
// Return true if activation succeeded, false to skip
|
||||
return true
|
||||
},
|
||||
|
||||
async deactivate(): Promise<void> {
|
||||
// Optional cleanup when brainy.close() is called
|
||||
}
|
||||
}
|
||||
|
||||
export default myPlugin
|
||||
```
|
||||
|
||||
### 2. Package exports
|
||||
|
||||
Your package must export the plugin as the default export so brainy's plugin loader can resolve it:
|
||||
|
||||
```typescript
|
||||
// index.ts
|
||||
export { default } from './plugin.js'
|
||||
```
|
||||
|
||||
### 3. Registration
|
||||
|
||||
**Config-based:** List your package name in the brainy config:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
plugins: ['my-brainy-plugin']
|
||||
})
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
**Programmatic registration:** For plugins not installed as npm packages, use `brain.use()`:
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import myPlugin from './my-plugin.js'
|
||||
|
||||
const brain = new Brainy()
|
||||
brain.use(myPlugin)
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
## Provider Keys Reference
|
||||
|
||||
Each key has a specific expected signature. Brainy checks for these during `init()` and wires them into the appropriate code paths.
|
||||
|
||||
### Core Providers
|
||||
|
||||
#### `distance`
|
||||
**Type:** `(a: number[], b: number[]) => number`
|
||||
|
||||
Replaces the default cosine distance function used in vector search and neural APIs. This is the highest-impact single provider — it's called for every vector comparison.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('distance', (a: number[], b: number[]): number => {
|
||||
// Your SIMD-accelerated or GPU distance calculation
|
||||
return myFastCosineDistance(a, b)
|
||||
})
|
||||
```
|
||||
|
||||
#### `embeddings`
|
||||
**Type:** `(text: string | string[]) => Promise<number[] | number[][]>`
|
||||
|
||||
Replaces the built-in WASM embedding engine. Called for every `brain.add()`, `brain.update()`, and `brain.find()` operation that involves text.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('embeddings', async (text: string | string[]) => {
|
||||
if (Array.isArray(text)) {
|
||||
return myEngine.embedBatch(text)
|
||||
}
|
||||
return myEngine.embed(text)
|
||||
})
|
||||
```
|
||||
|
||||
#### `embedBatch`
|
||||
**Type:** `(texts: string[]) => Promise<number[][]>`
|
||||
|
||||
Dedicated batch embedding provider. When registered, brainy uses this for bulk operations (import, reindex, batch add) instead of calling the `embeddings` provider N times. This enables true single-forward-pass batch processing.
|
||||
|
||||
Priority order for batch operations:
|
||||
1. `embedBatch` provider (single forward pass — fastest)
|
||||
2. `embeddings` provider with `Promise.all()` (N individual calls)
|
||||
3. Built-in WASM batch API (fallback)
|
||||
|
||||
```typescript
|
||||
context.registerProvider('embedBatch', async (texts: string[]) => {
|
||||
// Process all texts in a single forward pass
|
||||
return myEngine.batchEmbed(texts)
|
||||
})
|
||||
```
|
||||
|
||||
### Index Providers
|
||||
|
||||
> **Write-path invariant (the change-feed contract).** Every canonical
|
||||
> mutation flows through Brainy's generation-store commit points — index
|
||||
> providers are invoked *inside* that commit and never originate canonical
|
||||
> writes of their own. The `brain.onChange` change feed is emitted from those
|
||||
> commit points and relies on this: **a plugin must never introduce a write
|
||||
> path that bypasses the generation-store commit.** If a future provider ever
|
||||
> needs a direct native ingest path, it must either route through the commit
|
||||
> or emit equivalent change events — otherwise every `onChange` consumer
|
||||
> (live UIs, cache invalidation, realtime sync) silently develops a blind
|
||||
> spot.
|
||||
|
||||
#### `vector`
|
||||
**Type:** `(config: object, distanceFunction: Function, options: object) => VectorIndexProvider-compatible`
|
||||
|
||||
Factory function that creates a vector index instance. The returned object must implement the `VectorIndexProvider` public API:
|
||||
|
||||
- `addItem(item: { id: string, vector: number[] }): Promise<string>`
|
||||
- `search(queryVector: number[], k: number, filter?, options?): Promise<Array<[string, number]>>`
|
||||
- `removeItem(id: string): Promise<boolean>`
|
||||
- `size(): number`
|
||||
- `clear(): void`
|
||||
- `flush(): Promise<number>`
|
||||
- `rebuild(options?): Promise<void>`
|
||||
- `getDirtyNodeCount(): number`
|
||||
- `getPersistMode(): 'immediate' | 'deferred'`
|
||||
- `getEntryPointId(): string | null`
|
||||
- `getMaxLevel(): number`
|
||||
- `getDimension(): number | null`
|
||||
- `getConfig(): object`
|
||||
- `getDistanceFunction(): Function`
|
||||
- `enableCOW(parent): void`
|
||||
- `setUseParallelization(boolean): void`
|
||||
|
||||
For type-aware indexes (separate graph per noun type), also implement:
|
||||
- `getIndexForType(type: string): VectorIndexProvider` (duck-typed detection)
|
||||
- `search(queryVector, k, type?, filter?, options?): Promise<Array<[string, number]>>`
|
||||
|
||||
```typescript
|
||||
context.registerProvider('vector', (config, distanceFn, options) => {
|
||||
return new MyNativeVectorIndex(config, distanceFn, options)
|
||||
})
|
||||
```
|
||||
|
||||
#### The readiness contract (all three index providers)
|
||||
|
||||
A provider that **persists its derived index** should implement the optional readiness
|
||||
members so a warm reopen never pays a redundant rebuild-from-canonical:
|
||||
|
||||
- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
|
||||
`brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
|
||||
and **before the rebuild gate**.
|
||||
- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
|
||||
loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
|
||||
exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
|
||||
`totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
|
||||
while fully durable. Never return `true` if the durable state failed to load: the
|
||||
signal is honest in both directions, and a not-ready provider gets its rebuild even
|
||||
when `size() > 0`.
|
||||
- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
|
||||
migration); brainy skips its rebuild entirely.
|
||||
|
||||
Providers that implement none of these keep the size/count heuristics — correct for
|
||||
engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
|
||||
|
||||
#### `metadataIndex`
|
||||
**Type:** `(storage: StorageAdapter) => MetadataIndexManager-compatible`
|
||||
|
||||
Factory function that creates a metadata index. The returned object must implement the `MetadataIndexManager` interface including `init()`, `addEntity()`, `removeEntity()`, `query()`, `flush()`, `clear()`, etc.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('metadataIndex', (storage) => {
|
||||
return new MyNativeMetadataIndex(storage)
|
||||
})
|
||||
```
|
||||
|
||||
#### `graphIndex`
|
||||
**Type:** `(storage: StorageAdapter) => GraphAdjacencyIndex-compatible`
|
||||
|
||||
Factory function that creates a graph adjacency index for relationship tracking (verbs/triples). Must implement the `GraphAdjacencyIndex` interface including `addVerb()`, `getVerbsBySource()`, `getVerbsByTarget()`, `flush()`, etc.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('graphIndex', (storage) => {
|
||||
return new MyNativeGraphIndex(storage)
|
||||
})
|
||||
```
|
||||
|
||||
#### `aggregation`
|
||||
**Type:** `(storage: StorageAdapter) => AggregationProvider-compatible`
|
||||
|
||||
Factory function that creates an aggregation engine for write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows. The returned object must implement the `AggregationProvider` interface.
|
||||
|
||||
```typescript
|
||||
context.registerProvider('aggregation', (storage) => {
|
||||
return new MyNativeAggregationEngine(storage)
|
||||
})
|
||||
```
|
||||
|
||||
When provided by an optional native acceleration plugin (such as `@soulcraft/cor`), this enables:
|
||||
- Compiled source filters (vs per-entity JS object traversal)
|
||||
- Precise MIN/MAX via sorted data structures (vs lazy recompute)
|
||||
- Parallel aggregate rebuild across CPU cores
|
||||
- SIMD-accelerated timestamp bucketing
|
||||
|
||||
### Utility Providers
|
||||
|
||||
#### `cache`
|
||||
**Type:** `UnifiedCache`
|
||||
|
||||
Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`).
|
||||
|
||||
```typescript
|
||||
import type { UnifiedCache } from '@soulcraft/brainy/internals'
|
||||
|
||||
context.registerProvider('cache', myNativeCache)
|
||||
```
|
||||
|
||||
#### `entityIdMapper`
|
||||
**Type:** `(storage: StorageAdapter) => EntityIdMapper-compatible`
|
||||
|
||||
Factory for bidirectional UUID ↔ integer mapping used by roaring bitmaps. Must implement `getOrAssign()`, `getUuid()`, `getInt()`, `has()`, `remove()`, `flush()`, `clear()`.
|
||||
|
||||
#### `roaring`
|
||||
**Type:** `RoaringBitmap32 class`
|
||||
|
||||
Replacement for the roaring bitmap implementation. Used internally by the metadata index for set operations. Must be API-compatible with `roaring-wasm`.
|
||||
|
||||
#### `msgpack`
|
||||
**Type:** `{ encode: (data: any) => Buffer, decode: (buffer: Buffer) => any }`
|
||||
|
||||
Native msgpack encode/decode for SSTable serialization.
|
||||
|
||||
### Analytics Providers (Native-Only)
|
||||
|
||||
These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cor`) is installed.
|
||||
|
||||
Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it.
|
||||
|
||||
#### `analytics:hyperloglog`
|
||||
Approximate distinct counts. Count unique values (e.g., unique merchants) across millions of records using ~16KB of memory with ~1% error. Each update is O(1).
|
||||
|
||||
#### `analytics:tdigest`
|
||||
Streaming percentiles. Compute P50/P90/P95/P99 from streaming data without storing all values. Uses ~4KB per digest with ~1% accuracy at the tails.
|
||||
|
||||
#### `analytics:countmin`
|
||||
Frequency estimation. Find the most common values (e.g., top-K merchants) using ~40KB with 0.1% error. O(1) per update.
|
||||
|
||||
#### `analytics:anomaly`
|
||||
Real-time anomaly detection. Flag statistically unusual values at write-time using exponentially weighted moving averages. 64 bytes per group, sub-microsecond decisions.
|
||||
|
||||
#### `aggregation:mmap`
|
||||
Persistent aggregate storage via memory-mapped files. Aggregate state survives process crashes without explicit flush. Zero serialization overhead.
|
||||
|
||||
---
|
||||
|
||||
## Storage Adapter Plugins
|
||||
|
||||
Plugins can register custom storage backends that users reference by name.
|
||||
|
||||
### Implementing a Storage Adapter
|
||||
|
||||
```typescript
|
||||
import type { StorageAdapterFactory } from '@soulcraft/brainy/plugin'
|
||||
import type { StorageAdapter } from '@soulcraft/brainy'
|
||||
|
||||
class MyStorageAdapter implements StorageAdapter {
|
||||
async init(): Promise<void> { /* ... */ }
|
||||
async saveNoun(noun: HNSWNoun): Promise<void> { /* ... */ }
|
||||
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> { /* ... */ }
|
||||
async deleteNoun(id: string): Promise<void> { /* ... */ }
|
||||
// ... implement all StorageAdapter methods
|
||||
}
|
||||
```
|
||||
|
||||
### Registering a Storage Adapter
|
||||
|
||||
```typescript
|
||||
context.registerProvider('storage:my-backend', {
|
||||
name: 'my-backend',
|
||||
create: (config: Record<string, unknown>) => {
|
||||
return new MyStorageAdapter(config)
|
||||
}
|
||||
} satisfies StorageAdapterFactory)
|
||||
```
|
||||
|
||||
Users can then use your storage:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: 'my-backend', myBackendOption: 'value' })
|
||||
```
|
||||
|
||||
## Import Paths
|
||||
|
||||
Brainy provides three entry points for plugin developers:
|
||||
|
||||
| Import Path | Contents | Stability |
|
||||
|-------------|----------|-----------|
|
||||
| `@soulcraft/brainy` | Public API, types, StorageAdapter | Stable (semver) |
|
||||
| `@soulcraft/brainy/plugin` | BrainyPlugin, BrainyPluginContext, StorageAdapterFactory | Stable (semver) |
|
||||
| `@soulcraft/brainy/internals` | UnifiedCache, EntityIdMapper, logger utilities | Internal (may change between minor versions) |
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Brainy provides a `diagnostics()` method to verify plugin wiring:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
const diag = brain.diagnostics()
|
||||
console.log(diag)
|
||||
// {
|
||||
// version: '7.14.0',
|
||||
// plugins: { active: ['my-plugin'], count: 1 },
|
||||
// providers: {
|
||||
// metadataIndex: { source: 'default' },
|
||||
// graphIndex: { source: 'default' },
|
||||
// embeddings: { source: 'plugin' },
|
||||
// embedBatch: { source: 'plugin' },
|
||||
// distance: { source: 'plugin' },
|
||||
// vector: { source: 'default' },
|
||||
// ...
|
||||
// },
|
||||
// indexes: {
|
||||
// vector: { size: 0, type: 'JsHnswVectorIndex' },
|
||||
// metadata: { type: 'MetadataIndexManager', initialized: true },
|
||||
// graph: { type: 'GraphAdjacencyIndex', initialized: true, wiredToStorage: true }
|
||||
// }
|
||||
// }
|
||||
```
|
||||
|
||||
The CLI also supports diagnostics:
|
||||
|
||||
```bash
|
||||
brainy diagnostics
|
||||
```
|
||||
|
||||
### Init-Time Summary
|
||||
|
||||
When a plugin is active, brainy automatically logs a provider summary after `init()`:
|
||||
|
||||
```
|
||||
[brainy] Plugin activated: @soulcraft/cor
|
||||
[brainy] Providers: 8/10 native (@soulcraft/cor) | default: vector, cache
|
||||
```
|
||||
|
||||
This tells you at a glance how many subsystems are accelerated and which ones are falling back to JavaScript. The log respects `config.silent`.
|
||||
|
||||
### Fail-Fast for Production
|
||||
|
||||
Use `requireProviders()` after `init()` to guarantee specific providers are plugin-supplied. This prevents silent fallback to JavaScript in deployments where you expect native acceleration:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Throws immediately if any of these are using JS fallback
|
||||
brain.requireProviders(['distance', 'embeddings', 'metadataIndex', 'graphIndex'])
|
||||
```
|
||||
|
||||
If a required provider is missing, the error message tells you exactly what's wrong:
|
||||
|
||||
```
|
||||
[brainy] Required providers using JS fallback: graphIndex.
|
||||
Active plugins: @soulcraft/cor.
|
||||
These providers must be supplied by a plugin for this deployment.
|
||||
Check plugin installation, license, and native module availability.
|
||||
```
|
||||
|
||||
This is the recommended pattern for production deployments with paid plugins — fail at startup rather than silently degrading performance.
|
||||
|
||||
## Complete Example: Distance Acceleration Plugin
|
||||
|
||||
A minimal but useful plugin that provides SIMD-accelerated distance calculations:
|
||||
|
||||
```typescript
|
||||
// simd-distance-plugin/src/plugin.ts
|
||||
import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
|
||||
|
||||
// Hypothetical native module
|
||||
import { simdCosineDistance } from './native.js'
|
||||
|
||||
const simdDistancePlugin: BrainyPlugin = {
|
||||
name: 'brainy-simd-distance',
|
||||
|
||||
async activate(context: BrainyPluginContext): Promise<boolean> {
|
||||
// Check if SIMD is available on this platform
|
||||
if (!checkSimdSupport()) {
|
||||
console.log('[simd-distance] SIMD not available, skipping')
|
||||
return false // Don't activate — brainy uses JS fallback
|
||||
}
|
||||
|
||||
context.registerProvider('distance', simdCosineDistance)
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
export default simdDistancePlugin
|
||||
```
|
||||
|
||||
```json
|
||||
// simd-distance-plugin/package.json
|
||||
{
|
||||
"name": "brainy-simd-distance",
|
||||
"main": "./dist/plugin.js",
|
||||
"types": "./dist/plugin.d.ts",
|
||||
"peerDependencies": {
|
||||
"@soulcraft/brainy": ">=7.0.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({ plugins: ['brainy-simd-distance'] })
|
||||
await brain.init()
|
||||
|
||||
// Verify it's active
|
||||
const diag = brain.diagnostics()
|
||||
console.log(diag.providers.distance) // { source: 'plugin' }
|
||||
```
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Brainy works perfectly without plugins.** Every provider has a JavaScript fallback. Plugins only improve performance or add capabilities.
|
||||
|
||||
2. **Provider keys are string-based.** The plugin system is not coupled to any specific plugin. Any package can register any provider.
|
||||
|
||||
3. **Clean separation.** Plugins access brainy through the documented `BrainyPluginContext` interface. No direct access to internal classes is needed.
|
||||
|
||||
4. **Fail-safe activation.** If a plugin throws during `activate()`, brainy logs a warning and continues with defaults. A broken plugin never prevents brainy from working.
|
||||
|
||||
5. **Lifecycle management.** `deactivate()` is called during `brainy.close()` for resource cleanup. Native resources, connections, and file handles should be released here.
|
||||
562
docs/PRODUCTION_SERVICE_ARCHITECTURE.md
Normal file
562
docs/PRODUCTION_SERVICE_ARCHITECTURE.md
Normal file
|
|
@ -0,0 +1,562 @@
|
|||
# Production Service Architecture Guide
|
||||
|
||||
**How to use Brainy optimally in production services (Bun, Node.js, Deno)**
|
||||
|
||||
> **Recommended Runtime:** [Bun](https://bun.sh) provides best performance with Brainy's Candle WASM engine. All examples work with both Bun and Node.js.
|
||||
|
||||
---
|
||||
|
||||
## The Problem: Instance-per-Request Anti-Pattern
|
||||
|
||||
### ❌ What NOT to Do
|
||||
|
||||
```typescript
|
||||
// WRONG - Creates new instance EVERY request
|
||||
app.get('/api/entities', async (req, res) => {
|
||||
const brain = new Brainy({ storage: { path: './brainy-data' } })
|
||||
await brain.init() // FULL INITIALIZATION EVERY TIME!
|
||||
const entities = await brain.find(...)
|
||||
res.json(entities)
|
||||
})
|
||||
```
|
||||
|
||||
### Why This is Terrible
|
||||
|
||||
After 40 API calls:
|
||||
- **40 Brainy instances** running simultaneously
|
||||
- **20GB memory** (40 × 500MB per instance)
|
||||
- **2 seconds wasted** (40 × 50ms initialization)
|
||||
- **Zero cache benefit** (each instance has its own empty cache)
|
||||
- **Index rebuilding** on every request (TypeAware HNSW, LSM-trees, etc.)
|
||||
- **Memory leaks** (old instances may not GC properly)
|
||||
|
||||
---
|
||||
|
||||
## ✅ The Solution: Singleton Pattern
|
||||
|
||||
**ONE Brainy instance per service, shared across ALL requests.**
|
||||
|
||||
### Performance Comparison
|
||||
|
||||
| Metric | Instance-per-Request | Singleton (Optimal) |
|
||||
|--------|---------------------|---------------------|
|
||||
| Memory (40 requests) | 20GB | 500MB |
|
||||
| Request 1 latency | 60ms | 60ms (one-time init) |
|
||||
| Request 2+ latency | 60ms (no cache!) | 2ms (80% cache hit!) |
|
||||
| Cache hit rate | 0% | 80%+ |
|
||||
| Speedup | - | **30x faster** |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Patterns
|
||||
|
||||
### Pattern 1: Simple Singleton (Recommended)
|
||||
|
||||
```typescript
|
||||
// server.ts
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// SINGLETON INSTANCE
|
||||
let brainInstance: Brainy | null = null
|
||||
|
||||
async function getBrain(): Promise<Brainy> {
|
||||
if (brainInstance) {
|
||||
return brainInstance
|
||||
}
|
||||
|
||||
console.log('🧠 Initializing Brainy singleton...')
|
||||
|
||||
brainInstance = new Brainy({
|
||||
storage: {
|
||||
path: './brainy-data',
|
||||
autoOptimize: true
|
||||
},
|
||||
cache: {
|
||||
maxSize: 1000, // Shared across ALL requests
|
||||
ttl: 3600000, // 1 hour
|
||||
enableMetrics: true
|
||||
},
|
||||
augmentations: {
|
||||
include: ['cache', 'metrics', 'display', 'vfs']
|
||||
}
|
||||
})
|
||||
|
||||
await brainInstance.init()
|
||||
console.log('✅ Brainy ready')
|
||||
|
||||
return brainInstance
|
||||
}
|
||||
|
||||
// Initialize BEFORE starting server
|
||||
async function startServer() {
|
||||
await getBrain() // One-time initialization
|
||||
|
||||
app.get('/api/entities', async (req, res) => {
|
||||
const brain = await getBrain() // Reuses same instance!
|
||||
const entities = await brain.find(req.query)
|
||||
res.json(entities)
|
||||
})
|
||||
|
||||
app.listen(3000)
|
||||
}
|
||||
|
||||
startServer()
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ Simple to implement
|
||||
- ✅ Thread-safe (async initialization)
|
||||
- ✅ Shared cache and indexes
|
||||
- ✅ 40x memory reduction
|
||||
|
||||
---
|
||||
|
||||
### Pattern 2: Service Class (Production-Grade)
|
||||
|
||||
```typescript
|
||||
// services/BrainService.ts
|
||||
export class BrainService {
|
||||
private brain: Brainy | null = null
|
||||
private initPromise: Promise<Brainy> | null = null
|
||||
|
||||
async getInstance(): Promise<Brainy> {
|
||||
if (this.brain) return this.brain
|
||||
if (this.initPromise) return this.initPromise
|
||||
|
||||
this.initPromise = this.initialize()
|
||||
return this.initPromise
|
||||
}
|
||||
|
||||
private async initialize(): Promise<Brainy> {
|
||||
this.brain = new Brainy({
|
||||
storage: {
|
||||
path: process.env.BRAINY_DATA_PATH || './brainy-data'
|
||||
},
|
||||
cache: { maxSize: 1000, ttl: 3600000 }
|
||||
})
|
||||
await this.brain.init()
|
||||
return this.brain
|
||||
}
|
||||
|
||||
async shutdown(): Promise<void> {
|
||||
if (this.brain) {
|
||||
// Cleanup if needed
|
||||
this.brain = null
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// server.ts
|
||||
const brainService = new BrainService()
|
||||
|
||||
app.get('/api/entities', async (req, res) => {
|
||||
const brain = await brainService.getInstance()
|
||||
const entities = await brain.find(req.query)
|
||||
res.json(entities)
|
||||
})
|
||||
|
||||
// Graceful shutdown
|
||||
process.on('SIGTERM', async () => {
|
||||
await brainService.shutdown()
|
||||
process.exit(0)
|
||||
})
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ Prevents race conditions (multiple simultaneous inits)
|
||||
- ✅ Testable (can inject mock)
|
||||
- ✅ Clean shutdown handling
|
||||
- ✅ Environment-configurable
|
||||
|
||||
---
|
||||
|
||||
### Pattern 3: Bun Server (Recommended)
|
||||
|
||||
```typescript
|
||||
// server.ts - Clean Bun implementation
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
let brain: Brainy | null = null
|
||||
|
||||
async function getBrain(): Promise<Brainy> {
|
||||
if (!brain) {
|
||||
brain = new Brainy({ storage: { path: './brainy-data' } })
|
||||
await brain.init()
|
||||
}
|
||||
return brain
|
||||
}
|
||||
|
||||
// Initialize before server starts
|
||||
await getBrain()
|
||||
|
||||
Bun.serve({
|
||||
port: 3000,
|
||||
async fetch(req) {
|
||||
const url = new URL(req.url)
|
||||
|
||||
if (url.pathname === '/api/entities') {
|
||||
const b = await getBrain()
|
||||
const entities = await b.find({})
|
||||
return Response.json(entities)
|
||||
}
|
||||
|
||||
if (url.pathname === '/api/entity' && req.method === 'POST') {
|
||||
const b = await getBrain()
|
||||
const body = await req.json()
|
||||
const id = await b.add(body)
|
||||
return Response.json({ id })
|
||||
}
|
||||
|
||||
return new Response('Not Found', { status: 404 })
|
||||
}
|
||||
})
|
||||
|
||||
console.log('Server running on http://localhost:3000')
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ Native Bun runtime performance
|
||||
- ✅ No framework dependencies
|
||||
- ✅ Pure WASM — no native binaries, bundler-friendly
|
||||
- ✅ Built-in TypeScript support
|
||||
|
||||
### Pattern 4: Express/Node.js Middleware (Legacy)
|
||||
|
||||
```typescript
|
||||
// middleware/brainy.ts
|
||||
let brainInstance: Brainy | null = null
|
||||
|
||||
export async function initBrainy() {
|
||||
if (!brainInstance) {
|
||||
brainInstance = new Brainy({ storage: { path: './brainy-data' } })
|
||||
await brainInstance.init()
|
||||
}
|
||||
}
|
||||
|
||||
export function brainMiddleware(req, res, next) {
|
||||
if (!brainInstance) {
|
||||
return res.status(500).json({ error: 'Brainy not initialized' })
|
||||
}
|
||||
req.brain = brainInstance // Attach to request
|
||||
next()
|
||||
}
|
||||
|
||||
// Type extension
|
||||
declare global {
|
||||
namespace Express {
|
||||
interface Request {
|
||||
brain: Brainy
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// server.ts
|
||||
import { initBrainy, brainMiddleware } from './middleware/brainy'
|
||||
|
||||
async function startServer() {
|
||||
await initBrainy() // Initialize first
|
||||
|
||||
app.use('/api', brainMiddleware) // Apply to API routes
|
||||
|
||||
app.get('/api/entities', async (req, res) => {
|
||||
const entities = await req.brain.find(req.query) // Type-safe!
|
||||
res.json(entities)
|
||||
})
|
||||
|
||||
app.listen(3000)
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ Clean separation of concerns
|
||||
- ✅ Type-safe (`req.brain` is typed)
|
||||
- ✅ Easy to add auth/validation
|
||||
|
||||
---
|
||||
|
||||
## Optimization Strategies
|
||||
|
||||
### 1. Configure Cache for Your Workload
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
cache: {
|
||||
maxSize: 1000, // Number of entities to cache
|
||||
ttl: 3600000, // Cache lifetime (1 hour)
|
||||
enableMetrics: true, // Track hit rate
|
||||
evictionPolicy: 'lru' // Least recently used
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Cache sizing:**
|
||||
- Small service (< 100 req/min): `maxSize: 500`
|
||||
- Medium service (< 1000 req/min): `maxSize: 1000`
|
||||
- Large service (> 1000 req/min): `maxSize: 5000`
|
||||
|
||||
### 2. Lazy Load Augmentations
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
augmentations: {
|
||||
// Only load what you actually use
|
||||
include: ['cache', 'metrics', 'display', 'vfs'],
|
||||
exclude: ['neuralImport', 'intelligentImport'] // Skip heavy features
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Memory savings:**
|
||||
- With all augmentations: ~800MB
|
||||
- With minimal set: ~400MB
|
||||
|
||||
### 3. Warm Up Indexes
|
||||
|
||||
```typescript
|
||||
async function startServer() {
|
||||
const brain = await getBrain()
|
||||
|
||||
// Pre-warm frequently-used indexes
|
||||
await brain.find({ type: 'person', limit: 1 })
|
||||
await brain.find({ type: 'organization', limit: 1 })
|
||||
|
||||
console.log('✅ Indexes pre-warmed')
|
||||
|
||||
app.listen(3000)
|
||||
}
|
||||
```
|
||||
|
||||
**Benefit:** First requests are fast (no cold-start index building)
|
||||
|
||||
### 4. Memory-Aware Configuration
|
||||
|
||||
```typescript
|
||||
import os from 'os'
|
||||
|
||||
const totalMemory = os.totalmem()
|
||||
const availableMemory = os.freemem()
|
||||
|
||||
const brain = new Brainy({
|
||||
cache: {
|
||||
// Use 10% of total RAM for cache
|
||||
maxSize: Math.floor(totalMemory * 0.1 / (1024 * 1024))
|
||||
},
|
||||
indexes: {
|
||||
// Lazy load indexes if low memory
|
||||
lazyLoad: availableMemory < totalMemory * 0.5,
|
||||
preload: ['person', 'organization'] // Only preload common types
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Concurrency & Thread Safety
|
||||
|
||||
Brainy is **designed** for concurrent access. A single instance can handle:
|
||||
|
||||
```typescript
|
||||
// Multiple concurrent requests - all using same instance
|
||||
app.get('/api/read/:id', async (req, res) => {
|
||||
const brain = getBrain()
|
||||
const entity = await brain.get(req.params.id) // Safe - no state mutation
|
||||
res.json(entity)
|
||||
})
|
||||
|
||||
app.post('/api/write', async (req, res) => {
|
||||
const brain = getBrain()
|
||||
const id = await brain.add(req.body) // Safe - internal locking
|
||||
res.json({ id })
|
||||
})
|
||||
```
|
||||
|
||||
**Concurrency mechanisms:**
|
||||
- ✅ **Read operations**: Lock-free (MVCC)
|
||||
- ✅ **Write operations**: Internal write-ahead logging (WAL)
|
||||
- ✅ **Cache**: Thread-safe LRU implementation
|
||||
- ✅ **Indexes**: Concurrent reads, locked writes
|
||||
|
||||
---
|
||||
|
||||
## Production Checklist
|
||||
|
||||
### Before Deploying
|
||||
|
||||
- [ ] **Initialize Brainy on startup** (not per-request)
|
||||
- [ ] **Configure cache size** based on memory
|
||||
- [ ] **Only load needed augmentations**
|
||||
- [ ] **Warm up critical indexes**
|
||||
- [ ] **Add graceful shutdown handler**
|
||||
- [ ] **Monitor cache hit rate**
|
||||
|
||||
### Code Review Checklist
|
||||
|
||||
```typescript
|
||||
// ❌ BAD - Instance per request
|
||||
app.get('/api/route', async (req, res) => {
|
||||
const brain = new Brainy(...) // RED FLAG!
|
||||
await brain.init() // RED FLAG!
|
||||
})
|
||||
|
||||
// ✅ GOOD - Singleton pattern
|
||||
app.get('/api/route', async (req, res) => {
|
||||
const brain = await getBrain() // Reuses instance ✓
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Metrics
|
||||
|
||||
```typescript
|
||||
// Add metrics endpoint
|
||||
app.get('/api/metrics', (req, res) => {
|
||||
const brain = getBrain()
|
||||
|
||||
res.json({
|
||||
cache: {
|
||||
size: brain.cache?.size || 0,
|
||||
maxSize: brain.cache?.maxSize || 0,
|
||||
hitRate: brain.metrics?.cacheHitRate || 0 // Target: >70%
|
||||
},
|
||||
storage: brain.storage.getStats(),
|
||||
memory: {
|
||||
heapUsed: Math.round(process.memoryUsage().heapUsed / 1024 / 1024),
|
||||
heapTotal: Math.round(process.memoryUsage().heapTotal / 1024 / 1024)
|
||||
}
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
**Key metrics to track:**
|
||||
- **Cache hit rate**: Should be >70% after warm-up
|
||||
- **Memory usage**: Should stay constant (~500MB for singleton)
|
||||
- **Request latency**: Should be <10ms for cached entities
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### 1. Creating instances in routes
|
||||
```typescript
|
||||
// ❌ NEVER do this
|
||||
app.get('/api/entities', async (req, res) => {
|
||||
const brain = new Brainy(...) // Creates new instance every time!
|
||||
})
|
||||
```
|
||||
|
||||
### 2. Not awaiting initialization
|
||||
```typescript
|
||||
// ❌ Race condition - server starts before Brainy ready
|
||||
app.listen(3000)
|
||||
getBrain() // Async init happens AFTER server starts!
|
||||
|
||||
// ✅ Correct - wait for init
|
||||
await getBrain()
|
||||
app.listen(3000)
|
||||
```
|
||||
|
||||
### 3. Multiple instances for different purposes
|
||||
```typescript
|
||||
// ❌ Wasteful - creates 2 instances
|
||||
const readBrain = new Brainy(...)
|
||||
const writeBrain = new Brainy(...)
|
||||
|
||||
// ✅ One instance handles both
|
||||
const brain = new Brainy(...)
|
||||
await brain.get(id) // Read
|
||||
await brain.add(data) // Write
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### Current (Anti-Pattern)
|
||||
```typescript
|
||||
// Probably in multiple route files
|
||||
async function handler(req, res) {
|
||||
const brain = new Brainy({ storage: { path: './brainy-data' } })
|
||||
await brain.init()
|
||||
// ... use brain
|
||||
}
|
||||
```
|
||||
|
||||
### Step 1: Create Singleton Module
|
||||
```typescript
|
||||
// lib/brainy.ts
|
||||
let instance: Brainy | null = null
|
||||
|
||||
export async function getBrain(): Promise<Brainy> {
|
||||
if (!instance) {
|
||||
instance = new Brainy({ storage: { path: './brainy-data' } })
|
||||
await instance.init()
|
||||
}
|
||||
return instance
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Update Server Startup
|
||||
```typescript
|
||||
// server.ts
|
||||
import { getBrain } from './lib/brainy'
|
||||
|
||||
async function startServer() {
|
||||
// Initialize Brainy FIRST
|
||||
await getBrain()
|
||||
console.log('✅ Brainy initialized')
|
||||
|
||||
// THEN start server
|
||||
app.listen(3000)
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Update All Routes
|
||||
```typescript
|
||||
// Before
|
||||
async function handler(req, res) {
|
||||
const brain = new Brainy(...) // Remove this
|
||||
await brain.init() // Remove this
|
||||
|
||||
// ... rest of code
|
||||
}
|
||||
|
||||
// After
|
||||
import { getBrain } from './lib/brainy'
|
||||
|
||||
async function handler(req, res) {
|
||||
const brain = await getBrain() // Add this
|
||||
|
||||
// ... rest of code stays same
|
||||
}
|
||||
```
|
||||
|
||||
**Expected results:**
|
||||
- ✅ 40x memory reduction (20GB → 500MB)
|
||||
- ✅ 30x faster requests (60ms → 2ms average)
|
||||
- ✅ 80%+ cache hit rate
|
||||
- ✅ Your service can scale to 1000s of requests/minute
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**DO:**
|
||||
- ✅ Initialize Brainy ONCE on server startup
|
||||
- ✅ Share single instance across all requests
|
||||
- ✅ Configure cache for your workload
|
||||
- ✅ Monitor cache hit rate
|
||||
- ✅ Handle graceful shutdown
|
||||
|
||||
**DON'T:**
|
||||
- ❌ Create new Brainy instance per request
|
||||
- ❌ Create multiple instances
|
||||
- ❌ Start server before Brainy is initialized
|
||||
- ❌ Load augmentations you don't use
|
||||
|
||||
**Result:** 40x less memory, 30x faster requests, Brainy optimizations actually work!
|
||||
|
||||
---
|
||||
|
||||
**Questions? Issues?**
|
||||
- Report issues: https://github.com/soulcraftlabs/brainy/issues
|
||||
298
docs/QUERY_OPERATORS.md
Normal file
298
docs/QUERY_OPERATORS.md
Normal file
|
|
@ -0,0 +1,298 @@
|
|||
# Query Operators (BFO)
|
||||
|
||||
> Brainy Field Operators — the complete reference for `where` filters in `find()`.
|
||||
|
||||
All operators work with `find({ where: { ... } })` and filter on **metadata fields** (not `data`).
|
||||
|
||||
---
|
||||
|
||||
## Equality
|
||||
|
||||
| Operator | Alias | Description | Example |
|
||||
|----------|-------|-------------|---------|
|
||||
| `eq` | `equals` | Exact match | `{ status: { eq: 'active' } }` |
|
||||
| `ne` | `notEquals` | Not equal | `{ status: { ne: 'deleted' } }` |
|
||||
|
||||
**Shorthand:** A bare value is treated as `equals`:
|
||||
|
||||
```typescript
|
||||
// These are equivalent:
|
||||
brain.find({ where: { status: 'active' } })
|
||||
brain.find({ where: { status: { equals: 'active' } } })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Comparison
|
||||
|
||||
| Operator | Alias | Description | Example |
|
||||
|----------|-------|-------------|---------|
|
||||
| `gt` | `greaterThan` | Greater than | `{ age: { gt: 18 } }` |
|
||||
| `gte` | `greaterThanOrEqual` | Greater or equal | `{ score: { gte: 90 } }` |
|
||||
| `lt` | `lessThan` | Less than | `{ price: { lt: 100 } }` |
|
||||
| `lte` | `lessThanOrEqual` | Less or equal | `{ rating: { lte: 3 } }` |
|
||||
| `between` | — | Inclusive range `[min, max]` | `{ year: { between: [2020, 2025] } }` |
|
||||
|
||||
```typescript
|
||||
// Range query
|
||||
const recent = await brain.find({
|
||||
where: {
|
||||
createdAt: { between: [Date.now() - 86400000, Date.now()] }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Array / Set
|
||||
|
||||
| Operator | Alias | Description | Example |
|
||||
|----------|-------|-------------|---------|
|
||||
| `oneOf` | `in` | Value is one of the given options | `{ color: { oneOf: ['red', 'blue'] } }` |
|
||||
| `noneOf` | — | Value is NOT one of the given options | `{ status: { noneOf: ['deleted', 'archived'] } }` |
|
||||
| `contains` | — | Array field contains value | `{ tags: { contains: 'ai' } }` |
|
||||
| `excludes` | — | Array field does NOT contain value | `{ tags: { excludes: 'spam' } }` |
|
||||
| `hasAll` | — | Array field contains ALL listed values | `{ skills: { hasAll: ['js', 'ts'] } }` |
|
||||
|
||||
```typescript
|
||||
// Find entities tagged with 'ai'
|
||||
const aiEntities = await brain.find({
|
||||
where: { tags: { contains: 'ai' } }
|
||||
})
|
||||
|
||||
// Find entities of specific types
|
||||
const people = await brain.find({
|
||||
where: { noun: { oneOf: ['Person', 'Agent'] } }
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Existence
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `exists: true` | Field exists (has any value) | `{ email: { exists: true } }` |
|
||||
| `exists: false` | Field does NOT exist | `{ email: { exists: false } }` |
|
||||
| `missing: true` | Field does NOT exist (alias for `exists: false`) | `{ email: { missing: true } }` |
|
||||
| `missing: false` | Field exists (alias for `exists: true`) | `{ email: { missing: false } }` |
|
||||
|
||||
```typescript
|
||||
// Find entities that have an email field
|
||||
const withEmail = await brain.find({
|
||||
where: { email: { exists: true } }
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pattern (In-Memory Only)
|
||||
|
||||
These operators work via the in-memory filter path. They are applied **after** the indexed query, so use them with other indexed operators for best performance.
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `matches` | Regex or string pattern match | `{ name: { matches: /^Dr\./ } }` |
|
||||
| `startsWith` | String prefix | `{ name: { startsWith: 'John' } }` |
|
||||
| `endsWith` | String suffix | `{ email: { endsWith: '@gmail.com' } }` |
|
||||
|
||||
```typescript
|
||||
const doctors = await brain.find({
|
||||
where: {
|
||||
type: NounType.Person, // Indexed — fast
|
||||
name: { startsWith: 'Dr.' } // In-memory — applied after
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Logical
|
||||
|
||||
Combine multiple conditions:
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `allOf` | ALL sub-filters must match (AND) | `{ allOf: [{ status: 'active' }, { role: 'admin' }] }` |
|
||||
| `anyOf` | ANY sub-filter must match (OR) | `{ anyOf: [{ role: 'admin' }, { role: 'owner' }] }` |
|
||||
| `not` | Invert a filter | `{ not: { status: 'deleted' } }` |
|
||||
|
||||
```typescript
|
||||
// Complex OR query
|
||||
const adminsOrOwners = await brain.find({
|
||||
where: {
|
||||
anyOf: [
|
||||
{ role: 'admin' },
|
||||
{ role: 'owner' }
|
||||
]
|
||||
}
|
||||
})
|
||||
|
||||
// NOT query
|
||||
const notDeleted = await brain.find({
|
||||
where: {
|
||||
not: { status: 'deleted' }
|
||||
}
|
||||
})
|
||||
|
||||
// Combined AND + OR
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
allOf: [
|
||||
{ department: 'engineering' },
|
||||
{ anyOf: [
|
||||
{ level: 'senior' },
|
||||
{ yearsExperience: { greaterThan: 5 } }
|
||||
]}
|
||||
]
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Indexed vs In-Memory Operators
|
||||
|
||||
Brainy's MetadataIndex supports a subset of operators natively for O(1) field lookups. Other operators fall back to in-memory filtering.
|
||||
|
||||
| Operator | MetadataIndex (Indexed) | In-Memory Fallback |
|
||||
|----------|:-----------------------:|:------------------:|
|
||||
| `equals` / `eq` | Yes | Yes |
|
||||
| `notEquals` / `ne` | — | Yes |
|
||||
| `greaterThan` / `gt` | Yes | Yes |
|
||||
| `greaterThanOrEqual` / `gte` | Yes | Yes |
|
||||
| `lessThan` / `lt` | Yes | Yes |
|
||||
| `lessThanOrEqual` / `lte` | Yes | Yes |
|
||||
| `between` | Yes | Yes |
|
||||
| `oneOf` / `in` | Yes | Yes |
|
||||
| `noneOf` | — | Yes |
|
||||
| `contains` | Yes | Yes |
|
||||
| `exists` / `missing` | Yes | Yes |
|
||||
| `matches` | — | Yes |
|
||||
| `startsWith` | — | Yes |
|
||||
| `endsWith` | — | Yes |
|
||||
| `allOf` | Partial | Yes |
|
||||
| `anyOf` | Partial | Yes |
|
||||
| `not` | — | Yes |
|
||||
|
||||
**Performance tip:** Combine indexed operators (equals, greaterThan, oneOf, between, contains, exists) with pattern operators for optimal speed — the index narrows results first, then patterns filter in memory.
|
||||
|
||||
---
|
||||
|
||||
## Practical Examples
|
||||
|
||||
### Filter by entity type
|
||||
|
||||
```typescript
|
||||
// Using the type shorthand (recommended)
|
||||
brain.find({ type: NounType.Person })
|
||||
|
||||
// Using where.noun directly
|
||||
brain.find({ where: { noun: NounType.Person } })
|
||||
|
||||
// Multiple types
|
||||
brain.find({ type: [NounType.Person, NounType.Agent] })
|
||||
```
|
||||
|
||||
### Filter by subtype
|
||||
|
||||
`subtype` is a top-level standard field — takes the column-store fast path, not the metadata fallback. Pair with `type` for the typical "Person who is an employee" query:
|
||||
|
||||
```typescript
|
||||
// Equality on subtype:
|
||||
brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// Set membership:
|
||||
brain.find({ type: NounType.Person, subtype: ['employee', 'contractor'] })
|
||||
|
||||
// Operator-form predicates use `where`:
|
||||
brain.find({
|
||||
type: NounType.Person,
|
||||
where: { subtype: { exists: true } }
|
||||
})
|
||||
```
|
||||
|
||||
See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the full surface.
|
||||
|
||||
### Filter relationships by subtype (7.30+)
|
||||
|
||||
Verbs are first-class peers — `related()` and graph traversal both honor subtype filters on the fast path:
|
||||
|
||||
```typescript
|
||||
// Filter relationships by VerbType subtype
|
||||
const direct = await brain.related({
|
||||
from: ceoId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: 'direct'
|
||||
})
|
||||
|
||||
// Set membership on verb subtype
|
||||
const all = await brain.related({
|
||||
from: ceoId,
|
||||
type: VerbType.ReportsTo,
|
||||
subtype: ['direct', 'dotted-line']
|
||||
})
|
||||
|
||||
// Graph traversal — subtype filters traversal edges (depth-1 in 7.30 JS path;
|
||||
// multi-hop subtype filtering lands on Cor native)
|
||||
const reports = await brain.find({
|
||||
connected: {
|
||||
from: ceoId,
|
||||
via: VerbType.ReportsTo,
|
||||
subtype: 'direct',
|
||||
depth: 1
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Combine semantic search with filters
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
query: 'machine learning engineer', // Semantic search (on data)
|
||||
type: NounType.Person, // Type filter (indexed)
|
||||
where: {
|
||||
department: 'engineering', // Exact match (indexed)
|
||||
yearsExperience: { greaterThan: 3 } // Range filter (indexed)
|
||||
},
|
||||
limit: 10
|
||||
})
|
||||
```
|
||||
|
||||
### Temporal queries
|
||||
|
||||
```typescript
|
||||
const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000
|
||||
const recentEntities = await brain.find({
|
||||
where: {
|
||||
createdAt: { greaterThan: lastWeek }
|
||||
},
|
||||
orderBy: 'createdAt',
|
||||
order: 'desc',
|
||||
limit: 50
|
||||
})
|
||||
```
|
||||
|
||||
### Graph + metadata combination
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
connected: {
|
||||
from: teamLeadId,
|
||||
via: VerbType.WorksWith,
|
||||
depth: 2
|
||||
},
|
||||
where: {
|
||||
role: { oneOf: ['engineer', 'designer'] },
|
||||
active: true
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Data Model](./DATA_MODEL.md) — Entity structure, data vs metadata
|
||||
- [API Reference](./api/README.md) — Complete API documentation
|
||||
- [Find System](./FIND_SYSTEM.md) — Natural language find() details
|
||||
232
docs/README.md
232
docs/README.md
|
|
@ -1,122 +1,130 @@
|
|||
# Brainy Documentation
|
||||
|
||||
Welcome to the comprehensive documentation for Brainy - the intelligent vector graph database with zero-configuration setup and production-scale performance.
|
||||
> The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API.
|
||||
|
||||
## 📚 Documentation Structure
|
||||
## Quick Start
|
||||
|
||||
### 🚀 [Getting Started](getting-started/)
|
||||
Quick setup guides and first steps with Brainy.
|
||||
```typescript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
- **[Quick Start Guide](getting-started/quick-start.md)** - Get up and running in minutes
|
||||
- **[Installation Guide](getting-started/installation.md)** - Installation options and requirements
|
||||
- **[Environment Setup](getting-started/environment-setup.md)** - Configure your development environment
|
||||
- **[First Steps](getting-started/first-steps.md)** - Your first Brainy application
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
### 📖 [User Guides](user-guides/)
|
||||
Comprehensive guides for using Brainy features.
|
||||
// Add entities — data is embedded for semantic search, metadata is indexed for filtering
|
||||
const id = await brain.add({
|
||||
data: 'Revolutionary AI Breakthrough',
|
||||
type: NounType.Document,
|
||||
metadata: { category: 'technology', rating: 4.8 }
|
||||
})
|
||||
|
||||
- **[Search and Metadata Guide](user-guides/SEARCH_AND_METADATA_GUIDE.md)** - Advanced search techniques
|
||||
- **[Write-Only Mode](user-guides/WRITEONLY_MODE_IMPLEMENTATION.md)** - Optimized data ingestion
|
||||
- **[Read-Only & Frozen Modes](guides/readonly-frozen-modes.md)** - Immutability control for production
|
||||
- **[Per-Service Statistics](guides/per-service-statistics.md)** - Multi-service tracking and monitoring
|
||||
- **[Cache Configuration](guides/cache-configuration.md)** - Memory and caching optimization
|
||||
- **[JSON Document Search](guides/json-document-search.md)** - Searching within JSON documents
|
||||
- **[HNSW Field Search](guides/hnsw-field-search.md)** - Field-specific vector search
|
||||
- **[Model Management](guides/model-management.md)** - Managing AI models and embeddings
|
||||
- **[Production Migration](guides/production-migration-guide.md)** - Moving to production
|
||||
|
||||
### ⚡ [Optimization Guides](optimization-guides/)
|
||||
Performance optimization and scaling strategies.
|
||||
|
||||
- **[Large-Scale Optimizations](optimization-guides/large-scale-optimizations.md)** - Enterprise-grade performance
|
||||
- **[Auto-Configuration System](optimization-guides/auto-configuration.md)** - Zero-config intelligence
|
||||
- **[Semantic Partitioning](optimization-guides/semantic-partitioning.md)** - Intelligent data clustering
|
||||
- **[Distributed Search](optimization-guides/distributed-search.md)** - Parallel processing
|
||||
- **[Memory Optimization](optimization-guides/memory-optimization.md)** - Efficient memory usage
|
||||
- **[Storage Optimization](optimization-guides/storage-optimization.md)** - S3 and storage strategies
|
||||
- **[S3 Migration Guide](optimization-guides/s3-migration-guide.md)** - Migrating existing data with shared buckets
|
||||
|
||||
### 🔧 [API Reference](api-reference/)
|
||||
Complete API documentation and examples.
|
||||
|
||||
- **[Core API](api-reference/core-api.md)** - Main BrainyData class methods
|
||||
- **[Vector Operations](api-reference/vector-operations.md)** - Vector storage and search
|
||||
- **[Graph Operations](api-reference/graph-operations.md)** - Noun and verb relationships
|
||||
- **[Configuration API](api-reference/configuration.md)** - System configuration options
|
||||
- **[Storage Adapters](api-reference/storage-adapters.md)** - Universal storage compatibility and custom adapters
|
||||
- **[Augmentations API](api-reference/augmentations.md)** - Extension system
|
||||
|
||||
### 🛠️ [Development](development/)
|
||||
Development, testing, and contribution guides.
|
||||
|
||||
- **[Developer Guide](development/DEVELOPERS.md)** - Setting up development environment
|
||||
- **[Testing Guide](development/testing.md)** - Running and writing tests
|
||||
- **[Documentation Standards](development/DOCUMENTATION_STANDARDS.md)** - Documentation conventions
|
||||
- **[Publishing CLI](development/publishing-cli.md)** - CLI package publishing
|
||||
- **[Expected Test Messages](development/EXPECTED_TEST_MESSAGES.md)** - Test output reference
|
||||
|
||||
### 🔬 [Technical Reference](technical/)
|
||||
Deep technical documentation and implementation details.
|
||||
|
||||
- **[Technical Guides Overview](technical/TECHNICAL_GUIDES.md)** - Technical documentation index
|
||||
- **[Architecture](technical/architecture.md)** - System architecture overview
|
||||
- **[HNSW Implementation](technical/hnsw-implementation.md)** - Vector index details
|
||||
- **[Threading](technical/THREADING.md)** - Multi-threading implementation
|
||||
- **[Storage Systems](technical/storage-systems.md)** - Storage architecture
|
||||
- **[Performance Analysis](technical/performance-analysis.md)** - Performance benchmarks
|
||||
- **[Compatibility](technical/COMPATIBILITY.md)** - Platform compatibility matrix
|
||||
|
||||
### 💡 [Examples](examples/)
|
||||
Code examples and tutorials.
|
||||
|
||||
- **[Basic Usage](examples/basic-usage.md)** - Simple examples to get started
|
||||
- **[Advanced Patterns](examples/advanced-patterns.md)** - Complex use cases
|
||||
- **[Integration Examples](examples/integrations.md)** - Third-party integrations
|
||||
- **[Performance Examples](examples/performance.md)** - Optimization examples
|
||||
|
||||
### 🔍 [Troubleshooting](troubleshooting/)
|
||||
Common issues and solutions.
|
||||
|
||||
- **[Common Issues](troubleshooting/common-issues.md)** - Frequently encountered problems
|
||||
- **[Performance Issues](troubleshooting/performance.md)** - Performance troubleshooting
|
||||
- **[Environment Issues](troubleshooting/environment.md)** - Platform-specific problems
|
||||
- **[Error Messages](troubleshooting/error-messages.md)** - Error code reference
|
||||
|
||||
## 🗺️ Quick Navigation
|
||||
|
||||
### New to Brainy?
|
||||
1. **[Installation Guide](getting-started/installation.md)** - Install Brainy
|
||||
2. **[Quick Start Guide](getting-started/quick-start.md)** - Your first vector database
|
||||
3. **[Basic Usage Examples](examples/basic-usage.md)** - Simple code examples
|
||||
|
||||
### Need Performance?
|
||||
1. **[Large-Scale Optimizations](optimization-guides/large-scale-optimizations.md)** - Enterprise features
|
||||
2. **[Auto-Configuration](optimization-guides/auto-configuration.md)** - Zero-config setup
|
||||
3. **[Performance Examples](examples/performance.md)** - Optimization patterns
|
||||
|
||||
### Building Applications?
|
||||
1. **[API Reference](api-reference/)** - Complete API documentation
|
||||
2. **[User Guides](user-guides/)** - Feature documentation
|
||||
3. **[Integration Examples](examples/integrations.md)** - Real-world usage
|
||||
|
||||
### Contributing?
|
||||
1. **[Developer Guide](development/DEVELOPERS.md)** - Development setup
|
||||
2. **[Documentation Standards](development/DOCUMENTATION_STANDARDS.md)** - Writing docs
|
||||
3. **[Testing Guide](development/testing.md)** - Testing practices
|
||||
|
||||
## 🔄 Recently Updated
|
||||
|
||||
- **[S3 Migration Guide](optimization-guides/s3-migration-guide.md)** - NEW: Complete migration guide for shared S3 buckets
|
||||
- **[Large-Scale Optimizations](optimization-guides/large-scale-optimizations.md)** - Complete rewrite with auto-configuration
|
||||
- **[Quick Start Guide](getting-started/quick-start.md)** - Updated with zero-config setup
|
||||
- **[API Reference](api-reference/)** - New auto-configuration APIs
|
||||
|
||||
## 📞 Getting Help
|
||||
|
||||
- **[GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)** - Bug reports and feature requests
|
||||
- **[Discussions](https://github.com/soulcraftlabs/brainy/discussions)** - Community support
|
||||
- **[Examples](examples/)** - Code examples and tutorials
|
||||
// Search with Triple Intelligence
|
||||
const results = await brain.find({
|
||||
query: 'artificial intelligence', // Semantic search (on data)
|
||||
where: { rating: { greaterThan: 4.0 } }, // Metadata filter
|
||||
connected: { from: authorId, depth: 2 } // Graph traversal
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**📝 Note**: This documentation is automatically organized and regularly updated. All guides include practical examples and are tested with the latest version of Brainy.
|
||||
## Core Documentation
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| **[API Reference](./api/README.md)** | Complete API documentation — **start here** |
|
||||
| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
|
||||
| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
|
||||
| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
|
||||
| [Consistency Model](./concepts/consistency-model.md) | The Db API guarantees — snapshot isolation, atomic transactions, time travel |
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Architecture Overview](./architecture/overview.md) | High-level system design |
|
||||
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Metadata unified query |
|
||||
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | 42 nouns + 127 verbs type system |
|
||||
| [Stage 3 Canonical Taxonomy](./STAGE3-CANONICAL-TAXONOMY.md) | Complete type reference |
|
||||
| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
|
||||
| [Index Architecture](./architecture/index-architecture.md) | Vector, Graph, and Metadata indexing |
|
||||
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
|
||||
|
||||
---
|
||||
|
||||
## Virtual Filesystem (VFS)
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [VFS Quick Start](./vfs/QUICK_START.md) | Get started in 30 seconds |
|
||||
| [VFS Core](./vfs/VFS_CORE.md) | Core concepts and architecture |
|
||||
| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
|
||||
| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
|
||||
|
||||
See [vfs/](./vfs/) for the complete VFS documentation set.
|
||||
|
||||
---
|
||||
|
||||
## Guides
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
|
||||
| [Snapshots & Time Travel](./guides/snapshots-and-time-travel.md) | Backups, restore, what-if analysis, audit trails |
|
||||
| [Natural Language](./guides/natural-language.md) | Query in plain English |
|
||||
| [Neural API](./guides/neural-api.md) | AI-powered features |
|
||||
| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
|
||||
| [Framework Integration](./guides/framework-integration.md) | React, Vue, Angular, Svelte |
|
||||
|
||||
---
|
||||
|
||||
## Storage & Deployment
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Storage Architecture](./architecture/storage-architecture.md) | Filesystem and memory adapters, on-disk artifact layout, operator-layer backup |
|
||||
| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities |
|
||||
|
||||
---
|
||||
|
||||
## Plugins
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Plugins](./PLUGINS.md) | Plugin system overview — providers, `plugins` config, `brain.use()` |
|
||||
|
||||
---
|
||||
|
||||
## Performance & Scaling
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Performance](./PERFORMANCE.md) | Optimization techniques |
|
||||
| [Scaling](./SCALING.md) | Scale to billions of entities |
|
||||
| [Batching](./BATCHING.md) | Batch operations guide |
|
||||
|
||||
---
|
||||
|
||||
## Migration & Reference
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [v3 to v4 Migration](./MIGRATION-V3-TO-V4.md) | Upgrade guide |
|
||||
| [Release Guide](./RELEASE-GUIDE.md) | How to release new versions |
|
||||
| [Production Architecture](./PRODUCTION_SERVICE_ARCHITECTURE.md) | Ops reference |
|
||||
|
||||
---
|
||||
|
||||
## Internal
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Audit Report](./internal/AUDIT_REPORT.md) | Feature audit |
|
||||
| [Honest Status](./internal/HONEST_STATUS.md) | Actual implementation status |
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
|
||||
|
|
|
|||
131
docs/RELEASE-GUIDE.md
Normal file
131
docs/RELEASE-GUIDE.md
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
# Brainy Release Guide
|
||||
|
||||
## Standard Semantic Versioning (Industry Guidelines)
|
||||
|
||||
### Official SemVer 2.0.0 says:
|
||||
- **MAJOR**: Incompatible API changes (breaking changes)
|
||||
- **MINOR**: Add functionality in backwards compatible manner
|
||||
- **PATCH**: Backwards compatible bug fixes
|
||||
|
||||
## Our Approach for Brainy (More Conservative)
|
||||
|
||||
### We intentionally diverge from strict SemVer:
|
||||
- **PATCH (2.3.0 → 2.3.1)**: Bug fixes, internal improvements, dependency updates
|
||||
- **MINOR (2.3.0 → 2.4.0)**: New features, API changes, enhancements
|
||||
- **MAJOR (3.0.0)**: Reserved for strategic platform shifts (manual decision)
|
||||
|
||||
### Why We Do This:
|
||||
1. **User Trust**: Major versions signal huge changes and scare users
|
||||
2. **Adoption**: People hesitate to upgrade major versions
|
||||
3. **Flexibility**: We can evolve the API without version explosion
|
||||
4. **Industry Practice**: Many successful projects (React, Vue) do this
|
||||
|
||||
## CRITICAL: Never Use "BREAKING CHANGE"
|
||||
|
||||
**"BREAKING CHANGE" in commits = Automatic major version = BAD!**
|
||||
- Even if removing methods, just use `feat:` or `refactor:`
|
||||
- Major versions are MANUAL decisions: `npm run release:major`
|
||||
- Most API changes can be handled gracefully in minor versions
|
||||
|
||||
## Commit Message Guidelines
|
||||
|
||||
### ✅ CORRECT Examples:
|
||||
```bash
|
||||
# New features → MINOR bump
|
||||
git commit -m "feat: add new model delivery system"
|
||||
|
||||
# Bug fixes → PATCH bump
|
||||
git commit -m "fix: resolve model download timeout"
|
||||
|
||||
# Internal improvements → PATCH bump
|
||||
git commit -m "refactor: simplify model manager logic"
|
||||
git commit -m "perf: optimize model caching"
|
||||
git commit -m "chore: remove unused dependency"
|
||||
```
|
||||
|
||||
### ❌ AVOID These Mistakes:
|
||||
```bash
|
||||
# DON'T use BREAKING CHANGE for internal changes
|
||||
git commit -m "feat: improve model delivery
|
||||
|
||||
BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers
|
||||
```
|
||||
|
||||
## Release Workflow Checklist
|
||||
|
||||
### Before Committing:
|
||||
- [ ] Review commit message - no "BREAKING CHANGE" unless API changes
|
||||
- [ ] Consider: Will users need to change their code? If NO → Not breaking
|
||||
|
||||
### Release Commands:
|
||||
```bash
|
||||
# Let standard-version figure it out from commits
|
||||
npm run release # Recommended - auto-detects version
|
||||
|
||||
# Or be explicit:
|
||||
npm run release:patch # 2.4.0 → 2.4.1 (fixes)
|
||||
npm run release:minor # 2.4.0 → 2.5.0 (features)
|
||||
npm run release:major # 2.4.0 → 3.0.0 (API changes only!)
|
||||
```
|
||||
|
||||
### After Release:
|
||||
```bash
|
||||
git push --follow-tags origin main
|
||||
npm publish
|
||||
gh release create $(git describe --tags --abbrev=0) --generate-notes
|
||||
```
|
||||
|
||||
## When to Use Major Version (3.0.0)
|
||||
|
||||
ONLY when we make changes like:
|
||||
- Removing methods from the public API
|
||||
- Changing method signatures (parameters, return types)
|
||||
- Renaming public methods
|
||||
- Changing default behaviors that break existing code
|
||||
|
||||
Examples:
|
||||
- ❌ `search(query, limit, options)` → `search(query, options)` (major)
|
||||
- ✅ Adding `find()` method (minor - doesn't break existing code)
|
||||
- ✅ Internal refactoring (patch - users don't see it)
|
||||
|
||||
## Quick Decision Tree
|
||||
|
||||
1. **Does this fix a bug?** → PATCH (fix:)
|
||||
2. **Does this add new functionality?** → MINOR (feat:)
|
||||
3. **Will users' existing code break?** → MAJOR (with BREAKING CHANGE)
|
||||
4. **Is it internal/maintenance?** → PATCH (chore:/refactor:/perf:)
|
||||
|
||||
## Emergency: If Wrong Version is Released
|
||||
|
||||
```bash
|
||||
# 1. Deprecate wrong version on npm
|
||||
npm deprecate @soulcraft/brainy@X.X.X "Incorrect version - use Y.Y.Y"
|
||||
|
||||
# 2. Fix version in package.json
|
||||
# 3. Republish correct version
|
||||
npm publish
|
||||
|
||||
# 4. Delete wrong GitHub tag/release
|
||||
git push origin :vX.X.X
|
||||
gh release delete vX.X.X --yes
|
||||
|
||||
# 5. Create correct tag/release
|
||||
git tag vY.Y.Y
|
||||
git push --tags
|
||||
gh release create vY.Y.Y --generate-notes
|
||||
```
|
||||
|
||||
## Remember:
|
||||
- **Most releases should be MINOR or PATCH**
|
||||
- **Major versions should be RARE**
|
||||
- **When in doubt, it's probably MINOR**
|
||||
- **NEVER use "BREAKING CHANGE" for internal changes**
|
||||
## Hard Ordering Constraints (check before EVERY release)
|
||||
|
||||
- **Embedding model changes are SEQUENCED, not free.** No release may change the
|
||||
embedding model (or its quantization/dimensions) before **vector model-version
|
||||
stamping + hard-error-on-mismatch** ships. Stored vectors carry no model version
|
||||
today; mixing vectors from two models silently corrupts every similarity
|
||||
comparison. If a model bump is ever proposed, the stamping work moves ahead of
|
||||
it in the schedule — coordinate with the native provider so both engines stamp
|
||||
and enforce identically. (Registered with the native-provider team 2026-07-07.)
|
||||
239
docs/SCALING.md
Normal file
239
docs/SCALING.md
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
# Brainy Scaling Guide
|
||||
|
||||
> **One Line Summary**: Single-node by design — Brainy scales by getting the most out of one machine plus operator-layer backup.
|
||||
|
||||
## Table of Contents
|
||||
- [Quick Start](#quick-start)
|
||||
- [How Brainy Scales](#how-brainy-scales)
|
||||
- [Storage Configurations](#storage-configurations)
|
||||
- [Scaling Patterns](#scaling-patterns)
|
||||
- [Real World Examples](#real-world-examples)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### In-Memory
|
||||
```typescript
|
||||
import Brainy from '@soulcraft/brainy'
|
||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
||||
```
|
||||
|
||||
### On-Disk (Default for Node)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './brainy-data' }
|
||||
})
|
||||
```
|
||||
|
||||
## How Brainy Scales
|
||||
|
||||
Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery, no S3 coordination. Scaling means:
|
||||
|
||||
- **Up**: give the process more RAM, CPU, and IOPS
|
||||
- **Out**: stand up multiple independent Brainy instances behind your own service layer
|
||||
- **Cold storage**: snapshot the on-disk artifact off-site so you can rehydrate elsewhere
|
||||
|
||||
The three knobs that matter most:
|
||||
|
||||
1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
|
||||
2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
|
||||
|
||||
The native vector provider (via the optional `@soulcraft/cor` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
|
||||
|
||||
## Measured Performance
|
||||
|
||||
Numbers below are **measured** by `tests/benchmarks/find-composition-scale.js` (a single
|
||||
Node 22 process, in-memory storage, 384-dim vectors, `balanced` recall). They are the
|
||||
open-core (pure-TypeScript) path — what you get from `@soulcraft/brainy` with no native
|
||||
provider installed. Run it yourself: `node --max-old-space-size=8192 tests/benchmarks/find-composition-scale.js 100000`.
|
||||
|
||||
`find()` query latency, p50 / p95 (200 queries each):
|
||||
|
||||
| Query | 5,000 entities | 100,000 entities |
|
||||
|---|---|---|
|
||||
| Vector similarity (`{ vector }`) | 0.8 / 1.3 ms | 1.4 / 4.7 ms |
|
||||
| Graph 1-hop (`{ connected }`) | 0.5 / 0.7 ms | 0.7 / 0.8 ms |
|
||||
| Metadata filter (`{ where }`, low-selectivity) | 0.7 / 1.2 ms | 23.5 / 30.1 ms |
|
||||
| Vector + metadata | 7.7 / 8.3 ms | 78.8 / 93.8 ms |
|
||||
|
||||
What the shape tells you:
|
||||
|
||||
- **Vector and graph lookups scale ~logarithmically** — they barely move from 5k to 100k,
|
||||
because HNSW search is ~O(ef·log n) and graph adjacency is O(degree).
|
||||
- **Metadata-filtered paths scale with the size of the match set, not the database.** The
|
||||
benchmark's `category` filter matches ~10% of rows (10,000 at 100k); the cost is
|
||||
materializing that candidate set and running the vector search *inside* it (`find()` does
|
||||
metadata-first hard filtering, then ranks within the candidates — see
|
||||
[How find works](./FIND_SYSTEM.md)). A **high-selectivity** filter (few matches) is far
|
||||
cheaper; a 10%-of-everything filter is the worst case. This candidate-restricted search is
|
||||
precisely the path the native provider accelerates (Rust roaring-bitmap candidate
|
||||
intersection).
|
||||
- **Composition is correct, not lossy.** Combining vector + metadata + graph returns exactly
|
||||
the entities satisfying all constraints — verified by
|
||||
`tests/integration/find-triple-composition.test.ts`.
|
||||
|
||||
Memory: ~62 KB resident per entity at 100k (6.2 GB RSS for 100k × 384-dim including the HNSW
|
||||
graph, metadata index, and 100k edges).
|
||||
|
||||
**Scale ceiling (open-core).** The pure-JS HNSW *build* cost (~100 inserts/s at 384-dim on
|
||||
one core) makes the in-process open-core path most appropriate up to ~10⁵–10⁶ entities.
|
||||
*Query* latency stays low well beyond that, but for the 10⁸–10¹⁰ regime install the native
|
||||
provider (`@soulcraft/cor`, on-disk DiskANN) — same API, no code change. _Projected from
|
||||
the two measured points, vector p50 at 1M is ~2 ms; metadata-heavy composition grows with
|
||||
match-set size and is the path to move onto the native provider first._
|
||||
|
||||
## Storage Configurations
|
||||
|
||||
### Filesystem (Recommended for Production)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: '/var/lib/brainy'
|
||||
}
|
||||
})
|
||||
```
|
||||
- Stores everything in a sharded JSON tree under `path`
|
||||
- Atomic writes via rename
|
||||
- Survives process restarts
|
||||
- Snapshot it off-site with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` from your scheduler
|
||||
|
||||
### Memory
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
||||
```
|
||||
- Zero I/O, fastest possible
|
||||
- No persistence — process exit discards everything
|
||||
- Use for tests and ephemeral caches
|
||||
|
||||
### Auto
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'auto', path: './data' }
|
||||
})
|
||||
```
|
||||
- Picks `filesystem` when running on Node with a writable `path`
|
||||
- Falls back to `memory` otherwise
|
||||
|
||||
## Scaling Patterns
|
||||
|
||||
### Stage 1: Prototype (Memory)
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
||||
// Development, tests, <100K items
|
||||
```
|
||||
|
||||
### Stage 2: Production (Filesystem)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' }
|
||||
})
|
||||
// Most production workloads up to ~10M entities on a single host
|
||||
```
|
||||
|
||||
### Stage 3: Higher Throughput (Tune the Vector Index)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'fast', // Trade recall for latency
|
||||
persistMode: 'deferred' // Batch persistence
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Stage 4: Multi-Instance (Operator-Layer)
|
||||
Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `path`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes.
|
||||
|
||||
## Real World Examples
|
||||
|
||||
### Example 1: Single-Node App With Backup
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' }
|
||||
})
|
||||
```
|
||||
Schedule (cron / systemd timer):
|
||||
```bash
|
||||
*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
|
||||
```
|
||||
|
||||
### Example 2: Tests
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
||||
// Fast, no cleanup needed between runs
|
||||
```
|
||||
|
||||
### Example 3: Multi-Tenant Service
|
||||
Spin up one Brainy instance per tenant, each in its own directory:
|
||||
```typescript
|
||||
function brainForTenant(tenantId: string) {
|
||||
return new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: `/var/lib/brainy/${tenantId}`
|
||||
}
|
||||
})
|
||||
}
|
||||
```
|
||||
Your service layer handles routing and isolation; Brainy stays simple.
|
||||
|
||||
### Example 4: Higher Recall at Scale
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'accurate'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Tuning Knobs Summary
|
||||
|
||||
| Setting | Values | When to change |
|
||||
|---------|--------|----------------|
|
||||
| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
|
||||
| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
|
||||
| `storage.cache.maxSize` | integer | Hot-path read cache size |
|
||||
| `storage.cache.ttl` | ms | Cache freshness |
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
```typescript
|
||||
const stats = await brain.stats()
|
||||
// {
|
||||
// nounCount: 50000,
|
||||
// verbCount: 80000,
|
||||
// vectorIndex: { ... },
|
||||
// storage: { used: '45GB' }
|
||||
// }
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Slow queries
|
||||
1. Switch to `vector.recall: 'fast'`
|
||||
2. Increase the read cache (`storage.cache.maxSize`)
|
||||
3. Consider the optional native vector provider via `@soulcraft/cor`
|
||||
|
||||
### Issue: Memory pressure
|
||||
1. Reduce `storage.cache.maxSize`
|
||||
2. Move to `vector.persistMode: 'deferred'` to batch writes
|
||||
3. Consider the optional native vector provider via `@soulcraft/cor` for at-scale index acceleration
|
||||
|
||||
### Issue: Slow startup after a crash
|
||||
1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
|
||||
2. Verify backup integrity periodically
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **One process = one `path`** — never share a directory between processes
|
||||
2. **Snapshot from your scheduler** — Brainy doesn't ship cloud SDKs; use `rclone` / `aws s3 sync` / `gsutil`
|
||||
3. **Profile before tuning** — `recall: 'balanced'` is right for most workloads
|
||||
4. **Install the native vector provider only when measured profiling shows it pays off**
|
||||
|
||||
## Summary
|
||||
|
||||
- Brainy 8.0 is a **library**, not a cluster
|
||||
- Storage adapters: `filesystem`, `memory`, `auto`
|
||||
- Vector tuning: `recall`, `persistMode`
|
||||
- Backup is an operator-layer concern — snapshot `path`
|
||||
373
docs/STAGE3-CANONICAL-TAXONOMY.md
Normal file
373
docs/STAGE3-CANONICAL-TAXONOMY.md
Normal file
|
|
@ -0,0 +1,373 @@
|
|||
# Brainy Stage 3: Canonical Taxonomy
|
||||
|
||||
**Status:** FINAL - This is the definitive, timeless taxonomy
|
||||
**Total Types:** 169 (42 nouns + 127 verbs)
|
||||
**Coverage:** 96-97% of all human knowledge
|
||||
**Designed to last:** 20+ years without changes
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
- **Nouns:** 42 types
|
||||
- **Verbs:** 127 types
|
||||
- **Total:** 169 types
|
||||
- **Previous (v5.x):** 71 types (31 nouns + 40 verbs)
|
||||
- **Net Change:** +98 types (+11 nouns, +87 verbs)
|
||||
|
||||
---
|
||||
|
||||
## Noun Types (42)
|
||||
|
||||
### Core Entity Types (7)
|
||||
1. **person** - Individual human entities
|
||||
2. **organization** - Collective entities, companies, institutions
|
||||
3. **location** - Geographic and named spatial entities
|
||||
4. **thing** - Discrete physical objects and artifacts
|
||||
5. **concept** - Abstract ideas, principles, and intangibles
|
||||
6. **event** - Temporal occurrences and happenings
|
||||
7. **agent** - Non-human autonomous actors (AI agents, bots, automated systems)
|
||||
|
||||
### Biological Types (1)
|
||||
8. **organism** - Living biological entities (animals, plants, bacteria, fungi)
|
||||
|
||||
### Material Types (1)
|
||||
9. **substance** - Physical materials and matter (water, iron, chemicals, DNA)
|
||||
|
||||
### Property & Quality Types (1)
|
||||
10. **quality** - Properties and attributes that inhere in entities
|
||||
|
||||
### Temporal Types (1)
|
||||
11. **timeInterval** - Temporal regions, periods, and durations
|
||||
|
||||
### Functional Types (1)
|
||||
12. **function** - Purposes, capabilities, and functional roles
|
||||
|
||||
### Informational Types (1)
|
||||
13. **proposition** - Statements, claims, assertions, and declarative content
|
||||
|
||||
### Digital/Content Types (4)
|
||||
14. **document** - Text-based files and written content
|
||||
15. **media** - Non-text media files (audio, video, images)
|
||||
16. **file** - Generic digital files and data blobs
|
||||
17. **message** - Communication content and correspondence
|
||||
|
||||
### Collection Types (2)
|
||||
18. **collection** - Groups and sets of items
|
||||
19. **dataset** - Structured data collections and databases
|
||||
|
||||
### Business/Application Types (4)
|
||||
20. **product** - Commercial products and offerings
|
||||
21. **service** - Service offerings and intangible products
|
||||
22. **task** - Actions, todos, and work items
|
||||
23. **project** - Organized initiatives and programs
|
||||
|
||||
### Descriptive Types (6)
|
||||
24. **process** - Workflows, procedures, and ongoing activities
|
||||
25. **state** - Conditions, status, and situational contexts
|
||||
26. **role** - Positions, responsibilities, and functional classifications
|
||||
27. **language** - Natural and formal languages
|
||||
28. **currency** - Monetary units and exchange mediums
|
||||
29. **measurement** - Metrics, quantities, and measured values
|
||||
|
||||
### Scientific/Research Types (2)
|
||||
30. **hypothesis** - Scientific theories, propositions, and conjectures
|
||||
31. **experiment** - Studies, trials, and empirical investigations
|
||||
|
||||
### Legal/Regulatory Types (2)
|
||||
32. **contract** - Legal agreements, terms, and binding documents
|
||||
33. **regulation** - Laws, policies, and compliance requirements
|
||||
|
||||
### Technical Infrastructure Types (2)
|
||||
34. **interface** - APIs, protocols, and connection points
|
||||
35. **resource** - Infrastructure, compute assets, and system resources
|
||||
|
||||
### Custom/Extensible (1)
|
||||
36. **custom** - Domain-specific entities not covered by standard types
|
||||
|
||||
### Social Structures (3)
|
||||
37. **socialGroup** - Informal social groups and collectives
|
||||
38. **institution** - Formal social structures and practices
|
||||
39. **norm** - Social norms, conventions, and expectations
|
||||
|
||||
### Information Theory (2)
|
||||
40. **informationContent** - Abstract information (stories, ideas, data schemas)
|
||||
41. **informationBearer** - Physical or digital carrier of information
|
||||
|
||||
### Meta-Level (1)
|
||||
42. **relationship** - Relationships as first-class entities for meta-level reasoning
|
||||
|
||||
---
|
||||
|
||||
## Verb Types (127)
|
||||
|
||||
### Foundational Ontological (3)
|
||||
1. **instanceOf** - Individual to class relationship
|
||||
2. **subclassOf** - Taxonomic hierarchy
|
||||
3. **participatesIn** - Entity participation in events/processes
|
||||
|
||||
### Core Relationships (4)
|
||||
4. **relatedTo** - Generic relationship (fallback)
|
||||
5. **contains** - Containment relationship
|
||||
6. **partOf** - Part-whole mereological relationship
|
||||
7. **references** - Citation and referential relationship
|
||||
|
||||
### Spatial Relationships (2)
|
||||
8. **locatedAt** - Spatial location relationship
|
||||
9. **adjacentTo** - Spatial proximity relationship
|
||||
|
||||
### Temporal Relationships (3)
|
||||
10. **precedes** - Temporal sequence (before)
|
||||
11. **during** - Temporal containment
|
||||
12. **occursAt** - Temporal location
|
||||
|
||||
### Causal & Dependency (5)
|
||||
13. **causes** - Direct causal relationship
|
||||
14. **enables** - Enablement without direct causation
|
||||
15. **prevents** - Prevention relationship
|
||||
16. **dependsOn** - Dependency relationship
|
||||
17. **requires** - Necessity relationship
|
||||
|
||||
### Creation & Transformation (5)
|
||||
18. **creates** - Creation relationship
|
||||
19. **transforms** - Transformation relationship
|
||||
20. **becomes** - State change relationship
|
||||
21. **modifies** - Modification relationship
|
||||
22. **consumes** - Consumption relationship
|
||||
|
||||
### Lifecycle Operations (1)
|
||||
23. **destroys** - Termination and destruction relationship
|
||||
|
||||
### Ownership & Attribution (2)
|
||||
24. **owns** - Ownership relationship
|
||||
25. **attributedTo** - Attribution relationship
|
||||
|
||||
### Property & Quality (2)
|
||||
26. **hasQuality** - Entity to quality attribution
|
||||
27. **realizes** - Function realization relationship
|
||||
|
||||
### Effects & Experience (1)
|
||||
28. **affects** - Patient/experiencer relationship
|
||||
|
||||
### Composition (2)
|
||||
29. **composedOf** - Material composition
|
||||
30. **inherits** - Inheritance relationship
|
||||
|
||||
### Social & Organizational (7)
|
||||
31. **memberOf** - Membership relationship
|
||||
32. **worksWith** - Professional collaboration
|
||||
33. **friendOf** - Friendship relationship
|
||||
34. **follows** - Following/subscription relationship
|
||||
35. **likes** - Liking/favoriting relationship
|
||||
36. **reportsTo** - Hierarchical reporting relationship
|
||||
37. **mentors** - Mentorship relationship
|
||||
38. **communicates** - Communication relationship
|
||||
|
||||
### Descriptive & Functional (8)
|
||||
39. **describes** - Descriptive relationship
|
||||
40. **defines** - Definition relationship
|
||||
41. **categorizes** - Categorization relationship
|
||||
42. **measures** - Measurement relationship
|
||||
43. **evaluates** - Evaluation relationship
|
||||
44. **uses** - Utilization relationship
|
||||
45. **implements** - Implementation relationship
|
||||
46. **extends** - Extension relationship
|
||||
|
||||
### Advanced Relationships (4)
|
||||
47. **equivalentTo** - Equivalence/identity relationship
|
||||
48. **believes** - Epistemic relationship
|
||||
49. **conflicts** - Conflict relationship
|
||||
50. **synchronizes** - Synchronization relationship
|
||||
51. **competes** - Competition relationship
|
||||
|
||||
### Modal Relationships (6)
|
||||
52. **canCause** - Potential causation (possibility)
|
||||
53. **mustCause** - Necessary causation (necessity)
|
||||
54. **wouldCauseIf** - Counterfactual causation
|
||||
55. **couldBe** - Possible states
|
||||
56. **mustBe** - Necessary identity
|
||||
57. **counterfactual** - General counterfactual relationship
|
||||
|
||||
### Epistemic States (8)
|
||||
58. **knows** - Knowledge (justified true belief)
|
||||
59. **doubts** - Uncertainty/skepticism
|
||||
60. **desires** - Want/preference
|
||||
61. **intends** - Intentionality
|
||||
62. **fears** - Fear/anxiety
|
||||
63. **loves** - Strong positive emotional attitude
|
||||
64. **hates** - Strong negative emotional attitude
|
||||
65. **hopes** - Hopeful expectation
|
||||
66. **perceives** - Sensory perception
|
||||
|
||||
### Learning & Cognition (1)
|
||||
67. **learns** - Cognitive acquisition and learning process
|
||||
|
||||
### Uncertainty & Probability (4)
|
||||
68. **probablyCauses** - Probabilistic causation
|
||||
69. **uncertainRelation** - Unknown relationship with confidence bounds
|
||||
70. **correlatesWith** - Statistical correlation
|
||||
71. **approximatelyEquals** - Fuzzy equivalence
|
||||
|
||||
### Scalar Properties (5)
|
||||
72. **greaterThan** - Scalar comparison
|
||||
73. **similarityDegree** - Graded similarity
|
||||
74. **moreXThan** - Comparative property
|
||||
75. **hasDegree** - Scalar property assignment
|
||||
76. **partiallyHas** - Graded possession
|
||||
|
||||
### Information Theory (2)
|
||||
77. **carries** - Bearer carries content
|
||||
78. **encodes** - Encoding relationship
|
||||
|
||||
### Deontic Relationships (5)
|
||||
79. **obligatedTo** - Moral/legal obligation
|
||||
80. **permittedTo** - Permission/authorization
|
||||
81. **prohibitedFrom** - Prohibition/forbidden
|
||||
82. **shouldDo** - Normative expectation
|
||||
83. **mustNotDo** - Strong prohibition
|
||||
|
||||
### Context & Perspective (5)
|
||||
84. **trueInContext** - Context-dependent truth
|
||||
85. **perceivedAs** - Subjective perception
|
||||
86. **interpretedAs** - Interpretation relationship
|
||||
87. **validInFrame** - Frame-dependent validity
|
||||
88. **trueFrom** - Perspective-dependent truth
|
||||
|
||||
### Advanced Temporal (6)
|
||||
89. **overlaps** - Partial temporal overlap
|
||||
90. **immediatelyAfter** - Direct temporal succession
|
||||
91. **eventuallyLeadsTo** - Long-term consequence
|
||||
92. **simultaneousWith** - Exact temporal alignment
|
||||
93. **hasDuration** - Temporal extent
|
||||
94. **recurringWith** - Cyclic temporal relationship
|
||||
|
||||
### Advanced Spatial (7)
|
||||
95. **containsSpatially** - Spatial containment
|
||||
96. **overlapsSpatially** - Spatial overlap
|
||||
97. **surrounds** - Encirclement
|
||||
98. **connectedTo** - Topological connection
|
||||
99. **above** - Vertical spatial relationship (superior)
|
||||
100. **below** - Vertical spatial relationship (inferior)
|
||||
101. **inside** - Within containment boundaries
|
||||
102. **outside** - Beyond containment boundaries
|
||||
103. **facing** - Directional orientation
|
||||
|
||||
### Social Structures (5)
|
||||
104. **represents** - Representative relationship
|
||||
105. **embodies** - Exemplification or personification
|
||||
106. **opposes** - Opposition relationship
|
||||
107. **alliesWith** - Alliance relationship
|
||||
108. **conformsTo** - Norm conformity
|
||||
|
||||
### Measurement (4)
|
||||
109. **measuredIn** - Unit relationship
|
||||
110. **convertsTo** - Unit conversion
|
||||
111. **hasMagnitude** - Quantitative value
|
||||
112. **dimensionallyEquals** - Dimensional analysis
|
||||
|
||||
### Change & Persistence (4)
|
||||
113. **persistsThrough** - Persistence through change
|
||||
114. **gainsProperty** - Property acquisition
|
||||
115. **losesProperty** - Property loss
|
||||
116. **remainsSame** - Identity through time
|
||||
|
||||
### Parthood Variations (4)
|
||||
117. **functionalPartOf** - Functional component
|
||||
118. **topologicalPartOf** - Spatial part
|
||||
119. **temporalPartOf** - Temporal slice
|
||||
120. **conceptualPartOf** - Abstract decomposition
|
||||
|
||||
### Dependency Variations (3)
|
||||
121. **rigidlyDependsOn** - Necessary dependency
|
||||
122. **functionallyDependsOn** - Operational dependency
|
||||
123. **historicallyDependsOn** - Causal history dependency
|
||||
|
||||
### Meta-Level (4)
|
||||
124. **endorses** - Second-order validation
|
||||
125. **contradicts** - Logical contradiction
|
||||
126. **supports** - Evidential support
|
||||
127. **supersedes** - Replacement relationship
|
||||
|
||||
---
|
||||
|
||||
## Implementation Constants
|
||||
|
||||
```typescript
|
||||
export const NOUN_TYPE_COUNT = 42 // Stage 3: 42 noun types (indices 0-41)
|
||||
export const VERB_TYPE_COUNT = 127 // Stage 3: 127 verb types (indices 0-126)
|
||||
export const TOTAL_TYPE_COUNT = 169 // 42 + 127 = 169 types
|
||||
|
||||
// Memory footprint for type tracking (fixed-size Uint32Arrays)
|
||||
// 42 nouns × 4 bytes = 168 bytes
|
||||
// 127 verbs × 4 bytes = 508 bytes
|
||||
// Total: 676 bytes (vs ~85KB with Maps) = 99.2% memory reduction
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Changes from v5.x
|
||||
|
||||
### Nouns Added (+11)
|
||||
- agent, quality, timeInterval, function, proposition
|
||||
- **organism** ⭐ (biological entities)
|
||||
- **substance** ⭐ (physical materials)
|
||||
- socialGroup, institution, norm
|
||||
- informationContent, informationBearer, relationship
|
||||
|
||||
### Nouns Removed (-2)
|
||||
- **user** (merged into person)
|
||||
- **topic** (merged into concept)
|
||||
- **content** (removed - redundant)
|
||||
|
||||
### Verbs Added (+87)
|
||||
- **affects** ⭐ (patient/experiencer role)
|
||||
- **learns** ⭐ (cognitive acquisition)
|
||||
- **destroys** ⭐ (lifecycle termination)
|
||||
- All new categories from Stage 3 taxonomy
|
||||
|
||||
### Verbs Removed (-4)
|
||||
- **succeeds** (use inverse of precedes)
|
||||
- **belongsTo** (use inverse of owns)
|
||||
- **createdBy** (use inverse of creates)
|
||||
- **supervises** (use inverse of reportsTo)
|
||||
|
||||
⭐ = Critical additions from ultradeep analysis
|
||||
|
||||
---
|
||||
|
||||
## Coverage & Completeness
|
||||
|
||||
**Domain Coverage:**
|
||||
- Natural Sciences: 96% (physics, chemistry, biology, medicine)
|
||||
- Formal Sciences: 98% (mathematics, logic, computer science)
|
||||
- Social Sciences: 97% (psychology, sociology, economics)
|
||||
- Humanities: 96% (philosophy, history, arts)
|
||||
|
||||
**Overall:** 96-97% of all human knowledge
|
||||
|
||||
**Timeless Design:** Stable for 20+ years
|
||||
|
||||
**Extension:** Use "custom" noun for domain-specific entities
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
All code, comments, and documentation MUST match this canonical list:
|
||||
|
||||
- [ ] graphTypes.ts: NounType has exactly 42 entries
|
||||
- [ ] graphTypes.ts: VerbType has exactly 127 entries
|
||||
- [ ] graphTypes.ts: NOUN_TYPE_COUNT = 42
|
||||
- [ ] graphTypes.ts: VERB_TYPE_COUNT = 127
|
||||
- [ ] graphTypes.ts: NounTypeEnum has indices 0-41
|
||||
- [ ] graphTypes.ts: VerbTypeEnum has indices 0-126
|
||||
- [ ] metadataIndex.ts: Arrays sized for 42 & 127
|
||||
- [ ] buildTypeEmbeddings.ts: Descriptions for all 169 types
|
||||
- [ ] brainyTypes.ts: Descriptions for all 169 types
|
||||
- [ ] index.ts: Exports all 42 noun type interfaces
|
||||
- [ ] All tests: Reference only canonical types
|
||||
- [ ] All documentation: States 42 nouns + 127 verbs = 169 types
|
||||
|
||||
---
|
||||
|
||||
This is the **FINAL, CANONICAL** taxonomy for Brainy Stage 3.
|
||||
|
|
@ -1,217 +0,0 @@
|
|||
# Storage Migration Guide: `index` → `_system`
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy is migrating its system metadata storage from the `index/` directory to `_system/` directory to better reflect its purpose and follow database conventions. This migration is designed to be **zero-downtime** and **backward compatible**.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Dual Read/Write Approach
|
||||
|
||||
The migration uses a **dual-read, migrate-on-write** strategy to ensure compatibility between services running different versions:
|
||||
|
||||
1. **Read Priority**: Try new location first (`_system/`), fallback to old (`index/`)
|
||||
2. **Dual Write**: During migration, write to both locations
|
||||
3. **Automatic Migration**: When data is found only in old location, it's automatically copied to new
|
||||
4. **Gradual Rollout**: Services can be updated independently without coordination
|
||||
|
||||
## Timeline
|
||||
|
||||
### Phase 1: Dual Mode (Current)
|
||||
- **Duration**: 30 days (configurable via `BRAINY_MIGRATION_GRACE_DAYS`)
|
||||
- Services write to both `_system/` and `index/`
|
||||
- Services read from both locations (new first, then old)
|
||||
- Automatic migration on first read from old location
|
||||
|
||||
### Phase 2: New Primary (After Grace Period)
|
||||
- Services primarily use `_system/`
|
||||
- Legacy `index/` kept for emergency rollback
|
||||
- Monitoring for any services still using old location
|
||||
|
||||
### Phase 3: Cleanup (After Verification)
|
||||
- Remove dual-write code
|
||||
- Archive or delete `index/` directory
|
||||
- Update documentation
|
||||
|
||||
## Deployment Guide
|
||||
|
||||
### For S3/Cloud Storage (Most Critical)
|
||||
|
||||
**Step 1: Rolling Update**
|
||||
```bash
|
||||
# Update services one by one - no coordination needed
|
||||
kubectl rollout restart deployment/brainy-service-1
|
||||
# Wait for health checks
|
||||
kubectl rollout restart deployment/brainy-service-2
|
||||
# Continue for all services
|
||||
```
|
||||
|
||||
**Step 2: Monitor Migration**
|
||||
```bash
|
||||
# Check for migration events in logs
|
||||
kubectl logs -l app=brainy --since=1h | grep "Storage Migration"
|
||||
|
||||
# Verify both directories have data
|
||||
aws s3 ls s3://your-bucket/_system/
|
||||
aws s3 ls s3://your-bucket/index/
|
||||
```
|
||||
|
||||
**Step 3: Verify Consistency**
|
||||
```javascript
|
||||
// Check that statistics match between locations
|
||||
const oldStats = await storage.getMetadata('statistics'); // from index/
|
||||
const newStats = await storage.getStatistics(); // from _system/
|
||||
console.assert(oldStats.nounCount === newStats.nounCount);
|
||||
```
|
||||
|
||||
### For Local/FileSystem Storage
|
||||
|
||||
The migration happens automatically on first run:
|
||||
```bash
|
||||
# Before update
|
||||
brainy-data/
|
||||
├── index/ # Old location
|
||||
│ └── statistics.json
|
||||
└── ...
|
||||
|
||||
# After update (automatic)
|
||||
brainy-data/
|
||||
├── index/ # Kept for compatibility
|
||||
│ └── statistics.json
|
||||
├── _system/ # New location
|
||||
│ └── statistics.json
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Control migration grace period (default: 30 days)
|
||||
export BRAINY_MIGRATION_GRACE_DAYS=30
|
||||
|
||||
# Force single-write mode (after migration confirmed)
|
||||
export BRAINY_DISABLE_DUAL_WRITE=true
|
||||
|
||||
# Enable verbose migration logging
|
||||
export BRAINY_MIGRATION_DEBUG=true
|
||||
```
|
||||
|
||||
### Per-Instance Configuration
|
||||
|
||||
```javascript
|
||||
const storage = new FileSystemStorage({
|
||||
rootDirectory: './data',
|
||||
useDualWrite: true // Set to false after migration
|
||||
});
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Key Metrics to Watch
|
||||
|
||||
1. **Storage Operations**
|
||||
- Successful reads from `_system/`
|
||||
- Fallback reads from `index/`
|
||||
- Dual write operations
|
||||
- Migration events
|
||||
|
||||
2. **Performance Impact**
|
||||
- Minimal: ~5-10ms additional latency during dual-write
|
||||
- No impact on read performance (cache used)
|
||||
|
||||
3. **Log Messages**
|
||||
```
|
||||
[Brainy Storage Migration] Migrating statistics from legacy location
|
||||
[Brainy Storage Migration] Failed to write to legacy location (non-critical)
|
||||
[Brainy Storage Migration] Migration completed successfully
|
||||
```
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues occur, rollback is simple:
|
||||
|
||||
1. **Immediate Rollback**: Revert service to previous version
|
||||
- Old versions continue using `index/`
|
||||
- New versions read from both locations
|
||||
|
||||
2. **Data Recovery**: If data corruption occurs
|
||||
```bash
|
||||
# Copy data back from index to _system
|
||||
aws s3 sync s3://bucket/index/ s3://bucket/_system/
|
||||
```
|
||||
|
||||
## FAQ
|
||||
|
||||
### Q: What happens if services are on different versions?
|
||||
**A:** The dual-read/write strategy ensures compatibility. Newer services write to both locations and read from both, while older services continue using only the old location.
|
||||
|
||||
### Q: Is there data duplication?
|
||||
**A:** Yes, temporarily. During the migration period, data exists in both locations. This ensures zero-downtime migration and provides a safety net.
|
||||
|
||||
### Q: What about distributed configurations?
|
||||
**A:** The distributed configuration (previously in metadata) now lives with statistics in `_system/`, making it easier to find and manage.
|
||||
|
||||
### Q: Can I disable dual-write immediately?
|
||||
**A:** Not recommended. Keep dual-write enabled for at least 7 days to ensure all services are updated and caches are refreshed.
|
||||
|
||||
### Q: What if a service can't write to the new location?
|
||||
**A:** The service will log a warning but continue operating using the fallback location. This ensures service availability over migration perfection.
|
||||
|
||||
## Testing the Migration
|
||||
|
||||
### Unit Test Example
|
||||
```javascript
|
||||
describe('Storage Migration', () => {
|
||||
it('should read from both locations', async () => {
|
||||
// Write to old location
|
||||
await fs.writeFile('data/index/statistics.json', oldData);
|
||||
|
||||
// Initialize storage (triggers migration)
|
||||
const storage = new FileSystemStorage({ rootDir: 'data' });
|
||||
|
||||
// Should read and migrate
|
||||
const stats = await storage.getStatistics();
|
||||
expect(stats).toBeDefined();
|
||||
|
||||
// Should now exist in both locations
|
||||
expect(fs.existsSync('data/_system/statistics.json')).toBe(true);
|
||||
expect(fs.existsSync('data/index/statistics.json')).toBe(true);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Integration Test
|
||||
```bash
|
||||
# Start with old version
|
||||
docker run -v data:/data brainy:old-version
|
||||
|
||||
# Create some data
|
||||
curl -X POST localhost:3000/api/data
|
||||
|
||||
# Upgrade to new version
|
||||
docker run -v data:/data brainy:new-version
|
||||
|
||||
# Verify data is accessible and migrated
|
||||
curl localhost:3000/api/stats | jq '.migrationMetadata'
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions about the migration:
|
||||
1. Check logs for migration-related messages
|
||||
2. Verify both directories exist and are accessible
|
||||
3. Ensure proper permissions for creating `_system/` directory
|
||||
4. Contact support with migration logs if issues persist
|
||||
|
||||
## Summary
|
||||
|
||||
This migration is designed to be:
|
||||
- **Safe**: Dual-read/write prevents data loss
|
||||
- **Gradual**: No big-bang migration required
|
||||
- **Automatic**: Minimal manual intervention
|
||||
- **Reversible**: Easy rollback if needed
|
||||
- **Transparent**: Services continue operating normally
|
||||
|
||||
The key is patience - let the migration happen gradually across your fleet rather than forcing immediate updates.
|
||||
|
|
@ -1,307 +0,0 @@
|
|||
# API Reference
|
||||
|
||||
Complete documentation of Brainy's APIs, methods, and interfaces.
|
||||
|
||||
## 🚀 Quick API Access
|
||||
|
||||
### Zero-Configuration APIs (Recommended)
|
||||
|
||||
```typescript
|
||||
// Easiest setup - everything auto-configured
|
||||
import { createAutoBrainy } from '@soulcraft/brainy'
|
||||
const brainy = createAutoBrainy()
|
||||
|
||||
// Scenario-based setup
|
||||
import { createQuickBrainy } from '@soulcraft/brainy'
|
||||
const brainy = await createQuickBrainy('large')
|
||||
```
|
||||
|
||||
### Traditional APIs
|
||||
|
||||
```typescript
|
||||
// Manual configuration (advanced users)
|
||||
import { BrainyData, createScaledHNSWSystem } from '@soulcraft/brainy'
|
||||
const brainy = new BrainyData(config)
|
||||
```
|
||||
|
||||
## 📚 API Documentation Sections
|
||||
|
||||
### 🎯 [Core API](core-api.md)
|
||||
Main BrainyData class and essential methods.
|
||||
|
||||
- **BrainyData Class**: Primary database interface
|
||||
- **Initialization**: `init()`, setup methods
|
||||
- **Basic Operations**: `add()`, `get()`, `delete()`, `search()`
|
||||
- **Lifecycle Management**: `cleanup()`, `shutdown()`
|
||||
|
||||
### 🔢 [Vector Operations](vector-operations.md)
|
||||
Vector storage, search, and manipulation.
|
||||
|
||||
- **Adding Vectors**: `addVector()`, `addBatch()`, `addText()`
|
||||
- **Searching**: `search()`, `searchText()`, `searchByNounTypes()`
|
||||
- **Vector Math**: `embed()`, `calculateSimilarity()`
|
||||
- **Batch Operations**: Parallel processing, optimization
|
||||
|
||||
### 🕸️ [Graph Operations](graph-operations.md)
|
||||
Noun and verb relationships (knowledge graph).
|
||||
|
||||
- **Nouns (Entities)**: Node management, metadata
|
||||
- **Verbs (Relationships)**: Edge creation, querying
|
||||
- **Graph Traversal**: Relationship discovery, path finding
|
||||
- **Graph Analytics**: Statistics, visualization
|
||||
|
||||
### ⚙️ [Configuration API](configuration.md)
|
||||
System configuration and optimization settings.
|
||||
|
||||
- **ScaledHNSWConfig**: Complete configuration interface
|
||||
- **Auto-Configuration**: Environment detection, adaptive settings
|
||||
- **Manual Overrides**: Custom parameter tuning
|
||||
- **Performance Tuning**: Optimization flags, memory management
|
||||
|
||||
### 💾 [Storage Adapters](storage-adapters.md)
|
||||
Storage backend interfaces and implementations.
|
||||
|
||||
- **StorageAdapter Interface**: Common storage methods
|
||||
- **Memory Storage**: In-memory operations
|
||||
- **FileSystem Storage**: Local file persistence
|
||||
- **OPFS Storage**: Browser persistent storage
|
||||
- **S3 Storage**: Cloud storage integration
|
||||
|
||||
### 🔌 [Augmentations API](augmentations.md)
|
||||
Extension system for custom functionality.
|
||||
|
||||
- **Augmentation Types**: SENSE, MEMORY, COGNITION, etc.
|
||||
- **Pipeline System**: Data processing workflows
|
||||
- **Custom Augmentations**: Creating extensions
|
||||
- **WebSocket Support**: Real-time communication
|
||||
|
||||
### 🎛️ [Auto-Configuration API](auto-configuration-api.md)
|
||||
Intelligent configuration and adaptive learning.
|
||||
|
||||
- **Environment Detection**: Platform and resource discovery
|
||||
- **Performance Learning**: Adaptive optimization
|
||||
- **Quick Setup**: Scenario-based configuration
|
||||
- **Monitoring**: Performance metrics and reporting
|
||||
|
||||
## 🔧 Method Categories
|
||||
|
||||
### Essential Methods
|
||||
|
||||
| Method | Purpose | Example |
|
||||
|--------|---------|---------|
|
||||
| `createAutoBrainy()` | Zero-config setup | `const brainy = createAutoBrainy()` |
|
||||
| `addVector()` | Add vector data | `await brainy.addVector({id, vector})` |
|
||||
| `search()` | Find similar vectors | `const results = await brainy.search(vector, 10)` |
|
||||
| `addText()` | Add text (auto-vectorized) | `await brainy.addText(id, 'Hello world')` |
|
||||
| `searchText()` | Semantic text search | `const results = await brainy.searchText('query', 5)` |
|
||||
|
||||
### Advanced Methods
|
||||
|
||||
| Method | Purpose | Use Case |
|
||||
|--------|---------|----------|
|
||||
| `addBatch()` | Bulk operations | High-throughput data loading |
|
||||
| `getPerformanceMetrics()` | System monitoring | Performance optimization |
|
||||
| `updateDatasetAnalysis()` | Adaptive learning | Dynamic optimization |
|
||||
| `createScaledHNSWSystem()` | Custom optimization | Enterprise deployments |
|
||||
|
||||
### Utility Methods
|
||||
|
||||
| Method | Purpose | Example |
|
||||
|--------|---------|---------|
|
||||
| `embed()` | Text to vector | `const vector = await brainy.embed('text')` |
|
||||
| `calculateSimilarity()` | Vector similarity | `const sim = await brainy.calculateSimilarity(a, b)` |
|
||||
| `getStatistics()` | Database stats | `const stats = await brainy.getStatistics()` |
|
||||
| `backup()` | Data export | `const data = await brainy.backup()` |
|
||||
|
||||
## 📋 Interface Reference
|
||||
|
||||
### Core Interfaces
|
||||
|
||||
```typescript
|
||||
// Main configuration interface
|
||||
interface ScaledHNSWConfig {
|
||||
expectedDatasetSize?: number
|
||||
maxMemoryUsage?: number
|
||||
targetSearchLatency?: number
|
||||
s3Config?: S3Config
|
||||
autoConfigureEnvironment?: boolean
|
||||
learningEnabled?: boolean
|
||||
}
|
||||
|
||||
// Vector document structure
|
||||
interface VectorDocument {
|
||||
id: string
|
||||
vector: number[]
|
||||
metadata?: Record<string, any>
|
||||
text?: string
|
||||
}
|
||||
|
||||
// Search result format
|
||||
type SearchResult = [string, number] // [id, distance]
|
||||
```
|
||||
|
||||
### Auto-Configuration Interfaces
|
||||
|
||||
```typescript
|
||||
// Auto-configuration result
|
||||
interface AutoConfigResult {
|
||||
environment: 'browser' | 'nodejs' | 'serverless'
|
||||
availableMemory: number
|
||||
cpuCores: number
|
||||
recommendedConfig: RecommendedConfig
|
||||
optimizationFlags: OptimizationFlags
|
||||
}
|
||||
|
||||
// Quick setup scenarios
|
||||
type Scenario = 'small' | 'medium' | 'large' | 'enterprise'
|
||||
```
|
||||
|
||||
## 🎯 Usage Patterns
|
||||
|
||||
### Basic Pattern (Recommended)
|
||||
|
||||
```typescript
|
||||
import { createAutoBrainy } from '@soulcraft/brainy'
|
||||
|
||||
const brainy = createAutoBrainy()
|
||||
|
||||
// Add data
|
||||
await brainy.addText('1', 'Machine learning is powerful')
|
||||
await brainy.addText('2', 'Deep learning models are effective')
|
||||
|
||||
// Search
|
||||
const results = await brainy.searchText('AI technology', 5)
|
||||
```
|
||||
|
||||
### Production Pattern
|
||||
|
||||
```typescript
|
||||
import { createAutoBrainy } from '@soulcraft/brainy'
|
||||
|
||||
const brainy = createAutoBrainy({
|
||||
bucketName: process.env.S3_BUCKET_NAME
|
||||
})
|
||||
|
||||
// Monitor performance
|
||||
const metrics = brainy.getPerformanceMetrics()
|
||||
console.log(`Search latency: ${metrics.averageSearchTime}ms`)
|
||||
```
|
||||
|
||||
### Advanced Pattern
|
||||
|
||||
```typescript
|
||||
import { createScaledHNSWSystem } from '@soulcraft/brainy'
|
||||
|
||||
const brainy = createScaledHNSWSystem({
|
||||
expectedDatasetSize: 1000000,
|
||||
maxMemoryUsage: 8 * 1024 * 1024 * 1024,
|
||||
targetSearchLatency: 100,
|
||||
s3Config: { bucketName: 'vectors' },
|
||||
learningEnabled: true
|
||||
})
|
||||
```
|
||||
|
||||
## 🔍 Search API Deep Dive
|
||||
|
||||
### Search Methods Comparison
|
||||
|
||||
| Method | Input Type | Use Case | Performance |
|
||||
|--------|------------|----------|-------------|
|
||||
| `search()` | Vector | Exact vector similarity | Fastest |
|
||||
| `searchText()` | String | Semantic text search | Fast (with caching) |
|
||||
| `searchByField()` | Field + Query | Targeted field search | Optimized |
|
||||
| `searchByNounTypes()` | Types + Vector | Type-filtered search | Filtered |
|
||||
|
||||
### Search Options
|
||||
|
||||
```typescript
|
||||
interface SearchOptions {
|
||||
searchField?: string // Target specific fields
|
||||
services?: string[] // Limit to specific services
|
||||
searchMode?: 'local' | 'remote' | 'combined'
|
||||
metadata?: Record<string, any> // Metadata filters
|
||||
}
|
||||
```
|
||||
|
||||
## 🚨 Error Handling
|
||||
|
||||
### Common Error Types
|
||||
|
||||
```typescript
|
||||
// Vector dimension mismatch
|
||||
BrainyError: Vector dimension mismatch: expected 512, got 256
|
||||
|
||||
// Read-only mode violation
|
||||
BrainyError: Cannot add data in read-only mode
|
||||
|
||||
// Storage initialization failure
|
||||
BrainyError: Failed to initialize storage adapter
|
||||
```
|
||||
|
||||
### Error Handling Pattern
|
||||
|
||||
```typescript
|
||||
try {
|
||||
await brainy.addVector({ id: '1', vector: [0.1, 0.2] })
|
||||
} catch (error) {
|
||||
if (error.message.includes('dimension mismatch')) {
|
||||
console.error('Vector has wrong dimensions')
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 Performance APIs
|
||||
|
||||
### Metrics Collection
|
||||
|
||||
```typescript
|
||||
// Get current performance metrics
|
||||
const metrics = brainy.getPerformanceMetrics()
|
||||
|
||||
// Available metrics
|
||||
interface PerformanceMetrics {
|
||||
totalSearches: number
|
||||
averageSearchTime: number
|
||||
cacheHitRate: number
|
||||
memoryUsage: number
|
||||
indexSize: number
|
||||
partitionStats?: PartitionStats[]
|
||||
}
|
||||
```
|
||||
|
||||
### Performance Monitoring
|
||||
|
||||
```typescript
|
||||
// Monitor performance over time
|
||||
setInterval(() => {
|
||||
const metrics = brainy.getPerformanceMetrics()
|
||||
|
||||
if (metrics.averageSearchTime > 500) {
|
||||
console.warn('Search performance degrading')
|
||||
}
|
||||
|
||||
if (metrics.cacheHitRate < 0.7) {
|
||||
console.warn('Low cache hit rate')
|
||||
}
|
||||
}, 60000) // Check every minute
|
||||
```
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- **[Getting Started](../getting-started/)** - Basic setup and usage
|
||||
- **[User Guides](../user-guides/)** - Feature-specific guides
|
||||
- **[Optimization Guides](../optimization-guides/)** - Performance tuning
|
||||
- **[Examples](../examples/)** - Working code samples
|
||||
- **[Technical Reference](../technical/)** - Implementation details
|
||||
|
||||
## 💡 API Design Principles
|
||||
|
||||
1. **Zero Configuration**: Sane defaults for immediate productivity
|
||||
2. **Progressive Enhancement**: Simple → Advanced as needed
|
||||
3. **Performance First**: Optimized for production workloads
|
||||
4. **Type Safety**: Full TypeScript support with generics
|
||||
5. **Error Resilience**: Graceful degradation and helpful error messages
|
||||
|
||||
---
|
||||
|
||||
**Explore the complete API documentation to unlock Brainy's full potential!** 🚀
|
||||
|
|
@ -1,609 +0,0 @@
|
|||
# Search Methods API Reference
|
||||
|
||||
Complete API documentation for Brainy's search methods with MongoDB-style metadata filtering.
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy provides powerful search capabilities that combine vector similarity with sophisticated metadata filtering. All search methods support the new `metadata` parameter for advanced filtering using MongoDB-style operators.
|
||||
|
||||
## Core Search Methods
|
||||
|
||||
### `search(queryVector, k, options)`
|
||||
|
||||
**Vector-based search with metadata filtering**
|
||||
|
||||
```typescript
|
||||
async search<T = any>(
|
||||
queryVector: Vector | any,
|
||||
k: number = 10,
|
||||
options: SearchOptions<T> = {}
|
||||
): Promise<SearchResult<T>[]>
|
||||
```
|
||||
|
||||
#### Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `queryVector` | `Vector \| any` | Required | Query vector or data to be embedded |
|
||||
| `k` | `number` | `10` | Maximum number of results to return |
|
||||
| `options` | `SearchOptions<T>` | `{}` | Search configuration options |
|
||||
|
||||
#### Options
|
||||
|
||||
```typescript
|
||||
interface SearchOptions<T> {
|
||||
nounTypes?: string[] // Filter by entity types
|
||||
includeVerbs?: boolean // Include relationships in results
|
||||
searchMode?: 'local' | 'remote' | 'combined'
|
||||
metadata?: MetadataFilter // MongoDB-style metadata filtering 🆕
|
||||
service?: string // Filter by service that created the data
|
||||
offset?: number // Pagination offset
|
||||
forceEmbed?: boolean // Force embedding even if input is a vector
|
||||
}
|
||||
```
|
||||
|
||||
#### Returns
|
||||
|
||||
```typescript
|
||||
interface SearchResult<T> {
|
||||
id: string
|
||||
score: number // Similarity score (0-1, higher = more similar)
|
||||
vector: Vector
|
||||
metadata: T
|
||||
nounType?: string
|
||||
createdBy?: any
|
||||
}
|
||||
```
|
||||
|
||||
#### Example
|
||||
|
||||
```javascript
|
||||
const results = await brainy.search("smartphone", 10, {
|
||||
metadata: {
|
||||
category: { $in: ["electronics", "mobile"] },
|
||||
brand: "Apple",
|
||||
price: { $lt: 1000 }
|
||||
},
|
||||
nounTypes: ["product"],
|
||||
includeVerbs: true
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `searchText(query, k, options)`
|
||||
|
||||
**Text-based search with automatic embedding**
|
||||
|
||||
```typescript
|
||||
async searchText(
|
||||
query: string,
|
||||
k: number = 10,
|
||||
options: TextSearchOptions = {}
|
||||
): Promise<SearchResult<T>[]>
|
||||
```
|
||||
|
||||
#### Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `query` | `string` | Required | Text query to search for |
|
||||
| `k` | `number` | `10` | Maximum number of results to return |
|
||||
| `options` | `TextSearchOptions` | `{}` | Search configuration options |
|
||||
|
||||
#### Options
|
||||
|
||||
```typescript
|
||||
interface TextSearchOptions {
|
||||
nounTypes?: string[] // Filter by entity types
|
||||
includeVerbs?: boolean // Include relationships in results
|
||||
searchMode?: 'local' | 'remote' | 'combined'
|
||||
metadata?: MetadataFilter // MongoDB-style metadata filtering 🆕
|
||||
}
|
||||
```
|
||||
|
||||
#### Example
|
||||
|
||||
```javascript
|
||||
const results = await brainy.searchText("gaming laptop", 15, {
|
||||
metadata: {
|
||||
availability: { $in: ["in_stock", "preorder"] },
|
||||
price: { $lte: 2000 },
|
||||
specs: { $all: ["RTX4060", "16GB RAM"] }
|
||||
},
|
||||
nounTypes: ["product"]
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `searchByNounTypes(queryVector, k, nounTypes, options)`
|
||||
|
||||
**Search within specific entity types with metadata filtering**
|
||||
|
||||
```typescript
|
||||
async searchByNounTypes(
|
||||
queryVectorOrData: Vector | any,
|
||||
k: number = 10,
|
||||
nounTypes: string[] | null = null,
|
||||
options: SearchByNounTypesOptions = {}
|
||||
): Promise<SearchResult<T>[]>
|
||||
```
|
||||
|
||||
#### Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `queryVectorOrData` | `Vector \| any` | Required | Query vector or data to be embedded |
|
||||
| `k` | `number` | `10` | Maximum number of results to return |
|
||||
| `nounTypes` | `string[] \| null` | `null` | Specific entity types to search within |
|
||||
| `options` | `SearchByNounTypesOptions` | `{}` | Search configuration options |
|
||||
|
||||
#### Options
|
||||
|
||||
```typescript
|
||||
interface SearchByNounTypesOptions {
|
||||
forceEmbed?: boolean // Force embedding even if input is a vector
|
||||
service?: string // Filter by service that created the data
|
||||
metadata?: MetadataFilter // MongoDB-style metadata filtering 🆕
|
||||
offset?: number // Pagination offset
|
||||
}
|
||||
```
|
||||
|
||||
#### Example
|
||||
|
||||
```javascript
|
||||
const results = await brainy.searchByNounTypes(
|
||||
"organic coffee beans",
|
||||
20,
|
||||
["product", "food_item"],
|
||||
{
|
||||
metadata: {
|
||||
$and: [
|
||||
{ certification: "organic" },
|
||||
{ origin: { $includes: "Ethiopia" } },
|
||||
{ availability: { $ne: "out_of_stock" } }
|
||||
]
|
||||
},
|
||||
service: "ecommerce_platform"
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metadata Filtering
|
||||
|
||||
### `MetadataFilter` Interface
|
||||
|
||||
The metadata filtering system supports MongoDB-style operators for complex queries:
|
||||
|
||||
```typescript
|
||||
type MetadataFilter = {
|
||||
[field: string]: any | {
|
||||
// Comparison operators
|
||||
$eq?: any
|
||||
$ne?: any
|
||||
$gt?: number | string | Date
|
||||
$gte?: number | string | Date
|
||||
$lt?: number | string | Date
|
||||
$lte?: number | string | Date
|
||||
|
||||
// Array operators
|
||||
$in?: any[]
|
||||
$nin?: any[]
|
||||
$all?: any[]
|
||||
$includes?: any
|
||||
$size?: number
|
||||
|
||||
// String operators
|
||||
$regex?: string
|
||||
$startsWith?: string
|
||||
$endsWith?: string
|
||||
$contains?: string
|
||||
|
||||
// Existence operators
|
||||
$exists?: boolean
|
||||
$type?: 'string' | 'number' | 'boolean' | 'object' | 'array'
|
||||
}
|
||||
|
||||
// Logical operators
|
||||
$and?: MetadataFilter[]
|
||||
$or?: MetadataFilter[]
|
||||
$not?: MetadataFilter
|
||||
$nor?: MetadataFilter[]
|
||||
}
|
||||
```
|
||||
|
||||
### Operator Examples
|
||||
|
||||
#### Comparison Operators
|
||||
|
||||
```javascript
|
||||
// Equal (implicit)
|
||||
{ category: "books" }
|
||||
|
||||
// Explicit comparison
|
||||
{
|
||||
price: { $lte: 50 },
|
||||
pages: { $gt: 200 },
|
||||
rating: { $ne: null }
|
||||
}
|
||||
```
|
||||
|
||||
#### Array Operators
|
||||
|
||||
```javascript
|
||||
{
|
||||
genres: { $in: ["Fiction", "Mystery", "Thriller"] }, // Has any of these genres
|
||||
awards: { $all: ["Hugo", "Nebula"] }, // Has all awards
|
||||
chapters: { $size: 12 }, // Exactly 12 chapters
|
||||
tags: { $includes: "bestseller" } // Array contains "bestseller"
|
||||
}
|
||||
```
|
||||
|
||||
#### String Operators
|
||||
|
||||
```javascript
|
||||
{
|
||||
isbn: { $regex: "^978-" }, // ISBN starts with 978
|
||||
title: { $startsWith: "The" }, // Title starts with "The"
|
||||
description: { $contains: "adventure" }, // Description contains "adventure"
|
||||
publisher: { $endsWith: "Press" } // Publisher ends with "Press"
|
||||
}
|
||||
```
|
||||
|
||||
#### Logical Operators
|
||||
|
||||
```javascript
|
||||
{
|
||||
$and: [
|
||||
{ category: "electronics" },
|
||||
{ warranty: true }
|
||||
],
|
||||
$or: [
|
||||
{ brand: "Apple" },
|
||||
{ brand: "Samsung" }
|
||||
],
|
||||
$not: { status: "discontinued" }
|
||||
}
|
||||
```
|
||||
|
||||
#### Nested Fields (Dot Notation)
|
||||
|
||||
```javascript
|
||||
{
|
||||
"specs.display.size": { $gte: 15 },
|
||||
"specs.processor.brand": "Intel",
|
||||
"ratings.average": { $gt: 4.5 }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Search Options
|
||||
|
||||
### Combining Multiple Filters
|
||||
|
||||
You can combine `metadata` filtering with other search options:
|
||||
|
||||
```javascript
|
||||
const results = await brainy.search("science textbook", 10, {
|
||||
// Entity type filtering
|
||||
nounTypes: ["book"],
|
||||
|
||||
// Service filtering
|
||||
service: "academic_platform",
|
||||
|
||||
// Metadata filtering
|
||||
metadata: {
|
||||
subject: { $in: ["physics", "chemistry"] },
|
||||
format: { $includes: "hardcover" },
|
||||
condition: { $ne: "damaged" }
|
||||
},
|
||||
|
||||
// Include relationships
|
||||
includeVerbs: true,
|
||||
|
||||
// Pagination
|
||||
offset: 20
|
||||
})
|
||||
```
|
||||
|
||||
### Search Modes
|
||||
|
||||
Control how search is performed:
|
||||
|
||||
```javascript
|
||||
const results = await brainy.searchText("cooking recipes", 10, {
|
||||
searchMode: "local", // Search only local data
|
||||
// searchMode: "remote", // Search only remote instances
|
||||
// searchMode: "combined", // Search local + remote (default)
|
||||
|
||||
metadata: {
|
||||
cuisine: { $includes: "italian" }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Index Optimization
|
||||
|
||||
Metadata filtering uses automatic indexing for optimal performance:
|
||||
|
||||
- **Pre-filtering**: Uses indexes to identify candidates before vector search
|
||||
- **Automatic maintenance**: Indexes update when data changes
|
||||
- **Memory efficient**: LRU caching with configurable limits
|
||||
|
||||
### Query Performance Tips
|
||||
|
||||
1. **Use specific filters first**:
|
||||
```javascript
|
||||
// Faster: specific equality
|
||||
{ category: "books" }
|
||||
|
||||
// Slower: negation
|
||||
{ category: { $ne: "magazines" } }
|
||||
```
|
||||
|
||||
2. **Combine filters efficiently**:
|
||||
```javascript
|
||||
// Faster: most selective filter first
|
||||
{
|
||||
isbn: "978-0123456789", // High selectivity
|
||||
genre: { $includes: "mystery" }, // Medium selectivity
|
||||
in_stock: true // Low selectivity
|
||||
}
|
||||
```
|
||||
|
||||
3. **Use appropriate operators**:
|
||||
```javascript
|
||||
// For arrays, use array operators
|
||||
{ genres: { $includes: "mystery" } } // ✅ Correct
|
||||
{ genres: "mystery" } // ❌ Won't match arrays
|
||||
```
|
||||
|
||||
### Index Configuration
|
||||
|
||||
Configure indexing behavior:
|
||||
|
||||
```javascript
|
||||
const brainy = new BrainyData({
|
||||
metadataIndex: {
|
||||
maxIndexSize: 50000, // Max entries per field+value
|
||||
rebuildThreshold: 0.05, // Rebuild when 5% stale
|
||||
autoOptimize: true, // Auto-cleanup unused entries
|
||||
indexedFields: ["category", "brand"], // Only index specific fields
|
||||
excludeFields: ["internal_id", "temp"] // Never index sensitive fields
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
```javascript
|
||||
try {
|
||||
const results = await brainy.search("query", 10, {
|
||||
metadata: { level: "senior" }
|
||||
})
|
||||
} catch (error) {
|
||||
if (error.message.includes('MetadataIndexError')) {
|
||||
// Index-related error
|
||||
console.error('Metadata index error:', error)
|
||||
} else if (error.message.includes('ValidationError')) {
|
||||
// Invalid query format
|
||||
console.error('Invalid metadata query:', error)
|
||||
} else {
|
||||
// Other search errors
|
||||
console.error('Search error:', error)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Query Validation
|
||||
|
||||
Brainy validates metadata queries and provides helpful error messages:
|
||||
|
||||
```javascript
|
||||
// Invalid operator
|
||||
{ price: { $invalid: 25 } }
|
||||
// Error: Unknown operator '$invalid'
|
||||
|
||||
// Type mismatch
|
||||
{ pages: { $gt: "not_a_number" } }
|
||||
// Error: $gt operator requires number, got string
|
||||
|
||||
// Invalid regex
|
||||
{ title: { $regex: "[invalid" } }
|
||||
// Error: Invalid regular expression
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Filter Discovery API (v0.49+)
|
||||
|
||||
### getFilterValues(field)
|
||||
|
||||
Get all available values for a specific field with O(1) field lookup:
|
||||
|
||||
```javascript
|
||||
// Get all categories in the database - O(1) field access
|
||||
const categories = await brainy.getFilterValues('category')
|
||||
// Returns: ['electronics', 'books', 'clothing', 'home', ...]
|
||||
|
||||
// Get all brands - O(1) field lookup + O(n) value retrieval
|
||||
const brands = await brainy.getFilterValues('brand')
|
||||
// Returns: ['apple', 'samsung', 'sony', ...]
|
||||
|
||||
// Use discovered values in filters
|
||||
const results = await brainy.search("products", 10, {
|
||||
metadata: {
|
||||
category: { $in: categories.slice(0, 3) }, // First 3 categories
|
||||
brand: brands[0] // Specific brand
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### getFilterFields()
|
||||
|
||||
Get all fields that have been indexed - O(1) operation:
|
||||
|
||||
```javascript
|
||||
// Discover what fields are available - O(1) direct access
|
||||
const fields = await brainy.getFilterFields()
|
||||
// Returns: ['category', 'price', 'brand', 'rating', 'tags', ...]
|
||||
|
||||
// Build dynamic filters based on available fields
|
||||
const filter = {}
|
||||
if (fields.includes('category')) {
|
||||
filter.category = 'electronics'
|
||||
}
|
||||
if (fields.includes('price')) {
|
||||
filter.price = { $lte: 1000 }
|
||||
}
|
||||
|
||||
const results = await brainy.search("query", 10, { metadata: filter })
|
||||
```
|
||||
|
||||
### Use Cases
|
||||
|
||||
1. **Dynamic UI Generation**: Build filter dropdowns from actual data
|
||||
2. **Data Exploration**: Understand what metadata exists
|
||||
3. **Validation**: Check if a field exists before filtering
|
||||
4. **Analytics**: See distribution of values
|
||||
|
||||
```javascript
|
||||
// Build a filter UI dynamically
|
||||
async function buildFilterUI() {
|
||||
const fields = await brainy.getFilterFields()
|
||||
|
||||
for (const field of fields) {
|
||||
const values = await brainy.getFilterValues(field)
|
||||
console.log(`${field}: ${values.length} unique values`)
|
||||
|
||||
// Create dropdown/checkbox for each field
|
||||
createFilterControl(field, values)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### From Simple Filtering
|
||||
|
||||
```javascript
|
||||
// Before (v0.47.x and earlier)
|
||||
const results = await brainy.search("laptop", 10, {
|
||||
filter: { brand: "Apple" } // Simple object matching
|
||||
})
|
||||
|
||||
// After (v0.48.x+) - Backward compatible!
|
||||
const results = await brainy.search("laptop", 10, {
|
||||
metadata: {
|
||||
brand: "Apple", // Same simple matching
|
||||
price: { $lte: 2000 }, // Plus MongoDB operators
|
||||
specs: { $includes: "SSD" } // Plus array operations
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Automatic Migration
|
||||
|
||||
- **No code changes required** - existing searches continue to work
|
||||
- **Indexes build automatically** - on first startup after upgrade
|
||||
- **Performance improves gradually** - as indexes populate
|
||||
|
||||
---
|
||||
|
||||
## Examples by Use Case
|
||||
|
||||
### Academic Library
|
||||
|
||||
```javascript
|
||||
// Find advanced textbooks
|
||||
const textbooks = await brainy.searchText("mathematics textbook", 20, {
|
||||
metadata: {
|
||||
level: { $in: ["undergraduate", "graduate", "advanced"] },
|
||||
subject: { $in: ["calculus", "algebra", "statistics"] },
|
||||
format: { $all: ["hardcover", "solutions_manual"] },
|
||||
availability: "in_stock"
|
||||
}
|
||||
})
|
||||
|
||||
// Find research papers with specific citations
|
||||
const researchPapers = await brainy.search(queryVector, 15, {
|
||||
metadata: {
|
||||
citations: { $gte: 10, $lte: 1000 },
|
||||
$or: [
|
||||
{ "author.degree": { $in: ["PhD", "Masters"] } },
|
||||
{ awards: { $size: { $gte: 1 } } }
|
||||
]
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### E-commerce Platform
|
||||
|
||||
```javascript
|
||||
// Product search with filters
|
||||
const products = await brainy.searchText("wireless headphones", 25, {
|
||||
metadata: {
|
||||
category: "electronics",
|
||||
price: { $lte: 200 },
|
||||
rating: { $gte: 4.0 },
|
||||
availability: { $ne: "out_of_stock" },
|
||||
features: { $all: ["bluetooth", "noise_canceling"] }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Content Management
|
||||
|
||||
```javascript
|
||||
// Find published articles
|
||||
const articles = await brainy.searchText("climate change", 10, {
|
||||
metadata: {
|
||||
status: "published",
|
||||
publish_date: { $gte: "2023-01-01" },
|
||||
author: { $in: ["Dr. Green", "Prof. Earth"] },
|
||||
tags: { $includes: "environment" },
|
||||
word_count: { $gte: 1000 }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Monitoring
|
||||
|
||||
### Index Statistics
|
||||
|
||||
```javascript
|
||||
// Get index performance metrics
|
||||
const stats = await brainy.metadataIndex?.getStats()
|
||||
console.log(`Index entries: ${stats.totalEntries}`)
|
||||
console.log(`Fields indexed: ${stats.fieldsIndexed}`)
|
||||
console.log(`Memory usage: ${stats.indexSize} bytes`)
|
||||
```
|
||||
|
||||
### Search Performance
|
||||
|
||||
```javascript
|
||||
// Monitor search performance
|
||||
const start = Date.now()
|
||||
const results = await brainy.search("query", 10, {
|
||||
metadata: { category: "electronics" }
|
||||
})
|
||||
const duration = Date.now() - start
|
||||
console.log(`Search completed in ${duration}ms`)
|
||||
console.log(`Found ${results.length} results`)
|
||||
```
|
||||
|
||||
This API reference provides complete documentation for all search methods with metadata filtering. The MongoDB-style operators give you powerful querying capabilities while maintaining high performance through automatic indexing.
|
||||
|
|
@ -1,331 +0,0 @@
|
|||
# Storage Adapters
|
||||
|
||||
Brainy's storage system is designed for universal compatibility through a simple, powerful abstraction layer. Any storage system that can handle key-value operations can be used as a Brainy backend.
|
||||
|
||||
## 🌍 Universal Storage Architecture
|
||||
|
||||
**One Interface, Any Storage System.** Brainy works with virtually any data store through its `StorageAdapter` interface.
|
||||
|
||||
### Currently Supported Storage Systems
|
||||
|
||||
| Storage Type | Environment | Use Case | Status |
|
||||
|--------------|-------------|----------|---------|
|
||||
| **S3-Compatible** | Production | AWS S3, Cloudflare R2, MinIO, DigitalOcean Spaces | ✅ Production Ready |
|
||||
| **File System** | Node.js/Server | Local development, dedicated servers | ✅ Production Ready |
|
||||
| **OPFS (Origin Private File System)** | Browser | Modern web apps, PWAs | ✅ Production Ready |
|
||||
| **Memory Storage** | Any | Testing, caching, temporary data | ✅ Production Ready |
|
||||
|
||||
### 🚀 Easily Add New Storage Backends
|
||||
|
||||
**Want to use a database that's not listed? No problem!** Creating a new storage adapter is straightforward because Brainy only requires these simple operations:
|
||||
|
||||
```typescript
|
||||
interface StorageAdapter {
|
||||
// Basic key-value operations
|
||||
saveMetadata(id: string, data: any): Promise<void>
|
||||
getMetadata(id: string): Promise<any | null>
|
||||
|
||||
// Entity operations (built on top of metadata)
|
||||
saveNoun(noun: HNSWNoun): Promise<void>
|
||||
getNoun(id: string): Promise<HNSWNoun | null>
|
||||
|
||||
// Pagination support
|
||||
getNouns(options?: PaginationOptions): Promise<PaginatedResult<HNSWNoun>>
|
||||
getVerbs(options?: PaginationOptions): Promise<PaginatedResult<GraphVerb>>
|
||||
|
||||
// Lifecycle
|
||||
init(): Promise<void>
|
||||
clear(): Promise<void>
|
||||
}
|
||||
```
|
||||
|
||||
## 🔌 Storage Systems You Can Add
|
||||
|
||||
### NoSQL Databases
|
||||
- **MongoDB** - Store index entries as documents
|
||||
- **DynamoDB** - Use partition keys for metadata indexes
|
||||
- **Firestore** - Collections for entities, subcollections for indexes
|
||||
- **CouchDB** - Document-based storage with views
|
||||
|
||||
### SQL Databases
|
||||
- **PostgreSQL** - JSON columns for metadata, tables for entities
|
||||
- **MySQL** - JSON fields with indexes
|
||||
- **SQLite** - Lightweight local storage
|
||||
- **SQL Server** - Enterprise integration
|
||||
|
||||
### Key-Value Stores
|
||||
- **Redis** - High-performance caching and storage
|
||||
- **LevelDB** - Embedded key-value database
|
||||
- **RocksDB** - High-performance key-value store
|
||||
- **etcd** - Distributed key-value store
|
||||
|
||||
### Graph Databases
|
||||
- **Neo4j** - Store entities as nodes, indexes as separate node types
|
||||
- **ArangoDB** - Multi-model database support
|
||||
- **Amazon Neptune** - AWS managed graph database
|
||||
|
||||
### Cloud Storage
|
||||
- **Google Cloud Storage** - Via S3-compatible API or native
|
||||
- **Azure Blob Storage** - Native Azure integration
|
||||
- **Backblaze B2** - Cost-effective cloud storage
|
||||
|
||||
### Search Engines
|
||||
- **Elasticsearch** - Complement vector search with text search
|
||||
- **Apache Solr** - Enterprise search integration
|
||||
- **Algolia** - Hosted search service
|
||||
|
||||
## 💡 How Storage Adapters Work
|
||||
|
||||
### Simple Key-Value Foundation
|
||||
|
||||
All Brainy storage is built on a simple principle: **everything is stored as JSON objects with unique keys**.
|
||||
|
||||
```typescript
|
||||
// This is all your storage needs to support:
|
||||
await storage.saveMetadata("user_123", {
|
||||
name: "John Doe",
|
||||
type: "person",
|
||||
email: "john@example.com"
|
||||
})
|
||||
|
||||
const user = await storage.getMetadata("user_123")
|
||||
// Returns: { name: "John Doe", type: "person", email: "john@example.com" }
|
||||
```
|
||||
|
||||
### Automatic Index Management
|
||||
|
||||
The metadata indexing system automatically handles complex operations:
|
||||
|
||||
```typescript
|
||||
// Brainy automatically creates these index entries:
|
||||
await storage.saveMetadata("__metadata_index__type_person_chunk0", {
|
||||
field: "type",
|
||||
value: "person",
|
||||
ids: ["user_123", "user_456", ...]
|
||||
})
|
||||
|
||||
await storage.saveMetadata("__metadata_field_index__type", {
|
||||
values: { "person": 50, "company": 23, "product": 12 }
|
||||
})
|
||||
```
|
||||
|
||||
### Directory Structure
|
||||
|
||||
Storage adapters use a logical directory structure that maps to your storage system:
|
||||
|
||||
```
|
||||
entities/
|
||||
├── nouns/
|
||||
│ ├── vectors/ # Vector data for semantic search
|
||||
│ │ ├── user_123.json
|
||||
│ │ └── company_456.json
|
||||
│ └── metadata/ # Metadata + automatic indexes
|
||||
│ ├── user_123.json # User metadata
|
||||
│ ├── __metadata_index__type_person_chunk0.json # Index entries
|
||||
│ └── __metadata_field_index__type.json # Field values
|
||||
└── verbs/
|
||||
├── vectors/ # Relationship vectors
|
||||
└── metadata/ # Relationship metadata + indexes
|
||||
```
|
||||
|
||||
## 🛠️ Creating a Custom Storage Adapter
|
||||
|
||||
### Step 1: Implement the Interface
|
||||
|
||||
```typescript
|
||||
import { BaseStorage } from '@soulcraft/brainy'
|
||||
|
||||
export class MyCustomStorage extends BaseStorage {
|
||||
private client: MyDatabaseClient
|
||||
|
||||
async init(): Promise<void> {
|
||||
this.client = new MyDatabaseClient(this.config)
|
||||
await this.client.connect()
|
||||
this.isInitialized = true
|
||||
}
|
||||
|
||||
async saveMetadata(id: string, metadata: any): Promise<void> {
|
||||
// Map to your database's put/insert operation
|
||||
await this.client.put(id, JSON.stringify(metadata))
|
||||
}
|
||||
|
||||
async getMetadata(id: string): Promise<any | null> {
|
||||
// Map to your database's get/select operation
|
||||
const result = await this.client.get(id)
|
||||
return result ? JSON.parse(result) : null
|
||||
}
|
||||
|
||||
// Implement other required methods...
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Use Your Custom Adapter
|
||||
|
||||
```typescript
|
||||
import { BrainyData } from '@soulcraft/brainy'
|
||||
import { MyCustomStorage } from './my-custom-storage'
|
||||
|
||||
const brainy = new BrainyData({
|
||||
storage: {
|
||||
custom: new MyCustomStorage({
|
||||
connectionString: 'your-db-connection-string',
|
||||
database: 'brainy-vectors'
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
await brainy.init()
|
||||
// Now Brainy uses your custom storage backend!
|
||||
```
|
||||
|
||||
## 🏗️ Implementation Examples
|
||||
|
||||
### Redis Storage Adapter
|
||||
|
||||
```typescript
|
||||
export class RedisStorage extends BaseStorage {
|
||||
private redis: Redis
|
||||
|
||||
async saveMetadata(id: string, data: any): Promise<void> {
|
||||
await this.redis.set(id, JSON.stringify(data))
|
||||
}
|
||||
|
||||
async getMetadata(id: string): Promise<any | null> {
|
||||
const result = await this.redis.get(id)
|
||||
return result ? JSON.parse(result) : null
|
||||
}
|
||||
|
||||
async getNouns(options?: PaginationOptions): Promise<PaginatedResult<HNSWNoun>> {
|
||||
const cursor = options?.cursor || '0'
|
||||
const limit = options?.limit || 100
|
||||
|
||||
const [newCursor, keys] = await this.redis.scan(cursor, 'MATCH', 'entities/nouns/vectors/*', 'COUNT', limit)
|
||||
|
||||
const nouns: HNSWNoun[] = []
|
||||
for (const key of keys) {
|
||||
const data = await this.redis.get(key)
|
||||
if (data) {
|
||||
nouns.push(this.parseNoun(JSON.parse(data)))
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
items: nouns,
|
||||
hasMore: newCursor !== '0',
|
||||
nextCursor: newCursor !== '0' ? newCursor : undefined
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### MongoDB Storage Adapter
|
||||
|
||||
```typescript
|
||||
export class MongoStorage extends BaseStorage {
|
||||
private db: MongoDatabase
|
||||
|
||||
async saveMetadata(id: string, data: any): Promise<void> {
|
||||
const collection = this.getCollectionForId(id)
|
||||
await collection.replaceOne(
|
||||
{ _id: id },
|
||||
{ _id: id, data },
|
||||
{ upsert: true }
|
||||
)
|
||||
}
|
||||
|
||||
async getMetadata(id: string): Promise<any | null> {
|
||||
const collection = this.getCollectionForId(id)
|
||||
const doc = await collection.findOne({ _id: id })
|
||||
return doc?.data || null
|
||||
}
|
||||
|
||||
private getCollectionForId(id: string): Collection {
|
||||
// Route different entity types to different collections for optimization
|
||||
if (id.startsWith('entities/nouns/')) return this.db.collection('nouns')
|
||||
if (id.startsWith('entities/verbs/')) return this.db.collection('verbs')
|
||||
if (id.startsWith('__metadata_index__')) return this.db.collection('indexes')
|
||||
return this.db.collection('metadata')
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 Why This Architecture is Powerful
|
||||
|
||||
### 1. **Storage-Agnostic Intelligence**
|
||||
All of Brainy's smart features work with any storage backend:
|
||||
- Metadata indexing
|
||||
- Filter discovery
|
||||
- Pagination
|
||||
- Caching
|
||||
- Real-time updates
|
||||
|
||||
### 2. **Performance Optimization**
|
||||
Each adapter can optimize for its storage type:
|
||||
- **Redis**: Leverage Redis pipelines and data structures
|
||||
- **MongoDB**: Use MongoDB aggregation pipelines
|
||||
- **SQL**: Optimize with proper indexes and queries
|
||||
- **S3**: Batch operations and intelligent prefixing
|
||||
|
||||
### 3. **Zero Migration Lock-In**
|
||||
Switch storage backends without changing your application code:
|
||||
```typescript
|
||||
// Development: File system
|
||||
const devBrainy = new BrainyData({ storage: { fileSystem: { path: './data' } } })
|
||||
|
||||
// Production: S3
|
||||
const prodBrainy = new BrainyData({ storage: { s3Storage: { bucketName: 'prod-vectors' } } })
|
||||
|
||||
// Same API, different storage!
|
||||
```
|
||||
|
||||
### 4. **Hybrid Deployments**
|
||||
Use different storage for different use cases:
|
||||
```typescript
|
||||
// Writers use S3 for durability
|
||||
const writer = new BrainyData({
|
||||
storage: { s3Storage: { bucketName: 'durable-storage' } }
|
||||
})
|
||||
|
||||
// Readers use Redis for speed
|
||||
const reader = new BrainyData({
|
||||
storage: { redis: { connectionString: 'redis://cache' } }
|
||||
})
|
||||
```
|
||||
|
||||
## 📊 Storage Performance Characteristics
|
||||
|
||||
| Storage Type | Write Speed | Read Speed | Scalability | Cost | Best For |
|
||||
|--------------|-------------|------------|-------------|------|----------|
|
||||
| Memory | ⚡⚡⚡⚡ | ⚡⚡⚡⚡ | ⚡ | 💰💰💰 | Testing, caching |
|
||||
| Redis | ⚡⚡⚡ | ⚡⚡⚡⚡ | ⚡⚡⚡ | 💰💰 | Real-time apps |
|
||||
| File System | ⚡⚡⚡ | ⚡⚡⚡ | ⚡⚡ | 💰 | Development, single server |
|
||||
| MongoDB | ⚡⚡ | ⚡⚡⚡ | ⚡⚡⚡⚡ | 💰💰 | Document-heavy apps |
|
||||
| PostgreSQL | ⚡⚡ | ⚡⚡ | ⚡⚡⚡ | 💰💰 | Complex queries, ACID |
|
||||
| S3-Compatible | ⚡ | ⚡⚡ | ⚡⚡⚡⚡ | 💰 | Large-scale, serverless |
|
||||
|
||||
## 🎯 Getting Started
|
||||
|
||||
### 1. **Choose Your Storage**
|
||||
Pick the storage system that matches your needs and infrastructure.
|
||||
|
||||
### 2. **Implement or Use Existing**
|
||||
Use a built-in adapter or create your own following the examples above.
|
||||
|
||||
### 3. **Configure and Deploy**
|
||||
```typescript
|
||||
const brainy = new BrainyData({
|
||||
storage: { yourStorage: yourConfig }
|
||||
})
|
||||
await brainy.init()
|
||||
```
|
||||
|
||||
### 4. **Scale as Needed**
|
||||
Switch storage backends as your requirements evolve - your application code stays the same.
|
||||
|
||||
## 💡 Need Help?
|
||||
|
||||
- **Built-in adapters**: Use File System, S3, OPFS, or Memory storage
|
||||
- **Custom adapters**: Follow the examples above or ask in [GitHub Discussions](https://github.com/soulcraftlabs/brainy/discussions)
|
||||
- **Performance tuning**: See [Storage Optimization Guide](../optimization-guides/storage-optimization.md)
|
||||
|
||||
**The power is in your hands** - Brainy adapts to your storage, not the other way around! 🚀
|
||||
File diff suppressed because it is too large
Load diff
2108
docs/api/README.md
Normal file
2108
docs/api/README.md
Normal file
File diff suppressed because it is too large
Load diff
114
docs/architecture/PERFORMANCE_ANALYSIS.md
Normal file
114
docs/architecture/PERFORMANCE_ANALYSIS.md
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
# Brainy Performance Analysis & Optimization
|
||||
|
||||
## Current Issues Found
|
||||
|
||||
### 1. ❌ CRITICAL: notEquals Operator is O(n)
|
||||
```javascript
|
||||
// PROBLEM: Gets ALL items to filter
|
||||
case 'notEquals':
|
||||
const allItemIds = await this.getAllIds() // O(n) - TERRIBLE!
|
||||
```
|
||||
|
||||
### 2. ❌ Soft Delete Performance
|
||||
- Every query adds `deleted: { notEquals: true }`
|
||||
- This makes EVERY query O(n) instead of O(log n)
|
||||
|
||||
### 3. ❌ exists Operator is Inefficient
|
||||
```javascript
|
||||
case 'exists':
|
||||
// Scans all cache entries - O(n)
|
||||
for (const [key, entry] of this.indexCache.entries()) {
|
||||
if (entry.field === field) {
|
||||
entry.ids.forEach(id => allIds.add(id))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4. ⚠️ Query Optimizer Not Smart Enough
|
||||
- `isSelectiveFilter()` needs to understand which filters are fast
|
||||
- Should prioritize O(1) and O(log n) operations
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### ✅ Fast Operations (Keep These)
|
||||
| Operation | Complexity | Example |
|
||||
|-----------|-----------|---------|
|
||||
| Vector Search (HNSW) | O(log n) | `like: "query"` |
|
||||
| Exact Match | O(1) | `where: { status: "active" }` |
|
||||
| Deleted Filter (NEW) | O(1) | `where: { deleted: false }` |
|
||||
| Range Query (sorted) | O(log n) | `where: { year: { gt: 2000 } }` |
|
||||
| Graph Traversal | O(k) | `connected: { from: id }` |
|
||||
|
||||
### ❌ Slow Operations (Need Fixing)
|
||||
| Operation | Current | Should Be | Fix |
|
||||
|-----------|---------|-----------|-----|
|
||||
| notEquals | O(n) | O(1) or O(log n) | Use complement index |
|
||||
| exists | O(n) | O(1) | Maintain field existence bitmap |
|
||||
| noneOf | O(n) | O(k) | Use set operations |
|
||||
|
||||
## Optimized Architecture
|
||||
|
||||
### Solution 1: Positive Indexing for Soft Delete ✅
|
||||
```javascript
|
||||
// Instead of: deleted !== true (O(n))
|
||||
// Use: deleted === false (O(1))
|
||||
where: { deleted: false }
|
||||
|
||||
// Ensure all items have deleted field
|
||||
if (!metadata.deleted) metadata.deleted = false
|
||||
```
|
||||
|
||||
### Solution 2: Complement Indices for notEquals
|
||||
```javascript
|
||||
class MetadataIndexManager {
|
||||
// For common notEquals queries, maintain complement sets
|
||||
private complementIndices: Map<string, Set<string>> = new Map()
|
||||
|
||||
// Example: Track non-deleted items separately
|
||||
private activeItems: Set<string> = new Set()
|
||||
private deletedItems: Set<string> = new Set()
|
||||
}
|
||||
```
|
||||
|
||||
### Solution 3: Field Existence Bitmap
|
||||
```javascript
|
||||
class FieldExistenceIndex {
|
||||
private fieldBitmaps: Map<string, BitSet> = new Map()
|
||||
|
||||
hasField(id: string, field: string): boolean {
|
||||
return this.fieldBitmaps.get(field)?.has(id) ?? false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Query Execution Strategy
|
||||
|
||||
### Progressive Search (When Metadata is Selective)
|
||||
```
|
||||
1. Field Filter (O(1) or O(log n)) → Small candidate set
|
||||
2. Vector Search within candidates (O(k log k))
|
||||
3. Fusion if needed
|
||||
```
|
||||
|
||||
### Parallel Search (When Nothing is Selective)
|
||||
```
|
||||
1. Vector Search (O(log n)) → Top K results
|
||||
2. Graph Traversal (O(m)) → Connected items
|
||||
3. Field Filter (O(1)) → Metadata matches
|
||||
4. Fusion: Intersection or Union
|
||||
```
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
1. **DONE** ✅ Fix soft delete to use `deleted: false`
|
||||
2. **TODO** 🔧 Optimize notEquals for common fields
|
||||
3. **TODO** 🔧 Add field existence index
|
||||
4. **TODO** 🔧 Improve query optimizer intelligence
|
||||
5. **TODO** 🔧 Add query explain mode for debugging
|
||||
|
||||
## Performance Targets
|
||||
|
||||
- Vector search: < 10ms for 1M items
|
||||
- Metadata filter: < 1ms for exact match
|
||||
- Combined query: < 20ms for complex queries
|
||||
- Soft delete overhead: < 0.1ms (O(1))
|
||||
242
docs/architecture/aggregation.md
Normal file
242
docs/architecture/aggregation.md
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
# Aggregation Architecture
|
||||
|
||||
> Write-time incremental aggregation with O(1) reads
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Write-time computation** — aggregates update on every `add()`, `update()`, and `delete()`, not as batch jobs
|
||||
2. **Incremental state** — running totals maintained per group, never rescanning the dataset
|
||||
3. **Provider interface** — TypeScript engine is the default; plugins can replace it with native implementations
|
||||
4. **Zero-allocation reads** — query results are computed from pre-aggregated state
|
||||
|
||||
## Component Overview
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Brainy │
|
||||
│ │
|
||||
│ add() / update() / delete() │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────┐ ┌────────────────────────┐ │
|
||||
│ │ AggregationIndex │ │ AggregateMaterializer │ │
|
||||
│ │ │───▶│ (debounced writes) │ │
|
||||
│ │ ├─ definitions Map │ └────────────────────────┘ │
|
||||
│ │ ├─ states Map │ │
|
||||
│ │ └─ staleMinMax Set │ ┌────────────────────────┐ │
|
||||
│ │ │ │ timeWindows.ts │ │
|
||||
│ │ Source filter ──────│───▶│ bucketTimestamp() │ │
|
||||
│ │ Group key ──────────│───▶│ parseBucketRange() │ │
|
||||
│ └──────────┬───────────┘ └────────────────────────┘ │
|
||||
│ │ │
|
||||
│ │ provider interface │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────┐ │
|
||||
│ │ AggregationProvider │ (optional, registered by │
|
||||
│ │ ├─ incrementalUpdate│ plugin like @soulcraft/cor) │
|
||||
│ │ ├─ rebuildAggregate │ │
|
||||
│ │ ├─ queryAggregate │ │
|
||||
│ │ └─ serialize/restore│ │
|
||||
│ └──────────────────────┘ │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## State Management
|
||||
|
||||
### Definitions
|
||||
|
||||
Registered via `brain.defineAggregate(def)`. Stored in a `Map<string, AggregateDefinition>` keyed by aggregate name. Persisted to storage under `__aggregation_definitions__` on flush.
|
||||
|
||||
### Group State
|
||||
|
||||
Each aggregate maintains a `Map<string, AggregateGroupState>` where keys are serialized group key values (e.g., `category=food|date=2024-01`). Each group holds per-metric `MetricState`:
|
||||
|
||||
```typescript
|
||||
interface MetricState {
|
||||
sum: number // Running total
|
||||
count: number // Entity count
|
||||
min: number // Minimum (Infinity if empty)
|
||||
max: number // Maximum (-Infinity if empty)
|
||||
m2?: number // Welford's M2 for stddev/variance
|
||||
}
|
||||
```
|
||||
|
||||
### Change Detection
|
||||
|
||||
On restart, definition hashes (FNV-1a 32-bit) are compared with the persisted hash. If a definition changed (different groupBy, metrics, or source), the aggregate state is reset and must be rebuilt.
|
||||
|
||||
## Write-Time Update Flow
|
||||
|
||||
When `brain.add(entity)` is called:
|
||||
|
||||
```
|
||||
1. For each registered aggregate:
|
||||
├─ Source filter check (type, service, where)
|
||||
│ └─ Skip if entity doesn't match
|
||||
├─ Aggregate entity check
|
||||
│ └─ Skip if entity.service === 'brainy:aggregation'
|
||||
│ or entity.metadata.__aggregate is set
|
||||
├─ Group key computation
|
||||
│ └─ Extract groupBy fields from metadata
|
||||
│ Apply time bucketing for windowed dimensions
|
||||
└─ Metric update
|
||||
└─ For each metric in the definition:
|
||||
├─ count: increment count
|
||||
├─ sum/avg: add value to sum, increment count
|
||||
├─ min/max: compare and update
|
||||
└─ stddev/variance: Welford's online update
|
||||
```
|
||||
|
||||
### Update Handling
|
||||
|
||||
On `brain.update(entity)`, the engine reverses the old entity's contribution and applies the new entity's contribution. This correctly handles:
|
||||
|
||||
- **Value changes**: old amount=10, new amount=20 — sum adjusts by +10
|
||||
- **Group key changes**: entity moves from category "food" to "drink" — both groups update
|
||||
- **Source filter changes**: entity type changes from Event to Document — removed from matching aggregates
|
||||
|
||||
### Delete Handling
|
||||
|
||||
On `brain.remove(id)`, the engine reverses the entity's contribution:
|
||||
|
||||
- `count` and `sum` are decremented
|
||||
- `min`/`max` may become stale (marked in `staleMinMax` for lazy recompute)
|
||||
- Welford's M2 is updated with the inverse formula
|
||||
- Empty groups (all metric counts at zero) are removed
|
||||
|
||||
## Algorithms
|
||||
|
||||
### Welford's Online Algorithm
|
||||
|
||||
Standard deviation and variance use Welford's numerically stable online algorithm with M2 tracking. This computes incrementally without storing individual values:
|
||||
|
||||
```
|
||||
On add(x):
|
||||
count += 1
|
||||
oldMean = (sum - x) / (count - 1) // mean before this value
|
||||
sum += x
|
||||
mean = sum / count // mean after this value
|
||||
M2 += (x - oldMean) * (x - mean)
|
||||
|
||||
On remove(x):
|
||||
oldMean = sum / count
|
||||
sum -= x
|
||||
count -= 1
|
||||
newMean = sum / count
|
||||
M2 = max(0, M2 - (x - oldMean) * (x - newMean))
|
||||
|
||||
Sample variance = M2 / (count - 1)
|
||||
Sample stddev = sqrt(variance)
|
||||
```
|
||||
|
||||
M2 is clamped to zero on remove to prevent floating-point drift from producing negative values.
|
||||
|
||||
### MIN/MAX Handling
|
||||
|
||||
The TypeScript engine uses simple comparison for add operations and marks MIN/MAX as potentially stale on delete (since removing the current min/max value requires a rescan). Stale values are lazily recomputed on the next query.
|
||||
|
||||
The Cor native engine uses a `BTreeMap<OrderedFloat<f64>, u64>` that tracks the exact frequency of every value, providing precise MIN/MAX after any sequence of operations without rescanning.
|
||||
|
||||
### Time Window Bucketing
|
||||
|
||||
Timestamps (Unix milliseconds) are bucketed using UTC-based formatting:
|
||||
|
||||
| Granularity | Bucket Key | Algorithm |
|
||||
|------------|-----------|-----------|
|
||||
| `hour` | `2024-01-15T14` | UTC year-month-day-hour |
|
||||
| `day` | `2024-01-15` | UTC year-month-day |
|
||||
| `week` | `2024-W03` | ISO 8601 week (Monday start, week 1 contains first Thursday) |
|
||||
| `month` | `2024-01` | UTC year-month |
|
||||
| `quarter` | `2024-Q1` | `ceil((month) / 3)` |
|
||||
| `year` | `2024` | UTC year |
|
||||
| `{ seconds: N }` | ISO timestamp | `floor(timestamp / interval) * interval` |
|
||||
|
||||
Bucket keys can be parsed back into `{ start, end }` timestamp ranges via `parseBucketRange()`.
|
||||
|
||||
## Provider Interface
|
||||
|
||||
The `AggregationProvider` interface defines the contract between Brainy's `AggregationIndex` and plugin-provided native implementations:
|
||||
|
||||
```typescript
|
||||
interface AggregationProvider {
|
||||
defineAggregate?(def: AggregateDefinition): void
|
||||
removeAggregate?(name: string): void
|
||||
|
||||
incrementalUpdate(
|
||||
name: string,
|
||||
def: AggregateDefinition,
|
||||
entity: Record<string, unknown>,
|
||||
op: 'add' | 'update' | 'delete',
|
||||
prev?: Record<string, unknown>
|
||||
): AggregateGroupState[]
|
||||
|
||||
computeGroupKey(
|
||||
entity: Record<string, unknown>,
|
||||
groupBy: GroupByDimension[]
|
||||
): Record<string, string | number>
|
||||
|
||||
rebuildAggregate(
|
||||
def: AggregateDefinition,
|
||||
entities: Array<Record<string, unknown>>
|
||||
): Map<string, AggregateGroupState>
|
||||
|
||||
queryAggregate(
|
||||
state: Map<string, AggregateGroupState>,
|
||||
params: AggregateQueryParams
|
||||
): AggregateResult[]
|
||||
|
||||
restoreState?(data: string): void
|
||||
serializeState?(): string
|
||||
}
|
||||
```
|
||||
|
||||
When a native provider is registered:
|
||||
|
||||
1. `AggregationIndex` delegates `incrementalUpdate()` to the provider instead of running TypeScript logic
|
||||
2. Provider returns updated `AggregateGroupState[]` which are applied back into the state maps
|
||||
3. Query execution is delegated via `queryAggregate()`
|
||||
4. State serialization is delegated via `serializeState()`/`restoreState()`
|
||||
|
||||
Brainy retains ownership of the state maps and persistence. The provider handles computation.
|
||||
|
||||
## Materialization
|
||||
|
||||
The `AggregateMaterializer` converts aggregate group states into `NounType.Measurement` entities:
|
||||
|
||||
1. When an aggregate group is updated and `materialize` is enabled, `scheduleMaterialize()` is called
|
||||
2. Materialization is debounced (default: 1000ms) to batch rapid updates during ingestion
|
||||
3. On trigger, the materializer either creates or updates a `NounType.Measurement` entity
|
||||
4. Materialized entities include `service: 'brainy:aggregation'` and `metadata.__aggregate` to prevent infinite loops
|
||||
|
||||
Materialized entities are automatically visible through:
|
||||
- OData endpoints
|
||||
- Google Sheets integration
|
||||
- Server-Sent Events (SSE)
|
||||
- Webhook notifications
|
||||
|
||||
## Persistence
|
||||
|
||||
### Storage Keys
|
||||
|
||||
| Key | Content |
|
||||
|-----|---------|
|
||||
| `__aggregation_definitions__` | Array of all definitions with FNV-1a hashes |
|
||||
| `__aggregation_state_{name}__` | Per-aggregate group states (array of `AggregateGroupState`) |
|
||||
| `__aggregation_native_state__` | Serialized native provider state (JSON string) |
|
||||
|
||||
### Lifecycle
|
||||
|
||||
1. **`init()`** — Load definitions, compare hashes, load matching state, restore native provider state
|
||||
2. **Write operations** — Mark modified aggregates as dirty
|
||||
3. **`flush()`** — Persist all dirty aggregate states and native provider state
|
||||
4. **`close()`** — Flush and release resources
|
||||
|
||||
## Source Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `src/aggregation/AggregationIndex.ts` | Core engine: definitions, state, write hooks, query |
|
||||
| `src/aggregation/materializer.ts` | Debounced materialization of results as entities |
|
||||
| `src/aggregation/timeWindows.ts` | Time bucketing and bucket range parsing |
|
||||
| `src/aggregation/index.ts` | Module exports |
|
||||
| `src/types/brainy.types.ts` | Type definitions for all aggregation interfaces |
|
||||
302
docs/architecture/augmentations-actual.md
Normal file
302
docs/architecture/augmentations-actual.md
Normal file
|
|
@ -0,0 +1,302 @@
|
|||
# Augmentations System - What Actually Exists
|
||||
|
||||
> **Important Update**: Investigation reveals Brainy has MORE augmentations than documented!
|
||||
|
||||
## ✅ Actually Implemented Augmentations (12+)
|
||||
|
||||
Full implementation with crash recovery, checkpointing, and replay.
|
||||
```typescript
|
||||
// Fully working with all features documented
|
||||
```
|
||||
|
||||
### 2. Entity Registry Augmentation ✅
|
||||
High-performance deduplication using bloom filters.
|
||||
```typescript
|
||||
import { EntityRegistryAugmentation } from 'brainy'
|
||||
// Complete with all features
|
||||
```
|
||||
|
||||
### 3. Auto-Register Entities Augmentation ✅
|
||||
Automatic entity extraction from text.
|
||||
```typescript
|
||||
import { AutoRegisterEntitiesAugmentation } from 'brainy'
|
||||
// Extracts and registers entities automatically
|
||||
```
|
||||
|
||||
### 4. Intelligent Verb Scoring Augmentation ✅
|
||||
Multi-factor relationship strength calculation.
|
||||
```typescript
|
||||
import { IntelligentVerbScoringAugmentation } from 'brainy'
|
||||
// Semantic, temporal, frequency scoring
|
||||
```
|
||||
|
||||
### 5. Batch Processing Augmentation ✅
|
||||
Dynamic batching with adaptive backpressure.
|
||||
```typescript
|
||||
import { BatchProcessingAugmentation } from 'brainy'
|
||||
// Smart batching with flow control
|
||||
```
|
||||
|
||||
### 6. Connection Pool Augmentation ✅
|
||||
Intelligent connection management.
|
||||
```typescript
|
||||
import { ConnectionPoolAugmentation } from 'brainy'
|
||||
// Auto-scaling connection pools
|
||||
```
|
||||
|
||||
### 7. Request Deduplicator Augmentation ✅
|
||||
Prevents duplicate operations.
|
||||
```typescript
|
||||
import { RequestDeduplicatorAugmentation } from 'brainy'
|
||||
// In-flight request deduplication
|
||||
```
|
||||
|
||||
### 8. WebSocket Conduit Augmentation ✅
|
||||
Real-time bidirectional streaming.
|
||||
```typescript
|
||||
import { WebSocketConduitAugmentation } from 'brainy'
|
||||
// Full WebSocket support
|
||||
```
|
||||
|
||||
### 9. WebRTC Conduit Augmentation ✅
|
||||
Peer-to-peer communication.
|
||||
```typescript
|
||||
import { WebRTCConduitAugmentation } from 'brainy'
|
||||
// P2P data channels
|
||||
```
|
||||
|
||||
### 10. Memory Storage Augmentation ✅
|
||||
Optimized in-memory operations.
|
||||
```typescript
|
||||
import { MemoryStorageAugmentation } from 'brainy'
|
||||
// Memory-specific optimizations
|
||||
```
|
||||
|
||||
### 11. Server Search Augmentation ✅
|
||||
Server-side search delegation over a conduit.
|
||||
```typescript
|
||||
import { ServerSearchConduitAugmentation } from 'brainy'
|
||||
// Forwards queries to a remote Brainy server
|
||||
```
|
||||
|
||||
### 12. Neural Import Augmentation ✅
|
||||
AI-powered data understanding and import.
|
||||
```typescript
|
||||
import { NeuralImportAugmentation } from 'brainy'
|
||||
// Full entity detection and classification
|
||||
```
|
||||
|
||||
## 🎯 Hidden Features in Augmentations
|
||||
|
||||
### Neural Import Capabilities (Fully Implemented!)
|
||||
```typescript
|
||||
const neuralImport = new NeuralImport(brain)
|
||||
|
||||
// These ALL work:
|
||||
await neuralImport.neuralImport('data.csv')
|
||||
await neuralImport.detectEntitiesWithNeuralAnalysis(data)
|
||||
await neuralImport.detectNounType(entity)
|
||||
await neuralImport.detectRelationships(entities)
|
||||
await neuralImport.generateInsights(data)
|
||||
```
|
||||
|
||||
### Operation Modes (Fully Implemented!)
|
||||
```typescript
|
||||
// Read-only mode with optimized caching
|
||||
const readerMode = new ReaderMode()
|
||||
// 80% cache, aggressive prefetch, 1hr TTL
|
||||
|
||||
// Write-only mode with batching
|
||||
const writerMode = new WriterMode()
|
||||
// Large write buffer, batch writes, minimal cache
|
||||
|
||||
// Hybrid mode
|
||||
const hybridMode = new HybridMode()
|
||||
// Balanced for mixed workloads
|
||||
```
|
||||
|
||||
### Advanced Caching (3-Level System!)
|
||||
```typescript
|
||||
const cacheManager = new CacheManager({
|
||||
hotCache: { size: 1000, ttl: 60000 }, // L1 - RAM
|
||||
warmCache: { size: 10000, ttl: 300000 }, // L2 - Fast storage
|
||||
coldCache: { size: 100000, ttl: null } // L3 - Persistent
|
||||
})
|
||||
```
|
||||
|
||||
### Performance Monitoring (Complete!)
|
||||
```typescript
|
||||
const monitor = new PerformanceMonitor(brain)
|
||||
|
||||
// All these metrics work:
|
||||
monitor.getMetrics() // Returns comprehensive stats
|
||||
monitor.getQueryPatterns() // Query analysis
|
||||
monitor.getCacheStats() // Cache performance
|
||||
monitor.getThrottlingMetrics() // Rate limiting info
|
||||
```
|
||||
|
||||
## 📊 Statistics System (Fully Working!)
|
||||
|
||||
```typescript
|
||||
const stats = await brain.getStats()
|
||||
// Returns comprehensive metrics:
|
||||
{
|
||||
nouns: {
|
||||
count: number,
|
||||
created: number,
|
||||
updated: number,
|
||||
deleted: number,
|
||||
size: number,
|
||||
avgSize: number
|
||||
},
|
||||
verbs: {
|
||||
count: number,
|
||||
created: number,
|
||||
types: Record<string, number>,
|
||||
weights: { min, max, avg }
|
||||
},
|
||||
vectors: {
|
||||
dimensions: 384,
|
||||
indexSize: number,
|
||||
partitions: number,
|
||||
avgSearchTime: number
|
||||
},
|
||||
cache: {
|
||||
hits: number,
|
||||
misses: number,
|
||||
evictions: number,
|
||||
hitRate: number,
|
||||
hotCacheSize: number,
|
||||
warmCacheSize: number
|
||||
},
|
||||
performance: {
|
||||
operations: number,
|
||||
avgAddTime: number,
|
||||
avgSearchTime: number,
|
||||
avgUpdateTime: number,
|
||||
p95Latency: number,
|
||||
p99Latency: number
|
||||
},
|
||||
storage: {
|
||||
used: number,
|
||||
available: number,
|
||||
compression: number,
|
||||
files: number
|
||||
},
|
||||
throttling: {
|
||||
delays: number,
|
||||
rateLimited: number,
|
||||
backoffMs: number,
|
||||
retries: number
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 GPU Support (Partial but Real!)
|
||||
|
||||
```typescript
|
||||
// GPU detection WORKS:
|
||||
const device = await detectBestDevice()
|
||||
// Returns: 'cpu' | 'webgpu' | 'cuda'
|
||||
|
||||
// WebGPU support in browser:
|
||||
if (device === 'webgpu') {
|
||||
// Transformer models can use WebGPU
|
||||
}
|
||||
|
||||
// CUDA detection in Node:
|
||||
if (device === 'cuda') {
|
||||
// Future: GPU acceleration support
|
||||
}
|
||||
```
|
||||
|
||||
## 🔄 Adaptive Systems (All Working!)
|
||||
|
||||
### Adaptive Backpressure
|
||||
```typescript
|
||||
const backpressure = new AdaptiveBackpressure()
|
||||
// Automatically adjusts flow based on system load
|
||||
```
|
||||
|
||||
### Adaptive Socket Manager
|
||||
```typescript
|
||||
const socketManager = new AdaptiveSocketManager()
|
||||
// Dynamic connection pooling based on traffic
|
||||
```
|
||||
|
||||
### Cache Auto-Configuration
|
||||
```typescript
|
||||
const cacheConfig = await getCacheAutoConfig()
|
||||
// Sizes cache based on available memory
|
||||
```
|
||||
|
||||
### S3 Throttling Protection
|
||||
```typescript
|
||||
// Built into S3 storage adapter
|
||||
// Automatic exponential backoff
|
||||
// Rate limit detection and adaptation
|
||||
```
|
||||
|
||||
## 🎨 How to Use Hidden Features
|
||||
|
||||
### Enable Reader / Writer Modes
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
mode: 'reader' // or 'writer' or 'hybrid'
|
||||
})
|
||||
```
|
||||
|
||||
### Use Neural Import
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
new NeuralImportAugmentation({
|
||||
confidenceThreshold: 0.7,
|
||||
autoDetect: true
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// Import with AI understanding
|
||||
await brain.neuralImport('data.csv')
|
||||
```
|
||||
|
||||
### Access Statistics
|
||||
```typescript
|
||||
// Get comprehensive stats
|
||||
const stats = await brain.getStats()
|
||||
|
||||
// Get specific service stats
|
||||
const nounStats = await brain.getStatistics({
|
||||
service: 'nouns'
|
||||
})
|
||||
|
||||
// Force refresh
|
||||
const freshStats = await brain.getStatistics({
|
||||
forceRefresh: true
|
||||
})
|
||||
```
|
||||
|
||||
## 📝 What Needs Documentation
|
||||
|
||||
These features EXIST but need better docs:
|
||||
1. Reader / writer operation modes
|
||||
2. Neural import full API
|
||||
3. 3-level cache configuration
|
||||
4. Performance monitoring API
|
||||
5. GPU acceleration setup
|
||||
6. Advanced statistics queries
|
||||
7. Throttling configuration
|
||||
8. Backpressure tuning
|
||||
|
||||
## 💡 The Truth
|
||||
|
||||
Brainy is MORE powerful than its own documentation suggests! Most "missing" features are actually implemented but hidden or not properly exposed. The codebase contains sophisticated systems for:
|
||||
- Reader / writer operation modes
|
||||
- AI-powered import
|
||||
- Advanced caching
|
||||
- Performance monitoring
|
||||
- GPU support
|
||||
- Adaptive optimization
|
||||
|
||||
The main work needed is integration and documentation, not implementation!
|
||||
494
docs/architecture/augmentations.md
Normal file
494
docs/architecture/augmentations.md
Normal file
|
|
@ -0,0 +1,494 @@
|
|||
# Augmentations System
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy's Augmentation System provides a powerful plugin architecture that extends core functionality without modifying the base code. Augmentations can intercept, modify, and enhance any operation in the database.
|
||||
|
||||
## Built-in Augmentations
|
||||
|
||||
> **Note**: This document shows both available and planned augmentations. Each section is marked with its current status.
|
||||
|
||||
### 1. Entity Registry Augmentation ✅ Available
|
||||
|
||||
High-performance deduplication for streaming data ingestion.
|
||||
|
||||
```typescript
|
||||
import { EntityRegistryAugmentation } from 'brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
new EntityRegistryAugmentation({
|
||||
maxCacheSize: 100000, // Track up to 100k unique entities
|
||||
ttl: 3600000, // 1-hour TTL for cache entries
|
||||
hashFields: ['id', 'url'] // Fields to use for deduplication
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// Automatically prevents duplicate entities
|
||||
await brain.add("Same content", { id: "123" }) // Added
|
||||
await brain.add("Same content", { id: "123" }) // Skipped (duplicate)
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- O(1) duplicate detection using bloom filters
|
||||
- Configurable cache size and TTL
|
||||
- Custom hash field selection
|
||||
- Perfect for real-time data streams
|
||||
|
||||
|
||||
Enterprise-grade durability and crash recovery.
|
||||
|
||||
```typescript
|
||||
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
checkpointInterval: 1000, // Checkpoint every 1000 operations
|
||||
compression: true, // Enable log compression
|
||||
maxLogSize: 100 * 1024 * 1024 // 100MB max log size
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// All operations are now durably logged
|
||||
|
||||
// Recover from crash
|
||||
const recovered = new Brainy({
|
||||
})
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- ACID compliance
|
||||
- Automatic crash recovery
|
||||
- Point-in-time recovery
|
||||
- Log compression and rotation
|
||||
- Minimal performance impact
|
||||
|
||||
### 3. Intelligent Verb Scoring Augmentation ✅ Available
|
||||
|
||||
AI-powered relationship strength calculation.
|
||||
|
||||
```typescript
|
||||
import { IntelligentVerbScoringAugmentation } from 'brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
new IntelligentVerbScoringAugmentation({
|
||||
factors: {
|
||||
semantic: 0.4, // Weight for semantic similarity
|
||||
temporal: 0.3, // Weight for time proximity
|
||||
frequency: 0.2, // Weight for interaction frequency
|
||||
explicit: 0.1 // Weight for explicit ratings
|
||||
}
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// Relationships automatically get intelligent scores
|
||||
await brain.relate(user1, product1, "viewed", { timestamp: Date.now() })
|
||||
await brain.relate(user1, product1, "purchased", { timestamp: Date.now() })
|
||||
// Automatically calculates relationship strength based on multiple factors
|
||||
|
||||
// Query using intelligent scores
|
||||
const strongRelationships = await brain.find({
|
||||
connected: {
|
||||
from: user1,
|
||||
minScore: 0.8 // Only highly relevant relationships
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Capabilities:**
|
||||
- Multi-factor relationship scoring
|
||||
- Temporal decay functions
|
||||
- Semantic similarity integration
|
||||
- Customizable weight factors
|
||||
|
||||
### 4. Auto-Register Entities Augmentation ⚠️ Basic Implementation
|
||||
|
||||
Automatically extracts and registers entities from text.
|
||||
|
||||
```typescript
|
||||
import { AutoRegisterEntitiesAugmentation } from 'brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
new AutoRegisterEntitiesAugmentation({
|
||||
types: ['person', 'organization', 'location', 'product'],
|
||||
confidence: 0.8,
|
||||
createRelationships: true
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// Automatically extracts and registers entities
|
||||
await brain.add(
|
||||
"Apple CEO Tim Cook announced the new iPhone 15 in Cupertino",
|
||||
{ type: "news" }
|
||||
)
|
||||
// Automatically creates:
|
||||
// - Noun: "Tim Cook" (person)
|
||||
// - Noun: "Apple" (organization)
|
||||
// - Noun: "iPhone 15" (product)
|
||||
// - Noun: "Cupertino" (location)
|
||||
// - Verbs: relationships between entities
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- NER (Named Entity Recognition)
|
||||
- Automatic relationship inference
|
||||
- Configurable entity types
|
||||
- Confidence thresholds
|
||||
|
||||
### 5. Batch Processing Augmentation ✅ Available
|
||||
|
||||
Optimizes bulk operations for maximum throughput.
|
||||
|
||||
```typescript
|
||||
import { BatchProcessingAugmentation } from 'brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
new BatchProcessingAugmentation({
|
||||
batchSize: 100,
|
||||
flushInterval: 1000, // Flush every second
|
||||
parallel: true, // Parallel processing
|
||||
maxQueueSize: 10000
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// Operations are automatically batched
|
||||
for (let i = 0; i < 10000; i++) {
|
||||
await brain.add(`Item ${i}`) // Internally batched
|
||||
}
|
||||
// Processes in optimized batches of 100
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- 10-100x throughput improvement
|
||||
- Automatic batching
|
||||
- Configurable batch sizes
|
||||
- Memory-efficient queue management
|
||||
|
||||
### 6. Caching Augmentation 🚧 Coming Soon
|
||||
|
||||
Intelligent multi-level caching system.
|
||||
|
||||
```typescript
|
||||
import { CachingAugmentation } from 'brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
new CachingAugmentation({
|
||||
levels: {
|
||||
l1: { size: 100, ttl: 60000 }, // Hot cache: 100 items, 1 min
|
||||
l2: { size: 1000, ttl: 300000 }, // Warm cache: 1000 items, 5 min
|
||||
l3: { size: 10000, ttl: 3600000 } // Cold cache: 10k items, 1 hour
|
||||
},
|
||||
strategies: ['lru', 'lfu'], // Least Recently/Frequently Used
|
||||
preload: true // Preload popular items
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// Queries automatically use cache
|
||||
const results = await brain.find("popular query") // Cached
|
||||
const again = await brain.find("popular query") // From cache (instant)
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Multi-level cache hierarchy
|
||||
- Multiple eviction strategies
|
||||
- Query result caching
|
||||
- Embedding cache
|
||||
- Automatic cache invalidation
|
||||
|
||||
### 7. Compression Augmentation 🚧 Coming Soon
|
||||
|
||||
Reduces storage size while maintaining query performance.
|
||||
|
||||
```typescript
|
||||
import { CompressionAugmentation } from 'brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
new CompressionAugmentation({
|
||||
algorithm: 'brotli',
|
||||
level: 6, // Compression level (1-11)
|
||||
threshold: 1024, // Only compress items > 1KB
|
||||
excludeFields: ['id', 'type'] // Don't compress these
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// Data automatically compressed/decompressed
|
||||
await brain.add(largeDocument) // Compressed before storage
|
||||
const doc = await brain.getNoun(id) // Decompressed on retrieval
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- 60-80% storage reduction
|
||||
- Transparent compression
|
||||
- Selective field compression
|
||||
- Multiple algorithm support
|
||||
|
||||
### 8. Monitoring Augmentation 🚧 Coming Soon
|
||||
|
||||
Real-time performance monitoring and metrics.
|
||||
|
||||
```typescript
|
||||
import { MonitoringAugmentation } from 'brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
new MonitoringAugmentation({
|
||||
metrics: ['operations', 'latency', 'cache', 'memory'],
|
||||
interval: 5000, // Report every 5 seconds
|
||||
webhook: 'https://metrics.example.com/brainy',
|
||||
console: true // Also log to console
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// Automatic metric collection
|
||||
brain.on('metrics', (metrics) => {
|
||||
console.log(`
|
||||
Operations/sec: ${metrics.opsPerSecond}
|
||||
Avg latency: ${metrics.avgLatency}ms
|
||||
Cache hit rate: ${metrics.cacheHitRate}%
|
||||
Memory usage: ${metrics.memoryMB}MB
|
||||
`)
|
||||
})
|
||||
```
|
||||
|
||||
**Metrics:**
|
||||
- Operation throughput
|
||||
- Query latency percentiles
|
||||
- Cache hit rates
|
||||
- Memory usage
|
||||
- Storage growth
|
||||
- Error rates
|
||||
|
||||
## Neural Import Capabilities 🚧 Coming Soon
|
||||
|
||||
> **Note**: Import/Export features are currently in development. Expected Q1 2025.
|
||||
|
||||
### 1. Document Import with Auto-Structuring
|
||||
|
||||
```typescript
|
||||
import { NeuralImportAugmentation } from 'brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
new NeuralImportAugmentation({
|
||||
autoStructure: true,
|
||||
extractEntities: true,
|
||||
generateSummaries: true,
|
||||
detectLanguage: true
|
||||
})
|
||||
]
|
||||
})
|
||||
|
||||
// Import unstructured documents
|
||||
await brain.importDocument('./research-paper.pdf')
|
||||
// Automatically:
|
||||
// - Extracts text and metadata
|
||||
// - Identifies sections and structure
|
||||
// - Extracts entities and concepts
|
||||
// - Generates embeddings per section
|
||||
// - Creates relationship graph
|
||||
```
|
||||
|
||||
### 2. Database Migration Import
|
||||
|
||||
```typescript
|
||||
// Import from existing databases
|
||||
await brain.importFromSQL({
|
||||
connection: 'postgres://localhost/mydb',
|
||||
tables: {
|
||||
users: { type: 'person', idField: 'user_id' },
|
||||
products: { type: 'product', idField: 'sku' },
|
||||
orders: {
|
||||
type: 'relationship',
|
||||
from: 'user_id',
|
||||
to: 'product_id',
|
||||
verb: 'purchased'
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Import from MongoDB
|
||||
await brain.importFromMongo({
|
||||
uri: 'mongodb://localhost:27017',
|
||||
database: 'myapp',
|
||||
collections: {
|
||||
users: { type: 'person' },
|
||||
posts: { type: 'content' }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Stream Import
|
||||
|
||||
```typescript
|
||||
// Import from real-time streams
|
||||
await brain.importStream({
|
||||
source: 'kafka://localhost:9092/events',
|
||||
format: 'json',
|
||||
transform: (event) => ({
|
||||
noun: event.data,
|
||||
metadata: {
|
||||
type: event.type,
|
||||
timestamp: event.timestamp
|
||||
}
|
||||
}),
|
||||
deduplication: true
|
||||
})
|
||||
```
|
||||
|
||||
### 4. Bulk CSV/JSON Import
|
||||
|
||||
```typescript
|
||||
// Import CSV with automatic type detection
|
||||
await brain.importCSV('./data.csv', {
|
||||
headers: true,
|
||||
typeColumn: 'entity_type',
|
||||
detectRelationships: true,
|
||||
batchSize: 1000
|
||||
})
|
||||
|
||||
// Import JSON with nested structure handling
|
||||
await brain.importJSON('./data.json', {
|
||||
rootPath: '$.entities',
|
||||
nounPath: '$.content',
|
||||
metadataPath: '$.properties',
|
||||
relationshipPath: '$.connections'
|
||||
})
|
||||
```
|
||||
|
||||
## Creating Custom Augmentations
|
||||
|
||||
```typescript
|
||||
import { Augmentation } from 'brainy'
|
||||
|
||||
class CustomAugmentation extends Augmentation {
|
||||
name = 'CustomAugmentation'
|
||||
|
||||
async onInit(brain: Brainy): Promise<void> {
|
||||
// Initialize augmentation
|
||||
console.log('Custom augmentation initialized')
|
||||
}
|
||||
|
||||
async onBeforeAddNoun(content: any, metadata: any): Promise<[any, any]> {
|
||||
// Modify before adding noun
|
||||
metadata.processed = true
|
||||
metadata.timestamp = Date.now()
|
||||
return [content, metadata]
|
||||
}
|
||||
|
||||
async onAfterAddNoun(id: string, noun: any): Promise<void> {
|
||||
// React to noun addition
|
||||
console.log(`Noun ${id} added`)
|
||||
}
|
||||
|
||||
async onBeforeSearch(query: any): Promise<any> {
|
||||
// Modify search query
|
||||
query.boost = 'recent'
|
||||
return query
|
||||
}
|
||||
|
||||
async onAfterSearch(results: any[]): Promise<any[]> {
|
||||
// Process search results
|
||||
return results.map(r => ({
|
||||
...r,
|
||||
customScore: r.score * 1.5
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
// Use custom augmentation
|
||||
const brain = new Brainy({
|
||||
augmentations: [new CustomAugmentation()]
|
||||
})
|
||||
```
|
||||
|
||||
## Augmentation Lifecycle Hooks
|
||||
|
||||
### Available Hooks
|
||||
|
||||
```typescript
|
||||
interface AugmentationHooks {
|
||||
// Initialization
|
||||
onInit(brain: Brainy): Promise<void>
|
||||
onShutdown(): Promise<void>
|
||||
|
||||
// Noun operations
|
||||
onBeforeAddNoun(content, metadata): Promise<[content, metadata]>
|
||||
onAfterAddNoun(id, noun): Promise<void>
|
||||
onBeforeGetNoun(id): Promise<string>
|
||||
onAfterGetNoun(noun): Promise<any>
|
||||
onBeforeUpdateNoun(id, updates): Promise<[string, any]>
|
||||
onAfterUpdateNoun(id, noun): Promise<void>
|
||||
onBeforeDeleteNoun(id): Promise<string>
|
||||
onAfterDeleteNoun(id): Promise<void>
|
||||
|
||||
// Verb operations
|
||||
onBeforeAddVerb(source, target, type, metadata): Promise<[any, any, string, any]>
|
||||
onAfterAddVerb(id, verb): Promise<void>
|
||||
onBeforeGetVerb(id): Promise<string>
|
||||
onAfterGetVerb(verb): Promise<any>
|
||||
|
||||
// Search operations
|
||||
onBeforeSearch(query): Promise<any>
|
||||
onAfterSearch(results): Promise<any[]>
|
||||
onBeforeFind(query): Promise<any>
|
||||
onAfterFind(results): Promise<any[]>
|
||||
|
||||
// Storage operations
|
||||
onBeforeSave(data): Promise<any>
|
||||
onAfterLoad(data): Promise<any>
|
||||
|
||||
// Events
|
||||
onError(error): Promise<void>
|
||||
onMetric(metric): Promise<void>
|
||||
}
|
||||
```
|
||||
|
||||
## Augmentation Composition
|
||||
|
||||
```typescript
|
||||
// Combine multiple augmentations
|
||||
const brain = new Brainy({
|
||||
augmentations: [
|
||||
// Order matters - executed in sequence
|
||||
new EntityRegistryAugmentation(), // Deduplication first
|
||||
new AutoRegisterEntitiesAugmentation(), // Entity extraction
|
||||
new IntelligentVerbScoringAugmentation(), // Scoring
|
||||
new CompressionAugmentation(), // Compression
|
||||
new CachingAugmentation(), // Caching
|
||||
new MonitoringAugmentation() // Monitoring last
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
1. **Order Matters**: Place filtering augmentations early
|
||||
2. **Resource Usage**: Monitor memory with many augmentations
|
||||
3. **Async Operations**: Use parallel processing where possible
|
||||
4. **Caching**: Enable caching augmentation for read-heavy workloads
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Single Responsibility**: Each augmentation should do one thing well
|
||||
2. **Non-Blocking**: Avoid blocking operations in hooks
|
||||
3. **Error Handling**: Always handle errors gracefully
|
||||
4. **Configuration**: Make augmentations configurable
|
||||
5. **Documentation**: Document augmentation behavior and options
|
||||
|
||||
## See Also
|
||||
|
||||
- [Architecture Overview](./overview.md)
|
||||
- [API Reference](../api/README.md)
|
||||
- [Performance Guide](../guides/performance.md)
|
||||
388
docs/architecture/data-storage-architecture.md
Normal file
388
docs/architecture/data-storage-architecture.md
Normal file
|
|
@ -0,0 +1,388 @@
|
|||
# Brainy Data Storage Architecture (8.0)
|
||||
|
||||
**Complete on-disk reference for the 8.0 layout.**
|
||||
|
||||
This document describes what a Brainy 8.0 data directory actually contains: the
|
||||
canonical entity records, the system area, the generational-MVCC bookkeeping,
|
||||
the column store, the blob area, and the lock files — plus how the in-memory
|
||||
indexes rebuild from them. The authoritative design records are
|
||||
[ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and
|
||||
[index-architecture.md](./index-architecture.md); this document is the on-disk
|
||||
map that ties them together.
|
||||
|
||||
8.0 removed the 7.x copy-on-write subsystem (`_cow/`, `branches/{branch}/…`
|
||||
paths) and the cloud/OPFS storage adapters. The two storage backends are
|
||||
**filesystem** and **memory**; both speak the same path vocabulary (memory
|
||||
storage keys its internal map by the identical path strings).
|
||||
|
||||
---
|
||||
|
||||
## 1. Directory Tree
|
||||
|
||||
A real 8.0 filesystem store (`path`, default `./brainy-data`):
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
│
|
||||
├── entities/ # Canonical records (current state)
|
||||
│ ├── nouns/
|
||||
│ │ └── {shard}/ # 256 shards: first 2 hex chars of the UUID
|
||||
│ │ └── {id}/ # One directory per entity UUID
|
||||
│ │ ├── vectors.json.gz # Embedding + HNSW node state
|
||||
│ │ └── metadata.json.gz # Everything else (type, subtype, data, fields, _rev)
|
||||
│ └── verbs/
|
||||
│ └── {shard}/
|
||||
│ └── {id}/
|
||||
│ ├── vectors.json.gz # Relationship embedding (when present)
|
||||
│ └── metadata.json.gz # sourceId, targetId, verb, subtype, weight, data…
|
||||
│
|
||||
├── _system/ # System singletons + bucketed system keys
|
||||
│ ├── generation.json(.gz) # { generation, updatedAt } — the write watermark
|
||||
│ ├── manifest.json # { version, generation, … } — MVCC commit point
|
||||
│ ├── tx-log.jsonl # One line per committed transact() batch (append-only)
|
||||
│ ├── counts.json # Entity/verb totals
|
||||
│ ├── type-statistics.json.gz # Per-NounType counts
|
||||
│ ├── subtype-statistics.json.gz # Per-(NounType, subtype) counts
|
||||
│ ├── verb-subtype-statistics.json.gz # Per-(VerbType, subtype) counts
|
||||
│ ├── statistics.json # Aggregate statistics blob (counts, index sizes)
|
||||
│ ├── hnsw-system.json # Vector-index entry point + max level
|
||||
│ ├── __metadata_field_registry__.json.gz # Which metadata fields are indexed
|
||||
│ ├── brainy:entityIdMapper.json.gz # UUID ↔ u64 mapping for native index providers
|
||||
│ └── idx/
|
||||
│ └── {bucket}/ # 256 buckets: FNV-1a hash of the key
|
||||
│ ├── __metadata_field_index__field_{name}.json.gz # Sparse field indexes
|
||||
│ ├── __chunk__*.json.gz # Metadata-index roaring-bitmap chunks
|
||||
│ ├── __sparse_index__*.json.gz# Zone maps + bloom filters
|
||||
│ └── graph-lsm-verbs-{source|target}-*.json.gz # Graph LSM SSTables + manifest
|
||||
│
|
||||
├── _generations/ # MVCC history (written ONLY by transact())
|
||||
│ └── {N}/ # One directory per committed generation N
|
||||
│ ├── tx.json # The generation-N delta (immutable)
|
||||
│ └── prev/
|
||||
│ └── {id}.json # Before-image of each touched record (immutable)
|
||||
│
|
||||
├── _column_index/ # Column store manifests (one dir per field)
|
||||
│ └── {field}/
|
||||
│ └── MANIFEST.json.gz # Run list + zone metadata for that column
|
||||
│
|
||||
├── _blobs/ # Binary blob area (`<key>.bin` convention)
|
||||
│ ├── _column_index/
|
||||
│ │ └── {field}/
|
||||
│ │ └── L0-000001.bin # Column-store runs (level-0 segments)
|
||||
│ └── … # VFS file content and other binary blobs
|
||||
│
|
||||
└── locks/ # Process coordination (NEVER snapshotted)
|
||||
├── _writer.lock # Single-writer lock: pid, hostname, heartbeat
|
||||
├── _flush_requests/ # Reader→writer flush RPC (.req files)
|
||||
└── _flush_responses/ # Writer acks (.ack files)
|
||||
```
|
||||
|
||||
Most JSON objects are gzip-compressed (`.json.gz`) — compression is on by
|
||||
default for filesystem storage (`storage.options.compression`, zlib level 6).
|
||||
A few hot singletons (`manifest.json`, `counts.json`, `hnsw-system.json`,
|
||||
`tx-log.jsonl`) are written uncompressed for cheap partial reads and appends.
|
||||
|
||||
---
|
||||
|
||||
## 2. Canonical Entity Records
|
||||
|
||||
Each entity (noun) and relationship (verb) is **two files** under one
|
||||
ID-first directory. The split keeps vector I/O (large, append-mostly) separate
|
||||
from metadata I/O (small, read-heavy).
|
||||
|
||||
### Noun vector file — `entities/nouns/{shard}/{id}/vectors.json.gz`
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "421d92e7-4241-470a-80f4-4b39414e7a83",
|
||||
"vector": [-0.139, -0.056, 0.028, "…384 dims…"],
|
||||
"connections": { "0": ["neighbor-uuid…"] },
|
||||
"level": 0
|
||||
}
|
||||
```
|
||||
|
||||
The HNSW node state (`connections`, `level`) is persisted with the vector so
|
||||
the vector index can rebuild without recomputing the graph.
|
||||
|
||||
### Noun metadata file — `entities/nouns/{shard}/{id}/metadata.json.gz`
|
||||
|
||||
```json
|
||||
{
|
||||
"data": "React is a JavaScript library for building user interfaces",
|
||||
"noun": "concept",
|
||||
"subtype": "cli-add",
|
||||
"createdAt": 1781198053726,
|
||||
"updatedAt": 1781198053726,
|
||||
"_rev": 1
|
||||
}
|
||||
```
|
||||
|
||||
- `noun` is the NounType. **Type lives in metadata, not in the path** — lookup
|
||||
by ID is a single path construction, no type needed.
|
||||
- `subtype` is the per-product sub-classification (required on write by
|
||||
default in 8.0).
|
||||
- `_rev` increments on every write and backs `update({ ifRev })` CAS.
|
||||
- Consumer metadata fields sit alongside the standard ones.
|
||||
|
||||
### Verb files — `entities/verbs/{shard}/{id}/…`
|
||||
|
||||
Same two-file split. The verb metadata record carries the graph edge:
|
||||
`sourceId`, `targetId`, `verb` (VerbType), `subtype`, `weight`, `data`,
|
||||
`metadata`, timestamps, `_rev`. Verb IDs are Brainy-generated UUIDs by
|
||||
contract (8.0 rejects caller-supplied verb ids) because native graph providers
|
||||
intern the raw UUID bytes as u64 handles.
|
||||
|
||||
### Path construction
|
||||
|
||||
```typescript
|
||||
const shard = id.substring(0, 2) // '42'
|
||||
const metadataPath = `entities/nouns/${shard}/${id}/metadata.json`
|
||||
const vectorsPath = `entities/nouns/${shard}/${id}/vectors.json`
|
||||
// Verbs: same shape under entities/verbs/
|
||||
```
|
||||
|
||||
Implemented in `src/storage/baseStorage.ts` (path generators) and
|
||||
`src/storage/sharding.ts` (`getShardIdFromUuid`).
|
||||
|
||||
---
|
||||
|
||||
## 3. The `_system/` Area
|
||||
|
||||
Two kinds of keys live here, resolved by `BaseStorage.parsePath()`:
|
||||
|
||||
1. **Singletons** — well-known keys written at `_system/<key>.json`:
|
||||
`counts`, `statistics`, `type-statistics`, `hnsw-system`,
|
||||
`__metadata_field_registry__`, `brainy:entityIdMapper`, plus the MVCC
|
||||
trio (`generation.json`, `manifest.json`, `tx-log.jsonl`).
|
||||
2. **Bucketed system keys** — everything else (field indexes, bitmap chunks,
|
||||
sparse-index segments, graph LSM SSTables) hashes into one of 256
|
||||
`_system/idx/{bucket}/` directories via FNV-1a, so no single directory
|
||||
accumulates unbounded entries.
|
||||
|
||||
Notable singletons:
|
||||
|
||||
| File | Contents |
|
||||
|------|----------|
|
||||
| `generation.json` | `{ generation, updatedAt }` — monotonic watermark, bumped by **every** write batch |
|
||||
| `manifest.json` | MVCC commit point: highest *committed* generation (see §4) |
|
||||
| `tx-log.jsonl` | One JSON line per committed `transact()` batch: generation, timestamp, `meta` |
|
||||
| `type-statistics.json.gz` | Per-NounType counts (backs `brain.counts.byType`) |
|
||||
| `subtype-statistics.json.gz` | `{ counts: { [type]: { [subtype]: n } }, updatedAt }` (contract-bound shape) |
|
||||
| `verb-subtype-statistics.json.gz` | Same shape for VerbTypes |
|
||||
| `hnsw-system.json` | `{ entryPointId, maxLevel }` — per-node state lives in each entity's `vectors.json` |
|
||||
| `brainy:entityIdMapper.json.gz` | UUID ↔ u64 interning table for the BigInt provider contract |
|
||||
| `__metadata_field_registry__.json.gz` | Registry of indexed metadata field names |
|
||||
|
||||
---
|
||||
|
||||
## 4. Generational MVCC (`_generations/` + the `_system` trio)
|
||||
|
||||
Full design in [ADR-001](../ADR-001-generational-mvcc.md). The on-disk shape:
|
||||
|
||||
```
|
||||
_system/generation.json { generation, updatedAt } atomic tmp+rename
|
||||
_system/manifest.json { version, generation, … } atomic tmp+rename — THE commit point
|
||||
_system/tx-log.jsonl one line per committed transact() append-only
|
||||
_generations/{N}/tx.json the generation-N delta immutable once written
|
||||
_generations/{N}/prev/{id}.json before-image of {id} immutable once written
|
||||
```
|
||||
|
||||
Commit protocol (writer side): stage before-images and the delta under
|
||||
`_generations/N/`, fsync, apply the delta to the canonical `entities/…`
|
||||
records, then atomically rename `manifest.json` to publish generation N. The
|
||||
tx-log line is appended last (advisory). Crash recovery on open discards any
|
||||
`_generations/{N}` newer than the manifest.
|
||||
|
||||
Two write classes share the generation clock:
|
||||
|
||||
- **Single-operation writes** (`add`/`update`/`remove`/`relate` outside
|
||||
`transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound,
|
||||
but write **no** history — they are not visible to `db.since()` and remain
|
||||
visible through earlier pins.
|
||||
- **`transact()` batches** write the full `_generations/{N}` record and a
|
||||
tx-log line, and are the unit of time travel (`brain.asOf()`).
|
||||
|
||||
Snapshots (`db.persist(path)`) hard-link the entire store **except `locks/`**
|
||||
into a self-contained directory openable via `Brainy.load(path)`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Column Store (`_column_index/` + `_blobs/_column_index/`)
|
||||
|
||||
The metadata index persists per-field columnar runs for O(log n) range and
|
||||
membership queries at scale:
|
||||
|
||||
- `_column_index/{field}/MANIFEST.json.gz` — the run list and zone metadata
|
||||
for one field (`createdAt`, `subtype`, `noun`, `_rev`, consumer fields,
|
||||
`__words__` for tokenized text…).
|
||||
- `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run
|
||||
segments, stored through the shared `_blobs/<key>.bin` binary convention.
|
||||
|
||||
Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments
|
||||
additionally live as bucketed keys under `_system/idx/` (see §3). Which path
|
||||
serves a given `where` clause is the query planner's decision — inspect it
|
||||
with `brainy inspect explain <dir> --where '…'`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Blob Area (`_blobs/`)
|
||||
|
||||
`_blobs/<key>.bin` is the flat binary-blob convention shared by every storage
|
||||
adapter (`saveBinaryBlob`/`getBinaryBlob` in the storage contract):
|
||||
|
||||
- **VFS file content** — VFS entities are regular nouns (path, ownership, and
|
||||
timestamps in entity metadata); the file *bytes* are blobs.
|
||||
- **Column-store runs** (under the `_column_index/` key prefix, §5).
|
||||
- Any other binary payload an index provider persists.
|
||||
|
||||
Writes use unique temp names + rename, so concurrent writers of the same key
|
||||
cannot tear each other's blobs.
|
||||
|
||||
---
|
||||
|
||||
## 7. Locks (`locks/`)
|
||||
|
||||
```
|
||||
locks/_writer.lock # single-writer lock: { pid, hostname, startedAt, heartbeat, version }
|
||||
locks/_flush_requests/ # readers drop <uuid>.req to ask the writer to flush
|
||||
locks/_flush_responses/ # writer answers with <uuid>.ack
|
||||
```
|
||||
|
||||
- One **writer** per data directory, enforced at `init()`; stale locks (dead
|
||||
PID / stale heartbeat) are reclaimed automatically.
|
||||
- Read-only processes (`Brainy.openReadOnly()`, the `brainy inspect` CLI
|
||||
family) can ask the live writer to flush via the request/response files, so
|
||||
out-of-process diagnostics see fresh state.
|
||||
- `locks/` is excluded from snapshots (`SNAPSHOT_EXCLUDED_TOP_DIRS` in
|
||||
`src/storage/adapters/fileSystemStorage.ts`).
|
||||
|
||||
---
|
||||
|
||||
## 8. In-Memory Indexes and What Rebuilds From What
|
||||
|
||||
| Index | In memory | Persisted state | Rebuild source |
|
||||
|-------|-----------|-----------------|----------------|
|
||||
| **Vector (HNSW)** | Graph of vector connections | `_system/hnsw-system.json` + per-entity `vectors.json` | Walk entity vector files; lazy mode loads structure only and pages vectors on demand |
|
||||
| **Metadata index** | Field → value bitmaps + column-store readers | `_system/idx/` chunks + `_column_index/` manifests + `_blobs/_column_index/` runs | Loaded directly; full rebuild re-scans entity metadata |
|
||||
| **Graph adjacency** | sourceId/targetId → verb-id LSM trees | `graph-lsm-verbs-{source,target}-*` SSTables under `_system/idx/` | Loaded from SSTables; full rebuild re-scans verb metadata |
|
||||
| **Counts/statistics** | Per-type and per-subtype maps | `_system/{type,subtype,verb-subtype}-statistics.json.gz`, `counts.json` | Recomputable by scanning entities (`brainy inspect repair`) |
|
||||
|
||||
A pluggable index provider (the 8.0 plugin contract in
|
||||
`@soulcraft/brainy/plugin`) may replace any of the JS implementations; the
|
||||
persisted formats above are contract-bound so JS and native implementations
|
||||
can interleave on the same directory.
|
||||
|
||||
---
|
||||
|
||||
## 9. Sharding Strategy
|
||||
|
||||
**Entities:** first 2 hex characters of the UUID → 256 uniform shards.
|
||||
Deterministic, configuration-free, and keeps per-directory entry counts low
|
||||
(at 1M entities: ~3,900 directories per shard). Paginated whole-store walks
|
||||
(`getNouns`/`getVerbs`) iterate shards `00`–`ff` in order.
|
||||
|
||||
**System keys:** FNV-1a hash of the key → 256 `_system/idx/` buckets. Same
|
||||
motivation, different keyspace (system keys are not UUIDs).
|
||||
|
||||
**What is never sharded:** the `_system/` singletons, `_generations/{N}`
|
||||
directories (keyed by generation number), `_column_index/{field}` manifests
|
||||
(keyed by field name), and `locks/`.
|
||||
|
||||
---
|
||||
|
||||
## 10. Durability and Atomicity
|
||||
|
||||
- **Per-object atomicity:** every JSON object and blob is written to a unique
|
||||
temp file then `rename()`d — readers never observe torn objects.
|
||||
- **Transaction atomicity:** the `manifest.json` rename is the single commit
|
||||
point for `transact()` batches (§4); everything staged before it is
|
||||
discarded by crash recovery if the rename never lands.
|
||||
- **Compression:** gzip per object (`.json.gz`), transparent to all readers.
|
||||
Native index providers that mmap binary formats use the uncompressed
|
||||
`_blobs/` area instead.
|
||||
|
||||
---
|
||||
|
||||
## 11. `clear()` Semantics
|
||||
|
||||
`brain.clear()` removes all entities, relationships, indexes, statistics, and
|
||||
MVCC history, then re-resolves every index exactly as `init()` does —
|
||||
including plugin-provided vector/metadata/id-mapper factories and VFS root
|
||||
re-creation. The data directory afterwards contains a fresh, empty store (the
|
||||
writer lock remains held by the running process).
|
||||
|
||||
---
|
||||
|
||||
## 12. Common Scenarios
|
||||
|
||||
### Adding an entity
|
||||
|
||||
```
|
||||
brain.add({ data, type, subtype })
|
||||
1. Generate UUID → shard = first 2 hex chars
|
||||
2. Embed data → 384-dim vector
|
||||
3. Write entities/nouns/{shard}/{id}/vectors.json.gz (vector + HNSW node state)
|
||||
4. Write entities/nouns/{shard}/{id}/metadata.json.gz (type/subtype/data/fields, _rev: 1)
|
||||
5. Update in-memory indexes (HNSW insert, metadata index, statistics)
|
||||
6. Bump _system/generation.json (no _generations/ entry — single-op write)
|
||||
```
|
||||
|
||||
### Committing a transaction
|
||||
|
||||
```
|
||||
await brain.transact(tx => { tx.add(…); tx.update(…) })
|
||||
1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync
|
||||
2. Apply the delta to canonical entities/… records
|
||||
3. Atomic-rename _system/manifest.json → generation N is committed
|
||||
4. Append one line to _system/tx-log.jsonl
|
||||
```
|
||||
|
||||
### Cold start
|
||||
|
||||
```
|
||||
await brain.init()
|
||||
1. Acquire locks/_writer.lock (or open read-only)
|
||||
2. Crash recovery: drop _generations/{N} newer than manifest.json
|
||||
3. Load _system singletons (counts, statistics, field registry, id mapper)
|
||||
4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large)
|
||||
5. Graph adjacency: load LSM SSTables from _system/idx/
|
||||
6. Metadata index: column-store manifests + bitmap chunks on demand
|
||||
```
|
||||
|
||||
### Snapshot and restore
|
||||
|
||||
```
|
||||
const db = brain.now(); await db.persist('/backups/today'); await db.release()
|
||||
→ hard-links everything except locks/ into a self-contained directory
|
||||
|
||||
await Brainy.load('/backups/today') // open snapshot read-only as a Db
|
||||
await brain.restore('/backups/today', { confirm: true }) // replace store state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13. Summary
|
||||
|
||||
- **Two backends** (filesystem, memory), one path vocabulary.
|
||||
- **Two files per entity** under ID-first `entities/{kind}/{shard}/{id}/`.
|
||||
- **Type and subtype are metadata**, not directory structure; type queries go
|
||||
through the metadata index, not the filesystem.
|
||||
- **`_system/`** holds singletons plus 256 hash buckets of index state.
|
||||
- **`_generations/` + `manifest.json` + `tx-log.jsonl`** implement
|
||||
generational MVCC. History is per-write: every `add()`/`update()`/`remove()`/
|
||||
`relate()` gets its own generation, and `transact()` groups several ops into
|
||||
one atomic generation. (Single-op retention has been the model since 8.0;
|
||||
you never need to route a write through `transact()` just to keep its history.)
|
||||
- **`_column_index/` + `_blobs/`** hold the columnar metadata runs and binary
|
||||
blobs (VFS content included).
|
||||
- **`locks/`** coordinates the single writer and reader flush requests, and
|
||||
never travels with snapshots.
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [ADR-001 — Generational MVCC](../ADR-001-generational-mvcc.md)
|
||||
- [Index Architecture](./index-architecture.md)
|
||||
- [Consistency Model](../concepts/consistency-model.md)
|
||||
- [VFS Guide](../vfs/README.md)
|
||||
538
docs/architecture/finite-type-system.md
Normal file
538
docs/architecture/finite-type-system.md
Normal file
|
|
@ -0,0 +1,538 @@
|
|||
# 🎯 Brainy's Finite Noun/Verb Type System
|
||||
|
||||
> **Why Brainy's Finite Type System is Revolutionary for Knowledge Graphs at Billion Scale**
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy introduces a **finite type system** that sits between traditional schemaless NoSQL and rigid relational databases. This approach unlocks unprecedented optimization opportunities while maintaining semantic flexibility.
|
||||
|
||||
---
|
||||
|
||||
## The Three-Way Comparison
|
||||
|
||||
### 1. Traditional NoSQL (Schemaless)
|
||||
|
||||
```typescript
|
||||
// Complete freedom, zero optimization
|
||||
{
|
||||
id: '123',
|
||||
randomField1: 'value',
|
||||
anotherWeirdKey: 42,
|
||||
whoKnowsWhatElse: { nested: 'chaos' }
|
||||
}
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- ❌ No index optimization possible
|
||||
- ❌ Tools can't understand data structure
|
||||
- ❌ Incompatible augmentations/extensions
|
||||
- ❌ Memory explosion with billions of unique keys
|
||||
- ❌ No semantic understanding
|
||||
- ❌ Query planning impossible
|
||||
|
||||
### 2. Traditional Relational (Rigid Schema)
|
||||
|
||||
```sql
|
||||
CREATE TABLE entities (
|
||||
id UUID PRIMARY KEY,
|
||||
field1 VARCHAR(255),
|
||||
field2 INTEGER,
|
||||
...
|
||||
field50 TEXT
|
||||
);
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- ❌ Must define schema upfront
|
||||
- ❌ Schema migrations are painful
|
||||
- ❌ Can't handle heterogeneous data
|
||||
- ❌ Requires restart for schema changes
|
||||
- ❌ Fixed columns waste space
|
||||
|
||||
### 3. Brainy's Finite Type System (Semantic Structure)
|
||||
|
||||
```typescript
|
||||
// Finite noun types (extensible but constrained)
|
||||
type NounType =
|
||||
| 'person' | 'place' | 'organization' | 'document'
|
||||
| 'event' | 'concept' | 'thing' | ...
|
||||
|
||||
// Finite verb types (semantic relationships)
|
||||
type VerbType =
|
||||
| 'relatedTo' | 'contains' | 'isA' | 'causedBy'
|
||||
| 'precedes' | 'influences' | ...
|
||||
|
||||
// Example usage
|
||||
const entity = {
|
||||
id: '123',
|
||||
nounType: 'person', // Finite! Known type
|
||||
vector: [...], // Semantic embedding
|
||||
metadata: {
|
||||
noun: 'person', // Required type field
|
||||
name: 'Alice', // Custom fields allowed
|
||||
occupation: 'Engineer' // Flexible metadata
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ **Index Optimization**: Fixed-size Uint32Arrays for type tracking (99.76% memory reduction)
|
||||
- ✅ **Semantic Understanding**: Types have meaning, not just structure
|
||||
- ✅ **Tool Compatibility**: All augmentations understand core types
|
||||
- ✅ **Concept Extraction**: NLP can map text to known types
|
||||
- ✅ **Explicit Types**: Clear type specification in API
|
||||
- ✅ **Query Optimization**: Type-aware query planning
|
||||
- ✅ **Flexible Metadata**: Any fields within typed structure
|
||||
- ✅ **Billion-Scale Ready**: Type tracking scales linearly
|
||||
|
||||
---
|
||||
|
||||
## Revolutionary Benefits in Detail
|
||||
|
||||
### 1. Index Optimization at Billion Scale
|
||||
|
||||
**The Problem**: Traditional NoSQL stores arbitrary field names in indexes:
|
||||
|
||||
```typescript
|
||||
// Memory explosion with unique keys
|
||||
Map<string, Set<string>> {
|
||||
"user_preference_notification_email_enabled": Set(['id1', 'id2', ...]),
|
||||
"customer_shipping_address_line_1": Set(['id3', 'id4', ...]),
|
||||
// Billions of unique, unpredictable keys!
|
||||
}
|
||||
```
|
||||
|
||||
**Brainy's Solution**: Fixed noun/verb types enable fixed-size tracking:
|
||||
|
||||
```typescript
|
||||
// 99.76% memory reduction with Uint32Arrays
|
||||
class TypeAwareMetadataIndex {
|
||||
// Fixed size: nounTypes × verbTypes × fieldCount
|
||||
private nounTypeBitmaps: RoaringBitmap32[] // One per noun type
|
||||
private verbTypeBitmaps: RoaringBitmap32[] // One per verb type
|
||||
|
||||
// Example: 100 noun types × 50 verb types = 5KB overhead
|
||||
// vs 500MB+ for arbitrary keys!
|
||||
}
|
||||
```
|
||||
|
||||
**Real-World Impact (PROJECTED - not yet benchmarked)**:
|
||||
- **Before**: 500MB memory for 1M entities with diverse keys
|
||||
- **After**: PROJECTED 1.2MB memory for same dataset (385x reduction - calculated from Uint32Array size, not measured)
|
||||
- **Scales to billions**: Memory grows with entity count, not key diversity
|
||||
|
||||
### 2. Explicit Type System
|
||||
|
||||
**The Design**: Specify types clearly in your API calls:
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
// Add entity with explicit type
|
||||
await brain.add({
|
||||
data: { name: 'Alice', role: 'CEO of Acme Corp' },
|
||||
type: NounType.Person // Explicit type specification
|
||||
})
|
||||
|
||||
// Query with type filtering
|
||||
await brain.find({
|
||||
query: 'Alice',
|
||||
type: NounType.Person // Type-optimized search
|
||||
})
|
||||
```
|
||||
|
||||
**Why Explicit Types?**:
|
||||
1. **Deterministic**: You control exactly how entities are classified
|
||||
2. **Predictable**: No inference surprises or edge cases
|
||||
3. **Fast**: No neural processing overhead on every add/query
|
||||
4. **Smaller**: No embedded keyword models needed
|
||||
|
||||
**Real-World Use Case**:
|
||||
```typescript
|
||||
// Import data with known types
|
||||
await brain.add({
|
||||
data: { name: 'Apple Inc.', industry: 'Technology' },
|
||||
type: NounType.Organization
|
||||
})
|
||||
|
||||
await brain.add({
|
||||
data: { name: 'Cupertino', country: 'USA' },
|
||||
type: NounType.Location
|
||||
})
|
||||
|
||||
// Create relationship
|
||||
await brain.relate({
|
||||
from: appleId,
|
||||
to: cupertinoId,
|
||||
type: VerbType.LocatedIn
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Tool & Augmentation Compatibility
|
||||
|
||||
**The Problem with Schemaless**: Every tool must handle infinite variations:
|
||||
|
||||
```typescript
|
||||
// Incompatible tools
|
||||
const tool1Data = { type: 'person', name: 'Alice' }
|
||||
const tool2Data = { kind: 'human', fullName: 'Alice' }
|
||||
const tool3Data = { entity_type: 'individual', person_name: 'Alice' }
|
||||
|
||||
// Tools can't understand each other!
|
||||
```
|
||||
|
||||
**Brainy's Solution**: Finite types create a common language:
|
||||
|
||||
```typescript
|
||||
// All tools/augmentations understand core types
|
||||
interface NounMetadata {
|
||||
noun: NounType // Agreed-upon type system
|
||||
// ... custom fields
|
||||
}
|
||||
|
||||
// Augmentation 1: Adds caching for 'person' entities
|
||||
class PersonCacheAugmentation {
|
||||
execute(op, params) {
|
||||
if (params.noun?.metadata?.noun === 'person') {
|
||||
// All person entities are understood!
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Augmentation 2: Enriches 'organization' entities
|
||||
class OrgEnrichmentAugmentation {
|
||||
execute(op, params) {
|
||||
if (params.noun?.metadata?.noun === 'organization') {
|
||||
// Fetch industry data, employees, etc.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Augmentations compose seamlessly!
|
||||
```
|
||||
|
||||
**Ecosystem Benefits**:
|
||||
- Third-party augmentations are **interoperable**
|
||||
- Type-specific optimizations are **portable**
|
||||
- Query builders understand **semantic structure**
|
||||
- Visualization tools render **type-appropriate** displays
|
||||
- Import/export tools map to **universal types**
|
||||
|
||||
### 4. Concept Extraction & NLP Integration
|
||||
|
||||
**Traditional Approach**: Extract entities, ignore types:
|
||||
|
||||
```typescript
|
||||
// Generic NER (Named Entity Recognition)
|
||||
"Alice works at Google"
|
||||
// → ['Alice', 'Google'] // What are these?
|
||||
```
|
||||
|
||||
**Brainy's Approach**: Extract **typed** concepts:
|
||||
|
||||
```typescript
|
||||
import { NaturalLanguageProcessor } from '@soulcraft/brainy'
|
||||
|
||||
const nlp = new NaturalLanguageProcessor()
|
||||
const concepts = await nlp.extractConcepts("Alice works at Google in San Francisco")
|
||||
|
||||
// Returns typed entities:
|
||||
[
|
||||
{ text: 'Alice', nounType: 'person', confidence: 0.95 },
|
||||
{ text: 'Google', nounType: 'organization', confidence: 0.98 },
|
||||
{ text: 'San Francisco', nounType: 'place', confidence: 0.92 }
|
||||
]
|
||||
|
||||
// And typed relationships:
|
||||
[
|
||||
{
|
||||
from: 'Alice',
|
||||
to: 'Google',
|
||||
verbType: 'worksAt',
|
||||
confidence: 0.88
|
||||
},
|
||||
{
|
||||
from: 'Google',
|
||||
to: 'San Francisco',
|
||||
verbType: 'locatedIn',
|
||||
confidence: 0.85
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Downstream Benefits**:
|
||||
- **Smart Clustering**: Group by semantic type, not arbitrary keys
|
||||
- **Type-Aware Queries**: "Find all organizations in California"
|
||||
- **Relationship Reasoning**: "Who works at companies in SF?"
|
||||
- **Automatic Ontology**: Types form natural hierarchy
|
||||
|
||||
### 5. Query Optimization & Planning
|
||||
|
||||
**The Problem**: Schemaless queries are guesswork:
|
||||
|
||||
```sql
|
||||
-- MongoDB: No idea what fields exist
|
||||
db.collection.find({ someField: 'value' })
|
||||
// Full collection scan!
|
||||
```
|
||||
|
||||
**Brainy's Solution**: Type-aware query planning:
|
||||
|
||||
```typescript
|
||||
// Query planner knows types exist!
|
||||
brain.find({
|
||||
where: { noun: 'person' } // Type index lookup: O(1)!
|
||||
})
|
||||
|
||||
// Multi-type queries are optimized
|
||||
brain.find({
|
||||
where: {
|
||||
noun: ['person', 'organization'], // Bitmap union
|
||||
location: 'California' // Then filter
|
||||
}
|
||||
})
|
||||
|
||||
// Relationship traversal is type-aware
|
||||
brain.find({
|
||||
verb: 'worksAt', // Verb type index
|
||||
sourceType: 'person', // Source noun type index
|
||||
targetType: 'organization' // Target noun type index
|
||||
})
|
||||
```
|
||||
|
||||
**Query Performance**:
|
||||
- **Type Filtering**: O(1) bitmap intersection
|
||||
- **Join Planning**: Type-aware join order optimization
|
||||
- **Index Selection**: Automatic best index for type
|
||||
- **Cardinality Estimation**: Type statistics guide planning
|
||||
|
||||
### 6. Architecture & Development Benefits
|
||||
|
||||
#### Memory-Efficient Type Tracking
|
||||
|
||||
```typescript
|
||||
// Traditional approach: Map per field
|
||||
class TraditionalIndex {
|
||||
private fieldIndexes: Map<string, Map<any, Set<string>>>
|
||||
// Memory: O(unique_fields × unique_values × entities)
|
||||
}
|
||||
|
||||
// Brainy approach: Fixed Uint32Array per type
|
||||
class TypeAwareIndex {
|
||||
private nounTypeTracking: Uint32Array // Fixed size!
|
||||
private typeIndexes: RoaringBitmap32[] // One per type
|
||||
// Memory: O(noun_types) + O(entities_per_type)
|
||||
// PROJECTED: 385x smaller at billion scale (calculated from architecture, not benchmarked)
|
||||
}
|
||||
```
|
||||
|
||||
#### Type-Driven Code Organization
|
||||
|
||||
```typescript
|
||||
// Natural code structure follows types
|
||||
/src
|
||||
/nouns
|
||||
/person
|
||||
personStorage.ts // Type-specific storage
|
||||
personQueries.ts // Type-specific queries
|
||||
personAugmentation.ts // Type-specific logic
|
||||
/organization
|
||||
orgStorage.ts
|
||||
orgQueries.ts
|
||||
orgAugmentation.ts
|
||||
/verbs
|
||||
/worksAt
|
||||
worksAtValidation.ts // Relationship rules
|
||||
worksAtInference.ts // Type inference
|
||||
```
|
||||
|
||||
#### Type Safety in TypeScript
|
||||
|
||||
```typescript
|
||||
// Compiler-enforced type correctness
|
||||
function processPerson(noun: Noun) {
|
||||
if (noun.metadata.noun === 'person') {
|
||||
// TypeScript narrows type!
|
||||
const name: string = noun.metadata.name // Safe access
|
||||
}
|
||||
}
|
||||
|
||||
// Exhaustive type checking
|
||||
function processNoun(noun: Noun) {
|
||||
switch (noun.metadata.noun) {
|
||||
case 'person': return handlePerson(noun)
|
||||
case 'place': return handlePlace(noun)
|
||||
case 'organization': return handleOrg(noun)
|
||||
// Compiler error if missing cases!
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Public API: Type System
|
||||
|
||||
The type system is **fully public** for developers and augmentation authors:
|
||||
|
||||
```typescript
|
||||
import {
|
||||
NounType,
|
||||
VerbType,
|
||||
getNounTypes,
|
||||
getVerbTypes,
|
||||
BrainyTypes,
|
||||
suggestType
|
||||
} from '@soulcraft/brainy'
|
||||
|
||||
// Get all available noun types
|
||||
const nounTypes = getNounTypes()
|
||||
// → ['Person', 'Organization', 'Location', 'Thing', 'Concept', ...]
|
||||
|
||||
// Get all available verb types
|
||||
const verbTypes = getVerbTypes()
|
||||
// → ['RelatedTo', 'Contains', 'CreatedBy', 'LocatedIn', ...]
|
||||
|
||||
// Use types directly
|
||||
await brain.add({
|
||||
data: { name: 'Alice' },
|
||||
type: NounType.Person
|
||||
})
|
||||
|
||||
// Query by type
|
||||
await brain.find({
|
||||
type: NounType.Person,
|
||||
where: { name: 'Alice' }
|
||||
})
|
||||
```
|
||||
|
||||
**Use Cases**:
|
||||
- **Type-Safe Code**: Use TypeScript enums for compile-time checking
|
||||
- **Import Tools**: Specify entity types during data import
|
||||
- **Query Builders**: Filter by known types
|
||||
- **Augmentations**: Type-specific processing pipelines
|
||||
- **Visualization**: Type-appropriate rendering
|
||||
|
||||
---
|
||||
|
||||
## Real-World Performance Comparison
|
||||
|
||||
### Scenario: 1 Billion Entities with Rich Metadata
|
||||
|
||||
| Aspect | NoSQL (Schemaless) | Relational (Fixed) | Brainy (Finite Types) |
|
||||
|--------|-------------------|-------------------|----------------------|
|
||||
| **Memory (Indexes)** | 500GB+ | 250GB | 1.3GB |
|
||||
| **Type Lookup** | Full scan | O(log n) | O(1) bitmap |
|
||||
| **Add New Type** | Zero cost | Schema migration! | Register type |
|
||||
| **Query Planning** | Impossible | Table statistics | Type statistics |
|
||||
| **Tool Compatibility** | None | SQL only | Full ecosystem |
|
||||
| **Semantic Understanding** | None | None | Built-in |
|
||||
| **Concept Extraction** | Manual | Manual | Via SmartExtractor |
|
||||
| **Flexibility** | Infinite | Zero | Optimal balance |
|
||||
|
||||
---
|
||||
|
||||
## Design Principles
|
||||
|
||||
### 1. Finite but Extensible
|
||||
|
||||
```typescript
|
||||
// Core types are finite
|
||||
const coreNounTypes = [
|
||||
'person', 'place', 'organization', 'thing', ...
|
||||
]
|
||||
|
||||
// But easily extended
|
||||
brain.registerNounType('chemical_compound', {
|
||||
keywords: ['molecule', 'compound', 'element'],
|
||||
synonyms: ['substance', 'material'],
|
||||
parentType: 'thing'
|
||||
})
|
||||
```
|
||||
|
||||
### 1a. Subtypes — sub-classification without hierarchy
|
||||
|
||||
The 42-type taxonomy is intentionally coarse. Per-product vocabulary fits on the **`subtype`** axis — a top-level standard string field on every entity. Flat by design — no hierarchy, no parent chain, no recursive resolution. That preserves the Uint32Array-backed O(1) type stats while giving consumers a place to put `'employee'` / `'customer'` / `'invoice'` / `'milestone'` without burning a slot in the global enum.
|
||||
|
||||
```typescript
|
||||
// Same NounType, different subtypes:
|
||||
await brain.add({ type: NounType.Person, subtype: 'employee' })
|
||||
await brain.add({ type: NounType.Person, subtype: 'customer' })
|
||||
await brain.add({ type: NounType.Document, subtype: 'invoice' })
|
||||
|
||||
// Fast path — column-store hit, not metadata fallback:
|
||||
await brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// Per-NounType-per-subtype counts maintained incrementally:
|
||||
brain.counts.bySubtype(NounType.Person)
|
||||
// → { employee: 12, customer: 847 }
|
||||
```
|
||||
|
||||
Subtype has its own statistics rollup (`_system/subtype-statistics.json`) maintained alongside `nounCountsByType`, so per-subtype counts stay O(1) at billion scale.
|
||||
|
||||
The same principle applies to **VerbTypes** (7.30+): the 127-verb taxonomy is intentionally coarse, and `subtype` is the per-product axis for relationships too. A `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'colleague'`. Verb-side rollup lives at `_system/verb-subtype-statistics.json` with identical shape to the noun-side rollup. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`. Brainy's design is fully symmetric — nouns and verbs are first-class peers with identical capability surfaces.
|
||||
|
||||
Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
|
||||
|
||||
### 2. Semantic not Structural
|
||||
|
||||
```typescript
|
||||
// NOT structural types
|
||||
type Person = {
|
||||
name: string
|
||||
age: number
|
||||
// Fixed structure
|
||||
}
|
||||
|
||||
// Semantic types
|
||||
type Noun = {
|
||||
nounType: 'person', // Semantic meaning!
|
||||
metadata: {
|
||||
noun: 'person', // Required type
|
||||
// Any custom fields!
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Optimizable yet Flexible
|
||||
|
||||
```typescript
|
||||
// Optimized type tracking
|
||||
const typeIndex = new RoaringBitmap32() // 99.76% smaller!
|
||||
|
||||
// Flexible metadata
|
||||
const metadata = {
|
||||
noun: 'person', // Required type
|
||||
customField1: 'value', // Your fields
|
||||
customField2: 123, // Any structure
|
||||
nested: { ... } // Full flexibility
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
Brainy's **Finite Noun/Verb Type System** is revolutionary because it achieves the impossible:
|
||||
|
||||
1. ✅ **Billion-scale performance** (99.76% memory reduction)
|
||||
2. ✅ **Semantic understanding** (NLP integration)
|
||||
3. ✅ **Tool compatibility** (ecosystem interoperability)
|
||||
4. ✅ **Query optimization** (type-aware planning)
|
||||
5. ✅ **Concept extraction** (via SmartExtractor for imports)
|
||||
6. ✅ **Developer experience** (clean architecture)
|
||||
7. ✅ **Flexibility** (metadata freedom within types)
|
||||
|
||||
It's not schemaless chaos. It's not rigid relational constraints. It's **semantic structure** - the perfect balance for knowledge graphs at scale.
|
||||
|
||||
---
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Storage Architecture](./storage-architecture.md) - How types enable billion-scale storage
|
||||
- [Augmentation System](./augmentations.md) - Building type-aware augmentations
|
||||
- [Query Optimization](../api/query-optimization.md) - Type-aware query planning
|
||||
- [Import Flow](../guides/import-flow.md) - How types work in the import pipeline
|
||||
|
||||
---
|
||||
|
||||
*Brainy's finite type system: The foundation of billion-scale, semantically-aware knowledge graphs.*
|
||||
941
docs/architecture/index-architecture.md
Normal file
941
docs/architecture/index-architecture.md
Normal file
|
|
@ -0,0 +1,941 @@
|
|||
# Index Architecture
|
||||
|
||||
Brainy uses a sophisticated **3-tier index architecture** that enables "Triple Intelligence" - the unified combination of vector similarity, graph relationships, and metadata filtering. This document provides a comprehensive architectural overview of how these indexes work internally and coordinate with each other.
|
||||
|
||||
## Overview: The Three Main Indexes + Sub-Indexes
|
||||
|
||||
Brainy has **3 main indexes** at the top level, each with multiple sub-indexes managed automatically:
|
||||
|
||||
### Main Indexes (Level 1)
|
||||
|
||||
| Index | Purpose | Data Structure | Complexity | File Location | rebuild() Method |
|
||||
|-------|---------|----------------|------------|---------------|------------------|
|
||||
| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
|
||||
| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 |
|
||||
| **GraphAdjacencyIndex** | Relationship traversal | 2 verb-id LSM-trees + tombstone-filtered adjacency derivation | O(degree) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
|
||||
|
||||
### Sub-Indexes (Level 2)
|
||||
|
||||
**TypeAwareVectorIndex contains:**
|
||||
- **42 type-specific vector indexes** - One per NounType (automatically rebuilt via parent)
|
||||
|
||||
**MetadataIndexManager contains:**
|
||||
- **ChunkManager** - Adaptive chunked sparse indexing
|
||||
- **EntityIdMapper** - UUID ↔ integer mapping for roaring bitmaps
|
||||
- **FieldTypeInference** - DuckDB-inspired value-based field type detection
|
||||
- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count)
|
||||
- **Sorted Indexes** - Support orderBy queries (automatically maintained)
|
||||
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes
|
||||
|
||||
**GraphAdjacencyIndex contains:**
|
||||
- **lsmTreeSource** - Source → Targets (outgoing edges)
|
||||
- **lsmTreeTarget** - Target → Sources (incoming edges)
|
||||
- **lsmTreeVerbsBySource** - Source → Verb IDs
|
||||
- **lsmTreeVerbsByTarget** - Target → Verb IDs
|
||||
|
||||
All indexes share a **UnifiedCache** for coordinated memory management, ensuring fair resource allocation and preventing any single index from monopolizing memory.
|
||||
|
||||
## 1. MetadataIndex - Fast Field Filtering
|
||||
|
||||
**Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing.
|
||||
|
||||
### Internal Architecture
|
||||
|
||||
```typescript
|
||||
class MetadataIndexManager {
|
||||
// Chunked sparse indices: field → SparseIndex (replaces flat files)
|
||||
private sparseIndices = new Map<string, SparseIndex>()
|
||||
|
||||
// Chunk management
|
||||
private chunkManager: ChunkManager
|
||||
private chunkingStrategy: AdaptiveChunkingStrategy
|
||||
|
||||
// Lightweight field statistics
|
||||
private fieldIndexes = new Map<string, FieldIndexData>() // value → count
|
||||
private fieldStats = new Map<string, FieldStats>() // cardinality tracking
|
||||
|
||||
// Type-field affinity for NLP understanding
|
||||
private typeFieldAffinity = new Map<string, Map<string, number>>()
|
||||
|
||||
// Shared memory management
|
||||
private unifiedCache: UnifiedCache
|
||||
}
|
||||
```
|
||||
|
||||
### Key Data Structures
|
||||
|
||||
#### Chunked Sparse Index
|
||||
```typescript
|
||||
// SparseIndex: Directory of chunks for a field
|
||||
// Example: field="status"
|
||||
class SparseIndex {
|
||||
field: string
|
||||
chunks: ChunkDescriptor[] // Metadata about each chunk
|
||||
bloomFilters: BloomFilter[] // Fast membership testing
|
||||
}
|
||||
|
||||
// ChunkDescriptor: Metadata about a chunk
|
||||
interface ChunkDescriptor {
|
||||
chunkId: number
|
||||
valueCount: number // How many unique values in this chunk
|
||||
idCount: number // Total entity IDs
|
||||
zoneMap: ZoneMap // Min/max for range queries
|
||||
lastUpdated: number
|
||||
}
|
||||
|
||||
// Actual chunk data stored separately
|
||||
class ChunkData {
|
||||
chunkId: number
|
||||
field: string
|
||||
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (roaring bitmaps!)
|
||||
}
|
||||
```
|
||||
|
||||
**Performance**:
|
||||
- O(1) exact lookup with bloom filters (1% false positive rate)
|
||||
- O(log n) range queries with zone maps
|
||||
- 630x file reduction (560k flat files → 89 chunk files)
|
||||
|
||||
#### Roaring Bitmap Optimization
|
||||
|
||||
**Problem Solved**: JavaScript `Set<string>` for storing entity IDs was inefficient:
|
||||
- Memory overhead: ~40 bytes per UUID string (36 chars + overhead)
|
||||
- Slow intersection: JavaScript array filtering for multi-field queries
|
||||
- No hardware acceleration
|
||||
|
||||
**Solution**: Replace `Set<string>` with `RoaringBitmap32` (WebAssembly implementation) for 90% memory savings and hardware-accelerated operations. Uses `roaring-wasm` package for universal compatibility (Node.js, browsers, serverless) without requiring native compilation.
|
||||
|
||||
```typescript
|
||||
// EntityIdMapper: UUID ↔ Integer mapping
|
||||
class EntityIdMapper {
|
||||
private uuidToInt = new Map<string, number>()
|
||||
private intToUuid = new Map<number, string>()
|
||||
private nextId = 1
|
||||
|
||||
getOrAssign(uuid: string): number {
|
||||
// O(1) mapping: UUIDs → integers for bitmap storage
|
||||
let intId = this.uuidToInt.get(uuid)
|
||||
if (!intId) {
|
||||
intId = this.nextId++
|
||||
this.uuidToInt.set(uuid, intId)
|
||||
this.intToUuid.set(intId, uuid)
|
||||
}
|
||||
return intId
|
||||
}
|
||||
|
||||
intsIterableToUuids(ints: Iterable<number>): string[] {
|
||||
// Convert bitmap results back to UUIDs
|
||||
const result: string[] = []
|
||||
for (const intId of ints) {
|
||||
const uuid = this.intToUuid.get(intId)
|
||||
if (uuid) result.push(uuid)
|
||||
}
|
||||
return result
|
||||
}
|
||||
}
|
||||
|
||||
// ChunkData now uses RoaringBitmap32 instead of Set<string>
|
||||
class ChunkData {
|
||||
chunkId: number
|
||||
field: string
|
||||
entries: Map<string, RoaringBitmap32> // value → bitmap of integer IDs
|
||||
}
|
||||
```
|
||||
|
||||
**Key Benefits**:
|
||||
- **90% memory savings**: Roaring bitmaps compress much better than UUID strings
|
||||
- **Hardware-accelerated operations**: SIMD instructions (AVX2/SSE4.2) for ultra-fast bitmap AND/OR
|
||||
- **Portable serialization**: Cross-platform compatible format (Java/Go/Node.js)
|
||||
- **Lazy conversion**: UUIDs converted to integers only once, not per query
|
||||
|
||||
**Multi-Field Intersection (THE BIG WIN!)**:
|
||||
```typescript
|
||||
// Before: JavaScript array filtering
|
||||
async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string[]> {
|
||||
// 1. Fetch UUID arrays for each field
|
||||
const statusIds = await this.getIds('status', 'active') // ["uuid1", "uuid2", ...]
|
||||
const roleIds = await this.getIds('role', 'admin') // ["uuid2", "uuid3", ...]
|
||||
|
||||
// 2. JavaScript intersection (SLOW!)
|
||||
return statusIds.filter(id => roleIds.includes(id)) // O(n*m) array filtering
|
||||
}
|
||||
|
||||
// After: Roaring bitmap intersection
|
||||
async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
|
||||
// 1. Fetch roaring bitmaps (integers, not UUIDs)
|
||||
const bitmaps: RoaringBitmap32[] = []
|
||||
for (const {field, value} of pairs) {
|
||||
const bitmap = await this.getBitmapFromChunks(field, value)
|
||||
if (!bitmap) return [] // Short-circuit if any field has no matches
|
||||
bitmaps.push(bitmap)
|
||||
}
|
||||
|
||||
// 2. Hardware-accelerated intersection (FAST! AVX2/SSE4.2 SIMD)
|
||||
const result = RoaringBitmap32.and(...bitmaps) // O(1) hardware operation!
|
||||
|
||||
// 3. Convert final bitmap to UUIDs (once, not per-field)
|
||||
return this.idMapper.intsIterableToUuids(result)
|
||||
}
|
||||
```
|
||||
|
||||
**Performance Impact**:
|
||||
- Multi-field intersection: **1.4x average speedup**, up to 3.3x on 10K entities
|
||||
- Memory usage: **90% reduction** (17.17 MB → 2.01 MB for 100K entities)
|
||||
- Hardware acceleration: SIMD instructions make bitmap operations nearly free
|
||||
|
||||
**Benchmark Results** — example output from a single run of `tests/performance/roaring-bitmap-benchmark.ts` (1,000 queries per size, one machine; absolute times vary by hardware, the relative speedup and memory savings are the durable signal):
|
||||
| Dataset Size | Operation | Set Time | Roaring Time | Speedup | Memory Savings |
|
||||
|--------------|-----------|----------|--------------|---------|----------------|
|
||||
| 10,000 entities | 3-field intersection | 3.74ms | 1.14ms | **3.3x faster** | 90% |
|
||||
| 100,000 entities | 3-field intersection | 2.60ms | 1.78ms | **1.5x faster** | 88% |
|
||||
|
||||
**Implementation**: See `src/utils/entityIdMapper.ts` and benchmark at `tests/performance/roaring-bitmap-benchmark.ts`
|
||||
|
||||
#### Bloom Filter (Probabilistic Membership Testing)
|
||||
```typescript
|
||||
class BloomFilter {
|
||||
bits: Uint8Array // Bit array
|
||||
size: number // Total bits
|
||||
hashCount: number // Number of hash functions (FNV-1a, DJB2)
|
||||
|
||||
mightContain(value): boolean // ~1% false positive, 0% false negative
|
||||
}
|
||||
```
|
||||
|
||||
**Use case**: Quickly skip chunks that definitely don't contain a value
|
||||
|
||||
#### Zone Map (Range Query Optimization)
|
||||
```typescript
|
||||
interface ZoneMap {
|
||||
min: any | null // Minimum value in chunk
|
||||
max: any | null // Maximum value in chunk
|
||||
count: number // Number of entries
|
||||
hasNulls: boolean // Whether chunk contains null values
|
||||
}
|
||||
```
|
||||
|
||||
**Use case**: Skip entire chunks during range queries (ClickHouse-inspired)
|
||||
|
||||
#### Type-Field Affinity
|
||||
```typescript
|
||||
// Tracks which fields are commonly used with which types
|
||||
// Example:
|
||||
// typeFieldAffinity.get('character') → {
|
||||
// 'name': 127, // 127 characters have a 'name' field
|
||||
// 'age': 89, // 89 characters have an 'age' field
|
||||
// 'alignment': 45 // 45 characters have an 'alignment' field
|
||||
// }
|
||||
```
|
||||
|
||||
**Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field
|
||||
|
||||
#### Word Index (`__words__`) -
|
||||
```typescript
|
||||
// Special field for text/keyword search
|
||||
// Entity text content is tokenized and indexed as word hashes
|
||||
|
||||
// Tokenization:
|
||||
// "David Smith is a software engineer" → ["david", "smith", "is", "software", "engineer"]
|
||||
|
||||
// Word Hashing (FNV-1a):
|
||||
// "david" → hashWord("david") → 1234567 (int32)
|
||||
// "smith" → hashWord("smith") → 9876543 (int32)
|
||||
|
||||
// Index structure (same as other fields):
|
||||
// __words__ → 1234567 → RoaringBitmap{entity1, entity5, ...}
|
||||
// __words__ → 9876543 → RoaringBitmap{entity1, entity3, ...}
|
||||
```
|
||||
|
||||
**Design Decisions**:
|
||||
- **Max 50 words per entity**: Prevents index bloat for large documents
|
||||
- **FNV-1a hashing**: Fast, low collision rate, int32 output
|
||||
- **Min word length 2 chars**: Filters out noise words
|
||||
- **Lowercase normalization**: Case-insensitive matching
|
||||
- **Automatic integration**: Words extracted via `extractIndexableFields()`
|
||||
|
||||
**Hybrid Search**: Text results combined with vector results using Reciprocal Rank Fusion (RRF):
|
||||
```typescript
|
||||
// RRF formula: score(d) = sum(1 / (k + rank(d)))
|
||||
// where k = 60 (standard constant)
|
||||
// alpha = weight for semantic (0 = text only, 1 = semantic only)
|
||||
```
|
||||
|
||||
### Query Algorithm
|
||||
|
||||
**Exact Match Query**:
|
||||
```typescript
|
||||
async getIds(field: string, value: any): Promise<string[]> {
|
||||
// 1. Load sparse index for field
|
||||
const sparseIndex = await this.loadSparseIndex(field)
|
||||
|
||||
// 2. Find candidate chunks using bloom filters
|
||||
const candidateChunks = sparseIndex.findChunksForValue(value)
|
||||
// → Bloom filter checks all chunks (~1ms)
|
||||
// → Returns only chunks that *might* contain value
|
||||
|
||||
// 3. Load candidate chunks and collect IDs
|
||||
const results = []
|
||||
for (const chunkId of candidateChunks) {
|
||||
const chunk = await this.chunkManager.loadChunk(field, chunkId)
|
||||
const ids = chunk.entries.get(value)
|
||||
if (ids) results.push(...ids)
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
```
|
||||
|
||||
**Range Query**:
|
||||
```typescript
|
||||
async getIdsForRange(field: string, min: any, max: any): Promise<string[]> {
|
||||
// 1. Load sparse index for field
|
||||
const sparseIndex = await this.loadSparseIndex(field)
|
||||
|
||||
// 2. Find candidate chunks using zone maps
|
||||
const candidateChunks = sparseIndex.findChunksForRange(min, max)
|
||||
// → Check zoneMap.min and zoneMap.max for each chunk
|
||||
// → Skip chunks where max < min or min > max
|
||||
|
||||
// 3. Load chunks and filter values
|
||||
const results = []
|
||||
for (const chunkId of candidateChunks) {
|
||||
const chunk = await this.chunkManager.loadChunk(field, chunkId)
|
||||
for (const [value, ids] of chunk.entries) {
|
||||
if (value >= min && value <= max) {
|
||||
results.push(...ids)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- Bloom filters: Skip 99% of irrelevant chunks (exact match)
|
||||
- Zone maps: Skip entire chunks that fall outside range
|
||||
- Adaptive chunking: ~50 values per chunk optimizes I/O
|
||||
- Immediate flushing: No need for dirty tracking or batch writes
|
||||
|
||||
### Temporal Bucketing
|
||||
|
||||
**Problem Solved**: High-cardinality timestamp fields created massive file pollution.
|
||||
- Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!)
|
||||
|
||||
**Solution**: Automatic bucketing of temporal fields to 1-minute intervals.
|
||||
|
||||
```typescript
|
||||
// In normalizeValue(value, field):
|
||||
if (field && typeof value === 'number') {
|
||||
const fieldLower = field.toLowerCase()
|
||||
const isTemporal = fieldLower.includes('time') ||
|
||||
fieldLower.includes('date') ||
|
||||
fieldLower.includes('accessed') ||
|
||||
fieldLower.includes('modified') ||
|
||||
fieldLower.includes('created') ||
|
||||
fieldLower.includes('updated')
|
||||
|
||||
if (isTemporal) {
|
||||
// Bucket to 1-minute intervals
|
||||
const bucketSize = 60000 // milliseconds
|
||||
const bucketed = Math.floor(value / bucketSize) * bucketSize
|
||||
return bucketed.toString()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- ✅ Reduces 575 unique timestamps → ~10 buckets
|
||||
- ✅ File count: 358,407 → ~4,600 (98.7% reduction)
|
||||
- ✅ Zero configuration - automatic field name detection
|
||||
- ✅ Still enables range queries (not excluded like before)
|
||||
- ✅ 1-minute precision sufficient for most use cases
|
||||
|
||||
**Field Name Detection**: Automatically buckets fields with these keywords:
|
||||
- `time`, `date`, `accessed`, `modified`, `created`, `updated`
|
||||
- Examples: `timestamp`, `createdAt`, `lastModified`, `birthdate`, `eventTime`
|
||||
|
||||
### Operations
|
||||
|
||||
```typescript
|
||||
// Add to index (src/brainy.ts:387)
|
||||
await this.metadataIndex.addToIndex(id, metadata)
|
||||
|
||||
// Query exact match
|
||||
const ids = await this.metadataIndex.getIds('status', 'active')
|
||||
|
||||
// Query range
|
||||
const ids = await this.metadataIndex.getIdsForFilter({
|
||||
publishDate: { greaterThan: 1640995200000 }
|
||||
})
|
||||
|
||||
// Filter discovery (what values exist for a field)
|
||||
const values = await this.metadataIndex.getFilterValues('status')
|
||||
// → ['active', 'archived', 'draft']
|
||||
|
||||
// Statistics (O(1))
|
||||
const totalEntities = this.metadataIndex.getTotalEntityCount()
|
||||
const typeBreakdown = this.metadataIndex.getAllEntityCounts()
|
||||
// → Map { 'character': 127, 'item': 89, 'location': 45 }
|
||||
```
|
||||
|
||||
### Excluded Fields
|
||||
|
||||
Some fields are excluded from indexing to prevent pollution:
|
||||
|
||||
```typescript
|
||||
const DEFAULT_EXCLUDE_FIELDS = [
|
||||
'id', // Primary key (redundant to index)
|
||||
'uuid', // Alternative primary key
|
||||
'vector', // High-dimensional data
|
||||
'embedding', // Same as vector
|
||||
'content', // Large text content
|
||||
'description', // Large text content
|
||||
'metadata', // Nested object (too large)
|
||||
'data' // Generic nested object
|
||||
]
|
||||
```
|
||||
|
||||
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
|
||||
|
||||
## 2. Vector Index - Vector Similarity Search
|
||||
|
||||
**Purpose**: O(log n) semantic similarity search using vector embeddings.
|
||||
|
||||
The default JS implementation is `JsHnswVectorIndex`; an optional native acceleration package (`@soulcraft/cor`) can register a higher-performing `VectorIndexProvider` through the plugin system. The public API stays the same either way.
|
||||
|
||||
### Internal Architecture
|
||||
|
||||
```typescript
|
||||
class JsHnswVectorIndex {
|
||||
// Per-noun indexes for efficiency
|
||||
private nouns: Map<string, HNSWNoun> = new Map()
|
||||
|
||||
// Global entry point for search
|
||||
private entryPointId: string | null = null
|
||||
private maxLevel = 0
|
||||
|
||||
// Shared memory management
|
||||
private unifiedCache: UnifiedCache
|
||||
private storage: BaseStorage | null = null
|
||||
}
|
||||
|
||||
// Each noun has its own HNSW graph
|
||||
class HNSWNoun {
|
||||
noun: string
|
||||
nodes: Map<string, HNSWNode>
|
||||
entryPointId: string | null
|
||||
maxLevel: number
|
||||
}
|
||||
|
||||
// Each node in the graph
|
||||
class HNSWNode {
|
||||
id: string
|
||||
vector: Vector | null // Lazy-loaded from storage
|
||||
level: number
|
||||
connections: Map<number, string[]> // level → neighbor IDs
|
||||
}
|
||||
```
|
||||
|
||||
### Hierarchical Graph Structure
|
||||
|
||||
The default vector index builds a multi-layered graph:
|
||||
|
||||
```
|
||||
Layer 2: [entry] ←→ [node1] (sparse, long-range connections)
|
||||
↓ ↓
|
||||
Layer 1: [entry] ←→ [node1] ←→ [node2] ←→ [node3] (medium density)
|
||||
↓ ↓ ↓ ↓
|
||||
Layer 0: [entry] ←→ [node1] ←→ [node2] ←→ [node3] ←→ [node4] ←→ [node5] (dense, all nodes)
|
||||
```
|
||||
|
||||
**Search Algorithm**:
|
||||
1. Start at entry point in top layer
|
||||
2. Greedy search for nearest neighbor in current layer
|
||||
3. Move down to next layer with found neighbor
|
||||
4. Repeat until reaching layer 0
|
||||
5. Return k nearest neighbors
|
||||
|
||||
**Complexity**: O(log n) due to hierarchical structure
|
||||
|
||||
### Adaptive Vector Loading
|
||||
|
||||
Vectors are lazy-loaded on demand based on memory availability:
|
||||
|
||||
```typescript
|
||||
private async getVectorSafe(noun: HNSWNoun): Promise<Vector> {
|
||||
// Check UnifiedCache first
|
||||
const cached = this.unifiedCache.get(noun.id)
|
||||
if (cached) return cached
|
||||
|
||||
// Load from storage if memory available
|
||||
if (this.unifiedCache.canCache()) {
|
||||
const vector = await this.storage.loadVector(noun.id)
|
||||
this.unifiedCache.set(noun.id, vector)
|
||||
return vector
|
||||
}
|
||||
|
||||
// Load transiently if memory pressure
|
||||
return await this.storage.loadVector(noun.id)
|
||||
}
|
||||
```
|
||||
|
||||
### Operations
|
||||
|
||||
```typescript
|
||||
// Add entity (src/brainy.ts:add)
|
||||
await this.index.addEntity(id, vector, noun)
|
||||
|
||||
// Search for similar vectors
|
||||
const results = await this.index.search(queryVector, k, threshold)
|
||||
// Returns: Array<{id: string, similarity: number}>
|
||||
|
||||
// Rebuild from storage
|
||||
await this.index.rebuild()
|
||||
```
|
||||
|
||||
## 3. GraphAdjacencyIndex - O(1) Relationship Traversal
|
||||
|
||||
**Purpose**: Constant-time neighbor lookups regardless of graph size.
|
||||
|
||||
### Internal Architecture
|
||||
|
||||
```typescript
|
||||
class GraphAdjacencyIndex {
|
||||
// O(1) bidirectional lookups
|
||||
private sourceIndex = new Map<string, Set<string>>() // sourceId → targetIds
|
||||
private targetIndex = new Map<string, Set<string>>() // targetId → sourceIds
|
||||
|
||||
// Full relationship data
|
||||
private verbIndex = new Map<string, GraphVerb>() // verbId → metadata
|
||||
|
||||
// Statistics
|
||||
private relationshipCountsByType = new Map<string, number>()
|
||||
|
||||
// Shared memory
|
||||
private unifiedCache: UnifiedCache
|
||||
private storage: BaseStorage
|
||||
}
|
||||
```
|
||||
|
||||
### Key Innovation: Bidirectional Adjacency
|
||||
|
||||
**Core Insight**: Store BOTH directions of each relationship for O(1) lookups.
|
||||
|
||||
```typescript
|
||||
// Example: Alice KNOWS Bob
|
||||
// verbId = "verb-123"
|
||||
|
||||
// Source index: Alice → Bob
|
||||
sourceIndex.set('alice', Set(['bob']))
|
||||
|
||||
// Target index: Bob ← Alice
|
||||
targetIndex.set('bob', Set(['alice']))
|
||||
|
||||
// Full metadata
|
||||
verbIndex.set('verb-123', {
|
||||
id: 'verb-123',
|
||||
verb: 'knows',
|
||||
source: 'alice',
|
||||
target: 'bob',
|
||||
metadata: { since: 2020 }
|
||||
})
|
||||
```
|
||||
|
||||
**Result**: Finding Alice's friends OR Bob's friends is O(1) - just one Map lookup!
|
||||
|
||||
### Operations
|
||||
|
||||
```typescript
|
||||
// Add relationship (src/brainy.ts:relate)
|
||||
await this.graphIndex.addRelationship(verbId, sourceId, targetId, verb)
|
||||
|
||||
// Get neighbors (O(1) per hop)
|
||||
const outgoing = await this.graphIndex.getNeighbors(id, 'out') // Who does id point to?
|
||||
const incoming = await this.graphIndex.getNeighbors(id, 'in') // Who points to id?
|
||||
const both = await this.graphIndex.getNeighbors(id, 'both') // All neighbors
|
||||
|
||||
// Get relationships
|
||||
const verbs = await this.graphIndex.getRelationships(sourceId, targetId)
|
||||
|
||||
// Statistics (O(1))
|
||||
const totalRelationships = this.graphIndex.getTotalRelationshipCount()
|
||||
const byType = this.graphIndex.getRelationshipCountsByType()
|
||||
// → Map { 'knows': 45, 'created': 23, 'located_at': 12 }
|
||||
```
|
||||
|
||||
### Graph Traversal
|
||||
|
||||
The index supports multi-hop traversal:
|
||||
|
||||
```typescript
|
||||
// Find all entities within 2 hops
|
||||
const reachable = await this.graphIndex.traverse({
|
||||
startId: 'alice',
|
||||
depth: 2,
|
||||
direction: 'out'
|
||||
})
|
||||
// Complexity: O(V + E) breadth-first search, but each neighbor lookup is O(1)
|
||||
```
|
||||
|
||||
## Shared Memory Management: UnifiedCache
|
||||
|
||||
All three main indexes share a single **UnifiedCache** instance for coordinated memory management.
|
||||
|
||||
### Architecture
|
||||
|
||||
```typescript
|
||||
class UnifiedCache {
|
||||
private cache: Map<string, CachedItem> = new Map()
|
||||
private maxSize: number
|
||||
private currentSize: number = 0
|
||||
private evictionPolicy: 'LRU' | 'LFU' = 'LRU'
|
||||
}
|
||||
|
||||
// Each index gets the same cache instance
|
||||
const unifiedCache = new UnifiedCache({ maxSize: 1000 })
|
||||
this.metadataIndex = new MetadataIndexManager(storage, { unifiedCache })
|
||||
this.vectorIndex = new JsHnswVectorIndex(storage, { unifiedCache })
|
||||
this.graphIndex = new GraphAdjacencyIndex(storage, { unifiedCache })
|
||||
```
|
||||
|
||||
### Benefits
|
||||
|
||||
1. **Fair Resource Allocation**: All indexes compete for the same memory pool
|
||||
2. **Prevents Monopolization**: No single index can starve others of memory
|
||||
3. **Coordinated Eviction**: LRU eviction across all cached items system-wide
|
||||
4. **Memory Pressure Handling**: Automatic cache shrinking when memory is tight
|
||||
5. **Adaptive Loading**: Indexes load data transiently under memory pressure
|
||||
|
||||
### Cache Key Patterns
|
||||
|
||||
Each index uses different key prefixes:
|
||||
|
||||
```typescript
|
||||
// Metadata index
|
||||
cache.set(`meta:${field}:${value}`, indexEntry)
|
||||
|
||||
// Vector index
|
||||
cache.set(`vector:${id}`, vectorData)
|
||||
|
||||
// Graph index
|
||||
cache.set(`graph:${sourceId}`, neighbors)
|
||||
|
||||
// Deleted items (no caching needed - uses Set)
|
||||
```
|
||||
|
||||
## How Indexes Work Together
|
||||
|
||||
### 1. Entity Creation (`brainy.add()`)
|
||||
|
||||
```typescript
|
||||
// src/brainy.ts:add()
|
||||
async add(params: AddParams): Promise<string> {
|
||||
const id = generateId()
|
||||
const vector = await this.embedder(params.content)
|
||||
|
||||
// Add to metadata index (field filtering)
|
||||
await this.metadataIndex.addToIndex(id, params.metadata)
|
||||
|
||||
// Add to vector index (vector search)
|
||||
await this.index.addEntity(id, vector, params.noun)
|
||||
|
||||
// Relationships added via separate relate() calls
|
||||
|
||||
return id
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Entity Search (`brainy.find()`)
|
||||
|
||||
```typescript
|
||||
// src/brainy.ts:find()
|
||||
async find(query: FindQuery): Promise<Result[]> {
|
||||
let results: Result[] = []
|
||||
|
||||
// Step 1: Metadata filtering (fast pre-filter)
|
||||
if (query.where) {
|
||||
const filteredIds = await this.metadataIndex.getIdsForFilter(query.where)
|
||||
results = await this.getEntitiesByIds(filteredIds)
|
||||
}
|
||||
|
||||
// Step 2: Vector similarity search (semantic ranking)
|
||||
if (query.like) {
|
||||
const queryVector = await this.embedder(query.like)
|
||||
const vectorResults = await this.index.search(queryVector, query.limit)
|
||||
|
||||
// Intersect or union with metadata results
|
||||
results = this.combineResults(results, vectorResults)
|
||||
}
|
||||
|
||||
// Step 3: Graph traversal (relationship filtering)
|
||||
if (query.connected) {
|
||||
const connectedIds = await this.graphIndex.traverse(query.connected)
|
||||
results = results.filter(r => connectedIds.includes(r.id))
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Entity Update (`brainy.update()`)
|
||||
|
||||
```typescript
|
||||
// src/brainy.ts:update()
|
||||
async update(params: UpdateParams): Promise<void> {
|
||||
const existing = await this.get(params.id)
|
||||
|
||||
// Update metadata index (remove old, add new)
|
||||
await this.metadataIndex.removeFromIndex(params.id, existing.metadata)
|
||||
await this.metadataIndex.addToIndex(params.id, params.metadata)
|
||||
|
||||
// Update vector index (re-embed if content changed)
|
||||
if (params.content) {
|
||||
const newVector = await this.embedder(params.content)
|
||||
await this.index.updateEntity(params.id, newVector)
|
||||
}
|
||||
|
||||
// Graph relationships unchanged (managed separately)
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Statistics (`brainy.stats()`)
|
||||
|
||||
All indexes provide O(1) statistics:
|
||||
|
||||
```typescript
|
||||
// src/brainy.ts:stats()
|
||||
async stats(): Promise<Statistics> {
|
||||
return {
|
||||
// From metadata index
|
||||
entities: this.metadataIndex.getTotalEntityCount(),
|
||||
entityTypes: this.metadataIndex.getAllEntityCounts(),
|
||||
|
||||
// From graph index
|
||||
relationships: this.graphIndex.getTotalRelationshipCount(),
|
||||
relationshipTypes: this.graphIndex.getRelationshipCountsByType(),
|
||||
|
||||
// From vector index
|
||||
vectorIndexSize: this.index.getSize()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Index Rebuilding (Lazy Loading Support)
|
||||
|
||||
**Two modes of index loading:**
|
||||
|
||||
#### Mode 1: Auto-Rebuild on init() (default)
|
||||
|
||||
```typescript
|
||||
// src/brainy.ts:init()
|
||||
async init(): Promise<void> {
|
||||
// When disableAutoRebuild: false (default)
|
||||
const metadataStats = await this.metadataIndex.getStats()
|
||||
const vectorIndexSize = this.index.size()
|
||||
const graphIndexSize = await this.graphIndex.size()
|
||||
|
||||
if (metadataStats.totalEntries === 0 ||
|
||||
vectorIndexSize === 0 ||
|
||||
graphIndexSize === 0) {
|
||||
// Rebuild all indexes in parallel
|
||||
await Promise.all([
|
||||
metadataStats.totalEntries === 0 ? this.metadataIndex.rebuild() : Promise.resolve(),
|
||||
vectorIndexSize === 0 ? this.index.rebuild() : Promise.resolve(),
|
||||
graphIndexSize === 0 ? this.graphIndex.rebuild() : Promise.resolve()
|
||||
])
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Mode 2: Lazy Loading on First Query
|
||||
|
||||
```typescript
|
||||
// When disableAutoRebuild: true
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem' },
|
||||
disableAutoRebuild: true // Enable lazy loading
|
||||
})
|
||||
|
||||
await brain.init() // Returns instantly, indexes empty
|
||||
|
||||
// First query triggers lazy rebuild
|
||||
const results = await brain.find({ limit: 10 })
|
||||
// → Calls ensureIndexesLoaded() (line 4617)
|
||||
// → Rebuilds all 3 main indexes with concurrency control
|
||||
// → Subsequent queries are instant (0ms check)
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- First query with lazy loading: ~50-200ms rebuild (1K-10K entities)
|
||||
- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
|
||||
- Subsequent queries: 0ms check (instant)
|
||||
|
||||
See [initialization-and-rebuild.md](./initialization-and-rebuild.md) for detailed lazy loading implementation.
|
||||
|
||||
## Triple Intelligence Integration
|
||||
|
||||
The **TripleIntelligenceSystem** (`src/triple/TripleIntelligenceSystem.ts`) combines all three core indexes:
|
||||
|
||||
```typescript
|
||||
class TripleIntelligenceSystem {
|
||||
constructor(
|
||||
private metadataIndex: MetadataIndexManager,
|
||||
private vectorIndex: VectorIndexProvider,
|
||||
private graphIndex: GraphAdjacencyIndex,
|
||||
private embedder: EmbedderFunction,
|
||||
private storage: BaseStorage
|
||||
) {}
|
||||
|
||||
async query(nlpQuery: string): Promise<Result[]> {
|
||||
// Parse natural language
|
||||
const parsed = await this.parseQuery(nlpQuery)
|
||||
|
||||
// Execute across all three indexes
|
||||
const [metadataResults, vectorResults, graphResults] = await Promise.all([
|
||||
this.metadataIndex.getIdsForFilter(parsed.filters),
|
||||
this.vectorIndex.search(parsed.vector, parsed.limit),
|
||||
this.graphIndex.traverse(parsed.graphConstraints)
|
||||
])
|
||||
|
||||
// Fuse results with weighted scoring
|
||||
return this.fuseResults(metadataResults, vectorResults, graphResults)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Operation Complexity by Index
|
||||
|
||||
| Operation | MetadataIndexManager | TypeAwareVectorIndex | GraphAdjacencyIndex |
|
||||
|-----------|---------------------|-------------------|---------------------|
|
||||
| **Add** | O(1) per field | O(log n) | O(1) |
|
||||
| **Remove** | O(1) per field | O(log n) | O(1) |
|
||||
| **Exact lookup** | O(1) | N/A | O(1) |
|
||||
| **Range query** | O(log n) + O(k) | N/A | N/A |
|
||||
| **Similarity search** | N/A | O(log n) | N/A |
|
||||
| **Neighbor lookup** | N/A | N/A | O(1) |
|
||||
| **Statistics** | O(1) | O(1) | O(1) |
|
||||
| **Rebuild** | O(n) | O(n) | O(n) |
|
||||
|
||||
Where:
|
||||
- n = total number of entities
|
||||
- k = number of matching results
|
||||
|
||||
**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for the vector index).
|
||||
|
||||
### Memory Footprint
|
||||
|
||||
| Index | Per-Entity Memory | Notes |
|
||||
|-------|-------------------|-------|
|
||||
| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) |
|
||||
| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
|
||||
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional verb-id references in 2 LSM-trees |
|
||||
|
||||
**Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship
|
||||
|
||||
**Sub-index memory:**
|
||||
- ChunkManager: ~20 bytes per chunk descriptor
|
||||
- EntityIdMapper: ~32 bytes per UUID mapping (50-90% savings vs Set\<string\>)
|
||||
- LSM-trees: ~200 bytes per relationship (SSTable storage)
|
||||
|
||||
### Scalability
|
||||
|
||||
All indexes scale gracefully. The cost of each stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion:
|
||||
|
||||
| Query stage | Complexity | Scaling behavior |
|
||||
|-------------|------------|------------------|
|
||||
| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
|
||||
| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
|
||||
| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
|
||||
| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
|
||||
| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
|
||||
|
||||
**Key observations**:
|
||||
- Graph queries stay O(1) regardless of scale
|
||||
- Metadata filtering scales sub-linearly
|
||||
- Vector search degrades gracefully due to the hierarchical index
|
||||
- Combined queries remain fast even at scale
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When to Use Each Index
|
||||
|
||||
**MetadataIndex**:
|
||||
- Filtering by exact field values (status, type, category)
|
||||
- Range queries on numeric/temporal fields (dates, prices, counts)
|
||||
- Field discovery (what filters are available)
|
||||
- Type-based querying (find all characters, all items)
|
||||
|
||||
**Vector Index**:
|
||||
- Semantic similarity search ("find similar documents")
|
||||
- Content-based retrieval ("find posts about AI")
|
||||
- Fuzzy matching (when exact matches aren't required)
|
||||
- Recommendation systems (find related items)
|
||||
|
||||
**GraphAdjacencyIndex**:
|
||||
- Relationship queries ("who knows whom")
|
||||
- Path finding ("how are these entities connected")
|
||||
- Network analysis ("find communities")
|
||||
- Multi-hop traversal ("friends of friends")
|
||||
|
||||
**Note**: Soft-delete functionality is not currently integrated. Brainy uses hard deletes via storage layer.
|
||||
|
||||
### Query Optimization
|
||||
|
||||
1. **Start with metadata filters** - They're fastest and most selective
|
||||
2. **Use graph constraints** - O(1) lookups significantly reduce search space
|
||||
3. **Vector search last** - Most expensive, best used on pre-filtered set
|
||||
4. **Leverage temporal bucketing** - Timestamp range queries work efficiently
|
||||
5. **Monitor statistics** - Use O(1) stats methods for cardinality estimation
|
||||
|
||||
### Memory Management
|
||||
|
||||
1. **Configure UnifiedCache appropriately** - Balance between speed and memory
|
||||
2. **Use lazy loading** - Vector index loads vectors on-demand
|
||||
3. **Monitor cache hit rates** - Adjust cache size if hit rate is low
|
||||
4. **Consider storage adapter** - Memory = fastest, filesystem = persistent
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Find System](../FIND_SYSTEM.md) - Query-centric view of index usage
|
||||
- [Triple Intelligence](./triple-intelligence.md) - Advanced query system
|
||||
- [Storage Architecture](./storage-architecture.md) - Storage layer details
|
||||
- [Performance Guide](../PERFORMANCE.md) - Performance tuning
|
||||
- [Overview](./overview.md) - High-level architecture
|
||||
|
||||
## Summary: Index Hierarchy
|
||||
|
||||
### Level 1: Main Indexes (3)
|
||||
All have rebuild() methods and are covered by lazy loading:
|
||||
1. **TypeAwareVectorIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403`
|
||||
2. **MetadataIndexManager** - `src/utils/metadataIndex.ts:2318`
|
||||
3. **GraphAdjacencyIndex** - `src/graph/graphAdjacencyIndex.ts:389`
|
||||
|
||||
### Level 2: Sub-Indexes (~50+)
|
||||
Automatically managed by parent rebuild():
|
||||
- **42 type-specific vector indexes** (one per NounType)
|
||||
- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes)
|
||||
- **2 LSM-trees** (lsmTreeVerbsBySource, lsmTreeVerbsByTarget — the verb set is the single adjacency source of truth; neighbor reads derive from live verbs so removals are honored)
|
||||
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
|
||||
|
||||
### Lazy Loading
|
||||
- **Mode 1**: Auto-rebuild on init() (default)
|
||||
- **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`)
|
||||
- **Concurrency-safe**: Mutex prevents duplicate rebuilds
|
||||
- **Performance**: First query ~50-200ms, subsequent queries instant
|
||||
|
||||
### Total Functional Index Count
|
||||
- **3 main indexes** with independent rebuild() methods
|
||||
- **~50+ sub-components** managed automatically
|
||||
- **All covered** by rebuildIndexesIfNeeded() or built-in lazy initialization
|
||||
|
||||
## Version History
|
||||
|
||||
- **v5.7.7** (November 2025): Added production-scale lazy loading with concurrency control. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added `ensureIndexesLoaded()` helper and `getIndexStatus()` diagnostic.
|
||||
- **v3.43.0** (October 2025): Migrated from `roaring` (native C++) to `roaring-wasm` (WebAssembly) for universal compatibility. No API changes - maintains identical RoaringBitmap32 interface. Benefits: works in all environments (Node.js, browsers, serverless) without build tools, zero compilation errors, simpler developer experience. 90% memory savings and hardware-accelerated operations unchanged.
|
||||
- **v3.42.0** (October 2025): Replaced flat file indexing with adaptive chunked sparse indexing. Bloom filters + zone maps for O(1) exact match and O(log n) range queries. 630x file reduction (560k → 89 files). Removed dual code paths.
|
||||
- **v3.41.0** (October 2025): Added automatic temporal bucketing to MetadataIndex
|
||||
- **v3.40.0** (October 2025): Enhanced batch processing for imports
|
||||
- **v3.0.0** (September 2025): Introduced 3-tier index architecture with UnifiedCache
|
||||
713
docs/architecture/initialization-and-rebuild.md
Normal file
713
docs/architecture/initialization-and-rebuild.md
Normal file
|
|
@ -0,0 +1,713 @@
|
|||
# Initialization and Rebuild Processes
|
||||
|
||||
This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
||||
|
||||
## Core Principle: All Indexes Are Disk-Based
|
||||
|
||||
**KEY INSIGHT**: All indexes in Brainy are already disk-based. There is no need for snapshots or separate backup mechanisms. Initialization simply loads the right amount of data from storage into memory based on available resources.
|
||||
|
||||
### What Gets Persisted
|
||||
|
||||
| Index | Persisted Data | Storage Method | Since Version |
|
||||
|-------|---------------|----------------|---------------|
|
||||
| **MetadataIndex** | Field registry + chunked sparse indices with bloom filters + zone maps | `storage.saveMetadata()` | v3.42.0 (chunks), v4.2.1 (registry) |
|
||||
| **Vector Index** | Vector embeddings + graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
|
||||
| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
|
||||
| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
|
||||
|
||||
#### MetadataIndex Persistence Details
|
||||
|
||||
The MetadataIndex now persists two components:
|
||||
|
||||
1. **Field Registry** (`__metadata_field_registry__`): Directory of indexed fields for O(1) discovery
|
||||
- Size: ~4-8KB (50-200 fields typical)
|
||||
- Enables instant cold starts by discovering persisted indices
|
||||
- Auto-saved during every flush operation
|
||||
|
||||
2. **Sparse Indices** (`__sparse_index__<field>`): Per-field index directories
|
||||
- Contains chunk metadata, zone maps, and bloom filters
|
||||
- Lazy-loaded via UnifiedCache on first query
|
||||
|
||||
3. **Chunks** (`__metadata_chunk__<field>_<chunkId>`): Actual inverted index data
|
||||
- Roaring bitmaps for compressed entity ID storage
|
||||
- Loaded on-demand based on query patterns
|
||||
|
||||
All storage operations use the **StorageAdapter** interface, which works with FileSystem and Memory backends.
|
||||
|
||||
## Initialization Process
|
||||
|
||||
### 1. Lazy Initialization Pattern
|
||||
|
||||
All indexes use lazy initialization - they don't load data until first use:
|
||||
|
||||
```typescript
|
||||
// Example: GraphAdjacencyIndex
|
||||
class GraphAdjacencyIndex {
|
||||
private initialized = false
|
||||
|
||||
private async ensureInitialized(): Promise<void> {
|
||||
if (this.initialized) return
|
||||
|
||||
// Initialize LSM-trees from storage
|
||||
await this.lsmTreeSource.init()
|
||||
await this.lsmTreeTarget.init()
|
||||
|
||||
this.initialized = true
|
||||
}
|
||||
|
||||
// Every public method calls ensureInitialized() first
|
||||
async getNeighbors(id: string): Promise<string[]> {
|
||||
await this.ensureInitialized() // Lazy init!
|
||||
// ... actual logic
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- Zero-cost abstraction: No initialization overhead if index not used
|
||||
- Faster startup: Indexes initialize in parallel on first use
|
||||
- Lower memory: Only used indexes consume memory
|
||||
|
||||
### 2. Brain Initialization Flow
|
||||
|
||||
When you create a `Brain` instance and call `init()`, behavior depends on the `disableAutoRebuild` configuration:
|
||||
|
||||
#### Mode 1: Auto-Rebuild on init() (Default)
|
||||
|
||||
```typescript
|
||||
// src/brainy.ts (lines 192-237)
|
||||
async init(): Promise<void> {
|
||||
const initStartTime = Date.now()
|
||||
|
||||
// STEP 1: Initialize storage and unified cache
|
||||
await this.storage.init()
|
||||
|
||||
// STEP 2: Check index sizes (lazy initialization triggers here)
|
||||
const metadataStats = await this.metadataIndex.getStats()
|
||||
const vectorIndexSize = this.index.size()
|
||||
const graphIndexSize = await this.graphIndex.size()
|
||||
|
||||
// STEP 3: Rebuild empty indexes from storage in parallel
|
||||
if (metadataStats.totalEntries === 0 ||
|
||||
vectorIndexSize === 0 ||
|
||||
graphIndexSize === 0) {
|
||||
|
||||
const rebuildStartTime = Date.now()
|
||||
await Promise.all([
|
||||
metadataStats.totalEntries === 0
|
||||
? this.metadataIndex.rebuild()
|
||||
: Promise.resolve(),
|
||||
vectorIndexSize === 0
|
||||
? this.index.rebuild()
|
||||
: Promise.resolve(),
|
||||
graphIndexSize === 0
|
||||
? this.graphIndex.rebuild()
|
||||
: Promise.resolve()
|
||||
])
|
||||
|
||||
const rebuildDuration = Date.now() - rebuildStartTime
|
||||
console.log(`✅ All indexes rebuilt in ${rebuildDuration}ms`)
|
||||
}
|
||||
|
||||
// STEP 4: Log statistics
|
||||
const stats = await this.stats()
|
||||
console.log(`📊 Brain initialized with ${stats.entities} entities`)
|
||||
}
|
||||
```
|
||||
|
||||
**Timeline** (typical cold start with 10K entities):
|
||||
- 0-50ms: Storage adapter initialization
|
||||
- 50-100ms: Field registry loading (O(1) discovery of persisted indices)
|
||||
- 100-200ms: Index lazy initialization (LSM-tree loading)
|
||||
- 200-500ms: Cache warming (preload common fields)
|
||||
- **No rebuild needed!** Registry discovers existing indices
|
||||
- Total: ~0.5-1 second (instant cold starts)
|
||||
|
||||
**Timeline** (cold start WITHOUT field registry - first run only):
|
||||
- 0-50ms: Storage adapter initialization
|
||||
- 50-100ms: Index lazy initialization
|
||||
- 100-2000ms: One-time rebuild to create indices
|
||||
- Total: ~1-3 seconds (one time only)
|
||||
|
||||
#### Mode 2: Lazy Loading on First Query
|
||||
|
||||
When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query:
|
||||
|
||||
```typescript
|
||||
// User code
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem' },
|
||||
disableAutoRebuild: true // Enable lazy loading
|
||||
})
|
||||
|
||||
await brain.init() // Returns instantly (0-10ms)
|
||||
|
||||
// First query triggers lazy rebuild
|
||||
const results = await brain.find({ limit: 10 })
|
||||
// → Calls ensureIndexesLoaded() internally (brainy.ts:4617)
|
||||
// → Rebuilds all 3 main indexes with concurrency control
|
||||
// → Returns results (~50-200ms total for 1K-10K entities)
|
||||
|
||||
// Subsequent queries are instant
|
||||
const more = await brain.find({ limit: 100 }) // 0ms check, instant
|
||||
```
|
||||
|
||||
**ensureIndexesLoaded() Implementation** (brainy.ts:4617-4664):
|
||||
```typescript
|
||||
private async ensureIndexesLoaded(): Promise<void> {
|
||||
// Fast path: Already loaded
|
||||
if (this.lazyRebuildCompleted) {
|
||||
return // 0ms
|
||||
}
|
||||
|
||||
// Concurrency control: Wait for in-progress rebuild
|
||||
if (this.lazyRebuildInProgress && this.lazyRebuildPromise) {
|
||||
await this.lazyRebuildPromise // Wait for same rebuild
|
||||
return
|
||||
}
|
||||
|
||||
// Check if storage has data
|
||||
const entities = await this.storage.getNouns({ pagination: { limit: 1 } })
|
||||
const hasData = (entities.totalCount && entities.totalCount > 0) || entities.items.length > 0
|
||||
|
||||
if (!hasData) {
|
||||
this.lazyRebuildCompleted = true
|
||||
return
|
||||
}
|
||||
|
||||
// Start lazy rebuild with mutex
|
||||
this.lazyRebuildInProgress = true
|
||||
this.lazyRebuildPromise = this.rebuildIndexesIfNeeded(true)
|
||||
.then(() => {
|
||||
this.lazyRebuildCompleted = true
|
||||
})
|
||||
.finally(() => {
|
||||
this.lazyRebuildInProgress = false
|
||||
this.lazyRebuildPromise = null
|
||||
})
|
||||
|
||||
await this.lazyRebuildPromise
|
||||
}
|
||||
```
|
||||
|
||||
**Lazy Loading Performance:**
|
||||
- First query: ~50-200ms (1K-10K entities) - triggers rebuild
|
||||
- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
|
||||
- Subsequent queries: 0ms check (instant)
|
||||
- Zero-config: Works automatically, no code changes needed
|
||||
|
||||
**Use Cases for Lazy Loading:**
|
||||
- **Serverless/Edge**: Minimize cold start time, indexes load on demand
|
||||
- **Development**: Faster restarts during development
|
||||
- **Large datasets**: Defer index loading until actually needed
|
||||
- **Read-heavy workloads**: Write operations don't wait for index rebuild
|
||||
|
||||
## Rebuild Process
|
||||
|
||||
### What "Rebuild" Actually Means
|
||||
|
||||
**IMPORTANT**: "Rebuild" does NOT mean recomputing data. It means:
|
||||
1. **Load persisted data** from storage (vector index connections, metadata chunks, LSM-tree SSTables)
|
||||
2. **Populate in-memory structures** (Maps, Sets, graphs)
|
||||
3. **Apply adaptive caching** (preload vectors if small dataset, lazy load if large)
|
||||
|
||||
**Complexity**: O(N) - linear scan through storage, NOT O(N log N) recomputation!
|
||||
|
||||
### 1. Vector Index Rebuild (Correct Pattern)
|
||||
|
||||
```typescript
|
||||
// src/hnsw/hnswIndex.ts (lines 809-947)
|
||||
public async rebuild(options: {
|
||||
lazy?: boolean
|
||||
batchSize?: number
|
||||
onProgress?: (loaded: number, total: number) => void
|
||||
} = {}): Promise<void> {
|
||||
// STEP 1: Clear in-memory structures
|
||||
this.clear()
|
||||
|
||||
// STEP 2: Load system data (entry point, max level)
|
||||
const systemData = await this.storage.getHNSWSystem()
|
||||
this.entryPointId = systemData.entryPointId
|
||||
this.maxLevel = systemData.maxLevel
|
||||
|
||||
// STEP 3: Determine preloading strategy (adaptive caching)
|
||||
const totalNouns = await this.storage.getNounCount()
|
||||
const vectorMemory = totalNouns * 384 * 4 // 384 dims × 4 bytes
|
||||
const availableCache = this.unifiedCache.getRemainingCapacity()
|
||||
const shouldPreload = vectorMemory < availableCache * 0.3
|
||||
|
||||
// STEP 4: Load entities with persisted vector index connections
|
||||
let hasMore = true
|
||||
let cursor: string | undefined = undefined
|
||||
|
||||
while (hasMore) {
|
||||
const result = await this.storage.getNouns({
|
||||
pagination: { limit: 1000, cursor }
|
||||
})
|
||||
|
||||
for (const nounData of result.items) {
|
||||
// Load vector graph data from storage (NOT recomputed!)
|
||||
const hnswData = await this.storage.getHNSWData(nounData.id)
|
||||
|
||||
// Create noun with restored connections
|
||||
const noun: HNSWNoun = {
|
||||
id: nounData.id,
|
||||
vector: shouldPreload ? nounData.vector : [], // Adaptive!
|
||||
connections: new Map(),
|
||||
level: hnswData.level
|
||||
}
|
||||
|
||||
// Restore connections from persisted data
|
||||
for (const [levelStr, nounIds] of Object.entries(hnswData.connections)) {
|
||||
const level = parseInt(levelStr, 10)
|
||||
noun.connections.set(level, new Set<string>(nounIds))
|
||||
}
|
||||
|
||||
// Just add to memory (no recomputation!)
|
||||
this.nouns.set(nounData.id, noun)
|
||||
}
|
||||
|
||||
hasMore = result.hasMore
|
||||
cursor = result.nextCursor
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
- ✅ Loads vector index connections from storage via `getHNSWData()`
|
||||
- ✅ Uses adaptive caching (preload vectors if < 30% of available cache)
|
||||
- ✅ O(N) complexity - just loads existing data
|
||||
- ❌ Does NOT call `addItem()` which would recompute connections (O(N log N))
|
||||
|
||||
### 2. TypeAwareVectorIndex Rebuild (Fixed in v3.45.0)
|
||||
|
||||
**Critical Architectural Fix**: The type-aware vector index previously had TWO major bugs:
|
||||
|
||||
1. **Bug #1**: Called `addItem()` during rebuild → O(N log N) recomputation instead of O(N) loading
|
||||
2. **Bug #2**: Loaded ALL nouns 31 times in parallel (once per type) → O(31*N) complexity causing timeouts
|
||||
|
||||
Both were fixed in v3.45.0 by loading ALL nouns ONCE and routing to correct type indexes:
|
||||
|
||||
```typescript
|
||||
// src/hnsw/typeAwareHNSWIndex.ts (lines 379-571)
|
||||
public async rebuild(options?: {
|
||||
lazy?: boolean
|
||||
batchSize?: number
|
||||
onProgress?: (loaded: number, total: number) => void
|
||||
}): Promise<void> {
|
||||
// STEP 1: Clear all type-specific indexes
|
||||
for (const index of this.typeIndexes.values()) {
|
||||
index.clear()
|
||||
}
|
||||
|
||||
// STEP 2: Determine preloading strategy (same as vector index)
|
||||
const totalNouns = await this.storage.getNounCount()
|
||||
const vectorMemory = totalNouns * 384 * 4
|
||||
const availableCache = this.unifiedCache.getRemainingCapacity()
|
||||
const shouldPreload = vectorMemory < availableCache * 0.3
|
||||
|
||||
// STEP 3: Load entities grouped by type
|
||||
for (const nounType of ALL_NOUN_TYPES) {
|
||||
const index = this.getOrCreateIndex(nounType)
|
||||
let hasMore = true
|
||||
let cursor: string | undefined = undefined
|
||||
|
||||
while (hasMore) {
|
||||
const result = await this.storage.getNouns({
|
||||
type: nounType,
|
||||
pagination: { limit: 1000, cursor }
|
||||
})
|
||||
|
||||
for (const nounData of result.items) {
|
||||
// CORRECT: Load persisted vector index data (not recomputed!)
|
||||
const hnswData = await this.storage.getHNSWData(nounData.id)
|
||||
|
||||
const noun = {
|
||||
id: nounData.id,
|
||||
vector: shouldPreload ? nounData.vector : [],
|
||||
connections: new Map(),
|
||||
level: hnswData.level
|
||||
}
|
||||
|
||||
// Restore connections from storage
|
||||
for (const [levelStr, nounIds] of Object.entries(hnswData.connections)) {
|
||||
const level = parseInt(levelStr, 10)
|
||||
noun.connections.set(level, new Set<string>(nounIds))
|
||||
}
|
||||
|
||||
// Add to in-memory index (no recomputation!)
|
||||
index.nouns.set(nounData.id, noun)
|
||||
}
|
||||
|
||||
hasMore = result.hasMore
|
||||
cursor = result.nextCursor
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Bug Fix**: Changed from `index.addItem()` (recomputation) to direct `nouns.set()` (restoration).
|
||||
|
||||
**Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities)
|
||||
|
||||
**Correct Pattern**:
|
||||
```typescript
|
||||
// Load ALL nouns ONCE (not 31 times!)
|
||||
while (hasMore) {
|
||||
const result = await storage.getNounsWithPagination({ limit: 1000, cursor })
|
||||
|
||||
for (const noun of result.items) {
|
||||
const type = noun.nounType || noun.metadata?.noun
|
||||
const index = this.getIndexForType(type)
|
||||
|
||||
// Load persisted HNSW data
|
||||
const hnswData = await storage.getHNSWData(noun.id)
|
||||
|
||||
// Restore connections (not recompute!)
|
||||
const restoredNoun = {
|
||||
id: noun.id,
|
||||
vector: shouldPreload ? noun.vector : [],
|
||||
connections: restoreConnections(hnswData),
|
||||
level: hnswData.level
|
||||
}
|
||||
|
||||
// Add to correct type index
|
||||
index.nouns.set(noun.id, restoredNoun)
|
||||
}
|
||||
|
||||
cursor = result.nextCursor
|
||||
hasMore = result.hasMore
|
||||
}
|
||||
```
|
||||
|
||||
**Performance Improvements**:
|
||||
- 31x speedup: Load nouns ONCE instead of 31 times (O(N) vs O(31*N))
|
||||
- 200-600x speedup: Load from storage instead of recomputing (O(N) vs O(N log N))
|
||||
- **Combined**: ~6000x speedup! (150 minutes → 1.5 seconds for 10K entities)
|
||||
|
||||
### 3. MetadataIndex Rebuild (v4.2.1+ with Field Registry)
|
||||
|
||||
**v4.2.1 Critical Fix**: Field registry persistence eliminates unnecessary rebuilds!
|
||||
|
||||
```typescript
|
||||
// src/utils/metadataIndex.ts (lines 202-216)
|
||||
async init(): Promise<void> {
|
||||
// STEP 1: Load field registry to discover persisted indices
|
||||
// This is THE KEY FIX - O(1) discovery of existing indices
|
||||
await this.loadFieldRegistry()
|
||||
|
||||
// If registry found, fieldIndexes Map is now populated
|
||||
// getStats() will return totalEntries > 0 → skips rebuild!
|
||||
|
||||
// STEP 2: Initialize EntityIdMapper
|
||||
await this.idMapper.init()
|
||||
|
||||
// STEP 3: Warm cache with discovered fields
|
||||
await this.warmCache()
|
||||
}
|
||||
|
||||
async loadFieldRegistry(): Promise<void> {
|
||||
const registry = await this.storage.getMetadata('__metadata_field_registry__')
|
||||
|
||||
if (registry?.fields) {
|
||||
// Populate fieldIndexes Map from discovered fields
|
||||
// Sparse indices are lazy-loaded when first accessed
|
||||
for (const field of registry.fields) {
|
||||
this.fieldIndexes.set(field, {
|
||||
values: {},
|
||||
lastUpdated: registry.lastUpdated
|
||||
})
|
||||
}
|
||||
// Result: getStats() now returns totalEntries > 0
|
||||
// → Brain skips rebuild, cold start in 2-3 seconds!
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Rebuild Only Happens If**:
|
||||
1. **First run** (no field registry exists yet)
|
||||
2. **Registry corruption** (rare)
|
||||
3. **Explicit rebuild request** (manual operation)
|
||||
|
||||
```typescript
|
||||
// Only runs if field registry not found
|
||||
async rebuild(): Promise<void> {
|
||||
// STEP 1: Clear in-memory structures
|
||||
this.fieldIndexes.clear()
|
||||
|
||||
// STEP 2: Load all entity metadata and rebuild indices
|
||||
// Sequential batching (25/batch) to prevent socket exhaustion
|
||||
// After rebuild: Field registry saved during next flush()
|
||||
|
||||
// One-time cost: ~2-3 seconds for 1K entities
|
||||
}
|
||||
```
|
||||
|
||||
**Performance Comparison**:
|
||||
|
||||
| Version | Cold Start | Discovery Method | Rebuild Needed? |
|
||||
|---------|------------|------------------|-----------------|
|
||||
| v4.2.0 | 8-9 min | None (always rebuild) | Always |
|
||||
| v4.2.1 | 2-3 sec | Field registry O(1) | First run only |
|
||||
|
||||
**Key Points**:
|
||||
- ✅ Field registry enables O(1) discovery (4-8KB file)
|
||||
- ✅ Sparse indices lazy-loaded on first query
|
||||
- ✅ Bloom filters + zone maps loaded for fast filtering
|
||||
- ✅ One-time rebuild on first run, then instant restarts forever
|
||||
- ✅ Automatic: No configuration needed
|
||||
|
||||
### 4. GraphAdjacencyIndex Rebuild
|
||||
|
||||
```typescript
|
||||
// src/graph/graphAdjacencyIndex.ts (lines 279-336)
|
||||
async rebuild(): Promise<void> {
|
||||
// STEP 1: Clear in-memory caches
|
||||
this.verbIndex.clear()
|
||||
this.relationshipCountsByType.clear()
|
||||
|
||||
// STEP 2: Load all verbs from storage
|
||||
let hasMore = true
|
||||
let cursor: string | undefined = undefined
|
||||
|
||||
while (hasMore) {
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { limit: 1000, cursor }
|
||||
})
|
||||
|
||||
for (const verb of result.items) {
|
||||
// Add to index (which updates LSM-trees)
|
||||
await this.addVerb(verb)
|
||||
}
|
||||
|
||||
hasMore = result.hasMore
|
||||
cursor = result.nextCursor
|
||||
}
|
||||
|
||||
// Note: LSM-trees (lsmTreeSource, lsmTreeTarget) are already
|
||||
// initialized from persisted SSTables during ensureInitialized()
|
||||
}
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
- ✅ LSM-tree SSTables already loaded during `init()`
|
||||
- ✅ Rebuild just repopulates verb cache
|
||||
- ✅ O(E) complexity where E = number of edges
|
||||
|
||||
## Adaptive Memory Management
|
||||
|
||||
### Strategy: Preload vs Lazy Load
|
||||
|
||||
All indexes use the **UnifiedCache** to determine memory allocation:
|
||||
|
||||
```typescript
|
||||
// Decision logic (in all indexes)
|
||||
const totalDataSize = estimateDataSize()
|
||||
const availableCache = unifiedCache.getRemainingCapacity()
|
||||
|
||||
if (totalDataSize < availableCache * 0.3) {
|
||||
// PRELOAD: Dataset is small relative to available memory
|
||||
// Load everything into memory for maximum performance
|
||||
shouldPreload = true
|
||||
} else {
|
||||
// LAZY LOAD: Dataset is large
|
||||
// Load on-demand with LRU eviction
|
||||
shouldPreload = false
|
||||
}
|
||||
```
|
||||
|
||||
**Thresholds**:
|
||||
- **< 30% of available cache**: Preload all vectors
|
||||
- **> 30% of available cache**: Lazy load on demand
|
||||
|
||||
**Example** (default 100MB cache):
|
||||
- 10K entities × 1.5KB = 15MB → **Preload** (15MB < 30MB)
|
||||
- 100K entities × 1.5KB = 150MB → **Lazy load** (150MB > 30MB)
|
||||
|
||||
### UnifiedCache Integration
|
||||
|
||||
```typescript
|
||||
// All indexes share the same cache
|
||||
const unifiedCache = getGlobalCache() // Singleton, 100MB default
|
||||
|
||||
// MetadataIndex
|
||||
this.unifiedCache = unifiedCache
|
||||
|
||||
// Vector index
|
||||
this.unifiedCache = unifiedCache
|
||||
|
||||
// GraphAdjacencyIndex
|
||||
this.unifiedCache = unifiedCache
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- Fair resource allocation across indexes
|
||||
- Prevents any single index from monopolizing memory
|
||||
- Coordinated LRU eviction system-wide
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Rebuild Times (Typical Hardware)
|
||||
|
||||
| Dataset Size | Metadata | Vector | Graph | Total (Parallel) |
|
||||
|--------------|----------|------|-------|------------------|
|
||||
| 1K entities | 50ms | 100ms | 30ms | **150ms** |
|
||||
| 10K entities | 200ms | 500ms | 150ms | **600ms** |
|
||||
| 100K entities | 1s | 3s | 1s | **3.5s** |
|
||||
| 1M entities | 8s | 25s | 10s | **28s** |
|
||||
|
||||
**Note**: Parallel rebuild means total time ≈ max(individual times), not sum.
|
||||
|
||||
### Memory Overhead
|
||||
|
||||
| Index | In-Memory Overhead | Disk Storage |
|
||||
|-------|-------------------|--------------|
|
||||
| **MetadataIndex** | ~100 bytes/entity | ~500 bytes/entity (chunks) |
|
||||
| **Vector Index** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
|
||||
| **GraphAdjacencyIndex** | ~128 bytes/relationship | ~200 bytes/relationship (LSM-tree) |
|
||||
| **DeletedItemsIndex** | ~40 bytes/deleted ID | ~50 bytes/deleted ID |
|
||||
|
||||
**Total overhead** (lazy loading):
|
||||
- **In-memory**: ~300 bytes per entity + ~128 bytes per relationship
|
||||
- **On-disk**: ~2 KB per entity + ~200 bytes per relationship
|
||||
|
||||
### O(N) vs O(N log N) Comparison
|
||||
|
||||
**Before fix** (TypeAwareVectorIndex bug):
|
||||
```typescript
|
||||
// BAD: Recomputes vector index connections during rebuild
|
||||
for (const noun of nouns) {
|
||||
await index.addItem(noun) // O(log N) per item → O(N log N) total
|
||||
}
|
||||
// 10K entities: ~5 minutes
|
||||
```
|
||||
|
||||
**After fix** (correct pattern):
|
||||
```typescript
|
||||
// GOOD: Loads connections from storage
|
||||
for (const noun of nouns) {
|
||||
const hnswData = await storage.getHNSWData(noun.id) // O(1) per item
|
||||
noun.connections = restoreConnections(hnswData) // O(1) per item
|
||||
index.nouns.set(noun.id, noun) // O(1) per item
|
||||
}
|
||||
// 10K entities: ~500ms (600x faster!)
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Cold Start (Empty Storage)
|
||||
|
||||
```typescript
|
||||
const brain = new Brain({ storage })
|
||||
|
||||
// First init: All indexes are empty
|
||||
await brain.init()
|
||||
// → No rebuild needed, indexes start empty
|
||||
|
||||
// Add data
|
||||
await brain.add({ content: 'Hello', noun: 'message' })
|
||||
|
||||
// Second init: Indexes populated
|
||||
const brain2 = new Brain({ storage })
|
||||
await brain2.init()
|
||||
// → Rebuilds all indexes from storage (~1-3s for 10K entities)
|
||||
```
|
||||
|
||||
### Warm Start (Storage Already Populated)
|
||||
|
||||
```typescript
|
||||
const brain = new Brain({ storage })
|
||||
|
||||
// Init with existing data
|
||||
await brain.init()
|
||||
// → Detects non-empty storage
|
||||
// → Rebuilds indexes in parallel
|
||||
// → Uses adaptive caching (preload if small, lazy if large)
|
||||
```
|
||||
|
||||
### Manual Rebuild
|
||||
|
||||
```typescript
|
||||
const brain = new Brain({ storage })
|
||||
await brain.init()
|
||||
|
||||
// Force rebuild (e.g., after data corruption)
|
||||
await brain.metadataIndex.rebuild()
|
||||
await brain.index.rebuild()
|
||||
await brain.graphIndex.rebuild()
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Slow Rebuild Times
|
||||
|
||||
**Symptom**: Rebuild takes minutes instead of seconds
|
||||
|
||||
**Diagnosis**:
|
||||
```typescript
|
||||
// Check if rebuild is recomputing instead of loading
|
||||
console.time('rebuild')
|
||||
await brain.index.rebuild()
|
||||
console.timeEnd('rebuild')
|
||||
|
||||
// For 10K entities:
|
||||
// - Expected: 500-800ms (loading from storage)
|
||||
// - Bug: 5-10 minutes (recomputing vector index connections)
|
||||
```
|
||||
|
||||
**Solution**: Ensure index is loading from storage, not calling `addItem()` during rebuild.
|
||||
|
||||
### High Memory Usage
|
||||
|
||||
**Symptom**: Memory usage exceeds expectations
|
||||
|
||||
**Diagnosis**:
|
||||
```typescript
|
||||
// Check if vectors are being preloaded
|
||||
const stats = brain.index.getStats()
|
||||
console.log('Preloaded vectors:', stats.preloadedVectors)
|
||||
|
||||
// Expected:
|
||||
// - Small dataset (< 30% cache): Most vectors preloaded
|
||||
// - Large dataset (> 30% cache): Few vectors preloaded
|
||||
```
|
||||
|
||||
**Solution**: Adjust `UnifiedCache` size or force lazy loading:
|
||||
```typescript
|
||||
const brain = new Brain({
|
||||
storage,
|
||||
cache: { maxSize: 50 * 1024 * 1024 } // 50MB cache
|
||||
})
|
||||
```
|
||||
|
||||
### Missing Data After Rebuild
|
||||
|
||||
**Symptom**: Entities disappear after restart
|
||||
|
||||
**Diagnosis**:
|
||||
```typescript
|
||||
// Check storage persistence
|
||||
const nouns = await storage.getNouns({ pagination: { limit: 10 } })
|
||||
console.log('Nouns in storage:', nouns.items.length)
|
||||
|
||||
// If empty: Storage not persisting
|
||||
// If populated: Rebuild not loading correctly
|
||||
```
|
||||
|
||||
**Solution**: Verify storage adapter is configured correctly (e.g., FileSystem path exists).
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Index Architecture](./index-architecture.md) - Data structures and operations
|
||||
- [Storage Architecture](./storage-architecture.md) - Storage layer details
|
||||
- [Performance Guide](../PERFORMANCE.md) - Performance tuning
|
||||
- [Scaling Guide](../SCALING.md) - Large dataset optimization
|
||||
|
||||
## Version History
|
||||
|
||||
- **v5.7.7** (November 2025): Added production-scale lazy loading with `ensureIndexesLoaded()` helper. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added concurrency control (mutex) to prevent duplicate rebuilds from concurrent queries. Added `getIndexStatus()` diagnostic method. Zero-config operation - works automatically.
|
||||
- **v3.45.0** (October 2025): Fixed type-aware vector index `rebuild()` to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
|
||||
- **v3.44.0** (October 2025): GraphAdjacencyIndex migrated to LSM-tree storage for billion-scale relationships
|
||||
- **v3.42.0** (October 2025): MetadataIndex migrated to chunked sparse indexing
|
||||
- **v3.35.0** (August 2025): Vector index connections first persisted to storage
|
||||
- **v3.0.0** (September 2025): Initial 3-tier index architecture
|
||||
134
docs/architecture/multiprocess-storage-mixin.md
Normal file
134
docs/architecture/multiprocess-storage-mixin.md
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# Design note: multi-process storage mixin
|
||||
|
||||
**Status:** Proposed (future minor)
|
||||
**Owner:** Brainy core
|
||||
**Filed:** 2026-05-15
|
||||
**Companion:** [`concepts/storage-adapters`](../concepts/storage-adapters.md)
|
||||
|
||||
## Context
|
||||
|
||||
Brainy 7.21 added seven storage-adapter methods to support multi-process
|
||||
safety:
|
||||
|
||||
```
|
||||
supportsMultiProcessLocking()
|
||||
acquireWriterLock(opts)
|
||||
releaseWriterLock()
|
||||
readWriterLock()
|
||||
startFlushRequestWatcher(cb)
|
||||
stopFlushRequestWatcher()
|
||||
requestFlushOverFilesystem(timeoutMs)
|
||||
```
|
||||
|
||||
They live on `BaseStorage` as no-op defaults and are overridden on
|
||||
`FileSystemStorage` with real implementations. Any adapter extending
|
||||
`FileSystemStorage` (e.g. Cor's `MmapFileSystemStorage`) inherits the
|
||||
real ones for free.
|
||||
|
||||
This works correctly today. The question is whether the methods *belong*
|
||||
on `BaseStorage`.
|
||||
|
||||
## The case for moving them out
|
||||
|
||||
`BaseStorage` already mixes several concerns:
|
||||
- entity / verb CRUD primitives
|
||||
- generational record hooks (8.0 MVCC)
|
||||
- type-statistics tracking
|
||||
- count persistence
|
||||
- multi-process safety (new)
|
||||
|
||||
Adapters that have no notion of multi-process semantics — `MemoryStorage`,
|
||||
cloud adapters (S3, GCS, R2, Azure, OPFS) — still carry seven inherited
|
||||
no-ops on their prototype chain. A reader can't tell from the class
|
||||
declaration whether a given adapter participates in the locking protocol;
|
||||
it has to call `supportsMultiProcessLocking()` and trust the answer.
|
||||
|
||||
A cleaner separation:
|
||||
|
||||
```typescript
|
||||
interface MultiProcessSafeStorage {
|
||||
supportsMultiProcessLocking(): boolean
|
||||
acquireWriterLock(opts?: { force?: boolean }): Promise<WriterLockInfo | null>
|
||||
releaseWriterLock(): Promise<void>
|
||||
readWriterLock(): Promise<WriterLockInfo | null>
|
||||
startFlushRequestWatcher(cb: () => Promise<void>): void
|
||||
stopFlushRequestWatcher(): void
|
||||
requestFlushOverFilesystem(timeoutMs: number): Promise<boolean>
|
||||
}
|
||||
|
||||
function isMultiProcessSafe(s: BaseStorage): s is BaseStorage & MultiProcessSafeStorage {
|
||||
return typeof (s as any).supportsMultiProcessLocking === 'function'
|
||||
&& (s as any).supportsMultiProcessLocking()
|
||||
}
|
||||
```
|
||||
|
||||
Brainy's call sites become:
|
||||
|
||||
```typescript
|
||||
if (this.config.mode !== 'reader' && isMultiProcessSafe(this.storage)) {
|
||||
await this.storage.acquireWriterLock({ force: this.config.force })
|
||||
// ... TypeScript narrows the rest correctly ...
|
||||
}
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- Type system enforces the capability — no more `(this.storage as any).X()`.
|
||||
- Adapters that opt out (memory, cloud) are visibly clean.
|
||||
- `hasStorageMethod()` defensive helper can stay (still guards
|
||||
build/install artifacts) but doesn't carry the conceptual weight of
|
||||
"did the plugin implement the interface."
|
||||
- ADR-style trail for future capability additions: each new capability
|
||||
gets its own interface, opted into explicitly.
|
||||
|
||||
## The case against doing it now
|
||||
|
||||
- Breaking change for any adapter that already overrides these methods.
|
||||
`FileSystemStorage` is the only one in-tree, but Cor's
|
||||
`MmapFileSystemStorage` inherits from it — interface relocation would
|
||||
ripple through the plugin ecosystem.
|
||||
- The current state works. The real failure modes seen in the field
|
||||
were build/install artifacts, not type-system failures.
|
||||
- 7.22.0 just shipped a clean fix. Stacking another refactor before
|
||||
consumers absorb it adds churn without urgency.
|
||||
- The `hasStorageMethod()` guard accomplishes the same runtime safety the
|
||||
interface narrowing would in TypeScript-aware code.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Defer.** Keep the current architecture through the 7.x line. Revisit
|
||||
when:
|
||||
- A second multi-process capability lands (e.g. distributed-readers
|
||||
coordination) and the natural surface area is more than seven
|
||||
methods. Five+ becomes the moment a separate interface earns its
|
||||
keep.
|
||||
- A v8 major is on the table for unrelated reasons. Bundle the
|
||||
interface extraction with that release so consumers absorb both
|
||||
changes in one upgrade.
|
||||
|
||||
Until then:
|
||||
- Document the inheritance contract (done — see
|
||||
[`concepts/storage-adapters`](../concepts/storage-adapters.md)).
|
||||
- Keep `hasStorageMethod()` as the runtime guard.
|
||||
- Don't add new methods to `BaseStorage` defaults without re-evaluating
|
||||
the surface-area boundary.
|
||||
|
||||
## Migration sketch (when we do it)
|
||||
|
||||
For reference, a clean migration path:
|
||||
|
||||
1. Add `MultiProcessSafeStorage` interface to `src/storage/coreTypes.ts`.
|
||||
2. Move the seven method signatures from `BaseStorage` to the new
|
||||
interface. Default implementations stay on `BaseStorage` but only as
|
||||
private helpers consumed by `FileSystemStorage`'s explicit
|
||||
implementations.
|
||||
3. `FileSystemStorage implements MultiProcessSafeStorage` becomes
|
||||
explicit; methods get the `public` modifier with full JSDoc.
|
||||
4. Brainy call sites switch from `hasStorageMethod` to
|
||||
`isMultiProcessSafe` type-guard. Keep `hasStorageMethod` for
|
||||
build/install artifact protection.
|
||||
5. Document the new contract in `concepts/storage-adapters.md`.
|
||||
6. Major-version-bump the `@soulcraft/brainy` peerDep range expected by
|
||||
plugins.
|
||||
|
||||
Estimated work: ~half a day of code, ~2 hours of doc/example updates,
|
||||
ecosystem coordination via the platform handoff.
|
||||
1556
docs/architecture/noun-verb-taxonomy.md
Normal file
1556
docs/architecture/noun-verb-taxonomy.md
Normal file
File diff suppressed because it is too large
Load diff
150
docs/architecture/overview.md
Normal file
150
docs/architecture/overview.md
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
# Architecture Overview
|
||||
|
||||
Brainy is a multi-dimensional AI database that combines vector similarity, graph relationships, and metadata filtering into a unified query system. This document provides a comprehensive overview of the system architecture.
|
||||
|
||||
## Core Components
|
||||
|
||||
### Brainy (Main Entry Point)
|
||||
The central orchestrator that manages all subsystems:
|
||||
- **4-Index Architecture**: MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md))
|
||||
- **Storage System**: FileSystem and Memory adapters
|
||||
- **Augmentation System**: Extensible plugin architecture
|
||||
- **Triple Intelligence**: Unified query engine
|
||||
|
||||
### Triple Intelligence Engine
|
||||
Brainy's revolutionary feature that unifies three types of search:
|
||||
- **Vector Search**: Semantic similarity via the pluggable vector index
|
||||
- **Graph Traversal**: Relationship-based queries
|
||||
- **Field Filtering**: Precise metadata filtering with O(1) performance
|
||||
|
||||
```typescript
|
||||
// Single query combining all three intelligence types
|
||||
const results = await brain.find({
|
||||
like: "machine learning papers", // Vector similarity
|
||||
connected: { to: "research-team", depth: 2 }, // Graph traversal
|
||||
where: { published: { $gte: "2024-01-01" } } // Metadata filtering
|
||||
})
|
||||
```
|
||||
|
||||
### Storage Architecture
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
├── _system/ # System management
|
||||
│ └── statistics.json
|
||||
├── nouns/ # Entity data storage
|
||||
│ └── {uuid}.json
|
||||
├── metadata/ # Metadata and indexing
|
||||
│ ├── {uuid}.json
|
||||
│ ├── __entity_registry__.json
|
||||
│ └── __metadata_index__*.json
|
||||
├── verbs/ # Relationship storage
|
||||
└── locks/ # Concurrent access control
|
||||
```
|
||||
|
||||
### Vector Index
|
||||
Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph:
|
||||
- **Performance**: O(log n) search complexity
|
||||
- **Configurable recall**: `fast` / `balanced` / `accurate` presets trade recall for latency
|
||||
- **Scalable**: Handles millions of vectors per process
|
||||
- **Persistent**: Serializable to storage
|
||||
- **Swappable**: Replace with a native implementation (such as `@soulcraft/cor`) via the plugin system without changing application code
|
||||
|
||||
### Metadata Index Manager
|
||||
High-performance field indexing system:
|
||||
- **O(1) Lookups**: Inverted index for field→value→IDs mapping
|
||||
- **Query Support**: equals, anyOf, allOf, range queries
|
||||
- **Chunked Storage**: Supports massive datasets
|
||||
- **Auto-indexing**: Automatically maintains indexes on updates
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Operation Complexity
|
||||
- **Vector Search**: O(log n) via the vector index
|
||||
- **Field Filtering**: O(1) via inverted indexes
|
||||
- **Graph Traversal**: O(V + E) for breadth-first search
|
||||
- **Add Operation**: O(log n) for index insertion
|
||||
- **Update Operation**: O(1) for metadata updates
|
||||
|
||||
### Memory Usage
|
||||
- **Base Memory**: ~50MB for core system
|
||||
- **Per Vector**: ~1KB (384 dimensions × 4 bytes)
|
||||
- **Index Overhead**: ~20% of vector data
|
||||
- **Cache Size**: Configurable (default 1000 entries)
|
||||
|
||||
### Throughput
|
||||
- **Writes**: 1000+ ops/second (with batching)
|
||||
- **Reads**: 10,000+ ops/second
|
||||
- **Search**: 100+ queries/second (varies by complexity)
|
||||
|
||||
## Augmentation System
|
||||
|
||||
Brainy's extensible plugin architecture allows for powerful enhancements:
|
||||
|
||||
### Core Augmentations
|
||||
- **Entity Registry**: High-speed deduplication for streaming data
|
||||
- **Batch Processing**: Optimized bulk operations
|
||||
- **Request Deduplicator**: Prevents duplicate processing
|
||||
|
||||
### Creating Custom Augmentations
|
||||
```typescript
|
||||
class CustomAugmentation extends BrainyAugmentation {
|
||||
async onInit(brain: Brainy): Promise<void> {
|
||||
// Initialize augmentation
|
||||
}
|
||||
|
||||
async onAdd(item: any, brain: Brainy): Promise<any> {
|
||||
// Process item before adding
|
||||
return item
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Caching Strategy
|
||||
|
||||
Multi-layered caching for optimal performance:
|
||||
- **Search Cache**: LRU cache for query results
|
||||
- **Metadata Cache**: Field index caching
|
||||
- **Pattern Cache**: NLP pattern matching cache
|
||||
- **Entity Cache**: In-memory entity registry
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Key Objects for Extensions
|
||||
- `brain.index`: Access the vector index
|
||||
- `brain.metadataIndex`: Access field indexing
|
||||
- `brain.graphIndex`: Access graph adjacency index
|
||||
- `brain.storage`: Access storage layer
|
||||
- `brain.augmentations`: Access augmentation manager
|
||||
|
||||
For detailed information about each index, see [Index Architecture](./index-architecture.md).
|
||||
|
||||
### Event System
|
||||
```typescript
|
||||
brain.on('add', (item) => console.log('Item added:', item))
|
||||
brain.on('search', (query) => console.log('Search performed:', query))
|
||||
brain.on('error', (error) => console.error('Error:', error))
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When Adding Features
|
||||
1. Check if similar functionality exists
|
||||
2. Consider if it should be an augmentation
|
||||
3. Use existing indexes and caches
|
||||
4. Avoid duplicating functionality
|
||||
5. Follow the established patterns
|
||||
|
||||
### Performance Optimization
|
||||
1. Use batch operations for bulk data
|
||||
2. Enable appropriate caching
|
||||
3. Choose the right storage adapter
|
||||
4. Configure index parameters for your use case
|
||||
5. Monitor statistics for bottlenecks
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Index Architecture](./index-architecture.md) - Deep dive into the 4-index system
|
||||
- [Storage Architecture](./storage-architecture.md) - Deep dive into storage system
|
||||
- [Triple Intelligence](./triple-intelligence.md) - Advanced query system
|
||||
- [API Reference](../api/README.md) - Complete API documentation
|
||||
318
docs/architecture/storage-architecture.md
Normal file
318
docs/architecture/storage-architecture.md
Normal file
|
|
@ -0,0 +1,318 @@
|
|||
# Storage Architecture
|
||||
|
||||
> **Updated**: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup
|
||||
|
||||
## Storage Structure
|
||||
|
||||
### Architecture: Metadata/Vector Separation
|
||||
|
||||
Entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
├── _system/ # System metadata (not sharded)
|
||||
│ ├── statistics.json # Performance metrics
|
||||
│ ├── __metadata_field_index__*.json # Field indexes
|
||||
│ └── __metadata_sorted_index__*.json # Sorted indexes
|
||||
│
|
||||
├── entities/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # Vector graph data (sharded by UUID)
|
||||
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
|
||||
│ │ │ │ ├── 00123456-....json # Vector + graph connections
|
||||
│ │ │ │ └── 00abcdef-....json
|
||||
│ │ │ ├── 01/ ... ff/ # 256 shards total
|
||||
│ │ │
|
||||
│ │ └── metadata/ # Business data (sharded by UUID)
|
||||
│ │ ├── 00/
|
||||
│ │ │ ├── 00123456-....json # Entity metadata only
|
||||
│ │ │ └── 00abcdef-....json
|
||||
│ │ ├── 01/ ... ff/
|
||||
│ │
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (sharded)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ │
|
||||
│ └── metadata/ # Relationship data (sharded)
|
||||
│ ├── 00/ ... ff/
|
||||
```
|
||||
|
||||
### Why Split Metadata and Vectors?
|
||||
|
||||
**Performance at scale:**
|
||||
- **Vector search operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
|
||||
- **Filtering**: Only load metadata during filtering, not vectors
|
||||
- **Pagination**: Load metadata IDs first, fetch vectors/metadata on-demand
|
||||
- **Result**: 60-70% reduction in I/O for typical queries at million-entity scale
|
||||
|
||||
### UUID-Based Sharding (256 Shards)
|
||||
|
||||
**How it works:**
|
||||
```typescript
|
||||
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
|
||||
const shard = uuid.substring(0, 2) // "3f"
|
||||
|
||||
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
|
||||
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- **Uniform distribution**: ~3,900 entities per shard (at 1M scale)
|
||||
- **Filesystem optimization**: avoids huge flat directories that bog down `readdir`
|
||||
- **Parallel operations**: walk 256 shards in parallel
|
||||
- **Predictable**: Deterministic shard assignment
|
||||
|
||||
## Storage Adapters
|
||||
|
||||
Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` interface:
|
||||
|
||||
### FileSystem Storage (Node.js, default)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Server applications, CLI tools, single-node deployments
|
||||
- **Performance**: Direct file I/O
|
||||
- **Persistence**: Permanent on disk
|
||||
- **Features**:
|
||||
- **Batch Delete**: Efficient bulk deletion with retries
|
||||
- **UUID Sharding**: Automatic 256-shard distribution
|
||||
|
||||
### Memory Storage
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'memory'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Tests, ephemeral workloads, single-process caches
|
||||
- **Performance**: No I/O — all data lives in process memory
|
||||
- **Persistence**: None — data is lost when the process exits
|
||||
|
||||
### Auto
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'auto',
|
||||
path: './data'
|
||||
}
|
||||
})
|
||||
```
|
||||
`'auto'` picks `'filesystem'` when running on Node.js with a writable `path`, and falls back to `'memory'` otherwise.
|
||||
|
||||
## Backup and Off-Site Replication
|
||||
|
||||
Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `path` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:
|
||||
|
||||
- `gsutil rsync -r ./data gs://my-bucket/brainy-data`
|
||||
- `aws s3 sync ./data s3://my-bucket/brainy-data`
|
||||
- `rclone sync ./data remote:brainy-data`
|
||||
- Periodic `tar` snapshots to any object store
|
||||
|
||||
Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory.
|
||||
|
||||
## Metadata Indexing System
|
||||
|
||||
### Field Discovery Index
|
||||
Tracks all unique values for each field:
|
||||
|
||||
```json
|
||||
// __metadata_field_index__field_category.json
|
||||
{
|
||||
"values": {
|
||||
"technology": 45,
|
||||
"science": 32,
|
||||
"business": 28
|
||||
},
|
||||
"lastUpdated": 1699564234567
|
||||
}
|
||||
```
|
||||
|
||||
### Value-Based Indexes
|
||||
Maps field+value combinations to entity IDs:
|
||||
|
||||
```json
|
||||
// __metadata_index__category_technology_chunk0.json
|
||||
{
|
||||
"field": "category",
|
||||
"value": "technology",
|
||||
"ids": ["uuid1", "uuid2", "uuid3", ...],
|
||||
"chunk": 0,
|
||||
"total": 45
|
||||
}
|
||||
```
|
||||
|
||||
### Index Chunking
|
||||
Large indexes automatically chunk for performance:
|
||||
- **Chunk size**: 10,000 IDs per chunk
|
||||
- **Auto-splitting**: Transparent to queries
|
||||
- **Parallel loading**: Chunks load on demand
|
||||
|
||||
## Entity Registry
|
||||
|
||||
High-performance deduplication system for streaming data:
|
||||
|
||||
### Registry Structure
|
||||
```json
|
||||
// __entity_registry__.json
|
||||
{
|
||||
"mappings": {
|
||||
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
|
||||
},
|
||||
"stats": {
|
||||
"totalMappings": 10000,
|
||||
"lastSync": 1699564234567
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Performance Characteristics
|
||||
- **Lookup**: O(1) in-memory hash map
|
||||
- **Persistence**: Configurable (memory/storage/hybrid)
|
||||
- **Cache**: LRU with configurable TTL
|
||||
- **Sync**: Periodic or on-demand
|
||||
|
||||
## Durability
|
||||
|
||||
Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `path` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).
|
||||
|
||||
## Storage Optimization
|
||||
|
||||
### 1. Batch Operations
|
||||
|
||||
```typescript
|
||||
// Efficient batch delete
|
||||
await storage.batchDelete([
|
||||
'entities/nouns/vectors/00/00123456-....json',
|
||||
'entities/nouns/metadata/00/00123456-....json'
|
||||
// ...
|
||||
])
|
||||
|
||||
// Batch writes for performance
|
||||
await brain.addBatch([
|
||||
{ content: "item1", metadata: {} },
|
||||
{ content: "item2", metadata: {} },
|
||||
{ content: "item3", metadata: {} }
|
||||
])
|
||||
// Single transaction, optimized I/O
|
||||
```
|
||||
|
||||
### 2. Caching Strategy
|
||||
|
||||
```typescript
|
||||
// Configure caching
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
ttl: 300000, // 5 minutes
|
||||
strategy: 'lru' // Least recently used
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Concurrent Access
|
||||
|
||||
### Locking Mechanism
|
||||
```typescript
|
||||
// Automatic locking for write operations
|
||||
await brain.storage.withLock('resource-id', async () => {
|
||||
// Exclusive access to resource
|
||||
await brain.storage.saveNoun(id, data)
|
||||
})
|
||||
```
|
||||
|
||||
### Read-Write Separation
|
||||
- **Reads**: Non-blocking, parallel
|
||||
- **Writes**: Serialized with locks
|
||||
- **Hybrid**: Read-heavy optimization
|
||||
|
||||
## Migration and Backup
|
||||
|
||||
Backup and restore go through the Db API — see
|
||||
[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) for the full
|
||||
recipe book.
|
||||
|
||||
### Snapshot (backup)
|
||||
```typescript
|
||||
// Instant, self-contained snapshot (hard links on filesystem storage)
|
||||
const db = brain.now()
|
||||
await db.persist('/backups/2026-06-11')
|
||||
await db.release()
|
||||
```
|
||||
|
||||
### Restore
|
||||
```typescript
|
||||
// Replace the store's entire state from a snapshot (destructive — confirm required)
|
||||
await brain.restore('/backups/2026-06-11', { confirm: true })
|
||||
```
|
||||
|
||||
### Move to a new directory
|
||||
```typescript
|
||||
// A snapshot directory is a complete store: restore it into a fresh brain
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', path: './new' } })
|
||||
await brain.init()
|
||||
await brain.restore('/backups/2026-06-11', { confirm: true })
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### FileSystem Optimizations
|
||||
- **Directory sharding**: 256 shards spread files across subdirectories
|
||||
- **Async I/O**: Non-blocking file operations
|
||||
- **Buffer pooling**: Reuse buffers for efficiency
|
||||
|
||||
### Monitoring
|
||||
|
||||
```typescript
|
||||
// Get storage statistics
|
||||
const stats = await brain.storage.getStatistics()
|
||||
console.log(stats)
|
||||
// {
|
||||
// totalSize: 1048576,
|
||||
// entityCount: 1000,
|
||||
// indexSize: 204800,
|
||||
// walSize: 10240,
|
||||
// cacheHitRate: 0.85
|
||||
// }
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Choose the Right Adapter
|
||||
1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence
|
||||
2. **Single-process production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
|
||||
3. **Horizontal scaling**: Brainy runs in one process — there is no built-in cluster. Run independent instances behind a service layer and replicate the on-disk artifact with your operator tooling; or run many reader processes against one shared store with a single writer
|
||||
|
||||
### Optimize for Your Use Case
|
||||
1. **Read-heavy**: Enable caching and let the OS page cache do its job
|
||||
2. **Write-heavy**: Batch operations and tune the cache `maxSize`
|
||||
3. **Real-time**: FileSystem with periodic snapshots
|
||||
4. **Archival**: Snapshot `path` to cold object storage on a schedule
|
||||
5. **Large-scale**: Rely on metadata/vector separation + UUID sharding
|
||||
|
||||
### Monitor and Maintain
|
||||
1. Regular statistics collection
|
||||
2. Watch disk usage and shard balance
|
||||
3. Index optimization
|
||||
4. Cache tuning based on hit rates
|
||||
5. Verify backup runs (test restore quarterly)
|
||||
|
||||
## API Reference
|
||||
|
||||
See the [Storage API](../api/storage.md) for complete method documentation.
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026
|
||||
**Key Features**: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup
|
||||
372
docs/architecture/triple-intelligence.md
Normal file
372
docs/architecture/triple-intelligence.md
Normal file
|
|
@ -0,0 +1,372 @@
|
|||
---
|
||||
title: Triple Intelligence
|
||||
slug: concepts/triple-intelligence
|
||||
public: true
|
||||
category: concepts
|
||||
template: concept
|
||||
order: 1
|
||||
description: Unified vector similarity, graph traversal, and metadata filtering in one query. Auto-optimizes between parallel execution and progressive filtering.
|
||||
next:
|
||||
- concepts/noun-types
|
||||
- api/reference
|
||||
---
|
||||
|
||||
# Triple Intelligence System
|
||||
|
||||
The Triple Intelligence System is Brainy's revolutionary query engine that unifies vector similarity, graph relationships, and metadata filtering into a single, optimized query interface.
|
||||
|
||||
## Overview
|
||||
|
||||
Traditional databases force you to choose between vector search, graph traversal, OR metadata filtering. Brainy combines all three intelligences into one magical API that automatically optimizes execution for maximum performance.
|
||||
|
||||
## Query Interface
|
||||
|
||||
### Unified Query Structure
|
||||
|
||||
`find()` accepts a single `FindParams` object (or a natural-language string). One
|
||||
object combines all three intelligences:
|
||||
|
||||
```typescript
|
||||
interface FindParams {
|
||||
// Vector intelligence — semantic similarity
|
||||
query?: string // Natural-language / semantic query (embedded, matched via HNSW + text index)
|
||||
vector?: number[] // Pre-computed embedding for direct vector search
|
||||
|
||||
// Metadata intelligence — structured field filters
|
||||
type?: NounType | NounType[] // Filter by entity type
|
||||
subtype?: string | string[] // Filter by per-product subtype
|
||||
where?: Record<string, any> // Field predicates with bare operators (gte, lt, in, contains, exists…)
|
||||
|
||||
// Graph intelligence — relationship traversal
|
||||
connected?: {
|
||||
to?: string // Reachable to this entity
|
||||
from?: string // Reachable from this entity
|
||||
via?: VerbType | VerbType[] // Relationship type(s) to traverse (alias: type)
|
||||
depth?: number // Max traversal depth (default: 1)
|
||||
direction?: 'in' | 'out' | 'both'
|
||||
}
|
||||
|
||||
// Proximity — nearest neighbours of a known entity
|
||||
near?: { id: string; threshold?: number }
|
||||
|
||||
// Control
|
||||
limit?: number // Max results (default: 10)
|
||||
offset?: number // Skip N results
|
||||
orderBy?: string // Field to sort by (e.g. 'createdAt')
|
||||
order?: 'asc' | 'desc' // Sort direction
|
||||
}
|
||||
```
|
||||
|
||||
### Example Queries
|
||||
|
||||
#### Natural Language Queries with find()
|
||||
```typescript
|
||||
// Brainy understands natural language and extracts intent
|
||||
const results = await brain.find("research papers about neural networks from 2023")
|
||||
// Automatically interprets: document type, topic, time range
|
||||
|
||||
// Complex temporal and numeric queries
|
||||
const reports = await brain.find("quarterly reports from Q3 2024 with revenue over 10M")
|
||||
// Automatically extracts: report type, date range, numeric filters
|
||||
|
||||
// Multi-condition natural language
|
||||
const articles = await brain.find("verified articles by John Smith about machine learning published this year")
|
||||
// Automatically identifies: author, topic, verification status, time range
|
||||
```
|
||||
|
||||
#### Simple Vector Search
|
||||
```typescript
|
||||
const results = await brain.find("machine learning concepts")
|
||||
```
|
||||
|
||||
#### Combined Intelligence Query
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
query: "neural networks",
|
||||
where: {
|
||||
category: "research",
|
||||
year: { gte: 2023 }
|
||||
},
|
||||
connected: {
|
||||
to: "deep-learning-team",
|
||||
depth: 2
|
||||
},
|
||||
limit: 20
|
||||
})
|
||||
```
|
||||
|
||||
## Query Optimization
|
||||
|
||||
### Automatic Plan Generation
|
||||
|
||||
The Triple Intelligence engine analyzes each query to create an optimal execution plan:
|
||||
|
||||
1. **Selectivity Analysis**: Identifies the most selective filters
|
||||
2. **Cost Estimation**: Estimates computational cost for each operation
|
||||
3. **Strategy Selection**: Chooses between parallel or progressive execution
|
||||
4. **Plan Caching**: Caches successful plans for similar queries
|
||||
|
||||
### Execution Strategies
|
||||
|
||||
#### Parallel Execution
|
||||
All three search types execute simultaneously:
|
||||
- **Best for**: Balanced queries with multiple signals
|
||||
- **Performance**: Maximum speed through parallelization
|
||||
- **Use case**: Complex queries needing all intelligence types
|
||||
|
||||
```typescript
|
||||
// Parallel execution for balanced query
|
||||
const results = await brain.find({
|
||||
query: "AI research", // ~1000 potential matches
|
||||
where: { kind: "paper" }, // ~500 potential matches
|
||||
connected: { to: "stanford" } // ~200 potential matches
|
||||
})
|
||||
// All three execute in parallel, results fused
|
||||
```
|
||||
|
||||
#### Progressive Filtering
|
||||
Operations chain for maximum efficiency:
|
||||
- **Best for**: Queries with highly selective filters
|
||||
- **Performance**: Reduces search space at each step
|
||||
- **Use case**: Large datasets with specific criteria
|
||||
|
||||
```typescript
|
||||
// Progressive execution for selective query
|
||||
const results = await brain.find({
|
||||
where: { userId: "user123" }, // Very selective (1-10 matches)
|
||||
query: "recent posts", // Applied to filtered set
|
||||
limit: 5
|
||||
})
|
||||
// Metadata filter first, then vector search on results
|
||||
```
|
||||
|
||||
## Fusion Ranking
|
||||
|
||||
### Score Combination
|
||||
|
||||
When multiple intelligence types return results, scores are intelligently combined:
|
||||
|
||||
```typescript
|
||||
fusionScore = (
|
||||
vectorScore * vectorWeight + // Semantic relevance (0.4)
|
||||
graphScore * graphWeight + // Relationship strength (0.3)
|
||||
fieldScore * fieldWeight // Exact match confidence (0.3)
|
||||
) / totalWeight
|
||||
```
|
||||
|
||||
### Adaptive Weights
|
||||
|
||||
Weights adjust based on query characteristics:
|
||||
- **Text-heavy query**: Higher vector weight
|
||||
- **Relationship query**: Higher graph weight
|
||||
- **Specific filters**: Higher field weight
|
||||
|
||||
## Natural Language Processing
|
||||
|
||||
### Pattern Recognition
|
||||
|
||||
Brainy includes 220+ embedded patterns for natural language understanding:
|
||||
|
||||
```typescript
|
||||
// Natural language automatically parsed
|
||||
const results = await brain.find(
|
||||
"show me recent AI papers from Stanford published this year"
|
||||
)
|
||||
// Automatically converts to:
|
||||
// {
|
||||
// query: "AI papers",
|
||||
// where: {
|
||||
// institution: "Stanford",
|
||||
// published: { gte: "2024-01-01" }
|
||||
// }
|
||||
// }
|
||||
```
|
||||
|
||||
### Intent Detection
|
||||
|
||||
The NLP processor identifies query intent:
|
||||
- **Informational**: "what is", "how does"
|
||||
- **Navigational**: "find", "show me"
|
||||
- **Transactional**: "create", "update"
|
||||
- **Analytical**: "compare", "analyze"
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Query Plan Caching
|
||||
|
||||
Successful execution plans are cached:
|
||||
```typescript
|
||||
// First call parses the natural-language query and builds an execution plan
|
||||
await brain.find("machine learning papers")
|
||||
|
||||
// A structurally similar query reuses that plan, skipping plan generation
|
||||
await brain.find("deep learning papers")
|
||||
```
|
||||
|
||||
### Self-Optimization
|
||||
|
||||
Brainy uses itself to optimize queries:
|
||||
- Query patterns stored in separate brain instance
|
||||
- Execution times tracked and analyzed
|
||||
- Plans automatically improved based on performance
|
||||
|
||||
### Index Utilization
|
||||
|
||||
Triple Intelligence leverages all available indexes:
|
||||
- **HNSW Index**: For vector similarity
|
||||
- **Metadata Index**: For metadata filtering
|
||||
- **Graph Index**: For relationship traversal
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Explain Mode
|
||||
|
||||
Diagnose how a query's `where` fields map to the index. Run `brain.explain()`
|
||||
first whenever `find()` returns surprising or empty results:
|
||||
|
||||
```typescript
|
||||
const plan = await brain.explain({
|
||||
query: "quantum computing",
|
||||
where: { category: "research" }
|
||||
})
|
||||
|
||||
console.log(plan.fieldPlan)
|
||||
// [
|
||||
// { field: 'category', path: 'column-store', notes: '...' }
|
||||
// ]
|
||||
|
||||
console.log(plan.warnings)
|
||||
// e.g. ['Field "category" has no index entries. find() will return [] silently...']
|
||||
```
|
||||
|
||||
### Result Ordering
|
||||
|
||||
Sort results by any stored field with `orderBy` / `order`:
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
query: "news articles",
|
||||
where: { verified: true },
|
||||
orderBy: 'createdAt', // Newest first
|
||||
order: 'desc'
|
||||
})
|
||||
```
|
||||
|
||||
### Similarity Threshold
|
||||
|
||||
Find the nearest neighbours of a known entity and keep only close matches with
|
||||
`near`:
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
near: { id: anchorId, threshold: 0.9 }, // Only results >= 0.9 similarity
|
||||
limit: 10
|
||||
})
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Query Design
|
||||
|
||||
1. **Start specific**: Use selective filters when possible
|
||||
2. **Combine intelligently**: Don't force all three types if not needed
|
||||
3. **Use limits**: Always specify reasonable result limits
|
||||
4. **Cache results**: For repeated queries, cache at application level
|
||||
|
||||
### Performance Tips
|
||||
|
||||
1. **Index first**: Ensure fields used in `where` clauses are indexed
|
||||
2. **Batch operations**: Use batch methods for bulk queries
|
||||
3. **Monitor plans**: Use explain mode to understand performance
|
||||
4. **Optimize patterns**: Train custom patterns for your domain
|
||||
|
||||
### Common Patterns
|
||||
|
||||
#### Semantic Search with Filtering
|
||||
```typescript
|
||||
// Find similar content with constraints
|
||||
const results = await brain.find({
|
||||
query: searchText,
|
||||
where: {
|
||||
status: 'published',
|
||||
language: 'en'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
#### Related Items Discovery
|
||||
```typescript
|
||||
// Find items related to a specific item
|
||||
const results = await brain.find({
|
||||
connected: {
|
||||
to: itemId,
|
||||
depth: 2,
|
||||
via: VerbType.RelatedTo
|
||||
},
|
||||
limit: 20
|
||||
})
|
||||
```
|
||||
|
||||
#### Time-based Queries
|
||||
```typescript
|
||||
// Recent items matching criteria
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
timestamp: { gte: Date.now() - 86400000 }
|
||||
},
|
||||
query: "trending topics",
|
||||
orderBy: 'timestamp',
|
||||
order: 'desc'
|
||||
})
|
||||
```
|
||||
|
||||
## Natural Language Processing
|
||||
|
||||
The `find()` method includes advanced NLP capabilities powered by 220+ embedded patterns that understand natural language queries.
|
||||
|
||||
### Supported Query Types
|
||||
|
||||
```typescript
|
||||
// Temporal queries
|
||||
await brain.find("documents from last week")
|
||||
await brain.find("reports created yesterday")
|
||||
await brain.find("articles published in Q3 2024")
|
||||
await brain.find("data from January to March")
|
||||
|
||||
// Numeric filters
|
||||
await brain.find("products with price under $100")
|
||||
await brain.find("articles with more than 1000 views")
|
||||
await brain.find("reports showing revenue over 10M")
|
||||
|
||||
// Combined conditions
|
||||
await brain.find("verified research papers about AI from 2024 with high citations")
|
||||
await brain.find("recent customer reviews with rating above 4 stars")
|
||||
await brain.find("blog posts by John Smith about machine learning published this month")
|
||||
|
||||
// Relationship queries
|
||||
await brain.find("documents related to project X")
|
||||
await brain.find("people who work at TechCorp")
|
||||
await brain.find("products similar to iPhone")
|
||||
```
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Intent Detection**: Identifies what the user is looking for
|
||||
2. **Entity Extraction**: Extracts names, dates, numbers, categories
|
||||
3. **Temporal Parsing**: Converts "last week", "Q3 2024" to date ranges
|
||||
4. **Filter Generation**: Creates appropriate where clauses
|
||||
5. **Query Fusion**: Combines NLP understanding with vector search
|
||||
|
||||
### Pattern Coverage
|
||||
|
||||
Brainy includes 220+ pre-computed patterns covering:
|
||||
- **Temporal**: 40+ patterns for dates and time ranges
|
||||
- **Numeric**: 30+ patterns for comparisons and ranges
|
||||
- **Relationships**: 25+ patterns for connections
|
||||
- **Actions**: 35+ patterns for verbs and intents
|
||||
- **Entities**: 40+ patterns for people, places, things
|
||||
- **Domain-specific**: 50+ patterns for tech, business, social
|
||||
|
||||
## API Reference
|
||||
|
||||
See the [Triple Intelligence API](../api/triple-intelligence.md) for complete method documentation.
|
||||
157
docs/architecture/zero-config.md
Normal file
157
docs/architecture/zero-config.md
Normal file
|
|
@ -0,0 +1,157 @@
|
|||
---
|
||||
title: Zero Configuration
|
||||
slug: concepts/zero-config
|
||||
public: false
|
||||
category: concepts
|
||||
template: concept
|
||||
order: 3
|
||||
description: Brainy auto-detects storage, initializes embeddings, and builds indexes — no configuration required. Works in Node.js and Bun (server-only since 8.0).
|
||||
next:
|
||||
- getting-started/installation
|
||||
- guides/storage-adapters
|
||||
---
|
||||
|
||||
# Zero Configuration & Auto-Adaptation
|
||||
|
||||
> **"Zero config by default, fully tunable when you need it."** Construct a
|
||||
> `Brainy()` with no options and it picks sensible, environment-aware defaults.
|
||||
> Every default below is overridable through the constructor — see the
|
||||
> [API Reference](../api/README.md#configuration).
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy 8.0 is server-only (Node.js 22+ / Bun). With no configuration it:
|
||||
|
||||
- selects a storage adapter from the runtime,
|
||||
- initializes the embedding model (all-MiniLM-L6-v2, 384 dimensions),
|
||||
- builds and maintains the metadata, graph, and vector indexes,
|
||||
- sizes its caches and write buffers to the detected memory budget,
|
||||
- chooses a persistence mode that matches the storage backend, and
|
||||
- quiets its own logging when it detects a production environment.
|
||||
|
||||
There is no public config-generation function — adaptation happens inside the
|
||||
constructor and `init()`.
|
||||
|
||||
## Instant Start
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// That's it. No config needed.
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
await brain.add({ data: 'First entity', type: 'concept' })
|
||||
const results = await brain.find('first')
|
||||
```
|
||||
|
||||
## What Auto-Adaptation Covers
|
||||
|
||||
### 1. Storage auto-detection
|
||||
|
||||
With no `storage` option, Brainy uses `type: 'auto'`:
|
||||
|
||||
- **Filesystem** when running on a runtime with a writable Node filesystem and a
|
||||
resolvable root directory. This is the default for typical Node/Bun servers and
|
||||
persists across restarts.
|
||||
- **In-memory** otherwise (no filesystem access, or an explicit memory request).
|
||||
Fast, zero I/O, discarded on process exit — ideal for tests and ephemeral
|
||||
caches.
|
||||
|
||||
8.0 ships exactly two storage adapters — `memory` and `filesystem` — plus the
|
||||
`auto` selector that resolves to one of them. See
|
||||
[Storage Adapters](../concepts/storage-adapters.md) for the full contract.
|
||||
|
||||
```typescript
|
||||
// Explicit override when you want a specific root
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './brainy-data' }
|
||||
})
|
||||
```
|
||||
|
||||
### 2. HNSW quality from the `recall` preset
|
||||
|
||||
Vector-index quality comes from a single preset rather than hand-tuned graph
|
||||
parameters. `config.vector.recall` accepts `'fast'`, `'balanced'`, or
|
||||
`'accurate'` and defaults to `'balanced'`. The preset maps internally to the
|
||||
HNSW construction and search parameters (`M` / `efConstruction` / `efSearch`),
|
||||
so you trade recall against latency with one knob instead of three.
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
vector: { recall: 'fast' } // favor latency over recall
|
||||
})
|
||||
```
|
||||
|
||||
The default JS index is `JsHnswVectorIndex`. An optional native acceleration
|
||||
provider (the `@soulcraft/cor` package) can replace it with a
|
||||
higher-performing implementation; the public knobs stay the same. Quantization
|
||||
and other index-internal acceleration are the native provider's concern, not a
|
||||
Brainy configuration option.
|
||||
|
||||
### 3. Persistence mode follows the backend
|
||||
|
||||
`config.vector.persistMode` accepts `'immediate'` or `'deferred'`. Left unset,
|
||||
Brainy chooses for you:
|
||||
|
||||
- **Immediate** on filesystem storage, so the index file stays in lock-step with
|
||||
the data and survives a crash.
|
||||
- **Deferred** on in-memory storage, where there is nothing durable to sync to,
|
||||
so writes are batched for throughput.
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
vector: { persistMode: 'deferred' } // batch persistence for write-heavy loads
|
||||
})
|
||||
```
|
||||
|
||||
### 4. Memory-aware cache and buffer sizing
|
||||
|
||||
Brainy reads the container's memory budget — `CLOUD_RUN_MEMORY`, `MEMORY_LIMIT`,
|
||||
or the cgroup memory limit when running in a container — and sizes its read
|
||||
caches and write buffers to fit. On a small instance it stays conservative; on a
|
||||
large one it uses more of the available headroom. Query-result limits are capped
|
||||
against the same budget (roughly 25 KB per result) to keep a single oversized
|
||||
query from exhausting memory.
|
||||
|
||||
You can pin the cache explicitly:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
cache: { maxSize: 10000, ttl: 3_600_000 }
|
||||
})
|
||||
```
|
||||
|
||||
### 5. Logging quiets in production
|
||||
|
||||
Brainy detects production-style environments (for example `NODE_ENV` set to a
|
||||
non-development value) and reduces its own log verbosity automatically. This is
|
||||
logging-only behavior — it does not change indexing, storage, or query results.
|
||||
|
||||
## Configuration Override
|
||||
|
||||
Zero-config is the default, not a ceiling. Every adaptive decision above has an
|
||||
explicit constructor option:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'accurate',
|
||||
persistMode: 'immediate'
|
||||
},
|
||||
cache: { maxSize: 50000, ttl: 600_000 }
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
See the [API Reference](../api/README.md#configuration) for the complete option
|
||||
list.
|
||||
|
||||
## See Also
|
||||
|
||||
- [Architecture Overview](./overview.md)
|
||||
- [Storage Adapters](../concepts/storage-adapters.md)
|
||||
- [Scaling Guide](../SCALING.md)
|
||||
- [API Reference](../api/README.md)
|
||||
|
|
@ -1,122 +0,0 @@
|
|||
# Documentation Organization
|
||||
|
||||
This document explains the new documentation structure and workflow for managing markdown files in the Brainy project.
|
||||
|
||||
## Overview
|
||||
|
||||
The documentation has been reorganized from 29 scattered markdown files at the root level to a clean, organized structure with only essential files at the root and categorized documentation in the `docs/` directory.
|
||||
|
||||
## New Structure
|
||||
|
||||
### Root Level Files (GitHub/NPM Standards)
|
||||
- `README.md` - Main project documentation
|
||||
- `CONTRIBUTING.md` - Contribution guidelines
|
||||
- `CODE_OF_CONDUCT.md` - Community standards
|
||||
- `CHANGELOG.md` - Version history and release notes
|
||||
|
||||
### Documentation Directory Structure
|
||||
```
|
||||
docs/
|
||||
├── technical/ # Technical documentation and analysis
|
||||
│ ├── CONCURRENCY_ANALYSIS.md
|
||||
│ ├── STORAGE_CONCURRENCY_ANALYSIS.md
|
||||
│ ├── THREADING.md
|
||||
│ ├── STATISTICS.md
|
||||
│ ├── TESTING.md
|
||||
│ ├── VITEST_IMPROVEMENTS.md
|
||||
│ ├── REALTIME_UPDATES.md
|
||||
│ ├── METADATA_HANDLING.md
|
||||
│ ├── VECTOR_DIMENSION_STANDARDIZATION.md
|
||||
│ ├── USE_MODEL_LOADING_EXPLANATION.md
|
||||
│ ├── STORAGE_TESTING.md
|
||||
│ ├── TECHNICAL_GUIDES.md
|
||||
│ ├── ENVIRONMENT_TESTING.md
|
||||
│ └── SCALING_STRATEGY.md
|
||||
├── development/ # Development and contributor documentation
|
||||
│ ├── DEVELOPERS.md
|
||||
│ ├── DOCUMENTATION_STANDARDS.md
|
||||
│ ├── MARKDOWN_CONVENTIONS.md
|
||||
│ ├── EXPECTED_TEST_MESSAGES.md
|
||||
│ └── PRETTY_TEST_REPORTER.md
|
||||
└── guides/ # User guides and migration documentation
|
||||
├── cache-configuration.md
|
||||
├── hnsw-field-search.md
|
||||
├── json-document-search.md
|
||||
├── model-management.md
|
||||
├── optional-model-bundling.md
|
||||
├── production-migration-guide.md
|
||||
└── service-identification.md
|
||||
```
|
||||
|
||||
|
||||
## CHANGELOG.md Management
|
||||
|
||||
### Automated Workflow
|
||||
|
||||
The project now uses automated changelog management:
|
||||
|
||||
1. **Adding Changes**: Add entries to the `[Unreleased]` section in `CHANGELOG.md`
|
||||
2. **Version Bumping**: Use npm scripts that automatically update the changelog:
|
||||
- `npm run version:patch` - Patch version bump + changelog update
|
||||
- `npm run version:minor` - Minor version bump + changelog update
|
||||
- `npm run version:major` - Major version bump + changelog update
|
||||
|
||||
3. **Manual Updates**: Use `npm run changelog:update` to manually update the changelog
|
||||
|
||||
### Changelog Format
|
||||
|
||||
The changelog follows the [Keep a Changelog](https://keepachangelog.com/) standard:
|
||||
|
||||
- **Added** - New features
|
||||
- **Changed** - Changes in existing functionality
|
||||
- **Deprecated** - Soon-to-be removed features
|
||||
- **Removed** - Now removed features
|
||||
- **Fixed** - Bug fixes
|
||||
- **Security** - Vulnerability fixes
|
||||
|
||||
### GitHub Integration
|
||||
|
||||
The CHANGELOG.md is automatically used for:
|
||||
- GitHub releases (via `scripts/create-github-release.js`)
|
||||
- NPM package release notes
|
||||
- Version history tracking
|
||||
|
||||
## Benefits of New Structure
|
||||
|
||||
1. **Clean Root Directory**: Only 4 essential markdown files at root level
|
||||
2. **Better Organization**: Logical categorization of documentation in docs/ subdirectories
|
||||
3. **GitHub Compliance**: Follows GitHub and NPM best practices
|
||||
4. **Automated Maintenance**: Changelog updates are automated
|
||||
5. **Easy Navigation**: Clear directory structure for different doc types
|
||||
6. **Reduced Clutter**: Temporary and outdated files have been removed
|
||||
|
||||
## Workflow for Contributors
|
||||
|
||||
### Adding Documentation
|
||||
1. **Technical docs** → `docs/technical/`
|
||||
2. **Development docs** → `docs/development/`
|
||||
3. **User guides** → `docs/guides/`
|
||||
4. **Temporary files** → Should be avoided; use issues or PRs for temporary documentation
|
||||
|
||||
### Making Changes
|
||||
1. Add changes to `[Unreleased]` section in `CHANGELOG.md`
|
||||
2. Use appropriate category (Added, Changed, Fixed, etc.)
|
||||
3. When ready to release, use `npm run version:patch/minor/major`
|
||||
4. The changelog will be automatically updated with version and date
|
||||
|
||||
### Release Process
|
||||
1. Ensure `[Unreleased]` section has all changes
|
||||
2. Run `npm run version:patch/minor/major`
|
||||
3. Run `npm run deploy` to publish and create GitHub release
|
||||
4. GitHub release will use CHANGELOG.md content
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- All technical documentation organized in `docs/technical/`
|
||||
- Development documentation organized in `docs/development/`
|
||||
- User guides organized in `docs/guides/`
|
||||
- Temporary summary files and archived content have been cleaned up
|
||||
- Links in existing documentation may need updates
|
||||
- New automation ensures changelog stays current
|
||||
|
||||
This reorganization provides a sustainable, scalable approach to documentation management that follows industry best practices and integrates seamlessly with GitHub and NPM workflows.
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue