soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

docs(8.0): Phase F — deep clean across 21 docs

Browse source 
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.

Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
  cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
  HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
  sections replaced with single-node vector tuning + filesystem framing.

Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md

Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md

MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
David Snelling 2026-06-09 16:13:35 -07:00
parent 2626ab8d62
commit adda1570f3
22 changed files with 570 additions and 2658 deletions
Download patch file Download diff file Expand all files Collapse all files

530
docs/SCALING.md
View file

@ -1,502 +1,202 @@
# 🚀 Brainy Scaling Guide - Enterprise for Everyone
# Brainy Scaling Guide
> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required.
> **One Line Summary**: Single-node by design — Brainy scales by getting the most out of one machine plus operator-layer backup.
## 📖 Table of Contents
## Table of Contents
- [Quick Start](#quick-start)
- [How It Works](#how-it-works)
- [How Brainy Scales](#how-brainy-scales)
- [Storage Configurations](#storage-configurations)
- [Scaling Patterns](#scaling-patterns)
- [Real World Examples](#real-world-examples)
## Quick Start
### Single Node (Default)
### In-Memory
```typescript
import Brainy from '@soulcraft/brainy'
const brain = new Brainy() // That's it!
const brain = new Brainy({ storage: { type: 'memory' } })
```
### Multi-Node (Auto-Discovery)
### On-Disk (Default for Node)
```typescript
// Node 1
const brain = new Brainy() // Starts as primary
// Node 2 (different server)
const brain = new Brainy() // Auto-discovers Node 1, becomes replica!
```
**That's literally all you need!** Brainy handles everything else automatically.
## How It Works
### 🎯 The Magic: Zero Configuration
Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration:
1. **First node starts** → Becomes primary automatically
2. **Second node starts** → Discovers first node via UDP broadcast
3. **Nodes negotiate** → Elect leader, distribute shards
4. **Data flows** → Automatic replication and routing
5. **Node fails** → Automatic failover in <1 second
### 🔄 Automatic Node Discovery
```typescript
// Three ways Brainy finds other nodes (auto-selected):
// 1. LOCAL NETWORK (Default)
// Uses UDP broadcast on port 7946
// Perfect for: On-premise, same VPC
// 2. CLOUD NATIVE (Auto-detected)
// Kubernetes: Uses k8s DNS service discovery
// AWS: Uses EC2 tags or Route53
// Azure: Uses Azure DNS
// 3. EXPLICIT (When needed)
const brain = new Brainy({
peers: ['node1.example.com', 'node2.example.com']
storage: { type: 'filesystem', rootDirectory: './brainy-data' }
})
```
### 📊 Data Distribution
## How Brainy Scales
When you add data, Brainy automatically:
Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery, no S3 coordination. Scaling means:
```typescript
brain.add({ name: "John" }, 'person')
- **Up**: give the process more RAM, CPU, and IOPS
- **Out**: stand up multiple independent Brainy instances behind your own service layer
- **Cold storage**: snapshot the on-disk artifact off-site so you can rehydrate elsewhere
// Behind the scenes:
// 1. Hash ID to determine shard (consistent hashing)
// 2. Find nodes responsible for this shard
// 3. Write to primary shard owner
// 4. Replicate to N backup nodes (default: 2)
// 5. Confirm write when majority acknowledge
```
The three knobs that matter most:
1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
2. **`config.vector.quantization`** — `{ bits: 4 | 8 }` for memory savings on the open-core JS vector index
3. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
The native vector provider (via the optional `@soulcraft/cortex` package) extends this with a higher-performing index when installed.
## Storage Configurations
### 🗂️ Storage Adapter Patterns
Brainy intelligently adapts to your storage setup:
#### Pattern 1: Separate Storage Per Node (Recommended)
### Filesystem (Recommended for Production)
```typescript
// Node 1 - Own filesystem
const brain1 = new Brainy({
storage: '/data/node1' // or auto: './brainy-data'
})
// Node 2 - Own filesystem
const brain2 = new Brainy({
storage: '/data/node2' // or auto: './brainy-data'
})
// ✅ BENEFITS:
// - No conflicts between nodes
// - Fast local reads
// - True horizontal scaling
// - Survives network partitions
```
#### Pattern 2: Separate S3 Buckets Per Node
```typescript
// Node 1 - Own S3 bucket
const brain1 = new Brainy({
storage: 's3://brainy-node-1' // Auto-uses AWS credentials
})
// Node 2 - Own S3 bucket
const brain2 = new Brainy({
storage: 's3://brainy-node-2'
})
// ✅ BENEFITS:
// - Infinite storage capacity
// - Geographic distribution
// - No local disk needed
// - Built-in durability
```
#### Pattern 3: Shared S3 Bucket (Coordinated)
```typescript
// All nodes - Shared bucket with coordination
const brain = new Brainy({
storage: 's3://shared-brainy-data',
// Brainy automatically adds node-specific prefixes!
})
// What happens automatically:
// - Node 1 writes to: s3://shared-brainy-data/node-1/
// - Node 2 writes to: s3://shared-brainy-data/node-2/
// - Metadata in: s3://shared-brainy-data/_cluster/
// - Coordination via S3 conditional writes
// ✅ BENEFITS:
// - Single bucket to manage
// - Easy backup/restore
// - Cost effective
// - Automatic namespace isolation
```
#### Pattern 4: Mixed Storage (Hybrid)
```typescript
// Hot data on local SSD, cold data in S3
const brain = new Brainy({
storage: {
hot: '/fast-ssd/brainy', // Recent/frequent data
cold: 's3://brainy-archive' // Older data
type: 'filesystem',
rootDirectory: '/var/lib/brainy'
}
// Brainy automatically promotes/demotes data!
})
```
- Stores everything in a sharded JSON tree under `rootDirectory`
- Atomic writes via rename
- Survives process restarts
- Snapshot it off-site with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` from your scheduler
### 🌍 Cloud Provider Auto-Detection
### Memory
```typescript
const brain = new Brainy({ storage: { type: 'memory' } })
```
- Zero I/O, fastest possible
- No persistence — process exit discards everything
- Use for tests and ephemeral caches
### Auto
```typescript
const brain = new Brainy({
storage: 'cloud://brainy-data' // Auto-detects provider!
storage: { type: 'auto', rootDirectory: './data' }
})
// Automatically uses:
// - AWS: S3 + DynamoDB for metadata
// - Google Cloud: GCS + Firestore
// - Azure: Blob Storage + Cosmos DB
// - Cloudflare: R2 + D1
// - Vercel: Blob + KV
```
### 📝 Storage Coordination Rules
When multiple nodes share storage, Brainy automatically:
1. **Namespace Isolation**: Each node gets unique prefix
2. **Lock-Free Writes**: Uses atomic operations
3. **Consistent Metadata**: Coordinated via consensus
4. **Conflict Resolution**: Version vectors for conflicts
5. **Garbage Collection**: Automatic cleanup of old data
- Picks `filesystem` when running on Node with a writable `rootDirectory`
- Falls back to `memory` otherwise
## Scaling Patterns
### 📈 Progressive Scaling Journey
#### Stage 1: Prototype (1 node, memory)
### Stage 1: Prototype (Memory)
```typescript
const brain = new Brainy() // Memory storage, single node
// Perfect for: Development, testing, <1000 items
const brain = new Brainy({ storage: { type: 'memory' } })
// Development, tests, <100K items
```
#### Stage 2: Production (1 node, disk)
### Stage 2: Production (Filesystem)
```typescript
const brain = new Brainy({
storage: './data' // Persistent storage
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' }
})
// Perfect for: Small apps, <100K items
// Most production workloads up to ~10M entities on a single host
```
#### Stage 3: High Availability (2-3 nodes)
### Stage 3: Higher Throughput (Tune the Vector Index)
```typescript
// Just start same code on multiple servers!
const brain = new Brainy({
storage: './data' // Each node's own storage
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
vector: {
recall: 'fast', // Trade recall for latency
quantization: { bits: 8 }, // SQ8 quantization
persistMode: 'deferred' // Batch persistence
}
})
// Automatic: Leader election, replication, failover
// Perfect for: Critical apps, <1M items
```
#### Stage 4: Scale Out (N nodes)
```typescript
// Same code, more servers!
const brain = new Brainy({
storage: 's3://brainy-{{nodeId}}' // Template auto-filled
})
// Automatic: Sharding, load balancing, geo-distribution
// Perfect for: Large apps, unlimited items
```
### 🎯 Common Scaling Scenarios
#### Scenario: Read-Heavy Application
```typescript
// Brainy auto-detects read-heavy pattern and:
// 1. Increases cache size
// 2. Creates more read replicas
// 3. Routes reads to nearest node
// 4. Caches popular items on all nodes
const brain = new Brainy() // No config needed!
```
#### Scenario: Multi-Tenant SaaS
```typescript
// Brainy auto-detects tenant patterns and:
// 1. Shards by tenant ID
// 2. Isolates tenant data
// 3. Routes by tenant
// 4. Separate rate limits per tenant
const brain = new Brainy() // Detects from your queries!
```
#### Scenario: Geographic Distribution
```typescript
// Deploy nodes in different regions
// Brainy automatically:
// 1. Detects node locations (via latency)
// 2. Replicates data geographically
// 3. Routes to nearest node
// 4. Handles region failures
// US-East
const brain = new Brainy({ region: 'us-east' }) // Optional hint
// EU-West (auto-discovers US-East)
const brain = new Brainy({ region: 'eu-west' })
```
### Stage 4: Multi-Instance (Operator-Layer)
Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `rootDirectory`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes.
## Real World Examples
### Example 1: Blog Platform
### Example 1: Single-Node App With Backup
```typescript
// Day 1: Single server
const brain = new Brainy({
storage: './blog-data'
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' }
})
// Month 6: Add redundancy (on second server)
const brain = new Brainy({
storage: './blog-data' // Different machine!
})
// Automatically syncs with first server
// Year 2: Global scale
// US Server
const brain = new Brainy({
storage: 's3://blog-us/data'
})
// EU Server
const brain = new Brainy({
storage: 's3://blog-eu/data'
})
// Asia Server
const brain = new Brainy({
storage: 's3://blog-asia/data'
})
// All automatically coordinate!
```
Schedule (cron / systemd timer):
```bash
*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
```
### Example 2: E-Commerce Site
### Example 2: Tests
```typescript
// Development
const brain = new Brainy() // Memory storage
// Staging (Kubernetes)
const brain = new Brainy({
storage: process.env.STORAGE_PATH // Uses PVC
})
// Auto-discovers other pods via K8s DNS
// Production (AWS)
const brain = new Brainy({
storage: 's3://shop-data',
cache: 'elasticache://shop-cache' // Optional
})
// Auto-scales with ECS/EKS
const brain = new Brainy({ storage: { type: 'memory' } })
// Fast, no cleanup needed between runs
```
### Example 3: Analytics Platform
### Example 3: Multi-Tenant Service
Spin up one Brainy instance per tenant, each in its own directory:
```typescript
// Ingestion nodes (write-optimized)
const brain = new Brainy({
role: 'writer', // Hint for optimization
storage: '/fast-nvme/ingest'
})
// Query nodes (read-optimized)
const brain = new Brainy({
role: 'reader', // More cache, indexes
storage: 's3://analytics-archive'
})
// Automatically coordinates between writers and readers!
```
## 🔧 Storage Adapter Specifics
### Local Filesystem
```typescript
{
storage: './data' // or absolute: '/var/lib/brainy'
// Each node MUST have separate directory
// Can be network mounted (NFS, EFS)
function brainForTenant(tenantId: string) {
return new Brainy({
storage: {
type: 'filesystem',
rootDirectory: `/var/lib/brainy/${tenantId}`
}
})
}
```
Your service layer handles routing and isolation; Brainy stays simple.
### AWS S3
### Example 4: Higher Recall at Scale
```typescript
{
storage: 's3://bucket-name/prefix'
// Uses AWS SDK credentials (env, IAM role, etc)
// Supports S3-compatible (MinIO, Ceph)
}
```
### Cloudflare R2
```typescript
{
storage: 'r2://bucket-name'
// Uses Wrangler or API tokens
// Zero egress fees!
}
```
### Google Cloud Storage
```typescript
{
storage: 'gs://bucket-name'
// Uses Application Default Credentials
}
```
### Azure Blob Storage
```typescript
{
storage: 'azure://container-name'
// Uses DefaultAzureCredential
}
```
### Mixed/Tiered
```typescript
{
storage: {
hot: './local-cache', // Fast SSD
warm: 's3://regular-data', // Standard storage
cold: 's3://glacier-archive' // Cheap archive
const brain = new Brainy({
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
vector: {
recall: 'accurate',
quantization: { bits: 8 }
}
// Automatic tiering based on access patterns
}
```
## 🎭 Advanced Patterns
### Pattern: Blue-Green Deployment
```typescript
// Blue cluster (current)
const brain = new Brainy({
cluster: 'blue',
storage: 's3://prod-blue'
})
// Green cluster (new version)
const brain = new Brainy({
cluster: 'green',
storage: 's3://prod-green',
syncFrom: 'blue' // Real-time sync during migration
})
```
### Pattern: Federation
```typescript
// Region 1 Cluster
const brain1 = new Brainy({
federation: 'global',
region: 'us-east',
storage: 's3://us-east-data'
})
## Tuning Knobs Summary
// Region 2 Cluster
const brain2 = new Brainy({
federation: 'global',
region: 'eu-west',
storage: 's3://eu-west-data'
})
// Clusters coordinate for global queries!
```
| Setting | Values | When to change |
|---------|--------|----------------|
| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
| `vector.quantization.bits` | `4` / `8` | Smaller index, lower memory |
| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
| `storage.cache.maxSize` | integer | Hot-path read cache size |
| `storage.cache.ttl` | ms | Cache freshness |
### Pattern: Edge Computing
```typescript
// Edge nodes (in CDN POPs)
const brain = new Brainy({
mode: 'edge',
storage: 'memory', // RAM only
upstream: 'https://main-cluster.example.com'
})
// Caches frequently accessed data at edge
```
## 📊 Monitoring & Observability
Brainy automatically exposes metrics:
## Monitoring & Observability
```typescript
const metrics = brain.getMetrics()
const stats = await brain.stats()
// {
// nodes: { total: 5, healthy: 5 },
// shards: { total: 20, local: 4 },
// replication: { factor: 2, lag: 45 },
// operations: { reads: 10000, writes: 1000 },
// storage: { used: '45GB', available: '955GB' }
// nounCount: 50000,
// verbCount: 80000,
// vectorIndex: { ... },
// storage: { used: '45GB' }
// }
```
## 🚨 Troubleshooting
## Troubleshooting
### Issue: Nodes don't discover each other
```typescript
// Solution 1: Check network allows UDP 7946
// Solution 2: Use explicit peers
const brain = new Brainy({
peers: ['10.0.0.1:7946', '10.0.0.2:7946']
})
```
### Issue: Slow queries
1. Switch to `vector.recall: 'fast'`
2. Enable SQ8 quantization
3. Increase the read cache (`storage.cache.maxSize`)
4. Consider the optional native vector provider via `@soulcraft/cortex`
### Issue: Storage conflicts
```typescript
// Ensure each node has unique storage path
// ❌ WRONG: All nodes use './data'
// ✅ RIGHT: Node1: './data1', Node2: './data2'
// ✅ RIGHT: Use {{nodeId}} template
```
### Issue: Memory pressure
1. Enable `vector.quantization: { bits: 4 }` or `{ bits: 8 }`
2. Reduce `storage.cache.maxSize`
3. Move to `vector.persistMode: 'deferred'` to batch writes
### Issue: Slow performance
```typescript
// Brainy auto-tunes, but you can hint:
const brain = new Brainy({
profile: 'read-heavy' // or 'write-heavy', 'balanced'
})
```
### Issue: Slow startup after a crash
1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
2. Verify backup integrity periodically
## 🎯 Best Practices
## Best Practices
1. **Let Brainy Auto-Configure**: Don't over-configure
2. **Separate Storage Per Node**: Avoids conflicts
3. **Use S3 for Large Scale**: Infinite capacity
4. **Start Simple**: Single node → Scale when needed
5. **Monitor Metrics**: Watch for bottlenecks
6. **Trust Auto-Scaling**: It learns your patterns
1. **One process = one `rootDirectory`** — never share a directory between processes
2. **Snapshot from your scheduler** — Brainy doesn't ship cloud SDKs; use `rclone` / `aws s3 sync` / `gsutil`
3. **Profile before tuning** — `recall: 'balanced'` is right for most workloads
4. **Install the native vector provider only when measured profiling shows it pays off**
## 🚀 Summary
## Summary
- **Zero Config**: Just `new Brainy()` at any scale
- **Auto-Discovery**: Nodes find each other
- **Smart Storage**: Adapts to any backend
- **Progressive Scaling**: 1 → 100 nodes seamlessly
- **Self-Tuning**: Learns and optimizes
- **No DevOps**: It just works!
**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!**
---
*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)*
- Brainy 8.0 is a **library**, not a cluster
- Storage adapters: `filesystem`, `memory`, `auto`
- Vector tuning: `recall`, `quantization`, `persistMode`
- Backup is an operator-layer concern — snapshot `rootDirectory`