brainy/docs/SCALING.md

# 🚀 Brainy Scaling Guide - Enterprise for Everyone

> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required.

## 📖 Table of Contents
- [Quick Start](#quick-start)
- [How It Works](#how-it-works)
- [Storage Configurations](#storage-configurations)
- [Scaling Patterns](#scaling-patterns)
- [Real World Examples](#real-world-examples)

## Quick Start

### Single Node (Default)
```typescript
import Brainy from '@soulcraft/brainy'
const brain = new Brainy()  // That's it!
```

### Multi-Node (Auto-Discovery)
```typescript
// Node 1
const brain = new Brainy()  // Starts as primary

// Node 2 (different server)
const brain = new Brainy()  // Auto-discovers Node 1, becomes replica!
```

**That's literally all you need!** Brainy handles everything else automatically.

## How It Works

### 🎯 The Magic: Zero Configuration

Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration:

1. **First node starts** → Becomes primary automatically
2. **Second node starts** → Discovers first node via UDP broadcast
3. **Nodes negotiate** → Elect leader, distribute shards
4. **Data flows** → Automatic replication and routing
5. **Node fails** → Automatic failover in <1 second

### 🔄 Automatic Node Discovery

```typescript
// Three ways Brainy finds other nodes (auto-selected):

// 1. LOCAL NETWORK (Default)
// Uses UDP broadcast on port 7946
// Perfect for: On-premise, same VPC

// 2. CLOUD NATIVE (Auto-detected)
// Kubernetes: Uses k8s DNS service discovery
// AWS: Uses EC2 tags or Route53
// Azure: Uses Azure DNS

// 3. EXPLICIT (When needed)
const brain = new Brainy({
  peers: ['node1.example.com', 'node2.example.com']
})
```

### 📊 Data Distribution

When you add data, Brainy automatically:

```typescript
brain.add({ name: "John" }, 'person')

// Behind the scenes:
// 1. Hash ID to determine shard (consistent hashing)
// 2. Find nodes responsible for this shard
// 3. Write to primary shard owner
// 4. Replicate to N backup nodes (default: 2)
// 5. Confirm write when majority acknowledge
```

## Storage Configurations

### 🗂️ Storage Adapter Patterns

Brainy intelligently adapts to your storage setup:

#### Pattern 1: Separate Storage Per Node (Recommended)

```typescript
// Node 1 - Own filesystem
const brain1 = new Brainy({
  storage: '/data/node1'  // or auto: './brainy-data'
})

// Node 2 - Own filesystem  
const brain2 = new Brainy({
  storage: '/data/node2'  // or auto: './brainy-data'
})

// ✅ BENEFITS:
// - No conflicts between nodes
// - Fast local reads
// - True horizontal scaling
// - Survives network partitions
```

#### Pattern 2: Separate S3 Buckets Per Node

```typescript
// Node 1 - Own S3 bucket
const brain1 = new Brainy({
  storage: 's3://brainy-node-1'  // Auto-uses AWS credentials
})

// Node 2 - Own S3 bucket
const brain2 = new Brainy({
  storage: 's3://brainy-node-2'
})

// ✅ BENEFITS:
// - Infinite storage capacity
// - Geographic distribution
// - No local disk needed
// - Built-in durability
```

#### Pattern 3: Shared S3 Bucket (Coordinated)

```typescript
// All nodes - Shared bucket with coordination
const brain = new Brainy({
  storage: 's3://shared-brainy-data',
  // Brainy automatically adds node-specific prefixes!
})

// What happens automatically:
// - Node 1 writes to: s3://shared-brainy-data/node-1/
// - Node 2 writes to: s3://shared-brainy-data/node-2/
// - Metadata in: s3://shared-brainy-data/_cluster/
// - Coordination via S3 conditional writes

// ✅ BENEFITS:
// - Single bucket to manage
// - Easy backup/restore
// - Cost effective
// - Automatic namespace isolation
```

#### Pattern 4: Mixed Storage (Hybrid)

```typescript
// Hot data on local SSD, cold data in S3
const brain = new Brainy({
  storage: {
    hot: '/fast-ssd/brainy',     // Recent/frequent data
    cold: 's3://brainy-archive'  // Older data
  }
  // Brainy automatically promotes/demotes data!
})
```

### 🌍 Cloud Provider Auto-Detection

```typescript
const brain = new Brainy({
  storage: 'cloud://brainy-data'  // Auto-detects provider!
})

// Automatically uses:
// - AWS: S3 + DynamoDB for metadata
// - Google Cloud: GCS + Firestore
// - Azure: Blob Storage + Cosmos DB
// - Cloudflare: R2 + D1
// - Vercel: Blob + KV
```

### 📝 Storage Coordination Rules

When multiple nodes share storage, Brainy automatically:

1. **Namespace Isolation**: Each node gets unique prefix
2. **Lock-Free Writes**: Uses atomic operations
3. **Consistent Metadata**: Coordinated via consensus
4. **Conflict Resolution**: Version vectors for conflicts
5. **Garbage Collection**: Automatic cleanup of old data

## Scaling Patterns

### 📈 Progressive Scaling Journey

#### Stage 1: Prototype (1 node, memory)
```typescript
const brain = new Brainy()  // Memory storage, single node
// Perfect for: Development, testing, <1000 items
```

#### Stage 2: Production (1 node, disk)
```typescript
const brain = new Brainy({
  storage: './data'  // Persistent storage
})
// Perfect for: Small apps, <100K items
```

#### Stage 3: High Availability (2-3 nodes)
```typescript
// Just start same code on multiple servers!
const brain = new Brainy({
  storage: './data'  // Each node's own storage
})
// Automatic: Leader election, replication, failover
// Perfect for: Critical apps, <1M items
```

#### Stage 4: Scale Out (N nodes)
```typescript
// Same code, more servers!
const brain = new Brainy({
  storage: 's3://brainy-{{nodeId}}'  // Template auto-filled
})
// Automatic: Sharding, load balancing, geo-distribution
// Perfect for: Large apps, unlimited items
```

### 🎯 Common Scaling Scenarios

#### Scenario: Read-Heavy Application
```typescript
// Brainy auto-detects read-heavy pattern and:
// 1. Increases cache size
// 2. Creates more read replicas
// 3. Routes reads to nearest node
// 4. Caches popular items on all nodes

const brain = new Brainy()  // No config needed!
```

#### Scenario: Multi-Tenant SaaS
```typescript
// Brainy auto-detects tenant patterns and:
// 1. Shards by tenant ID
// 2. Isolates tenant data
// 3. Routes by tenant
// 4. Separate rate limits per tenant

const brain = new Brainy()  // Detects from your queries!
```

#### Scenario: Geographic Distribution
```typescript
// Deploy nodes in different regions
// Brainy automatically:
// 1. Detects node locations (via latency)
// 2. Replicates data geographically
// 3. Routes to nearest node
// 4. Handles region failures

// US-East
const brain = new Brainy({ region: 'us-east' })  // Optional hint

// EU-West (auto-discovers US-East)
const brain = new Brainy({ region: 'eu-west' })
```

## Real World Examples

### Example 1: Blog Platform

```typescript
// Day 1: Single server
const brain = new Brainy({
  storage: './blog-data'
})

// Month 6: Add redundancy (on second server)
const brain = new Brainy({
  storage: './blog-data'  // Different machine!
})
// Automatically syncs with first server

// Year 2: Global scale
// US Server
const brain = new Brainy({
  storage: 's3://blog-us/data'
})

// EU Server  
const brain = new Brainy({
  storage: 's3://blog-eu/data'
})

// Asia Server
const brain = new Brainy({
  storage: 's3://blog-asia/data'
})
// All automatically coordinate!
```

### Example 2: E-Commerce Site

```typescript
// Development
const brain = new Brainy()  // Memory storage

// Staging (Kubernetes)
const brain = new Brainy({
  storage: process.env.STORAGE_PATH  // Uses PVC
})
// Auto-discovers other pods via K8s DNS

// Production (AWS)
const brain = new Brainy({
  storage: 's3://shop-data',
  cache: 'elasticache://shop-cache'  // Optional
})
// Auto-scales with ECS/EKS
```

### Example 3: Analytics Platform

```typescript
// Ingestion nodes (write-optimized)
const brain = new Brainy({
  role: 'writer',  // Hint for optimization
  storage: '/fast-nvme/ingest'
})

// Query nodes (read-optimized)
const brain = new Brainy({
  role: 'reader',  // More cache, indexes
  storage: 's3://analytics-archive'
})

// Automatically coordinates between writers and readers!
```

## 🔧 Storage Adapter Specifics

### Local Filesystem
```typescript
{
  storage: './data'  // or absolute: '/var/lib/brainy'
  // Each node MUST have separate directory
  // Can be network mounted (NFS, EFS)
}
```

### AWS S3
```typescript
{
  storage: 's3://bucket-name/prefix'
  // Uses AWS SDK credentials (env, IAM role, etc)
  // Supports S3-compatible (MinIO, Ceph)
}
```

### Cloudflare R2
```typescript
{
  storage: 'r2://bucket-name'
  // Uses Wrangler or API tokens
  // Zero egress fees!
}
```

### Google Cloud Storage
```typescript
{
  storage: 'gs://bucket-name'
  // Uses Application Default Credentials
}
```

### Azure Blob Storage
```typescript
{
  storage: 'azure://container-name'
  // Uses DefaultAzureCredential
}
```

### Mixed/Tiered
```typescript
{
  storage: {
    hot: './local-cache',      // Fast SSD
    warm: 's3://regular-data',  // Standard storage
    cold: 's3://glacier-archive' // Cheap archive
  }
  // Automatic tiering based on access patterns
}
```

## 🎭 Advanced Patterns

### Pattern: Blue-Green Deployment
```typescript
// Blue cluster (current)
const brain = new Brainy({
  cluster: 'blue',
  storage: 's3://prod-blue'
})

// Green cluster (new version)
const brain = new Brainy({
  cluster: 'green',
  storage: 's3://prod-green',
  syncFrom: 'blue'  // Real-time sync during migration
})
```

### Pattern: Federation
```typescript
// Region 1 Cluster
const brain1 = new Brainy({
  federation: 'global',
  region: 'us-east',
  storage: 's3://us-east-data'
})

// Region 2 Cluster
const brain2 = new Brainy({
  federation: 'global',
  region: 'eu-west',
  storage: 's3://eu-west-data'
})
// Clusters coordinate for global queries!
```

### Pattern: Edge Computing
```typescript
// Edge nodes (in CDN POPs)
const brain = new Brainy({
  mode: 'edge',
  storage: 'memory',  // RAM only
  upstream: 'https://main-cluster.example.com'
})
// Caches frequently accessed data at edge
```

## 📊 Monitoring & Observability

Brainy automatically exposes metrics:

```typescript
const metrics = brain.getMetrics()
// {
//   nodes: { total: 5, healthy: 5 },
//   shards: { total: 20, local: 4 },
//   replication: { factor: 2, lag: 45 },
//   operations: { reads: 10000, writes: 1000 },
//   storage: { used: '45GB', available: '955GB' }
// }
```

## 🚨 Troubleshooting

### Issue: Nodes don't discover each other
```typescript
// Solution 1: Check network allows UDP 7946
// Solution 2: Use explicit peers
const brain = new Brainy({
  peers: ['10.0.0.1:7946', '10.0.0.2:7946']
})
```

### Issue: Storage conflicts
```typescript
// Ensure each node has unique storage path
// ❌ WRONG: All nodes use './data'
// ✅ RIGHT: Node1: './data1', Node2: './data2'
// ✅ RIGHT: Use {{nodeId}} template
```

### Issue: Slow performance
```typescript
// Brainy auto-tunes, but you can hint:
const brain = new Brainy({
  profile: 'read-heavy'  // or 'write-heavy', 'balanced'
})
```

## 🎯 Best Practices

1. **Let Brainy Auto-Configure**: Don't over-configure
2. **Separate Storage Per Node**: Avoids conflicts
3. **Use S3 for Large Scale**: Infinite capacity
4. **Start Simple**: Single node → Scale when needed
5. **Monitor Metrics**: Watch for bottlenecks
6. **Trust Auto-Scaling**: It learns your patterns

## 🚀 Summary

- **Zero Config**: Just `new Brainy()` at any scale
- **Auto-Discovery**: Nodes find each other
- **Smart Storage**: Adapts to any backend
- **Progressive Scaling**: 1 → 100 nodes seamlessly
- **Self-Tuning**: Learns and optimizes
- **No DevOps**: It just works!

**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!**

---

*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)*