diff --git a/docs/SCALING.md b/docs/SCALING.md new file mode 100644 index 00000000..d071a838 --- /dev/null +++ b/docs/SCALING.md @@ -0,0 +1,502 @@ +# 🚀 Brainy Scaling Guide - Enterprise for Everyone + +> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required. + +## 📖 Table of Contents +- [Quick Start](#quick-start) +- [How It Works](#how-it-works) +- [Storage Configurations](#storage-configurations) +- [Scaling Patterns](#scaling-patterns) +- [Real World Examples](#real-world-examples) + +## Quick Start + +### Single Node (Default) +```typescript +import Brainy from '@soulcraft/brainy' +const brain = new Brainy() // That's it! +``` + +### Multi-Node (Auto-Discovery) +```typescript +// Node 1 +const brain = new Brainy() // Starts as primary + +// Node 2 (different server) +const brain = new Brainy() // Auto-discovers Node 1, becomes replica! +``` + +**That's literally all you need!** Brainy handles everything else automatically. + +## How It Works + +### 🎯 The Magic: Zero Configuration + +Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration: + +1. **First node starts** → Becomes primary automatically +2. **Second node starts** → Discovers first node via UDP broadcast +3. **Nodes negotiate** → Elect leader, distribute shards +4. **Data flows** → Automatic replication and routing +5. **Node fails** → Automatic failover in <1 second + +### 🔄 Automatic Node Discovery + +```typescript +// Three ways Brainy finds other nodes (auto-selected): + +// 1. LOCAL NETWORK (Default) +// Uses UDP broadcast on port 7946 +// Perfect for: On-premise, same VPC + +// 2. CLOUD NATIVE (Auto-detected) +// Kubernetes: Uses k8s DNS service discovery +// AWS: Uses EC2 tags or Route53 +// Azure: Uses Azure DNS + +// 3. EXPLICIT (When needed) +const brain = new Brainy({ + peers: ['node1.example.com', 'node2.example.com'] +}) +``` + +### 📊 Data Distribution + +When you add data, Brainy automatically: + +```typescript +brain.add({ name: "John" }, 'person') + +// Behind the scenes: +// 1. Hash ID to determine shard (consistent hashing) +// 2. Find nodes responsible for this shard +// 3. Write to primary shard owner +// 4. Replicate to N backup nodes (default: 2) +// 5. Confirm write when majority acknowledge +``` + +## Storage Configurations + +### 🗂️ Storage Adapter Patterns + +Brainy intelligently adapts to your storage setup: + +#### Pattern 1: Separate Storage Per Node (Recommended) + +```typescript +// Node 1 - Own filesystem +const brain1 = new Brainy({ + storage: '/data/node1' // or auto: './brainy-data' +}) + +// Node 2 - Own filesystem +const brain2 = new Brainy({ + storage: '/data/node2' // or auto: './brainy-data' +}) + +// ✅ BENEFITS: +// - No conflicts between nodes +// - Fast local reads +// - True horizontal scaling +// - Survives network partitions +``` + +#### Pattern 2: Separate S3 Buckets Per Node + +```typescript +// Node 1 - Own S3 bucket +const brain1 = new Brainy({ + storage: 's3://brainy-node-1' // Auto-uses AWS credentials +}) + +// Node 2 - Own S3 bucket +const brain2 = new Brainy({ + storage: 's3://brainy-node-2' +}) + +// ✅ BENEFITS: +// - Infinite storage capacity +// - Geographic distribution +// - No local disk needed +// - Built-in durability +``` + +#### Pattern 3: Shared S3 Bucket (Coordinated) + +```typescript +// All nodes - Shared bucket with coordination +const brain = new Brainy({ + storage: 's3://shared-brainy-data', + // Brainy automatically adds node-specific prefixes! +}) + +// What happens automatically: +// - Node 1 writes to: s3://shared-brainy-data/node-1/ +// - Node 2 writes to: s3://shared-brainy-data/node-2/ +// - Metadata in: s3://shared-brainy-data/_cluster/ +// - Coordination via S3 conditional writes + +// ✅ BENEFITS: +// - Single bucket to manage +// - Easy backup/restore +// - Cost effective +// - Automatic namespace isolation +``` + +#### Pattern 4: Mixed Storage (Hybrid) + +```typescript +// Hot data on local SSD, cold data in S3 +const brain = new Brainy({ + storage: { + hot: '/fast-ssd/brainy', // Recent/frequent data + cold: 's3://brainy-archive' // Older data + } + // Brainy automatically promotes/demotes data! +}) +``` + +### 🌍 Cloud Provider Auto-Detection + +```typescript +const brain = new Brainy({ + storage: 'cloud://brainy-data' // Auto-detects provider! +}) + +// Automatically uses: +// - AWS: S3 + DynamoDB for metadata +// - Google Cloud: GCS + Firestore +// - Azure: Blob Storage + Cosmos DB +// - Cloudflare: R2 + D1 +// - Vercel: Blob + KV +``` + +### 📝 Storage Coordination Rules + +When multiple nodes share storage, Brainy automatically: + +1. **Namespace Isolation**: Each node gets unique prefix +2. **Lock-Free Writes**: Uses atomic operations +3. **Consistent Metadata**: Coordinated via consensus +4. **Conflict Resolution**: Version vectors for conflicts +5. **Garbage Collection**: Automatic cleanup of old data + +## Scaling Patterns + +### 📈 Progressive Scaling Journey + +#### Stage 1: Prototype (1 node, memory) +```typescript +const brain = new Brainy() // Memory storage, single node +// Perfect for: Development, testing, <1000 items +``` + +#### Stage 2: Production (1 node, disk) +```typescript +const brain = new Brainy({ + storage: './data' // Persistent storage +}) +// Perfect for: Small apps, <100K items +``` + +#### Stage 3: High Availability (2-3 nodes) +```typescript +// Just start same code on multiple servers! +const brain = new Brainy({ + storage: './data' // Each node's own storage +}) +// Automatic: Leader election, replication, failover +// Perfect for: Critical apps, <1M items +``` + +#### Stage 4: Scale Out (N nodes) +```typescript +// Same code, more servers! +const brain = new Brainy({ + storage: 's3://brainy-{{nodeId}}' // Template auto-filled +}) +// Automatic: Sharding, load balancing, geo-distribution +// Perfect for: Large apps, unlimited items +``` + +### 🎯 Common Scaling Scenarios + +#### Scenario: Read-Heavy Application +```typescript +// Brainy auto-detects read-heavy pattern and: +// 1. Increases cache size +// 2. Creates more read replicas +// 3. Routes reads to nearest node +// 4. Caches popular items on all nodes + +const brain = new Brainy() // No config needed! +``` + +#### Scenario: Multi-Tenant SaaS +```typescript +// Brainy auto-detects tenant patterns and: +// 1. Shards by tenant ID +// 2. Isolates tenant data +// 3. Routes by tenant +// 4. Separate rate limits per tenant + +const brain = new Brainy() // Detects from your queries! +``` + +#### Scenario: Geographic Distribution +```typescript +// Deploy nodes in different regions +// Brainy automatically: +// 1. Detects node locations (via latency) +// 2. Replicates data geographically +// 3. Routes to nearest node +// 4. Handles region failures + +// US-East +const brain = new Brainy({ region: 'us-east' }) // Optional hint + +// EU-West (auto-discovers US-East) +const brain = new Brainy({ region: 'eu-west' }) +``` + +## Real World Examples + +### Example 1: Blog Platform + +```typescript +// Day 1: Single server +const brain = new Brainy({ + storage: './blog-data' +}) + +// Month 6: Add redundancy (on second server) +const brain = new Brainy({ + storage: './blog-data' // Different machine! +}) +// Automatically syncs with first server + +// Year 2: Global scale +// US Server +const brain = new Brainy({ + storage: 's3://blog-us/data' +}) + +// EU Server +const brain = new Brainy({ + storage: 's3://blog-eu/data' +}) + +// Asia Server +const brain = new Brainy({ + storage: 's3://blog-asia/data' +}) +// All automatically coordinate! +``` + +### Example 2: E-Commerce Site + +```typescript +// Development +const brain = new Brainy() // Memory storage + +// Staging (Kubernetes) +const brain = new Brainy({ + storage: process.env.STORAGE_PATH // Uses PVC +}) +// Auto-discovers other pods via K8s DNS + +// Production (AWS) +const brain = new Brainy({ + storage: 's3://shop-data', + cache: 'elasticache://shop-cache' // Optional +}) +// Auto-scales with ECS/EKS +``` + +### Example 3: Analytics Platform + +```typescript +// Ingestion nodes (write-optimized) +const brain = new Brainy({ + role: 'writer', // Hint for optimization + storage: '/fast-nvme/ingest' +}) + +// Query nodes (read-optimized) +const brain = new Brainy({ + role: 'reader', // More cache, indexes + storage: 's3://analytics-archive' +}) + +// Automatically coordinates between writers and readers! +``` + +## 🔧 Storage Adapter Specifics + +### Local Filesystem +```typescript +{ + storage: './data' // or absolute: '/var/lib/brainy' + // Each node MUST have separate directory + // Can be network mounted (NFS, EFS) +} +``` + +### AWS S3 +```typescript +{ + storage: 's3://bucket-name/prefix' + // Uses AWS SDK credentials (env, IAM role, etc) + // Supports S3-compatible (MinIO, Ceph) +} +``` + +### Cloudflare R2 +```typescript +{ + storage: 'r2://bucket-name' + // Uses Wrangler or API tokens + // Zero egress fees! +} +``` + +### Google Cloud Storage +```typescript +{ + storage: 'gs://bucket-name' + // Uses Application Default Credentials +} +``` + +### Azure Blob Storage +```typescript +{ + storage: 'azure://container-name' + // Uses DefaultAzureCredential +} +``` + +### Mixed/Tiered +```typescript +{ + storage: { + hot: './local-cache', // Fast SSD + warm: 's3://regular-data', // Standard storage + cold: 's3://glacier-archive' // Cheap archive + } + // Automatic tiering based on access patterns +} +``` + +## 🎭 Advanced Patterns + +### Pattern: Blue-Green Deployment +```typescript +// Blue cluster (current) +const brain = new Brainy({ + cluster: 'blue', + storage: 's3://prod-blue' +}) + +// Green cluster (new version) +const brain = new Brainy({ + cluster: 'green', + storage: 's3://prod-green', + syncFrom: 'blue' // Real-time sync during migration +}) +``` + +### Pattern: Federation +```typescript +// Region 1 Cluster +const brain1 = new Brainy({ + federation: 'global', + region: 'us-east', + storage: 's3://us-east-data' +}) + +// Region 2 Cluster +const brain2 = new Brainy({ + federation: 'global', + region: 'eu-west', + storage: 's3://eu-west-data' +}) +// Clusters coordinate for global queries! +``` + +### Pattern: Edge Computing +```typescript +// Edge nodes (in CDN POPs) +const brain = new Brainy({ + mode: 'edge', + storage: 'memory', // RAM only + upstream: 'https://main-cluster.example.com' +}) +// Caches frequently accessed data at edge +``` + +## 📊 Monitoring & Observability + +Brainy automatically exposes metrics: + +```typescript +const metrics = brain.getMetrics() +// { +// nodes: { total: 5, healthy: 5 }, +// shards: { total: 20, local: 4 }, +// replication: { factor: 2, lag: 45 }, +// operations: { reads: 10000, writes: 1000 }, +// storage: { used: '45GB', available: '955GB' } +// } +``` + +## 🚨 Troubleshooting + +### Issue: Nodes don't discover each other +```typescript +// Solution 1: Check network allows UDP 7946 +// Solution 2: Use explicit peers +const brain = new Brainy({ + peers: ['10.0.0.1:7946', '10.0.0.2:7946'] +}) +``` + +### Issue: Storage conflicts +```typescript +// Ensure each node has unique storage path +// ❌ WRONG: All nodes use './data' +// ✅ RIGHT: Node1: './data1', Node2: './data2' +// ✅ RIGHT: Use {{nodeId}} template +``` + +### Issue: Slow performance +```typescript +// Brainy auto-tunes, but you can hint: +const brain = new Brainy({ + profile: 'read-heavy' // or 'write-heavy', 'balanced' +}) +``` + +## 🎯 Best Practices + +1. **Let Brainy Auto-Configure**: Don't over-configure +2. **Separate Storage Per Node**: Avoids conflicts +3. **Use S3 for Large Scale**: Infinite capacity +4. **Start Simple**: Single node → Scale when needed +5. **Monitor Metrics**: Watch for bottlenecks +6. **Trust Auto-Scaling**: It learns your patterns + +## 🚀 Summary + +- **Zero Config**: Just `new Brainy()` at any scale +- **Auto-Discovery**: Nodes find each other +- **Smart Storage**: Adapts to any backend +- **Progressive Scaling**: 1 → 100 nodes seamlessly +- **Self-Tuning**: Learns and optimizes +- **No DevOps**: It just works! + +**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!** + +--- + +*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)* \ No newline at end of file diff --git a/docs/architecture/distributed-storage.md b/docs/architecture/distributed-storage.md new file mode 100644 index 00000000..7f678cf2 --- /dev/null +++ b/docs/architecture/distributed-storage.md @@ -0,0 +1,550 @@ +# 🏗️ Distributed Storage Architecture + +> **Technical deep-dive**: How Brainy coordinates storage across multiple nodes and adapters + +## Storage Adapter Layer + +### Base Storage Interface + +Every storage adapter implements this interface: + +```typescript +interface StorageAdapter { + // Basic operations + get(key: string): Promise + set(key: string, value: any): Promise + delete(key: string): Promise + + // Batch operations + getBatch(keys: string[]): Promise> + setBatch(items: Map): Promise + + // Atomic operations (for coordination) + compareAndSwap(key: string, oldVal: any, newVal: any): Promise + increment(key: string, delta: number): Promise + + // Namespace support + withNamespace(namespace: string): StorageAdapter +} +``` + +### Storage Coordination Strategies + +#### Strategy 1: Isolated Storage (Default) + +Each node has completely separate storage: + +``` +Node-1 → Local FS: /data/node1/ + └── shards/ + ├── shard-001/ + ├── shard-045/ + └── shard-127/ + +Node-2 → Local FS: /data/node2/ + └── shards/ + ├── shard-023/ + ├── shard-067/ + └── shard-089/ +``` + +**Coordination**: Via network messages only +- Shard ownership tracked in distributed consensus +- Data transfer via direct node-to-node communication +- No storage-level conflicts possible + +#### Strategy 2: Shared Storage with Namespacing + +Multiple nodes share storage but use namespaces: + +``` +S3 Bucket: brainy-cluster/ +├── node-abc123/ +│ ├── shards/ +│ └── wal/ +├── node-def456/ +│ ├── shards/ +│ └── wal/ +└── _cluster/ + ├── topology.json + ├── shard-map.json + └── elections/ +``` + +**Coordination**: Via storage-level atomic operations +- Each node owns its namespace +- Cluster metadata in shared `_cluster/` namespace +- Atomic operations for leader election +- Conditional writes prevent conflicts + +#### Strategy 3: Shared Storage with Fine-Grained Locking + +Advanced mode for full shared storage: + +``` +S3 Bucket: brainy-shared/ +├── shards/ +│ ├── 001/ +│ │ ├── data.bin +│ │ └── .lock (atomic) +│ ├── 002/ +│ │ ├── data.bin +│ │ └── .lock +└── metadata/ + ├── index/ + └── locks/ +``` + +**Coordination**: Via distributed locking +- Shard-level locks using atomic operations +- Lock acquisition via compare-and-swap +- Automatic lock expiry (lease-based) +- Deadlock detection and recovery + +## Storage Adapter Implementations + +### 1. Filesystem Adapter + +```typescript +class FilesystemAdapter implements StorageAdapter { + constructor(private basePath: string) {} + + async get(key: string) { + const path = this.keyToPath(key) + return fs.readFile(path, 'json') + } + + async compareAndSwap(key: string, oldVal: any, newVal: any) { + // Use file locking for atomicity + const lockfile = `${this.keyToPath(key)}.lock` + await flock(lockfile, 'ex') // Exclusive lock + try { + const current = await this.get(key) + if (deepEqual(current, oldVal)) { + await this.set(key, newVal) + return true + } + return false + } finally { + await funlock(lockfile) + } + } + + withNamespace(ns: string) { + return new FilesystemAdapter(path.join(this.basePath, ns)) + } +} +``` + +### 2. S3 Adapter + +```typescript +class S3Adapter implements StorageAdapter { + constructor( + private bucket: string, + private prefix: string = '' + ) {} + + async get(key: string) { + const result = await s3.getObject({ + Bucket: this.bucket, + Key: `${this.prefix}${key}` + }) + return JSON.parse(result.Body) + } + + async compareAndSwap(key: string, oldVal: any, newVal: any) { + // Use S3's conditional writes + const fullKey = `${this.prefix}${key}` + + // Get current version + const head = await s3.headObject({ + Bucket: this.bucket, + Key: fullKey + }) + + // Conditional put with ETag + try { + await s3.putObject({ + Bucket: this.bucket, + Key: fullKey, + Body: JSON.stringify(newVal), + IfMatch: head.ETag // Only succeeds if unchanged + }) + return true + } catch (err) { + if (err.code === 'PreconditionFailed') { + return false + } + throw err + } + } + + withNamespace(ns: string) { + const newPrefix = `${this.prefix}${ns}/` + return new S3Adapter(this.bucket, newPrefix) + } +} +``` + +### 3. Cloudflare R2 Adapter + +```typescript +class R2Adapter implements StorageAdapter { + // Similar to S3 but with R2-specific optimizations + + async compareAndSwap(key: string, oldVal: any, newVal: any) { + // R2 supports conditional headers + const response = await fetch(`${this.endpoint}/${key}`, { + method: 'PUT', + body: JSON.stringify(newVal), + headers: { + 'If-Match': await this.getETag(key) + } + }) + return response.ok + } + + // R2-specific: Use Workers for edge computing + async getWithCache(key: string) { + // Check Cloudflare edge cache first + const cached = await caches.default.match(key) + if (cached) return cached.json() + + // Fallback to R2 + const value = await this.get(key) + + // Cache at edge + await caches.default.put(key, new Response(JSON.stringify(value))) + + return value + } +} +``` + +## Distributed Coordination Patterns + +### Pattern 1: Leader-Based Coordination + +```typescript +class LeaderCoordinator { + async acquireShardOwnership(shardId: string) { + if (!this.isLeader()) { + // Only leader assigns shards + return this.requestFromLeader('acquireShard', shardId) + } + + // Leader logic + const shardMap = await this.storage.get('_cluster/shard-map') + if (!shardMap[shardId].owner) { + shardMap[shardId].owner = this.nodeId + + // Atomic update + const success = await this.storage.compareAndSwap( + '_cluster/shard-map', + shardMap, + { ...shardMap, [shardId]: { owner: this.nodeId } } + ) + + if (success) { + this.broadcast('shardAssigned', { shardId, owner: this.nodeId }) + } + } + } +} +``` + +### Pattern 2: Consensus-Based Coordination + +```typescript +class ConsensusCoordinator { + async acquireShardOwnership(shardId: string) { + // Propose to all nodes + const proposal = { + type: 'ACQUIRE_SHARD', + shardId, + nodeId: this.nodeId, + term: this.currentTerm + } + + // Raft consensus + const votes = await this.gatherVotes(proposal) + + if (votes.length > this.nodes.length / 2) { + // Majority agreed + await this.commitProposal(proposal) + return true + } + + return false + } +} +``` + +### Pattern 3: Storage-Native Coordination + +```typescript +class StorageNativeCoordinator { + async acquireShardOwnership(shardId: string) { + // Use storage adapter's native coordination + const lockKey = `_locks/shard-${shardId}` + const lease = { + owner: this.nodeId, + expires: Date.now() + 30000 // 30 second lease + } + + // Try to acquire lock atomically + const acquired = await this.storage.compareAndSwap( + lockKey, + null, // Must not exist + lease + ) + + if (acquired) { + // Start lease renewal + this.startLeaseRenewal(lockKey, lease) + return true + } + + return false + } + + private startLeaseRenewal(key: string, lease: any) { + setInterval(async () => { + const renewed = await this.storage.compareAndSwap( + key, + lease, + { ...lease, expires: Date.now() + 30000 } + ) + + if (!renewed) { + // Lost lease + this.handleLeaseLoss(key) + } + }, 10000) // Renew every 10s + } +} +``` + +## Multi-Storage Patterns + +### Hybrid Storage (Hot/Cold) + +```typescript +class HybridStorageAdapter implements StorageAdapter { + constructor( + private hot: StorageAdapter, // Fast SSD + private cold: StorageAdapter // Cheap S3 + ) {} + + async get(key: string) { + // Try hot storage first + const hotValue = await this.hot.get(key).catch(() => null) + if (hotValue) { + this.updateAccessTime(key) + return hotValue + } + + // Fallback to cold storage + const coldValue = await this.cold.get(key) + + // Promote to hot storage if frequently accessed + if (this.shouldPromote(key)) { + await this.hot.set(key, coldValue) + } + + return coldValue + } + + async set(key: string, value: any) { + // Write to hot storage + await this.hot.set(key, value) + + // Async write to cold storage + setImmediate(() => { + this.cold.set(key, value).catch(console.error) + }) + } + + // Background process to demote cold data + async runTiering() { + const hotKeys = await this.hot.listKeys() + + for (const key of hotKeys) { + const lastAccess = await this.getAccessTime(key) + + if (Date.now() - lastAccess > 7 * 24 * 60 * 60 * 1000) { + // Not accessed in 7 days, demote to cold + await this.cold.set(key, await this.hot.get(key)) + await this.hot.delete(key) + } + } + } +} +``` + +### Geo-Distributed Storage + +```typescript +class GeoDistributedAdapter implements StorageAdapter { + constructor( + private regions: Map + ) {} + + async get(key: string) { + // Determine closest region + const region = await this.getClosestRegion() + + // Try local region first + const localValue = await this.regions.get(region) + .get(key) + .catch(() => null) + + if (localValue) return localValue + + // Fallback to other regions + for (const [name, adapter] of this.regions) { + if (name !== region) { + const value = await adapter.get(key).catch(() => null) + if (value) { + // Replicate to local region for next time + this.regions.get(region).set(key, value) + return value + } + } + } + + throw new Error('Key not found in any region') + } + + async set(key: string, value: any) { + // Write to local region immediately + const region = await this.getClosestRegion() + await this.regions.get(region).set(key, value) + + // Async replication to other regions + for (const [name, adapter] of this.regions) { + if (name !== region) { + adapter.set(key, value).catch(console.error) + } + } + } +} +``` + +## Storage Optimization Strategies + +### 1. Write Batching + +```typescript +class BatchingAdapter implements StorageAdapter { + private writeBatch = new Map() + private batchTimer?: NodeJS.Timeout + + async set(key: string, value: any) { + this.writeBatch.set(key, value) + + if (!this.batchTimer) { + this.batchTimer = setTimeout(() => this.flush(), 100) + } + + if (this.writeBatch.size >= 1000) { + await this.flush() + } + } + + private async flush() { + if (this.writeBatch.size === 0) return + + const batch = new Map(this.writeBatch) + this.writeBatch.clear() + + await this.underlying.setBatch(batch) + + if (this.batchTimer) { + clearTimeout(this.batchTimer) + this.batchTimer = undefined + } + } +} +``` + +### 2. Read Caching + +```typescript +class CachingAdapter implements StorageAdapter { + private cache = new LRU({ max: 10000 }) + + async get(key: string) { + // Check cache + if (this.cache.has(key)) { + return this.cache.get(key) + } + + // Read from storage + const value = await this.underlying.get(key) + + // Cache for next time + this.cache.set(key, value) + + return value + } + + async set(key: string, value: any) { + // Invalidate cache + this.cache.delete(key) + + // Write through + await this.underlying.set(key, value) + } +} +``` + +### 3. Compression + +```typescript +class CompressingAdapter implements StorageAdapter { + async set(key: string, value: any) { + const json = JSON.stringify(value) + + // Compress if beneficial + if (json.length > 1024) { + const compressed = await gzip(json) + await this.underlying.set(key, { + _compressed: true, + data: compressed.toString('base64') + }) + } else { + await this.underlying.set(key, value) + } + } + + async get(key: string) { + const stored = await this.underlying.get(key) + + if (stored._compressed) { + const compressed = Buffer.from(stored.data, 'base64') + const json = await gunzip(compressed) + return JSON.parse(json) + } + + return stored + } +} +``` + +## Summary + +Brainy's storage layer is designed for: + +1. **Flexibility**: Works with any storage backend +2. **Coordination**: Multiple strategies for different needs +3. **Performance**: Batching, caching, compression +4. **Scalability**: From single file to geo-distributed +5. **Simplicity**: Complexity hidden behind simple interface + +The key insight: **Storage is just a plugin**. The intelligence is in the coordination layer above it! + +--- + +*For user-facing documentation, see [SCALING.md](../SCALING.md)* \ No newline at end of file