docs: add comprehensive scaling and storage architecture documentation
- Add user-friendly SCALING.md explaining Enterprise for Everyone - Document zero-config philosophy and auto-discovery - Explain storage adapter patterns and coordination strategies - Add real-world examples and best practices - Create technical deep-dive on distributed storage architecture - Document how different storage backends work together - Explain coordination strategies for shared vs isolated storage
This commit is contained in:
parent
8ead063df8
commit
f65455fb22
2 changed files with 1052 additions and 0 deletions
502
docs/SCALING.md
Normal file
502
docs/SCALING.md
Normal file
|
|
@ -0,0 +1,502 @@
|
|||
# 🚀 Brainy Scaling Guide - Enterprise for Everyone
|
||||
|
||||
> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required.
|
||||
|
||||
## 📖 Table of Contents
|
||||
- [Quick Start](#quick-start)
|
||||
- [How It Works](#how-it-works)
|
||||
- [Storage Configurations](#storage-configurations)
|
||||
- [Scaling Patterns](#scaling-patterns)
|
||||
- [Real World Examples](#real-world-examples)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Single Node (Default)
|
||||
```typescript
|
||||
import Brainy from '@soulcraft/brainy'
|
||||
const brain = new Brainy() // That's it!
|
||||
```
|
||||
|
||||
### Multi-Node (Auto-Discovery)
|
||||
```typescript
|
||||
// Node 1
|
||||
const brain = new Brainy() // Starts as primary
|
||||
|
||||
// Node 2 (different server)
|
||||
const brain = new Brainy() // Auto-discovers Node 1, becomes replica!
|
||||
```
|
||||
|
||||
**That's literally all you need!** Brainy handles everything else automatically.
|
||||
|
||||
## How It Works
|
||||
|
||||
### 🎯 The Magic: Zero Configuration
|
||||
|
||||
Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration:
|
||||
|
||||
1. **First node starts** → Becomes primary automatically
|
||||
2. **Second node starts** → Discovers first node via UDP broadcast
|
||||
3. **Nodes negotiate** → Elect leader, distribute shards
|
||||
4. **Data flows** → Automatic replication and routing
|
||||
5. **Node fails** → Automatic failover in <1 second
|
||||
|
||||
### 🔄 Automatic Node Discovery
|
||||
|
||||
```typescript
|
||||
// Three ways Brainy finds other nodes (auto-selected):
|
||||
|
||||
// 1. LOCAL NETWORK (Default)
|
||||
// Uses UDP broadcast on port 7946
|
||||
// Perfect for: On-premise, same VPC
|
||||
|
||||
// 2. CLOUD NATIVE (Auto-detected)
|
||||
// Kubernetes: Uses k8s DNS service discovery
|
||||
// AWS: Uses EC2 tags or Route53
|
||||
// Azure: Uses Azure DNS
|
||||
|
||||
// 3. EXPLICIT (When needed)
|
||||
const brain = new Brainy({
|
||||
peers: ['node1.example.com', 'node2.example.com']
|
||||
})
|
||||
```
|
||||
|
||||
### 📊 Data Distribution
|
||||
|
||||
When you add data, Brainy automatically:
|
||||
|
||||
```typescript
|
||||
brain.add({ name: "John" }, 'person')
|
||||
|
||||
// Behind the scenes:
|
||||
// 1. Hash ID to determine shard (consistent hashing)
|
||||
// 2. Find nodes responsible for this shard
|
||||
// 3. Write to primary shard owner
|
||||
// 4. Replicate to N backup nodes (default: 2)
|
||||
// 5. Confirm write when majority acknowledge
|
||||
```
|
||||
|
||||
## Storage Configurations
|
||||
|
||||
### 🗂️ Storage Adapter Patterns
|
||||
|
||||
Brainy intelligently adapts to your storage setup:
|
||||
|
||||
#### Pattern 1: Separate Storage Per Node (Recommended)
|
||||
|
||||
```typescript
|
||||
// Node 1 - Own filesystem
|
||||
const brain1 = new Brainy({
|
||||
storage: '/data/node1' // or auto: './brainy-data'
|
||||
})
|
||||
|
||||
// Node 2 - Own filesystem
|
||||
const brain2 = new Brainy({
|
||||
storage: '/data/node2' // or auto: './brainy-data'
|
||||
})
|
||||
|
||||
// ✅ BENEFITS:
|
||||
// - No conflicts between nodes
|
||||
// - Fast local reads
|
||||
// - True horizontal scaling
|
||||
// - Survives network partitions
|
||||
```
|
||||
|
||||
#### Pattern 2: Separate S3 Buckets Per Node
|
||||
|
||||
```typescript
|
||||
// Node 1 - Own S3 bucket
|
||||
const brain1 = new Brainy({
|
||||
storage: 's3://brainy-node-1' // Auto-uses AWS credentials
|
||||
})
|
||||
|
||||
// Node 2 - Own S3 bucket
|
||||
const brain2 = new Brainy({
|
||||
storage: 's3://brainy-node-2'
|
||||
})
|
||||
|
||||
// ✅ BENEFITS:
|
||||
// - Infinite storage capacity
|
||||
// - Geographic distribution
|
||||
// - No local disk needed
|
||||
// - Built-in durability
|
||||
```
|
||||
|
||||
#### Pattern 3: Shared S3 Bucket (Coordinated)
|
||||
|
||||
```typescript
|
||||
// All nodes - Shared bucket with coordination
|
||||
const brain = new Brainy({
|
||||
storage: 's3://shared-brainy-data',
|
||||
// Brainy automatically adds node-specific prefixes!
|
||||
})
|
||||
|
||||
// What happens automatically:
|
||||
// - Node 1 writes to: s3://shared-brainy-data/node-1/
|
||||
// - Node 2 writes to: s3://shared-brainy-data/node-2/
|
||||
// - Metadata in: s3://shared-brainy-data/_cluster/
|
||||
// - Coordination via S3 conditional writes
|
||||
|
||||
// ✅ BENEFITS:
|
||||
// - Single bucket to manage
|
||||
// - Easy backup/restore
|
||||
// - Cost effective
|
||||
// - Automatic namespace isolation
|
||||
```
|
||||
|
||||
#### Pattern 4: Mixed Storage (Hybrid)
|
||||
|
||||
```typescript
|
||||
// Hot data on local SSD, cold data in S3
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
hot: '/fast-ssd/brainy', // Recent/frequent data
|
||||
cold: 's3://brainy-archive' // Older data
|
||||
}
|
||||
// Brainy automatically promotes/demotes data!
|
||||
})
|
||||
```
|
||||
|
||||
### 🌍 Cloud Provider Auto-Detection
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: 'cloud://brainy-data' // Auto-detects provider!
|
||||
})
|
||||
|
||||
// Automatically uses:
|
||||
// - AWS: S3 + DynamoDB for metadata
|
||||
// - Google Cloud: GCS + Firestore
|
||||
// - Azure: Blob Storage + Cosmos DB
|
||||
// - Cloudflare: R2 + D1
|
||||
// - Vercel: Blob + KV
|
||||
```
|
||||
|
||||
### 📝 Storage Coordination Rules
|
||||
|
||||
When multiple nodes share storage, Brainy automatically:
|
||||
|
||||
1. **Namespace Isolation**: Each node gets unique prefix
|
||||
2. **Lock-Free Writes**: Uses atomic operations
|
||||
3. **Consistent Metadata**: Coordinated via consensus
|
||||
4. **Conflict Resolution**: Version vectors for conflicts
|
||||
5. **Garbage Collection**: Automatic cleanup of old data
|
||||
|
||||
## Scaling Patterns
|
||||
|
||||
### 📈 Progressive Scaling Journey
|
||||
|
||||
#### Stage 1: Prototype (1 node, memory)
|
||||
```typescript
|
||||
const brain = new Brainy() // Memory storage, single node
|
||||
// Perfect for: Development, testing, <1000 items
|
||||
```
|
||||
|
||||
#### Stage 2: Production (1 node, disk)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: './data' // Persistent storage
|
||||
})
|
||||
// Perfect for: Small apps, <100K items
|
||||
```
|
||||
|
||||
#### Stage 3: High Availability (2-3 nodes)
|
||||
```typescript
|
||||
// Just start same code on multiple servers!
|
||||
const brain = new Brainy({
|
||||
storage: './data' // Each node's own storage
|
||||
})
|
||||
// Automatic: Leader election, replication, failover
|
||||
// Perfect for: Critical apps, <1M items
|
||||
```
|
||||
|
||||
#### Stage 4: Scale Out (N nodes)
|
||||
```typescript
|
||||
// Same code, more servers!
|
||||
const brain = new Brainy({
|
||||
storage: 's3://brainy-{{nodeId}}' // Template auto-filled
|
||||
})
|
||||
// Automatic: Sharding, load balancing, geo-distribution
|
||||
// Perfect for: Large apps, unlimited items
|
||||
```
|
||||
|
||||
### 🎯 Common Scaling Scenarios
|
||||
|
||||
#### Scenario: Read-Heavy Application
|
||||
```typescript
|
||||
// Brainy auto-detects read-heavy pattern and:
|
||||
// 1. Increases cache size
|
||||
// 2. Creates more read replicas
|
||||
// 3. Routes reads to nearest node
|
||||
// 4. Caches popular items on all nodes
|
||||
|
||||
const brain = new Brainy() // No config needed!
|
||||
```
|
||||
|
||||
#### Scenario: Multi-Tenant SaaS
|
||||
```typescript
|
||||
// Brainy auto-detects tenant patterns and:
|
||||
// 1. Shards by tenant ID
|
||||
// 2. Isolates tenant data
|
||||
// 3. Routes by tenant
|
||||
// 4. Separate rate limits per tenant
|
||||
|
||||
const brain = new Brainy() // Detects from your queries!
|
||||
```
|
||||
|
||||
#### Scenario: Geographic Distribution
|
||||
```typescript
|
||||
// Deploy nodes in different regions
|
||||
// Brainy automatically:
|
||||
// 1. Detects node locations (via latency)
|
||||
// 2. Replicates data geographically
|
||||
// 3. Routes to nearest node
|
||||
// 4. Handles region failures
|
||||
|
||||
// US-East
|
||||
const brain = new Brainy({ region: 'us-east' }) // Optional hint
|
||||
|
||||
// EU-West (auto-discovers US-East)
|
||||
const brain = new Brainy({ region: 'eu-west' })
|
||||
```
|
||||
|
||||
## Real World Examples
|
||||
|
||||
### Example 1: Blog Platform
|
||||
|
||||
```typescript
|
||||
// Day 1: Single server
|
||||
const brain = new Brainy({
|
||||
storage: './blog-data'
|
||||
})
|
||||
|
||||
// Month 6: Add redundancy (on second server)
|
||||
const brain = new Brainy({
|
||||
storage: './blog-data' // Different machine!
|
||||
})
|
||||
// Automatically syncs with first server
|
||||
|
||||
// Year 2: Global scale
|
||||
// US Server
|
||||
const brain = new Brainy({
|
||||
storage: 's3://blog-us/data'
|
||||
})
|
||||
|
||||
// EU Server
|
||||
const brain = new Brainy({
|
||||
storage: 's3://blog-eu/data'
|
||||
})
|
||||
|
||||
// Asia Server
|
||||
const brain = new Brainy({
|
||||
storage: 's3://blog-asia/data'
|
||||
})
|
||||
// All automatically coordinate!
|
||||
```
|
||||
|
||||
### Example 2: E-Commerce Site
|
||||
|
||||
```typescript
|
||||
// Development
|
||||
const brain = new Brainy() // Memory storage
|
||||
|
||||
// Staging (Kubernetes)
|
||||
const brain = new Brainy({
|
||||
storage: process.env.STORAGE_PATH // Uses PVC
|
||||
})
|
||||
// Auto-discovers other pods via K8s DNS
|
||||
|
||||
// Production (AWS)
|
||||
const brain = new Brainy({
|
||||
storage: 's3://shop-data',
|
||||
cache: 'elasticache://shop-cache' // Optional
|
||||
})
|
||||
// Auto-scales with ECS/EKS
|
||||
```
|
||||
|
||||
### Example 3: Analytics Platform
|
||||
|
||||
```typescript
|
||||
// Ingestion nodes (write-optimized)
|
||||
const brain = new Brainy({
|
||||
role: 'writer', // Hint for optimization
|
||||
storage: '/fast-nvme/ingest'
|
||||
})
|
||||
|
||||
// Query nodes (read-optimized)
|
||||
const brain = new Brainy({
|
||||
role: 'reader', // More cache, indexes
|
||||
storage: 's3://analytics-archive'
|
||||
})
|
||||
|
||||
// Automatically coordinates between writers and readers!
|
||||
```
|
||||
|
||||
## 🔧 Storage Adapter Specifics
|
||||
|
||||
### Local Filesystem
|
||||
```typescript
|
||||
{
|
||||
storage: './data' // or absolute: '/var/lib/brainy'
|
||||
// Each node MUST have separate directory
|
||||
// Can be network mounted (NFS, EFS)
|
||||
}
|
||||
```
|
||||
|
||||
### AWS S3
|
||||
```typescript
|
||||
{
|
||||
storage: 's3://bucket-name/prefix'
|
||||
// Uses AWS SDK credentials (env, IAM role, etc)
|
||||
// Supports S3-compatible (MinIO, Ceph)
|
||||
}
|
||||
```
|
||||
|
||||
### Cloudflare R2
|
||||
```typescript
|
||||
{
|
||||
storage: 'r2://bucket-name'
|
||||
// Uses Wrangler or API tokens
|
||||
// Zero egress fees!
|
||||
}
|
||||
```
|
||||
|
||||
### Google Cloud Storage
|
||||
```typescript
|
||||
{
|
||||
storage: 'gs://bucket-name'
|
||||
// Uses Application Default Credentials
|
||||
}
|
||||
```
|
||||
|
||||
### Azure Blob Storage
|
||||
```typescript
|
||||
{
|
||||
storage: 'azure://container-name'
|
||||
// Uses DefaultAzureCredential
|
||||
}
|
||||
```
|
||||
|
||||
### Mixed/Tiered
|
||||
```typescript
|
||||
{
|
||||
storage: {
|
||||
hot: './local-cache', // Fast SSD
|
||||
warm: 's3://regular-data', // Standard storage
|
||||
cold: 's3://glacier-archive' // Cheap archive
|
||||
}
|
||||
// Automatic tiering based on access patterns
|
||||
}
|
||||
```
|
||||
|
||||
## 🎭 Advanced Patterns
|
||||
|
||||
### Pattern: Blue-Green Deployment
|
||||
```typescript
|
||||
// Blue cluster (current)
|
||||
const brain = new Brainy({
|
||||
cluster: 'blue',
|
||||
storage: 's3://prod-blue'
|
||||
})
|
||||
|
||||
// Green cluster (new version)
|
||||
const brain = new Brainy({
|
||||
cluster: 'green',
|
||||
storage: 's3://prod-green',
|
||||
syncFrom: 'blue' // Real-time sync during migration
|
||||
})
|
||||
```
|
||||
|
||||
### Pattern: Federation
|
||||
```typescript
|
||||
// Region 1 Cluster
|
||||
const brain1 = new Brainy({
|
||||
federation: 'global',
|
||||
region: 'us-east',
|
||||
storage: 's3://us-east-data'
|
||||
})
|
||||
|
||||
// Region 2 Cluster
|
||||
const brain2 = new Brainy({
|
||||
federation: 'global',
|
||||
region: 'eu-west',
|
||||
storage: 's3://eu-west-data'
|
||||
})
|
||||
// Clusters coordinate for global queries!
|
||||
```
|
||||
|
||||
### Pattern: Edge Computing
|
||||
```typescript
|
||||
// Edge nodes (in CDN POPs)
|
||||
const brain = new Brainy({
|
||||
mode: 'edge',
|
||||
storage: 'memory', // RAM only
|
||||
upstream: 'https://main-cluster.example.com'
|
||||
})
|
||||
// Caches frequently accessed data at edge
|
||||
```
|
||||
|
||||
## 📊 Monitoring & Observability
|
||||
|
||||
Brainy automatically exposes metrics:
|
||||
|
||||
```typescript
|
||||
const metrics = brain.getMetrics()
|
||||
// {
|
||||
// nodes: { total: 5, healthy: 5 },
|
||||
// shards: { total: 20, local: 4 },
|
||||
// replication: { factor: 2, lag: 45 },
|
||||
// operations: { reads: 10000, writes: 1000 },
|
||||
// storage: { used: '45GB', available: '955GB' }
|
||||
// }
|
||||
```
|
||||
|
||||
## 🚨 Troubleshooting
|
||||
|
||||
### Issue: Nodes don't discover each other
|
||||
```typescript
|
||||
// Solution 1: Check network allows UDP 7946
|
||||
// Solution 2: Use explicit peers
|
||||
const brain = new Brainy({
|
||||
peers: ['10.0.0.1:7946', '10.0.0.2:7946']
|
||||
})
|
||||
```
|
||||
|
||||
### Issue: Storage conflicts
|
||||
```typescript
|
||||
// Ensure each node has unique storage path
|
||||
// ❌ WRONG: All nodes use './data'
|
||||
// ✅ RIGHT: Node1: './data1', Node2: './data2'
|
||||
// ✅ RIGHT: Use {{nodeId}} template
|
||||
```
|
||||
|
||||
### Issue: Slow performance
|
||||
```typescript
|
||||
// Brainy auto-tunes, but you can hint:
|
||||
const brain = new Brainy({
|
||||
profile: 'read-heavy' // or 'write-heavy', 'balanced'
|
||||
})
|
||||
```
|
||||
|
||||
## 🎯 Best Practices
|
||||
|
||||
1. **Let Brainy Auto-Configure**: Don't over-configure
|
||||
2. **Separate Storage Per Node**: Avoids conflicts
|
||||
3. **Use S3 for Large Scale**: Infinite capacity
|
||||
4. **Start Simple**: Single node → Scale when needed
|
||||
5. **Monitor Metrics**: Watch for bottlenecks
|
||||
6. **Trust Auto-Scaling**: It learns your patterns
|
||||
|
||||
## 🚀 Summary
|
||||
|
||||
- **Zero Config**: Just `new Brainy()` at any scale
|
||||
- **Auto-Discovery**: Nodes find each other
|
||||
- **Smart Storage**: Adapts to any backend
|
||||
- **Progressive Scaling**: 1 → 100 nodes seamlessly
|
||||
- **Self-Tuning**: Learns and optimizes
|
||||
- **No DevOps**: It just works!
|
||||
|
||||
**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!**
|
||||
|
||||
---
|
||||
|
||||
*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)*
|
||||
550
docs/architecture/distributed-storage.md
Normal file
550
docs/architecture/distributed-storage.md
Normal file
|
|
@ -0,0 +1,550 @@
|
|||
# 🏗️ Distributed Storage Architecture
|
||||
|
||||
> **Technical deep-dive**: How Brainy coordinates storage across multiple nodes and adapters
|
||||
|
||||
## Storage Adapter Layer
|
||||
|
||||
### Base Storage Interface
|
||||
|
||||
Every storage adapter implements this interface:
|
||||
|
||||
```typescript
|
||||
interface StorageAdapter {
|
||||
// Basic operations
|
||||
get(key: string): Promise<any>
|
||||
set(key: string, value: any): Promise<void>
|
||||
delete(key: string): Promise<void>
|
||||
|
||||
// Batch operations
|
||||
getBatch(keys: string[]): Promise<Map<string, any>>
|
||||
setBatch(items: Map<string, any>): Promise<void>
|
||||
|
||||
// Atomic operations (for coordination)
|
||||
compareAndSwap(key: string, oldVal: any, newVal: any): Promise<boolean>
|
||||
increment(key: string, delta: number): Promise<number>
|
||||
|
||||
// Namespace support
|
||||
withNamespace(namespace: string): StorageAdapter
|
||||
}
|
||||
```
|
||||
|
||||
### Storage Coordination Strategies
|
||||
|
||||
#### Strategy 1: Isolated Storage (Default)
|
||||
|
||||
Each node has completely separate storage:
|
||||
|
||||
```
|
||||
Node-1 → Local FS: /data/node1/
|
||||
└── shards/
|
||||
├── shard-001/
|
||||
├── shard-045/
|
||||
└── shard-127/
|
||||
|
||||
Node-2 → Local FS: /data/node2/
|
||||
└── shards/
|
||||
├── shard-023/
|
||||
├── shard-067/
|
||||
└── shard-089/
|
||||
```
|
||||
|
||||
**Coordination**: Via network messages only
|
||||
- Shard ownership tracked in distributed consensus
|
||||
- Data transfer via direct node-to-node communication
|
||||
- No storage-level conflicts possible
|
||||
|
||||
#### Strategy 2: Shared Storage with Namespacing
|
||||
|
||||
Multiple nodes share storage but use namespaces:
|
||||
|
||||
```
|
||||
S3 Bucket: brainy-cluster/
|
||||
├── node-abc123/
|
||||
│ ├── shards/
|
||||
│ └── wal/
|
||||
├── node-def456/
|
||||
│ ├── shards/
|
||||
│ └── wal/
|
||||
└── _cluster/
|
||||
├── topology.json
|
||||
├── shard-map.json
|
||||
└── elections/
|
||||
```
|
||||
|
||||
**Coordination**: Via storage-level atomic operations
|
||||
- Each node owns its namespace
|
||||
- Cluster metadata in shared `_cluster/` namespace
|
||||
- Atomic operations for leader election
|
||||
- Conditional writes prevent conflicts
|
||||
|
||||
#### Strategy 3: Shared Storage with Fine-Grained Locking
|
||||
|
||||
Advanced mode for full shared storage:
|
||||
|
||||
```
|
||||
S3 Bucket: brainy-shared/
|
||||
├── shards/
|
||||
│ ├── 001/
|
||||
│ │ ├── data.bin
|
||||
│ │ └── .lock (atomic)
|
||||
│ ├── 002/
|
||||
│ │ ├── data.bin
|
||||
│ │ └── .lock
|
||||
└── metadata/
|
||||
├── index/
|
||||
└── locks/
|
||||
```
|
||||
|
||||
**Coordination**: Via distributed locking
|
||||
- Shard-level locks using atomic operations
|
||||
- Lock acquisition via compare-and-swap
|
||||
- Automatic lock expiry (lease-based)
|
||||
- Deadlock detection and recovery
|
||||
|
||||
## Storage Adapter Implementations
|
||||
|
||||
### 1. Filesystem Adapter
|
||||
|
||||
```typescript
|
||||
class FilesystemAdapter implements StorageAdapter {
|
||||
constructor(private basePath: string) {}
|
||||
|
||||
async get(key: string) {
|
||||
const path = this.keyToPath(key)
|
||||
return fs.readFile(path, 'json')
|
||||
}
|
||||
|
||||
async compareAndSwap(key: string, oldVal: any, newVal: any) {
|
||||
// Use file locking for atomicity
|
||||
const lockfile = `${this.keyToPath(key)}.lock`
|
||||
await flock(lockfile, 'ex') // Exclusive lock
|
||||
try {
|
||||
const current = await this.get(key)
|
||||
if (deepEqual(current, oldVal)) {
|
||||
await this.set(key, newVal)
|
||||
return true
|
||||
}
|
||||
return false
|
||||
} finally {
|
||||
await funlock(lockfile)
|
||||
}
|
||||
}
|
||||
|
||||
withNamespace(ns: string) {
|
||||
return new FilesystemAdapter(path.join(this.basePath, ns))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. S3 Adapter
|
||||
|
||||
```typescript
|
||||
class S3Adapter implements StorageAdapter {
|
||||
constructor(
|
||||
private bucket: string,
|
||||
private prefix: string = ''
|
||||
) {}
|
||||
|
||||
async get(key: string) {
|
||||
const result = await s3.getObject({
|
||||
Bucket: this.bucket,
|
||||
Key: `${this.prefix}${key}`
|
||||
})
|
||||
return JSON.parse(result.Body)
|
||||
}
|
||||
|
||||
async compareAndSwap(key: string, oldVal: any, newVal: any) {
|
||||
// Use S3's conditional writes
|
||||
const fullKey = `${this.prefix}${key}`
|
||||
|
||||
// Get current version
|
||||
const head = await s3.headObject({
|
||||
Bucket: this.bucket,
|
||||
Key: fullKey
|
||||
})
|
||||
|
||||
// Conditional put with ETag
|
||||
try {
|
||||
await s3.putObject({
|
||||
Bucket: this.bucket,
|
||||
Key: fullKey,
|
||||
Body: JSON.stringify(newVal),
|
||||
IfMatch: head.ETag // Only succeeds if unchanged
|
||||
})
|
||||
return true
|
||||
} catch (err) {
|
||||
if (err.code === 'PreconditionFailed') {
|
||||
return false
|
||||
}
|
||||
throw err
|
||||
}
|
||||
}
|
||||
|
||||
withNamespace(ns: string) {
|
||||
const newPrefix = `${this.prefix}${ns}/`
|
||||
return new S3Adapter(this.bucket, newPrefix)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Cloudflare R2 Adapter
|
||||
|
||||
```typescript
|
||||
class R2Adapter implements StorageAdapter {
|
||||
// Similar to S3 but with R2-specific optimizations
|
||||
|
||||
async compareAndSwap(key: string, oldVal: any, newVal: any) {
|
||||
// R2 supports conditional headers
|
||||
const response = await fetch(`${this.endpoint}/${key}`, {
|
||||
method: 'PUT',
|
||||
body: JSON.stringify(newVal),
|
||||
headers: {
|
||||
'If-Match': await this.getETag(key)
|
||||
}
|
||||
})
|
||||
return response.ok
|
||||
}
|
||||
|
||||
// R2-specific: Use Workers for edge computing
|
||||
async getWithCache(key: string) {
|
||||
// Check Cloudflare edge cache first
|
||||
const cached = await caches.default.match(key)
|
||||
if (cached) return cached.json()
|
||||
|
||||
// Fallback to R2
|
||||
const value = await this.get(key)
|
||||
|
||||
// Cache at edge
|
||||
await caches.default.put(key, new Response(JSON.stringify(value)))
|
||||
|
||||
return value
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Distributed Coordination Patterns
|
||||
|
||||
### Pattern 1: Leader-Based Coordination
|
||||
|
||||
```typescript
|
||||
class LeaderCoordinator {
|
||||
async acquireShardOwnership(shardId: string) {
|
||||
if (!this.isLeader()) {
|
||||
// Only leader assigns shards
|
||||
return this.requestFromLeader('acquireShard', shardId)
|
||||
}
|
||||
|
||||
// Leader logic
|
||||
const shardMap = await this.storage.get('_cluster/shard-map')
|
||||
if (!shardMap[shardId].owner) {
|
||||
shardMap[shardId].owner = this.nodeId
|
||||
|
||||
// Atomic update
|
||||
const success = await this.storage.compareAndSwap(
|
||||
'_cluster/shard-map',
|
||||
shardMap,
|
||||
{ ...shardMap, [shardId]: { owner: this.nodeId } }
|
||||
)
|
||||
|
||||
if (success) {
|
||||
this.broadcast('shardAssigned', { shardId, owner: this.nodeId })
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: Consensus-Based Coordination
|
||||
|
||||
```typescript
|
||||
class ConsensusCoordinator {
|
||||
async acquireShardOwnership(shardId: string) {
|
||||
// Propose to all nodes
|
||||
const proposal = {
|
||||
type: 'ACQUIRE_SHARD',
|
||||
shardId,
|
||||
nodeId: this.nodeId,
|
||||
term: this.currentTerm
|
||||
}
|
||||
|
||||
// Raft consensus
|
||||
const votes = await this.gatherVotes(proposal)
|
||||
|
||||
if (votes.length > this.nodes.length / 2) {
|
||||
// Majority agreed
|
||||
await this.commitProposal(proposal)
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Storage-Native Coordination
|
||||
|
||||
```typescript
|
||||
class StorageNativeCoordinator {
|
||||
async acquireShardOwnership(shardId: string) {
|
||||
// Use storage adapter's native coordination
|
||||
const lockKey = `_locks/shard-${shardId}`
|
||||
const lease = {
|
||||
owner: this.nodeId,
|
||||
expires: Date.now() + 30000 // 30 second lease
|
||||
}
|
||||
|
||||
// Try to acquire lock atomically
|
||||
const acquired = await this.storage.compareAndSwap(
|
||||
lockKey,
|
||||
null, // Must not exist
|
||||
lease
|
||||
)
|
||||
|
||||
if (acquired) {
|
||||
// Start lease renewal
|
||||
this.startLeaseRenewal(lockKey, lease)
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
private startLeaseRenewal(key: string, lease: any) {
|
||||
setInterval(async () => {
|
||||
const renewed = await this.storage.compareAndSwap(
|
||||
key,
|
||||
lease,
|
||||
{ ...lease, expires: Date.now() + 30000 }
|
||||
)
|
||||
|
||||
if (!renewed) {
|
||||
// Lost lease
|
||||
this.handleLeaseLoss(key)
|
||||
}
|
||||
}, 10000) // Renew every 10s
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Multi-Storage Patterns
|
||||
|
||||
### Hybrid Storage (Hot/Cold)
|
||||
|
||||
```typescript
|
||||
class HybridStorageAdapter implements StorageAdapter {
|
||||
constructor(
|
||||
private hot: StorageAdapter, // Fast SSD
|
||||
private cold: StorageAdapter // Cheap S3
|
||||
) {}
|
||||
|
||||
async get(key: string) {
|
||||
// Try hot storage first
|
||||
const hotValue = await this.hot.get(key).catch(() => null)
|
||||
if (hotValue) {
|
||||
this.updateAccessTime(key)
|
||||
return hotValue
|
||||
}
|
||||
|
||||
// Fallback to cold storage
|
||||
const coldValue = await this.cold.get(key)
|
||||
|
||||
// Promote to hot storage if frequently accessed
|
||||
if (this.shouldPromote(key)) {
|
||||
await this.hot.set(key, coldValue)
|
||||
}
|
||||
|
||||
return coldValue
|
||||
}
|
||||
|
||||
async set(key: string, value: any) {
|
||||
// Write to hot storage
|
||||
await this.hot.set(key, value)
|
||||
|
||||
// Async write to cold storage
|
||||
setImmediate(() => {
|
||||
this.cold.set(key, value).catch(console.error)
|
||||
})
|
||||
}
|
||||
|
||||
// Background process to demote cold data
|
||||
async runTiering() {
|
||||
const hotKeys = await this.hot.listKeys()
|
||||
|
||||
for (const key of hotKeys) {
|
||||
const lastAccess = await this.getAccessTime(key)
|
||||
|
||||
if (Date.now() - lastAccess > 7 * 24 * 60 * 60 * 1000) {
|
||||
// Not accessed in 7 days, demote to cold
|
||||
await this.cold.set(key, await this.hot.get(key))
|
||||
await this.hot.delete(key)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Geo-Distributed Storage
|
||||
|
||||
```typescript
|
||||
class GeoDistributedAdapter implements StorageAdapter {
|
||||
constructor(
|
||||
private regions: Map<string, StorageAdapter>
|
||||
) {}
|
||||
|
||||
async get(key: string) {
|
||||
// Determine closest region
|
||||
const region = await this.getClosestRegion()
|
||||
|
||||
// Try local region first
|
||||
const localValue = await this.regions.get(region)
|
||||
.get(key)
|
||||
.catch(() => null)
|
||||
|
||||
if (localValue) return localValue
|
||||
|
||||
// Fallback to other regions
|
||||
for (const [name, adapter] of this.regions) {
|
||||
if (name !== region) {
|
||||
const value = await adapter.get(key).catch(() => null)
|
||||
if (value) {
|
||||
// Replicate to local region for next time
|
||||
this.regions.get(region).set(key, value)
|
||||
return value
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error('Key not found in any region')
|
||||
}
|
||||
|
||||
async set(key: string, value: any) {
|
||||
// Write to local region immediately
|
||||
const region = await this.getClosestRegion()
|
||||
await this.regions.get(region).set(key, value)
|
||||
|
||||
// Async replication to other regions
|
||||
for (const [name, adapter] of this.regions) {
|
||||
if (name !== region) {
|
||||
adapter.set(key, value).catch(console.error)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Storage Optimization Strategies
|
||||
|
||||
### 1. Write Batching
|
||||
|
||||
```typescript
|
||||
class BatchingAdapter implements StorageAdapter {
|
||||
private writeBatch = new Map()
|
||||
private batchTimer?: NodeJS.Timeout
|
||||
|
||||
async set(key: string, value: any) {
|
||||
this.writeBatch.set(key, value)
|
||||
|
||||
if (!this.batchTimer) {
|
||||
this.batchTimer = setTimeout(() => this.flush(), 100)
|
||||
}
|
||||
|
||||
if (this.writeBatch.size >= 1000) {
|
||||
await this.flush()
|
||||
}
|
||||
}
|
||||
|
||||
private async flush() {
|
||||
if (this.writeBatch.size === 0) return
|
||||
|
||||
const batch = new Map(this.writeBatch)
|
||||
this.writeBatch.clear()
|
||||
|
||||
await this.underlying.setBatch(batch)
|
||||
|
||||
if (this.batchTimer) {
|
||||
clearTimeout(this.batchTimer)
|
||||
this.batchTimer = undefined
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Read Caching
|
||||
|
||||
```typescript
|
||||
class CachingAdapter implements StorageAdapter {
|
||||
private cache = new LRU({ max: 10000 })
|
||||
|
||||
async get(key: string) {
|
||||
// Check cache
|
||||
if (this.cache.has(key)) {
|
||||
return this.cache.get(key)
|
||||
}
|
||||
|
||||
// Read from storage
|
||||
const value = await this.underlying.get(key)
|
||||
|
||||
// Cache for next time
|
||||
this.cache.set(key, value)
|
||||
|
||||
return value
|
||||
}
|
||||
|
||||
async set(key: string, value: any) {
|
||||
// Invalidate cache
|
||||
this.cache.delete(key)
|
||||
|
||||
// Write through
|
||||
await this.underlying.set(key, value)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Compression
|
||||
|
||||
```typescript
|
||||
class CompressingAdapter implements StorageAdapter {
|
||||
async set(key: string, value: any) {
|
||||
const json = JSON.stringify(value)
|
||||
|
||||
// Compress if beneficial
|
||||
if (json.length > 1024) {
|
||||
const compressed = await gzip(json)
|
||||
await this.underlying.set(key, {
|
||||
_compressed: true,
|
||||
data: compressed.toString('base64')
|
||||
})
|
||||
} else {
|
||||
await this.underlying.set(key, value)
|
||||
}
|
||||
}
|
||||
|
||||
async get(key: string) {
|
||||
const stored = await this.underlying.get(key)
|
||||
|
||||
if (stored._compressed) {
|
||||
const compressed = Buffer.from(stored.data, 'base64')
|
||||
const json = await gunzip(compressed)
|
||||
return JSON.parse(json)
|
||||
}
|
||||
|
||||
return stored
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
Brainy's storage layer is designed for:
|
||||
|
||||
1. **Flexibility**: Works with any storage backend
|
||||
2. **Coordination**: Multiple strategies for different needs
|
||||
3. **Performance**: Batching, caching, compression
|
||||
4. **Scalability**: From single file to geo-distributed
|
||||
5. **Simplicity**: Complexity hidden behind simple interface
|
||||
|
||||
The key insight: **Storage is just a plugin**. The intelligence is in the coordination layer above it!
|
||||
|
||||
---
|
||||
|
||||
*For user-facing documentation, see [SCALING.md](../SCALING.md)*
|
||||
Loading…
Add table
Add a link
Reference in a new issue