feat(docs): add S3 migration guide for optimized data transfer strategies

2025-08-03 18:01:30 -07:00 · 2025-08-03 18:01:30 -07:00 · e24a6eb527
commit e24a6eb527
parent 2cf04b239f
3 changed files with 449 additions and 0 deletions
--- a/docs/README.md
+++ b/docs/README.md
@ -32,6 +32,7 @@ Performance optimization and scaling strategies.
 - **[Distributed Search](optimization-guides/distributed-search.md)** - Parallel processing
 - **[Memory Optimization](optimization-guides/memory-optimization.md)** - Efficient memory usage
 - **[Storage Optimization](optimization-guides/storage-optimization.md)** - S3 and storage strategies
+- **[S3 Migration Guide](optimization-guides/s3-migration-guide.md)** - Migrating existing data with shared buckets

 ### 🔧 [API Reference](api-reference/)
 Complete API documentation and examples.
@ -103,6 +104,7 @@ Common issues and solutions.

 ## 🔄 Recently Updated

+- **[S3 Migration Guide](optimization-guides/s3-migration-guide.md)** - NEW: Complete migration guide for shared S3 buckets
 - **[Large-Scale Optimizations](optimization-guides/large-scale-optimizations.md)** - Complete rewrite with auto-configuration
 - **[Quick Start Guide](getting-started/quick-start.md)** - Updated with zero-config setup
 - **[API Reference](api-reference/)** - New auto-configuration APIs
--- a/docs/optimization-guides/README.md
+++ b/docs/optimization-guides/README.md
@ -67,6 +67,14 @@ Continuous performance learning and optimization.
 - Usage pattern recognition
 - Self-optimization algorithms

+### 🗄️ [S3 Migration Guide](s3-migration-guide.md)
+Complete guide for migrating existing data to optimized system.
+
+- Migration strategies for shared S3 buckets
+- Zero-downtime migration options
+- Namespace isolation techniques
+- Rollback and verification procedures
+
 ## 📊 Performance Impact Overview

 | Optimization | Performance Gain | Memory Reduction | Setup Complexity |
--- a/docs/optimization-guides/s3-migration-guide.md
+++ b/docs/optimization-guides/s3-migration-guide.md
@ -0,0 +1,439 @@
+# S3 Migration Guide
+
+Complete guide for migrating existing Brainy data to the new optimized system, with special considerations for shared S3 buckets.
+
+## 🎯 Overview
+
+When upgrading to Brainy v0.36.0+ with the new optimization system, you can migrate existing data without starting from scratch. This guide covers migration strategies, shared bucket considerations, and best practices.
+
+## ✅ Key Points
+
+- **No data loss** - All existing data is preserved and enhanced
+- **Automatic optimization** - System applies all optimizations during migration
+- **Backward compatible** - Can read data from any previous version
+- **Zero-downtime options** - Multiple strategies for production systems
+
+## 🔄 Migration Methods
+
+### Method 1: Backup & Restore (Recommended)
+
+The safest and most reliable migration approach.
+
+```typescript
+// Step 1: Export from existing Brainy instance
+const oldBrainy = new BrainyData({ /* existing config */ })
+const backupData = await oldBrainy.backup()
+
+// Step 2: Save backup (optional)
+import fs from 'fs'
+fs.writeFileSync('brainy-backup.json', JSON.stringify(backupData))
+
+// Step 3: Create new optimized instance
+import { createAutoBrainy } from '@soulcraft/brainy'
+const newBrainy = createAutoBrainy({
+  bucketName: 'my-optimized-vectors'
+})
+
+// Step 4: Restore data with optimizations
+const result = await newBrainy.restore(backupData, {
+  clearExisting: true
+})
+
+console.log(`Migrated ${result.nounsRestored} vectors successfully`)
+```
+
+**Benefits:**
+- ✅ Clean migration with no conflicts
+- ✅ Can test before switching production
+- ✅ Rollback option available
+- ✅ Works with any storage backend
+
+### Method 2: In-Place Upgrade
+
+Upgrade existing data without moving it.
+
+```typescript
+import { createAutoBrainy } from '@soulcraft/brainy'
+
+// Point to existing data location
+const brainy = createAutoBrainy({
+  bucketName: 'existing-bucket' // Same as old instance
+})
+
+// System automatically:
+// 1. Detects existing data format
+// 2. Rebuilds optimized HNSW index
+// 3. Applies semantic partitioning
+// 4. Enables all optimizations
+
+// First operation triggers optimization
+const results = await brainy.search([0.1, 0.2, 0.3], 10)
+```
+
+**Benefits:**
+- ✅ No data movement required
+- ✅ Immediate optimization benefits
+- ✅ Minimal downtime
+
+**Limitations:**
+- ⚠️ Requires exclusive access during initial optimization
+- ⚠️ No rollback without backup
+
+### Method 3: Sparse Data Import
+
+For migrating raw data without vectors.
+
+```typescript
+// Import data without vectors (they'll be regenerated)
+const sparseData = {
+  nouns: [
+    {
+      id: 'doc-1',
+      metadata: {
+        text: 'Machine learning algorithms',
+        noun: 'Thing',
+        category: 'technology'
+      }
+      // No vector field - will be auto-generated
+    }
+  ],
+  verbs: [],
+  version: '1.0.0'
+}
+
+const brainy = createAutoBrainy()
+const result = await brainy.importSparseData(sparseData)
+```
+
+**Benefits:**
+- ✅ Smaller backup files
+- ✅ Ensures vectors use latest embedding model
+- ✅ Good for data format migrations
+
+## 🗄️ Shared S3 Bucket Considerations
+
+When multiple Brainy instances share the same S3 bucket, special care is needed during migration.
+
+### Understanding Shared Bucket Architecture
+
+```
+shared-bucket/
+├── nouns/              # Vector data
+├── verbs/              # Relationships  
+├── metadata/           # Additional metadata
+├── index/              # HNSW index data
+├── statistics/         # System statistics
+├── change-log/         # Change tracking with instance IDs
+└── locks/              # Distributed locks for coordination
+```
+
+### Built-in Safety Features
+
+1. **Distributed Locking**
+   - Prevents concurrent modifications
+   - 30-second TTL with automatic cleanup
+   - Coordinates between multiple instances
+
+2. **Instance Tracking**
+   ```typescript
+   // Each modification includes instance identification
+   {
+     timestamp: 1234567890,
+     operation: 'add',
+     instanceId: 'process-123' // or 'browser'
+   }
+   ```
+
+3. **Conflict Detection**
+   - Change logs track all modifications
+   - Enables conflict resolution
+   - Maintains data consistency
+
+### ⚠️ Potential Issues with Shared Buckets
+
+| Issue | Impact | Solution |
+|-------|--------|----------|
+| **Namespace Collision** | Different apps overwrite data | Use unique prefixes |
+| **Index Corruption** | Multiple rebuilds conflict | Coordinate migrations |
+| **Lock Contention** | Performance degradation | Stagger instance updates |
+| **Statistics Conflicts** | Incorrect metrics | Use distributed locking |
+
+## 📋 Migration Strategies for Shared Buckets
+
+### Strategy 1: Dedicated Namespace (Recommended)
+
+Create isolation using bucket subfolders.
+
+```typescript
+// Each application uses its own namespace
+const brainy = createAutoBrainy({
+  bucketName: 'shared-bucket/app-name/v2',
+  region: 'us-east-1'
+})
+
+// Data structure becomes:
+// shared-bucket/
+//   app-name/
+//     v2/
+//       nouns/
+//       verbs/
+//       ...
+```
+
+### Strategy 2: Blue-Green Migration
+
+Run old and new systems in parallel.
+
+```typescript
+// Phase 1: Create new optimized instance in separate bucket
+const newBrainy = createAutoBrainy({
+  bucketName: 'optimized-vectors',
+  region: 'us-east-1'
+})
+
+// Phase 2: Sync data from old to new
+const backupData = await oldBrainy.backup()
+await newBrainy.restore(backupData)
+
+// Phase 3: Run both systems in parallel
+// - Old system: Handles writes
+// - New system: Handles reads (testing)
+
+// Phase 4: Switch traffic to new system
+// Phase 5: Decommission old system
+```
+
+### Strategy 3: Rolling Migration
+
+Gradually migrate instances with zero downtime.
+
+```typescript
+// Step 1: Set read-only mode on secondary instances
+const readOnlyInstances = instances.map(instance => {
+  instance.setReadOnly(true)
+  return instance
+})
+
+// Step 2: Migrate primary write instance
+const primaryBrainy = createAutoBrainy({
+  bucketName: 'shared-bucket',
+  learningEnabled: true
+})
+
+// Step 3: Gradually migrate read instances
+for (const instance of readOnlyInstances) {
+  const optimized = createAutoBrainy({
+    bucketName: 'shared-bucket',
+    readOnlyMode: true
+  })
+  // Replace old instance with optimized
+}
+```
+
+### Strategy 4: Coordinated In-Place Migration
+
+Migrate with careful coordination.
+
+```typescript
+async function coordinatedMigration() {
+  // Step 1: Announce maintenance window
+  console.log('Starting coordinated migration...')
+  
+  // Step 2: Stop all write operations
+  const instances = await getAllInstances()
+  instances.forEach(i => i.setReadOnly(true))
+  
+  // Step 3: Wait for in-flight operations
+  await new Promise(resolve => setTimeout(resolve, 5000))
+  
+  // Step 4: Perform migration
+  const brainy = createAutoBrainy({
+    bucketName: 'shared-bucket',
+    autoConfigureEnvironment: true,
+    learningEnabled: true
+  })
+  
+  // Step 5: Verify migration
+  const stats = await brainy.getStatistics()
+  console.log(`Migrated ${stats.nounCount} vectors`)
+  
+  // Step 6: Resume operations
+  instances.forEach(i => i.setReadOnly(false))
+}
+```
+
+## 🛡️ Best Practices
+
+### 1. Pre-Migration Checklist
+
+- [ ] **Backup existing data** using `backup()` method
+- [ ] **Test migration** with subset of data
+- [ ] **Monitor S3 costs** during migration (increased API calls)
+- [ ] **Plan maintenance window** if using shared bucket
+- [ ] **Verify credentials** for S3 access
+- [ ] **Check available storage** for temporary duplication
+
+### 2. Performance Monitoring
+
+```typescript
+// Monitor migration progress and performance
+const brainy = createAutoBrainy({ /* config */ })
+
+// Check optimization status
+setInterval(async () => {
+  const metrics = brainy.getPerformanceMetrics()
+  console.log({
+    vectorsProcessed: metrics.indexSize,
+    searchLatency: metrics.averageSearchTime,
+    cacheHitRate: metrics.cacheHitRate,
+    memoryUsage: metrics.memoryUsage
+  })
+}, 10000)
+```
+
+### 3. Handling Large Datasets
+
+For datasets over 1M vectors:
+
+```typescript
+// Use scenario-based configuration
+const brainy = await createQuickBrainy('enterprise', {
+  bucketName: 'large-vectors',
+  region: 'us-east-1'
+})
+
+// Or manual configuration for fine control
+const brainy = createAutoBrainy({
+  expectedDatasetSize: 5000000,
+  maxMemoryUsage: 16 * 1024 * 1024 * 1024, // 16GB
+  targetSearchLatency: 500,
+  s3Config: {
+    bucketName: 'large-vectors',
+    region: 'us-east-1'
+  }
+})
+```
+
+### 4. Rollback Plan
+
+Always maintain ability to rollback:
+
+```typescript
+// Before migration
+const backupData = await oldBrainy.backup()
+fs.writeFileSync('backup-{timestamp}.json', JSON.stringify(backupData))
+
+// If rollback needed
+const oldBrainy = new BrainyData({ /* old config */ })
+await oldBrainy.restore(backupData, { clearExisting: true })
+```
+
+## 📊 Expected Performance Improvements
+
+After migration with optimizations:
+
+| Dataset Size | Before | After | Improvement |
+|-------------|--------|-------|-------------|
+| **10k vectors** | ~200ms | ~50ms | **4x faster** |
+| **100k vectors** | ~2s | ~200ms | **10x faster** |
+| **1M vectors** | ~10s | ~500ms | **20x faster** |
+| **Memory Usage** | 100% | 25-30% | **70-75% reduction** |
+| **S3 API Calls** | 100% | 10-50% | **50-90% reduction** |
+
+## 🔍 Troubleshooting
+
+### Common Issues and Solutions
+
+| Problem | Cause | Solution |
+|---------|-------|----------|
+| **"Lock timeout" errors** | Multiple instances competing | Stagger migration timing |
+| **High memory usage** | Large dataset loading | Enable compression, reduce cache |
+| **Slow initial searches** | Index rebuilding | Wait for optimization to complete |
+| **S3 rate limiting** | Too many concurrent operations | Reduce batch sizes |
+| **Missing vectors** | Sparse data import | Ensure embedding function available |
+
+### Debug Logging
+
+Enable detailed logging during migration:
+
+```typescript
+// Set environment variable
+process.env.BRAINY_DEBUG = 'true'
+
+const brainy = createAutoBrainy({
+  bucketName: 'my-bucket',
+  // Logs will show optimization decisions
+})
+```
+
+### Verification Steps
+
+After migration, verify success:
+
+```typescript
+async function verifyMigration(brainy) {
+  // 1. Check data integrity
+  const stats = await brainy.getStatistics()
+  console.log(`Vectors: ${stats.nounCount}`)
+  console.log(`Relationships: ${stats.verbCount}`)
+  
+  // 2. Test search performance
+  const testVector = [0.1, 0.2, 0.3]
+  const start = Date.now()
+  const results = await brainy.search(testVector, 10)
+  console.log(`Search time: ${Date.now() - start}ms`)
+  
+  // 3. Verify optimizations active
+  const metrics = brainy.getPerformanceMetrics()
+  console.log(`Cache hit rate: ${metrics.cacheHitRate}`)
+  console.log(`Compression ratio: ${metrics.compressionRatio}`)
+  
+  return stats.nounCount > 0 && results.length > 0
+}
+```
+
+## 🎯 Quick Decision Guide
+
+Choose your migration approach:
+
+```mermaid
+graph TD
+    A[Start Migration] --> B{Shared S3 Bucket?}
+    B -->|Yes| C{Can Stop All Instances?}
+    B -->|No| D[Use Backup & Restore]
+    C -->|Yes| E[In-Place Migration]
+    C -->|No| F{Need Zero Downtime?}
+    F -->|Yes| G[Blue-Green Migration]
+    F -->|No| H[Rolling Migration]
+    D --> I[Complete]
+    E --> I
+    G --> I
+    H --> I
+```
+
+## 💡 Tips for Success
+
+1. **Start Small**: Test with 1% of data first
+2. **Monitor Metrics**: Use `getPerformanceMetrics()` frequently
+3. **Use Read-Only Mode**: For instances that only search
+4. **Plan S3 Costs**: Migration causes temporary spike in API calls
+5. **Keep Backups**: Always maintain rollback capability
+6. **Leverage Auto-Config**: Let system optimize automatically
+7. **Document Process**: Record settings for reproducibility
+
+## 🔗 Related Documentation
+
+- [Large-Scale Optimizations](./large-scale-optimizations.md) - Understanding the optimization system
+- [Auto-Configuration Guide](./auto-configuration.md) - How automatic optimization works
+- [Storage Optimization](./storage-optimization.md) - S3 and storage best practices
+- [Production Migration Guide](../guides/production-migration-guide.md) - General production deployment
+
+## 🆘 Getting Help
+
+- **GitHub Issues**: Report migration problems
+- **Discussions**: Share migration experiences
+- **Documentation**: Check other guides for specific features
+
+---
+
+**Ready to migrate?** Start with a backup, test thoroughly, and enjoy 10-20x performance improvements! 🚀