brainy/docs/optimization-guides/s3-migration-guide.md

S3 Migration Guide

Complete guide for migrating existing Brainy data to the new optimized system, with special considerations for shared S3 buckets.

🎯 Overview

When upgrading to Brainy v0.36.0+ with the new optimization system, you can migrate existing data without starting from scratch. This guide covers migration strategies, shared bucket considerations, and best practices.

✅ Key Points

• No data loss - All existing data is preserved and enhanced
• Automatic optimization - System applies all optimizations during migration
• Backward compatible - Can read data from any previous version
• Zero-downtime options - Multiple strategies for production systems

🔄 Migration Methods

Method 1: Backup & Restore (Recommended)

The safest and most reliable migration approach.

// Step 1: Export from existing Brainy instance
const oldBrainy = new BrainyData({ /* existing config */ })
const backupData = await oldBrainy.backup()

// Step 2: Save backup (optional)
import fs from 'fs'
fs.writeFileSync('brainy-backup.json', JSON.stringify(backupData))

// Step 3: Create new optimized instance
import { createAutoBrainy } from '@soulcraft/brainy'
const newBrainy = createAutoBrainy({
  bucketName: 'my-optimized-vectors'
})

// Step 4: Restore data with optimizations
const result = await newBrainy.restore(backupData, {
  clearExisting: true
})

console.log(`Migrated ${result.nounsRestored} vectors successfully`)

Benefits:

• ✅ Clean migration with no conflicts
• ✅ Can test before switching production
• ✅ Rollback option available
• ✅ Works with any storage backend

Method 2: In-Place Upgrade

Upgrade existing data without moving it.

import { createAutoBrainy } from '@soulcraft/brainy'

// Point to existing data location
const brainy = createAutoBrainy({
  bucketName: 'existing-bucket' // Same as old instance
})

// System automatically:
// 1. Detects existing data format
// 2. Rebuilds optimized HNSW index
// 3. Applies semantic partitioning
// 4. Enables all optimizations

// First operation triggers optimization
const results = await brainy.search([0.1, 0.2, 0.3], 10)

Benefits:

• ✅ No data movement required
• ✅ Immediate optimization benefits
• ✅ Minimal downtime

Limitations:

• ⚠️ Requires exclusive access during initial optimization
• ⚠️ No rollback without backup

Method 3: Sparse Data Import

For migrating raw data without vectors.

// Import data without vectors (they'll be regenerated)
const sparseData = {
  nouns: [
    {
      id: 'doc-1',
      metadata: {
        text: 'Machine learning algorithms',
        noun: 'Thing',
        category: 'technology'
      }
      // No vector field - will be auto-generated
    }
  ],
  verbs: [],
  version: '1.0.0'
}

const brainy = createAutoBrainy()
const result = await brainy.importSparseData(sparseData)

Benefits:

• ✅ Smaller backup files
• ✅ Ensures vectors use latest embedding model
• ✅ Good for data format migrations

🗄️ Shared S3 Bucket Considerations

When multiple Brainy instances share the same S3 bucket, special care is needed during migration.

Understanding Shared Bucket Architecture

shared-bucket/
├── nouns/              # Vector data
├── verbs/              # Relationships  
├── metadata/           # Additional metadata
├── index/              # HNSW index data
├── statistics/         # System statistics
├── change-log/         # Change tracking with instance IDs
└── locks/              # Distributed locks for coordination

Built-in Safety Features

1. Distributed Locking

   • Prevents concurrent modifications
   • 30-second TTL with automatic cleanup
   • Coordinates between multiple instances

2. Instance Tracking

   // Each modification includes instance identification
   {
     timestamp: 1234567890,
     operation: 'add',
     instanceId: 'process-123' // or 'browser'
   }
   

3. Conflict Detection

   • Change logs track all modifications
   • Enables conflict resolution
   • Maintains data consistency

⚠️ Potential Issues with Shared Buckets

Issue	Impact	Solution
Namespace Collision	Different apps overwrite data	Use unique prefixes
Index Corruption	Multiple rebuilds conflict	Coordinate migrations
Lock Contention	Performance degradation	Stagger instance updates
Statistics Conflicts	Incorrect metrics	Use distributed locking

📋 Migration Strategies for Shared Buckets

Strategy 1: Dedicated Namespace (Recommended)

Create isolation using bucket subfolders.

// Each application uses its own namespace
const brainy = createAutoBrainy({
  bucketName: 'shared-bucket/app-name/v2',
  region: 'us-east-1'
})

// Data structure becomes:
// shared-bucket/
//   app-name/
//     v2/
//       nouns/
//       verbs/
//       ...

Strategy 2: Blue-Green Migration

Run old and new systems in parallel.

// Phase 1: Create new optimized instance in separate bucket
const newBrainy = createAutoBrainy({
  bucketName: 'optimized-vectors',
  region: 'us-east-1'
})

// Phase 2: Sync data from old to new
const backupData = await oldBrainy.backup()
await newBrainy.restore(backupData)

// Phase 3: Run both systems in parallel
// - Old system: Handles writes
// - New system: Handles reads (testing)

// Phase 4: Switch traffic to new system
// Phase 5: Decommission old system

Strategy 3: Rolling Migration

Gradually migrate instances with zero downtime.

// Step 1: Set read-only mode on secondary instances
const readOnlyInstances = instances.map(instance => {
  instance.setReadOnly(true)
  return instance
})

// Step 2: Migrate primary write instance
const primaryBrainy = createAutoBrainy({
  bucketName: 'shared-bucket',
  learningEnabled: true
})

// Step 3: Gradually migrate read instances
for (const instance of readOnlyInstances) {
  const optimized = createAutoBrainy({
    bucketName: 'shared-bucket',
    readOnlyMode: true
  })
  // Replace old instance with optimized
}

Strategy 4: Coordinated In-Place Migration

Migrate with careful coordination.

async function coordinatedMigration() {
  // Step 1: Announce maintenance window
  console.log('Starting coordinated migration...')
  
  // Step 2: Stop all write operations
  const instances = await getAllInstances()
  instances.forEach(i => i.setReadOnly(true))
  
  // Step 3: Wait for in-flight operations
  await new Promise(resolve => setTimeout(resolve, 5000))
  
  // Step 4: Perform migration
  const brainy = createAutoBrainy({
    bucketName: 'shared-bucket',
    autoConfigureEnvironment: true,
    learningEnabled: true
  })
  
  // Step 5: Verify migration
  const stats = await brainy.getStatistics()
  console.log(`Migrated ${stats.nounCount} vectors`)
  
  // Step 6: Resume operations
  instances.forEach(i => i.setReadOnly(false))
}

🛡️ Best Practices

1. Pre-Migration Checklist

• Backup existing data using backup() method
• Test migration with subset of data
• Monitor S3 costs during migration (increased API calls)
• Plan maintenance window if using shared bucket
• Verify credentials for S3 access
• Check available storage for temporary duplication

2. Performance Monitoring

// Monitor migration progress and performance
const brainy = createAutoBrainy({ /* config */ })

// Check optimization status
setInterval(async () => {
  const metrics = brainy.getPerformanceMetrics()
  console.log({
    vectorsProcessed: metrics.indexSize,
    searchLatency: metrics.averageSearchTime,
    cacheHitRate: metrics.cacheHitRate,
    memoryUsage: metrics.memoryUsage
  })
}, 10000)

3. Handling Large Datasets

For datasets over 1M vectors:

// Use scenario-based configuration
const brainy = await createQuickBrainy('enterprise', {
  bucketName: 'large-vectors',
  region: 'us-east-1'
})

// Or manual configuration for fine control
const brainy = createAutoBrainy({
  expectedDatasetSize: 5000000,
  maxMemoryUsage: 16 * 1024 * 1024 * 1024, // 16GB
  targetSearchLatency: 500,
  s3Config: {
    bucketName: 'large-vectors',
    region: 'us-east-1'
  }
})

4. Rollback Plan

Always maintain ability to rollback:

// Before migration
const backupData = await oldBrainy.backup()
fs.writeFileSync('backup-{timestamp}.json', JSON.stringify(backupData))

// If rollback needed
const oldBrainy = new BrainyData({ /* old config */ })
await oldBrainy.restore(backupData, { clearExisting: true })

📊 Expected Performance Improvements

After migration with optimizations:

Dataset Size	Before	After	Improvement
10k vectors	~200ms	~50ms	4x faster
100k vectors	~2s	~200ms	10x faster
1M vectors	~10s	~500ms	20x faster
Memory Usage	100%	25-30%	70-75% reduction
S3 API Calls	100%	10-50%	50-90% reduction

🔍 Troubleshooting

Common Issues and Solutions

Problem	Cause	Solution
"Lock timeout" errors	Multiple instances competing	Stagger migration timing
High memory usage	Large dataset loading	Enable compression, reduce cache
Slow initial searches	Index rebuilding	Wait for optimization to complete
S3 rate limiting	Too many concurrent operations	Reduce batch sizes
Missing vectors	Sparse data import	Ensure embedding function available

Debug Logging

Enable detailed logging during migration:

// Set environment variable
process.env.BRAINY_DEBUG = 'true'

const brainy = createAutoBrainy({
  bucketName: 'my-bucket',
  // Logs will show optimization decisions
})

Verification Steps

After migration, verify success:

async function verifyMigration(brainy) {
  // 1. Check data integrity
  const stats = await brainy.getStatistics()
  console.log(`Vectors: ${stats.nounCount}`)
  console.log(`Relationships: ${stats.verbCount}`)
  
  // 2. Test search performance
  const testVector = [0.1, 0.2, 0.3]
  const start = Date.now()
  const results = await brainy.search(testVector, 10)
  console.log(`Search time: ${Date.now() - start}ms`)
  
  // 3. Verify optimizations active
  const metrics = brainy.getPerformanceMetrics()
  console.log(`Cache hit rate: ${metrics.cacheHitRate}`)
  console.log(`Compression ratio: ${metrics.compressionRatio}`)
  
  return stats.nounCount > 0 && results.length > 0
}

🎯 Quick Decision Guide

Choose your migration approach:

graph TD
    A[Start Migration] --> B{Shared S3 Bucket?}
    B -->|Yes| C{Can Stop All Instances?}
    B -->|No| D[Use Backup & Restore]
    C -->|Yes| E[In-Place Migration]
    C -->|No| F{Need Zero Downtime?}
    F -->|Yes| G[Blue-Green Migration]
    F -->|No| H[Rolling Migration]
    D --> I[Complete]
    E --> I
    G --> I
    H --> I

💡 Tips for Success

1. Start Small: Test with 1% of data first
2. Monitor Metrics: Use getPerformanceMetrics() frequently
3. Use Read-Only Mode: For instances that only search
4. Plan S3 Costs: Migration causes temporary spike in API calls
5. Keep Backups: Always maintain rollback capability
6. Leverage Auto-Config: Let system optimize automatically
7. Document Process: Record settings for reproducibility

🔗 Related Documentation

• Large-Scale Optimizations - Understanding the optimization system
• Auto-Configuration Guide - How automatic optimization works
• Storage Optimization - S3 and storage best practices
• Production Migration Guide - General production deployment

🆘 Getting Help

• GitHub Issues: Report migration problems
• Discussions: Share migration experiences
• Documentation: Check other guides for specific features

---

Ready to migrate? Start with a backup, test thoroughly, and enjoy 10-20x performance improvements! 🚀