brainy/docs/STORAGE_MIGRATION_GUIDE.md

Storage Migration Guide: index → _system

Overview

Brainy is migrating its system metadata storage from the index/ directory to _system/ directory to better reflect its purpose and follow database conventions. This migration is designed to be zero-downtime and backward compatible.

Migration Strategy

Dual Read/Write Approach

The migration uses a dual-read, migrate-on-write strategy to ensure compatibility between services running different versions:

1. Read Priority: Try new location first (_system/), fallback to old (index/)
2. Dual Write: During migration, write to both locations
3. Automatic Migration: When data is found only in old location, it's automatically copied to new
4. Gradual Rollout: Services can be updated independently without coordination

Timeline

Phase 1: Dual Mode (Current)

• Duration: 30 days (configurable via BRAINY_MIGRATION_GRACE_DAYS)
• Services write to both _system/ and index/
• Services read from both locations (new first, then old)
• Automatic migration on first read from old location

Phase 2: New Primary (After Grace Period)

• Services primarily use _system/
• Legacy index/ kept for emergency rollback
• Monitoring for any services still using old location

Phase 3: Cleanup (After Verification)

• Remove dual-write code
• Archive or delete index/ directory
• Update documentation

Deployment Guide

For S3/Cloud Storage (Most Critical)

Step 1: Rolling Update

# Update services one by one - no coordination needed
kubectl rollout restart deployment/brainy-service-1
# Wait for health checks
kubectl rollout restart deployment/brainy-service-2
# Continue for all services

Step 2: Monitor Migration

# Check for migration events in logs
kubectl logs -l app=brainy --since=1h | grep "Storage Migration"

# Verify both directories have data
aws s3 ls s3://your-bucket/_system/
aws s3 ls s3://your-bucket/index/

Step 3: Verify Consistency

// Check that statistics match between locations
const oldStats = await storage.getMetadata('statistics'); // from index/
const newStats = await storage.getStatistics(); // from _system/
console.assert(oldStats.nounCount === newStats.nounCount);

For Local/FileSystem Storage

The migration happens automatically on first run:

# Before update
brainy-data/
├── index/          # Old location
│   └── statistics.json
└── ...

# After update (automatic)
brainy-data/
├── index/          # Kept for compatibility
│   └── statistics.json
├── _system/        # New location
│   └── statistics.json
└── ...

Configuration Options

Environment Variables

# Control migration grace period (default: 30 days)
export BRAINY_MIGRATION_GRACE_DAYS=30

# Force single-write mode (after migration confirmed)
export BRAINY_DISABLE_DUAL_WRITE=true

# Enable verbose migration logging
export BRAINY_MIGRATION_DEBUG=true

Per-Instance Configuration

const storage = new FileSystemStorage({
  rootDirectory: './data',
  useDualWrite: true  // Set to false after migration
});

Monitoring

Key Metrics to Watch

1. Storage Operations

   • Successful reads from _system/
   • Fallback reads from index/
   • Dual write operations
   • Migration events

2. Performance Impact

   • Minimal: ~5-10ms additional latency during dual-write
   • No impact on read performance (cache used)

3. Log Messages

   [Brainy Storage Migration] Migrating statistics from legacy location
   [Brainy Storage Migration] Failed to write to legacy location (non-critical)
   [Brainy Storage Migration] Migration completed successfully
   

Rollback Plan

If issues occur, rollback is simple:

1. Immediate Rollback: Revert service to previous version

   • Old versions continue using index/
   • New versions read from both locations

2. Data Recovery: If data corruption occurs

   # Copy data back from index to _system
   aws s3 sync s3://bucket/index/ s3://bucket/_system/
   

FAQ

Q: What happens if services are on different versions?

A: The dual-read/write strategy ensures compatibility. Newer services write to both locations and read from both, while older services continue using only the old location.

Q: Is there data duplication?

A: Yes, temporarily. During the migration period, data exists in both locations. This ensures zero-downtime migration and provides a safety net.

Q: What about distributed configurations?

A: The distributed configuration (previously in metadata) now lives with statistics in _system/, making it easier to find and manage.

Q: Can I disable dual-write immediately?

A: Not recommended. Keep dual-write enabled for at least 7 days to ensure all services are updated and caches are refreshed.

Q: What if a service can't write to the new location?

A: The service will log a warning but continue operating using the fallback location. This ensures service availability over migration perfection.

Testing the Migration

Unit Test Example

describe('Storage Migration', () => {
  it('should read from both locations', async () => {
    // Write to old location
    await fs.writeFile('data/index/statistics.json', oldData);
    
    // Initialize storage (triggers migration)
    const storage = new FileSystemStorage({ rootDir: 'data' });
    
    // Should read and migrate
    const stats = await storage.getStatistics();
    expect(stats).toBeDefined();
    
    // Should now exist in both locations
    expect(fs.existsSync('data/_system/statistics.json')).toBe(true);
    expect(fs.existsSync('data/index/statistics.json')).toBe(true);
  });
});

Integration Test

# Start with old version
docker run -v data:/data brainy:old-version

# Create some data
curl -X POST localhost:3000/api/data

# Upgrade to new version
docker run -v data:/data brainy:new-version

# Verify data is accessible and migrated
curl localhost:3000/api/stats | jq '.migrationMetadata'

Support

For issues or questions about the migration:

1. Check logs for migration-related messages
2. Verify both directories exist and are accessible
3. Ensure proper permissions for creating _system/ directory
4. Contact support with migration logs if issues persist

Summary

This migration is designed to be:

• Safe: Dual-read/write prevents data loss
• Gradual: No big-bang migration required
• Automatic: Minimal manual intervention
• Reversible: Easy rollback if needed
• Transparent: Services continue operating normally

The key is patience - let the migration happen gradually across your fleet rather than forcing immediate updates.