brainy/docs/planning/STORAGE_UNIFICATION_PLAN.md

Storage Unification Plan: Everything as Augmentations

Executive Summary

Unify storage adapters and memory augmentations into a single augmentation-based system while maintaining 100% backward compatibility and zero-config philosophy.

Current State Analysis

Two Parallel Systems

1. Storage Adapters (src/storage/adapters/)

   • Direct implementation of StorageAdapter interface
   • Selected via createStorage() during initialization
   • 67 direct calls to this.storage throughout BrainyData

2. Memory Augmentations (src/augmentations/memoryAugmentations.ts)

   • Wrap storage adapters as augmentations
   • Use timing: 'replace' for storage operations
   • Redundant with storage adapters

Initialization Order Problem

// Current flow in BrainyData.init()
1. Create/initialize storage (line 1463-1503)
2. Initialize augmentations with storage context (line 1508)
3. Storage passed to augmentations via context (line 782)

Problem: Augmentations need storage in context, but we want augmentations to provide storage!

Proposed Solution: Two-Phase Initialization

Phase 1: Pre-Registration (No Context)

// Early in init(), before storage creation
this.registerDefaultAugmentations() // Register but don't initialize

Phase 2: Storage Resolution

// Check for storage augmentations
const storageAug = this.augmentations.findByOperation('storage')

if (storageAug) {
  // Get storage from augmentation
  this.storage = await storageAug.provideStorage()
} else if (this.config.storageAdapter) {
  // Use provided adapter (backward compat)
  this.storage = this.config.storageAdapter
} else {
  // Zero-config: create and wrap in augmentation
  this.storage = await createStorage(this.storageConfig)
  
  // Auto-register as augmentation for consistency
  const autoAug = new DynamicStorageAugmentation(this.storage)
  this.augmentations.register(autoAug)
}

await this.storage.init()

Phase 3: Full Augmentation Initialization

// Now initialize all augmentations with context
const context = {
  brain: this,
  storage: this.storage,
  config: this.config,
  log: this.log
}

await this.augmentations.initializeAll(context)

Implementation Steps

Step 1: Create DynamicStorageAugmentation

// Wraps any storage adapter as an augmentation
class DynamicStorageAugmentation extends BaseAugmentation {
  constructor(private adapter: StorageAdapter) {
    super()
    this.name = `${adapter.constructor.name}Augmentation`
    this.timing = 'replace'
    this.operations = ['storage']
    this.priority = 100
  }
  
  async provideStorage(): Promise<StorageAdapter> {
    return this.adapter
  }
  
  async execute(op, params, next) {
    if (op === 'storage') {
      return this.adapter
    }
    return next()
  }
}

Step 2: Modify AugmentationRegistry

class AugmentationRegistry {
  // Add method to find augmentations before initialization
  findByOperation(operation: string): BrainyAugmentation | null {
    return this.augmentations.find(aug => 
      aug.operations.includes(operation) || 
      aug.operations.includes('all')
    ) || null
  }
  
  // Split registration from initialization
  register(augmentation: BrainyAugmentation): void {
    this.augmentations.push(augmentation)
    // Don't initialize yet
  }
  
  async initializeAll(context: AugmentationContext): Promise<void> {
    for (const aug of this.augmentations) {
      if (aug.initialize) {
        await aug.initialize(context)
      }
    }
  }
}

Step 3: Update BrainyData.init()

async init(): Promise<void> {
  // ... existing validation ...
  
  // Step 1: Register default augmentations (no init)
  this.registerDefaultAugmentations()
  
  // Step 2: Resolve storage
  await this.resolveStorage()
  
  // Step 3: Initialize augmentations with context
  await this.initializeAugmentations()
  
  // ... rest of init ...
}

private async resolveStorage(): Promise<void> {
  // Check for storage augmentation
  const storageAug = this.augmentations.findByOperation('storage')
  
  if (storageAug && storageAug.provideStorage) {
    // Get storage from augmentation
    this.storage = await storageAug.provideStorage()
  } else if (!this.storage) {
    // No storage augmentation and no provided adapter
    // Use zero-config
    const storageOptions = this.buildStorageOptions()
    this.storage = await createStorage(storageOptions)
    
    // Wrap in augmentation for consistency
    const wrapper = new DynamicStorageAugmentation(this.storage)
    this.augmentations.register(wrapper)
  }
  
  // Initialize storage
  await this.storage!.init()
}

Usage Examples

Zero-Config (No Change)

const brain = new BrainyData()
await brain.init()
// Automatically selects best storage for environment

Explicit Storage Adapter (Backward Compatible)

const brain = new BrainyData({
  storageAdapter: new S3Storage(config)
})
await brain.init()

Storage via Augmentation (New)

const brain = new BrainyData()
brain.augmentations.register(new S3StorageAugmentation(config))
await brain.init()

Storage Config (Backward Compatible)

const brain = new BrainyData({
  storage: {
    s3Storage: {
      bucketName: 'my-bucket',
      accessKeyId: 'xxx',
      secretAccessKey: 'yyy'
    }
  }
})
await brain.init()

Benefits

1. Unified Architecture: Everything is an augmentation
2. Backward Compatible: All existing code continues to work
3. Zero-Config Maintained: Intelligent selection still works
4. Extensible: Easy to add new storage types as augmentations
5. Middleware Capable: Storage operations can be intercepted
6. Premium Ready: Premium storage augmentations can be added to marketplace

Migration Path

Phase 1: Implement Infrastructure (No Breaking Changes)

• Add DynamicStorageAugmentation
• Update AugmentationRegistry with new methods
• Modify BrainyData.init() to support both paths

Phase 2: Deprecate Direct Storage Config

• Mark storageAdapter config as deprecated
• Encourage augmentation approach in docs
• Keep working for 2-3 major versions

Phase 3: Remove Legacy Code

• Remove storageAdapter from config
• Remove createStorage() direct calls
• All storage through augmentations

Testing Strategy

1. Backward Compatibility Tests

   • Ensure all existing storage config methods work
   • Test zero-config in different environments
   • Verify no breaking changes

2. New Functionality Tests

   • Test storage augmentation registration
   • Test override behavior
   • Test middleware capabilities

3. Performance Tests

   • Ensure no performance regression
   • Measure augmentation overhead

Risk Mitigation

1. Risk: Circular dependency between storage and augmentations Mitigation: Two-phase initialization breaks the cycle

2. Risk: Breaking existing code Mitigation: Keep this.storage and all direct calls unchanged

3. Risk: Performance overhead Mitigation: Storage augmentation is registered once, minimal overhead

4. Risk: Confusion about which approach to use Mitigation: Clear documentation, deprecation warnings, migration guide

Timeline

• Week 1: Implement core infrastructure
• Week 2: Update documentation and examples
• Week 3: Testing and optimization
• Week 4: Release as minor version (non-breaking)

Conclusion

This unification maintains all existing behaviors while providing a cleaner, more extensible architecture. The augmentation approach aligns with Brainy's philosophy and enables future enhancements without breaking changes.