ORGANIZE: Move documentation to proper directories

- Moved API design docs to docs/api-design/ - Moved planning docs to docs/planning/ - Root now only contains standard repo files (README, LICENSE, etc.) - Keeps CLAUDE.md and PLAN.md uncommitted for privacy Clean root directory for better project organization.
2025-08-25 09:53:41 -07:00 · 2025-08-25 09:53:41 -07:00 · 2c4b34e9fb
commit 2c4b34e9fb
parent 26c7d61185
19 changed files with 0 additions and 0 deletions
--- a/docs/planning/STORAGE_UNIFICATION_PLAN.md
+++ b/docs/planning/STORAGE_UNIFICATION_PLAN.md
@ -0,0 +1,270 @@
+# Storage Unification Plan: Everything as Augmentations
+
+## Executive Summary
+Unify storage adapters and memory augmentations into a single augmentation-based system while maintaining 100% backward compatibility and zero-config philosophy.
+
+## Current State Analysis
+
+### Two Parallel Systems
+1. **Storage Adapters** (`src/storage/adapters/`)
+   - Direct implementation of StorageAdapter interface
+   - Selected via `createStorage()` during initialization
+   - 67 direct calls to `this.storage` throughout BrainyData
+
+2. **Memory Augmentations** (`src/augmentations/memoryAugmentations.ts`)
+   - Wrap storage adapters as augmentations
+   - Use `timing: 'replace'` for storage operations
+   - Redundant with storage adapters
+
+### Initialization Order Problem
+```typescript
+// Current flow in BrainyData.init()
+1. Create/initialize storage (line 1463-1503)
+2. Initialize augmentations with storage context (line 1508)
+3. Storage passed to augmentations via context (line 782)
+```
+
+**Problem:** Augmentations need storage in context, but we want augmentations to provide storage!
+
+## Proposed Solution: Two-Phase Initialization
+
+### Phase 1: Pre-Registration (No Context)
+```typescript
+// Early in init(), before storage creation
+this.registerDefaultAugmentations() // Register but don't initialize
+```
+
+### Phase 2: Storage Resolution
+```typescript
+// Check for storage augmentations
+const storageAug = this.augmentations.findByOperation('storage')
+
+if (storageAug) {
+  // Get storage from augmentation
+  this.storage = await storageAug.provideStorage()
+} else if (this.config.storageAdapter) {
+  // Use provided adapter (backward compat)
+  this.storage = this.config.storageAdapter
+} else {
+  // Zero-config: create and wrap in augmentation
+  this.storage = await createStorage(this.storageConfig)
+  
+  // Auto-register as augmentation for consistency
+  const autoAug = new DynamicStorageAugmentation(this.storage)
+  this.augmentations.register(autoAug)
+}
+
+await this.storage.init()
+```
+
+### Phase 3: Full Augmentation Initialization
+```typescript
+// Now initialize all augmentations with context
+const context = {
+  brain: this,
+  storage: this.storage,
+  config: this.config,
+  log: this.log
+}
+
+await this.augmentations.initializeAll(context)
+```
+
+## Implementation Steps
+
+### Step 1: Create DynamicStorageAugmentation
+```typescript
+// Wraps any storage adapter as an augmentation
+class DynamicStorageAugmentation extends BaseAugmentation {
+  constructor(private adapter: StorageAdapter) {
+    super()
+    this.name = `${adapter.constructor.name}Augmentation`
+    this.timing = 'replace'
+    this.operations = ['storage']
+    this.priority = 100
+  }
+  
+  async provideStorage(): Promise<StorageAdapter> {
+    return this.adapter
+  }
+  
+  async execute(op, params, next) {
+    if (op === 'storage') {
+      return this.adapter
+    }
+    return next()
+  }
+}
+```
+
+### Step 2: Modify AugmentationRegistry
+```typescript
+class AugmentationRegistry {
+  // Add method to find augmentations before initialization
+  findByOperation(operation: string): BrainyAugmentation | null {
+    return this.augmentations.find(aug => 
+      aug.operations.includes(operation) || 
+      aug.operations.includes('all')
+    ) || null
+  }
+  
+  // Split registration from initialization
+  register(augmentation: BrainyAugmentation): void {
+    this.augmentations.push(augmentation)
+    // Don't initialize yet
+  }
+  
+  async initializeAll(context: AugmentationContext): Promise<void> {
+    for (const aug of this.augmentations) {
+      if (aug.initialize) {
+        await aug.initialize(context)
+      }
+    }
+  }
+}
+```
+
+### Step 3: Update BrainyData.init()
+```typescript
+async init(): Promise<void> {
+  // ... existing validation ...
+  
+  // Step 1: Register default augmentations (no init)
+  this.registerDefaultAugmentations()
+  
+  // Step 2: Resolve storage
+  await this.resolveStorage()
+  
+  // Step 3: Initialize augmentations with context
+  await this.initializeAugmentations()
+  
+  // ... rest of init ...
+}
+
+private async resolveStorage(): Promise<void> {
+  // Check for storage augmentation
+  const storageAug = this.augmentations.findByOperation('storage')
+  
+  if (storageAug && storageAug.provideStorage) {
+    // Get storage from augmentation
+    this.storage = await storageAug.provideStorage()
+  } else if (!this.storage) {
+    // No storage augmentation and no provided adapter
+    // Use zero-config
+    const storageOptions = this.buildStorageOptions()
+    this.storage = await createStorage(storageOptions)
+    
+    // Wrap in augmentation for consistency
+    const wrapper = new DynamicStorageAugmentation(this.storage)
+    this.augmentations.register(wrapper)
+  }
+  
+  // Initialize storage
+  await this.storage!.init()
+}
+```
+
+## Usage Examples
+
+### Zero-Config (No Change)
+```typescript
+const brain = new BrainyData()
+await brain.init()
+// Automatically selects best storage for environment
+```
+
+### Explicit Storage Adapter (Backward Compatible)
+```typescript
+const brain = new BrainyData({
+  storageAdapter: new S3Storage(config)
+})
+await brain.init()
+```
+
+### Storage via Augmentation (New)
+```typescript
+const brain = new BrainyData()
+brain.augmentations.register(new S3StorageAugmentation(config))
+await brain.init()
+```
+
+### Storage Config (Backward Compatible)
+```typescript
+const brain = new BrainyData({
+  storage: {
+    s3Storage: {
+      bucketName: 'my-bucket',
+      accessKeyId: 'xxx',
+      secretAccessKey: 'yyy'
+    }
+  }
+})
+await brain.init()
+```
+
+## Benefits
+
+1. **Unified Architecture:** Everything is an augmentation
+2. **Backward Compatible:** All existing code continues to work
+3. **Zero-Config Maintained:** Intelligent selection still works
+4. **Extensible:** Easy to add new storage types as augmentations
+5. **Middleware Capable:** Storage operations can be intercepted
+6. **Premium Ready:** Premium storage augmentations can be added to marketplace
+
+## Migration Path
+
+### Phase 1: Implement Infrastructure (No Breaking Changes)
+- Add DynamicStorageAugmentation
+- Update AugmentationRegistry with new methods
+- Modify BrainyData.init() to support both paths
+
+### Phase 2: Deprecate Direct Storage Config
+- Mark `storageAdapter` config as deprecated
+- Encourage augmentation approach in docs
+- Keep working for 2-3 major versions
+
+### Phase 3: Remove Legacy Code
+- Remove `storageAdapter` from config
+- Remove `createStorage()` direct calls
+- All storage through augmentations
+
+## Testing Strategy
+
+1. **Backward Compatibility Tests**
+   - Ensure all existing storage config methods work
+   - Test zero-config in different environments
+   - Verify no breaking changes
+
+2. **New Functionality Tests**
+   - Test storage augmentation registration
+   - Test override behavior
+   - Test middleware capabilities
+
+3. **Performance Tests**
+   - Ensure no performance regression
+   - Measure augmentation overhead
+
+## Risk Mitigation
+
+1. **Risk:** Circular dependency between storage and augmentations
+   **Mitigation:** Two-phase initialization breaks the cycle
+
+2. **Risk:** Breaking existing code
+   **Mitigation:** Keep `this.storage` and all direct calls unchanged
+
+3. **Risk:** Performance overhead
+   **Mitigation:** Storage augmentation is registered once, minimal overhead
+
+4. **Risk:** Confusion about which approach to use
+   **Mitigation:** Clear documentation, deprecation warnings, migration guide
+
+## Timeline
+
+- **Week 1:** Implement core infrastructure
+- **Week 2:** Update documentation and examples
+- **Week 3:** Testing and optimization
+- **Week 4:** Release as minor version (non-breaking)
+
+## Conclusion
+
+This unification maintains all existing behaviors while providing a cleaner, more extensible architecture. The augmentation approach aligns with Brainy's philosophy and enables future enhancements without breaking changes.