brainy/docs/STORAGE-AUGMENTATIONS-GUIDE.md

Storage Augmentations Guide

Overview

Brainy uses a unified augmentation system for storage backends. This guide explains the difference between built-in storage augmentations and how to create custom ones.

Built-in Storage Augmentations

These wrap existing, battle-tested storage adapters from /storage/adapters/:

Augmentation	Underlying Adapter	Environment	Description
MemoryStorageAugmentation	MemoryStorage	Universal	Fast in-memory storage (not persistent)
FileSystemStorageAugmentation	FileSystemStorage	Node.js	Persistent file-based storage
OPFSStorageAugmentation	OPFSStorage	Browser	Browser persistent storage
S3StorageAugmentation	S3CompatibleStorage	Universal	Amazon S3 with throttling & caching
R2StorageAugmentation	R2Storage	Universal	Cloudflare R2 storage
GCSStorageAugmentation	S3CompatibleStorage	Universal	Google Cloud Storage

Architecture of Built-in Storage

BrainyData
    ↓
StorageAugmentation (thin wrapper)
    ↓
StorageAdapter (actual implementation in /storage/adapters/)
    ↓
Actual Storage (filesystem, S3, memory, etc.)

Why This Design?

1. Preserve existing code - Storage adapters have years of bug fixes
2. Complex features intact - S3 throttling, caching, retry logic preserved
3. Minimal wrapper - Augmentations are just 20-30 lines
4. Zero feature loss - All 30+ StorageAdapter methods work unchanged

Using Built-in Storage

1. Zero-Config (Auto-Selection)

const brain = new BrainyData()
await brain.init()
// Automatically selects:
// - Node.js → FileSystemStorage
// - Browser → OPFSStorage (or Memory fallback)

2. Configuration-Based

const brain = new BrainyData({
  storage: {
    s3Storage: {
      bucketName: 'my-bucket',
      accessKeyId: 'xxx',
      secretAccessKey: 'yyy'
    }
  }
})

3. Augmentation Override

const brain = new BrainyData()
brain.augmentations.register(new S3StorageAugmentation({
  bucketName: 'my-bucket',
  region: 'us-east-1',
  accessKeyId: 'xxx',
  secretAccessKey: 'yyy'
}))
await brain.init()

Creating Custom Storage Augmentations

Custom storage augmentations can either:

1. Wrap an existing adapter (like built-ins do)
2. Implement the StorageAdapter interface directly

Option 1: Wrapping an Existing Adapter

import { StorageAugmentation } from 'brainy'
import { CustomAdapter } from './my-custom-adapter'

export class CustomStorageAugmentation extends StorageAugmentation {
  private config: CustomConfig
  
  constructor(config: CustomConfig) {
    super('custom-storage')
    this.config = config
  }
  
  async provideStorage(): Promise<StorageAdapter> {
    // Create and return your adapter
    const adapter = new CustomAdapter(this.config)
    return adapter
  }
}

Option 2: Self-Contained Implementation

import { StorageAugmentation, StorageAdapter } from 'brainy'
import Redis from 'ioredis'

export class RedisStorageAugmentation extends StorageAugmentation {
  private redis: Redis
  
  constructor(config: RedisConfig) {
    super('redis-storage')
    this.redis = new Redis(config)
  }
  
  async provideStorage(): Promise<StorageAdapter> {
    // Return an object implementing StorageAdapter
    return {
      async init() {
        await this.redis.ping()
      },
      
      async saveNoun(noun) {
        await this.redis.set(
          `noun:${noun.id}`,
          JSON.stringify(noun)
        )
      },
      
      async getNoun(id) {
        const data = await this.redis.get(`noun:${id}`)
        return data ? JSON.parse(data) : null
      },
      
      async deleteNoun(id) {
        await this.redis.del(`noun:${id}`)
      },
      
      // ... implement all 30+ required methods
      // See StorageAdapter interface in coreTypes.ts
    }
  }
}

StorageAdapter Interface Requirements

Your custom storage must implement these core methods:

interface StorageAdapter {
  // Initialization
  init(): Promise<void>
  
  // Noun operations
  saveNoun(noun: HNSWNoun): Promise<void>
  getNoun(id: string): Promise<HNSWNoun | null>
  deleteNoun(id: string): Promise<void>
  getNounsByNounType(type: string): Promise<HNSWNoun[]>
  
  // Verb operations
  saveVerb(verb: HNSWVerb): Promise<void>
  getVerb(id: string): Promise<HNSWVerb | null>
  deleteVerb(id: string): Promise<void>
  getVerbsBySource(sourceId: string): Promise<HNSWVerb[]>
  getVerbsByTarget(targetId: string): Promise<HNSWVerb[]>
  
  // Metadata operations
  saveMetadata(id: string, metadata: any): Promise<void>
  getMetadata(id: string): Promise<any | null>
  saveVerbMetadata(id: string, metadata: any): Promise<void>
  getVerbMetadata(id: string): Promise<any | null>
  
  // Pagination
  getNouns(options?: PaginationOptions): Promise<PaginatedResult>
  getVerbs(options?: PaginationOptions): Promise<PaginatedResult>
  
  // Statistics
  getStatistics(): Promise<StatisticsData | null>
  saveStatistics(stats: StatisticsData): Promise<void>
  incrementStatistic(type: string, service: string): Promise<void>
  
  // Utility
  clear(): Promise<void>
  getStorageStatus(): Promise<StorageStatus>
  
  // ... plus ~10 more methods
}

Publishing to Brain Cloud (Future)

Custom storage augmentations can be published to the Brain Cloud marketplace:

// package.json
{
  "name": "@brain-cloud/redis-storage",
  "version": "1.0.0",
  "brainy": {
    "type": "augmentation",
    "category": "storage",
    "implements": "StorageAdapter"
  }
}

Users will be able to install via:

brainy augment install redis-storage

Best Practices

1. Use existing adapters when possible - They're well-tested
2. Implement all methods - StorageAdapter has 30+ required methods
3. Handle errors gracefully - Storage is critical infrastructure
4. Include connection pooling - For network-based storage
5. Add retry logic - Network operations can fail
6. Implement caching - Reduce latency for hot data
7. Track statistics - Use BaseStorageAdapter if possible
8. Document configuration - Make it easy for users

Examples in the Wild

MongoDB Storage (Community)

class MongoStorageAugmentation extends StorageAugmentation {
  async provideStorage() {
    const client = new MongoClient(this.uri)
    const db = client.db('brainy')
    
    return {
      async saveNoun(noun) {
        await db.collection('nouns').replaceOne(
          { _id: noun.id },
          noun,
          { upsert: true }
        )
      },
      // ... full implementation
    }
  }
}

PostgreSQL Storage (Premium)

class PostgreSQLStorageAugmentation extends StorageAugmentation {
  async provideStorage() {
    const pool = new Pool(this.config)
    
    return {
      async saveNoun(noun) {
        await pool.query(
          'INSERT INTO nouns (id, data) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET data = $2',
          [noun.id, JSON.stringify(noun)]
        )
      },
      // ... full implementation
    }
  }
}

Summary

• Built-in augmentations wrap existing adapters (thin layer)
• Custom augmentations can wrap OR implement directly
• Storage adapters in /storage/adapters/ are for core only
• Premium storage comes as self-contained augmentations
• Everything uses the same StorageAdapter interface
• Zero-config still works perfectly

This design provides maximum flexibility while preserving all existing functionality!