brainy/STORAGE_ADAPTER_QUICK_REFERENCE.md

Brainy Storage Adapter - Quick Reference Guide

File Locations

src/storage/
├── baseStorageAdapter.ts          # Abstract base class (1,156 lines)
├── baseStorage.ts                 # Implementation layer (1,098 lines)
├── storageFactory.ts              # Factory for adapter selection
├── sharding.ts                    # UUID-based sharding utilities
├── cacheManager.ts                # Cache implementation
└── adapters/
    ├── fileSystemStorage.ts       # Node.js file system (2,677 lines)
    ├── memoryStorage.ts           # In-memory storage (822 lines)
    ├── s3CompatibleStorage.ts     # AWS S3 / R2 / GCS compat (5000+ lines)
    ├── gcsStorage.ts              # Google Cloud Storage native (1,835 lines)
    ├── opfsStorage.ts             # Browser OPFS storage
    └── baseStorageAdapter.ts      # Base class

src/coreTypes.ts
└── StorageAdapter interface (27 methods)

Storage Adapter Hierarchy

┌─ StorageAdapter (interface)
│  ├─ StorageAdapter.init()
│  ├─ StorageAdapter.saveNoun()
│  ├─ StorageAdapter.getNouns()
│  └─ ... (23 more methods)
│
└─ BaseStorageAdapter (abstract class)
   ├─ Statistics management
   ├─ Throttling detection
   ├─ Count tracking (O(1))
   ├─ Service-level statistics
   └─ Abstract methods for subclasses
   
   └─ BaseStorage (abstract class)
      ├─ 2-file system routing
      ├─ Pagination support
      ├─ Metadata handling
      ├─ Public API (saveNoun, getNoun, etc.)
      └─ Abstract internal methods
      
      └─ Concrete Adapters
         ├─ FileSystemStorage
         ├─ MemoryStorage
         ├─ S3CompatibleStorage
         ├─ GcsStorage
         └─ OPFSStorage

Abstract Methods to Implement

When creating a new adapter, extend BaseStorage and implement:

Noun/Verb Operations (6 methods)

protected abstract saveNoun_internal(noun: HNSWNoun): Promise<void>
protected abstract getNoun_internal(id: string): Promise<HNSWNoun | null>
protected abstract deleteNoun_internal(id: string): Promise<void>
protected abstract saveVerb_internal(verb: HNSWVerb): Promise<void>
protected abstract getVerb_internal(id: string): Promise<HNSWVerb | null>
protected abstract deleteVerb_internal(id: string): Promise<void>

Path Operations (4 methods)

protected abstract writeObjectToPath(path: string, data: any): Promise<void>
protected abstract readObjectFromPath(path: string): Promise<any | null>
protected abstract deleteObjectFromPath(path: string): Promise<void>
protected abstract listObjectsUnderPath(prefix: string): Promise<string[]>

Count Management (2 methods)

protected abstract initializeCounts(): Promise<void>
protected abstract persistCounts(): Promise<void>

Statistics (2 methods)

protected abstract saveStatisticsData(statistics: StatisticsData): Promise<void>
protected abstract getStatisticsData(): Promise<StatisticsData | null>

Lifecycle (3 methods)

abstract init(): Promise<void>
abstract clear(): Promise<void>
abstract getStorageStatus(): Promise<StorageStatus>

Total: 17 abstract methods to implement

Storage Path Structure

Modern Entity-Based Structure

storage-root/
├── entities/
│   ├── nouns/vectors/{shard}/{id}.json        ← Vector data
│   ├── nouns/metadata/{shard}/{id}.json       ← Metadata
│   ├── nouns/hnsw/{shard}/{id}.json           ← HNSW graph
│   ├── verbs/vectors/{shard}/{id}.json
│   ├── verbs/metadata/{shard}/{id}.json
│   └── verbs/hnsw/{shard}/{id}.json
├── indexes/
│   ├── metadata/...                           ← Search indexes
│   └── graph/...
└── _system/
    ├── statistics.json                        ← Aggregate stats
    ├── counts.json                            ← O(1) totals
    └── hnsw-system.json                       ← HNSW metadata

Shard Format

• First 2 hex chars of UUID (00-ff) = 256 shards
• Example: ab123456-... → stored in ab/ directory
• Enables 2.5M+ entities with consistent performance

2-File System Design

Vector File (always loaded with HNSW)

{
  "id": "ab123456-...",
  "vector": [0.1, 0.2, ...],
  "connections": { "0": [...], "1": [...] },
  "level": 2
}

Metadata File (loaded separately)

{
  "noun": "Person",
  "name": "Alice",
  "email": "alice@example.com",
  "createdAt": "...",
  "service": "user-service"
}

Benefit: Decouple vector operations from flexible metadata queries

Existing Adapters Overview

FileSystemStorage (Node.js)

• 2,677 lines
• Sharding with migration support
• File-based locking for multi-process
• Production-ready

MemoryStorage (Testing)

• 822 lines
• In-memory Maps
• Fast for testing
• No persistence

S3CompatibleStorage (Cloud)

• 5,000+ lines
• AWS S3, Cloudflare R2, GCS (via S3 API)
• Adaptive batching, request coalescing
• High-volume mode, write buffers

GcsStorage (Google Cloud)

• 1,835 lines
• Native @google-cloud/storage SDK
• ADC, service account, HMAC auth
• Cache managers, backpressure

OPFSStorage (Browser)

• Browser Origin Private File System
• Persistent across sessions
• Modern browsers only

Factory Integration

// src/storage/storageFactory.ts
const storage = await createStorage({
  type: 'filesystem',           // auto, memory, filesystem, s3, gcs, gcs-native, opfs
  path: './data',
  s3Storage: { bucketName, region, ... },
  gcsStorage: { bucketName, credentials, ... },
})

Adding TypeAwareStorageAdapter

Recommended Approach: Direct Implementation

// src/storage/adapters/typeAwareStorageAdapter.ts
export class TypeAwareStorageAdapter extends BaseStorage {
  // Implement 17 abstract methods
  // Add type indexing logic
  // Track noun/verb types in separate indexes
}

Integration Steps

1. Create /src/storage/adapters/typeAwareStorageAdapter.ts
2. Add to factory in /src/storage/storageFactory.ts
3. Update StorageOptions interface with type: 'type-aware'
4. No changes to Brainy.ts or existing adapters needed

Key Features Inherited from BaseStorageAdapter

• Statistics Caching: Batches updates for efficiency
• Throttling Detection: Handles 429/503 errors
• Count Management: O(1) operations with persistence
• Service Tracking: Per-service statistics
• Field Name Tracking: Metadata field discovery

Performance Characteristics

O(1) Operations

• getNounCount() - total noun count
• getVerbCount() - total verb count

O(n) Operations

• getNouns() - paginated listing (n = page size)
• getVerbs() - paginated listing
• getNounsByNounType() - filter by type
• getVerbsBySource() - filter by source

Cloud Storage Features (GCS, S3)

• High-volume mode detection
• Adaptive batching
• Request coalescing for deduplication
• Write buffers for bulk operations
• Backpressure management
• Socket pool management

Testing Storage Adapters

All adapters implement the same interface, so:

// Test with MemoryStorage (fastest)
const storage = new MemoryStorage()

// Test with FileSystemStorage (persistent)
const storage = new FileSystemStorage('./test-data')

// All adapters support the same operations
await storage.init()
await storage.saveNoun(noun)
const result = await storage.getNoun(id)
await storage.clear()

Brainy Integration

export class Brainy {
  private storage!: BaseStorage
  
  async init(config: BrainyConfig): Promise<void> {
    // Factory creates appropriate adapter
    this.storage = await createStorage(config.storage) as BaseStorage
    await this.storage.init()
    
    // Pass to HNSW index
    this.index = new HNSWIndex(this.storage, ...)
  }
}

Brainy depends on BaseStorage interface, not specific adapters.

Design Patterns Used

1. Factory Pattern - createStorage() selects adapter
2. Strategy Pattern - Adapters are interchangeable
3. Template Method - BaseStorage defines skeleton
4. Decorator Pattern - Can wrap adapters (e.g., TypeAware wrapper)
5. Adapter Pattern - Maps different storage backends to same interface

Conclusion

TypeAwareStorageAdapter can be added as a new adapter alongside existing ones without:

• Modifying Brainy.ts
• Replacing existing adapters
• Breaking the StorageAdapter interface
• Changing how storage is used throughout the codebase

Simply extend BaseStorage, implement 17 abstract methods, and register in storageFactory.ts.