brainy/STORAGE_FILES_REFERENCE.md

Brainy Storage System - Complete File Reference

Core Storage Files

1. Storage Interface Definition

File: src/coreTypes.ts

• Lines: ~250 (StorageAdapter interface)
• Type: Interface definition
• Content:
  • StorageAdapter interface (27 methods)
  • Supporting types: HNSWNoun, HNSWVerb, GraphVerb
  • Statistics types: StatisticsData, ServiceStatistics

2. Base Storage Adapter (Abstract Class)

File: src/storage/adapters/baseStorageAdapter.ts

• Lines: 1,156
• Type: Abstract base class
• Purpose: Common functionality for all adapters
• Key Features:
  • Statistics caching and batching
  • Throttling detection and backoff
  • Count management (O(1) operations)
  • Service-level statistics tracking
  • Field name discovery
• Implements: StorageAdapter interface
• Key Methods:
  • flushStatistics() - Write statistics to storage
  • isThrottlingError() - Detect cloud storage throttling
  • handleThrottling() - Exponential backoff
  • trackThrottlingEvent() - Record throttling events
  • incrementStatistic() - Increment service stats
  • decrementStatistic() - Decrement service stats

3. Base Storage Implementation

File: src/storage/baseStorage.ts

• Lines: 1,098
• Type: Abstract class extending BaseStorageAdapter
• Purpose: Core storage logic (routing, sharding, metadata)
• Key Features:
  • 2-file system implementation (vectors + metadata)
  • UUID-based sharding (256 directories)
  • Metadata routing and separation
  • Pagination support
  • Backward compatibility with legacy paths
• Implements:
  • Public API (saveNoun, getNoun, etc.)
  • Metadata operations (saveNounMetadata, etc.)
• Key Abstract Methods:
  • saveNoun_internal() - Adapter-specific noun save
  • getNoun_internal() - Adapter-specific noun read
  • writeObjectToPath() - Generic write
  • readObjectFromPath() - Generic read
  • listObjectsUnderPath() - Listing support

4. Storage Factory

File: src/storage/storageFactory.ts

• Lines: ~200
• Purpose: Factory function for adapter selection
• Function: createStorage(options: StorageOptions): Promise<StorageAdapter>
• Selection Logic:
  1. Forced memory/filesystem (testing)
  2. Explicit type selection
  3. Auto-detection (browser vs Node.js)
• Supported Types:
  • 'memory' - MemoryStorage
  • 'filesystem' - FileSystemStorage
  • 's3' - S3CompatibleStorage (AWS S3, R2)
  • 'gcs' - S3CompatibleStorage (GCS S3 API)
  • 'gcs-native' - GcsStorage (native SDK)
  • 'opfs' - OPFSStorage (browser)

5. Sharding Utilities

File: src/storage/sharding.ts

• Purpose: UUID-based sharding helpers
• Key Functions:
  • getShardIdFromUuid(id: string): string - Get first 2 hex chars
  • getShardIdByIndex(index: number): string - Get shard by index
  • getAllShardIds(): string[] - All 256 shard IDs
• Constants:
  • TOTAL_SHARDS = 256
  • MIN_SHARD_ID = '00'
  • MAX_SHARD_ID = 'ff'

6. Cache Manager

File: src/storage/cacheManager.ts

• Purpose: Generic LRU cache for storage
• Type: Generic class CacheManager<T>
• Used By: GcsStorage, S3CompatibleStorage
• Features:
  • LRU eviction
  • TTL support
  • Max size limits
  • Batch operations

Storage Adapter Implementations

FileSystemStorage (Node.js)

File: src/storage/adapters/fileSystemStorage.ts

• Lines: 2,677
• Type: Concrete implementation extending BaseStorage
• Platform: Node.js only
• Storage Target: Local file system
• Key Features:
  • Sharding with automatic migration
  • File-based locking (multi-process)
  • O(1) count persistence
  • HNSW index persistence
  • Dual-write for migrations
  • Path depth migration (0→1, 2→1)

Key Methods:

private getNounPath(id: string, depth: number): string
private getVerbPath(id: string, depth: number): string
private getNounMetadataPath(id: string, depth: number): string
async migrateShardingStructure(fromDepth: number, toDepth: number): Promise<void>
async migrateFromOldStructure(): Promise<void>

Statistics Tracking:

• Persists counts to _system/counts.json
• Tracks noun/verb type distributions
• Service-level activity timestamps
• Field name discovery

MemoryStorage (Testing)

File: src/storage/adapters/memoryStorage.ts

• Lines: 822
• Type: Concrete implementation extending BaseStorage
• Platform: Browser & Node.js
• Storage Target: In-memory Maps
• Key Features:
  • Fast for testing
  • No persistence (ephemeral)
  • Full pagination support
  • Filtering capabilities
  • 2-file system simulation

Key Maps:

private nouns: Map<string, HNSWNoun>
private verbs: Map<string, HNSWVerb>
private objectStore: Map<string, any>  // Unified metadata store

Methods:

• getNouns() - Paginated noun listing
• getVerbs() - Paginated verb listing
• getMetadataBatch() - Batch metadata loading

S3CompatibleStorage (Cloud)

File: src/storage/adapters/s3CompatibleStorage.ts

• Lines: 5,000+
• Type: Concrete implementation extending BaseStorage
• Platform: Node.js (server-side)
• Storage Targets:
  • Amazon S3
  • Cloudflare R2 (via S3 API)
  • Google Cloud Storage (via S3 API)

Key Features:

• Adaptive batching (10-1000 items)
• Request coalescing for deduplication
• High-volume mode detection
• Write buffers for bulk operations
• Socket pool management
• Backpressure system
• Change log tracking
• Cache management (nouns & verbs)

Performance Optimization:

• nounWriteBuffer - Batches noun writes
• verbWriteBuffer - Batches verb writes
• requestCoalescer - Deduplicates requests
• highVolumeMode - Activates at >20 pending ops
• baseBatchSize - Adaptive from 10 to 1000

GcsStorage (Google Cloud Native)

File: src/storage/adapters/gcsStorage.ts

• Lines: 1,835
• Type: Concrete implementation extending BaseStorage
• Platform: Node.js (server-side)
• Storage Target: Google Cloud Storage
• SDK: @google-cloud/storage (native)

Key Features:

• Application Default Credentials (ADC)
• Service Account Key File support
• Service Account Credentials Object
• HMAC Keys (backward compatible)
• Multi-level cache managers
• Backpressure management
• High-volume mode
• Request coalescing

Authentication Priority:

1. ADC (Application Default Credentials)
2. Service Account Key File
3. Service Account Credentials
4. HMAC Keys

Cache Managers:

private nounCacheManager: CacheManager<HNSWNode>
private verbCacheManager: CacheManager<Edge>

OPFSStorage (Browser Storage)

File: src/storage/adapters/opfsStorage.ts

• Type: Concrete implementation extending BaseStorage
• Platform: Browser only
• Storage Target: Origin Private File System (OPFS)
• Fallback: MemoryStorage if OPFS unavailable

Browser Compatibility:

• Chrome 96+
• Edge 96+
• Safari 15.1+

Storage Patterns and Utilities

Backward Compatibility

File: src/storage/backwardCompatibility.ts

• Purpose: Handle legacy storage paths
• Features:
  • Path migration detection
  • Dual-write during transition
  • Graceful fallback to old locations
  • Read-from-new, fallback-to-old

Metadata Index

File: src/utils/metadataIndex.ts

• Purpose: Build searchable indexes from metadata
• Used By: Brainy for fast metadata queries
• Features:
  • Field name discovery
  • Standard field mapping
  • Service-level statistics

Storage Discovery (Distributed)

File: src/distributed/storageDiscovery.ts

• Purpose: Discover storage config in distributed systems
• Features:
  • Node coordination
  • Storage synchronization

Adaptive Backpressure

File: src/utils/adaptiveBackpressure.ts

• Purpose: Flow control for storage operations
• Used By: All cloud storage adapters
• Features:
  • Request queuing
  • Throttling detection
  • Backoff scheduling

Write Buffer

File: src/utils/writeBuffer.ts

• Purpose: Batch write operations
• Used By: S3, GCS adapters
• Features:
  • Configurable batch size
  • Flush on timeout
  • Deduplication

Request Coalescer

File: src/utils/requestCoalescer.ts

• Purpose: Deduplicate concurrent requests
• Used By: S3, GCS adapters
• Features:
  • Request deduplication
  • Batch processing

Integration Points

In Brainy.ts (Main Class)

private storage!: BaseStorage

async init(): Promise<void> {
  // Create storage from factory
  const storageAdapter = await createStorage(this.config.storage)
  this.storage = storageAdapter as BaseStorage
  
  // Initialize
  await this.storage.init()
  
  // Pass to HNSW index
  this.index = new HNSWIndex(this.storage, ...)
}

In HNSW Index

export class HNSWIndex {
  constructor(private storage: StorageAdapter, ...)
  
  // Uses storage for node/edge persistence
  async saveNode(noun: HNSWNoun): Promise<void>
  async getNode(id: string): Promise<HNSWNoun>
}

In Metadata Index

export class MetadataIndexManager {
  constructor(storage: StorageAdapter, ...)
  
  // Uses storage for metadata queries
  async getNounsByFilter(filter: Filter): Promise<HNSWNoun[]>
}

Data Flow

Saving a Noun

Brainy.add()
  ↓
HNSWIndex.insert()
  ↓
BaseStorage.saveNoun()
  ├─ saveNoun_internal() → adapter-specific
  ├─ saveNounMetadata() → path routing
  └─ updateStatistics()

FileSystemStorage.saveNoun_internal()
  ├─ Create shard directory (ab/)
  ├─ Write JSON file
  └─ Update counts

Querying Nouns

Brainy.search()
  ↓
HNSW.search()
  ↓
BaseStorage.getNoun()
  ├─ getNoun_internal() → adapter-specific
  └─ getNounMetadata() → path routing

S3CompatibleStorage.getNoun_internal()
  ├─ Check cache
  ├─ Download from S3
  ├─ Parse JSON
  └─ Update cache

Storage Statistics Tracking

Stored in _system/statistics.json

{
  "nounCount": { "Person": 5, "Company": 2 },
  "verbCount": { "knows": 10, "works_at": 3 },
  "metadataCount": { "user-service": 50 },
  "hnswIndexSize": 15,
  "totalNodes": 7,
  "totalEdges": 13,
  "services": [
    { "name": "user-service", "totalNouns": 5, ... }
  ],
  "lastUpdated": "2024-10-15T..."
}

Stored in _system/counts.json

{
  "totalNounCount": 7,
  "totalVerbCount": 13,
  "entityCounts": { "Person": 5, "Company": 2 },
  "verbCounts": { "knows": 10, "works_at": 3 }
}

Type Definitions

HNSWNoun (Vector + HNSW)

interface HNSWNoun {
  id: string
  vector: number[]
  connections: Map<number, Set<string>>  // level → node IDs
  level: number
  metadata?: any  // Optional in vector file, separate file system
}

GraphVerb (Relationship)

interface GraphVerb {
  id: string
  sourceId: string
  targetId: string
  vector: number[]
  type?: string
  weight?: number
  metadata?: any
  // Plus aliases: source, target, verb, embedding
  createdAt?: Timestamp
  createdBy?: { augmentation: string; version: string }
}

Summary Statistics

Component	File	Lines	Purpose
StorageAdapter Interface	coreTypes.ts	250	Interface definition
BaseStorageAdapter	baseStorageAdapter.ts	1,156	Common functionality
BaseStorage	baseStorage.ts	1,098	Core routing & pagination
storageFactory	storageFactory.ts	200	Adapter selection
FileSystemStorage	fileSystemStorage.ts	2,677	Node.js FS
MemoryStorage	memoryStorage.ts	822	In-memory (test)
S3CompatibleStorage	s3CompatibleStorage.ts	5,000+	AWS S3 / R2 / GCS
GcsStorage	gcsStorage.ts	1,835	Google Cloud native
sharding.ts	sharding.ts	~100	UUID sharding
cacheManager.ts	cacheManager.ts	~200	LRU cache
TOTAL		~13,000+	Complete storage system

Conclusion

Brainy's storage architecture is:

• Well-layered (Interface → Abstract → Concrete)
• Extensible (factory pattern)
• Flexible (multiple backends)
• Scalable (sharding, caching, batching)
• Type-safe (full TypeScript support)

New adapters like TypeAwareStorageAdapter simply extend BaseStorage and implement 17 abstract methods.