brainy/docs/architecture/storage-architecture.md

Storage Architecture

Storage Structure

brainy-data/
├── _system/                        # System management
│   └── statistics.json            # Performance metrics and statistics
├── nouns/                         # Primary entity storage
│   └── {uuid}.json               # Individual entity documents
├── metadata/                      # Metadata and indexing system
│   ├── {uuid}.json               # Entity metadata
│   ├── __entity_registry__.json  # Entity deduplication registry
│   ├── __metadata_field_index__field_{field}.json    # Field discovery
│   └── __metadata_index__{field}_{value}_chunk{n}.json # Value indexes
├── verbs/                         # Relationship/action storage
│   └── {uuid}.json               # Relationship documents
│   └── wal_{timestamp}_{id}.wal # Transaction logs
└── locks/                        # Concurrent access control
    └── {resource}.lock          # Resource locks

Storage Adapters

Brainy provides multiple storage adapters with identical APIs:

FileSystem Storage (Node.js)

const brain = new Brainy({
  storage: {
    type: 'filesystem',
    path: './data'
  }
})

• Use case: Server applications, CLI tools
• Performance: Direct file I/O
• Persistence: Permanent on disk

S3 Compatible Storage

const brain = new Brainy({
  storage: {
    type: 's3',
    bucket: 'my-brainy-data',
    region: 'us-east-1',
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
    }
  }
})

• Use case: Distributed applications, cloud deployments
• Performance: Network dependent, with intelligent caching
• Persistence: Cloud storage durability

Origin Private File System (Browser)

const brain = new Brainy({
  storage: {
    type: 'opfs'
  }
})

• Use case: Browser applications, PWAs
• Performance: Near-native file system speed
• Persistence: Permanent in browser (with quota limits)

Memory Storage

const brain = new Brainy({
  storage: {
    type: 'memory'
  }
})

• Use case: Testing, temporary processing
• Performance: Fastest possible
• Persistence: Volatile (lost on restart)

Metadata Indexing System

Field Discovery Index

Tracks all unique values for each field:

// __metadata_field_index__field_category.json
{
  "values": {
    "technology": 45,
    "science": 32,
    "business": 28
  },
  "lastUpdated": 1699564234567
}

Value-Based Indexes

Maps field+value combinations to entity IDs:

// __metadata_index__category_technology_chunk0.json
{
  "field": "category",
  "value": "technology",
  "ids": ["uuid1", "uuid2", "uuid3", ...],
  "chunk": 0,
  "total": 45
}

Index Chunking

Large indexes automatically chunk for performance:

• Chunk size: 10,000 IDs per chunk
• Auto-splitting: Transparent to queries
• Parallel loading: Chunks load on demand

Entity Registry

High-performance deduplication system for streaming data:

Registry Structure

// __entity_registry__.json
{
  "mappings": {
    "did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
    "handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
  },
  "stats": {
    "totalMappings": 10000,
    "lastSync": 1699564234567
  }
}

Performance Characteristics

• Lookup: O(1) in-memory hash map
• Persistence: Configurable (memory/storage/hybrid)
• Cache: LRU with configurable TTL
• Sync: Periodic or on-demand

Ensures durability and enables recovery:

{
  "timestamp": 1699564234567,
  "operation": "add",
  "data": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "content": "...",
    "metadata": {}
  },
  "checksum": "sha256:..."
}

Recovery Process

2. Replay operations from last checkpoint
3. Verify checksums for integrity

Storage Optimization

Compression

• JSON: Automatic minification
• Vectors: Float32 to Uint8 quantization option
• Indexes: Binary format for large datasets

Caching Strategy

// Configure caching per storage type
const brain = new Brainy({
  storage: {
    type: 'filesystem',
    cache: {
      enabled: true,
      maxSize: 1000,      // Maximum cached items
      ttl: 300000,        // 5 minutes
      strategy: 'lru'     // Least recently used
    }
  }
})

Batch Operations

// Batch writes for performance
await brain.addBatch([
  { content: "item1", metadata: {} },
  { content: "item2", metadata: {} },
  { content: "item3", metadata: {} }
])
// Single transaction, optimized I/O

Concurrent Access

Locking Mechanism

// Automatic locking for write operations
await brain.storage.withLock('resource-id', async () => {
  // Exclusive access to resource
  await brain.storage.saveNoun(id, data)
})

Read-Write Separation

• Reads: Non-blocking, parallel
• Writes: Serialized with locks
• Hybrid: Read-heavy optimization

Migration and Backup

Export Data

// Export entire database
const backup = await brain.export({
  format: 'json',
  includeVectors: true,
  includeIndexes: false
})

Import Data

// Import from backup
await brain.import(backup, {
  mode: 'merge',  // or 'replace'
  validateSchema: true
})

Storage Migration

// Migrate between storage types
const oldBrain = new Brainy({ storage: { type: 'filesystem' } })
const newBrain = new Brainy({ storage: { type: 's3' } })

await oldBrain.init()
await newBrain.init()

// Transfer all data
const data = await oldBrain.export()
await newBrain.import(data)

Performance Tuning

Storage-Specific Optimizations

FileSystem

• Directory sharding: Split files across subdirectories
• Async I/O: Non-blocking file operations
• Buffer pooling: Reuse buffers for efficiency

S3

• Multipart uploads: For large objects
• Request batching: Combine small operations
• CDN integration: Edge caching for reads

OPFS

• Quota management: Monitor and request increases
• Worker offloading: Heavy operations in workers
• Transaction batching: Group operations

Monitoring

// Get storage statistics
const stats = await brain.storage.getStatistics()
console.log(stats)
// {
//   totalSize: 1048576,
//   entityCount: 1000,
//   indexSize: 204800,
//   walSize: 10240,
//   cacheHitRate: 0.85
// }

Best Practices

Choose the Right Adapter

1. Development: Memory or FileSystem
2. Production Server: FileSystem or S3
3. Browser Apps: OPFS or Memory
4. Distributed: S3 with caching

Optimize for Your Use Case

1. Read-heavy: Enable aggressive caching
2. Real-time: Memory with periodic persistence
3. Archival: S3 with compression

Monitor and Maintain

1. Regular statistics collection
2. Index optimization
3. Cache tuning based on hit rates

API Reference

See the Storage API for complete method documentation.