brainy/docs/architecture/storage-architecture.md

Storage Architecture

Updated: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup

Storage Structure

Architecture: Metadata/Vector Separation

Entities and relationships are split into 2 separate files for optimal performance at billion-entity scale:

brainy-data/
├── _system/                              # System metadata (not sharded)
│   ├── statistics.json                   # Performance metrics
│   ├── __metadata_field_index__*.json    # Field indexes
│   └── __metadata_sorted_index__*.json   # Sorted indexes
│
├── entities/
│   ├── nouns/
│   │   ├── vectors/                      # Vector graph data (sharded by UUID)
│   │   │   ├── 00/                       # Shard 00 (first 2 hex digits)
│   │   │   │   ├── 00123456-....json     # Vector + graph connections
│   │   │   │   └── 00abcdef-....json
│   │   │   ├── 01/ ... ff/               # 256 shards total
│   │   │
│   │   └── metadata/                     # Business data (sharded by UUID)
│   │       ├── 00/
│   │       │   ├── 00123456-....json     # Entity metadata only
│   │       │   └── 00abcdef-....json
│   │       ├── 01/ ... ff/
│   │
│   └── verbs/
│       ├── vectors/                      # Relationship vectors (sharded)
│       │   ├── 00/ ... ff/
│       │
│       └── metadata/                     # Relationship data (sharded)
│           ├── 00/ ... ff/

Why Split Metadata and Vectors?

Performance at scale:

• Vector search operations: Only load vectors (4KB) during search, not metadata (2-10KB)
• Filtering: Only load metadata during filtering, not vectors
• Pagination: Load metadata IDs first, fetch vectors/metadata on-demand
• Result: 60-70% reduction in I/O for typical queries at million-entity scale

UUID-Based Sharding (256 Shards)

How it works:

const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
const shard = uuid.substring(0, 2) // "3f"

// Vector path:   entities/nouns/vectors/3f/3fa85f64-....json
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json

Benefits:

• Uniform distribution: ~3,900 entities per shard (at 1M scale)
• Filesystem optimization: avoids huge flat directories that bog down readdir
• Parallel operations: walk 256 shards in parallel
• Predictable: Deterministic shard assignment

Storage Adapters

Brainy 8.0 ships two adapters, both implementing the same StorageAdapter interface:

FileSystem Storage (Node.js, default)

const brain = new Brainy({
  storage: {
    type: 'filesystem',
    rootDirectory: './data'
  }
})

• Use case: Server applications, CLI tools, single-node deployments
• Performance: Direct file I/O
• Persistence: Permanent on disk
• Features:
  • Batch Delete: Efficient bulk deletion with retries
  • UUID Sharding: Automatic 256-shard distribution

Memory Storage

const brain = new Brainy({
  storage: {
    type: 'memory'
  }
})

• Use case: Tests, ephemeral workloads, single-process caches
• Performance: No I/O — all data lives in process memory
• Persistence: None — data is lost when the process exits

Auto

const brain = new Brainy({
  storage: {
    type: 'auto',
    rootDirectory: './data'
  }
})

'auto' picks 'filesystem' when running on Node.js with a writable rootDirectory, and falls back to 'memory' otherwise.

Backup and Off-Site Replication

Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at rootDirectory is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:

• gsutil rsync -r ./data gs://my-bucket/brainy-data
• aws s3 sync ./data s3://my-bucket/brainy-data
• rclone sync ./data remote:brainy-data
• Periodic tar snapshots to any object store

Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory.

Metadata Indexing System

Field Discovery Index

Tracks all unique values for each field:

// __metadata_field_index__field_category.json
{
  "values": {
    "technology": 45,
    "science": 32,
    "business": 28
  },
  "lastUpdated": 1699564234567
}

Value-Based Indexes

Maps field+value combinations to entity IDs:

// __metadata_index__category_technology_chunk0.json
{
  "field": "category",
  "value": "technology",
  "ids": ["uuid1", "uuid2", "uuid3", ...],
  "chunk": 0,
  "total": 45
}

Index Chunking

Large indexes automatically chunk for performance:

• Chunk size: 10,000 IDs per chunk
• Auto-splitting: Transparent to queries
• Parallel loading: Chunks load on demand

Entity Registry

High-performance deduplication system for streaming data:

Registry Structure

// __entity_registry__.json
{
  "mappings": {
    "did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
    "handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
  },
  "stats": {
    "totalMappings": 10000,
    "lastSync": 1699564234567
  }
}

Performance Characteristics

• Lookup: O(1) in-memory hash map
• Persistence: Configurable (memory/storage/hybrid)
• Cache: LRU with configurable TTL
• Sync: Periodic or on-demand

Durability

Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot rootDirectory (see Backup and Off-Site Replication).

Storage Optimization

1. Batch Operations

// Efficient batch delete
await storage.batchDelete([
  'entities/nouns/vectors/00/00123456-....json',
  'entities/nouns/metadata/00/00123456-....json'
  // ...
])

// Batch writes for performance
await brain.addBatch([
  { content: "item1", metadata: {} },
  { content: "item2", metadata: {} },
  { content: "item3", metadata: {} }
])
// Single transaction, optimized I/O

2. Caching Strategy

// Configure caching
const brain = new Brainy({
  storage: {
    type: 'filesystem',
    rootDirectory: './data',
    cache: {
      enabled: true,
      maxSize: 1000,   // Maximum cached items
      ttl: 300000,     // 5 minutes
      strategy: 'lru'  // Least recently used
    }
  }
})

Concurrent Access

Locking Mechanism

// Automatic locking for write operations
await brain.storage.withLock('resource-id', async () => {
  // Exclusive access to resource
  await brain.storage.saveNoun(id, data)
})

Read-Write Separation

• Reads: Non-blocking, parallel
• Writes: Serialized with locks
• Hybrid: Read-heavy optimization

Migration and Backup

Export Data

// Export entire database
const backup = await brain.export({
  format: 'json',
  includeVectors: true,
  includeIndexes: false
})

Import Data

// Import from backup
await brain.import(backup, {
  mode: 'merge',        // or 'replace'
  validateSchema: true
})

Storage Migration

// Migrate between storage types
const oldBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './old' } })
const newBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })

await oldBrain.init()
await newBrain.init()

// Transfer all data
const data = await oldBrain.export()
await newBrain.import(data)

Performance Tuning

FileSystem Optimizations

• Directory sharding: 256 shards spread files across subdirectories
• Async I/O: Non-blocking file operations
• Buffer pooling: Reuse buffers for efficiency

Monitoring

// Get storage statistics
const stats = await brain.storage.getStatistics()
console.log(stats)
// {
//   totalSize: 1048576,
//   entityCount: 1000,
//   indexSize: 204800,
//   walSize: 10240,
//   cacheHitRate: 0.85
// }

Best Practices

Choose the Right Adapter

1. Development & tests: memory for speed, filesystem when you need persistence
2. Single-node production: filesystem with off-site backup via gsutil / aws s3 sync / rclone
3. Multi-node production: Run Brainy behind a service layer; replicate the on-disk artifact via your operator tooling

Optimize for Your Use Case

1. Read-heavy: Enable caching and let the OS page cache do its job
2. Write-heavy: Batch operations and tune the cache maxSize
3. Real-time: FileSystem with periodic snapshots
4. Archival: Snapshot rootDirectory to cold object storage on a schedule
5. Large-scale: Rely on metadata/vector separation + UUID sharding

Monitor and Maintain

1. Regular statistics collection
2. Watch disk usage and shard balance
3. Index optimization
4. Cache tuning based on hit rates
5. Verify backup runs (test restore quarterly)

API Reference

See the Storage API for complete method documentation.

---

Last Updated: 2026 Key Features: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup