brainy/docs/architecture/storage-architecture.md

# Storage Architecture

> **Updated**: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup

## Storage Structure

### Architecture: Metadata/Vector Separation

Entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:

```
brainy-data/
├── _system/                              # System metadata (not sharded)
│   ├── statistics.json                   # Performance metrics
│   ├── __metadata_field_index__*.json    # Field indexes
│   └── __metadata_sorted_index__*.json   # Sorted indexes
│
├── entities/
│   ├── nouns/
│   │   ├── vectors/                      # Vector graph data (sharded by UUID)
│   │   │   ├── 00/                       # Shard 00 (first 2 hex digits)
│   │   │   │   ├── 00123456-....json     # Vector + graph connections
│   │   │   │   └── 00abcdef-....json
│   │   │   ├── 01/ ... ff/               # 256 shards total
│   │   │
│   │   └── metadata/                     # Business data (sharded by UUID)
│   │       ├── 00/
│   │       │   ├── 00123456-....json     # Entity metadata only
│   │       │   └── 00abcdef-....json
│   │       ├── 01/ ... ff/
│   │
│   └── verbs/
│       ├── vectors/                      # Relationship vectors (sharded)
│       │   ├── 00/ ... ff/
│       │
│       └── metadata/                     # Relationship data (sharded)
│           ├── 00/ ... ff/
```

### Why Split Metadata and Vectors?

**Performance at scale:**
- **Vector search operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
- **Filtering**: Only load metadata during filtering, not vectors
- **Pagination**: Load metadata IDs first, fetch vectors/metadata on-demand
- **Result**: 60-70% reduction in I/O for typical queries at million-entity scale

### UUID-Based Sharding (256 Shards)

**How it works:**
```typescript
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
const shard = uuid.substring(0, 2) // "3f"

// Vector path:   entities/nouns/vectors/3f/3fa85f64-....json
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
```

**Benefits:**
- **Uniform distribution**: ~3,900 entities per shard (at 1M scale)
- **Filesystem optimization**: avoids huge flat directories that bog down `readdir`
- **Parallel operations**: walk 256 shards in parallel
- **Predictable**: Deterministic shard assignment

## Storage Adapters

Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` interface:

### FileSystem Storage (Node.js, default)
```typescript
const brain = new Brainy({
  storage: {
    type: 'filesystem',
    rootDirectory: './data'
  }
})
```
- **Use case**: Server applications, CLI tools, single-node deployments
- **Performance**: Direct file I/O
- **Persistence**: Permanent on disk
- **Features**:
  - **Batch Delete**: Efficient bulk deletion with retries
  - **UUID Sharding**: Automatic 256-shard distribution

### Memory Storage
```typescript
const brain = new Brainy({
  storage: {
    type: 'memory'
  }
})
```
- **Use case**: Tests, ephemeral workloads, single-process caches
- **Performance**: No I/O — all data lives in process memory
- **Persistence**: None — data is lost when the process exits

### Auto
```typescript
const brain = new Brainy({
  storage: {
    type: 'auto',
    rootDirectory: './data'
  }
})
```
`'auto'` picks `'filesystem'` when running on Node.js with a writable `rootDirectory`, and falls back to `'memory'` otherwise.

## Backup and Off-Site Replication

Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `rootDirectory` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:

- `gsutil rsync -r ./data gs://my-bucket/brainy-data`
- `aws s3 sync ./data s3://my-bucket/brainy-data`
- `rclone sync ./data remote:brainy-data`
- Periodic `tar` snapshots to any object store

Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory.

## Metadata Indexing System

### Field Discovery Index
Tracks all unique values for each field:

```json
// __metadata_field_index__field_category.json
{
  "values": {
    "technology": 45,
    "science": 32,
    "business": 28
  },
  "lastUpdated": 1699564234567
}
```

### Value-Based Indexes
Maps field+value combinations to entity IDs:

```json
// __metadata_index__category_technology_chunk0.json
{
  "field": "category",
  "value": "technology",
  "ids": ["uuid1", "uuid2", "uuid3", ...],
  "chunk": 0,
  "total": 45
}
```

### Index Chunking
Large indexes automatically chunk for performance:
- **Chunk size**: 10,000 IDs per chunk
- **Auto-splitting**: Transparent to queries
- **Parallel loading**: Chunks load on demand

## Entity Registry

High-performance deduplication system for streaming data:

### Registry Structure
```json
// __entity_registry__.json
{
  "mappings": {
    "did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
    "handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
  },
  "stats": {
    "totalMappings": 10000,
    "lastSync": 1699564234567
  }
}
```

### Performance Characteristics
- **Lookup**: O(1) in-memory hash map
- **Persistence**: Configurable (memory/storage/hybrid)
- **Cache**: LRU with configurable TTL
- **Sync**: Periodic or on-demand

## Durability

Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `rootDirectory` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).

## Storage Optimization

### 1. Batch Operations

```typescript
// Efficient batch delete
await storage.batchDelete([
  'entities/nouns/vectors/00/00123456-....json',
  'entities/nouns/metadata/00/00123456-....json'
  // ...
])

// Batch writes for performance
await brain.addBatch([
  { content: "item1", metadata: {} },
  { content: "item2", metadata: {} },
  { content: "item3", metadata: {} }
])
// Single transaction, optimized I/O
```

### 2. Caching Strategy

```typescript
// Configure caching
const brain = new Brainy({
  storage: {
    type: 'filesystem',
    rootDirectory: './data',
    cache: {
      enabled: true,
      maxSize: 1000,   // Maximum cached items
      ttl: 300000,     // 5 minutes
      strategy: 'lru'  // Least recently used
    }
  }
})
```

## Concurrent Access

### Locking Mechanism
```typescript
// Automatic locking for write operations
await brain.storage.withLock('resource-id', async () => {
  // Exclusive access to resource
  await brain.storage.saveNoun(id, data)
})
```

### Read-Write Separation
- **Reads**: Non-blocking, parallel
- **Writes**: Serialized with locks
- **Hybrid**: Read-heavy optimization

## Migration and Backup

### Export Data
```typescript
// Export entire database
const backup = await brain.export({
  format: 'json',
  includeVectors: true,
  includeIndexes: false
})
```

### Import Data
```typescript
// Import from backup
await brain.import(backup, {
  mode: 'merge',        // or 'replace'
  validateSchema: true
})
```

### Storage Migration
```typescript
// Migrate between storage types
const oldBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './old' } })
const newBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })

await oldBrain.init()
await newBrain.init()

// Transfer all data
const data = await oldBrain.export()
await newBrain.import(data)
```

## Performance Tuning

### FileSystem Optimizations
- **Directory sharding**: 256 shards spread files across subdirectories
- **Async I/O**: Non-blocking file operations
- **Buffer pooling**: Reuse buffers for efficiency

### Monitoring

```typescript
// Get storage statistics
const stats = await brain.storage.getStatistics()
console.log(stats)
// {
//   totalSize: 1048576,
//   entityCount: 1000,
//   indexSize: 204800,
//   walSize: 10240,
//   cacheHitRate: 0.85
// }
```

## Best Practices

### Choose the Right Adapter
1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence
2. **Single-node production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
3. **Multi-node production**: Run Brainy behind a service layer; replicate the on-disk artifact via your operator tooling

### Optimize for Your Use Case
1. **Read-heavy**: Enable caching and let the OS page cache do its job
2. **Write-heavy**: Batch operations and tune the cache `maxSize`
3. **Real-time**: FileSystem with periodic snapshots
4. **Archival**: Snapshot `rootDirectory` to cold object storage on a schedule
5. **Large-scale**: Rely on metadata/vector separation + UUID sharding

### Monitor and Maintain
1. Regular statistics collection
2. Watch disk usage and shard balance
3. Index optimization
4. Cache tuning based on hit rates
5. Verify backup runs (test restore quarterly)

## API Reference

See the [Storage API](../api/storage.md) for complete method documentation.

---

**Last Updated**: 2026
**Key Features**: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup