brainy/docs/architecture/storage-architecture.md

# Storage Architecture

> **Updated**: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup

## Storage Structure

### Architecture: Metadata/Vector Separation

Entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:

```
brainy-data/
├── _system/                              # System metadata (not sharded)
│   ├── statistics.json                   # Performance metrics
│   ├── __metadata_field_index__*.json    # Field indexes
│   └── __metadata_sorted_index__*.json   # Sorted indexes
│
├── entities/
│   ├── nouns/
│   │   ├── vectors/                      # Vector graph data (sharded by UUID)
│   │   │   ├── 00/                       # Shard 00 (first 2 hex digits)
│   │   │   │   ├── 00123456-....json     # Vector + graph connections
│   │   │   │   └── 00abcdef-....json
│   │   │   ├── 01/ ... ff/               # 256 shards total
│   │   │
│   │   └── metadata/                     # Business data (sharded by UUID)
│   │       ├── 00/
│   │       │   ├── 00123456-....json     # Entity metadata only
│   │       │   └── 00abcdef-....json
│   │       ├── 01/ ... ff/
│   │
│   └── verbs/
│       ├── vectors/                      # Relationship vectors (sharded)
│       │   ├── 00/ ... ff/
│       │
│       └── metadata/                     # Relationship data (sharded)
│           ├── 00/ ... ff/
```

### Why Split Metadata and Vectors?

**Performance at scale:**
- **Vector search operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
- **Filtering**: Only load metadata during filtering, not vectors
- **Pagination**: Load metadata IDs first, fetch vectors/metadata on-demand
- **Result**: 60-70% reduction in I/O for typical queries at million-entity scale

### UUID-Based Sharding (256 Shards)

**How it works:**
```typescript
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
const shard = uuid.substring(0, 2) // "3f"

// Vector path:   entities/nouns/vectors/3f/3fa85f64-....json
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
```

**Benefits:**
- **Uniform distribution**: ~3,900 entities per shard (at 1M scale)
- **Filesystem optimization**: avoids huge flat directories that bog down `readdir`
- **Parallel operations**: walk 256 shards in parallel
- **Predictable**: Deterministic shard assignment

## Storage Adapters

Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` interface:

### FileSystem Storage (Node.js, default)
```typescript
const brain = new Brainy({
  storage: {
    type: 'filesystem',
    rootDirectory: './data'
  }
})
```
- **Use case**: Server applications, CLI tools, single-node deployments
- **Performance**: Direct file I/O
- **Persistence**: Permanent on disk
- **Features**:
  - **Batch Delete**: Efficient bulk deletion with retries
  - **UUID Sharding**: Automatic 256-shard distribution

### Memory Storage
```typescript
const brain = new Brainy({
  storage: {
    type: 'memory'
  }
})
```
- **Use case**: Tests, ephemeral workloads, single-process caches
- **Performance**: No I/O — all data lives in process memory
- **Persistence**: None — data is lost when the process exits

### Auto
```typescript
const brain = new Brainy({
  storage: {
    type: 'auto',
    rootDirectory: './data'
  }
})
```
`'auto'` picks `'filesystem'` when running on Node.js with a writable `rootDirectory`, and falls back to `'memory'` otherwise.

## Backup and Off-Site Replication

Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `rootDirectory` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:

- `gsutil rsync -r ./data gs://my-bucket/brainy-data`
- `aws s3 sync ./data s3://my-bucket/brainy-data`
- `rclone sync ./data remote:brainy-data`
- Periodic `tar` snapshots to any object store

Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory.

## Metadata Indexing System

### Field Discovery Index
Tracks all unique values for each field:

```json
// __metadata_field_index__field_category.json
{
  "values": {
    "technology": 45,
    "science": 32,
    "business": 28
  },
  "lastUpdated": 1699564234567
}
```

### Value-Based Indexes
Maps field+value combinations to entity IDs:

```json
// __metadata_index__category_technology_chunk0.json
{
  "field": "category",
  "value": "technology",
  "ids": ["uuid1", "uuid2", "uuid3", ...],
  "chunk": 0,
  "total": 45
}
```

### Index Chunking
Large indexes automatically chunk for performance:
- **Chunk size**: 10,000 IDs per chunk
- **Auto-splitting**: Transparent to queries
- **Parallel loading**: Chunks load on demand

## Entity Registry

High-performance deduplication system for streaming data:

### Registry Structure
```json
// __entity_registry__.json
{
  "mappings": {
    "did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
    "handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
  },
  "stats": {
    "totalMappings": 10000,
    "lastSync": 1699564234567
  }
}
```

### Performance Characteristics
- **Lookup**: O(1) in-memory hash map
- **Persistence**: Configurable (memory/storage/hybrid)
- **Cache**: LRU with configurable TTL
- **Sync**: Periodic or on-demand

## Durability

Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `rootDirectory` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).

## Storage Optimization

### 1. Batch Operations

```typescript
// Efficient batch delete
await storage.batchDelete([
  'entities/nouns/vectors/00/00123456-....json',
  'entities/nouns/metadata/00/00123456-....json'
  // ...
])

// Batch writes for performance
await brain.addBatch([
  { content: "item1", metadata: {} },
  { content: "item2", metadata: {} },
  { content: "item3", metadata: {} }
])
// Single transaction, optimized I/O
```

### 2. Caching Strategy

```typescript
// Configure caching
const brain = new Brainy({
  storage: {
    type: 'filesystem',
    rootDirectory: './data',
    cache: {
      enabled: true,
      maxSize: 1000,   // Maximum cached items
      ttl: 300000,     // 5 minutes
      strategy: 'lru'  // Least recently used
    }
  }
})
```

## Concurrent Access

### Locking Mechanism
```typescript
// Automatic locking for write operations
await brain.storage.withLock('resource-id', async () => {
  // Exclusive access to resource
  await brain.storage.saveNoun(id, data)
})
```

### Read-Write Separation
- **Reads**: Non-blocking, parallel
- **Writes**: Serialized with locks
- **Hybrid**: Read-heavy optimization

## Migration and Backup

Backup and restore go through the Db API — see
[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) for the full
recipe book.

### Snapshot (backup)
```typescript
// Instant, self-contained snapshot (hard links on filesystem storage)
const db = brain.now()
await db.persist('/backups/2026-06-11')
await db.release()
```

### Restore
```typescript
// Replace the store's entire state from a snapshot (destructive — confirm required)
await brain.restore('/backups/2026-06-11', { confirm: true })
```

### Move to a new directory
```typescript
// A snapshot directory is a complete store: restore it into a fresh brain
const brain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })
await brain.init()
await brain.restore('/backups/2026-06-11', { confirm: true })
```

## Performance Tuning

### FileSystem Optimizations
- **Directory sharding**: 256 shards spread files across subdirectories
- **Async I/O**: Non-blocking file operations
- **Buffer pooling**: Reuse buffers for efficiency

### Monitoring

```typescript
// Get storage statistics
const stats = await brain.storage.getStatistics()
console.log(stats)
// {
//   totalSize: 1048576,
//   entityCount: 1000,
//   indexSize: 204800,
//   walSize: 10240,
//   cacheHitRate: 0.85
// }
```

## Best Practices

### Choose the Right Adapter
1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence
2. **Single-process production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
3. **Horizontal scaling**: Brainy runs in one process — there is no built-in cluster. Run independent instances behind a service layer and replicate the on-disk artifact with your operator tooling; or run many reader processes against one shared store with a single writer

### Optimize for Your Use Case
1. **Read-heavy**: Enable caching and let the OS page cache do its job
2. **Write-heavy**: Batch operations and tune the cache `maxSize`
3. **Real-time**: FileSystem with periodic snapshots
4. **Archival**: Snapshot `rootDirectory` to cold object storage on a schedule
5. **Large-scale**: Rely on metadata/vector separation + UUID sharding

### Monitor and Maintain
1. Regular statistics collection
2. Watch disk usage and shard balance
3. Index optimization
4. Cache tuning based on hit rates
5. Verify backup runs (test restore quarterly)

## API Reference

See the [Storage API](../api/storage.md) for complete method documentation.

---

**Last Updated**: 2026
**Key Features**: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup