docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
parent
2626ab8d62
commit
adda1570f3
22 changed files with 570 additions and 2658 deletions
|
|
@ -37,10 +37,10 @@ brain.augmentations.register(augmentation)
|
|||
|
||||
### 4. Auto-Configuration ✅
|
||||
```typescript
|
||||
new Brainy({
|
||||
new Brainy({
|
||||
cache: true, // Auto-registers CacheAugmentation
|
||||
index: true, // Auto-registers IndexAugmentation
|
||||
storage: 's3' // Auto-registers S3StorageAugmentation
|
||||
storage: { type: 'filesystem', rootDirectory: './data' } // Auto-registers FileSystemStorageAugmentation
|
||||
})
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
# Brainy Data Storage Architecture
|
||||
|
||||
**Complete file structure reference for all storage backends**
|
||||
**Complete file structure reference**
|
||||
|
||||
This document explains how Brainy stores, indexes, and scales data across all storage backends (GCS, S3, R2, Azure, filesystem, OPFS, memory).
|
||||
This document explains how Brainy stores, indexes, and scales data on disk and in memory.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -22,10 +22,10 @@ This document explains how Brainy stores, indexes, and scales data across all st
|
|||
|
||||
## 1. Complete File Structure
|
||||
|
||||
### v5.11.0 Full Directory Tree
|
||||
### Full Directory Tree
|
||||
|
||||
```
|
||||
brainy-data/ # Root directory (or bucket name for cloud)
|
||||
brainy-data/ # Root directory
|
||||
│
|
||||
├── branches/ # Branch-scoped storage (v5.4.0+, COW always-on)
|
||||
│ ├── main/ # Main branch (default)
|
||||
|
|
@ -34,7 +34,7 @@ brainy-data/ # Root directory (or bucket name f
|
|||
│ │ │ ├── Character/ # Type-first: entities organized by type
|
||||
│ │ │ │ ├── vectors/
|
||||
│ │ │ │ │ ├── 00/ # UUID-based sharding (256 shards)
|
||||
│ │ │ │ │ │ ├── 001234...uuid.json # HNSW vector + connections
|
||||
│ │ │ │ │ │ ├── 001234...uuid.json # Vector + graph connections
|
||||
│ │ │ │ │ │ └── 00abcd...uuid.json
|
||||
│ │ │ │ │ ├── 01/ ... fe/
|
||||
│ │ │ │ │ └── ff/
|
||||
|
|
@ -111,11 +111,11 @@ brainy-data/ # Root directory (or bucket name f
|
|||
├── statistics.json # Global statistics
|
||||
├── counts.json # Entity/verb counts by type
|
||||
│
|
||||
├── hnsw/ # HNSW index metadata
|
||||
├── vector/ # Vector index metadata
|
||||
│ ├── system.json # Entry point, max level
|
||||
│ └── nodes/
|
||||
│ ├── 00/
|
||||
│ │ └── 001234...uuid.json # Per-node HNSW data
|
||||
│ │ └── 001234...uuid.json # Per-node graph data
|
||||
│ ├── 01/ ... fe/
|
||||
│ └── ff/
|
||||
│
|
||||
|
|
@ -151,15 +151,15 @@ Each entity is stored as **2 separate files** for optimal performance.
|
|||
{
|
||||
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
|
||||
"vector": [0.1, 0.2, 0.3, ...], // 384-dimensional embedding
|
||||
"connections": { // HNSW graph connections
|
||||
"connections": { // Vector index graph connections
|
||||
"0": ["uuid1", "uuid2"], // Layer 0 neighbors
|
||||
"1": ["uuid3", "uuid4"] // Layer 1 neighbors
|
||||
},
|
||||
"level": 2 // HNSW max level for this node
|
||||
"level": 2 // Max level for this node
|
||||
}
|
||||
```
|
||||
|
||||
**Purpose**: HNSW graph navigation for semantic search
|
||||
**Purpose**: Vector index navigation for semantic search
|
||||
**Size**: ~4KB per entity (384 dims × 4 bytes × 2.6 overhead)
|
||||
**Scale**: Millions of entities
|
||||
|
||||
|
|
@ -202,7 +202,7 @@ Each relationship is also stored as **2 separate files**.
|
|||
"id": "7b2f5e3c-8d4a-4f1e-9c2b-5a6d7e8f9a0b",
|
||||
"vector": [0.5, 0.3, 0.7, ...], // Relationship embedding
|
||||
"connections": {
|
||||
"0": ["verb-uuid1", "verb-uuid2"] // Verb-to-verb HNSW connections
|
||||
"0": ["verb-uuid1", "verb-uuid2"] // Verb-to-verb vector connections
|
||||
},
|
||||
"level": 1
|
||||
}
|
||||
|
|
@ -334,7 +334,7 @@ Unlike entities and relationships, system metadata consists of **index files** t
|
|||
"Character": 50000,
|
||||
"Place": 30000
|
||||
},
|
||||
"hnswIndexSize": 204800,
|
||||
"vectorIndexSize": 204800,
|
||||
"totalNodes": 100000,
|
||||
"totalEdges": 175000,
|
||||
"lastUpdated": "2025-11-18T..."
|
||||
|
|
@ -369,8 +369,8 @@ Unlike entities and relationships, system metadata consists of **index files** t
|
|||
|
||||
**Purpose**: Fast entity/verb counts by type without scanning storage
|
||||
|
||||
#### HNSW System Metadata
|
||||
**Location**: `_system/hnsw/system.json`
|
||||
#### Vector Index System Metadata
|
||||
**Location**: `_system/vector/system.json`
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
@ -381,10 +381,10 @@ Unlike entities and relationships, system metadata consists of **index files** t
|
|||
}
|
||||
```
|
||||
|
||||
**Purpose**: HNSW index entry point and global parameters
|
||||
**Purpose**: Vector index entry point and global parameters
|
||||
|
||||
#### HNSW Node Data
|
||||
**Location**: `_system/hnsw/nodes/{shard}/{uuid}.json`
|
||||
#### Vector Index Node Data
|
||||
**Location**: `_system/vector/nodes/{shard}/{uuid}.json`
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
@ -398,7 +398,7 @@ Unlike entities and relationships, system metadata consists of **index files** t
|
|||
}
|
||||
```
|
||||
|
||||
**Purpose**: Per-node HNSW graph connections (persisted for fast rebuild)
|
||||
**Purpose**: Per-node vector index graph connections (persisted for fast rebuild)
|
||||
|
||||
#### Field Indexes (Hash Indexes)
|
||||
**Location**: `_system/metadata_indexes/__metadata_field_index__{field}.json`
|
||||
|
|
@ -491,11 +491,11 @@ const tagRefPath = `_cow/refs/tags/${tagName}.json`
|
|||
// System files never use sharding or branching
|
||||
const statsPath = `_system/statistics.json`
|
||||
const countsPath = `_system/counts.json`
|
||||
const hnswSystemPath = `_system/hnsw/system.json`
|
||||
const vectorSystemPath = `_system/vector/system.json`
|
||||
const fieldIndexPath = `_system/metadata_indexes/__metadata_field_index__${fieldName}.json`
|
||||
|
||||
// HNSW node data IS sharded (entity UUID-based)
|
||||
const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
|
||||
// Vector node data IS sharded (entity UUID-based)
|
||||
const vectorNodePath = `_system/vector/nodes/${shard}/${entityId}.json`
|
||||
```
|
||||
|
||||
### Path Patterns Summary
|
||||
|
|
@ -512,8 +512,8 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
|
|||
| **COW ref** | `_cow/refs/heads/{branch}.json` | ❌ No | ❌ No |
|
||||
| **Statistics** | `_system/statistics.json` | ❌ No | ❌ No |
|
||||
| **Counts** | `_system/counts.json` | ❌ No | ❌ No |
|
||||
| **HNSW system** | `_system/hnsw/system.json` | ❌ No | ❌ No |
|
||||
| **HNSW node** | `_system/hnsw/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No |
|
||||
| **Vector system** | `_system/vector/system.json` | ❌ No | ❌ No |
|
||||
| **Vector node** | `_system/vector/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No |
|
||||
| **Field index** | `_system/metadata_indexes/__metadata_field_index__{field}.json` | ❌ No | ❌ No |
|
||||
|
||||
### Key Principles
|
||||
|
|
@ -521,7 +521,7 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
|
|||
1. **Shard Extraction**: Always use first 2 hex characters of UUID/SHA-256
|
||||
2. **ID-First**: Shard + ID come BEFORE type (type is in metadata)
|
||||
3. **Branch Isolation**: Only entity data uses branches/
|
||||
4. **System Isolation**: System files never use sharding or branching (except HNSW nodes)
|
||||
4. **System Isolation**: System files never use sharding or branching (except vector index nodes)
|
||||
5. **Content-Addressable**: COW uses SHA-256 hash as filename
|
||||
|
||||
---
|
||||
|
|
@ -530,7 +530,7 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
|
|||
|
||||
Brainy uses four complementary index systems for different query patterns.
|
||||
|
||||
### 3.1 HNSW Vector Index (In-Memory with Lazy Loading)
|
||||
### 3.1 Vector Index (In-Memory with Lazy Loading)
|
||||
|
||||
**Purpose**: Semantic similarity search
|
||||
**Location**: RAM (rebuilt from storage on startup)
|
||||
|
|
@ -538,7 +538,7 @@ Brainy uses four complementary index systems for different query patterns.
|
|||
|
||||
**How It Works**:
|
||||
1. Loads `branches/{branch}/entities/nouns/{type}/vectors/**/*.json` files
|
||||
2. Builds HNSW graph structure in memory
|
||||
2. Builds vector index graph structure in memory
|
||||
3. Enables O(log n) approximate nearest neighbor search
|
||||
4. Vectors loaded on-demand in lazy mode (zero configuration)
|
||||
|
||||
|
|
@ -634,10 +634,10 @@ const users = await brain.getNouns({
|
|||
|
||||
### 4.1 Why Shard?
|
||||
|
||||
**Cloud Storage Limitations**:
|
||||
- GCS/S3: Listing 100K files in one directory = 10-30 seconds
|
||||
- GCS/S3: Max recommended files per directory = 1,000-10,000
|
||||
- Network: Parallel operations faster than sequential
|
||||
**Filesystem Limitations**:
|
||||
- Listing 100K files in one directory is slow (`readdir` walks the whole entry list)
|
||||
- Most filesystems prefer ≤10,000 entries per directory for fast lookups
|
||||
- Parallel operations across shards beat serial scans
|
||||
|
||||
**Solution**: Split into 256 shards = ~3,900 files per shard at 1M scale
|
||||
|
||||
|
|
@ -786,7 +786,7 @@ _cow/blobs/ab/abc123...sha256.bin (used by both entities)
|
|||
|
||||
### 6.1 What is ID-First?
|
||||
|
||||
**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused 20-21 second delays on cloud storage.
|
||||
**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused long delays when an entity's type was unknown.
|
||||
|
||||
**Old type-first structure** (v5.12.0):
|
||||
```
|
||||
|
|
@ -807,13 +807,13 @@ branches/main/entities/nouns/00/001234...uuid/metadata.json
|
|||
**1. 40x Faster Lookups**
|
||||
```typescript
|
||||
// v5.x: Had to search 42 types if type unknown
|
||||
// Result: 21 seconds on GCS (42 types × 500ms)
|
||||
// Result: 21 seconds (42 types × 500ms)
|
||||
|
||||
// Direct path from ID
|
||||
const id = '001234...'
|
||||
const shard = id.substring(0, 2) // '00'
|
||||
const path = `branches/main/entities/nouns/${shard}/${id}/metadata.json`
|
||||
// Result: <500ms on GCS - 40x faster!
|
||||
// Result: <500ms - 40x faster!
|
||||
```
|
||||
|
||||
**2. Simpler Code**
|
||||
|
|
@ -980,9 +980,9 @@ const related = await brain.find('financial projections for Q2')
|
|||
|
||||
## 8. Storage Backend Mapping
|
||||
|
||||
### 8.1 All Backends Use Same Structure
|
||||
### 8.1 Both Adapters Use the Same Structure
|
||||
|
||||
**Filesystem** (local):
|
||||
**Filesystem** (default for Node.js):
|
||||
```
|
||||
/path/to/brainy-data/
|
||||
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
|
||||
|
|
@ -990,70 +990,21 @@ const related = await brain.find('financial projections for Q2')
|
|||
└── _system/statistics.json
|
||||
```
|
||||
|
||||
**Google Cloud Storage** (GCS):
|
||||
```
|
||||
gs://my-bucket/
|
||||
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
|
||||
├── _cow/commits/00/00a1b2c3...sha256.json
|
||||
└── _system/statistics.json
|
||||
```
|
||||
|
||||
**AWS S3** / **MinIO** / **DigitalOcean Spaces**:
|
||||
```
|
||||
s3://my-bucket/
|
||||
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
|
||||
├── _cow/commits/00/00a1b2c3...sha256.json
|
||||
└── _system/statistics.json
|
||||
```
|
||||
|
||||
**Cloudflare R2**:
|
||||
```
|
||||
r2://my-bucket/
|
||||
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
|
||||
├── _cow/commits/00/00a1b2c3...sha256.json
|
||||
└── _system/statistics.json
|
||||
```
|
||||
|
||||
**Azure Blob Storage**:
|
||||
```
|
||||
azure://my-container/
|
||||
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
|
||||
├── _cow/commits/00/00a1b2c3...sha256.json
|
||||
└── _system/statistics.json
|
||||
```
|
||||
|
||||
**OPFS** (browser):
|
||||
```
|
||||
opfs://root/brainy/
|
||||
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
|
||||
├── _cow/commits/00/00a1b2c3...sha256.json
|
||||
└── _system/statistics.json
|
||||
```
|
||||
|
||||
**Memory Storage** (in-memory):
|
||||
- Uses same path structure
|
||||
- Stored in `Map<string, any>`
|
||||
- Key = full path (e.g., "branches/main/entities/nouns/Character/vectors/00/001234...uuid.json")
|
||||
|
||||
For off-site backup of the filesystem artifact, snapshot the `rootDirectory` from your scheduler using `gsutil`, `aws s3 sync`, `rclone`, or `tar` — Brainy itself stays local.
|
||||
|
||||
---
|
||||
|
||||
### 8.2 Backend-Specific Optimizations
|
||||
|
||||
**Cloud Storage (GCS, S3, R2, Azure)**:
|
||||
- Lifecycle policies for automatic archival (96% cost savings)
|
||||
- Intelligent-Tiering (S3) or Autoclass (GCS) for access-pattern optimization
|
||||
- Batch operations (1000 objects per request for S3)
|
||||
- Parallel uploads/downloads
|
||||
### 8.2 Adapter Characteristics
|
||||
|
||||
**Filesystem**:
|
||||
- Optional gzip compression (60-80% space savings)
|
||||
- Direct file I/O (fastest for local)
|
||||
- Atomic writes with rename
|
||||
|
||||
**OPFS**:
|
||||
- Quota monitoring (browser storage limits)
|
||||
- Persistent storage (survives page refresh)
|
||||
- Worker-based I/O (non-blocking)
|
||||
- Sharded directory layout keeps `readdir` fast
|
||||
|
||||
**Memory**:
|
||||
- No I/O overhead (instant access)
|
||||
|
|
@ -1090,7 +1041,7 @@ opfs://root/brainy/
|
|||
| Get entity by ID | 15-30s | 100-150ms | **200x faster** |
|
||||
| List all entities | 30-60s | 30-60s | Same |
|
||||
| Filter by metadata | 10-30s | 5-50ms | **100-600x faster** (via indexes) |
|
||||
| Semantic search | N/A | 1-10ms | N/A (requires HNSW) |
|
||||
| Semantic search | N/A | 1-10ms | N/A (requires vector index) |
|
||||
| Type filtering | 30-60s | 120-200ms | **150-500x faster** (type-first) |
|
||||
| Graph query (getVerbsBySource) | O(total_verbs) | <1ms | **O(1) via index** |
|
||||
|
||||
|
|
@ -1113,15 +1064,10 @@ opfs://root/brainy/
|
|||
|
||||
| Storage Backend | Max Entities (No Optimization) | Max Entities (Full Optimization) |
|
||||
|----------------|-------------------------------|----------------------------------|
|
||||
| GCS | ~10,000 | **10M+** |
|
||||
| S3 | ~10,000 | **10M+** |
|
||||
| R2 | ~10,000 | **10M+** |
|
||||
| Azure | ~10,000 | **10M+** |
|
||||
| Filesystem | ~100,000 | **10M+** |
|
||||
| OPFS | ~50,000 | **1M+** (browser limits) |
|
||||
| Memory | Limited by RAM | Limited by RAM |
|
||||
|
||||
**Full optimization** = Sharding + Type-first + COW + Lifecycle policies + Lazy mode
|
||||
**Full optimization** = Sharding + Type-first + COW + Lazy mode
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -1129,7 +1075,7 @@ opfs://root/brainy/
|
|||
|
||||
| Component | Standard Mode | Lazy Mode | Savings |
|
||||
|-----------|---------------|-----------|---------|
|
||||
| **HNSW Index (100K entities)** | 149MB | 15-33MB | 5-10x |
|
||||
| **Vector Index (100K entities)** | 149MB | 15-33MB | 5-10x |
|
||||
| **Graph Index (100K verbs)** | 100MB | 100MB | N/A |
|
||||
| **Metadata Indexes** | 10-50MB | 10-50MB | N/A |
|
||||
| **UnifiedCache** | 2GB | 2GB | N/A |
|
||||
|
|
@ -1149,17 +1095,15 @@ opfs://root/brainy/
|
|||
- Use UUIDs for all entities and relationships
|
||||
- Let Brainy handle sharding automatically (type-first + UUID sharding)
|
||||
- Use metadata indexes for filtering
|
||||
- Enable lifecycle policies for cloud storage (96% cost savings)
|
||||
- Use batch operations for bulk deletions
|
||||
- Enable compression for FileSystem storage (60-80% space savings)
|
||||
- Snapshot `rootDirectory` to off-site storage from your scheduler (`gsutil` / `aws s3 sync` / `rclone`)
|
||||
- Create branches for experimentation (instant, zero-cost)
|
||||
|
||||
❌ **Don't**:
|
||||
- Try to organize files manually
|
||||
- Assume file paths are predictable (use IDs, not paths)
|
||||
- Store large binary data in metadata (use blob storage or VFS)
|
||||
- Disable COW (can't be disabled in v5.11.0+, always enabled)
|
||||
- Forget to monitor OPFS quota in browser applications
|
||||
- Store large binary data in metadata (use VFS for blobs)
|
||||
- Disable COW (always enabled)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -1170,7 +1114,7 @@ opfs://root/brainy/
|
|||
✅ Deletes:
|
||||
- `branches/` → ALL entity data (all types, all shards, all branches, all forks)
|
||||
- `_cow/` → ALL version control (commits, trees, blobs, refs)
|
||||
- `_system/` → ALL indexes (statistics, HNSW, metadata)
|
||||
- `_system/` → ALL indexes (statistics, vector index, metadata)
|
||||
|
||||
✅ Resets:
|
||||
- COW managers (refManager, blobStorage, commitLog) → `undefined`
|
||||
|
|
@ -1237,7 +1181,7 @@ await brain.add({ data: 'Alice', type: 'person' })
|
|||
→ _cow/refs/heads/main.json (points to new commit)
|
||||
10. Update statistics:
|
||||
→ _system/statistics.json (increment person count)
|
||||
11. Update HNSW index (in-memory):
|
||||
11. Update vector index (in-memory):
|
||||
→ Connect to nearest neighbors
|
||||
12. Update graph index (in-memory):
|
||||
→ Add to adjacency maps
|
||||
|
|
@ -1281,7 +1225,7 @@ const results = await brain.find('medieval castle', { k: 10 })
|
|||
**What happens in storage**:
|
||||
```
|
||||
1. Compute query vector: [0.1, 0.2, 0.3, ...]
|
||||
2. Use HNSW index (in-memory):
|
||||
2. Use vector index (in-memory):
|
||||
→ Navigate graph from entry point
|
||||
→ Find 10 nearest neighbors (1-10ms)
|
||||
3. Load vectors from cache or storage:
|
||||
|
|
@ -1363,7 +1307,7 @@ await brain.storage.clear()
|
|||
→ Result: ALL commits, trees, blobs, refs deleted
|
||||
3. Delete all indexes:
|
||||
→ Remove: _system/ (entire directory)
|
||||
→ Result: Statistics, HNSW, metadata indexes deleted
|
||||
→ Result: Statistics, vector index, metadata indexes deleted
|
||||
4. Reset COW managers in memory:
|
||||
→ refManager = undefined
|
||||
→ blobStorage = undefined
|
||||
|
|
@ -1393,13 +1337,13 @@ await brain.init()
|
|||
**What happens in storage**:
|
||||
```
|
||||
1. Check for persisted indexes:
|
||||
→ Load: _system/hnsw/system.json (entry point, max level)
|
||||
→ Load: _system/hnsw/nodes/** (graph connections)
|
||||
→ Load: _system/vector/system.json (entry point, max level)
|
||||
→ Load: _system/vector/nodes/** (graph connections)
|
||||
→ Load: _system/statistics.json (entity counts)
|
||||
2. Decide standard vs lazy mode:
|
||||
→ Check: entityCount × vectorSize vs. available cache
|
||||
→ Auto-enable lazy mode if needed
|
||||
3. Rebuild HNSW index:
|
||||
3. Rebuild vector index:
|
||||
→ Standard mode: Load all vectors into memory
|
||||
→ Lazy mode: Load only graph structure (~24 bytes/node)
|
||||
4. Rebuild Graph Adjacency index:
|
||||
|
|
@ -1441,7 +1385,7 @@ await brain.addBatch([
|
|||
→ 1 tree object (or tree fan-out for large trees)
|
||||
→ 10,000+ blobs (deduplicated)
|
||||
5. Update indexes in batch:
|
||||
→ HNSW: Batch insert (optimized)
|
||||
→ Vector index: Batch insert (optimized)
|
||||
→ Graph: Batch update
|
||||
→ Metadata: Batch index update
|
||||
```
|
||||
|
|
@ -1455,25 +1399,23 @@ await brain.addBatch([
|
|||
**Complete Storage Structure**:
|
||||
- **3 storage layers**: branches/ (data), _cow/ (versions), _system/ (indexes)
|
||||
- **2 files per entity**: metadata.json + vector.json (optimized I/O)
|
||||
- **4 indexes**: HNSW (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields)
|
||||
- **4 indexes**: Vector (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields)
|
||||
- **256 shards**: UUID-based (uniform distribution)
|
||||
- **42 noun types + 127 verb types**: Type is metadata, not storage structure
|
||||
- **Git-like COW**: Branches, commits, trees, blobs, refs
|
||||
- **VFS support**: Traditional file/folder hierarchies
|
||||
|
||||
**Scalability (v6.0.0 Improvements)**:
|
||||
- **ID-First Storage**: 40x faster on cloud storage (eliminates 42-type search)
|
||||
- Sharding: 200x faster for cloud storage
|
||||
**Scalability**:
|
||||
- **ID-First Storage**: 40x faster directory lookups (eliminates 42-type search)
|
||||
- Sharding: 200x faster file lookups
|
||||
- Type filtering: Still O(type_count) via metadata index
|
||||
- Lazy mode: 5-10x less memory for large datasets
|
||||
- COW: Instant branches, efficient forks
|
||||
- Deduplication: 30-90% storage savings
|
||||
|
||||
**Production Features**:
|
||||
- Lifecycle policies (96% cost savings on cloud storage)
|
||||
- Batch operations (efficient API usage)
|
||||
- Compression (60-80% space savings on filesystem)
|
||||
- Quota monitoring (OPFS browser limits)
|
||||
- Batch operations (efficient I/O)
|
||||
- Operator-layer backup via `gsutil` / `aws s3 sync` / `rclone` against `rootDirectory`
|
||||
- Auto-reinitialization (COW always-on, can't be broken)
|
||||
- **Clean architecture**: Removed 500+ lines of type cache complexity
|
||||
|
||||
|
|
@ -1481,7 +1423,7 @@ await brain.addBatch([
|
|||
|
||||
## Next Steps
|
||||
|
||||
- [Storage Adapters](./storage-architecture.md) - Configure cloud storage backends
|
||||
- [Storage Architecture](./storage-architecture.md) - Adapter and backup details
|
||||
- [VFS Guide](../vfs/README.md) - Use Virtual File System features
|
||||
- [Triple Intelligence](../vfs/TRIPLE_INTELLIGENCE.md) - Semantic file extraction
|
||||
- [Scaling Guide](../SCALING.md) - Handle 10M+ entities
|
||||
|
|
@ -1489,6 +1431,5 @@ await brain.addBatch([
|
|||
|
||||
---
|
||||
|
||||
**Version**: v6.0.0
|
||||
**Last Updated**: 2025-11-19
|
||||
**Key Features**: ID-first storage, COW always-on, metadata-based type index, 4-index architecture, VFS support, billion-scale optimization, 40x cloud performance improvement
|
||||
**Last Updated**: 2026
|
||||
**Key Features**: ID-first storage, COW always-on, metadata-based type index, 4-index architecture, VFS support, billion-scale optimization
|
||||
|
|
|
|||
|
|
@ -1,550 +0,0 @@
|
|||
# 🏗️ Distributed Storage Architecture
|
||||
|
||||
> **Technical deep-dive**: How Brainy coordinates storage across multiple nodes and adapters
|
||||
|
||||
## Storage Adapter Layer
|
||||
|
||||
### Base Storage Interface
|
||||
|
||||
Every storage adapter implements this interface:
|
||||
|
||||
```typescript
|
||||
interface StorageAdapter {
|
||||
// Basic operations
|
||||
get(key: string): Promise<any>
|
||||
set(key: string, value: any): Promise<void>
|
||||
delete(key: string): Promise<void>
|
||||
|
||||
// Batch operations
|
||||
getBatch(keys: string[]): Promise<Map<string, any>>
|
||||
setBatch(items: Map<string, any>): Promise<void>
|
||||
|
||||
// Atomic operations (for coordination)
|
||||
compareAndSwap(key: string, oldVal: any, newVal: any): Promise<boolean>
|
||||
increment(key: string, delta: number): Promise<number>
|
||||
|
||||
// Namespace support
|
||||
withNamespace(namespace: string): StorageAdapter
|
||||
}
|
||||
```
|
||||
|
||||
### Storage Coordination Strategies
|
||||
|
||||
#### Strategy 1: Isolated Storage (Default)
|
||||
|
||||
Each node has completely separate storage:
|
||||
|
||||
```
|
||||
Node-1 → Local FS: /data/node1/
|
||||
└── shards/
|
||||
├── shard-001/
|
||||
├── shard-045/
|
||||
└── shard-127/
|
||||
|
||||
Node-2 → Local FS: /data/node2/
|
||||
└── shards/
|
||||
├── shard-023/
|
||||
├── shard-067/
|
||||
└── shard-089/
|
||||
```
|
||||
|
||||
**Coordination**: Via network messages only
|
||||
- Shard ownership tracked in distributed consensus
|
||||
- Data transfer via direct node-to-node communication
|
||||
- No storage-level conflicts possible
|
||||
|
||||
#### Strategy 2: Shared Storage with Namespacing
|
||||
|
||||
Multiple nodes share storage but use namespaces:
|
||||
|
||||
```
|
||||
S3 Bucket: brainy-cluster/
|
||||
├── node-abc123/
|
||||
│ ├── shards/
|
||||
│ └── wal/
|
||||
├── node-def456/
|
||||
│ ├── shards/
|
||||
│ └── wal/
|
||||
└── _cluster/
|
||||
├── topology.json
|
||||
├── shard-map.json
|
||||
└── elections/
|
||||
```
|
||||
|
||||
**Coordination**: Via storage-level atomic operations
|
||||
- Each node owns its namespace
|
||||
- Cluster metadata in shared `_cluster/` namespace
|
||||
- Atomic operations for leader election
|
||||
- Conditional writes prevent conflicts
|
||||
|
||||
#### Strategy 3: Shared Storage with Fine-Grained Locking
|
||||
|
||||
Advanced mode for full shared storage:
|
||||
|
||||
```
|
||||
S3 Bucket: brainy-shared/
|
||||
├── shards/
|
||||
│ ├── 001/
|
||||
│ │ ├── data.bin
|
||||
│ │ └── .lock (atomic)
|
||||
│ ├── 002/
|
||||
│ │ ├── data.bin
|
||||
│ │ └── .lock
|
||||
└── metadata/
|
||||
├── index/
|
||||
└── locks/
|
||||
```
|
||||
|
||||
**Coordination**: Via distributed locking
|
||||
- Shard-level locks using atomic operations
|
||||
- Lock acquisition via compare-and-swap
|
||||
- Automatic lock expiry (lease-based)
|
||||
- Deadlock detection and recovery
|
||||
|
||||
## Storage Adapter Implementations
|
||||
|
||||
### 1. Filesystem Adapter
|
||||
|
||||
```typescript
|
||||
class FilesystemAdapter implements StorageAdapter {
|
||||
constructor(private basePath: string) {}
|
||||
|
||||
async get(key: string) {
|
||||
const path = this.keyToPath(key)
|
||||
return fs.readFile(path, 'json')
|
||||
}
|
||||
|
||||
async compareAndSwap(key: string, oldVal: any, newVal: any) {
|
||||
// Use file locking for atomicity
|
||||
const lockfile = `${this.keyToPath(key)}.lock`
|
||||
await flock(lockfile, 'ex') // Exclusive lock
|
||||
try {
|
||||
const current = await this.get(key)
|
||||
if (deepEqual(current, oldVal)) {
|
||||
await this.set(key, newVal)
|
||||
return true
|
||||
}
|
||||
return false
|
||||
} finally {
|
||||
await funlock(lockfile)
|
||||
}
|
||||
}
|
||||
|
||||
withNamespace(ns: string) {
|
||||
return new FilesystemAdapter(path.join(this.basePath, ns))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. S3 Adapter
|
||||
|
||||
```typescript
|
||||
class S3Adapter implements StorageAdapter {
|
||||
constructor(
|
||||
private bucket: string,
|
||||
private prefix: string = ''
|
||||
) {}
|
||||
|
||||
async get(key: string) {
|
||||
const result = await s3.getObject({
|
||||
Bucket: this.bucket,
|
||||
Key: `${this.prefix}${key}`
|
||||
})
|
||||
return JSON.parse(result.Body)
|
||||
}
|
||||
|
||||
async compareAndSwap(key: string, oldVal: any, newVal: any) {
|
||||
// Use S3's conditional writes
|
||||
const fullKey = `${this.prefix}${key}`
|
||||
|
||||
// Get current version
|
||||
const head = await s3.headObject({
|
||||
Bucket: this.bucket,
|
||||
Key: fullKey
|
||||
})
|
||||
|
||||
// Conditional put with ETag
|
||||
try {
|
||||
await s3.putObject({
|
||||
Bucket: this.bucket,
|
||||
Key: fullKey,
|
||||
Body: JSON.stringify(newVal),
|
||||
IfMatch: head.ETag // Only succeeds if unchanged
|
||||
})
|
||||
return true
|
||||
} catch (err) {
|
||||
if (err.code === 'PreconditionFailed') {
|
||||
return false
|
||||
}
|
||||
throw err
|
||||
}
|
||||
}
|
||||
|
||||
withNamespace(ns: string) {
|
||||
const newPrefix = `${this.prefix}${ns}/`
|
||||
return new S3Adapter(this.bucket, newPrefix)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Cloudflare R2 Adapter
|
||||
|
||||
```typescript
|
||||
class R2Adapter implements StorageAdapter {
|
||||
// Similar to S3 but with R2-specific optimizations
|
||||
|
||||
async compareAndSwap(key: string, oldVal: any, newVal: any) {
|
||||
// R2 supports conditional headers
|
||||
const response = await fetch(`${this.endpoint}/${key}`, {
|
||||
method: 'PUT',
|
||||
body: JSON.stringify(newVal),
|
||||
headers: {
|
||||
'If-Match': await this.getETag(key)
|
||||
}
|
||||
})
|
||||
return response.ok
|
||||
}
|
||||
|
||||
// R2-specific: Use Workers for edge computing
|
||||
async getWithCache(key: string) {
|
||||
// Check Cloudflare edge cache first
|
||||
const cached = await caches.default.match(key)
|
||||
if (cached) return cached.json()
|
||||
|
||||
// Fallback to R2
|
||||
const value = await this.get(key)
|
||||
|
||||
// Cache at edge
|
||||
await caches.default.put(key, new Response(JSON.stringify(value)))
|
||||
|
||||
return value
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Distributed Coordination Patterns
|
||||
|
||||
### Pattern 1: Leader-Based Coordination
|
||||
|
||||
```typescript
|
||||
class LeaderCoordinator {
|
||||
async acquireShardOwnership(shardId: string) {
|
||||
if (!this.isLeader()) {
|
||||
// Only leader assigns shards
|
||||
return this.requestFromLeader('acquireShard', shardId)
|
||||
}
|
||||
|
||||
// Leader logic
|
||||
const shardMap = await this.storage.get('_cluster/shard-map')
|
||||
if (!shardMap[shardId].owner) {
|
||||
shardMap[shardId].owner = this.nodeId
|
||||
|
||||
// Atomic update
|
||||
const success = await this.storage.compareAndSwap(
|
||||
'_cluster/shard-map',
|
||||
shardMap,
|
||||
{ ...shardMap, [shardId]: { owner: this.nodeId } }
|
||||
)
|
||||
|
||||
if (success) {
|
||||
this.broadcast('shardAssigned', { shardId, owner: this.nodeId })
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: Consensus-Based Coordination
|
||||
|
||||
```typescript
|
||||
class ConsensusCoordinator {
|
||||
async acquireShardOwnership(shardId: string) {
|
||||
// Propose to all nodes
|
||||
const proposal = {
|
||||
type: 'ACQUIRE_SHARD',
|
||||
shardId,
|
||||
nodeId: this.nodeId,
|
||||
term: this.currentTerm
|
||||
}
|
||||
|
||||
// Raft consensus
|
||||
const votes = await this.gatherVotes(proposal)
|
||||
|
||||
if (votes.length > this.nodes.length / 2) {
|
||||
// Majority agreed
|
||||
await this.commitProposal(proposal)
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Storage-Native Coordination
|
||||
|
||||
```typescript
|
||||
class StorageNativeCoordinator {
|
||||
async acquireShardOwnership(shardId: string) {
|
||||
// Use storage adapter's native coordination
|
||||
const lockKey = `_locks/shard-${shardId}`
|
||||
const lease = {
|
||||
owner: this.nodeId,
|
||||
expires: Date.now() + 30000 // 30 second lease
|
||||
}
|
||||
|
||||
// Try to acquire lock atomically
|
||||
const acquired = await this.storage.compareAndSwap(
|
||||
lockKey,
|
||||
null, // Must not exist
|
||||
lease
|
||||
)
|
||||
|
||||
if (acquired) {
|
||||
// Start lease renewal
|
||||
this.startLeaseRenewal(lockKey, lease)
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
private startLeaseRenewal(key: string, lease: any) {
|
||||
setInterval(async () => {
|
||||
const renewed = await this.storage.compareAndSwap(
|
||||
key,
|
||||
lease,
|
||||
{ ...lease, expires: Date.now() + 30000 }
|
||||
)
|
||||
|
||||
if (!renewed) {
|
||||
// Lost lease
|
||||
this.handleLeaseLoss(key)
|
||||
}
|
||||
}, 10000) // Renew every 10s
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Multi-Storage Patterns
|
||||
|
||||
### Hybrid Storage (Hot/Cold)
|
||||
|
||||
```typescript
|
||||
class HybridStorageAdapter implements StorageAdapter {
|
||||
constructor(
|
||||
private hot: StorageAdapter, // Fast SSD
|
||||
private cold: StorageAdapter // Cheap S3
|
||||
) {}
|
||||
|
||||
async get(key: string) {
|
||||
// Try hot storage first
|
||||
const hotValue = await this.hot.get(key).catch(() => null)
|
||||
if (hotValue) {
|
||||
this.updateAccessTime(key)
|
||||
return hotValue
|
||||
}
|
||||
|
||||
// Fallback to cold storage
|
||||
const coldValue = await this.cold.get(key)
|
||||
|
||||
// Promote to hot storage if frequently accessed
|
||||
if (this.shouldPromote(key)) {
|
||||
await this.hot.set(key, coldValue)
|
||||
}
|
||||
|
||||
return coldValue
|
||||
}
|
||||
|
||||
async set(key: string, value: any) {
|
||||
// Write to hot storage
|
||||
await this.hot.set(key, value)
|
||||
|
||||
// Async write to cold storage
|
||||
setImmediate(() => {
|
||||
this.cold.set(key, value).catch(console.error)
|
||||
})
|
||||
}
|
||||
|
||||
// Background process to demote cold data
|
||||
async runTiering() {
|
||||
const hotKeys = await this.hot.listKeys()
|
||||
|
||||
for (const key of hotKeys) {
|
||||
const lastAccess = await this.getAccessTime(key)
|
||||
|
||||
if (Date.now() - lastAccess > 7 * 24 * 60 * 60 * 1000) {
|
||||
// Not accessed in 7 days, demote to cold
|
||||
await this.cold.set(key, await this.hot.get(key))
|
||||
await this.hot.delete(key)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Geo-Distributed Storage
|
||||
|
||||
```typescript
|
||||
class GeoDistributedAdapter implements StorageAdapter {
|
||||
constructor(
|
||||
private regions: Map<string, StorageAdapter>
|
||||
) {}
|
||||
|
||||
async get(key: string) {
|
||||
// Determine closest region
|
||||
const region = await this.getClosestRegion()
|
||||
|
||||
// Try local region first
|
||||
const localValue = await this.regions.get(region)
|
||||
.get(key)
|
||||
.catch(() => null)
|
||||
|
||||
if (localValue) return localValue
|
||||
|
||||
// Fallback to other regions
|
||||
for (const [name, adapter] of this.regions) {
|
||||
if (name !== region) {
|
||||
const value = await adapter.get(key).catch(() => null)
|
||||
if (value) {
|
||||
// Replicate to local region for next time
|
||||
this.regions.get(region).set(key, value)
|
||||
return value
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error('Key not found in any region')
|
||||
}
|
||||
|
||||
async set(key: string, value: any) {
|
||||
// Write to local region immediately
|
||||
const region = await this.getClosestRegion()
|
||||
await this.regions.get(region).set(key, value)
|
||||
|
||||
// Async replication to other regions
|
||||
for (const [name, adapter] of this.regions) {
|
||||
if (name !== region) {
|
||||
adapter.set(key, value).catch(console.error)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Storage Optimization Strategies
|
||||
|
||||
### 1. Write Batching
|
||||
|
||||
```typescript
|
||||
class BatchingAdapter implements StorageAdapter {
|
||||
private writeBatch = new Map()
|
||||
private batchTimer?: NodeJS.Timeout
|
||||
|
||||
async set(key: string, value: any) {
|
||||
this.writeBatch.set(key, value)
|
||||
|
||||
if (!this.batchTimer) {
|
||||
this.batchTimer = setTimeout(() => this.flush(), 100)
|
||||
}
|
||||
|
||||
if (this.writeBatch.size >= 1000) {
|
||||
await this.flush()
|
||||
}
|
||||
}
|
||||
|
||||
private async flush() {
|
||||
if (this.writeBatch.size === 0) return
|
||||
|
||||
const batch = new Map(this.writeBatch)
|
||||
this.writeBatch.clear()
|
||||
|
||||
await this.underlying.setBatch(batch)
|
||||
|
||||
if (this.batchTimer) {
|
||||
clearTimeout(this.batchTimer)
|
||||
this.batchTimer = undefined
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Read Caching
|
||||
|
||||
```typescript
|
||||
class CachingAdapter implements StorageAdapter {
|
||||
private cache = new LRU({ max: 10000 })
|
||||
|
||||
async get(key: string) {
|
||||
// Check cache
|
||||
if (this.cache.has(key)) {
|
||||
return this.cache.get(key)
|
||||
}
|
||||
|
||||
// Read from storage
|
||||
const value = await this.underlying.get(key)
|
||||
|
||||
// Cache for next time
|
||||
this.cache.set(key, value)
|
||||
|
||||
return value
|
||||
}
|
||||
|
||||
async set(key: string, value: any) {
|
||||
// Invalidate cache
|
||||
this.cache.delete(key)
|
||||
|
||||
// Write through
|
||||
await this.underlying.set(key, value)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Compression
|
||||
|
||||
```typescript
|
||||
class CompressingAdapter implements StorageAdapter {
|
||||
async set(key: string, value: any) {
|
||||
const json = JSON.stringify(value)
|
||||
|
||||
// Compress if beneficial
|
||||
if (json.length > 1024) {
|
||||
const compressed = await gzip(json)
|
||||
await this.underlying.set(key, {
|
||||
_compressed: true,
|
||||
data: compressed.toString('base64')
|
||||
})
|
||||
} else {
|
||||
await this.underlying.set(key, value)
|
||||
}
|
||||
}
|
||||
|
||||
async get(key: string) {
|
||||
const stored = await this.underlying.get(key)
|
||||
|
||||
if (stored._compressed) {
|
||||
const compressed = Buffer.from(stored.data, 'base64')
|
||||
const json = await gunzip(compressed)
|
||||
return JSON.parse(json)
|
||||
}
|
||||
|
||||
return stored
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
Brainy's storage layer is designed for:
|
||||
|
||||
1. **Flexibility**: Works with any storage backend
|
||||
2. **Coordination**: Multiple strategies for different needs
|
||||
3. **Performance**: Batching, caching, compression
|
||||
4. **Scalability**: From single file to geo-distributed
|
||||
5. **Simplicity**: Complexity hidden behind simple interface
|
||||
|
||||
The key insight: **Storage is just a plugin**. The intelligence is in the coordination layer above it!
|
||||
|
||||
---
|
||||
|
||||
*For user-facing documentation, see [SCALING.md](../SCALING.md)*
|
||||
|
|
@ -10,14 +10,14 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
|
|||
|
||||
| Index | Purpose | Data Structure | Complexity | File Location | rebuild() Method |
|
||||
|-------|---------|----------------|------------|---------------|------------------|
|
||||
| **TypeAwareHNSWIndex** | Type-aware vector similarity search | 42 type-specific HNSW hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
|
||||
| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
|
||||
| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 |
|
||||
| **GraphAdjacencyIndex** | Relationship traversal | 4 LSM-trees + bidirectional adjacency maps | O(1) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
|
||||
|
||||
### Sub-Indexes (Level 2)
|
||||
|
||||
**TypeAwareHNSWIndex contains:**
|
||||
- **42 type-specific HNSW indexes** - One per NounType (automatically rebuilt via parent)
|
||||
**TypeAwareVectorIndex contains:**
|
||||
- **42 type-specific vector indexes** - One per NounType (automatically rebuilt via parent)
|
||||
|
||||
**MetadataIndexManager contains:**
|
||||
- **ChunkManager** - Adaptive chunked sparse indexing
|
||||
|
|
@ -398,14 +398,16 @@ const DEFAULT_EXCLUDE_FIELDS = [
|
|||
|
||||
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
|
||||
|
||||
## 2. HNSWIndex - Vector Similarity Search
|
||||
## 2. Vector Index - Vector Similarity Search
|
||||
|
||||
**Purpose**: O(log n) semantic similarity search using vector embeddings.
|
||||
|
||||
The default JS implementation is `JsHnswVectorIndex`; an optional native acceleration package (`@soulcraft/cortex`) can register a higher-performing `VectorIndexProvider` through the plugin system. The public API stays the same either way.
|
||||
|
||||
### Internal Architecture
|
||||
|
||||
```typescript
|
||||
class HNSWIndex {
|
||||
class JsHnswVectorIndex {
|
||||
// Per-noun indexes for efficiency
|
||||
private nouns: Map<string, HNSWNoun> = new Map()
|
||||
|
||||
|
|
@ -437,7 +439,7 @@ class HNSWNode {
|
|||
|
||||
### Hierarchical Graph Structure
|
||||
|
||||
HNSW builds a multi-layered graph:
|
||||
The default vector index builds a multi-layered graph:
|
||||
|
||||
```
|
||||
Layer 2: [entry] ←→ [node1] (sparse, long-range connections)
|
||||
|
|
@ -603,7 +605,7 @@ class UnifiedCache {
|
|||
// Each index gets the same cache instance
|
||||
const unifiedCache = new UnifiedCache({ maxSize: 1000 })
|
||||
this.metadataIndex = new MetadataIndexManager(storage, { unifiedCache })
|
||||
this.hnswIndex = new HNSWIndex(storage, { unifiedCache })
|
||||
this.vectorIndex = new JsHnswVectorIndex(storage, { unifiedCache })
|
||||
this.graphIndex = new GraphAdjacencyIndex(storage, { unifiedCache })
|
||||
```
|
||||
|
||||
|
|
@ -623,7 +625,7 @@ Each index uses different key prefixes:
|
|||
// Metadata index
|
||||
cache.set(`meta:${field}:${value}`, indexEntry)
|
||||
|
||||
// HNSW index
|
||||
// Vector index
|
||||
cache.set(`vector:${id}`, vectorData)
|
||||
|
||||
// Graph index
|
||||
|
|
@ -645,7 +647,7 @@ async add(params: AddParams): Promise<string> {
|
|||
// Add to metadata index (field filtering)
|
||||
await this.metadataIndex.addToIndex(id, params.metadata)
|
||||
|
||||
// Add to HNSW index (vector search)
|
||||
// Add to vector index (vector search)
|
||||
await this.index.addEntity(id, vector, params.noun)
|
||||
|
||||
// Relationships added via separate relate() calls
|
||||
|
|
@ -700,7 +702,7 @@ async update(params: UpdateParams): Promise<void> {
|
|||
await this.metadataIndex.removeFromIndex(params.id, existing.metadata)
|
||||
await this.metadataIndex.addToIndex(params.id, params.metadata)
|
||||
|
||||
// Update HNSW index (re-embed if content changed)
|
||||
// Update vector index (re-embed if content changed)
|
||||
if (params.content) {
|
||||
const newVector = await this.embedder(params.content)
|
||||
await this.index.updateEntity(params.id, newVector)
|
||||
|
|
@ -729,7 +731,7 @@ async stats(): Promise<Statistics> {
|
|||
// From deleted items index
|
||||
deletedItems: this.deletedItemsIndex.getDeletedCount(),
|
||||
|
||||
// From HNSW index
|
||||
// From vector index
|
||||
vectorIndexSize: this.index.getSize()
|
||||
}
|
||||
}
|
||||
|
|
@ -746,16 +748,16 @@ async stats(): Promise<Statistics> {
|
|||
async init(): Promise<void> {
|
||||
// When disableAutoRebuild: false (default)
|
||||
const metadataStats = await this.metadataIndex.getStats()
|
||||
const hnswIndexSize = this.index.size()
|
||||
const vectorIndexSize = this.index.size()
|
||||
const graphIndexSize = await this.graphIndex.size()
|
||||
|
||||
if (metadataStats.totalEntries === 0 ||
|
||||
hnswIndexSize === 0 ||
|
||||
vectorIndexSize === 0 ||
|
||||
graphIndexSize === 0) {
|
||||
// Rebuild all indexes in parallel
|
||||
await Promise.all([
|
||||
metadataStats.totalEntries === 0 ? this.metadataIndex.rebuild() : Promise.resolve(),
|
||||
hnswIndexSize === 0 ? this.index.rebuild() : Promise.resolve(),
|
||||
vectorIndexSize === 0 ? this.index.rebuild() : Promise.resolve(),
|
||||
graphIndexSize === 0 ? this.graphIndex.rebuild() : Promise.resolve()
|
||||
])
|
||||
}
|
||||
|
|
@ -795,7 +797,7 @@ The **TripleIntelligenceSystem** (`src/triple/TripleIntelligenceSystem.ts`) comb
|
|||
class TripleIntelligenceSystem {
|
||||
constructor(
|
||||
private metadataIndex: MetadataIndexManager,
|
||||
private hnswIndex: HNSWIndex,
|
||||
private vectorIndex: VectorIndexProvider,
|
||||
private graphIndex: GraphAdjacencyIndex,
|
||||
private embedder: EmbedderFunction,
|
||||
private storage: BaseStorage
|
||||
|
|
@ -808,7 +810,7 @@ class TripleIntelligenceSystem {
|
|||
// Execute across all three indexes
|
||||
const [metadataResults, vectorResults, graphResults] = await Promise.all([
|
||||
this.metadataIndex.getIdsForFilter(parsed.filters),
|
||||
this.hnswIndex.search(parsed.vector, parsed.limit),
|
||||
this.vectorIndex.search(parsed.vector, parsed.limit),
|
||||
this.graphIndex.traverse(parsed.graphConstraints)
|
||||
])
|
||||
|
||||
|
|
@ -822,7 +824,7 @@ class TripleIntelligenceSystem {
|
|||
|
||||
### Operation Complexity by Index
|
||||
|
||||
| Operation | MetadataIndexManager | TypeAwareHNSWIndex | GraphAdjacencyIndex |
|
||||
| Operation | MetadataIndexManager | TypeAwareVectorIndex | GraphAdjacencyIndex |
|
||||
|-----------|---------------------|-------------------|---------------------|
|
||||
| **Add** | O(1) per field | O(log n) | O(1) |
|
||||
| **Remove** | O(1) per field | O(log n) | O(1) |
|
||||
|
|
@ -837,14 +839,14 @@ Where:
|
|||
- n = total number of entities
|
||||
- k = number of matching results
|
||||
|
||||
**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for HNSW).
|
||||
**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for the vector index).
|
||||
|
||||
### Memory Footprint
|
||||
|
||||
| Index | Per-Entity Memory | Notes |
|
||||
|-------|-------------------|-------|
|
||||
| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) |
|
||||
| **TypeAwareHNSWIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
|
||||
| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
|
||||
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional references + metadata in 4 LSM-trees |
|
||||
|
||||
**Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship
|
||||
|
|
@ -868,7 +870,7 @@ All indexes scale gracefully:
|
|||
**Key observations**:
|
||||
- Graph queries stay O(1) regardless of scale
|
||||
- Metadata filtering scales sub-linearly
|
||||
- Vector search degrades gracefully due to HNSW
|
||||
- Vector search degrades gracefully due to the hierarchical index
|
||||
- Combined queries remain fast even at scale
|
||||
|
||||
## Best Practices
|
||||
|
|
@ -881,7 +883,7 @@ All indexes scale gracefully:
|
|||
- Field discovery (what filters are available)
|
||||
- Type-based querying (find all characters, all items)
|
||||
|
||||
**HNSWIndex**:
|
||||
**Vector Index**:
|
||||
- Semantic similarity search ("find similar documents")
|
||||
- Content-based retrieval ("find posts about AI")
|
||||
- Fuzzy matching (when exact matches aren't required)
|
||||
|
|
@ -906,9 +908,9 @@ All indexes scale gracefully:
|
|||
### Memory Management
|
||||
|
||||
1. **Configure UnifiedCache appropriately** - Balance between speed and memory
|
||||
2. **Use lazy loading** - HNSW loads vectors on-demand
|
||||
2. **Use lazy loading** - Vector index loads vectors on-demand
|
||||
3. **Monitor cache hit rates** - Adjust cache size if hit rate is low
|
||||
4. **Consider storage adapter** - Memory storage = fastest, S3 = most scalable
|
||||
4. **Consider storage adapter** - Memory = fastest, filesystem = persistent
|
||||
|
||||
## Related Documentation
|
||||
|
||||
|
|
@ -922,13 +924,13 @@ All indexes scale gracefully:
|
|||
|
||||
### Level 1: Main Indexes (3)
|
||||
All have rebuild() methods and are covered by lazy loading:
|
||||
1. **TypeAwareHNSWIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403`
|
||||
1. **TypeAwareVectorIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403`
|
||||
2. **MetadataIndexManager** - `src/utils/metadataIndex.ts:2318`
|
||||
3. **GraphAdjacencyIndex** - `src/graph/graphAdjacencyIndex.ts:389`
|
||||
|
||||
### Level 2: Sub-Indexes (~50+)
|
||||
Automatically managed by parent rebuild():
|
||||
- **42 type-specific HNSW indexes** (one per NounType)
|
||||
- **42 type-specific vector indexes** (one per NounType)
|
||||
- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes)
|
||||
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
|
||||
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Initialization and Rebuild Processes
|
||||
|
||||
This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
||||
This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
||||
|
||||
## Core Principle: All Indexes Are Disk-Based
|
||||
|
||||
|
|
@ -11,7 +11,7 @@ This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, Grap
|
|||
| Index | Persisted Data | Storage Method | Since Version |
|
||||
|-------|---------------|----------------|---------------|
|
||||
| **MetadataIndex** | Field registry + chunked sparse indices with bloom filters + zone maps | `storage.saveMetadata()` | v3.42.0 (chunks), v4.2.1 (registry) |
|
||||
| **HNSWIndex** | Vector embeddings + HNSW graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
|
||||
| **Vector Index** | Vector embeddings + graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
|
||||
| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
|
||||
| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
|
||||
|
||||
|
|
@ -32,7 +32,7 @@ The MetadataIndex now persists two components:
|
|||
- Roaring bitmaps for compressed entity ID storage
|
||||
- Loaded on-demand based on query patterns
|
||||
|
||||
All storage operations use the **StorageAdapter** interface, which works with FileSystem, OPFS, S3, GCS, R2, and Memory backends.
|
||||
All storage operations use the **StorageAdapter** interface, which works with FileSystem and Memory backends.
|
||||
|
||||
## Initialization Process
|
||||
|
||||
|
|
@ -84,12 +84,12 @@ async init(): Promise<void> {
|
|||
|
||||
// STEP 2: Check index sizes (lazy initialization triggers here)
|
||||
const metadataStats = await this.metadataIndex.getStats()
|
||||
const hnswIndexSize = this.index.size()
|
||||
const vectorIndexSize = this.index.size()
|
||||
const graphIndexSize = await this.graphIndex.size()
|
||||
|
||||
// STEP 3: Rebuild empty indexes from storage in parallel
|
||||
if (metadataStats.totalEntries === 0 ||
|
||||
hnswIndexSize === 0 ||
|
||||
vectorIndexSize === 0 ||
|
||||
graphIndexSize === 0) {
|
||||
|
||||
const rebuildStartTime = Date.now()
|
||||
|
|
@ -97,7 +97,7 @@ async init(): Promise<void> {
|
|||
metadataStats.totalEntries === 0
|
||||
? this.metadataIndex.rebuild()
|
||||
: Promise.resolve(),
|
||||
hnswIndexSize === 0
|
||||
vectorIndexSize === 0
|
||||
? this.index.rebuild()
|
||||
: Promise.resolve(),
|
||||
graphIndexSize === 0
|
||||
|
|
@ -207,13 +207,13 @@ private async ensureIndexesLoaded(): Promise<void> {
|
|||
### What "Rebuild" Actually Means
|
||||
|
||||
**IMPORTANT**: "Rebuild" does NOT mean recomputing data. It means:
|
||||
1. **Load persisted data** from storage (HNSW connections, metadata chunks, LSM-tree SSTables)
|
||||
1. **Load persisted data** from storage (vector index connections, metadata chunks, LSM-tree SSTables)
|
||||
2. **Populate in-memory structures** (Maps, Sets, graphs)
|
||||
3. **Apply adaptive caching** (preload vectors if small dataset, lazy load if large)
|
||||
|
||||
**Complexity**: O(N) - linear scan through storage, NOT O(N log N) recomputation!
|
||||
|
||||
### 1. HNSWIndex Rebuild (Correct Pattern)
|
||||
### 1. Vector Index Rebuild (Correct Pattern)
|
||||
|
||||
```typescript
|
||||
// src/hnsw/hnswIndex.ts (lines 809-947)
|
||||
|
|
@ -236,7 +236,7 @@ public async rebuild(options: {
|
|||
const availableCache = this.unifiedCache.getRemainingCapacity()
|
||||
const shouldPreload = vectorMemory < availableCache * 0.3
|
||||
|
||||
// STEP 4: Load entities with persisted HNSW connections
|
||||
// STEP 4: Load entities with persisted vector index connections
|
||||
let hasMore = true
|
||||
let cursor: string | undefined = undefined
|
||||
|
||||
|
|
@ -246,7 +246,7 @@ public async rebuild(options: {
|
|||
})
|
||||
|
||||
for (const nounData of result.items) {
|
||||
// Load HNSW graph data from storage (NOT recomputed!)
|
||||
// Load vector graph data from storage (NOT recomputed!)
|
||||
const hnswData = await this.storage.getHNSWData(nounData.id)
|
||||
|
||||
// Create noun with restored connections
|
||||
|
|
@ -274,14 +274,14 @@ public async rebuild(options: {
|
|||
```
|
||||
|
||||
**Key Points**:
|
||||
- ✅ Loads HNSW connections from storage via `getHNSWData()`
|
||||
- ✅ Loads vector index connections from storage via `getHNSWData()`
|
||||
- ✅ Uses adaptive caching (preload vectors if < 30% of available cache)
|
||||
- ✅ O(N) complexity - just loads existing data
|
||||
- ❌ Does NOT call `addItem()` which would recompute connections (O(N log N))
|
||||
|
||||
### 2. TypeAwareHNSWIndex Rebuild (Fixed in v3.45.0)
|
||||
### 2. TypeAwareVectorIndex Rebuild (Fixed in v3.45.0)
|
||||
|
||||
**Critical Architectural Fix**: TypeAwareHNSWIndex previously had TWO major bugs:
|
||||
**Critical Architectural Fix**: The type-aware vector index previously had TWO major bugs:
|
||||
|
||||
1. **Bug #1**: Called `addItem()` during rebuild → O(N log N) recomputation instead of O(N) loading
|
||||
2. **Bug #2**: Loaded ALL nouns 31 times in parallel (once per type) → O(31*N) complexity causing timeouts
|
||||
|
|
@ -300,7 +300,7 @@ public async rebuild(options?: {
|
|||
index.clear()
|
||||
}
|
||||
|
||||
// STEP 2: Determine preloading strategy (same as HNSWIndex)
|
||||
// STEP 2: Determine preloading strategy (same as vector index)
|
||||
const totalNouns = await this.storage.getNounCount()
|
||||
const vectorMemory = totalNouns * 384 * 4
|
||||
const availableCache = this.unifiedCache.getRemainingCapacity()
|
||||
|
|
@ -319,7 +319,7 @@ public async rebuild(options?: {
|
|||
})
|
||||
|
||||
for (const nounData of result.items) {
|
||||
// CORRECT: Load persisted HNSW data (not recomputed!)
|
||||
// CORRECT: Load persisted vector index data (not recomputed!)
|
||||
const hnswData = await this.storage.getHNSWData(nounData.id)
|
||||
|
||||
const noun = {
|
||||
|
|
@ -533,7 +533,7 @@ const unifiedCache = getGlobalCache() // Singleton, 100MB default
|
|||
// MetadataIndex
|
||||
this.unifiedCache = unifiedCache
|
||||
|
||||
// HNSWIndex
|
||||
// Vector index
|
||||
this.unifiedCache = unifiedCache
|
||||
|
||||
// GraphAdjacencyIndex
|
||||
|
|
@ -549,7 +549,7 @@ this.unifiedCache = unifiedCache
|
|||
|
||||
### Rebuild Times (Typical Hardware)
|
||||
|
||||
| Dataset Size | Metadata | HNSW | Graph | Total (Parallel) |
|
||||
| Dataset Size | Metadata | Vector | Graph | Total (Parallel) |
|
||||
|--------------|----------|------|-------|------------------|
|
||||
| 1K entities | 50ms | 100ms | 30ms | **150ms** |
|
||||
| 10K entities | 200ms | 500ms | 150ms | **600ms** |
|
||||
|
|
@ -563,7 +563,7 @@ this.unifiedCache = unifiedCache
|
|||
| Index | In-Memory Overhead | Disk Storage |
|
||||
|-------|-------------------|--------------|
|
||||
| **MetadataIndex** | ~100 bytes/entity | ~500 bytes/entity (chunks) |
|
||||
| **HNSWIndex** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
|
||||
| **Vector Index** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
|
||||
| **GraphAdjacencyIndex** | ~128 bytes/relationship | ~200 bytes/relationship (LSM-tree) |
|
||||
| **DeletedItemsIndex** | ~40 bytes/deleted ID | ~50 bytes/deleted ID |
|
||||
|
||||
|
|
@ -573,9 +573,9 @@ this.unifiedCache = unifiedCache
|
|||
|
||||
### O(N) vs O(N log N) Comparison
|
||||
|
||||
**Before fix** (TypeAwareHNSWIndex bug):
|
||||
**Before fix** (TypeAwareVectorIndex bug):
|
||||
```typescript
|
||||
// BAD: Recomputes HNSW connections during rebuild
|
||||
// BAD: Recomputes vector index connections during rebuild
|
||||
for (const noun of nouns) {
|
||||
await index.addItem(noun) // O(log N) per item → O(N log N) total
|
||||
}
|
||||
|
|
@ -652,7 +652,7 @@ console.timeEnd('rebuild')
|
|||
|
||||
// For 10K entities:
|
||||
// - Expected: 500-800ms (loading from storage)
|
||||
// - Bug: 5-10 minutes (recomputing HNSW connections)
|
||||
// - Bug: 5-10 minutes (recomputing vector index connections)
|
||||
```
|
||||
|
||||
**Solution**: Ensure index is loading from storage, not calling `addItem()` during rebuild.
|
||||
|
|
@ -706,8 +706,8 @@ console.log('Nouns in storage:', nouns.items.length)
|
|||
## Version History
|
||||
|
||||
- **v5.7.7** (November 2025): Added production-scale lazy loading with `ensureIndexesLoaded()` helper. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added concurrency control (mutex) to prevent duplicate rebuilds from concurrent queries. Added `getIndexStatus()` diagnostic method. Zero-config operation - works automatically.
|
||||
- **v3.45.0** (October 2025): Fixed TypeAwareHNSWIndex.rebuild() to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
|
||||
- **v3.45.0** (October 2025): Fixed type-aware vector index `rebuild()` to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
|
||||
- **v3.44.0** (October 2025): GraphAdjacencyIndex migrated to LSM-tree storage for billion-scale relationships
|
||||
- **v3.42.0** (October 2025): MetadataIndex migrated to chunked sparse indexing
|
||||
- **v3.35.0** (August 2025): HNSW connections first persisted to storage
|
||||
- **v3.35.0** (August 2025): Vector index connections first persisted to storage
|
||||
- **v3.0.0** (September 2025): Initial 3-tier index architecture
|
||||
|
|
|
|||
|
|
@ -6,14 +6,14 @@ Brainy is a multi-dimensional AI database that combines vector similarity, graph
|
|||
|
||||
### Brainy (Main Entry Point)
|
||||
The central orchestrator that manages all subsystems:
|
||||
- **4-Index Architecture**: MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md))
|
||||
- **Storage System**: Universal storage adapters (FileSystem, S3, OPFS, Memory)
|
||||
- **4-Index Architecture**: MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md))
|
||||
- **Storage System**: FileSystem and Memory adapters
|
||||
- **Augmentation System**: Extensible plugin architecture
|
||||
- **Triple Intelligence**: Unified query engine
|
||||
|
||||
### Triple Intelligence Engine
|
||||
Brainy's revolutionary feature that unifies three types of search:
|
||||
- **Vector Search**: Semantic similarity using HNSW indexing
|
||||
- **Vector Search**: Semantic similarity via the pluggable vector index
|
||||
- **Graph Traversal**: Relationship-based queries
|
||||
- **Field Filtering**: Precise metadata filtering with O(1) performance
|
||||
|
||||
|
|
@ -42,12 +42,13 @@ brainy-data/
|
|||
└── locks/ # Concurrent access control
|
||||
```
|
||||
|
||||
### HNSW Index
|
||||
Hierarchical Navigable Small World index for efficient vector search:
|
||||
### Vector Index
|
||||
Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph:
|
||||
- **Performance**: O(log n) search complexity
|
||||
- **Memory Efficient**: Product quantization support
|
||||
- **Scalable**: Handles millions of vectors
|
||||
- **Memory Efficient**: SQ4/SQ8 scalar quantization support
|
||||
- **Scalable**: Handles millions of vectors per process
|
||||
- **Persistent**: Serializable to storage
|
||||
- **Swappable**: Replace with a native implementation (such as `@soulcraft/cortex`) via the plugin system without changing application code
|
||||
|
||||
### Metadata Index Manager
|
||||
High-performance field indexing system:
|
||||
|
|
@ -59,7 +60,7 @@ High-performance field indexing system:
|
|||
## Performance Characteristics
|
||||
|
||||
### Operation Complexity
|
||||
- **Vector Search**: O(log n) via HNSW
|
||||
- **Vector Search**: O(log n) via the vector index
|
||||
- **Field Filtering**: O(1) via inverted indexes
|
||||
- **Graph Traversal**: O(V + E) for breadth-first search
|
||||
- **Add Operation**: O(log n) for index insertion
|
||||
|
|
@ -83,7 +84,6 @@ Brainy's extensible plugin architecture allows for powerful enhancements:
|
|||
### Core Augmentations
|
||||
- **Entity Registry**: High-speed deduplication for streaming data
|
||||
- **Batch Processing**: Optimized bulk operations
|
||||
- **Connection Pool**: Efficient resource management
|
||||
- **Request Deduplicator**: Prevents duplicate processing
|
||||
|
||||
### Creating Custom Augmentations
|
||||
|
|
@ -111,7 +111,7 @@ Multi-layered caching for optimal performance:
|
|||
## Integration Points
|
||||
|
||||
### Key Objects for Extensions
|
||||
- `brain.index`: Access HNSW vector index
|
||||
- `brain.index`: Access the vector index
|
||||
- `brain.metadataIndex`: Access field indexing
|
||||
- `brain.graphIndex`: Access graph adjacency index
|
||||
- `brain.storage`: Access storage layer
|
||||
|
|
|
|||
|
|
@ -1,46 +1,46 @@
|
|||
# Storage Architecture
|
||||
|
||||
> **Updated**: Metadata/vector separation, UUID-based sharding, lifecycle management
|
||||
> **Updated**: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup
|
||||
|
||||
## Storage Structure
|
||||
|
||||
### Architecture: Metadata/Vector Separation
|
||||
|
||||
entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
|
||||
Entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
├── _system/ # System metadata (not sharded)
|
||||
│ ├── statistics.json # Performance metrics
|
||||
│ ├── __metadata_field_index__*.json # Field indexes
|
||||
│ └── __metadata_sorted_index__*.json # Sorted indexes
|
||||
├── _system/ # System metadata (not sharded)
|
||||
│ ├── statistics.json # Performance metrics
|
||||
│ ├── __metadata_field_index__*.json # Field indexes
|
||||
│ └── __metadata_sorted_index__*.json # Sorted indexes
|
||||
│
|
||||
├── entities/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # HNSW graph data (sharded by UUID)
|
||||
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
|
||||
│ │ │ │ ├── 00123456-....json # Vector + HNSW connections
|
||||
│ │ │ │ └── 00abcdef-....json
|
||||
│ │ │ ├── 01/ ... ff/ # 256 shards total
|
||||
│ │ │
|
||||
│ │ └── metadata/ # Business data (sharded by UUID)
|
||||
│ │ ├── 00/
|
||||
│ │ │ ├── 00123456-....json # Entity metadata only
|
||||
│ │ │ └── 00abcdef-....json
|
||||
│ │ ├── 01/ ... ff/
|
||||
│ │
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (sharded)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ │
|
||||
│ └── metadata/ # Relationship data (sharded)
|
||||
│ ├── 00/ ... ff/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # Vector graph data (sharded by UUID)
|
||||
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
|
||||
│ │ │ │ ├── 00123456-....json # Vector + graph connections
|
||||
│ │ │ │ └── 00abcdef-....json
|
||||
│ │ │ ├── 01/ ... ff/ # 256 shards total
|
||||
│ │ │
|
||||
│ │ └── metadata/ # Business data (sharded by UUID)
|
||||
│ │ ├── 00/
|
||||
│ │ │ ├── 00123456-....json # Entity metadata only
|
||||
│ │ │ └── 00abcdef-....json
|
||||
│ │ ├── 01/ ... ff/
|
||||
│ │
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (sharded)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ │
|
||||
│ └── metadata/ # Relationship data (sharded)
|
||||
│ ├── 00/ ... ff/
|
||||
```
|
||||
|
||||
### Why Split Metadata and Vectors?
|
||||
|
||||
**Performance at scale:**
|
||||
- **HNSW operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
|
||||
- **Vector search operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
|
||||
- **Filtering**: Only load metadata during filtering, not vectors
|
||||
- **Pagination**: Load metadata IDs first, fetch vectors/metadata on-demand
|
||||
- **Result**: 60-70% reduction in I/O for typical queries at million-entity scale
|
||||
|
|
@ -52,115 +52,69 @@ brainy-data/
|
|||
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
|
||||
const shard = uuid.substring(0, 2) // "3f"
|
||||
|
||||
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
|
||||
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
|
||||
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- **Uniform distribution**: ~3,900 entities per shard (at 1M scale)
|
||||
- **Cloud storage optimization**: 200x faster than unsharded (30s → 150ms)
|
||||
- **Parallel operations**: Load 256 shards in parallel
|
||||
- **Filesystem optimization**: avoids huge flat directories that bog down `readdir`
|
||||
- **Parallel operations**: walk 256 shards in parallel
|
||||
- **Predictable**: Deterministic shard assignment
|
||||
|
||||
## Storage Adapters
|
||||
|
||||
Brainy provides multiple storage adapters with identical APIs and production features:
|
||||
Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` interface:
|
||||
|
||||
### FileSystem Storage (Node.js)
|
||||
### FileSystem Storage (Node.js, default)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // Gzip compression (60-80% space savings)
|
||||
}
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './data'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Server applications, CLI tools
|
||||
- **Performance**: Direct file I/O with optional compression
|
||||
- **Use case**: Server applications, CLI tools, single-node deployments
|
||||
- **Performance**: Direct file I/O
|
||||
- **Persistence**: Permanent on disk
|
||||
- **Features**:
|
||||
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
|
||||
- **Batch Delete**: Efficient bulk deletion with retries
|
||||
- **UUID Sharding**: Automatic 256-shard distribution
|
||||
- **Batch Delete**: Efficient bulk deletion with retries
|
||||
- **UUID Sharding**: Automatic 256-shard distribution
|
||||
|
||||
### S3 Compatible Storage (AWS, MinIO, R2)
|
||||
### Memory Storage
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucket: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
credentials: {
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
storage: {
|
||||
type: 'memory'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Distributed applications, cloud deployments
|
||||
- **Performance**: Network dependent, with intelligent caching
|
||||
- **Persistence**: Cloud storage durability (99.999999999%)
|
||||
- **Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
|
||||
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
|
||||
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
|
||||
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
|
||||
- **Use case**: Tests, ephemeral workloads, single-process caches
|
||||
- **Performance**: No I/O — all data lives in process memory
|
||||
- **Persistence**: None — data is lost when the process exits
|
||||
|
||||
### Google Cloud Storage (GCS)
|
||||
### Auto
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
bucketName: 'my-brainy-data',
|
||||
keyFilename: './service-account.json' // Or use ADC
|
||||
}
|
||||
storage: {
|
||||
type: 'auto',
|
||||
rootDirectory: './data'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Google Cloud deployments
|
||||
- **Performance**: Global CDN with edge caching
|
||||
- **Persistence**: 99.999999999% durability
|
||||
- **Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
|
||||
- **Autoclass**: Intelligent automatic tier optimization
|
||||
- **Batch Delete**: Efficient bulk operations
|
||||
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
|
||||
`'auto'` picks `'filesystem'` when running on Node.js with a writable `rootDirectory`, and falls back to `'memory'` otherwise.
|
||||
|
||||
### Azure Blob Storage
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'azure',
|
||||
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
|
||||
containerName: 'brainy-data'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Azure cloud deployments
|
||||
- **Performance**: Global replication with CDN
|
||||
- **Persistence**: LRS, ZRS, GRS, RA-GRS options
|
||||
- **Features**:
|
||||
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
|
||||
- **Lifecycle Policies**: Automatic tier transitions and deletions
|
||||
- **Batch Delete**: BlobBatchClient for efficient bulk operations
|
||||
- **Batch Tier Changes**: Move thousands of blobs efficiently
|
||||
- **Archive Rehydration**: Smart rehydration with priority options
|
||||
## Backup and Off-Site Replication
|
||||
|
||||
### Origin Private File System (Browser)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'opfs'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Browser applications, PWAs
|
||||
- **Performance**: Near-native file system speed
|
||||
- **Persistence**: Permanent in browser (with quota limits)
|
||||
- **Features**:
|
||||
- **Quota Monitoring**: Real-time quota tracking and warnings
|
||||
- **Batch Delete**: Efficient bulk deletion
|
||||
- **Storage Status**: Detailed usage/available reporting
|
||||
Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `rootDirectory` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:
|
||||
|
||||
- `gsutil rsync -r ./data gs://my-bucket/brainy-data`
|
||||
- `aws s3 sync ./data s3://my-bucket/brainy-data`
|
||||
- `rclone sync ./data remote:brainy-data`
|
||||
- Periodic `tar` snapshots to any object store
|
||||
|
||||
Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory.
|
||||
|
||||
## Metadata Indexing System
|
||||
|
||||
|
|
@ -170,12 +124,12 @@ Tracks all unique values for each field:
|
|||
```json
|
||||
// __metadata_field_index__field_category.json
|
||||
{
|
||||
"values": {
|
||||
"technology": 45,
|
||||
"science": 32,
|
||||
"business": 28
|
||||
},
|
||||
"lastUpdated": 1699564234567
|
||||
"values": {
|
||||
"technology": 45,
|
||||
"science": 32,
|
||||
"business": 28
|
||||
},
|
||||
"lastUpdated": 1699564234567
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -185,11 +139,11 @@ Maps field+value combinations to entity IDs:
|
|||
```json
|
||||
// __metadata_index__category_technology_chunk0.json
|
||||
{
|
||||
"field": "category",
|
||||
"value": "technology",
|
||||
"ids": ["uuid1", "uuid2", "uuid3", ...],
|
||||
"chunk": 0,
|
||||
"total": 45
|
||||
"field": "category",
|
||||
"value": "technology",
|
||||
"ids": ["uuid1", "uuid2", "uuid3", ...],
|
||||
"chunk": 0,
|
||||
"total": 45
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -207,14 +161,14 @@ High-performance deduplication system for streaming data:
|
|||
```json
|
||||
// __entity_registry__.json
|
||||
{
|
||||
"mappings": {
|
||||
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
|
||||
},
|
||||
"stats": {
|
||||
"totalMappings": 10000,
|
||||
"lastSync": 1699564234567
|
||||
}
|
||||
"mappings": {
|
||||
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
|
||||
},
|
||||
"stats": {
|
||||
"totalMappings": 10000,
|
||||
"lastSync": 1699564234567
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -224,216 +178,46 @@ High-performance deduplication system for streaming data:
|
|||
- **Cache**: LRU with configurable TTL
|
||||
- **Sync**: Periodic or on-demand
|
||||
|
||||
## Durability
|
||||
|
||||
Ensures durability and enables recovery:
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": 1699564234567,
|
||||
"operation": "add",
|
||||
"data": {
|
||||
"id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"content": "...",
|
||||
"metadata": {}
|
||||
},
|
||||
"checksum": "sha256:..."
|
||||
}
|
||||
```
|
||||
|
||||
### Recovery Process
|
||||
2. Replay operations from last checkpoint
|
||||
3. Verify checksums for integrity
|
||||
Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `rootDirectory` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).
|
||||
|
||||
## Storage Optimization
|
||||
|
||||
### 1. Lifecycle Policies (Cloud Storage)
|
||||
|
||||
**Automatic cost optimization through tier transitions:**
|
||||
|
||||
```typescript
|
||||
// S3: Set lifecycle policy for automatic archival
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-data',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
|
||||
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
|
||||
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// GCS: Set lifecycle policy
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
condition: { age: 30 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||||
}, {
|
||||
condition: { age: 90 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||||
}, {
|
||||
condition: { age: 365 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||||
}]
|
||||
})
|
||||
|
||||
// Azure: Set lifecycle policy
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
name: 'archiveOldData',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: { blobTypes: ['blockBlob'] },
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 30 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 90 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
**Cost Impact (500TB dataset):**
|
||||
| Storage | Before | After | Savings |
|
||||
|---------|--------|-------|---------|
|
||||
| **AWS S3** | $138,000/yr | $5,940/yr | **96%** |
|
||||
| **GCS** | $138,000/yr | $8,300/yr | **94%** |
|
||||
| **Azure** | $107,520/yr | $5,016/yr | **95%** |
|
||||
|
||||
### 2. Intelligent-Tiering (S3)
|
||||
|
||||
**Automatic optimization without retrieval fees:**
|
||||
|
||||
```typescript
|
||||
// Enable S3 Intelligent-Tiering
|
||||
await storage.enableIntelligentTiering('entities/', 'auto-optimize')
|
||||
|
||||
// Benefits:
|
||||
// - Automatic tier transitions based on access patterns
|
||||
// - No retrieval fees (unlike Glacier)
|
||||
// - Up to 95% cost savings
|
||||
// - No performance impact on frequently accessed data
|
||||
```
|
||||
|
||||
### 3. Autoclass (GCS)
|
||||
|
||||
**Google Cloud's intelligent automatic optimization:**
|
||||
|
||||
```typescript
|
||||
// Enable GCS Autoclass
|
||||
await storage.enableAutoclass({
|
||||
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
|
||||
})
|
||||
|
||||
// Benefits:
|
||||
// - Automatic optimization based on access patterns
|
||||
// - No data retrieval delays
|
||||
// - Transparent tier transitions
|
||||
// - Up to 94% cost savings
|
||||
```
|
||||
|
||||
### 4. Compression (FileSystem)
|
||||
|
||||
```typescript
|
||||
// Enable gzip compression for local storage
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // 60-80% space savings
|
||||
}
|
||||
})
|
||||
|
||||
// Performance impact:
|
||||
// - Write: +10-20ms per file (gzip compression)
|
||||
// - Read: +5-10ms per file (gzip decompression)
|
||||
// - Space savings: 60-80% for typical JSON data
|
||||
// - CPU overhead: Minimal (~5% CPU)
|
||||
```
|
||||
|
||||
### 5. Batch Operations
|
||||
### 1. Batch Operations
|
||||
|
||||
```typescript
|
||||
// Efficient batch delete
|
||||
await storage.batchDelete([
|
||||
'entities/nouns/vectors/00/00123456-....json',
|
||||
'entities/nouns/metadata/00/00123456-....json',
|
||||
// ... up to 1000 objects
|
||||
'entities/nouns/vectors/00/00123456-....json',
|
||||
'entities/nouns/metadata/00/00123456-....json'
|
||||
// ...
|
||||
])
|
||||
|
||||
// Benefits:
|
||||
// - S3: 1000 objects per request (vs 1 per request)
|
||||
// - GCS: 100 objects per request
|
||||
// - Azure: 256 objects per batch
|
||||
// - Automatic retry logic with exponential backoff
|
||||
// - Throttling protection
|
||||
|
||||
// Batch writes for performance
|
||||
await brain.addBatch([
|
||||
{ content: "item1", metadata: {} },
|
||||
{ content: "item2", metadata: {} },
|
||||
{ content: "item3", metadata: {} }
|
||||
{ content: "item1", metadata: {} },
|
||||
{ content: "item2", metadata: {} },
|
||||
{ content: "item3", metadata: {} }
|
||||
])
|
||||
// Single transaction, optimized I/O
|
||||
```
|
||||
|
||||
### 6. Quota Monitoring (OPFS)
|
||||
### 2. Caching Strategy
|
||||
|
||||
```typescript
|
||||
// Get quota status for browser storage
|
||||
const status = await storage.getStorageStatus()
|
||||
|
||||
console.log(status)
|
||||
// {
|
||||
// type: 'opfs',
|
||||
// available: true,
|
||||
// details: {
|
||||
// usage: 45829120, // 43.7 MB used
|
||||
// quota: 536870912, // 512 MB available
|
||||
// usagePercent: 8.5,
|
||||
// quotaExceeded: false
|
||||
// }
|
||||
// }
|
||||
|
||||
// Proactive quota management:
|
||||
// - Monitor usage before writes
|
||||
// - Warn users when approaching quota
|
||||
// - Automatically clean up old data
|
||||
```
|
||||
|
||||
### 7. Tier Management (Azure)
|
||||
|
||||
```typescript
|
||||
// Change blob tier for cost optimization
|
||||
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
|
||||
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
|
||||
|
||||
// Batch tier changes (efficient)
|
||||
await storage.batchChangeTier([blob1, blob2, blob3], 'Cool')
|
||||
|
||||
// Rehydrate from Archive when needed
|
||||
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
|
||||
```
|
||||
|
||||
### 8. Caching Strategy
|
||||
|
||||
```typescript
|
||||
// Configure caching per storage type
|
||||
// Configure caching
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
ttl: 300000, // 5 minutes
|
||||
strategy: 'lru' // Least recently used
|
||||
}
|
||||
}
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './data',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
ttl: 300000, // 5 minutes
|
||||
strategy: 'lru' // Least recently used
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -443,8 +227,8 @@ const brain = new Brainy({
|
|||
```typescript
|
||||
// Automatic locking for write operations
|
||||
await brain.storage.withLock('resource-id', async () => {
|
||||
// Exclusive access to resource
|
||||
await brain.storage.saveNoun(id, data)
|
||||
// Exclusive access to resource
|
||||
await brain.storage.saveNoun(id, data)
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -459,9 +243,9 @@ await brain.storage.withLock('resource-id', async () => {
|
|||
```typescript
|
||||
// Export entire database
|
||||
const backup = await brain.export({
|
||||
format: 'json',
|
||||
includeVectors: true,
|
||||
includeIndexes: false
|
||||
format: 'json',
|
||||
includeVectors: true,
|
||||
includeIndexes: false
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -469,16 +253,16 @@ const backup = await brain.export({
|
|||
```typescript
|
||||
// Import from backup
|
||||
await brain.import(backup, {
|
||||
mode: 'merge', // or 'replace'
|
||||
validateSchema: true
|
||||
mode: 'merge', // or 'replace'
|
||||
validateSchema: true
|
||||
})
|
||||
```
|
||||
|
||||
### Storage Migration
|
||||
```typescript
|
||||
// Migrate between storage types
|
||||
const oldBrain = new Brainy({ storage: { type: 'filesystem' } })
|
||||
const newBrain = new Brainy({ storage: { type: 's3' } })
|
||||
const oldBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './old' } })
|
||||
const newBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })
|
||||
|
||||
await oldBrain.init()
|
||||
await newBrain.init()
|
||||
|
|
@ -490,23 +274,11 @@ await newBrain.import(data)
|
|||
|
||||
## Performance Tuning
|
||||
|
||||
### Storage-Specific Optimizations
|
||||
|
||||
#### FileSystem
|
||||
- **Directory sharding**: Split files across subdirectories
|
||||
### FileSystem Optimizations
|
||||
- **Directory sharding**: 256 shards spread files across subdirectories
|
||||
- **Async I/O**: Non-blocking file operations
|
||||
- **Buffer pooling**: Reuse buffers for efficiency
|
||||
|
||||
#### S3
|
||||
- **Multipart uploads**: For large objects
|
||||
- **Request batching**: Combine small operations
|
||||
- **CDN integration**: Edge caching for reads
|
||||
|
||||
#### OPFS
|
||||
- **Quota management**: Monitor and request increases
|
||||
- **Worker offloading**: Heavy operations in workers
|
||||
- **Transaction batching**: Group operations
|
||||
|
||||
### Monitoring
|
||||
|
||||
```typescript
|
||||
|
|
@ -514,58 +286,34 @@ await newBrain.import(data)
|
|||
const stats = await brain.storage.getStatistics()
|
||||
console.log(stats)
|
||||
// {
|
||||
// totalSize: 1048576,
|
||||
// entityCount: 1000,
|
||||
// indexSize: 204800,
|
||||
// walSize: 10240,
|
||||
// cacheHitRate: 0.85
|
||||
// totalSize: 1048576,
|
||||
// entityCount: 1000,
|
||||
// indexSize: 204800,
|
||||
// walSize: 10240,
|
||||
// cacheHitRate: 0.85
|
||||
// }
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Choose the Right Adapter
|
||||
1. **Development**: FileSystem with compression (local persistence, small storage footprint)
|
||||
2. **Production Server**: FileSystem with compression or cloud storage with lifecycle policies
|
||||
3. **Browser Apps**: OPFS with quota monitoring
|
||||
4. **Distributed**: S3/GCS/Azure with Intelligent-Tiering/Autoclass
|
||||
1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence
|
||||
2. **Single-node production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
|
||||
3. **Multi-node production**: Run Brainy behind a service layer; replicate the on-disk artifact via your operator tooling
|
||||
|
||||
### Optimize for Your Use Case
|
||||
1. **Read-heavy**: Enable aggressive caching + cloud CDN
|
||||
2. **Write-heavy**: Batch operations + async writes
|
||||
1. **Read-heavy**: Enable caching and let the OS page cache do its job
|
||||
2. **Write-heavy**: Batch operations and tune the cache `maxSize`
|
||||
3. **Real-time**: FileSystem with periodic snapshots
|
||||
4. **Archival**: Cloud storage with lifecycle policies (96% cost savings!)
|
||||
5. **Large-scale**: Metadata/vector separation + UUID sharding + lifecycle policies
|
||||
|
||||
### Cost Optimization
|
||||
1. **Enable lifecycle policies** for cloud storage (automated cost reduction)
|
||||
2. **Use Intelligent-Tiering (S3)** or Autoclass (GCS) for automatic optimization
|
||||
3. **Enable compression** for FileSystem storage (60-80% space savings)
|
||||
4. **Monitor quota** for OPFS (prevent quota exceeded errors)
|
||||
5. **Use batch operations** for bulk deletions (efficient API usage)
|
||||
6. **Consider tier management** for Azure (Hot/Cool/Archive tiers)
|
||||
|
||||
**Example Cost Savings (500TB dataset):**
|
||||
- Without lifecycle policies: **$138,000/year**
|
||||
- With lifecycle policies: **$5,940/year**
|
||||
- **Savings: $132,060/year (96%)**
|
||||
4. **Archival**: Snapshot `rootDirectory` to cold object storage on a schedule
|
||||
5. **Large-scale**: Rely on metadata/vector separation + UUID sharding
|
||||
|
||||
### Monitor and Maintain
|
||||
1. Regular statistics collection
|
||||
2. Monitor lifecycle policy effectiveness
|
||||
2. Watch disk usage and shard balance
|
||||
3. Index optimization
|
||||
4. Cache tuning based on hit rates
|
||||
5. Track storage costs and tier distribution
|
||||
6. Review quota usage (OPFS) and storage growth patterns
|
||||
|
||||
### Production Deployment Checklist
|
||||
- ✅ Enable lifecycle policies on cloud storage
|
||||
- ✅ Configure batch delete for cleanup operations
|
||||
- ✅ Enable compression for FileSystem storage
|
||||
- ✅ Set up quota monitoring for OPFS
|
||||
- ✅ Configure appropriate tier transitions
|
||||
- ✅ Enable Intelligent-Tiering (S3) or Autoclass (GCS)
|
||||
- ✅ Monitor storage costs and optimize regularly
|
||||
5. Verify backup runs (test restore quarterly)
|
||||
|
||||
## API Reference
|
||||
|
||||
|
|
@ -573,6 +321,5 @@ See the [Storage API](../api/storage.md) for complete method documentation.
|
|||
|
||||
---
|
||||
|
||||
**Version**: 4.0.0
|
||||
**Last Updated**: 2025-10-17
|
||||
**Key Features**: Metadata/vector separation, UUID sharding, lifecycle management, tier optimization
|
||||
**Last Updated**: 2026
|
||||
**Key Features**: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue