soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

docs(8.0): Phase F — deep clean across 21 docs

Browse source 
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.

Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
  cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
  HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
  sections replaced with single-node vector tuning + filesystem framing.

Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md

Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md

MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
David Snelling 2026-06-09 16:13:35 -07:00
parent 2626ab8d62
commit adda1570f3
22 changed files with 570 additions and 2658 deletions
Download patch file Download diff file Expand all files Collapse all files

183
docs/architecture/data-storage-architecture.md
View file

@ -1,8 +1,8 @@
# Brainy Data Storage Architecture
**Complete file structure reference for all storage backends**
**Complete file structure reference**
This document explains how Brainy stores, indexes, and scales data across all storage backends (GCS, S3, R2, Azure, filesystem, OPFS, memory).
This document explains how Brainy stores, indexes, and scales data on disk and in memory.
---
@ -22,10 +22,10 @@ This document explains how Brainy stores, indexes, and scales data across all st
## 1. Complete File Structure
### v5.11.0 Full Directory Tree
### Full Directory Tree
```
brainy-data/ # Root directory (or bucket name for cloud)
brainy-data/ # Root directory
│
├── branches/ # Branch-scoped storage (v5.4.0+, COW always-on)
│ ├── main/ # Main branch (default)
@ -34,7 +34,7 @@ brainy-data/ # Root directory (or bucket name f
│ │ │ ├── Character/ # Type-first: entities organized by type
│ │ │ │ ├── vectors/
│ │ │ │ │ ├── 00/ # UUID-based sharding (256 shards)
│ │ │ │ │ │ ├── 001234...uuid.json # HNSW vector + connections
│ │ │ │ │ │ ├── 001234...uuid.json # Vector + graph connections
│ │ │ │ │ │ └── 00abcd...uuid.json
│ │ │ │ │ ├── 01/ ... fe/
│ │ │ │ │ └── ff/
@ -111,11 +111,11 @@ brainy-data/ # Root directory (or bucket name f
├── statistics.json # Global statistics
├── counts.json # Entity/verb counts by type
│
├── hnsw/ # HNSW index metadata
├── vector/ # Vector index metadata
│ ├── system.json # Entry point, max level
│ └── nodes/
│ ├── 00/
│ │ └── 001234...uuid.json # Per-node HNSW data
│ │ └── 001234...uuid.json # Per-node graph data
│ ├── 01/ ... fe/
│ └── ff/
│
@ -151,15 +151,15 @@ Each entity is stored as **2 separate files** for optimal performance.
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"vector": [0.1, 0.2, 0.3, ...], // 384-dimensional embedding
"connections": { // HNSW graph connections
"connections": { // Vector index graph connections
"0": ["uuid1", "uuid2"], // Layer 0 neighbors
"1": ["uuid3", "uuid4"] // Layer 1 neighbors
},
"level": 2 // HNSW max level for this node
"level": 2 // Max level for this node
}
```
**Purpose**: HNSW graph navigation for semantic search
**Purpose**: Vector index navigation for semantic search
**Size**: ~4KB per entity (384 dims × 4 bytes × 2.6 overhead)
**Scale**: Millions of entities
@ -202,7 +202,7 @@ Each relationship is also stored as **2 separate files**.
"id": "7b2f5e3c-8d4a-4f1e-9c2b-5a6d7e8f9a0b",
"vector": [0.5, 0.3, 0.7, ...], // Relationship embedding
"connections": {
"0": ["verb-uuid1", "verb-uuid2"] // Verb-to-verb HNSW connections
"0": ["verb-uuid1", "verb-uuid2"] // Verb-to-verb vector connections
},
"level": 1
}
@ -334,7 +334,7 @@ Unlike entities and relationships, system metadata consists of **index files** t
"Character": 50000,
"Place": 30000
},
"hnswIndexSize": 204800,
"vectorIndexSize": 204800,
"totalNodes": 100000,
"totalEdges": 175000,
"lastUpdated": "2025-11-18T..."
@ -369,8 +369,8 @@ Unlike entities and relationships, system metadata consists of **index files** t
**Purpose**: Fast entity/verb counts by type without scanning storage
#### HNSW System Metadata
**Location**: `_system/hnsw/system.json`
#### Vector Index System Metadata
**Location**: `_system/vector/system.json`
```json
{
@ -381,10 +381,10 @@ Unlike entities and relationships, system metadata consists of **index files** t
}
```
**Purpose**: HNSW index entry point and global parameters
**Purpose**: Vector index entry point and global parameters
#### HNSW Node Data
**Location**: `_system/hnsw/nodes/{shard}/{uuid}.json`
#### Vector Index Node Data
**Location**: `_system/vector/nodes/{shard}/{uuid}.json`
```json
{
@ -398,7 +398,7 @@ Unlike entities and relationships, system metadata consists of **index files** t
}
```
**Purpose**: Per-node HNSW graph connections (persisted for fast rebuild)
**Purpose**: Per-node vector index graph connections (persisted for fast rebuild)
#### Field Indexes (Hash Indexes)
**Location**: `_system/metadata_indexes/__metadata_field_index__{field}.json`
@ -491,11 +491,11 @@ const tagRefPath = `_cow/refs/tags/${tagName}.json`
// System files never use sharding or branching
const statsPath = `_system/statistics.json`
const countsPath = `_system/counts.json`
const hnswSystemPath = `_system/hnsw/system.json`
const vectorSystemPath = `_system/vector/system.json`
const fieldIndexPath = `_system/metadata_indexes/__metadata_field_index__${fieldName}.json`
// HNSW node data IS sharded (entity UUID-based)
const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
// Vector node data IS sharded (entity UUID-based)
const vectorNodePath = `_system/vector/nodes/${shard}/${entityId}.json`
```
### Path Patterns Summary
@ -512,8 +512,8 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
| **COW ref** | `_cow/refs/heads/{branch}.json` | ❌ No | ❌ No |
| **Statistics** | `_system/statistics.json` | ❌ No | ❌ No |
| **Counts** | `_system/counts.json` | ❌ No | ❌ No |
| **HNSW system** | `_system/hnsw/system.json` | ❌ No | ❌ No |
| **HNSW node** | `_system/hnsw/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No |
| **Vector system** | `_system/vector/system.json` | ❌ No | ❌ No |
| **Vector node** | `_system/vector/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No |
| **Field index** | `_system/metadata_indexes/__metadata_field_index__{field}.json` | ❌ No | ❌ No |
### Key Principles
@ -521,7 +521,7 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
1. **Shard Extraction**: Always use first 2 hex characters of UUID/SHA-256
2. **ID-First**: Shard + ID come BEFORE type (type is in metadata)
3. **Branch Isolation**: Only entity data uses branches/
4. **System Isolation**: System files never use sharding or branching (except HNSW nodes)
4. **System Isolation**: System files never use sharding or branching (except vector index nodes)
5. **Content-Addressable**: COW uses SHA-256 hash as filename
---
@ -530,7 +530,7 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
Brainy uses four complementary index systems for different query patterns.
### 3.1 HNSW Vector Index (In-Memory with Lazy Loading)
### 3.1 Vector Index (In-Memory with Lazy Loading)
**Purpose**: Semantic similarity search
**Location**: RAM (rebuilt from storage on startup)
@ -538,7 +538,7 @@ Brainy uses four complementary index systems for different query patterns.
**How It Works**:
1. Loads `branches/{branch}/entities/nouns/{type}/vectors/**/*.json` files
2. Builds HNSW graph structure in memory
2. Builds vector index graph structure in memory
3. Enables O(log n) approximate nearest neighbor search
4. Vectors loaded on-demand in lazy mode (zero configuration)
@ -634,10 +634,10 @@ const users = await brain.getNouns({
### 4.1 Why Shard?
**Cloud Storage Limitations**:
- GCS/S3: Listing 100K files in one directory = 10-30 seconds
- GCS/S3: Max recommended files per directory = 1,000-10,000
- Network: Parallel operations faster than sequential
**Filesystem Limitations**:
- Listing 100K files in one directory is slow (`readdir` walks the whole entry list)
- Most filesystems prefer ≤10,000 entries per directory for fast lookups
- Parallel operations across shards beat serial scans
**Solution**: Split into 256 shards = ~3,900 files per shard at 1M scale
@ -786,7 +786,7 @@ _cow/blobs/ab/abc123...sha256.bin (used by both entities)
### 6.1 What is ID-First?
**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused 20-21 second delays on cloud storage.
**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused long delays when an entity's type was unknown.
**Old type-first structure** (v5.12.0):
```
@ -807,13 +807,13 @@ branches/main/entities/nouns/00/001234...uuid/metadata.json
**1. 40x Faster Lookups**
```typescript
// v5.x: Had to search 42 types if type unknown
// Result: 21 seconds on GCS (42 types × 500ms)
// Result: 21 seconds (42 types × 500ms)
// Direct path from ID
const id = '001234...'
const shard = id.substring(0, 2) // '00'
const path = `branches/main/entities/nouns/${shard}/${id}/metadata.json`
// Result: <500ms on GCS - 40x faster!
// Result: <500ms - 40x faster!
```
**2. Simpler Code**
@ -980,9 +980,9 @@ const related = await brain.find('financial projections for Q2')
## 8. Storage Backend Mapping
### 8.1 All Backends Use Same Structure
### 8.1 Both Adapters Use the Same Structure
**Filesystem** (local):
**Filesystem** (default for Node.js):
```
/path/to/brainy-data/
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
@ -990,70 +990,21 @@ const related = await brain.find('financial projections for Q2')
└── _system/statistics.json
```
**Google Cloud Storage** (GCS):
```
gs://my-bucket/
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
├── _cow/commits/00/00a1b2c3...sha256.json
└── _system/statistics.json
```
**AWS S3** / **MinIO** / **DigitalOcean Spaces**:
```
s3://my-bucket/
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
├── _cow/commits/00/00a1b2c3...sha256.json
└── _system/statistics.json
```
**Cloudflare R2**:
```
r2://my-bucket/
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
├── _cow/commits/00/00a1b2c3...sha256.json
└── _system/statistics.json
```
**Azure Blob Storage**:
```
azure://my-container/
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
├── _cow/commits/00/00a1b2c3...sha256.json
└── _system/statistics.json
```
**OPFS** (browser):
```
opfs://root/brainy/
├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json
├── _cow/commits/00/00a1b2c3...sha256.json
└── _system/statistics.json
```
**Memory Storage** (in-memory):
- Uses same path structure
- Stored in `Map<string, any>`
- Key = full path (e.g., "branches/main/entities/nouns/Character/vectors/00/001234...uuid.json")
For off-site backup of the filesystem artifact, snapshot the `rootDirectory` from your scheduler using `gsutil`, `aws s3 sync`, `rclone`, or `tar` — Brainy itself stays local.
---
### 8.2 Backend-Specific Optimizations
**Cloud Storage (GCS, S3, R2, Azure)**:
- Lifecycle policies for automatic archival (96% cost savings)
- Intelligent-Tiering (S3) or Autoclass (GCS) for access-pattern optimization
- Batch operations (1000 objects per request for S3)
- Parallel uploads/downloads
### 8.2 Adapter Characteristics
**Filesystem**:
- Optional gzip compression (60-80% space savings)
- Direct file I/O (fastest for local)
- Atomic writes with rename
**OPFS**:
- Quota monitoring (browser storage limits)
- Persistent storage (survives page refresh)
- Worker-based I/O (non-blocking)
- Sharded directory layout keeps `readdir` fast
**Memory**:
- No I/O overhead (instant access)
@ -1090,7 +1041,7 @@ opfs://root/brainy/
| Get entity by ID | 15-30s | 100-150ms | **200x faster** |
| List all entities | 30-60s | 30-60s | Same |
| Filter by metadata | 10-30s | 5-50ms | **100-600x faster** (via indexes) |
| Semantic search | N/A | 1-10ms | N/A (requires HNSW) |
| Semantic search | N/A | 1-10ms | N/A (requires vector index) |
| Type filtering | 30-60s | 120-200ms | **150-500x faster** (type-first) |
| Graph query (getVerbsBySource) | O(total_verbs) | <1ms | **O(1) via index** |
@ -1113,15 +1064,10 @@ opfs://root/brainy/
| Storage Backend | Max Entities (No Optimization) | Max Entities (Full Optimization) |
|----------------|-------------------------------|----------------------------------|
| GCS | ~10,000 | **10M+** |
| S3 | ~10,000 | **10M+** |
| R2 | ~10,000 | **10M+** |
| Azure | ~10,000 | **10M+** |
| Filesystem | ~100,000 | **10M+** |
| OPFS | ~50,000 | **1M+** (browser limits) |
| Memory | Limited by RAM | Limited by RAM |
**Full optimization** = Sharding + Type-first + COW + Lifecycle policies + Lazy mode
**Full optimization** = Sharding + Type-first + COW + Lazy mode
---
@ -1129,7 +1075,7 @@ opfs://root/brainy/
| Component | Standard Mode | Lazy Mode | Savings |
|-----------|---------------|-----------|---------|
| **HNSW Index (100K entities)** | 149MB | 15-33MB | 5-10x |
| **Vector Index (100K entities)** | 149MB | 15-33MB | 5-10x |
| **Graph Index (100K verbs)** | 100MB | 100MB | N/A |
| **Metadata Indexes** | 10-50MB | 10-50MB | N/A |
| **UnifiedCache** | 2GB | 2GB | N/A |
@ -1149,17 +1095,15 @@ opfs://root/brainy/
- Use UUIDs for all entities and relationships
- Let Brainy handle sharding automatically (type-first + UUID sharding)
- Use metadata indexes for filtering
- Enable lifecycle policies for cloud storage (96% cost savings)
- Use batch operations for bulk deletions
- Enable compression for FileSystem storage (60-80% space savings)
- Snapshot `rootDirectory` to off-site storage from your scheduler (`gsutil` / `aws s3 sync` / `rclone`)
- Create branches for experimentation (instant, zero-cost)
❌ **Don't**:
- Try to organize files manually
- Assume file paths are predictable (use IDs, not paths)
- Store large binary data in metadata (use blob storage or VFS)
- Disable COW (can't be disabled in v5.11.0+, always enabled)
- Forget to monitor OPFS quota in browser applications
- Store large binary data in metadata (use VFS for blobs)
- Disable COW (always enabled)
---
@ -1170,7 +1114,7 @@ opfs://root/brainy/
✅ Deletes:
- `branches/` → ALL entity data (all types, all shards, all branches, all forks)
- `_cow/` → ALL version control (commits, trees, blobs, refs)
- `_system/` → ALL indexes (statistics, HNSW, metadata)
- `_system/` → ALL indexes (statistics, vector index, metadata)
✅ Resets:
- COW managers (refManager, blobStorage, commitLog) → `undefined`
@ -1237,7 +1181,7 @@ await brain.add({ data: 'Alice', type: 'person' })
→ _cow/refs/heads/main.json (points to new commit)
10. Update statistics:
→ _system/statistics.json (increment person count)
11. Update HNSW index (in-memory):
11. Update vector index (in-memory):
→ Connect to nearest neighbors
12. Update graph index (in-memory):
→ Add to adjacency maps
@ -1281,7 +1225,7 @@ const results = await brain.find('medieval castle', { k: 10 })
**What happens in storage**:
```
1. Compute query vector: [0.1, 0.2, 0.3, ...]
2. Use HNSW index (in-memory):
2. Use vector index (in-memory):
→ Navigate graph from entry point
→ Find 10 nearest neighbors (1-10ms)
3. Load vectors from cache or storage:
@ -1363,7 +1307,7 @@ await brain.storage.clear()
→ Result: ALL commits, trees, blobs, refs deleted
3. Delete all indexes:
→ Remove: _system/ (entire directory)
→ Result: Statistics, HNSW, metadata indexes deleted
→ Result: Statistics, vector index, metadata indexes deleted
4. Reset COW managers in memory:
→ refManager = undefined
→ blobStorage = undefined
@ -1393,13 +1337,13 @@ await brain.init()
**What happens in storage**:
```
1. Check for persisted indexes:
→ Load: _system/hnsw/system.json (entry point, max level)
→ Load: _system/hnsw/nodes/** (graph connections)
→ Load: _system/vector/system.json (entry point, max level)
→ Load: _system/vector/nodes/** (graph connections)
→ Load: _system/statistics.json (entity counts)
2. Decide standard vs lazy mode:
→ Check: entityCount × vectorSize vs. available cache
→ Auto-enable lazy mode if needed
3. Rebuild HNSW index:
3. Rebuild vector index:
→ Standard mode: Load all vectors into memory
→ Lazy mode: Load only graph structure (~24 bytes/node)
4. Rebuild Graph Adjacency index:
@ -1441,7 +1385,7 @@ await brain.addBatch([
→ 1 tree object (or tree fan-out for large trees)
→ 10,000+ blobs (deduplicated)
5. Update indexes in batch:
→ HNSW: Batch insert (optimized)
→ Vector index: Batch insert (optimized)
→ Graph: Batch update
→ Metadata: Batch index update
```
@ -1455,25 +1399,23 @@ await brain.addBatch([
**Complete Storage Structure**:
- **3 storage layers**: branches/ (data), _cow/ (versions), _system/ (indexes)
- **2 files per entity**: metadata.json + vector.json (optimized I/O)
- **4 indexes**: HNSW (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields)
- **4 indexes**: Vector (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields)
- **256 shards**: UUID-based (uniform distribution)
- **42 noun types + 127 verb types**: Type is metadata, not storage structure
- **Git-like COW**: Branches, commits, trees, blobs, refs
- **VFS support**: Traditional file/folder hierarchies
**Scalability (v6.0.0 Improvements)**:
- **ID-First Storage**: 40x faster on cloud storage (eliminates 42-type search)
- Sharding: 200x faster for cloud storage
**Scalability**:
- **ID-First Storage**: 40x faster directory lookups (eliminates 42-type search)
- Sharding: 200x faster file lookups
- Type filtering: Still O(type_count) via metadata index
- Lazy mode: 5-10x less memory for large datasets
- COW: Instant branches, efficient forks
- Deduplication: 30-90% storage savings
**Production Features**:
- Lifecycle policies (96% cost savings on cloud storage)
- Batch operations (efficient API usage)
- Compression (60-80% space savings on filesystem)
- Quota monitoring (OPFS browser limits)
- Batch operations (efficient I/O)
- Operator-layer backup via `gsutil` / `aws s3 sync` / `rclone` against `rootDirectory`
- Auto-reinitialization (COW always-on, can't be broken)
- **Clean architecture**: Removed 500+ lines of type cache complexity
@ -1481,7 +1423,7 @@ await brain.addBatch([
## Next Steps
- [Storage Adapters](./storage-architecture.md) - Configure cloud storage backends
- [Storage Architecture](./storage-architecture.md) - Adapter and backup details
- [VFS Guide](../vfs/README.md) - Use Virtual File System features
- [Triple Intelligence](../vfs/TRIPLE_INTELLIGENCE.md) - Semantic file extraction
- [Scaling Guide](../SCALING.md) - Handle 10M+ entities
@ -1489,6 +1431,5 @@ await brain.addBatch([
---
**Version**: v6.0.0
**Last Updated**: 2025-11-19
**Key Features**: ID-first storage, COW always-on, metadata-based type index, 4-index architecture, VFS support, billion-scale optimization, 40x cloud performance improvement
**Last Updated**: 2026
**Key Features**: ID-first storage, COW always-on, metadata-based type index, 4-index architecture, VFS support, billion-scale optimization