docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
parent
2626ab8d62
commit
adda1570f3
22 changed files with 570 additions and 2658 deletions
|
|
@ -11,7 +11,7 @@ Brainy achieves industry-leading performance through carefully optimized data st
|
|||
| **Metadata Index** | Exact match | **O(1)** | 0.8ms | `Map<string, Set<string>>` |
|
||||
| **Metadata Index** | Range query | **O(log n) + O(k)** | 0.6ms | Sorted array + binary search |
|
||||
| **Graph Index** | Get neighbors | **O(1)** | 0.09ms | `Map<string, Set<string>>` |
|
||||
| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | HNSW hierarchical graph |
|
||||
| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | Hierarchical graph |
|
||||
| **NLP Parser** | Query parsing | **O(m)** | 8.9ms | 220 pre-computed patterns |
|
||||
| **Type-Field Affinity** | Field matching | **O(f)** | 0.1ms | Type-specific field cache |
|
||||
| **Type Detection** | Noun/Verb matching | **O(t)** | 0.3ms | Pre-embedded type vectors |
|
||||
|
|
@ -55,7 +55,7 @@ const entity = await brain.get(id)
|
|||
const entity = await brain.get(id, { includeVectors: true })
|
||||
// - Computing similarity on THIS entity
|
||||
// - Manual vector operations
|
||||
// - HNSW graph traversal
|
||||
// - Vector index graph traversal
|
||||
```
|
||||
|
||||
## Architecture Deep Dive
|
||||
|
|
@ -143,14 +143,14 @@ class GraphAdjacencyIndex {
|
|||
|
||||
**Key Innovation:** Pure Map/Set operations - no database queries, no loops, just direct memory access.
|
||||
|
||||
### 4. HNSW Vector Search - O(log n)
|
||||
### 4. Vector Index - O(log n)
|
||||
|
||||
Hierarchical Navigable Small World graphs provide logarithmic approximate nearest neighbor search:
|
||||
The default vector index (`JsHnswVectorIndex`) provides logarithmic approximate nearest neighbor search through a hierarchical graph:
|
||||
|
||||
```typescript
|
||||
class HNSWIndex {
|
||||
class JsHnswVectorIndex {
|
||||
private nouns: Map<string, HNSWNoun> = new Map()
|
||||
|
||||
|
||||
interface HNSWNoun {
|
||||
id: string
|
||||
vector: number[]
|
||||
|
|
@ -238,7 +238,7 @@ const results = await Promise.all(searchPromises)
|
|||
|-----------|--------------|---------|
|
||||
| Metadata Index | ~40 bytes/entry | `(key_size + 8) × unique_values + 8 × total_items` |
|
||||
| Graph Index | ~24 bytes/edge | `16 × edges + 8 × nodes` |
|
||||
| HNSW | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
|
||||
| Vector Index | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` |
|
||||
| Pattern Library | 394KB fixed | Pre-computed, shared across instances |
|
||||
| Type Embeddings | ~60KB fixed | 70 types × 384 dimensions × 4 bytes, cached |
|
||||
| Field Embeddings | ~5KB dynamic | Actual fields × 384 dimensions × 4 bytes |
|
||||
|
|
@ -279,7 +279,7 @@ const results = await Promise.all(searchPromises)
|
|||
|
||||
| System | Metadata Filter | Graph Traversal | Vector Search | Natural Language |
|
||||
|--------|-----------------|-----------------|---------------|------------------|
|
||||
| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) HNSW | 220 patterns |
|
||||
| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) vector index | 220 patterns |
|
||||
| Neo4j | O(log n) B-tree | O(k) traversal | Not native | Not native |
|
||||
| Elasticsearch | O(log n) inverted | Not native | O(n) brute force* | Basic tokenization |
|
||||
| PostgreSQL | O(log n) B-tree | O(k) recursive | O(n) brute force* | Full-text only |
|
||||
|
|
@ -305,7 +305,7 @@ const results = await Promise.all(searchPromises)
|
|||
- ✅ **No Network Calls**: Everything runs locally, including embeddings
|
||||
- ✅ **Thread-Safe**: Immutable data structures where possible
|
||||
- ✅ **Memory Bounded**: Configurable cache sizes and automatic cleanup
|
||||
- ✅ **Horizontally Scalable**: Stateless operations support clustering
|
||||
- ✅ **Single-Node by Design**: One process owns one `rootDirectory`; scale out at the service layer
|
||||
- ✅ **Zero Stubs**: Every line of code is production-ready
|
||||
|
||||
## Lazy Loading Performance
|
||||
|
|
@ -375,110 +375,48 @@ const brain = new Brainy({ disableAutoRebuild: true })
|
|||
await brain.init() // Instant (0-10ms)
|
||||
```
|
||||
|
||||
### Automatic Self-Tuning (Current & Planned)
|
||||
### Automatic Self-Tuning
|
||||
|
||||
**✅ Currently Implemented:**
|
||||
- **Metadata Index**: Auto-builds sorted indices for range queries on first use
|
||||
- **Graph Index**: Auto-flushes every 30 seconds
|
||||
- **Default Tuning**: Research-based defaults (M=16, ef=200)
|
||||
- **Default Tuning**: Research-based vector index defaults
|
||||
- **Lazy Loading**: Indices built only when needed
|
||||
- **Cache Management**: LRU caches with TTL
|
||||
|
||||
**🚧 Planned Enhancements:**
|
||||
- **Dynamic Storage Selection**: Auto-switch between memory/disk based on size
|
||||
- **Adaptive Index Parameters**: Adjust M and ef based on query patterns
|
||||
- **Smart Cache Sizing**: Scale caches based on available memory
|
||||
- **Predictive Optimization**: Learn from usage patterns
|
||||
|
||||
### Intelligent Defaults
|
||||
|
||||
All defaults are research-based and production-tested:
|
||||
- **HNSW M=16**: Optimal balance of recall/speed for most datasets
|
||||
- **efConstruction=200**: High quality graph construction
|
||||
- **Cache TTL=5min**: Balances freshness with performance
|
||||
- **Flush Interval=30s**: Non-blocking background persistence
|
||||
- **Vector recall** = `'balanced'` (M=16, ef=200): right for most datasets
|
||||
- **Cache TTL** = 5 min: balances freshness and performance
|
||||
- **Flush interval** = 30 s: non-blocking background persistence
|
||||
|
||||
### Progressive Enhancement
|
||||
### Vector Index Tuning Knobs
|
||||
|
||||
Brainy learns and improves over time:
|
||||
1. **Query Pattern Learning**: Frequently used patterns get cached
|
||||
2. **Index Optimization**: Auto-rebuilds indices when fragmented
|
||||
3. **Memory Management**: Coordinates caches across all components
|
||||
4. **Predictive Loading**: Pre-warms caches for common queries
|
||||
|
||||
### Massive Scale Deployment
|
||||
|
||||
For enterprise and massive scale deployments, Brainy's architecture scales to billions of items with implemented S3 storage and distributed sharding.
|
||||
|
||||
**Currently Implemented:**
|
||||
- Memory storage (production-ready)
|
||||
- Disk storage (production-ready)
|
||||
- S3-compatible storage (AWS S3, Cloudflare R2, Google Cloud Storage, MinIO, Backblaze B2)
|
||||
- Distributed sharding with ConsistentHashRing
|
||||
- Single-node deployment (scales to ~1M items)
|
||||
- Multi-node deployment with sharding (scales to billions)
|
||||
|
||||
**Available Today:**
|
||||
Brainy 8.0 exposes exactly three knobs on `config.vector`:
|
||||
|
||||
```javascript
|
||||
// S3-compatible storage for unlimited scale - WORKS NOW
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucketName: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
credentials: {
|
||||
accessKeyId: 'YOUR_ACCESS_KEY',
|
||||
secretAccessKey: 'YOUR_SECRET_KEY'
|
||||
}
|
||||
// Works with: AWS S3, MinIO, Cloudflare R2, Backblaze B2, Google Cloud Storage
|
||||
}
|
||||
})
|
||||
|
||||
// Cloudflare R2 storage - WORKS NOW
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'r2',
|
||||
bucketName: 'my-brainy-data',
|
||||
accountId: 'YOUR_ACCOUNT_ID',
|
||||
accessKeyId: 'YOUR_R2_ACCESS_KEY',
|
||||
secretAccessKey: 'YOUR_R2_SECRET_KEY'
|
||||
}
|
||||
})
|
||||
|
||||
// Google Cloud Storage - WORKS NOW
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
bucketName: 'my-brainy-data',
|
||||
region: 'us-central1',
|
||||
credentials: {
|
||||
accessKeyId: 'YOUR_ACCESS_KEY',
|
||||
secretAccessKey: 'YOUR_SECRET_KEY'
|
||||
}
|
||||
const brain = new Brainy({
|
||||
vector: {
|
||||
recall: 'fast', // 'fast' | 'balanced' | 'accurate'
|
||||
quantization: { bits: 8 }, // 4 | 8 (SQ4 / SQ8)
|
||||
persistMode: 'deferred' // 'immediate' | 'deferred'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cortex`) can replace it with a higher-performing implementation; the public knobs stay the same.
|
||||
|
||||
### Scale Scenarios
|
||||
|
||||
| Scale | Items | Storage Strategy | Performance | Status |
|
||||
|-------|-------|-----------------|-------------|--------|
|
||||
| **Small** | <10K | Memory (automatic) | Sub-millisecond | ✅ Implemented |
|
||||
| **Medium** | 10K-1M | Disk with memory cache | 1-5ms | ✅ Implemented |
|
||||
| **Large** | 1M-100M | S3 with memory cache | 2-10ms | ✅ Implemented |
|
||||
| **Massive** | 100M-10B | S3 + distributed sharding | 5-20ms | ✅ Implemented |
|
||||
| **Planetary** | 10B+ | Multi-region S3 + Edge cache | 10-50ms | 🚧 Roadmap |
|
||||
| Scale | Items | Storage Strategy | Performance |
|
||||
|-------|-------|------------------|-------------|
|
||||
| **Small** | <10K | Memory | Sub-millisecond |
|
||||
| **Medium** | 10K-1M | Filesystem | 1-5ms |
|
||||
| **Large** | 1M-10M | Filesystem + tuned cache | 2-10ms |
|
||||
| **Massive** | 10M+ | Filesystem + native vector provider + service-layer sharding | 5-20ms |
|
||||
|
||||
### S3-Compatible Storage Benefits
|
||||
For >10M entities, run multiple Brainy processes behind your own routing layer — Brainy 8.0 doesn't ship cluster coordination.
|
||||
|
||||
- **Unlimited Scale**: No practical limit on dataset size
|
||||
- **Cost Effective**: $0.023/GB/month for standard storage
|
||||
- **Durability**: 99.999999999% (11 9's) durability
|
||||
- **Global**: Multi-region replication available
|
||||
- **Compatible**: Works with any S3-compatible API (MinIO, R2, B2)
|
||||
|
||||
### Distributed Architecture (Implemented)
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
|
|
@ -490,103 +428,51 @@ const brain = new Brainy({
|
|||
│ Brainy Core │
|
||||
│ (Triple Intelligence Engine) │
|
||||
├─────────────────────────────────────────┤
|
||||
│ Memory │ Shard │ Metadata │
|
||||
│ Cache │ Manager │ Index │
|
||||
│ Memory │ Vector │ Metadata │
|
||||
│ Cache │ Index │ Index │
|
||||
└─────────────┬───────────────────────────┘
|
||||
│
|
||||
┌─────────────▼───────────────────────────┐
|
||||
│ Storage Layer │
|
||||
├──────────┬──────────┬──────────────────┤
|
||||
│ HNSW │ Graph │ Objects │
|
||||
│ Vectors │ Edges │ (S3/R2/GCS) │
|
||||
│ Vectors │ Graph │ Files │
|
||||
│ (sharded)│ Edges │ (filesystem) │
|
||||
└──────────┴──────────┴──────────────────┘
|
||||
```
|
||||
|
||||
**Distributed Sharding (Implemented):**
|
||||
- ConsistentHashRing with 150 virtual nodes
|
||||
- 64 shards by default
|
||||
- Replication factor of 3
|
||||
- Automatic rebalancing on node addition/removal
|
||||
|
||||
### Auto-Sharding for Horizontal Scale (Implemented)
|
||||
|
||||
Brainy includes a complete sharding implementation with ConsistentHashRing:
|
||||
|
||||
```javascript
|
||||
import { ShardManager } from '@soulcraft/brainy/distributed'
|
||||
|
||||
// Create shard manager with custom configuration
|
||||
const shardManager = new ShardManager({
|
||||
shardCount: 64, // Default: 64 shards
|
||||
replicationFactor: 3, // Default: 3 replicas
|
||||
virtualNodes: 150, // Default: 150 virtual nodes
|
||||
autoRebalance: true // Default: true
|
||||
})
|
||||
|
||||
// Add nodes to the cluster
|
||||
shardManager.addNode('node-1')
|
||||
shardManager.addNode('node-2')
|
||||
shardManager.addNode('node-3')
|
||||
|
||||
// Sharding automatically:
|
||||
// - Uses consistent hashing for even distribution
|
||||
// - Maintains replicas for fault tolerance
|
||||
// - Rebalances on node changes
|
||||
// - Provides O(1) shard lookups
|
||||
```
|
||||
For off-site replication, snapshot `rootDirectory` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`).
|
||||
|
||||
### Performance at Scale
|
||||
|
||||
Even at massive scale, Brainy maintains excellent performance:
|
||||
|
||||
- **Metadata queries**: Still O(1) with distributed hash tables
|
||||
- **Graph traversal**: O(1) with edge locality optimization
|
||||
- **Vector search**: O(log n) with hierarchical sharding
|
||||
- **Write throughput**: 100K+ writes/second with S3 batching
|
||||
- **Metadata queries**: O(1) HashMap
|
||||
- **Graph traversal**: O(1) adjacency lookup
|
||||
- **Vector search**: O(log n)
|
||||
- **Write throughput**: 50K+ writes/second per process (filesystem, batched)
|
||||
- **Read throughput**: 1M+ reads/second with caching
|
||||
|
||||
### Zero-Config with Autoscaling (Implemented)
|
||||
### Zero-Config with Autoscaling
|
||||
|
||||
Brainy includes extensive autoscaling capabilities:
|
||||
|
||||
**✅ Implemented Autoscaling:**
|
||||
- **AutoConfiguration System**: Detects environment and adjusts settings
|
||||
- **Learning from Performance**: `learnFromPerformance()` adapts based on metrics
|
||||
- **Auto-flush**: Graph index (30s), Metadata index (configurable)
|
||||
- **Auto-optimize**: Enabled by default in Graph and HNSW indices
|
||||
- **Auto-rebalance**: Shards automatically rebalance on node changes
|
||||
- **Auto-optimize**: Enabled by default in graph and vector indices
|
||||
- **Zero-config presets**: Production, development, minimal modes
|
||||
- **Adaptive memory**: Scales caches based on available memory
|
||||
- **Environment detection**: Browser vs Node.js vs Serverless
|
||||
|
||||
**🚧 Roadmap Autoscaling:**
|
||||
- Dynamic HNSW parameter adjustment (M, ef)
|
||||
- Predictive query pattern caching
|
||||
- Multi-region auto-replication
|
||||
- Automatic cross-node data migration
|
||||
|
||||
## Implementation Status
|
||||
|
||||
### ✅ Fully Implemented and Production-Ready
|
||||
### Fully Implemented and Production-Ready
|
||||
- **O(1) metadata lookups** via HashMaps (exact match)
|
||||
- **O(log n) range queries** via sorted arrays with lazy building
|
||||
- **O(1) graph traversal** via adjacency maps
|
||||
- **O(log n) vector search** via HNSW
|
||||
- **O(log n) vector search** via the default JS index, swappable for a native provider
|
||||
- **220 NLP patterns** with pre-computed embeddings
|
||||
- **S3-compatible storage** (AWS S3, R2, GCS, MinIO, B2)
|
||||
- **Distributed sharding** with ConsistentHashRing
|
||||
- **Filesystem and memory storage** adapters
|
||||
- **Auto-configuration system** with environment detection
|
||||
- **Zero-config operation** with intelligent defaults
|
||||
- **Auto-flush and auto-optimize** in indices
|
||||
- **Sub-2ms response times** for complex queries
|
||||
|
||||
### 🚧 Roadmap Features
|
||||
- Dynamic HNSW parameter tuning
|
||||
- Predictive query pattern caching
|
||||
- Multi-region S3 replication
|
||||
- Automatic cross-node data migration
|
||||
- Edge caching layer
|
||||
|
||||
## Conclusion
|
||||
|
||||
Brainy delivers on its promise of **production-ready Triple Intelligence** with measured, verified performance characteristics. All listed features are fully implemented, tested, and benchmarked. No stubs, no mocks, no theoretical claims - just real, working code with measured performance.
|
||||
Loading…
Add table
Add a link
Reference in a new issue