refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider
The distributed-clustering subsystem never ran in production: it was inert, orphaned dead code (faked consensus, stub replication, no live wiring, and it did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process library. Scale is single-process + the optional native provider (@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools + horizontal read scaling (many reader processes, one writer). Removed: - src/distributed/ entirely (coordinator, shardManager, cacheSync, readWriteSeparation, queryPlanner, healthMonitor, configManager, hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts (slimmed to the live surface). - src/types/distributedTypes.ts; config.distributed field + JSDoc; coreTypes distributedConfig; memoryStorage distributedConfig persistence. - DistributedRole enum + src/config/distributedPresets.ts and the orphaned src/config/extensibleConfig.ts (config/augmentation registry built on removed cloud adapters + distributed presets), plus their src/index.ts re-exports. - 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook; enableDistributedSearch (dead config flag); the metadata partition field; the distributed_ reserved key prefix. - Orphaned src/storage/readOnlyOptimizations.ts (zero importers). - Tests targeting the subsystem: distributed-demo, distributed-cluster helper, distributed-transactions, sharding-transactions. - Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/ shard-manager/multi-node prose from v3-features, enterprise-for-everyone, augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP, vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4, storage-architecture; reframed scale prose to the 8.0 model. Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via getShardIdFromUuid — used live by baseStorage, unrelated to clustering); the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering. RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0 scale model.
This commit is contained in:
parent
f8e0079d3f
commit
00d3203d68
51 changed files with 153 additions and 9889 deletions
|
|
@ -1,396 +0,0 @@
|
|||
# 🔌 Extending Brainy Storage with Augmentations
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy's zero-config system is **fully extensible**. Augmentations can register new storage providers, presets, and auto-detection logic that integrates seamlessly with the existing system.
|
||||
|
||||
## How Storage Extensions Work
|
||||
|
||||
### 1. Storage Provider Registration
|
||||
|
||||
When an augmentation is installed, it can register a new storage provider:
|
||||
|
||||
```typescript
|
||||
import { StorageProvider, registerStorageAugmentation } from '@soulcraft/brainy/config'
|
||||
|
||||
const redisProvider: StorageProvider = {
|
||||
type: 'redis',
|
||||
name: 'Redis Storage',
|
||||
description: 'High-performance in-memory data store',
|
||||
priority: 10, // Higher priority = checked first in auto-detection
|
||||
|
||||
// Auto-detection logic
|
||||
async detect(): Promise<boolean> {
|
||||
// Check if Redis is available
|
||||
if (process.env.REDIS_URL) {
|
||||
try {
|
||||
const redis = await import('ioredis')
|
||||
const client = new redis.default(process.env.REDIS_URL)
|
||||
await client.ping()
|
||||
await client.quit()
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return false
|
||||
},
|
||||
|
||||
// Configuration builder
|
||||
async getConfig(): Promise<any> {
|
||||
return {
|
||||
type: 'redis',
|
||||
redisStorage: {
|
||||
url: process.env.REDIS_URL,
|
||||
prefix: 'brainy:',
|
||||
ttl: 3600
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Register the provider
|
||||
registerStorageAugmentation(redisProvider)
|
||||
```
|
||||
|
||||
### 2. Using Extended Storage
|
||||
|
||||
Once registered, the new storage type works with zero-config:
|
||||
|
||||
```typescript
|
||||
// Auto-detection will now check Redis
|
||||
const brain = new Brainy() // Will use Redis if available!
|
||||
|
||||
// Or explicitly specify
|
||||
const brain = new Brainy({ storage: 'redis' })
|
||||
|
||||
// Or with custom config
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'redis',
|
||||
redisStorage: {
|
||||
url: 'redis://localhost:6379',
|
||||
prefix: 'myapp:'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
### Redis Augmentation
|
||||
|
||||
```typescript
|
||||
// @soulcraft/brainy-redis package
|
||||
export class RedisStorageAugmentation {
|
||||
async init() {
|
||||
// Register the storage provider
|
||||
registerStorageAugmentation({
|
||||
type: 'redis',
|
||||
name: 'Redis Storage',
|
||||
priority: 10,
|
||||
|
||||
async detect() {
|
||||
return !!(process.env.REDIS_URL || process.env.REDIS_HOST)
|
||||
},
|
||||
|
||||
async getConfig() {
|
||||
return {
|
||||
type: 'redis',
|
||||
redisStorage: {
|
||||
url: process.env.REDIS_URL ||
|
||||
`redis://${process.env.REDIS_HOST}:${process.env.REDIS_PORT || 6379}`
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Register Redis-specific presets
|
||||
registerPresetAugmentation('redis-cache', {
|
||||
storage: 'redis',
|
||||
model: ModelPrecision.Q8,
|
||||
features: ['core', 'cache'],
|
||||
distributed: true,
|
||||
description: 'Redis-backed cache layer',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### MongoDB Augmentation
|
||||
|
||||
```typescript
|
||||
// @soulcraft/brainy-mongodb package
|
||||
export class MongoStorageAugmentation {
|
||||
async init() {
|
||||
registerStorageAugmentation({
|
||||
type: 'mongodb',
|
||||
name: 'MongoDB Storage',
|
||||
priority: 8,
|
||||
|
||||
async detect() {
|
||||
return !!(process.env.MONGODB_URI || process.env.MONGO_URL)
|
||||
},
|
||||
|
||||
async getConfig() {
|
||||
return {
|
||||
type: 'mongodb',
|
||||
mongoStorage: {
|
||||
uri: process.env.MONGODB_URI,
|
||||
database: 'brainy',
|
||||
collection: 'vectors'
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### PostgreSQL + pgvector Augmentation
|
||||
|
||||
```typescript
|
||||
// @soulcraft/brainy-postgres package
|
||||
export class PostgresStorageAugmentation {
|
||||
async init() {
|
||||
registerStorageAugmentation({
|
||||
type: 'postgres',
|
||||
name: 'PostgreSQL + pgvector',
|
||||
priority: 9,
|
||||
|
||||
async detect() {
|
||||
const url = process.env.DATABASE_URL
|
||||
if (url?.includes('postgres')) {
|
||||
// Check for pgvector extension
|
||||
const client = new Client({ connectionString: url })
|
||||
await client.connect()
|
||||
const result = await client.query(
|
||||
"SELECT * FROM pg_extension WHERE extname = 'vector'"
|
||||
)
|
||||
await client.end()
|
||||
return result.rows.length > 0
|
||||
}
|
||||
return false
|
||||
},
|
||||
|
||||
async getConfig() {
|
||||
return {
|
||||
type: 'postgres',
|
||||
postgresStorage: {
|
||||
connectionString: process.env.DATABASE_URL,
|
||||
table: 'brainy_vectors'
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Auto-Detection Priority
|
||||
|
||||
Storage providers are checked in priority order:
|
||||
|
||||
1. **Custom providers** (highest priority first)
|
||||
2. **Cloud storage** (S3, GCS, R2)
|
||||
3. **Database storage** (Redis, MongoDB, PostgreSQL)
|
||||
4. **Local storage** (filesystem, OPFS)
|
||||
5. **Memory** (fallback)
|
||||
|
||||
```typescript
|
||||
// Example priority chain
|
||||
Redis (priority: 10) → PostgreSQL (9) → MongoDB (8) → S3 (5) → Filesystem (1) → Memory (0)
|
||||
```
|
||||
|
||||
## Creating Custom Presets
|
||||
|
||||
Augmentations can also register new presets:
|
||||
|
||||
```typescript
|
||||
registerPresetAugmentation('redis-cluster', {
|
||||
storage: 'redis',
|
||||
model: ModelPrecision.Q8,
|
||||
features: ['core', 'cache', 'cluster'],
|
||||
distributed: true,
|
||||
role: DistributedRole.HYBRID,
|
||||
cache: {
|
||||
hotCacheMaxSize: 100000, // Large distributed cache
|
||||
autoTune: true
|
||||
},
|
||||
description: 'Redis Cluster configuration',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
|
||||
// Users can then use:
|
||||
const brain = new Brainy('redis-cluster')
|
||||
```
|
||||
|
||||
## Type Safety with Extensions
|
||||
|
||||
To maintain type safety with dynamic storage types:
|
||||
|
||||
```typescript
|
||||
// Augmentation declares its types
|
||||
declare module '@soulcraft/brainy' {
|
||||
interface StorageTypes {
|
||||
redis: {
|
||||
url: string
|
||||
prefix?: string
|
||||
ttl?: number
|
||||
}
|
||||
}
|
||||
|
||||
interface PresetNames {
|
||||
'redis-cache': 'redis-cache'
|
||||
'redis-cluster': 'redis-cluster'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices for Storage Augmentations
|
||||
|
||||
1. **Always provide auto-detection** - Check environment variables and connectivity
|
||||
2. **Set appropriate priority** - Higher for specialized storage, lower for general
|
||||
3. **Handle failures gracefully** - Return false from detect() if not available
|
||||
4. **Document requirements** - List required packages and environment variables
|
||||
5. **Provide presets** - Include common configuration patterns
|
||||
6. **Maintain compatibility** - Ensure model precision matches across instances
|
||||
|
||||
## Example: Complete Redis Augmentation
|
||||
|
||||
```typescript
|
||||
import {
|
||||
StorageProvider,
|
||||
registerStorageAugmentation,
|
||||
registerPresetAugmentation,
|
||||
PresetCategory,
|
||||
ModelPrecision,
|
||||
DistributedRole
|
||||
} from '@soulcraft/brainy/config'
|
||||
import Redis from 'ioredis'
|
||||
|
||||
export class BrainyRedisAugmentation {
|
||||
private client: Redis
|
||||
|
||||
async init() {
|
||||
// Register storage provider
|
||||
registerStorageAugmentation({
|
||||
type: 'redis',
|
||||
name: 'Redis Vector Storage',
|
||||
description: 'Redis with RediSearch for vector similarity',
|
||||
priority: 10,
|
||||
|
||||
requirements: {
|
||||
env: ['REDIS_URL'],
|
||||
packages: ['ioredis', 'redis']
|
||||
},
|
||||
|
||||
async detect() {
|
||||
if (!process.env.REDIS_URL) return false
|
||||
|
||||
try {
|
||||
const client = new Redis(process.env.REDIS_URL)
|
||||
|
||||
// Check for RediSearch module
|
||||
const modules = await client.call('MODULE', 'LIST')
|
||||
const hasRediSearch = modules.some(m => m[1] === 'search')
|
||||
|
||||
await client.quit()
|
||||
return hasRediSearch
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
},
|
||||
|
||||
async getConfig() {
|
||||
return {
|
||||
type: 'redis',
|
||||
redisStorage: {
|
||||
url: process.env.REDIS_URL,
|
||||
prefix: process.env.REDIS_PREFIX || 'brainy:',
|
||||
index: process.env.REDIS_INDEX || 'brainy-vectors',
|
||||
ttl: process.env.REDIS_TTL ? parseInt(process.env.REDIS_TTL) : undefined
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Register presets
|
||||
this.registerPresets()
|
||||
}
|
||||
|
||||
private registerPresets() {
|
||||
// Fast cache preset
|
||||
registerPresetAugmentation('redis-fast-cache', {
|
||||
storage: 'redis' as any,
|
||||
model: ModelPrecision.Q8,
|
||||
features: ['core', 'cache', 'search'],
|
||||
distributed: false,
|
||||
cache: {
|
||||
hotCacheMaxSize: 10000,
|
||||
autoTune: true
|
||||
},
|
||||
description: 'Redis-backed fast cache',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
|
||||
// Distributed cache preset
|
||||
registerPresetAugmentation('redis-distributed', {
|
||||
storage: 'redis' as any,
|
||||
model: ModelPrecision.AUTO,
|
||||
features: ['core', 'cache', 'search', 'cluster'],
|
||||
distributed: true,
|
||||
role: DistributedRole.HYBRID,
|
||||
cache: {
|
||||
hotCacheMaxSize: 50000,
|
||||
autoTune: true
|
||||
},
|
||||
description: 'Redis distributed cache cluster',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
|
||||
// Session store preset
|
||||
registerPresetAugmentation('redis-sessions', {
|
||||
storage: 'redis' as any,
|
||||
model: ModelPrecision.Q8,
|
||||
features: ['core', 'cache'],
|
||||
distributed: false,
|
||||
cache: {
|
||||
hotCacheMaxSize: 5000,
|
||||
autoTune: false
|
||||
},
|
||||
description: 'Redis session storage',
|
||||
category: PresetCategory.SERVICE
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Usage after installing the augmentation:
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import '@soulcraft/brainy-redis' // Registers the augmentation
|
||||
|
||||
// Now Redis is automatically detected!
|
||||
const brain = new Brainy() // Uses Redis if REDIS_URL is set
|
||||
|
||||
// Or use a Redis preset
|
||||
const brain = new Brainy('redis-fast-cache')
|
||||
|
||||
// Or explicitly configure
|
||||
const brain = new Brainy({
|
||||
storage: 'redis',
|
||||
model: ModelPrecision.FP32
|
||||
})
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
The extensible configuration system allows:
|
||||
|
||||
1. **New storage types** via `registerStorageAugmentation()`
|
||||
2. **Custom presets** via `registerPresetAugmentation()`
|
||||
3. **Auto-detection logic** that integrates with zero-config
|
||||
4. **Type-safe extensions** with TypeScript declarations
|
||||
5. **Priority-based selection** for intelligent defaults
|
||||
|
||||
This ensures Brainy can grow with new storage technologies while maintaining its zero-configuration philosophy!
|
||||
|
|
@ -426,16 +426,6 @@ npm install @soulcraft/brainy@^3.50.0
|
|||
7. Verify data integrity thoroughly
|
||||
8. Enable lifecycle policies gradually
|
||||
|
||||
### Scenario 4: Multi-Node Distributed System
|
||||
|
||||
**Recommended approach:**
|
||||
1. Perform blue-green deployment:
|
||||
- Keep v3 nodes running (blue)
|
||||
- Deploy v4 nodes (green)
|
||||
- Migrate data once
|
||||
- Switch traffic to v4 nodes
|
||||
- Decommission v3 nodes
|
||||
|
||||
## Cost Savings After Migration
|
||||
|
||||
### Enable All v4.0.0 Features
|
||||
|
|
|
|||
|
|
@ -84,7 +84,6 @@ See [vfs/](./vfs/) for the complete VFS documentation set.
|
|||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Storage Architecture](./architecture/storage-architecture.md) | Filesystem and memory adapters, on-disk artifact layout, operator-layer backup |
|
||||
| [Extending Storage](./EXTENDING_STORAGE.md) | Create custom storage adapters |
|
||||
| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities |
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -73,10 +73,10 @@ import { MemoryStorageAugmentation } from 'brainy'
|
|||
```
|
||||
|
||||
### 11. Server Search Augmentation ✅
|
||||
Distributed search capabilities.
|
||||
Server-side search delegation over a conduit.
|
||||
```typescript
|
||||
import { ServerSearchConduitAugmentation } from 'brainy'
|
||||
// Distributed query execution
|
||||
// Forwards queries to a remote Brainy server
|
||||
```
|
||||
|
||||
### 12. Neural Import Augmentation ✅
|
||||
|
|
@ -100,7 +100,7 @@ await neuralImport.detectRelationships(entities)
|
|||
await neuralImport.generateInsights(data)
|
||||
```
|
||||
|
||||
### Distributed Operation Modes (Fully Implemented!)
|
||||
### Operation Modes (Fully Implemented!)
|
||||
```typescript
|
||||
// Read-only mode with optimized caching
|
||||
const readerMode = new ReaderMode()
|
||||
|
|
@ -239,15 +239,10 @@ const cacheConfig = await getCacheAutoConfig()
|
|||
|
||||
## 🎨 How to Use Hidden Features
|
||||
|
||||
### Enable Distributed Modes
|
||||
### Enable Reader / Writer Modes
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
mode: 'reader', // or 'writer' or 'hybrid'
|
||||
distributed: {
|
||||
role: 'reader',
|
||||
cacheStrategy: 'aggressive',
|
||||
prefetch: true
|
||||
}
|
||||
mode: 'reader' // or 'writer' or 'hybrid'
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -285,7 +280,7 @@ const freshStats = await brain.getStatistics({
|
|||
## 📝 What Needs Documentation
|
||||
|
||||
These features EXIST but need better docs:
|
||||
1. Distributed operation modes
|
||||
1. Reader / writer operation modes
|
||||
2. Neural import full API
|
||||
3. 3-level cache configuration
|
||||
4. Performance monitoring API
|
||||
|
|
@ -297,7 +292,7 @@ These features EXIST but need better docs:
|
|||
## 💡 The Truth
|
||||
|
||||
Brainy is MORE powerful than its own documentation suggests! Most "missing" features are actually implemented but hidden or not properly exposed. The codebase contains sophisticated systems for:
|
||||
- Distributed operations
|
||||
- Reader / writer operation modes
|
||||
- AI-powered import
|
||||
- Advanced caching
|
||||
- Performance monitoring
|
||||
|
|
|
|||
|
|
@ -291,8 +291,8 @@ console.log(stats)
|
|||
|
||||
### Choose the Right Adapter
|
||||
1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence
|
||||
2. **Single-node production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
|
||||
3. **Multi-node production**: Run Brainy behind a service layer; replicate the on-disk artifact via your operator tooling
|
||||
2. **Single-process production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
|
||||
3. **Horizontal scaling**: Brainy runs in one process — there is no built-in cluster. Run independent instances behind a service layer and replicate the on-disk artifact with your operator tooling; or run many reader processes against one shared store with a single writer
|
||||
|
||||
### Optimize for Your Use Case
|
||||
1. **Read-heavy**: Enable caching and let the OS page cache do its job
|
||||
|
|
|
|||
|
|
@ -66,7 +66,7 @@ import { BatchProcessingAugmentation } from 'brainy'
|
|||
```typescript
|
||||
import { ConnectionPoolAugmentation } from 'brainy'
|
||||
// Auto-scaling connection management
|
||||
// Optimized for distributed operations
|
||||
// Optimized for high-concurrency workloads
|
||||
```
|
||||
|
||||
### 7. Request Deduplicator ✅
|
||||
|
|
@ -100,8 +100,7 @@ import { MemoryStorageAugmentation } from 'brainy'
|
|||
### 11. Server Search Conduit ✅
|
||||
```typescript
|
||||
import { ServerSearchConduitAugmentation } from 'brainy'
|
||||
// Distributed query execution
|
||||
// Load balancing across nodes
|
||||
// Forwards queries to a remote Brainy server
|
||||
```
|
||||
|
||||
### 12. Neural Import ✅
|
||||
|
|
@ -155,7 +154,9 @@ await brain.init()
|
|||
- **No environment variables needed**
|
||||
- **Works in all environments** (Node, Browser, Workers)
|
||||
|
||||
## 🏢 Distributed Operation Modes
|
||||
## 🏢 Operation Modes
|
||||
|
||||
Run many reader processes against one shared on-disk store with a single writer.
|
||||
|
||||
### Reader Mode ✅
|
||||
```typescript
|
||||
|
|
@ -374,7 +375,7 @@ await brain.init()
|
|||
- ✅ All engines (vector, graph, field, neural)
|
||||
- ✅ All augmentations (12+)
|
||||
- ✅ All storage adapters
|
||||
- ✅ All distributed modes
|
||||
- ✅ Reader / writer / hybrid operation modes
|
||||
- ✅ Complete statistics
|
||||
- ✅ GPU support
|
||||
- ✅ No feature limitations
|
||||
|
|
|
|||
|
|
@ -34,58 +34,6 @@ brain.add({ name: "John", email: "john@example.com" }, 'entity')
|
|||
- Automatic query optimization
|
||||
- Pattern-based query rewriting
|
||||
|
||||
## 🏢 Enterprise Features
|
||||
|
||||
### Distributed Coordination ✅
|
||||
Raft consensus for multi-node deployments:
|
||||
```typescript
|
||||
import { DistributedCoordinator } from '@soulcraft/brainy'
|
||||
|
||||
const coordinator = createCoordinator({
|
||||
nodeId: 'node-1',
|
||||
peers: ['node-2', 'node-3'],
|
||||
electionTimeout: 500
|
||||
})
|
||||
// Automatic leader election and failover
|
||||
```
|
||||
|
||||
### Horizontal Sharding ✅
|
||||
Consistent hashing for data distribution:
|
||||
```typescript
|
||||
import { ShardManager } from '@soulcraft/brainy'
|
||||
|
||||
const shards = createShardManager({
|
||||
nodes: ['node-1', 'node-2', 'node-3'],
|
||||
replicationFactor: 2,
|
||||
virtualNodes: 150
|
||||
})
|
||||
// Automatic shard rebalancing on node changes
|
||||
```
|
||||
|
||||
### Read/Write Separation ✅
|
||||
Primary-replica architecture for scale:
|
||||
```typescript
|
||||
import { ReadWriteSeparation } from '@soulcraft/brainy'
|
||||
|
||||
const replication = createReadWriteSeparation({
|
||||
role: 'auto', // Automatic primary/replica detection
|
||||
consistencyLevel: 'strong', // or 'eventual'
|
||||
readPreference: 'nearest'
|
||||
})
|
||||
```
|
||||
|
||||
### Cross-Instance Cache Sync ✅
|
||||
Version vector-based cache synchronization:
|
||||
```typescript
|
||||
import { CacheSync } from '@soulcraft/brainy'
|
||||
|
||||
const cache = createCacheSync({
|
||||
nodeId: 'node-1',
|
||||
syncInterval: 100,
|
||||
conflictResolution: 'version-vector'
|
||||
})
|
||||
```
|
||||
|
||||
## 🔐 Security & Compliance
|
||||
|
||||
### Rate Limiting ✅
|
||||
|
|
@ -281,14 +229,12 @@ await brain.restore('backup.bin')
|
|||
- **10,000+ items**: Sub-10ms search
|
||||
- **1M+ operations**: Stable memory usage
|
||||
- **100+ concurrent users**: No performance degradation
|
||||
- **Multi-node clusters**: Automatic failover
|
||||
|
||||
## 🚫 NOT Implemented (Planned)
|
||||
|
||||
These features are documented but NOT yet implemented:
|
||||
- GraphQL API (use REST API instead)
|
||||
- Kubernetes operators (use Docker)
|
||||
- Some distributed features require manual configuration
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -165,32 +165,28 @@ await brain.syncWith({
|
|||
- **Webhook support**: React to changes
|
||||
- **API generation**: Auto-generate REST/GraphQL APIs
|
||||
|
||||
### 🌍 Enterprise Scale 🚧 Coming Soon
|
||||
### 🌍 Scale
|
||||
|
||||
**Everyone gets planetary scale:**
|
||||
**Everyone gets the same scale model:**
|
||||
|
||||
```typescript
|
||||
// Same architecture Netflix uses, free for you
|
||||
const brain = new Brainy({
|
||||
clustering: {
|
||||
enabled: true, // Distributed mode
|
||||
sharding: 'automatic', // Auto-sharding
|
||||
replication: 3, // Triple replication
|
||||
consensus: 'raft', // Strong consistency
|
||||
geoDistribution: true // Multi-region support
|
||||
}
|
||||
})
|
||||
// Pure JS by default; install the optional native provider for billions of vectors
|
||||
const brain = new Brainy()
|
||||
|
||||
// Handles everything from 1 to 1 billion entities
|
||||
// 1 → ~1M vectors: pure-JS HNSW, zero extra setup
|
||||
// 1M → 10B+ vectors: install @soulcraft/cortex for the native DiskANN provider
|
||||
```
|
||||
|
||||
**Scaling features:**
|
||||
- **Horizontal scaling**: Add nodes as needed
|
||||
- **Auto-sharding**: Distributes data automatically
|
||||
- **Multi-region**: Global distribution
|
||||
- **Load balancing**: Automatic request distribution
|
||||
- **Zero-downtime upgrades**: Rolling updates
|
||||
- **Infinite scale**: No upper limits
|
||||
**Scaling model:**
|
||||
- **Single process, no cluster**: Brainy runs in one process — no coordinator,
|
||||
no peer discovery, no consensus to operate
|
||||
- **Optional native provider**: install `@soulcraft/cortex` to back the index with
|
||||
on-disk DiskANN that scales to 10B+ vectors on one machine
|
||||
- **Per-tenant pools**: isolate tenants by giving each its own Brainy instance and
|
||||
storage directory
|
||||
- **Horizontal read scaling**: run many reader processes against one shared on-disk
|
||||
store (single writer, many readers); replicate the artifact with your operator
|
||||
tooling
|
||||
|
||||
### 🛡️ Enterprise Compliance 🚧 Coming Soon
|
||||
|
||||
|
|
|
|||
|
|
@ -219,7 +219,6 @@ Actual cache size: ~40GB (prevents waste on 128GB systems)
|
|||
|
||||
**Recommendations:**
|
||||
- ✅ Use FP32 model for maximum accuracy
|
||||
- ✅ Enable distributed storage (S3/GCS)
|
||||
- ✅ Monitor fairness violations (HNSW shouldn't dominate cache)
|
||||
- ✅ Consider sharding beyond 50M entities
|
||||
- ✅ Implement application-level caching for hot queries
|
||||
|
|
|
|||
|
|
@ -458,7 +458,6 @@ describe('Transaction Tests', () => {
|
|||
See `tests/transaction/integration/` for comprehensive integration tests covering:
|
||||
- Sharding integration (`sharding-transactions.test.ts`)
|
||||
- Type-aware integration (`typeaware-transactions.test.ts`)
|
||||
- Distributed scenarios (`distributed-transactions.test.ts`)
|
||||
|
||||
The atomicity guarantees of `brain.transact()` — including crash recovery
|
||||
through the real recovery path — are proven in
|
||||
|
|
|
|||
|
|
@ -332,7 +332,7 @@ const urgent = await vfs.search('', {
|
|||
|
||||
## Advanced Features
|
||||
|
||||
> **Note:** See [VFS ROADMAP](./ROADMAP.md) for planned advanced features like version history, distributed filesystem, and more.
|
||||
> **Note:** See [VFS ROADMAP](./ROADMAP.md) for planned advanced features like version history and more.
|
||||
|
||||
## Integration Possibilities
|
||||
|
||||
|
|
@ -351,7 +351,6 @@ Brainy VFS is designed for speed and scale:
|
|||
**PROJECTED at larger scales (not yet tested):**
|
||||
- **Vector search** <100ms for millions of files (projected)
|
||||
- **Streaming support** for files of any size (architecture supports, see [limitations in ROADMAP](./ROADMAP.md))
|
||||
- **Distributed sharding** for billions of files (architecture supports, not tested at scale)
|
||||
|
||||
See tests in `tests/vfs/` for actual measured performance.
|
||||
|
||||
|
|
@ -377,7 +376,7 @@ Brainy VFS fully leverages Brainy's revolutionary Triple Intelligence system:
|
|||
| Manual organization | Self-organizing with intelligent fusion |
|
||||
| No relationships | Rich knowledge graph with traversal |
|
||||
| Static metadata | Dynamic, queryable metadata with field intelligence |
|
||||
| Single server | Distributed & federated |
|
||||
| Manual scaling | Horizontal read scaling — many readers, one writer |
|
||||
|
||||
## Installation
|
||||
|
||||
|
|
@ -438,10 +437,10 @@ The VFS is built with production scalability in mind:
|
|||
- Parent-child relationship caching
|
||||
- Hot path detection and optimization
|
||||
|
||||
2. **Distributed Architecture**
|
||||
- Sharding by path prefix
|
||||
- Read replicas for hot directories
|
||||
- CDN integration for static files
|
||||
2. **Horizontal Read Scaling**
|
||||
- On-disk store partitioned into 256 hex directory buckets
|
||||
- Many reader processes against one shared store
|
||||
- Single writer keeps the store consistent
|
||||
|
||||
3. **Intelligent Indexing**
|
||||
- Compound indexes on (parent, name)
|
||||
|
|
@ -589,49 +588,6 @@ vfs.on('file:added', async (path) => {
|
|||
})
|
||||
```
|
||||
|
||||
#### **5. Distributed Team Workspace**
|
||||
|
||||
Collaborative file management:
|
||||
|
||||
```javascript
|
||||
// Track file ownership and access
|
||||
await vfs.writeFile('/projects/alpha/spec.md', content, {
|
||||
metadata: {
|
||||
owner: userId,
|
||||
team: 'engineering',
|
||||
permissions: {
|
||||
[userId]: 'rw',
|
||||
'team:engineering': 'r',
|
||||
'others': '-'
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Add collaborative features
|
||||
await vfs.addTodo('/projects/alpha/spec.md', {
|
||||
task: 'Review security section',
|
||||
assignee: 'alice@company.com',
|
||||
due: '2024-02-01',
|
||||
priority: 'high'
|
||||
})
|
||||
|
||||
// Track who's working on what
|
||||
await vfs.setxattr('/projects/alpha/spec.md', 'locks', {
|
||||
section3: {
|
||||
user: 'bob@company.com',
|
||||
since: Date.now()
|
||||
}
|
||||
})
|
||||
|
||||
// Find all files assigned to a user
|
||||
const assigned = await vfs.search('', {
|
||||
where: {
|
||||
'todos.assignee': 'alice@company.com',
|
||||
'todos.status': 'pending'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Monitoring & Operations (Planned)
|
||||
|
||||
> **Note:** Production monitoring features are planned for v1.1. See [ROADMAP](./ROADMAP.md).
|
||||
|
|
@ -665,27 +621,11 @@ await vfs.init({
|
|||
})
|
||||
```
|
||||
|
||||
#### **Distributed Cluster**
|
||||
```javascript
|
||||
const vfs = new VirtualFileSystem()
|
||||
await vfs.init({
|
||||
distributed: true,
|
||||
nodes: [
|
||||
'vfs1.internal:8080',
|
||||
'vfs2.internal:8080',
|
||||
'vfs3.internal:8080'
|
||||
],
|
||||
replication: 3,
|
||||
consistency: 'eventual'
|
||||
})
|
||||
```
|
||||
|
||||
## Roadmap & Future Features
|
||||
|
||||
See [VFS ROADMAP](./ROADMAP.md) for planned features including:
|
||||
- Enhanced streaming support (v1.1)
|
||||
- Version history (v1.2)
|
||||
- Distributed filesystem (v1.2)
|
||||
- AI-powered automation (v2.0)
|
||||
- FUSE driver (v2.0 research)
|
||||
- And more community-requested features
|
||||
|
|
|
|||
|
|
@ -65,27 +65,6 @@ const diff = await vfs.diffVersions('/important-doc.md', v1, v2)
|
|||
**Status:** Planned
|
||||
**Effort:** 4-5 weeks
|
||||
|
||||
### Distributed Filesystem
|
||||
Mount remote Brainy instances for federated file access.
|
||||
|
||||
```typescript
|
||||
// Planned API (not yet implemented)
|
||||
await vfs.mount('/remote-team', {
|
||||
type: 'brainy-remote',
|
||||
url: 'https://team-brainy.example.com',
|
||||
credentials: {...}
|
||||
})
|
||||
|
||||
// Federated search across mounted instances
|
||||
const results = await vfs.search('project docs', { includeMounted: true })
|
||||
|
||||
// Sync directories
|
||||
await vfs.sync('/local/docs', '/remote-team/docs')
|
||||
```
|
||||
|
||||
**Status:** Planned
|
||||
**Effort:** 6-8 weeks
|
||||
|
||||
### Backup & Recovery API
|
||||
Built-in backup and recovery operations.
|
||||
|
||||
|
|
@ -215,12 +194,6 @@ Vite integration with VFS for semantic module resolution.
|
|||
|
||||
## Far Future (v5.0+)
|
||||
|
||||
### Automatic Node Discovery
|
||||
Zero-config multi-node setup with automatic discovery via UDP broadcast, Kubernetes DNS, or cloud provider APIs.
|
||||
|
||||
### Automatic Failover
|
||||
Health monitoring and automatic failover in distributed deployments.
|
||||
|
||||
### Cloud Provider Auto-Detection
|
||||
```typescript
|
||||
// Far future concept
|
||||
|
|
|
|||
|
|
@ -358,7 +358,6 @@ VFS scales to millions of files:
|
|||
- O(1) path lookup with caching
|
||||
- Efficient graph traversal for directories
|
||||
- Chunked storage for large files
|
||||
- Distributed storage backend support
|
||||
- Vector search scales with HNSW index
|
||||
|
||||
## Method Availability
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue