refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider
The distributed-clustering subsystem never ran in production: it was inert, orphaned dead code (faked consensus, stub replication, no live wiring, and it did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process library. Scale is single-process + the optional native provider (@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools + horizontal read scaling (many reader processes, one writer). Removed: - src/distributed/ entirely (coordinator, shardManager, cacheSync, readWriteSeparation, queryPlanner, healthMonitor, configManager, hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts (slimmed to the live surface). - src/types/distributedTypes.ts; config.distributed field + JSDoc; coreTypes distributedConfig; memoryStorage distributedConfig persistence. - DistributedRole enum + src/config/distributedPresets.ts and the orphaned src/config/extensibleConfig.ts (config/augmentation registry built on removed cloud adapters + distributed presets), plus their src/index.ts re-exports. - 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook; enableDistributedSearch (dead config flag); the metadata partition field; the distributed_ reserved key prefix. - Orphaned src/storage/readOnlyOptimizations.ts (zero importers). - Tests targeting the subsystem: distributed-demo, distributed-cluster helper, distributed-transactions, sharding-transactions. - Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/ shard-manager/multi-node prose from v3-features, enterprise-for-everyone, augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP, vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4, storage-architecture; reframed scale prose to the 8.0 model. Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via getShardIdFromUuid — used live by baseStorage, unrelated to clustering); the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering. RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0 scale model.
This commit is contained in:
parent
f8e0079d3f
commit
00d3203d68
51 changed files with 153 additions and 9889 deletions
|
|
@ -66,7 +66,7 @@ import { BatchProcessingAugmentation } from 'brainy'
|
|||
```typescript
|
||||
import { ConnectionPoolAugmentation } from 'brainy'
|
||||
// Auto-scaling connection management
|
||||
// Optimized for distributed operations
|
||||
// Optimized for high-concurrency workloads
|
||||
```
|
||||
|
||||
### 7. Request Deduplicator ✅
|
||||
|
|
@ -100,8 +100,7 @@ import { MemoryStorageAugmentation } from 'brainy'
|
|||
### 11. Server Search Conduit ✅
|
||||
```typescript
|
||||
import { ServerSearchConduitAugmentation } from 'brainy'
|
||||
// Distributed query execution
|
||||
// Load balancing across nodes
|
||||
// Forwards queries to a remote Brainy server
|
||||
```
|
||||
|
||||
### 12. Neural Import ✅
|
||||
|
|
@ -155,7 +154,9 @@ await brain.init()
|
|||
- **No environment variables needed**
|
||||
- **Works in all environments** (Node, Browser, Workers)
|
||||
|
||||
## 🏢 Distributed Operation Modes
|
||||
## 🏢 Operation Modes
|
||||
|
||||
Run many reader processes against one shared on-disk store with a single writer.
|
||||
|
||||
### Reader Mode ✅
|
||||
```typescript
|
||||
|
|
@ -374,7 +375,7 @@ await brain.init()
|
|||
- ✅ All engines (vector, graph, field, neural)
|
||||
- ✅ All augmentations (12+)
|
||||
- ✅ All storage adapters
|
||||
- ✅ All distributed modes
|
||||
- ✅ Reader / writer / hybrid operation modes
|
||||
- ✅ Complete statistics
|
||||
- ✅ GPU support
|
||||
- ✅ No feature limitations
|
||||
|
|
|
|||
|
|
@ -34,58 +34,6 @@ brain.add({ name: "John", email: "john@example.com" }, 'entity')
|
|||
- Automatic query optimization
|
||||
- Pattern-based query rewriting
|
||||
|
||||
## 🏢 Enterprise Features
|
||||
|
||||
### Distributed Coordination ✅
|
||||
Raft consensus for multi-node deployments:
|
||||
```typescript
|
||||
import { DistributedCoordinator } from '@soulcraft/brainy'
|
||||
|
||||
const coordinator = createCoordinator({
|
||||
nodeId: 'node-1',
|
||||
peers: ['node-2', 'node-3'],
|
||||
electionTimeout: 500
|
||||
})
|
||||
// Automatic leader election and failover
|
||||
```
|
||||
|
||||
### Horizontal Sharding ✅
|
||||
Consistent hashing for data distribution:
|
||||
```typescript
|
||||
import { ShardManager } from '@soulcraft/brainy'
|
||||
|
||||
const shards = createShardManager({
|
||||
nodes: ['node-1', 'node-2', 'node-3'],
|
||||
replicationFactor: 2,
|
||||
virtualNodes: 150
|
||||
})
|
||||
// Automatic shard rebalancing on node changes
|
||||
```
|
||||
|
||||
### Read/Write Separation ✅
|
||||
Primary-replica architecture for scale:
|
||||
```typescript
|
||||
import { ReadWriteSeparation } from '@soulcraft/brainy'
|
||||
|
||||
const replication = createReadWriteSeparation({
|
||||
role: 'auto', // Automatic primary/replica detection
|
||||
consistencyLevel: 'strong', // or 'eventual'
|
||||
readPreference: 'nearest'
|
||||
})
|
||||
```
|
||||
|
||||
### Cross-Instance Cache Sync ✅
|
||||
Version vector-based cache synchronization:
|
||||
```typescript
|
||||
import { CacheSync } from '@soulcraft/brainy'
|
||||
|
||||
const cache = createCacheSync({
|
||||
nodeId: 'node-1',
|
||||
syncInterval: 100,
|
||||
conflictResolution: 'version-vector'
|
||||
})
|
||||
```
|
||||
|
||||
## 🔐 Security & Compliance
|
||||
|
||||
### Rate Limiting ✅
|
||||
|
|
@ -281,14 +229,12 @@ await brain.restore('backup.bin')
|
|||
- **10,000+ items**: Sub-10ms search
|
||||
- **1M+ operations**: Stable memory usage
|
||||
- **100+ concurrent users**: No performance degradation
|
||||
- **Multi-node clusters**: Automatic failover
|
||||
|
||||
## 🚫 NOT Implemented (Planned)
|
||||
|
||||
These features are documented but NOT yet implemented:
|
||||
- GraphQL API (use REST API instead)
|
||||
- Kubernetes operators (use Docker)
|
||||
- Some distributed features require manual configuration
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue