refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider

The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).

Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
  readWriteSeparation, queryPlanner, healthMonitor, configManager,
  hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
  transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
  (slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
  coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
  src/config/extensibleConfig.ts (config/augmentation registry built on removed
  cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
  enableDistributedSearch (dead config flag); the metadata partition field;
  the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
  distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
  shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
  augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
  vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
  storage-architecture; reframed scale prose to the 8.0 model.

Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.

RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
This commit is contained in:
David Snelling 2026-06-15 10:37:39 -07:00
parent f8e0079d3f
commit 00d3203d68
51 changed files with 153 additions and 9889 deletions

View file

@ -66,7 +66,7 @@ import { BatchProcessingAugmentation } from 'brainy'
```typescript
import { ConnectionPoolAugmentation } from 'brainy'
// Auto-scaling connection management
// Optimized for distributed operations
// Optimized for high-concurrency workloads
```
### 7. Request Deduplicator ✅
@ -100,8 +100,7 @@ import { MemoryStorageAugmentation } from 'brainy'
### 11. Server Search Conduit ✅
```typescript
import { ServerSearchConduitAugmentation } from 'brainy'
// Distributed query execution
// Load balancing across nodes
// Forwards queries to a remote Brainy server
```
### 12. Neural Import ✅
@ -155,7 +154,9 @@ await brain.init()
- **No environment variables needed**
- **Works in all environments** (Node, Browser, Workers)
## 🏢 Distributed Operation Modes
## 🏢 Operation Modes
Run many reader processes against one shared on-disk store with a single writer.
### Reader Mode ✅
```typescript
@ -374,7 +375,7 @@ await brain.init()
- ✅ All engines (vector, graph, field, neural)
- ✅ All augmentations (12+)
- ✅ All storage adapters
- ✅ All distributed modes
- ✅ Reader / writer / hybrid operation modes
- ✅ Complete statistics
- ✅ GPU support
- ✅ No feature limitations

View file

@ -34,58 +34,6 @@ brain.add({ name: "John", email: "john@example.com" }, 'entity')
- Automatic query optimization
- Pattern-based query rewriting
## 🏢 Enterprise Features
### Distributed Coordination ✅
Raft consensus for multi-node deployments:
```typescript
import { DistributedCoordinator } from '@soulcraft/brainy'
const coordinator = createCoordinator({
nodeId: 'node-1',
peers: ['node-2', 'node-3'],
electionTimeout: 500
})
// Automatic leader election and failover
```
### Horizontal Sharding ✅
Consistent hashing for data distribution:
```typescript
import { ShardManager } from '@soulcraft/brainy'
const shards = createShardManager({
nodes: ['node-1', 'node-2', 'node-3'],
replicationFactor: 2,
virtualNodes: 150
})
// Automatic shard rebalancing on node changes
```
### Read/Write Separation ✅
Primary-replica architecture for scale:
```typescript
import { ReadWriteSeparation } from '@soulcraft/brainy'
const replication = createReadWriteSeparation({
role: 'auto', // Automatic primary/replica detection
consistencyLevel: 'strong', // or 'eventual'
readPreference: 'nearest'
})
```
### Cross-Instance Cache Sync ✅
Version vector-based cache synchronization:
```typescript
import { CacheSync } from '@soulcraft/brainy'
const cache = createCacheSync({
nodeId: 'node-1',
syncInterval: 100,
conflictResolution: 'version-vector'
})
```
## 🔐 Security & Compliance
### Rate Limiting ✅
@ -281,14 +229,12 @@ await brain.restore('backup.bin')
- **10,000+ items**: Sub-10ms search
- **1M+ operations**: Stable memory usage
- **100+ concurrent users**: No performance degradation
- **Multi-node clusters**: Automatic failover
## 🚫 NOT Implemented (Planned)
These features are documented but NOT yet implemented:
- GraphQL API (use REST API instead)
- Kubernetes operators (use Docker)
- Some distributed features require manual configuration
---