refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider

The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).

Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
  readWriteSeparation, queryPlanner, healthMonitor, configManager,
  hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
  transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
  (slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
  coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
  src/config/extensibleConfig.ts (config/augmentation registry built on removed
  cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
  enableDistributedSearch (dead config flag); the metadata partition field;
  the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
  distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
  shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
  augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
  vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
  storage-architecture; reframed scale prose to the 8.0 model.

Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.

RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
This commit is contained in:
David Snelling 2026-06-15 10:37:39 -07:00
parent f8e0079d3f
commit 00d3203d68
51 changed files with 153 additions and 9889 deletions

View file

@ -165,32 +165,28 @@ await brain.syncWith({
- **Webhook support**: React to changes
- **API generation**: Auto-generate REST/GraphQL APIs
### 🌍 Enterprise Scale 🚧 Coming Soon
### 🌍 Scale
**Everyone gets planetary scale:**
**Everyone gets the same scale model:**
```typescript
// Same architecture Netflix uses, free for you
const brain = new Brainy({
clustering: {
enabled: true, // Distributed mode
sharding: 'automatic', // Auto-sharding
replication: 3, // Triple replication
consensus: 'raft', // Strong consistency
geoDistribution: true // Multi-region support
}
})
// Pure JS by default; install the optional native provider for billions of vectors
const brain = new Brainy()
// Handles everything from 1 to 1 billion entities
// 1 → ~1M vectors: pure-JS HNSW, zero extra setup
// 1M → 10B+ vectors: install @soulcraft/cortex for the native DiskANN provider
```
**Scaling features:**
- **Horizontal scaling**: Add nodes as needed
- **Auto-sharding**: Distributes data automatically
- **Multi-region**: Global distribution
- **Load balancing**: Automatic request distribution
- **Zero-downtime upgrades**: Rolling updates
- **Infinite scale**: No upper limits
**Scaling model:**
- **Single process, no cluster**: Brainy runs in one process — no coordinator,
no peer discovery, no consensus to operate
- **Optional native provider**: install `@soulcraft/cortex` to back the index with
on-disk DiskANN that scales to 10B+ vectors on one machine
- **Per-tenant pools**: isolate tenants by giving each its own Brainy instance and
storage directory
- **Horizontal read scaling**: run many reader processes against one shared on-disk
store (single writer, many readers); replicate the artifact with your operator
tooling
### 🛡️ Enterprise Compliance 🚧 Coming Soon