refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider

The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).

Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
  readWriteSeparation, queryPlanner, healthMonitor, configManager,
  hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
  transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
  (slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
  coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
  src/config/extensibleConfig.ts (config/augmentation registry built on removed
  cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
  enableDistributedSearch (dead config flag); the metadata partition field;
  the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
  distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
  shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
  augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
  vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
  storage-architecture; reframed scale prose to the 8.0 model.

Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.

RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
This commit is contained in:
David Snelling 2026-06-15 10:37:39 -07:00
parent f8e0079d3f
commit 00d3203d68
51 changed files with 153 additions and 9889 deletions

View file

@ -332,7 +332,7 @@ const urgent = await vfs.search('', {
## Advanced Features
> **Note:** See [VFS ROADMAP](./ROADMAP.md) for planned advanced features like version history, distributed filesystem, and more.
> **Note:** See [VFS ROADMAP](./ROADMAP.md) for planned advanced features like version history and more.
## Integration Possibilities
@ -351,7 +351,6 @@ Brainy VFS is designed for speed and scale:
**PROJECTED at larger scales (not yet tested):**
- **Vector search** <100ms for millions of files (projected)
- **Streaming support** for files of any size (architecture supports, see [limitations in ROADMAP](./ROADMAP.md))
- **Distributed sharding** for billions of files (architecture supports, not tested at scale)
See tests in `tests/vfs/` for actual measured performance.
@ -377,7 +376,7 @@ Brainy VFS fully leverages Brainy's revolutionary Triple Intelligence system:
| Manual organization | Self-organizing with intelligent fusion |
| No relationships | Rich knowledge graph with traversal |
| Static metadata | Dynamic, queryable metadata with field intelligence |
| Single server | Distributed & federated |
| Manual scaling | Horizontal read scaling — many readers, one writer |
## Installation
@ -438,10 +437,10 @@ The VFS is built with production scalability in mind:
- Parent-child relationship caching
- Hot path detection and optimization
2. **Distributed Architecture**
- Sharding by path prefix
- Read replicas for hot directories
- CDN integration for static files
2. **Horizontal Read Scaling**
- On-disk store partitioned into 256 hex directory buckets
- Many reader processes against one shared store
- Single writer keeps the store consistent
3. **Intelligent Indexing**
- Compound indexes on (parent, name)
@ -589,49 +588,6 @@ vfs.on('file:added', async (path) => {
})
```
#### **5. Distributed Team Workspace**
Collaborative file management:
```javascript
// Track file ownership and access
await vfs.writeFile('/projects/alpha/spec.md', content, {
metadata: {
owner: userId,
team: 'engineering',
permissions: {
[userId]: 'rw',
'team:engineering': 'r',
'others': '-'
}
}
})
// Add collaborative features
await vfs.addTodo('/projects/alpha/spec.md', {
task: 'Review security section',
assignee: 'alice@company.com',
due: '2024-02-01',
priority: 'high'
})
// Track who's working on what
await vfs.setxattr('/projects/alpha/spec.md', 'locks', {
section3: {
user: 'bob@company.com',
since: Date.now()
}
})
// Find all files assigned to a user
const assigned = await vfs.search('', {
where: {
'todos.assignee': 'alice@company.com',
'todos.status': 'pending'
}
})
```
### Monitoring & Operations (Planned)
> **Note:** Production monitoring features are planned for v1.1. See [ROADMAP](./ROADMAP.md).
@ -665,27 +621,11 @@ await vfs.init({
})
```
#### **Distributed Cluster**
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
distributed: true,
nodes: [
'vfs1.internal:8080',
'vfs2.internal:8080',
'vfs3.internal:8080'
],
replication: 3,
consistency: 'eventual'
})
```
## Roadmap & Future Features
See [VFS ROADMAP](./ROADMAP.md) for planned features including:
- Enhanced streaming support (v1.1)
- Version history (v1.2)
- Distributed filesystem (v1.2)
- AI-powered automation (v2.0)
- FUSE driver (v2.0 research)
- And more community-requested features