refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider
The distributed-clustering subsystem never ran in production: it was inert, orphaned dead code (faked consensus, stub replication, no live wiring, and it did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process library. Scale is single-process + the optional native provider (@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools + horizontal read scaling (many reader processes, one writer). Removed: - src/distributed/ entirely (coordinator, shardManager, cacheSync, readWriteSeparation, queryPlanner, healthMonitor, configManager, hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts (slimmed to the live surface). - src/types/distributedTypes.ts; config.distributed field + JSDoc; coreTypes distributedConfig; memoryStorage distributedConfig persistence. - DistributedRole enum + src/config/distributedPresets.ts and the orphaned src/config/extensibleConfig.ts (config/augmentation registry built on removed cloud adapters + distributed presets), plus their src/index.ts re-exports. - 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook; enableDistributedSearch (dead config flag); the metadata partition field; the distributed_ reserved key prefix. - Orphaned src/storage/readOnlyOptimizations.ts (zero importers). - Tests targeting the subsystem: distributed-demo, distributed-cluster helper, distributed-transactions, sharding-transactions. - Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/ shard-manager/multi-node prose from v3-features, enterprise-for-everyone, augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP, vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4, storage-architecture; reframed scale prose to the 8.0 model. Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via getShardIdFromUuid — used live by baseStorage, unrelated to clustering); the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering. RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0 scale model.
This commit is contained in:
parent
f8e0079d3f
commit
00d3203d68
51 changed files with 153 additions and 9889 deletions
|
|
@ -332,7 +332,7 @@ const urgent = await vfs.search('', {
|
|||
|
||||
## Advanced Features
|
||||
|
||||
> **Note:** See [VFS ROADMAP](./ROADMAP.md) for planned advanced features like version history, distributed filesystem, and more.
|
||||
> **Note:** See [VFS ROADMAP](./ROADMAP.md) for planned advanced features like version history and more.
|
||||
|
||||
## Integration Possibilities
|
||||
|
||||
|
|
@ -351,7 +351,6 @@ Brainy VFS is designed for speed and scale:
|
|||
**PROJECTED at larger scales (not yet tested):**
|
||||
- **Vector search** <100ms for millions of files (projected)
|
||||
- **Streaming support** for files of any size (architecture supports, see [limitations in ROADMAP](./ROADMAP.md))
|
||||
- **Distributed sharding** for billions of files (architecture supports, not tested at scale)
|
||||
|
||||
See tests in `tests/vfs/` for actual measured performance.
|
||||
|
||||
|
|
@ -377,7 +376,7 @@ Brainy VFS fully leverages Brainy's revolutionary Triple Intelligence system:
|
|||
| Manual organization | Self-organizing with intelligent fusion |
|
||||
| No relationships | Rich knowledge graph with traversal |
|
||||
| Static metadata | Dynamic, queryable metadata with field intelligence |
|
||||
| Single server | Distributed & federated |
|
||||
| Manual scaling | Horizontal read scaling — many readers, one writer |
|
||||
|
||||
## Installation
|
||||
|
||||
|
|
@ -438,10 +437,10 @@ The VFS is built with production scalability in mind:
|
|||
- Parent-child relationship caching
|
||||
- Hot path detection and optimization
|
||||
|
||||
2. **Distributed Architecture**
|
||||
- Sharding by path prefix
|
||||
- Read replicas for hot directories
|
||||
- CDN integration for static files
|
||||
2. **Horizontal Read Scaling**
|
||||
- On-disk store partitioned into 256 hex directory buckets
|
||||
- Many reader processes against one shared store
|
||||
- Single writer keeps the store consistent
|
||||
|
||||
3. **Intelligent Indexing**
|
||||
- Compound indexes on (parent, name)
|
||||
|
|
@ -589,49 +588,6 @@ vfs.on('file:added', async (path) => {
|
|||
})
|
||||
```
|
||||
|
||||
#### **5. Distributed Team Workspace**
|
||||
|
||||
Collaborative file management:
|
||||
|
||||
```javascript
|
||||
// Track file ownership and access
|
||||
await vfs.writeFile('/projects/alpha/spec.md', content, {
|
||||
metadata: {
|
||||
owner: userId,
|
||||
team: 'engineering',
|
||||
permissions: {
|
||||
[userId]: 'rw',
|
||||
'team:engineering': 'r',
|
||||
'others': '-'
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Add collaborative features
|
||||
await vfs.addTodo('/projects/alpha/spec.md', {
|
||||
task: 'Review security section',
|
||||
assignee: 'alice@company.com',
|
||||
due: '2024-02-01',
|
||||
priority: 'high'
|
||||
})
|
||||
|
||||
// Track who's working on what
|
||||
await vfs.setxattr('/projects/alpha/spec.md', 'locks', {
|
||||
section3: {
|
||||
user: 'bob@company.com',
|
||||
since: Date.now()
|
||||
}
|
||||
})
|
||||
|
||||
// Find all files assigned to a user
|
||||
const assigned = await vfs.search('', {
|
||||
where: {
|
||||
'todos.assignee': 'alice@company.com',
|
||||
'todos.status': 'pending'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Monitoring & Operations (Planned)
|
||||
|
||||
> **Note:** Production monitoring features are planned for v1.1. See [ROADMAP](./ROADMAP.md).
|
||||
|
|
@ -665,27 +621,11 @@ await vfs.init({
|
|||
})
|
||||
```
|
||||
|
||||
#### **Distributed Cluster**
|
||||
```javascript
|
||||
const vfs = new VirtualFileSystem()
|
||||
await vfs.init({
|
||||
distributed: true,
|
||||
nodes: [
|
||||
'vfs1.internal:8080',
|
||||
'vfs2.internal:8080',
|
||||
'vfs3.internal:8080'
|
||||
],
|
||||
replication: 3,
|
||||
consistency: 'eventual'
|
||||
})
|
||||
```
|
||||
|
||||
## Roadmap & Future Features
|
||||
|
||||
See [VFS ROADMAP](./ROADMAP.md) for planned features including:
|
||||
- Enhanced streaming support (v1.1)
|
||||
- Version history (v1.2)
|
||||
- Distributed filesystem (v1.2)
|
||||
- AI-powered automation (v2.0)
|
||||
- FUSE driver (v2.0 research)
|
||||
- And more community-requested features
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue