The distributed-clustering subsystem never ran in production: it was inert, orphaned dead code (faked consensus, stub replication, no live wiring, and it did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process library. Scale is single-process + the optional native provider (@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools + horizontal read scaling (many reader processes, one writer). Removed: - src/distributed/ entirely (coordinator, shardManager, cacheSync, readWriteSeparation, queryPlanner, healthMonitor, configManager, hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts (slimmed to the live surface). - src/types/distributedTypes.ts; config.distributed field + JSDoc; coreTypes distributedConfig; memoryStorage distributedConfig persistence. - DistributedRole enum + src/config/distributedPresets.ts and the orphaned src/config/extensibleConfig.ts (config/augmentation registry built on removed cloud adapters + distributed presets), plus their src/index.ts re-exports. - 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook; enableDistributedSearch (dead config flag); the metadata partition field; the distributed_ reserved key prefix. - Orphaned src/storage/readOnlyOptimizations.ts (zero importers). - Tests targeting the subsystem: distributed-demo, distributed-cluster helper, distributed-transactions, sharding-transactions. - Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/ shard-manager/multi-node prose from v3-features, enterprise-for-everyone, augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP, vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4, storage-architecture; reframed scale prose to the 8.0 model. Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via getShardIdFromUuid — used live by baseStorage, unrelated to clustering); the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering. RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0 scale model.
5.7 KiB
VFS Roadmap - Planned Features
Status: These features are planned but not yet implemented. Current Version: See main VFS README for implemented features.
v1.1 (Next Release)
Enhanced Streaming Support
Improve VFS streaming to support true chunk-by-chunk reads for large files without loading entire content into memory.
Status: Planned Effort: 3-4 weeks
VFS Security Integration
Integrate Brainy's SecurityAPI with VFS file operations.
// Planned API (not yet implemented)
await vfs.encrypt('/sensitive-data.json', { algorithm: 'AES-256' })
await vfs.setACL('/project', { user: 'alice', permissions: 'rw-' })
const auditLog = await vfs.getAuditLog('/project', { since: lastWeek })
Status: Planned
Note: SecurityAPI exists (src/api/SecurityAPI.ts) but not integrated with VFS operations.
Atomic Operations Layer
Add transaction support for compound VFS operations.
// Planned API (not yet implemented)
await vfs.transaction(async (tx) => {
await tx.move('/src/file.txt', '/dest/file.txt')
await tx.writeFile('/dest/metadata.json', metadata)
// Both succeed or both fail
})
Status: Planned Effort: 1-2 weeks
v1.2
Version History
Track file versions with history and restoration capabilities.
// Planned API (not yet implemented)
await vfs.enableVersioning('/important-doc.md')
const versions = await vfs.getVersions('/important-doc.md')
await vfs.restoreVersion('/important-doc.md', versions[2].id)
const diff = await vfs.diffVersions('/important-doc.md', v1, v2)
Features:
- Automatic versioning on file changes
- Version history with metadata
- Point-in-time restoration
- Version comparison/diff
Status: Planned Effort: 4-5 weeks
Backup & Recovery API
Built-in backup and recovery operations.
// Planned API (not yet implemented)
const changes = await vfs.getChangesSince(lastBackupTime)
await vfs.createSnapshot('/backup-snapshot-2024')
await vfs.restoreToTime('/project', timestamp)
await vfs.verifyIntegrity('/project')
const issues = await vfs.repair('/project')
Status: Planned Effort: 3-4 weeks
v2.0
AI-Powered Auto-Organization
Intelligent file organization based on content and usage patterns.
// Planned API (not yet implemented)
await vfs.autoOrganize('/downloads', {
strategy: 'by-content-type',
createFolders: true
})
const duplicates = await vfs.detectDuplicates('/photos')
await vfs.deduplicateFiles(duplicates, { keep: 'highest-quality' })
await vfs.optimizeStorage('/project', {
compress: ['*.log', '*.txt'],
archive: { olderThan: '90d' }
})
Features:
- Content-based organization
- Smart deduplication
- Content-aware compression
- Automatic archival
Status: Planned - Research phase Effort: 8-10 weeks
Smart Collections / Virtual Directories
Persistent query-based virtual directories that auto-update.
// Planned API (not yet implemented)
await vfs.createVirtualDirectory('/auth-related', {
query: { concepts: { contains: 'authentication' }},
autoUpdate: true
})
// Directory automatically updates as matching files are added/modified
Note: Current VFS supports query-based access (e.g., /by-concept/auth) but not persistent virtual directories.
Status: Planned Effort: 4-5 weeks
FUSE Driver
Mount VFS as a native filesystem on Linux/Mac/Windows.
// Planned (research phase)
import { mountVFS } from '@soulcraft/brainy/vfs/fuse'
await mountVFS(vfs, {
mountPoint: '/mnt/brainy',
options: { allowOther: true }
})
Challenges:
- FUSE requires synchronous operations (VFS is async-first)
- Kernel-level integration complexity
- Cross-platform support (FUSE/Dokan/WinFsp)
Status: Research phase Effort: 10-12 weeks + significant testing
Community Contributions Wanted
These features would benefit from community contributions. If you're interested in building any of these, please open an issue!
Express.js Static Middleware
// Wanted: Community contribution
import { createStaticMiddleware } from '@soulcraft/brainy/vfs/express'
app.use('/files', createStaticMiddleware(vfs, {
index: ['index.html', 'index.md'],
etag: true,
cacheControl: 'max-age=3600'
}))
VSCode Extension
// Wanted: Community contribution
import { VFSProvider } from '@soulcraft/brainy/vfs/vscode'
const provider = new VFSProvider(vfs)
vscode.workspace.registerFileSystemProvider('brainy', provider)
Features:
- Browse VFS in VSCode explorer
- Semantic search from command palette
- Concept highlighting
- Relationship visualization
Webpack Plugin
Semantic-aware webpack builds with dependency graph from VFS relationships.
Vite Plugin
Vite integration with VFS for semantic module resolution.
Far Future (v5.0+)
Cloud Provider Auto-Detection
// Far future concept
const brain = new Brainy({
storage: 'cloud://brainy-data' // Auto-detects AWS/GCP/Azure
})
Hot/Cold Storage Tiering
Automatic data movement between hot (SSD/local) and cold (S3/archive) storage based on access patterns.
How to Track Progress
- GitHub Issues: Track feature development
- Discussions: Discuss feature design and requirements
- Pull Requests: Submit contributions
Contributing
Interested in implementing a planned feature? Here's how to get started:
- Open an issue discussing the feature you want to implement
- Review the design - we'll help with architecture decisions
- Submit a PR with implementation + tests
- Celebrate! Your contribution helps everyone 🎉
Last Updated: 2025-10-29 VFS Version: 4.9.0