The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).
Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
readWriteSeparation, queryPlanner, healthMonitor, configManager,
hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
(slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
src/config/extensibleConfig.ts (config/augmentation registry built on removed
cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
enableDistributedSearch (dead config flag); the metadata partition field;
the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
storage-architecture; reframed scale prose to the 8.0 model.
Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.
RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:
1. validateConsistency() to falsely detect corruption on every startup,
triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
and report inflated totalEntries/totalIds stats
Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
Removed 9 undocumented feature sections from VFS docs (version history, distributed filesystem, AI auto-organization, etc.). Added status labels (Production/Beta/Experimental) to all features, updated performance claims with MEASURED vs PROJECTED labels, and created ROADMAP.md for planned features. Fixed storage adapter list to show only built-in adapters.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add brain.extract() and brain.extractConcepts() methods that use
NeuralEntityExtractor with embeddings and sophisticated NounType
taxonomy (30+ entity types) for semantic entity and concept extraction.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add safe tree operations that guarantee no directory appears as its own child:
- getDirectChildren() returns only immediate children
- getTreeStructure() builds safe tree with recursion protection
- getDescendants() gets all descendants efficiently
- inspect() provides comprehensive path information
Also includes VFSTreeUtils for building and validating tree structures.
This resolves the common infinite recursion issue when building file
explorers, as discovered by the Soulcraft Studio team.
Co-Authored-By: User <noreply@user.local>
Add production-ready Virtual File System with intelligent Knowledge Layer:
Core VFS Features:
- Complete file system operations (read, write, mkdir, etc.)
- Intelligent PathResolver with 4-layer caching system
- Chunked storage for large files with real compression
- Embedding generation for semantic operations
- File relationships and metadata tracking
- Import functionality from local filesystem
Knowledge Layer Integration:
- EventRecorder for complete file history and temporal coupling
- SemanticVersioning with content-based change detection
- PersistentEntitySystem for character/entity tracking across files
- ConceptSystem for universal concept mapping and graphs
- GitBridge for import/export between VFS and Git repositories
Architecture:
- KnowledgeAugmentation properly integrated into Brainy augmentation system
- KnowledgeLayer wrapper provides real-time VFS operation interception
- Background processing ensures VFS operations remain fast
- All components use real Brainy embed() method for embeddings
- Support for creative writing, coding projects, and project management
Technical Implementation:
- Fixed all stub/mock implementations with real working code
- TypeScript compilation passes without errors
- Comprehensive test suite demonstrating all features
- Documentation covering architecture and usage patterns
- Backwards compatible with existing Brainy functionality
This enables scenarios like writing books with persistent characters,
managing coding projects with concept tracking, and complete project
coordination with intelligent file relationships.
