Major optimization - all type embeddings now built into package:
Build-time generation:
- Created scripts/buildTypeEmbeddings.ts to generate all type embeddings
- Generates embeddings for 31 NounTypes + 40 VerbTypes at build time
- Stores as base64-encoded binary data in embeddedTypeEmbeddings.ts
- Added check script to rebuild only when needed
Updated all consumers:
- NeuralEntityExtractor: loads pre-computed embeddings (instant)
- BrainyTypes: loads pre-computed embeddings (instant init)
- NaturalLanguageProcessor: loads pre-computed embeddings (instant init)
Build process:
- Added npm run build:types to generate embeddings
- Added npm run build:types:if-needed for conditional rebuild
- Integrated into main build pipeline
- Auto-rebuilds only when types or build script change
Benefits:
- Zero runtime cost - embeddings loaded instantly
- Survives all container restarts
- All 71 types always available (31 nouns + 40 verbs)
- ~100KB memory overhead for permanent performance gain
- Eliminates 5-10 second initialization delay
This completes the type embedding optimization started in v3.32.5
Major performance improvement for large file imports:
- Neural entity extraction now only initializes requested types
- Reduces initialization from 31 types to 2-5 types for concept extraction
- Fixed apparent hang in Excel/PDF/Markdown imports with concept extraction
Technical changes:
- Modified NeuralEntityExtractor.initializeTypeEmbeddings() to accept requestedTypes parameter
- Updated extract() to pass options.types to initialization
- Re-enabled concept extraction by default in SmartExcelImporter
- Added enhanced GCS diagnostic logging for initialization troubleshooting
Performance impact:
- Small files (<100 rows): 5-20 seconds (was: appeared to hang)
- Medium files (100-500 rows): 20-100 seconds (was: timeout)
- Large files (500+ rows): Can be disabled if needed
Fixes critical production issue where brain.extractConcepts() caused timeouts
Critical fix for GCS/S3 native storage adapters crashing on metadata index keys.
Problem:
- GCS and S3 adapters crashed with "Invalid UUID format" errors
- System keys like __metadata_field_index__status are NOT UUIDs
- Adapters incorrectly tried to shard all metadata keys as UUIDs
Solution:
- Move sharding/routing logic from adapters to BaseStorage class
- Add analyzeKey() method to detect system keys vs entity UUIDs
- System keys route to _system/ directory (no sharding)
- Entity UUIDs route to sharded directories (256 shards)
- All adapters now implement 4 primitive operations:
* writeObjectToPath(path, data)
* readObjectFromPath(path)
* deleteObjectFromPath(path)
* listObjectsUnderPath(prefix)
Benefits:
- Impossible for future adapters to repeat this mistake
- Zero breaking changes, full backward compatibility
- No data migration required
- Cleaner architecture with better separation of concerns
Updated adapters: GcsStorage, S3CompatibleStorage, OPFSStorage,
FileSystemStorage, MemoryStorage
Added: docs/architecture/data-storage-architecture.md
Updated: README.md with architecture docs link
- Removed ImportManager class and exports (use brain.import() instead)
- Fixed all documentation: getStatistics() → getStats()
- Updated 41 files across codebase for consistency
- Removed ImportManager section from API docs
- Added v3.30.0 migration guide to CHANGELOG
Co-Authored-By: Claude <noreply@anthropic.com>
Removes all traces of BrainyData to prevent user confusion:
- Renamed brainyDataInterface.ts to brainyInterface.ts for clarity
- Updated all imports and type references across 5 files
- Removed BrainyData compiled artifacts (handled by clean build)
- Added deprecation notice to CHANGELOG with migration guide
BrainyData was never part of official Brainy 3.0 API but existed as
legacy compiled artifacts. Users mistakenly imported it thinking neural
API was missing, when it exists in modern Brainy class.
All users should migrate to: new Brainy() with await brain.init()
Neural API available via: brain.neural().visualize() etc.
Resolves confusion reported by Brain Studio team.
Implement comprehensive conversation management system enabling AI agents
like Claude Code to maintain infinite context and history. Provides semantic
search, smart context retrieval, and automatic artifact linking using Brainy's
existing Triple Intelligence infrastructure.
Core Features:
- ConversationManager API for message storage and retrieval
- MCP protocol integration with 6 tools for Claude Code
- Context ranking using semantic, temporal, and graph scoring
- Neural clustering for theme discovery and deduplication
- Virtual filesystem integration for code artifact linking
- CLI commands for setup and management
Zero new infrastructure required - uses existing Brainy features:
- Storage via brain.add() with NounType.Message
- Relationships via brain.relate() with VerbType.Precedes
- Search via brain.find() with Triple Intelligence
- Clustering via brain.neural()
- Artifacts via brain.vfs()
One-command setup: brainy conversation setup
Version: 3.19.0
- Fix FileSystemStorage sharding: getAllShardedFiles() for proper directory traversal
- Fix getNode() metadata: was filtering out metadata causing VFS entity failures
- Add production-scale streaming pagination for millions of entities
- Optimize sharding threshold from 1000 to 100 files for better performance
- Fix VFS readdir() and tree operations for complete directory structure support
- Add verb count tracking and persistence for performance optimizations
🚀 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix root directory metadata to always have vfsType: 'directory'
- Add compatibility layer for malformed entity metadata
- Ensure Contains relationships are maintained for all file operations
- Fix resolvePath() special case for root directory
- Add comprehensive documentation for VFS troubleshooting
- Document proper usage of standard NounType and VerbType enums
Resolves critical VFS bugs reported by brain-cloud team
Brainy 3.0 Production Release - World's first Triple Intelligence database
- Complete API redesign with add(), find(), update(), delete(), relate()
- Unified vector, graph, and document search in one API
- Zero-config validation system with production-ready type safety
- Advanced neural clustering with comprehensive algorithms
- Built-in augmentation system (cache, display, metrics)
- 100+ comprehensive tests covering all APIs and edge cases
BREAKING CHANGES: All 2.x APIs replaced with new 3.0 syntax
- Unified embedding system with single EmbeddingManager
- Q8 model support with 75% smaller footprint (23MB vs 90MB)
- Intelligent precision auto-selection based on environment
- Clean cached embeddings with TTL and memory management
- Zero-config setup with smart defaults
- Complete storage structure documentation
- Removed legacy worker and hybrid managers
- Streamlined model configuration and precision management
