BREAKING CHANGES:
- Removed deprecated getAllNouns() and getAllVerbs() methods
- All internal usage migrated to pagination-based methods
New Features:
- Filter Discovery API:
- getFilterValues(field): Get all available values for a field
- getFilterFields(): Get all filterable fields
- Enables dynamic filter UI generation with O(1) field discovery
- Hybrid metadata indexing with field-level indexes
- Adaptive auto-flush for optimal performance
- LRU caching for metadata indexes
Improvements:
- Fixed ENAMETOOLONG errors from vector-based filenames
- Safe filename generation using hash-based approach
- Scalable chunked value storage for millions of entries
- Performance optimization with adaptive flush thresholds
- Added support for $includes operator in metadata filters
Technical:
- Replaced vector-based filenames with safe hash approach
- Implemented MetadataIndexCache with existing SearchCache pattern
- Field indexes enable O(1) filter discovery
- Adaptive flush based on performance metrics (20-200 entries)
- All tests passing with improved metadata filtering
Add full support for tracking and analyzing data by service in multi-tenant deployments.
## Features Added
- **Service Statistics Tracking**: Track nouns, verbs, and metadata counts per service
- **Service Activity Monitoring**: Track first/last activity timestamps and operation counts
- **New API Methods**:
- `listServices()`: List all services with their statistics and status
- `getServiceStatistics(service)`: Get detailed stats for a specific service
- Enhanced `getStatistics()` with service filtering and breakdown
- **Service Filtering**: Filter search results and queries by service
- **Storage Enhancements**: BaseStorageAdapter tracks service activity with timestamps
- **Type Definitions**: Added ServiceStatistics interface and extended StatisticsData
## Implementation Details
- Services automatically tracked via defaultService config or per-operation override
- Service status detection (active/inactive/read-only) based on activity
- Memory-efficient tracking at statistics level, not per noun/verb
- Backward compatible - existing data tracked under 'default' service
## Documentation
- Comprehensive guide in docs/guides/per-service-statistics.md
- Examples for multi-tenant apps, health monitoring, and auditing
- API reference and migration guide included
## Testing
- Full test suite in tests/service-statistics.test.ts
- Coverage of all new methods and filtering capabilities
This enables better observability, debugging, and management of multi-service Brainy deployments, addressing the need to track individual service performance when multiple services share storage.
- Add frozen flag to separate data immutability from performance optimizations
- readOnly: prevents data mutations but allows index optimizations (default behavior)
- frozen: prevents ALL changes including statistics and index updates
- Smart default: frozen=false when readOnly=true for optimal performance
- Add comprehensive documentation for read-only and frozen modes
- Created docs/guides/readonly-frozen-modes.md with detailed guide
- Added examples for compliance, forensics, and testing use cases
- Updated all documentation indexes with new guide links
- Simplify README.md to emphasize unified API
- Clearer demonstration that same code works everywhere
- Simplified framework examples showing consistent API
- Better noun/verb examples for entities and relationships
- Collapsible sections for cloud platform examples
- Environment auto-detection table
- Add tests for frozen flag behavior
- Test readOnly without frozen (allows optimizations)
- Test frozen mode (complete immutability)
- Test dynamic mode switching
BREAKING CHANGE: readOnly behavior changed - now allows optimizations by default.
To get old behavior (complete immutability), use readOnly: true with frozen: true.
Add @soulcraft/brainy-models as optional dependency for zero-config offline reliability. Enhance robustModelLoader with hierarchical loading strategy (local → online → fail). Add comprehensive production deployment documentation and update README with clear benefits.
This solves critical production issues where Universal Sentence Encoder fails to load in Docker/Cloud Run environments due to network timeouts or blocked URLs. The solution provides 100% offline reliability while maintaining backward compatibility and requires no code changes from users.
- **Documentation**:
- Added a detailed explanation in `model-management.md` for resolving `"format"` field compatibility issues in TensorFlow.js.
- Introduced a dual-layer protection approach to mitigate errors like `RangeError: byte length of Float32Array should be a multiple of 4`.
- **Scripts**:
- Removed the redundant `release:minor` script entry from `package.json`.
- Enhanced `_deploy` script consistency.
- **Protection Mechanisms**:
- Updated `download-full-models.js` to inject a missing `"format"` field during downloads.
- Enhanced `RobustModelLoader` to validate and restore the `"format"` field automatically at runtime, ensuring persistence and compatibility
- Introduced new documentation files under `docs/`:
- `model-bundling-analysis.md`: Provides detailed analysis of current, bundled, hybrid, and dynamic model loading approaches, including pros, cons, and recommendations.
- `model-management.md`: Explains how Brainy manages Universal Sentence Encoder models, including setup, usage, and troubleshooting.
- `optional-model-bundling.md`: Details the `@soulcraft/brainy-models` package for offline reliability with pre-bundled models.
- Added `src/utils/robustModelLoader.ts`:
- Implements enhanced model loading with retry mechanisms, timeout handling, fallback URLs, and optional local model bundling.
- Supports Node.js and browser environments with exponential backoff logic.
- Key Updates:
- **Hybrid Loading Strategy**: Recommended for balancing reliability and flexibility via hybrid online/offline mechanisms.
- **Enhanced Fallback Scenarios**: Robust loader improves network-dependent reliability for embedding workflows.
- **Offline Reliability Support**: Optional model bundling eliminates dependency on external services, supporting air-gapped and edge environments.
**Purpose**: Introduce a hybrid model loading approach with robust options for
- Added `cache-configuration.md` under `docs/guides`:
- Detailed multi-level cache system (hot, warm, cold) overview.
- Explained new adaptive tuning capabilities:
- Memory-based adjustments across Node.js, Browser, and Worker environments.
- Dynamic sizing for read-heavy/write-heavy workloads.
- Environment-specific configurations for optimal caching.
- Included best practices for large datasets, memory-constrained and read-only environments.
- Added monitoring and advanced manual tuning instructions.
- Modified `cacheManager.ts`:
- Introduced `environmentConfig` for tailored per-environment cache settings.
- Enhanced auto-tuning with support for dynamic memory detection and cache hit/miss ratio.
- Added fine-grained tuning for eviction thresholds, TTLs, and batch sizes based on workload characteristics.
- Improved adaptive tuning with async memory detection and detailed cache statistics tracking.
**Purpose**: Provide developers with detailed guidance and dynamic tools for optimizing Brainy's cache system, ensuring better performance across diverse environments and workloads.
- Created `service-identification.md` in `docs/guides`:
- Detailed guidelines on how services should identify themselves within Brainy.
- Documented two identification methods: default service initialization and operation-specific service naming.
- Included service name conventions and common examples (`github`, `reddit`, `default`).
- Described benefits of proper service identification:
- Enhanced statistics tracking and JSON field discovery by service.
- Provided best practices for consistent and descriptive service naming.
- Explained internal implementation details, such as `getServiceName` usage and statistic tracking.
**Purpose**: Help users properly identify services to enable statistics tracking, field discovery, and improved data management in Brainy.
- Added support for field-specific and prioritized searches in `brainyData`:
- Introduced `searchField` option to enable targeted field-level searches.
- Implemented `priorityFields` option for weighted vectorization and query relevance.
- Developed utilities in `jsonProcessing.ts` and `fieldNameTracking.ts`:
- `extractTextFromJson` for text extraction with customizable depth and field prioritization.
- `extractFieldFromJson` to target specific fields in JSON documents.
- `prepareJsonForVectorization` for optimized JSON vectorization.
- Enhanced management of field names and mappings:
- Integrated `trackFieldNames` to associate fields with their services.
- Supported cross-service consistency through `standardFieldMappings`.
- Updated documentation:
- Added detailed guides for JSON search enhancements and HNSW limitations.
- Extended usage examples in `README.md` and `json-search-test.js`.
- Verified improvements with comprehensive tests:
- Created unit and integration tests demonstrating search behavior improvements.
- Addressed previous TypeScript errors related to search parameters.
**Purpose**: Improve search accuracy and usability when working with complex JSON documents by enabling field-specific searches and enhancing contextual relevance.
- Deleted `CHANGES.md`, `CHANGES_SUMMARY.md`, `CONCURRENCY_ANALYSIS.md`, `CONCURRENCY_IMPLEMENTATION_SUMMARY.md`, and related developer documentation files.
- Removed redundant or legacy content no longer aligned with the current codebase and workflows.
- Updated repository to reflect streamlined documentation approach, reducing clutter and improving maintainability.
**Purpose**: Simplify and declutter repository by removing obsolete documentation files, ensuring it remains focused and relevant.
