BREAKING CHANGE: System metadata location changed from 'index/' to '_system/' directory
- Rename INDEX_DIR to SYSTEM_DIR following database conventions
- Implement dual-read/write strategy for zero-downtime migration
- Add automatic migration from old to new location on first access
- Support mixed service versions sharing S3/cloud storage
- Add 30-day grace period for gradual rollout (configurable)
- Store distributed config alongside statistics in _system folder
- Add comprehensive migration guide and documentation
Migration features:
- Read from both locations (new first, fallback to old)
- Write to both during migration period
- Automatic data migration when found only in old location
- Services can update independently without coordination
- Full backward compatibility for production deployments
The change improves clarity ('_system' better represents system metadata than 'index')
and follows standard database conventions (MongoDB's _system, PostgreSQL's pg_*).
- Fix deprecated getAllNodes() warnings by using getNodesWithPagination() in S3 adapter
- Add getNounsWithPagination() method to S3CompatibleStorage for proper pagination support
- Create optimizedS3Search module for efficient pagination and filtering
- Update baseStorage to properly detect pagination support in adapters
- Add comprehensive documentation for performance and logging fixes
- Ensure backward compatibility with existing code
This resolves the following warnings in dependent projects:
- "getAllNodes() is deprecated and will be removed in a future version"
- "Storage adapter does not support pagination, falling back to loading all nouns"
- "Only returning the first 1000 nodes. There are more nodes available"
- Updated MemoryStorage and BaseStorage to handle HNSWVerb instead of GraphVerb.
- Introduced methods to save and retrieve verb metadata separately.
- Enhanced getVerb and getAllVerbs methods to convert HNSWVerb to GraphVerb with metadata.
- Improved data handling and filtering in various storage methods.
- **Documentation Additions**:
- Introduced `SEARCH_AND_METADATA_GUIDE.md` to provide an in-depth guide on Brainy's search and metadata retrieval system:
- Detailed explanation of search workflows, metadata structures (`GraphNoun`, `GraphVerb`), and core components like `SearchResult`.
- Usage examples showcasing search queries, filtering by noun/verb types, and advanced features like multi-modal search.
- Included performance tips on caching, HNSW indexing, lazy loading, and augmentation pipeline.
- **Storage System Updates**:
- Enhanced memory and file storage adapters to support dedicated noun and verb metadata handling:
- Added methods `saveN
- Added a new `CacheManager` class in `cacheManager.ts` to support three-level caching strategy:
- **Level 1**: Hot cache in RAM for most accessed nodes.
- **Level 2**: Warm cache using OPFS, Filesystem, or S3, depending on the environment.
- **Level 3**: Cold storage for longer-term data storage.
- Integrated features for dynamic tuning:
- Auto-detection of environment (Browser, Node.js, Worker) and memory availability.
- Parameter tuning for cache size, eviction thresholds, and TTL based on usage patterns.
- Enhanced support for:
- LRU-based eviction in hot cache.
- Batch-based operations with configurable batch sizes.
- Comprehensive logging and debug outputs for cache operations.
- Ensured robust fallback handling to manage storage in constrained environments.
- Improved extensibility for storage adapters (warm and cold storage detection and initialization).
**Purpose**: Optimize data access and storage across multiple environments with seamless scalability and dynamic parameter adjustments.
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
- **Core**: Improved verb creation logic by adding `createdAt`, `updatedAt`, and `createdBy` attributes. These fields include timestamped metadata (`seconds`, `nanoseconds`) and source augmentation/service information for better tracking.
- **Storage**: Refactored `BaseStorage` methods to utilize internal variants (e.g., `saveVerb_internal`, `getNoun_internal`). Added support for new verb attributes while maintaining backward compatibility with existing data structures.
- **Tests**:
- Updated `s3-storage.test.ts` and `opfs-storage.test.ts` to validate changes in verb attributes such as timestamps and augmentation metadata.
- Added assertions for `createdAt`, `updatedAt`, and `createdBy` fields in test cases.
- **Cleanup**: Replaced ambiguous type aliases like `Edge` and `HNSWNode` with clearer equivalents (`Verb` and `HNSWNoun_internal`) for consistency across storage adapters.
**Purpose**: Enhance metadata tracking and standardize attribute handling across storage and core modules to ensure accurate and consistent data throughout the system.
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
- Added `BaseStorage` abstract class to provide common functionality for storage adapters.
- Implemented `FileSystemStorage` adapter for Node.js environments, supporting persistent file-based storage.
- Added directory structure management methods to `FileSystemStorage` (`ensureDirectoryExists`, `clear`, etc.).
- Implemented `MemoryStorage` adapter for in-memory storage suitable for non-persistent environments.
- Supported CRUD operations for nouns, verbs, and metadata in both `FileSystemStorage` and `MemoryStorage`.
- Included utility methods like `mapToObject` for serialization and deserialization of complex data structures.
Purpose: Provide extensible, reusable storage solutions with both persistent and non-persistent options for diverse application needs.
