- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
- **Examples**: Added a new `flush-statistics-example.js` script to demonstrate the usage of the `flushStatistics` method for ensuring updated statistics after data insertion.
- **Core**:
- Implemented `flushStatistics` in the `BrainyData` class to allow immediate flushing of statistics to storage.
- Updated `BaseStorageAdapter` with `flushStatisticsToStorage` to support flushing cached statistics.
- Modified the `shutDown` method to ensure statistics are flushed before database shutdown.
- **Documentation**: Added `statistics-flush-solution.md` to explain the batch update mechanism, the issue with delayed statistics updates, and how to manually flush statistics in storage.
**Purpose**: Provide users with the ability to manually flush statistics for real-time accuracy, particularly useful for systems relying on immediate updates. Improved documentation and examples to guide developers in implementing and using this functionality effectively.
- **Tests**: Added new `statistics-storage.test.ts` to validate statistics storage functionality across scenarios including saving, retrieving, time-based partitioning, and backward compatibility. Ensured tests dynamically handle missing environment variables by skipping S3-related tests when credentials are unavailable.
- **Docs**: Enhanced `statistics.md` with detailed explanations of scalability improvements, including adaptive flush timing, batched updates, and time-based partitioning. Improved readability and structure.
- **Storage**: Updated all storage adapters to integrate time-based partitioning and maintain backward compatibility with legacy statistics storage formats.
- **Dependencies**: Added `dotenv` to support environmental variable management for storage adapter tests.
**Purpose**: Strengthen system reliability by adding comprehensive test coverage for statistics storage, improve scalability documentation, and ensure consistency across storage adapters with robust implementations.
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
