- Added support for field-specific and prioritized searches in `brainyData`:
- Introduced `searchField` option to enable targeted field-level searches.
- Implemented `priorityFields` option for weighted vectorization and query relevance.
- Developed utilities in `jsonProcessing.ts` and `fieldNameTracking.ts`:
- `extractTextFromJson` for text extraction with customizable depth and field prioritization.
- `extractFieldFromJson` to target specific fields in JSON documents.
- `prepareJsonForVectorization` for optimized JSON vectorization.
- Enhanced management of field names and mappings:
- Integrated `trackFieldNames` to associate fields with their services.
- Supported cross-service consistency through `standardFieldMappings`.
- Updated documentation:
- Added detailed guides for JSON search enhancements and HNSW limitations.
- Extended usage examples in `README.md` and `json-search-test.js`.
- Verified improvements with comprehensive tests:
- Created unit and integration tests demonstrating search behavior improvements.
- Addressed previous TypeScript errors related to search parameters.
**Purpose**: Improve search accuracy and usability when working with complex JSON documents by enabling field-specific searches and enhancing contextual relevance.
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
- **Examples**: Added a new `flush-statistics-example.js` script to demonstrate the usage of the `flushStatistics` method for ensuring updated statistics after data insertion.
- **Core**:
- Implemented `flushStatistics` in the `BrainyData` class to allow immediate flushing of statistics to storage.
- Updated `BaseStorageAdapter` with `flushStatisticsToStorage` to support flushing cached statistics.
- Modified the `shutDown` method to ensure statistics are flushed before database shutdown.
- **Documentation**: Added `statistics-flush-solution.md` to explain the batch update mechanism, the issue with delayed statistics updates, and how to manually flush statistics in storage.
**Purpose**: Provide users with the ability to manually flush statistics for real-time accuracy, particularly useful for systems relying on immediate updates. Improved documentation and examples to guide developers in implementing and using this functionality effectively.
- **Tests**: Added new `statistics-storage.test.ts` to validate statistics storage functionality across scenarios including saving, retrieving, time-based partitioning, and backward compatibility. Ensured tests dynamically handle missing environment variables by skipping S3-related tests when credentials are unavailable.
- **Docs**: Enhanced `statistics.md` with detailed explanations of scalability improvements, including adaptive flush timing, batched updates, and time-based partitioning. Improved readability and structure.
- **Storage**: Updated all storage adapters to integrate time-based partitioning and maintain backward compatibility with legacy statistics storage formats.
- **Dependencies**: Added `dotenv` to support environmental variable management for storage adapter tests.
**Purpose**: Strengthen system reliability by adding comprehensive test coverage for statistics storage, improve scalability documentation, and ensure consistency across storage adapters with robust implementations.
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
