Remove unused files and implement proper version handling:
- Remove unused files: tensorflowUtils.ts, patched-platform-node.ts, test reporters
- Fix 5 TODO items with centralized version management in utils/version.ts
- Clean up duplicate metadata definitions in examples/basicUsage.ts
- Fix rollup config to use @rollup/plugin-terser instead of deprecated package
- Add comprehensive migration plan for deprecated methods (12 methods identified)
This cleanup removes 370 lines of dead code while maintaining full API compatibility.
All tests pass and build system works correctly.
- Added support for field-specific and prioritized searches in `brainyData`:
- Introduced `searchField` option to enable targeted field-level searches.
- Implemented `priorityFields` option for weighted vectorization and query relevance.
- Developed utilities in `jsonProcessing.ts` and `fieldNameTracking.ts`:
- `extractTextFromJson` for text extraction with customizable depth and field prioritization.
- `extractFieldFromJson` to target specific fields in JSON documents.
- `prepareJsonForVectorization` for optimized JSON vectorization.
- Enhanced management of field names and mappings:
- Integrated `trackFieldNames` to associate fields with their services.
- Supported cross-service consistency through `standardFieldMappings`.
- Updated documentation:
- Added detailed guides for JSON search enhancements and HNSW limitations.
- Extended usage examples in `README.md` and `json-search-test.js`.
- Verified improvements with comprehensive tests:
- Created unit and integration tests demonstrating search behavior improvements.
- Addressed previous TypeScript errors related to search parameters.
**Purpose**: Improve search accuracy and usability when working with complex JSON documents by enabling field-specific searches and enhancing contextual relevance.
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
- Added batch embedding support with `defaultBatchEmbeddingFunction`, leveraging shared model instances for optimized performance.
- Integrated `isInitializing` flag to prevent recursive initialization and ensure smooth concurrent operation handling during `BrainyData` initialization.
- Pre-loaded Universal Sentence Encoder in `BrainyData` to prevent delays during embedding.
- Introduced fallback mechanisms in embedding initialization for better error resiliency and model reusability.
- Updated `addBatch` with support for batchSize and refactored text/vector processing logic for clearer separation and memory management.
- Improved GPU and CPU backend selection in Universal Sentence Encoder for compatibility across environments.
- Enhanced memory management by cleaning tensors after embedding operations.
- Updated README with instructions for batch embedding, threading updates, and GPU/CPU optimizations.
