- **Removed Files**:
- Deleted outdated statistics documentation files (`statistics.md`, `statistics-flush-solution.md`, `statistics-summary.md`) to clean up the repository and avoid confusion.
- **Added Standards**:
- Introduced `DOCUMENTATION_STANDARDS.md` to outline naming conventions and troubleshooting practices for more consistent and maintainable project documentation.
- **Tests**:
- Added a new test file `edge-cases.test.ts` to verify handling of edge cases, ensuring robust behavior against boundary values and invalid inputs.
**Purpose**: Cleans up deprecated documentation while introducing concrete standards for maintaining and updating documentation. Enhances test coverage for unusual or boundary inputs, improving overall system resilience.
- **Vector Handling Updates**:
- Added a `dimensions` property to `BrainyDataConfig` for specifying vector dimensions.
- Introduced validation for vector dimensions during database creation and insertion to ensure consistency.
- Enhanced error handling and logging for dimension mismatches.
- **Model Loading Improvements**:
- Implemented retry logic for Universal Sentence Encoder model loading to handle network instability and JSON parsing errors gracefully.
- Improved logging and debugging support for failures during model initialization and embedding operations.
- **Compatibility Enhancements**:
- Updated polyfills to support TensorFlow.js compatibility across diverse server environments (Node.js, serverless, etc.).
- Introduced and refactored global `TextEncoder`/`TextDecoder` definitions for seamless operation in non-browser environments.
- Simplified TensorFlow.js backend setup with streamlined imports and logging for GPU/WebGL fallback.
- **Purpose**:
- These updates improve BrainyData's robustness, enforce correct vector usage, and extend compatibility with varied runtime environments. The changes enhance the usability, reliability, and cross-platform readiness of core functionalities.
- Removed obsolete scripts: `test-all-environments.js`, `test-fallback-function.js`, `test-fallback-simple.js`, `test-fix.js`, `test-tensorflow-textencoder.js`, `test-unified-encoding.js`, and `test-worker-utils.js`.
- Introduced `scripts/comprehensive-test.js` as a unified testing script covering all environments: Browser, Node.js, and CLI.
- Added `examples/cli-wrapper-example.js` to demonstrate a proper CLI implementation with TensorFlow.js initialization.
This refactor simplifies the testing structure by consolidating redundant scripts into a single comprehensive script while ensuring robust cross-environment coverage.
- Added batch embedding support with `defaultBatchEmbeddingFunction`, leveraging shared model instances for optimized performance.
- Integrated `isInitializing` flag to prevent recursive initialization and ensure smooth concurrent operation handling during `BrainyData` initialization.
- Pre-loaded Universal Sentence Encoder in `BrainyData` to prevent delays during embedding.
- Introduced fallback mechanisms in embedding initialization for better error resiliency and model reusability.
- Updated `addBatch` with support for batchSize and refactored text/vector processing logic for clearer separation and memory management.
- Improved GPU and CPU backend selection in Universal Sentence Encoder for compatibility across environments.
- Enhanced memory management by cleaning tensors after embedding operations.
- Updated README with instructions for batch embedding, threading updates, and GPU/CPU optimizations.
- Integrated GPU acceleration via WebGL in `embedding.ts` and `distance.ts` for optimized performance.
- Enhanced fallback mechanisms for CPU processing when GPU is unavailable or fails to initialize.
- Added threading support for embedding and distance calculations in `embedding.ts` and `distance.ts` to improve performance and avoid blocking the main thread.
- Refactored TensorFlow.js imports to dynamically load modules (`@tensorflow/tfjs-core`, `@tensorflow/tfjs-backend-webgl`, `@tensorflow/tfjs-backend-cpu`) for modular dependency usage.
- Introduced GPU-accelerated batch distance calculations in `distance.ts` with appropriate error handling.
- Applied consistent formatting to improve code readability and maintainability while adhering to project style guidelines.
