BREAKING CHANGE: Complete migration from TensorFlow.js to Transformers.js for embedding generation
This is a major architectural change that replaces TensorFlow.js (USE model) with Transformers.js (all-MiniLM-L6-v2) for significantly improved performance and reduced complexity.
Key Changes:
- Replace TensorFlow.js Universal Sentence Encoder with Transformers.js all-MiniLM-L6-v2
- Reduce model size from 525MB to 87MB (83% reduction)
- Reduce embedding dimensions from 512 to 384 (faster distance calculations)
- Remove TensorFlow.js Float32Array patching (caused ONNX conflicts)
- Implement smart bundled model detection for offline operation
- Add explicit model download script for Docker deployments
- Remove complex environment variables in favor of simple configuration
- Update all distance functions to use optimized pure JavaScript
- Remove TensorFlow-specific utilities and type definitions
Performance Improvements:
- Model loading: 5x faster (87MB vs 525MB)
- Memory usage: 75% reduction (~200-400MB vs ~1.5GB)
- Distance calculations: Faster pure JS vs GPU overhead for small vectors
- Cold start performance: Significantly improved
Files Changed:
- Updated package.json: New dependencies, simplified scripts
- Rewrote src/utils/embedding.ts: Complete Transformers.js implementation
- Updated src/utils/distance.ts: Optimized JavaScript distance functions
- Simplified src/setup.ts: Removed TensorFlow-specific patching
- Simplified src/utils/textEncoding.ts: Only Node.js TextEncoder/Decoder patches
- Deleted src/utils/robustModelLoader.ts: TensorFlow-specific loader
- Deleted src/types/tensorflowTypes.ts: TensorFlow type definitions
- Added scripts/download-models.cjs: Docker-compatible model downloader
- Added comprehensive documentation: README.md, OFFLINE_MODELS.md, analysis docs
Testing:
- All 19 tests passing
- Removed test mocking in favor of real implementation testing
- Updated test environment for Transformers.js compatibility
- Performance tests validate improved efficiency
This migration resolves production issues with Docker egress limitations and provides a more robust, performant foundation for vector operations.
- Add complete Universal Sentence Encoder Lite model (27MB)
- Include vocab.json for tokenization support
- Update package to work with @tensorflow-models/universal-sentence-encoder
- Ensure offline model loading capability for Docker deployments
- Published to npm as @soulcraft/brainy-models@0.8.0
Remove development artifacts, test files, and redundant directories:
- Delete debug/reproduction scripts and temporary test files
- Remove brainy-models-package/ (redundant with main models/ directory)
- Remove test-consumer/ development testing directory
- Remove build artifacts (coverage/, test-results.json)
- Remove large brainy-data/ test artifact directory
This cleanup reduces repository size significantly and prepares the project for a clean release.
- **New Scripts**:
- Created `reproduce_race_condition.cjs` to demonstrate and debug race condition issues in `Brainy`. This includes:
- Scenarios where verbs arrive before nouns.
- Testing indexing delays and streaming simulations.
- Evaluation of the `autoCreateMissingNouns` feature.
- Added `reproduce_writeonly_issue.js` to reproduce and verify issues with write-only mode:
- Ensures add operations succeed while search operations give appropriate errors.
- Handles placeholder nouns and validates their replacement with real data.
- Developed `test_race_condition_fixes.cjs` to verify the implemented fixes:
- Covers scenarios for `writeOnlyMode`, fallback storage lookups, and missing noun auto-creation.
- **Documentation Updates**:
- Added `
- **Documentation**:
- Added a detailed explanation in `model-management.md` for resolving `"format"` field compatibility issues in TensorFlow.js.
- Introduced a dual-layer protection approach to mitigate errors like `RangeError: byte length of Float32Array should be a multiple of 4`.
- **Scripts**:
- Removed the redundant `release:minor` script entry from `package.json`.
- Enhanced `_deploy` script consistency.
- **Protection Mechanisms**:
- Updated `download-full-models.js` to inject a missing `"format"` field during downloads.
- Enhanced `RobustModelLoader` to validate and restore the `"format"` field automatically at runtime, ensuring persistence and compatibility
- **Scripts**:
- Refactored logging within `release-workflow.js` for improved readability and maintainability.
- Added a fallback mechanism in `download-full-models.js` to inject the "format" field into `model.json` for TensorFlow.js compatibility.
- **Documentation**:
- Updated `README.md` with a redesigned structure:
- Enhanced readability using emojis and a cleaner presentation.
- Expanded `Overview`, `Features`, and `Quick Start` sections.
- Refined "Use Cases" and added better explanations for bundled model benefits.
- **Models**:
- Adjusted `model.json` to include the "format" field for compatibility with TensorFlow.js.
- Updated `metadata.json` with a recent download timestamp.
**Purpose
- Added `CODE_OF_CONDUCT.md` to establish community standards for behavior and inclusivity.
- Added `CONTRIBUTING.md` with detailed guidelines for contributing to the `brainy-models-package`:
- Model quality, testing, and optimization requirements.
- Development setup instructions, including build and test scripts.
- Pull request and commit message conventions.
- Package-specific utility scripts and file structure overview.
**Purpose**: Provide clear contribution and community guidelines to foster collaboration and maintain project quality standards.
- Added `.versionrc.json` to define conventional release configurations for `brainy-models-package`:
- Configured tag prefix as `brainy-models-v`.
- Defined custom commit message format and section mapping for release notes.
- Added `LICENSE` file with MIT license for legal clarity.
- Updated `README.md`:
- Adjusted internal NPM script names to use underscore prefix for consistency (`_release:patch`, `_github-release`).
**Purpose**: Establish standardized versioning, licensing, and script conventions for the `brainy-models-package` to improve maintainability and alignment with project standards.
- Introduced `@soulcraft/brainy-models` package with pre-bundled TensorFlow models for enhanced offline reliability.
- Added `index.d.ts` and `index.js` allowing offline embedding workflows with the Universal Sentence Encoder model.
- Included utility scripts for model compression, size retrieval, and availability checks.
- Added `metadata.json` and `model.json` defining the Universal Sentence Encoder configuration with offline bundling.
- Ensured comprehensive model documentation, error handling, and robust logging for seamless integration.
- Supported optional model quantization placeholders for future TensorFlow.js enhancements.
**Purpose**: Enable fully offline-ready embedding workflows via pre-bundled Universal Sentence Encoder models, ensuring maximum reliability and air-gapped environment compatibility.
- Removed `demo-optional-model-bundling.js`:
- Obsolete demonstration of model bundling and offline loading.
- Replaced by comprehensive documentation and tools in `@soulcraft/brainy-models`.
- Added `create-github-release.js`:
- Automates GitHub release creation for `@soulcraft/brainy-models-package`.
- Includes features for generating release notes, tagging, and uploading with GitHub CLI.
- Updated `package-lock.json`:
- Reflects new dependencies and updates for GitHub release automation.
**Purpose**: Streamline repository by removing redundant scripts and introducing automated GitHub release workflows for efficient version management.
- Added new scripts under `brainy-models-package/scripts`:
- **`compress-models.js`**: Implements model compression with float16 and int8 precision to create optimized variants of Universal Sentence Encoder models.
- **`download-full-models.js`**: Downloads the complete Universal Sentence Encoder model for offline usage.
- **`download-model.js`**: Downloads reference files for TensorFlow Hub-based Universal Sentence Encoder.
- Introduced a demonstration script:
- **`demo-optional-model-bundling.js`**: Highlights the solution of bundling models to eliminate network dependency, ensuring reliability and offline capability.
- Key Features:
- **Compression**:
- Reduced model size with float16 (balanced precision and size) and int8 (low-memory environments) options.
- Generated compression summaries for quick insights into model variants and saved space.
- **Offline Reliability**:
- Bundled versions eliminate first-load delays, network dependencies, and failures.
- Ensures rapid initialization in offline and memory-constrained scenarios.
- **Dynamic Optimization**:
- Tailored optimization profiles for various use cases: general, low-memory, and high-performance.
- **Demonstration and Documentation**:
- Comprehensive demo showcasing benefits of bundled models over online loading.
- Examples for usage, testing, and integration with Brainy.
**Purpose**: Introduce essential scripts and tools to enable efficient, offline-ready model usage, streamlining the embedding workflow while ensuring reliability in production and resource-constrained environments.
