chore: clean up project for release
Remove development artifacts, test files, and redundant directories: - Delete debug/reproduction scripts and temporary test files - Remove brainy-models-package/ (redundant with main models/ directory) - Remove test-consumer/ development testing directory - Remove build artifacts (coverage/, test-results.json) - Remove large brainy-data/ test artifact directory This cleanup reduces repository size significantly and prepares the project for a clean release.
This commit is contained in:
parent
04e5001e42
commit
838a998b6a
36 changed files with 0 additions and 17849 deletions
|
|
@ -1,38 +0,0 @@
|
|||
{
|
||||
"tagPrefix": "brainy-models-v",
|
||||
"commitMessageFormat": "chore(brainy-models): release {{currentTag}}",
|
||||
"types": [
|
||||
{
|
||||
"type": "feat",
|
||||
"section": "Features"
|
||||
},
|
||||
{
|
||||
"type": "fix",
|
||||
"section": "Bug Fixes"
|
||||
},
|
||||
{
|
||||
"type": "chore",
|
||||
"hidden": true
|
||||
},
|
||||
{
|
||||
"type": "docs",
|
||||
"section": "Documentation"
|
||||
},
|
||||
{
|
||||
"type": "style",
|
||||
"hidden": true
|
||||
},
|
||||
{
|
||||
"type": "refactor",
|
||||
"section": "Code Refactoring"
|
||||
},
|
||||
{
|
||||
"type": "perf",
|
||||
"section": "Performance Improvements"
|
||||
},
|
||||
{
|
||||
"type": "test",
|
||||
"hidden": true
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,113 +0,0 @@
|
|||
# Changelog
|
||||
|
||||
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
||||
|
||||
## [1.0.0](https://github.com/soulcraft-research/brainy/compare/brainy-models-v0.7.0...brainy-models-v1.0.0) (2025-08-05)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add verb and noun metadata handling in storage adapters ([9778f1b](https://github.com/soulcraft-research/brainy/commit/9778f1bbf46de06bb71d87333f3861210558024e))
|
||||
* **auto-configuration:** implement automatic configuration system for optimal settings ([aa64f49](https://github.com/soulcraft-research/brainy/commit/aa64f490cbad98bc6c95c1f212b8f049d41aa32f))
|
||||
* **distributed:** add distributed mode with multi-instance coordination ([8e4b0ef](https://github.com/soulcraft-research/brainy/commit/8e4b0ef7d8b9c3a4de2b776adc25da3c0a7fc971))
|
||||
* **docs:** add comprehensive user guides and installation instructions for Brainy ([d4dafbf](https://github.com/soulcraft-research/brainy/commit/d4dafbf598a49c7b1b005f7773f14b47fe65bffa))
|
||||
* **docs:** add S3 migration guide for optimized data transfer strategies ([7b4c779](https://github.com/soulcraft-research/brainy/commit/7b4c7794f3a587e2038452ab0d2c908272cf9556))
|
||||
* **docs:** update README and add large-scale optimizations guide for v0.36.0 ([ae01bea](https://github.com/soulcraft-research/brainy/commit/ae01bea1aa5d7f3a042f67fa6123c17e59310d68))
|
||||
* **hnsw:** implement comprehensive large-scale search optimizations ([c39eee6](https://github.com/soulcraft-research/brainy/commit/c39eee624d40c4ee4d5112c26283c17b133dc3e1))
|
||||
* **pagination:** implement cursor-based pagination and enhance search caching ([0f538f3](https://github.com/soulcraft-research/brainy/commit/0f538f39ba7897d2efb0fadd39fc7218b1bbe72d))
|
||||
* **partitioning:** simplify partition strategies and enable auto-tuning of semantic clusters ([1015c33](https://github.com/soulcraft-research/brainy/commit/1015c33004c248dd45fdb9c37edc6b15ad95a5f8))
|
||||
* refactor verb storage to use HNSWVerb for improved performance ([75ccf0f](https://github.com/soulcraft-research/brainy/commit/75ccf0f7472f94cc7a76a9b64b377fb7bdc02624))
|
||||
* **safety:** enhance claude-commit with mandatory review and safety features ([c20cc39](https://github.com/soulcraft-research/brainy/commit/c20cc392620ebded69732c62a07295dc212de001))
|
||||
* **tools:** add claude-commit AI-powered git commit tool ([d05e320](https://github.com/soulcraft-research/brainy/commit/d05e320a52486c5b5620869940df5b7330fa1067))
|
||||
* **tools:** propagate safety features to all projects ([8854b37](https://github.com/soulcraft-research/brainy/commit/8854b3735fac75e695e76ec62edde2160c065f55))
|
||||
* **tools:** update feature description for clarity ([5e15dab](https://github.com/soulcraft-research/brainy/commit/5e15dabb54e9e5616cf49b759222a24734c58fbc))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **build:** resolve TypeScript compilation errors in optimization modules ([0e2bce1](https://github.com/soulcraft-research/brainy/commit/0e2bce19251f7ea5008695f058272ca4271805f8))
|
||||
* **core:** resolve TypeScript compilation errors and test failures ([67db734](https://github.com/soulcraft-research/brainy/commit/67db73461124c33bb6fc11cb5f8daf3ddbfd9f09))
|
||||
* **security:** resolve critical vulnerability in form-data dependency ([8450af5](https://github.com/soulcraft-research/brainy/commit/8450af5d9289391dd516d23779fb675ae81bf5f6))
|
||||
* **storage:** resolve pagination warnings and improve S3 adapter performance ([d7a1c1b](https://github.com/soulcraft-research/brainy/commit/d7a1c1bd27257522e0986e2e882eb169bc7e1d14))
|
||||
* **types:** add explicit ArrayBuffer type assertions for compression ([7196fe2](https://github.com/soulcraft-research/brainy/commit/7196fe2d6b756a4e9401cf87585db688145e46fe))
|
||||
* **types:** resolve remaining ArrayBuffer type issues in compression methods ([eb8c95e](https://github.com/soulcraft-research/brainy/commit/eb8c95ef3720afa52acc45af2c3dcc896d67decc))
|
||||
|
||||
|
||||
### Documentation
|
||||
|
||||
* add comprehensive performance docs and rebrand to Zero-to-Smart™ ([80ca8e3](https://github.com/soulcraft-research/brainy/commit/80ca8e35d257723853d0f9d6c95f49937b71aceb))
|
||||
* add distributed deployment architecture and enhancement proposals ([4bb7a9f](https://github.com/soulcraft-research/brainy/commit/4bb7a9f431edaf85e782144057d77b4b20b16b44))
|
||||
* add guidelines for Conventional Commit format and structured commit messages ([f9a8595](https://github.com/soulcraft-research/brainy/commit/f9a859587802dfd294b2eaeefcf251f75a460db4))
|
||||
* add revised distributed implementation plan with practical phases ([e3978e5](https://github.com/soulcraft-research/brainy/commit/e3978e570dcee9b9c7757e75d3fe2c2f55cd99e4))
|
||||
* streamline README for better readability and user engagement ([2492fe4](https://github.com/soulcraft-research/brainy/commit/2492fe4f30099dc1b9f9cf0e435b83d5ffc34dce))
|
||||
|
||||
## [0.7.0](https://github.com/soulcraft-research/brainy/compare/brainy-models-v0.6.0...brainy-models-v0.7.0) (2025-08-02)
|
||||
|
||||
## [0.6.0](https://github.com/soulcraft-research/brainy/compare/brainy-models-v0.5.0...brainy-models-v0.6.0) (2025-08-01)
|
||||
|
||||
## 0.5.0 (2025-08-01)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **core, tests:** add standalone getStatistics function and improve storage configuration ([e5a9ede](https://github.com/soulcraft-research/brainy/commit/e5a9edea1b292e2b87d1ad043bb917f01f6366d9))
|
||||
* **core:** enhance addVerb functionality with auto-creation of missing nouns ([a0c4d48](https://github.com/soulcraft-research/brainy/commit/a0c4d48b4aa0b8e236c18f1b7afdc2e54801a142))
|
||||
* **demo, docs:** introduce threading test demos for browser and fallback, enhance threading documentation ([de627c5](https://github.com/soulcraft-research/brainy/commit/de627c5dfaff57a0abba6a4dc3b68bd52cb150e1))
|
||||
* **demo/CNAME:** add CNAME files for domain configuration ([a681ab7](https://github.com/soulcraft-research/brainy/commit/a681ab7cdd08d318111977cecd2ae7ef7262a3da))
|
||||
* **docs:** add WebSocket augmentation examples in README ([d7ff1b2](https://github.com/soulcraft-research/brainy/commit/d7ff1b2053779265521894d351a950f7903a5e64))
|
||||
* **docs:** update README to highlight new consolidated storage structure ([fbfcaeb](https://github.com/soulcraft-research/brainy/commit/fbfcaeb8d097c58a3127e68e9a7c06421cf5f468))
|
||||
* enhance `cli.ts` with updated typings and improved search interface ([4a23c97](https://github.com/soulcraft-research/brainy/commit/4a23c97d2a5f33a2d34f622014cf03e8350f8781))
|
||||
* **README:** add GPU acceleration and detailed performance optimizations ([da1fe27](https://github.com/soulcraft-research/brainy/commit/da1fe27e25851233a2a49eaaeec1dac9f9882d59))
|
||||
* reformat imports and exports for consistency and readability ([7de62f4](https://github.com/soulcraft-research/brainy/commit/7de62f4bbcdc91c88bb22e39e64c624219df8be4))
|
||||
* **scripts, project:** add comprehensive code style enforcement script and update style-related workflows ([5052bbc](https://github.com/soulcraft-research/brainy/commit/5052bbc0b713f2294b3e15de43e696b575320ae0))
|
||||
* **src/brainyData, src/utils:** enhance embedding efficiency with batch processing and initialize safeguards ([bba9a0c](https://github.com/soulcraft-research/brainy/commit/bba9a0c219c308f82821e46c44c49ef560ca2e39))
|
||||
* **src/hnsw:** add GPU acceleration for distance calculations and improve fallback handling ([a2ad5fa](https://github.com/soulcraft-research/brainy/commit/a2ad5fabdfe8e5de545c2a07cef2f08a667ebf5e))
|
||||
* **src/utils:** add GPU acceleration and improve threading for embeddings and distance calculations ([6a8d044](https://github.com/soulcraft-research/brainy/commit/6a8d044970b36976fc3ed5712d61bdbf774f7716))
|
||||
* **storage:** implement base, file system, and memory storage adapters ([8c5f17b](https://github.com/soulcraft-research/brainy/commit/8c5f17b1d96a84f996ad62ec4b1e1e56893dfcbc))
|
||||
* **tests:** add robust mock implementations and expand test coverage for S3 and OPFS storage ([f01a355](https://github.com/soulcraft-research/brainy/commit/f01a35598788b53a0294ecf9b91e47d166363846))
|
||||
* **tests:** replace old test scripts with updated test suite for storage and reporting ([68680db](https://github.com/soulcraft-research/brainy/commit/68680db2c660b34d5b4e4ee86d163f7723d328a4))
|
||||
* **types:** extend FileSystemHandle and optimize imports for consistency ([aff1483](https://github.com/soulcraft-research/brainy/commit/aff1483d4d84406039e301fa594889f99e19c9db))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* handle optional `loggingConfig` in `getDefaultEmbeddingFunction` initialization ([d42596a](https://github.com/soulcraft-research/brainy/commit/d42596ad4730943cea60081ca71fea6ad2babd37))
|
||||
* **package.json:** update TensorFlow dependencies for optimized backend usage ([e000661](https://github.com/soulcraft-research/brainy/commit/e00066119ebeb336e4564d38b4ef67d9bfbefeb6))
|
||||
* prevent duplication of `ROOT_DIR` in file system storage initialization ([d495b95](https://github.com/soulcraft-research/brainy/commit/d495b95af836c124685f5cff8a59ed74a82b3dca))
|
||||
* **README, src/utils:** bump version to 0.9.11 ([fe4de2f](https://github.com/soulcraft-research/brainy/commit/fe4de2f7a00b3f1ec1dd0edd2deb434d510096c2))
|
||||
* **README:** correct formatting in custom domain configuration steps ([bba846a](https://github.com/soulcraft-research/brainy/commit/bba846ae230e86235b1321851eb06f298ca09170))
|
||||
* **src/augmentationPipeline:** remove redundant semicolons and enforce consistent formatting ([47ba4f4](https://github.com/soulcraft-research/brainy/commit/47ba4f4093a8a3ecc8f7fa28638322d7ee4ae740))
|
||||
* **src/augmentationPipeline:** remove unnecessary whitespace for formatting consistency ([99a8cbf](https://github.com/soulcraft-research/brainy/commit/99a8cbfe2c036a29076f1ebaf4e92b320a0cf427))
|
||||
* **src/augmentations:** enforce consistent formatting and improve code readability ([cacd179](https://github.com/soulcraft-research/brainy/commit/cacd1790dc6b29096187dac53309b83d89b2242d))
|
||||
* **src/brainyData:** enforce consistent formatting and improve code readability ([e884c58](https://github.com/soulcraft-research/brainy/commit/e884c5831003c7e0cae8260cd0d3d8cf72e7ef07))
|
||||
* **src/brainyData:** enforce consistent formatting and improve fallback mechanisms ([cabe0a3](https://github.com/soulcraft-research/brainy/commit/cabe0a3a08e4bfd5fddfe19a7dfb0f382d347103))
|
||||
* **src/index:** enforce consistent formatting and adjust code structure ([95be233](https://github.com/soulcraft-research/brainy/commit/95be23362af2665f4d2fed2657599980381434f2))
|
||||
* **src/storage:** enforce consistent formatting and improve code readability ([cbf025d](https://github.com/soulcraft-research/brainy/commit/cbf025dffb0ab5e8af211d7daadc057b5771d567))
|
||||
* **src/utils:** enforce consistent formatting and enhance worker script initialization ([bd123c4](https://github.com/soulcraft-research/brainy/commit/bd123c4bb91ddb6d30ee5ef240e1872affb5f795))
|
||||
* **src/utils:** enhance type definition and improve load function detection ([ef83af4](https://github.com/soulcraft-research/brainy/commit/ef83af4b556966949e9df24cdf524a49b1ccab29))
|
||||
* **src/utils:** improve readability of `findUSELoadFunction` parameters ([f946328](https://github.com/soulcraft-research/brainy/commit/f9463288234ff9cf6220db8443423df37651dc72))
|
||||
* **src/utils:** remove unused `sentenceEncoderModule` for cleanup ([e81979d](https://github.com/soulcraft-research/brainy/commit/e81979dc849d7c1cdb70eb02697d1ad76db31196))
|
||||
|
||||
|
||||
### Documentation
|
||||
|
||||
* update Node.js version requirement to 23.0.0 in documentation ([920439f](https://github.com/soulcraft-research/brainy/commit/920439f611d1dba45e7bcb3915eea0df5507ac20))
|
||||
|
||||
## [0.4.0](https://github.com/soulcraft-research/brainy/compare/v0.1.0...v0.4.0) (2025-08-01)
|
||||
|
||||
|
||||
### Changed
|
||||
|
||||
* **release:** 0.2.0 [skip ci] ([c9ca141](https://github.com/soulcraft-research/brainy/commit/c9ca14146ba5376812823185e55fc8b38be3785c))
|
||||
* **release:** 0.3.0 [skip ci] ([437360c](https://github.com/soulcraft-research/brainy/commit/437360c2570632204cf951001aa7a0228479255d))
|
||||
|
||||
## [0.3.0](https://github.com/soulcraft-research/brainy/compare/v0.1.0...v0.3.0) (2025-08-01)
|
||||
|
||||
|
||||
### Changed
|
||||
|
||||
* **release:** 0.2.0 [skip ci] ([c9ca141](https://github.com/soulcraft-research/brainy/commit/c9ca14146ba5376812823185e55fc8b38be3785c))
|
||||
|
||||
## [0.2.0](https://github.com/soulcraft-research/brainy/compare/v0.1.0...v0.2.0) (2025-08-01)
|
||||
|
||||
## [0.1.0](https://github.com/soulcraft-research/brainy/compare/v0.33.0...v0.1.0) (2025-08-01)
|
||||
|
|
@ -1,49 +0,0 @@
|
|||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
We as members, contributors, and leaders pledge to make participation in our
|
||||
community a harassment-free experience for everyone, regardless of age, body
|
||||
size, visible or invisible disability, ethnicity, sex characteristics, gender
|
||||
identity and expression, level of experience, education, socio-economic status,
|
||||
nationality, personal appearance, race, religion, or sexual identity
|
||||
and orientation.
|
||||
|
||||
We pledge to act and interact in ways that contribute to an open, welcoming,
|
||||
diverse, inclusive, and healthy community.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to a positive environment for our
|
||||
community include:
|
||||
|
||||
* Demonstrating empathy and kindness toward other people
|
||||
* Being respectful of differing opinions, viewpoints, and experiences
|
||||
* Giving and gracefully accepting constructive feedback
|
||||
* Accepting responsibility and apologizing to those affected by our mistakes,
|
||||
and learning from the experience
|
||||
* Focusing on what is best not just for us as individuals, but for the
|
||||
overall community
|
||||
|
||||
Examples of unacceptable behavior include:
|
||||
|
||||
* The use of sexualized language or imagery, and sexual attention or
|
||||
advances of any kind
|
||||
* Trolling, insulting or derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or email
|
||||
address, without their explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported to the project maintainers responsible for enforcement at
|
||||
conduct@soulcraft.com.
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
||||
version 2.0, available at
|
||||
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
|
|
@ -1,103 +0,0 @@
|
|||
# Contributing to @soulcraft/brainy-models
|
||||
|
||||
Thank you for your interest in contributing to the Brainy Models package! This package provides pre-bundled TensorFlow models for the Brainy vector database.
|
||||
|
||||
## Overview
|
||||
|
||||
The `@soulcraft/brainy-models` package is part of the larger Brainy ecosystem. For general contribution guidelines, please refer to the main [Brainy Contributing Guide](https://github.com/soulcraft-research/brainy/blob/main/CONTRIBUTING.md).
|
||||
|
||||
## Package-Specific Guidelines
|
||||
|
||||
### Model Contributions
|
||||
|
||||
When contributing to the models package, please consider:
|
||||
|
||||
- **Model Quality**: Ensure models are properly tested and validated
|
||||
- **Model Size**: Be mindful of package size impact (current package is ~25MB)
|
||||
- **Compatibility**: Ensure models work with the target TensorFlow.js versions
|
||||
- **Documentation**: Update README.md with new model information
|
||||
|
||||
### Development Setup
|
||||
|
||||
1. Fork and clone the main repository
|
||||
2. Navigate to the models package: `cd brainy-models-package`
|
||||
3. Install dependencies: `npm install`
|
||||
4. Download models: `npm run download-models`
|
||||
5. Build the package: `npm run build`
|
||||
6. Run tests: `npm test`
|
||||
|
||||
### Testing Models
|
||||
|
||||
Before submitting changes:
|
||||
|
||||
```bash
|
||||
# Test model functionality
|
||||
npm test
|
||||
|
||||
# Test model compression
|
||||
npm run compress-models
|
||||
|
||||
# Verify package integrity
|
||||
npm run pack
|
||||
```
|
||||
|
||||
### Model Scripts
|
||||
|
||||
The package includes several utility scripts:
|
||||
|
||||
- `download-models` - Download the Universal Sentence Encoder model
|
||||
- `compress-models` - Create optimized model variants
|
||||
- `test` - Verify model functionality
|
||||
|
||||
### Commit Guidelines
|
||||
|
||||
Follow the same commit message conventions as the main Brainy project:
|
||||
|
||||
- Use conventional commit format
|
||||
- Keep first line under 50 characters
|
||||
- Use imperative mood ("Add model" not "Added model")
|
||||
- Reference issues where appropriate
|
||||
|
||||
### Pull Request Process
|
||||
|
||||
1. Ensure your changes don't break existing functionality
|
||||
2. Update documentation if you're adding new models or features
|
||||
3. Test model loading and embedding generation
|
||||
4. Verify package size impact is acceptable
|
||||
5. Submit PR to the main Brainy repository
|
||||
|
||||
### Model Optimization
|
||||
|
||||
When working with models:
|
||||
|
||||
- **Float16**: For balanced performance and size
|
||||
- **Int8**: For memory-constrained environments
|
||||
- **Original**: For maximum accuracy
|
||||
|
||||
### File Structure
|
||||
|
||||
```
|
||||
brainy-models-package/
|
||||
├── models/ # Model files
|
||||
├── src/ # TypeScript source
|
||||
├── dist/ # Compiled output
|
||||
├── scripts/ # Utility scripts
|
||||
└── test/ # Test files
|
||||
```
|
||||
|
||||
## Code of Conduct
|
||||
|
||||
This project follows the same [Code of Conduct](CODE_OF_CONDUCT.md) as the main Brainy project.
|
||||
|
||||
## Questions and Support
|
||||
|
||||
For questions specific to the models package:
|
||||
|
||||
- [GitHub Issues](https://github.com/soulcraft-research/brainy/issues) - Use the `brainy-models` label
|
||||
- [Main Documentation](https://github.com/soulcraft-research/brainy)
|
||||
|
||||
For general Brainy questions, refer to the main repository.
|
||||
|
||||
## License
|
||||
|
||||
By contributing to this project, you agree that your contributions will be licensed under the MIT License.
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2025 Soulcraft Research
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -1,375 +0,0 @@
|
|||
<div align="center">
|
||||
<img src="../brainy.png" alt="Brainy Logo" width="200"/>
|
||||
<br/><br/>
|
||||
|
||||
[](../LICENSE)
|
||||
[](https://nodejs.org/)
|
||||
[](https://www.typescriptlang.org/)
|
||||
[](../CONTRIBUTING.md)
|
||||
|
||||
**Pre-bundled TensorFlow models for maximum reliability with Brainy vector database**
|
||||
|
||||
</div>
|
||||
|
||||
## ✨ Overview
|
||||
|
||||
This package provides offline access to the Universal Sentence Encoder model, eliminating network dependencies and ensuring consistent performance. It's designed as an optional companion to the main `@soulcraft/brainy` package for applications requiring maximum reliability.
|
||||
|
||||
### 🚀 Key Features
|
||||
|
||||
- 🔒 **Maximum Reliability**: Fully offline model loading with zero network dependencies
|
||||
- 📦 **Pre-bundled Models**: Complete Universal Sentence Encoder model (~25MB) included
|
||||
- 🗜️ **Model Compression**: Multiple optimized variants (float16, int8) for different use cases
|
||||
- ⚡ **Performance Optimized**: Use case-specific optimizations for memory and speed
|
||||
- 🛠️ **Easy Integration**: Drop-in replacement for online model loading
|
||||
- 📊 **Comprehensive Metrics**: Detailed model information and performance statistics
|
||||
|
||||
## 🔧 Installation
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/brainy-models
|
||||
```
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Node.js >= 18.0.0
|
||||
- `@soulcraft/brainy` >= 0.33.0
|
||||
|
||||
## 🏁 Quick Start
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```typescript
|
||||
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
|
||||
|
||||
// Create encoder instance
|
||||
const encoder = new BundledUniversalSentenceEncoder({
|
||||
verbose: true,
|
||||
preferCompressed: false
|
||||
})
|
||||
|
||||
// Load the bundled model
|
||||
await encoder.load()
|
||||
|
||||
// Generate embeddings
|
||||
const texts = ['Hello world', 'How are you?', 'Machine learning is amazing']
|
||||
const embeddings = await encoder.embedToArrays(texts)
|
||||
|
||||
console.log(`Generated ${embeddings.length} embeddings of ${embeddings[0].length} dimensions`)
|
||||
|
||||
// Clean up
|
||||
encoder.dispose()
|
||||
```
|
||||
|
||||
### Integration with Brainy
|
||||
|
||||
```typescript
|
||||
import Brainy from '@soulcraft/brainy'
|
||||
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
|
||||
|
||||
// Create bundled encoder
|
||||
const bundledEncoder = new BundledUniversalSentenceEncoder({ verbose: true })
|
||||
await bundledEncoder.load()
|
||||
|
||||
// Use with Brainy (custom integration)
|
||||
const brainy = new Brainy({
|
||||
// Configure Brainy to use the bundled encoder
|
||||
customEmbedding: async (texts) => {
|
||||
return await bundledEncoder.embedToArrays(texts)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Using Compressed Models
|
||||
|
||||
```typescript
|
||||
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
|
||||
|
||||
// Use compressed model for memory-constrained environments
|
||||
const encoder = new BundledUniversalSentenceEncoder({
|
||||
preferCompressed: true,
|
||||
verbose: true
|
||||
})
|
||||
|
||||
await encoder.load()
|
||||
|
||||
// The encoder will automatically use the most appropriate compressed variant
|
||||
const embeddings = await encoder.embedToArrays(['Sample text'])
|
||||
```
|
||||
|
||||
## 📚 API Reference
|
||||
|
||||
### BundledUniversalSentenceEncoder
|
||||
|
||||
Main class for loading and using bundled models.
|
||||
|
||||
#### Constructor
|
||||
|
||||
```typescript
|
||||
new BundledUniversalSentenceEncoder(options)
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `verbose?: boolean` - Enable detailed logging (default: false)
|
||||
- `preferCompressed?: boolean` - Prefer compressed model variants (default: false)
|
||||
|
||||
#### Methods
|
||||
|
||||
##### `load(): Promise<void>`
|
||||
|
||||
Load the bundled model from local files.
|
||||
|
||||
```typescript
|
||||
await encoder.load()
|
||||
```
|
||||
|
||||
##### `embed(texts: string[]): Promise<tf.Tensor2D>`
|
||||
|
||||
Generate embeddings as TensorFlow tensors.
|
||||
|
||||
```typescript
|
||||
const embeddings = await encoder.embed(['Hello world'])
|
||||
// Remember to dispose of tensors when done
|
||||
embeddings.dispose()
|
||||
```
|
||||
|
||||
##### `embedToArrays(texts: string[]): Promise<number[][]>`
|
||||
|
||||
Generate embeddings as JavaScript arrays (automatically disposes tensors).
|
||||
|
||||
```typescript
|
||||
const embeddings = await encoder.embedToArrays(['Hello world'])
|
||||
console.log(embeddings[0].length) // 512
|
||||
```
|
||||
|
||||
##### `getMetadata(): ModelMetadata | null`
|
||||
|
||||
Get model metadata information.
|
||||
|
||||
```typescript
|
||||
const metadata = encoder.getMetadata()
|
||||
console.log(metadata?.dimensions) // 512
|
||||
```
|
||||
|
||||
##### `isLoaded(): boolean`
|
||||
|
||||
Check if the model is loaded.
|
||||
|
||||
```typescript
|
||||
if (encoder.isLoaded()) {
|
||||
// Model is ready to use
|
||||
}
|
||||
```
|
||||
|
||||
##### `getModelInfo(): { inputShape: number[], outputShape: number[] } | null`
|
||||
|
||||
Get model input/output shape information.
|
||||
|
||||
```typescript
|
||||
const info = encoder.getModelInfo()
|
||||
console.log(info?.outputShape) // [-1, 512]
|
||||
```
|
||||
|
||||
##### `dispose(): void`
|
||||
|
||||
Clean up model resources.
|
||||
|
||||
```typescript
|
||||
encoder.dispose()
|
||||
```
|
||||
|
||||
### ModelCompressor
|
||||
|
||||
Utility class for model compression and optimization.
|
||||
|
||||
#### Static Methods
|
||||
|
||||
##### `quantizeModel(modelPath: string, outputPath: string, options?): Promise<void>`
|
||||
|
||||
Compress a model using quantization.
|
||||
|
||||
```typescript
|
||||
import { ModelCompressor } from '@soulcraft/brainy-models'
|
||||
|
||||
await ModelCompressor.quantizeModel(
|
||||
'/path/to/model.json',
|
||||
'/path/to/compressed/model.json',
|
||||
{ dtype: 'int8' }
|
||||
)
|
||||
```
|
||||
|
||||
##### `getModelSize(modelPath: string): Promise<ModelSizeInfo>`
|
||||
|
||||
Get detailed model size information.
|
||||
|
||||
```typescript
|
||||
const sizeInfo = await ModelCompressor.getModelSize('/path/to/model.json')
|
||||
console.log(`Total size: ${sizeInfo.totalSize} bytes`)
|
||||
```
|
||||
|
||||
### Utility Functions
|
||||
|
||||
#### `utils.checkModelsAvailable(): boolean`
|
||||
|
||||
Check if bundled models are available.
|
||||
|
||||
```typescript
|
||||
import { utils } from '@soulcraft/brainy-models'
|
||||
|
||||
if (utils.checkModelsAvailable()) {
|
||||
console.log('Models are ready to use')
|
||||
}
|
||||
```
|
||||
|
||||
#### `utils.listAvailableModels(): string[]`
|
||||
|
||||
List available bundled models.
|
||||
|
||||
```typescript
|
||||
const models = utils.listAvailableModels()
|
||||
console.log('Available models:', models)
|
||||
```
|
||||
|
||||
## 🎯 Model Variants
|
||||
|
||||
The package includes multiple model variants optimized for different use cases:
|
||||
|
||||
### Original (Float32)
|
||||
- **Size**: ~25MB
|
||||
- **Use case**: Maximum accuracy
|
||||
- **Memory**: High
|
||||
- **Speed**: Fast
|
||||
|
||||
### Float16 Compressed
|
||||
- **Size**: ~12-15MB
|
||||
- **Use case**: Balanced performance
|
||||
- **Memory**: Medium
|
||||
- **Speed**: Fast
|
||||
|
||||
### Int8 Quantized
|
||||
- **Size**: ~6-8MB
|
||||
- **Use case**: Memory-constrained environments
|
||||
- **Memory**: Low
|
||||
- **Speed**: Medium
|
||||
|
||||
## ⚙️ Scripts
|
||||
|
||||
The package includes several utility scripts:
|
||||
|
||||
### Download Models
|
||||
|
||||
Download the complete Universal Sentence Encoder model:
|
||||
|
||||
```bash
|
||||
npm run download-models
|
||||
```
|
||||
|
||||
### Compress Models
|
||||
|
||||
Create optimized model variants:
|
||||
|
||||
```bash
|
||||
npm run compress-models
|
||||
```
|
||||
|
||||
### Test Models
|
||||
|
||||
Verify model functionality:
|
||||
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
|
||||
## 🔨 Development
|
||||
|
||||
### Building the Package
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
|
||||
### Creating a Release
|
||||
|
||||
```bash
|
||||
npm run pack
|
||||
```
|
||||
|
||||
## ⚖️ Comparison with Online Loading
|
||||
|
||||
| Feature | Online Loading | Bundled Models |
|
||||
|---------|----------------|----------------|
|
||||
| **Reliability** | Network dependent | 100% offline |
|
||||
| **First load time** | 30-60 seconds | < 1 second |
|
||||
| **Subsequent loads** | Cached (~1 second) | < 1 second |
|
||||
| **Package size** | ~3KB | ~25MB |
|
||||
| **Network required** | Yes (first time) | No |
|
||||
| **Offline support** | Limited | Complete |
|
||||
|
||||
## 💡 Use Cases
|
||||
|
||||
### When to Use Bundled Models
|
||||
|
||||
- ✅ Production applications requiring maximum reliability
|
||||
- ✅ Offline or air-gapped environments
|
||||
- ✅ Applications with strict SLA requirements
|
||||
- ✅ Edge computing and IoT devices
|
||||
- ✅ Development environments with unreliable internet
|
||||
|
||||
### When to Use Online Loading
|
||||
|
||||
- ✅ Development and prototyping
|
||||
- ✅ Applications where package size matters
|
||||
- ✅ Environments with reliable internet connectivity
|
||||
- ✅ Applications that rarely use embeddings
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### Model Not Found Error
|
||||
|
||||
```
|
||||
Error: Bundled model not found. Please run "npm run download-models"
|
||||
```
|
||||
|
||||
**Solution**: Run the download script to fetch the model files:
|
||||
```bash
|
||||
cd node_modules/@soulcraft/brainy-models
|
||||
npm run download-models
|
||||
```
|
||||
|
||||
### Memory Issues
|
||||
|
||||
If you encounter memory issues, try using compressed models:
|
||||
|
||||
```typescript
|
||||
const encoder = new BundledUniversalSentenceEncoder({
|
||||
preferCompressed: true
|
||||
})
|
||||
```
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
For optimal performance:
|
||||
|
||||
1. **Memory-constrained**: Use int8 quantized models
|
||||
2. **Speed-critical**: Use original float32 models
|
||||
3. **Balanced**: Use float16 compressed models
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! Please see the main [Brainy repository](https://github.com/soulcraft-research/brainy) for contribution guidelines.
|
||||
|
||||
## 💬 Support
|
||||
|
||||
For issues and questions:
|
||||
- [GitHub Issues](https://github.com/soulcraft-research/brainy/issues)
|
||||
- [Documentation](https://github.com/soulcraft-research/brainy)
|
||||
103
brainy-models-package/dist/index.d.ts
vendored
103
brainy-models-package/dist/index.d.ts
vendored
|
|
@ -1,103 +0,0 @@
|
|||
/**
|
||||
* @soulcraft/brainy-models
|
||||
*
|
||||
* Pre-bundled TensorFlow models for maximum reliability with Brainy vector database.
|
||||
* This package provides offline access to the Universal Sentence Encoder model,
|
||||
* eliminating network dependencies and ensuring consistent performance.
|
||||
*/
|
||||
import * as tf from '@tensorflow/tfjs';
|
||||
export interface ModelMetadata {
|
||||
name: string;
|
||||
version: string;
|
||||
description: string;
|
||||
dimensions: number;
|
||||
downloadDate: string;
|
||||
source: string;
|
||||
approach: string;
|
||||
modelUrl: string;
|
||||
bundledLocally: boolean;
|
||||
reliability: string;
|
||||
}
|
||||
export interface BundledModelOptions {
|
||||
verbose?: boolean;
|
||||
preferCompressed?: boolean;
|
||||
}
|
||||
/**
|
||||
* Bundled Universal Sentence Encoder for offline use
|
||||
*/
|
||||
export declare class BundledUniversalSentenceEncoder {
|
||||
private model;
|
||||
private metadata;
|
||||
private options;
|
||||
constructor(options?: BundledModelOptions);
|
||||
/**
|
||||
* Load the bundled model from local files
|
||||
*/
|
||||
load(): Promise<void>;
|
||||
/**
|
||||
* Generate embeddings for the given texts
|
||||
*/
|
||||
embed(texts: string[]): Promise<tf.Tensor2D>;
|
||||
/**
|
||||
* Generate embeddings and return as JavaScript arrays
|
||||
*/
|
||||
embedToArrays(texts: string[]): Promise<number[][]>;
|
||||
/**
|
||||
* Get model metadata
|
||||
*/
|
||||
getMetadata(): ModelMetadata | null;
|
||||
/**
|
||||
* Check if the model is loaded
|
||||
*/
|
||||
isLoaded(): boolean;
|
||||
/**
|
||||
* Get model information
|
||||
*/
|
||||
getModelInfo(): {
|
||||
inputShape: number[];
|
||||
outputShape: number[];
|
||||
} | null;
|
||||
/**
|
||||
* Dispose of the model and free memory
|
||||
*/
|
||||
dispose(): void;
|
||||
}
|
||||
/**
|
||||
* Model compression utilities
|
||||
*/
|
||||
export declare class ModelCompressor {
|
||||
/**
|
||||
* Compress model weights using quantization
|
||||
* Note: TensorFlow.js doesn't currently support model quantization
|
||||
*/
|
||||
static quantizeModel(modelPath: string, outputPath: string, options?: {
|
||||
dtype?: 'int8' | 'int16';
|
||||
}): Promise<void>;
|
||||
/**
|
||||
* Get model size information by reading files from disk
|
||||
*/
|
||||
static getModelSize(modelPath: string): Promise<{
|
||||
totalSize: number;
|
||||
weightsSize: number;
|
||||
modelJsonSize: number;
|
||||
}>;
|
||||
}
|
||||
/**
|
||||
* Utility functions
|
||||
*/
|
||||
export declare const utils: {
|
||||
/**
|
||||
* Check if bundled models are available
|
||||
*/
|
||||
checkModelsAvailable(): boolean;
|
||||
/**
|
||||
* Get bundled models directory
|
||||
*/
|
||||
getModelsDirectory(): string;
|
||||
/**
|
||||
* List available bundled models
|
||||
*/
|
||||
listAvailableModels(): string[];
|
||||
};
|
||||
export default BundledUniversalSentenceEncoder;
|
||||
//# sourceMappingURL=index.d.ts.map
|
||||
1
brainy-models-package/dist/index.d.ts.map
vendored
1
brainy-models-package/dist/index.d.ts.map
vendored
|
|
@ -1 +0,0 @@
|
|||
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;GAMG;AAEH,OAAO,KAAK,EAAE,MAAM,kBAAkB,CAAA;AAwBtC,MAAM,WAAW,aAAa;IAC5B,IAAI,EAAE,MAAM,CAAA;IACZ,OAAO,EAAE,MAAM,CAAA;IACf,WAAW,EAAE,MAAM,CAAA;IACnB,UAAU,EAAE,MAAM,CAAA;IAClB,YAAY,EAAE,MAAM,CAAA;IACpB,MAAM,EAAE,MAAM,CAAA;IACd,QAAQ,EAAE,MAAM,CAAA;IAChB,QAAQ,EAAE,MAAM,CAAA;IAChB,cAAc,EAAE,OAAO,CAAA;IACvB,WAAW,EAAE,MAAM,CAAA;CACpB;AAED,MAAM,WAAW,mBAAmB;IAClC,OAAO,CAAC,EAAE,OAAO,CAAA;IACjB,gBAAgB,CAAC,EAAE,OAAO,CAAA;CAC3B;AAED;;GAEG;AACH,qBAAa,+BAA+B;IAC1C,OAAO,CAAC,KAAK,CAA6B;IAC1C,OAAO,CAAC,QAAQ,CAA6B;IAC7C,OAAO,CAAC,OAAO,CAAqB;gBAExB,OAAO,GAAE,mBAAwB;IAQ7C;;OAEG;IACG,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAwC3B;;OAEG;IACG,KAAK,CAAC,KAAK,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,EAAE,CAAC,QAAQ,CAAC;IAqBlD;;OAEG;IACG,aAAa,CAAC,KAAK,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC;IAOzD;;OAEG;IACH,WAAW,IAAI,aAAa,GAAG,IAAI;IAInC;;OAEG;IACH,QAAQ,IAAI,OAAO;IAInB;;OAEG;IACH,YAAY,IAAI;QAAE,UAAU,EAAE,MAAM,EAAE,CAAC;QAAC,WAAW,EAAE,MAAM,EAAE,CAAA;KAAE,GAAG,IAAI;IAWtE;;OAEG;IACH,OAAO,IAAI,IAAI;CAMhB;AAED;;GAEG;AACH,qBAAa,eAAe;IAC1B;;;OAGG;WACU,aAAa,CACxB,SAAS,EAAE,MAAM,EACjB,UAAU,EAAE,MAAM,EAClB,OAAO,GAAE;QAAE,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAA;KAAO,GACzC,OAAO,CAAC,IAAI,CAAC;IAwBhB;;OAEG;WACU,YAAY,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC;QACpD,SAAS,EAAE,MAAM,CAAA;QACjB,WAAW,EAAE,MAAM,CAAA;QACnB,aAAa,EAAE,MAAM,CAAA;KACtB,CAAC;CAuCH;AAED;;GAEG;AACH,eAAO,MAAM,KAAK;IAChB;;OAEG;4BACqB,OAAO;IAK/B;;OAEG;0BACmB,MAAM;IAI5B;;OAEG;2BACoB,MAAM,EAAE;CAUhC,CAAA;AAGD,eAAe,+BAA+B,CAAA"}
|
||||
237
brainy-models-package/dist/index.js
vendored
237
brainy-models-package/dist/index.js
vendored
|
|
@ -1,237 +0,0 @@
|
|||
/**
|
||||
* @soulcraft/brainy-models
|
||||
*
|
||||
* Pre-bundled TensorFlow models for maximum reliability with Brainy vector database.
|
||||
* This package provides offline access to the Universal Sentence Encoder model,
|
||||
* eliminating network dependencies and ensuring consistent performance.
|
||||
*/
|
||||
import * as tf from '@tensorflow/tfjs';
|
||||
import { readFileSync, existsSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
/**
|
||||
* Helper function to safely extract error message from unknown error type
|
||||
*/
|
||||
function getErrorMessage(error) {
|
||||
if (error instanceof Error) {
|
||||
return error.message;
|
||||
}
|
||||
if (typeof error === 'string') {
|
||||
return error;
|
||||
}
|
||||
return String(error);
|
||||
}
|
||||
// Get the package directory
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
const PACKAGE_ROOT = join(__dirname, '..');
|
||||
const MODELS_DIR = join(PACKAGE_ROOT, 'models');
|
||||
/**
|
||||
* Bundled Universal Sentence Encoder for offline use
|
||||
*/
|
||||
export class BundledUniversalSentenceEncoder {
|
||||
model = null;
|
||||
metadata = null;
|
||||
options;
|
||||
constructor(options = {}) {
|
||||
this.options = {
|
||||
verbose: false,
|
||||
preferCompressed: false,
|
||||
...options
|
||||
};
|
||||
}
|
||||
/**
|
||||
* Load the bundled model from local files
|
||||
*/
|
||||
async load() {
|
||||
try {
|
||||
const modelDir = join(MODELS_DIR, 'universal-sentence-encoder');
|
||||
const modelPath = join(modelDir, 'model.json');
|
||||
const metadataPath = join(modelDir, 'metadata.json');
|
||||
if (!existsSync(modelPath)) {
|
||||
throw new Error(`Bundled model not found at ${modelPath}. ` +
|
||||
'Please run "npm run download-models" to download the model files.');
|
||||
}
|
||||
if (this.options.verbose) {
|
||||
console.log('🔄 Loading bundled Universal Sentence Encoder model...');
|
||||
}
|
||||
// Load metadata
|
||||
if (existsSync(metadataPath)) {
|
||||
const metadataContent = readFileSync(metadataPath, 'utf8');
|
||||
this.metadata = JSON.parse(metadataContent);
|
||||
if (this.options.verbose) {
|
||||
console.log(`📋 Model metadata:`, this.metadata);
|
||||
}
|
||||
}
|
||||
// Load the model
|
||||
this.model = await tf.loadGraphModel(`file://${modelPath}`);
|
||||
if (this.options.verbose) {
|
||||
console.log('✅ Bundled model loaded successfully');
|
||||
console.log(`🔒 Reliability: Maximum (fully offline)`);
|
||||
}
|
||||
}
|
||||
catch (error) {
|
||||
throw new Error(`Failed to load bundled model: ${getErrorMessage(error)}`);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Generate embeddings for the given texts
|
||||
*/
|
||||
async embed(texts) {
|
||||
if (!this.model) {
|
||||
throw new Error('Model not loaded. Call load() first.');
|
||||
}
|
||||
try {
|
||||
// Convert texts to tensor
|
||||
const inputTensor = tf.tensor1d(texts, 'string');
|
||||
// Run inference
|
||||
const embeddings = this.model.predict(inputTensor);
|
||||
// Clean up input tensor
|
||||
inputTensor.dispose();
|
||||
return embeddings;
|
||||
}
|
||||
catch (error) {
|
||||
throw new Error(`Failed to generate embeddings: ${getErrorMessage(error)}`);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Generate embeddings and return as JavaScript arrays
|
||||
*/
|
||||
async embedToArrays(texts) {
|
||||
const embeddings = await this.embed(texts);
|
||||
const arrays = await embeddings.array();
|
||||
embeddings.dispose();
|
||||
return arrays;
|
||||
}
|
||||
/**
|
||||
* Get model metadata
|
||||
*/
|
||||
getMetadata() {
|
||||
return this.metadata;
|
||||
}
|
||||
/**
|
||||
* Check if the model is loaded
|
||||
*/
|
||||
isLoaded() {
|
||||
return this.model !== null;
|
||||
}
|
||||
/**
|
||||
* Get model information
|
||||
*/
|
||||
getModelInfo() {
|
||||
if (!this.model) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
inputShape: this.model.inputs[0].shape || [],
|
||||
outputShape: this.model.outputs[0].shape || []
|
||||
};
|
||||
}
|
||||
/**
|
||||
* Dispose of the model and free memory
|
||||
*/
|
||||
dispose() {
|
||||
if (this.model) {
|
||||
this.model.dispose();
|
||||
this.model = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Model compression utilities
|
||||
*/
|
||||
export class ModelCompressor {
|
||||
/**
|
||||
* Compress model weights using quantization
|
||||
* Note: TensorFlow.js doesn't currently support model quantization
|
||||
*/
|
||||
static async quantizeModel(modelPath, outputPath, options = {}) {
|
||||
const { dtype = 'int8' } = options;
|
||||
try {
|
||||
console.log(`🔄 Loading model for quantization: ${modelPath}`);
|
||||
const model = await tf.loadGraphModel(`file://${modelPath}`);
|
||||
console.log(`🗜️ Quantizing model to ${dtype}...`);
|
||||
// TensorFlow.js doesn't have built-in quantization or model serialization APIs yet
|
||||
// This is a placeholder implementation that acknowledges the limitation
|
||||
console.warn('⚠️ Model quantization is not yet supported in TensorFlow.js');
|
||||
console.log(`📋 Model loaded successfully from: ${modelPath}`);
|
||||
console.log(`📋 Target output path: ${outputPath}`);
|
||||
console.log(`📋 Target dtype: ${dtype}`);
|
||||
model.dispose();
|
||||
throw new Error('Model quantization is not yet supported in TensorFlow.js. This feature requires server-side processing with TensorFlow Python.');
|
||||
}
|
||||
catch (error) {
|
||||
throw new Error(`Failed to compress model: ${getErrorMessage(error)}`);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Get model size information by reading files from disk
|
||||
*/
|
||||
static async getModelSize(modelPath) {
|
||||
try {
|
||||
// Load model to verify it's valid
|
||||
const model = await tf.loadGraphModel(`file://${modelPath}`);
|
||||
model.dispose();
|
||||
// Get model.json size
|
||||
const modelJsonSize = existsSync(modelPath) ? readFileSync(modelPath).length : 0;
|
||||
// Calculate weights size by reading weight files
|
||||
let weightsSize = 0;
|
||||
const modelDir = dirname(modelPath);
|
||||
// Read model.json to get weight file names
|
||||
if (existsSync(modelPath)) {
|
||||
const modelJson = JSON.parse(readFileSync(modelPath, 'utf8'));
|
||||
if (modelJson.weightsManifest) {
|
||||
for (const manifest of modelJson.weightsManifest) {
|
||||
for (const path of manifest.paths) {
|
||||
const weightFilePath = join(modelDir, path);
|
||||
if (existsSync(weightFilePath)) {
|
||||
weightsSize += readFileSync(weightFilePath).length;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
const totalSize = weightsSize + modelJsonSize;
|
||||
return {
|
||||
totalSize,
|
||||
weightsSize,
|
||||
modelJsonSize
|
||||
};
|
||||
}
|
||||
catch (error) {
|
||||
throw new Error(`Failed to get model size: ${getErrorMessage(error)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Utility functions
|
||||
*/
|
||||
export const utils = {
|
||||
/**
|
||||
* Check if bundled models are available
|
||||
*/
|
||||
checkModelsAvailable() {
|
||||
const modelPath = join(MODELS_DIR, 'universal-sentence-encoder', 'model.json');
|
||||
return existsSync(modelPath);
|
||||
},
|
||||
/**
|
||||
* Get bundled models directory
|
||||
*/
|
||||
getModelsDirectory() {
|
||||
return MODELS_DIR;
|
||||
},
|
||||
/**
|
||||
* List available bundled models
|
||||
*/
|
||||
listAvailableModels() {
|
||||
const models = [];
|
||||
const useModelPath = join(MODELS_DIR, 'universal-sentence-encoder', 'model.json');
|
||||
if (existsSync(useModelPath)) {
|
||||
models.push('universal-sentence-encoder');
|
||||
}
|
||||
return models;
|
||||
}
|
||||
};
|
||||
// Default export for convenience
|
||||
export default BundledUniversalSentenceEncoder;
|
||||
//# sourceMappingURL=index.js.map
|
||||
1
brainy-models-package/dist/index.js.map
vendored
1
brainy-models-package/dist/index.js.map
vendored
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -1,12 +0,0 @@
|
|||
{
|
||||
"name": "universal-sentence-encoder",
|
||||
"version": "1.0.0",
|
||||
"description": "Complete Universal Sentence Encoder model bundled for offline use",
|
||||
"dimensions": 512,
|
||||
"downloadDate": "2025-08-02T21:42:25.604Z",
|
||||
"source": "tensorflow-models/universal-sentence-encoder",
|
||||
"approach": "full-bundle",
|
||||
"modelUrl": "https://storage.googleapis.com/tfjs-models/savedmodel/universal_sentence_encoder",
|
||||
"bundledLocally": true,
|
||||
"reliability": "maximum"
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
5337
brainy-models-package/package-lock.json
generated
5337
brainy-models-package/package-lock.json
generated
File diff suppressed because it is too large
Load diff
|
|
@ -1,77 +0,0 @@
|
|||
{
|
||||
"name": "@soulcraft/brainy-models",
|
||||
"version": "1.0.0",
|
||||
"description": "Pre-bundled TensorFlow models for maximum reliability with Brainy vector database",
|
||||
"main": "dist/index.js",
|
||||
"module": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"type": "module",
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
},
|
||||
"scripts": {
|
||||
"prebuild": "npm run download-models",
|
||||
"build": "tsc",
|
||||
"test": "node test/test-models.js",
|
||||
"prepare": "npm run build",
|
||||
"download-models": "node scripts/download-full-models.js",
|
||||
"compress-models": "node scripts/compress-models.js",
|
||||
"_pack": "npm pack",
|
||||
"_release": "standard-version",
|
||||
"_release:patch": "standard-version --release-as patch",
|
||||
"_release:minor": "standard-version --release-as minor",
|
||||
"_release:major": "standard-version --release-as major",
|
||||
"_release:dry-run": "standard-version --dry-run",
|
||||
"_github-release": "node scripts/create-github-release.js",
|
||||
"_workflow": "node scripts/release-workflow.js",
|
||||
"_workflow:patch": "node scripts/release-workflow.js patch",
|
||||
"_workflow:minor": "node scripts/release-workflow.js minor",
|
||||
"_workflow:major": "node scripts/release-workflow.js major",
|
||||
"_workflow:dry-run": "npm run build && npm test && npm run _release:dry-run",
|
||||
"_deploy": "npm run build && npm publish"
|
||||
},
|
||||
"keywords": [
|
||||
"tensorflow",
|
||||
"models",
|
||||
"universal-sentence-encoder",
|
||||
"embeddings",
|
||||
"brainy",
|
||||
"vector-database",
|
||||
"offline",
|
||||
"bundled"
|
||||
],
|
||||
"author": "David Snelling (david@soulcraft.com)",
|
||||
"license": "MIT",
|
||||
"private": false,
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"homepage": "https://github.com/soulcraft-research/brainy",
|
||||
"bugs": {
|
||||
"url": "https://github.com/soulcraft-research/brainy/issues"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/soulcraft-research/brainy.git",
|
||||
"directory": "brainy-models-package"
|
||||
},
|
||||
"files": [
|
||||
"dist/",
|
||||
"models/",
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
],
|
||||
"dependencies": {
|
||||
"@tensorflow-models/universal-sentence-encoder": "^1.3.3",
|
||||
"@tensorflow/tfjs": "^4.23.0-rc.0",
|
||||
"@tensorflow/tfjs-node": "^4.23.0-rc.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.11.30",
|
||||
"standard-version": "^9.5.0",
|
||||
"typescript": "^5.4.5"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@soulcraft/brainy": ">=0.33.0"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Reproduction script for the TensorFlow.js isNullOrUndefined error
|
||||
*/
|
||||
|
||||
import * as tf from '@tensorflow/tfjs-node'
|
||||
import * as use from '@tensorflow-models/universal-sentence-encoder'
|
||||
|
||||
console.log('🔍 Loading Universal Sentence Encoder model...')
|
||||
|
||||
try {
|
||||
const model = await use.load()
|
||||
console.log('✅ Model loaded successfully')
|
||||
|
||||
console.log('🧪 Testing model functionality...')
|
||||
const testEmbedding = await model.embed(['Hello world'])
|
||||
const testArray = await testEmbedding.array()
|
||||
console.log(
|
||||
`✅ Model test passed - embedding dimensions: ${testArray[0].length}`
|
||||
)
|
||||
testEmbedding.dispose()
|
||||
model.dispose()
|
||||
} catch (error) {
|
||||
console.error('❌ Error:', error)
|
||||
console.error('Stack trace:', error.stack)
|
||||
process.exit(1)
|
||||
}
|
||||
|
|
@ -1,278 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/* eslint-env node */
|
||||
/* eslint-disable no-console */
|
||||
|
||||
/**
|
||||
* Model Compression Script for @soulcraft/brainy-models
|
||||
*
|
||||
* This script implements model compression and optimization techniques
|
||||
* to reduce model size while maintaining accuracy.
|
||||
*/
|
||||
|
||||
import fs from 'fs'
|
||||
import path from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
import * as tf from '@tensorflow/tfjs-node'
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = path.dirname(__filename)
|
||||
|
||||
const MODELS_DIR = path.join(__dirname, '..', 'models')
|
||||
const USE_MODEL_DIR = path.join(MODELS_DIR, 'universal-sentence-encoder')
|
||||
const COMPRESSED_DIR = path.join(USE_MODEL_DIR, 'compressed')
|
||||
|
||||
// Ensure compressed directory exists
|
||||
if (!fs.existsSync(COMPRESSED_DIR)) {
|
||||
fs.mkdirSync(COMPRESSED_DIR, { recursive: true })
|
||||
}
|
||||
|
||||
console.log('🗜️ Starting model compression for @soulcraft/brainy-models...')
|
||||
console.log('This will create optimized versions of the bundled models.\n')
|
||||
|
||||
/**
|
||||
* Get file size in MB
|
||||
*/
|
||||
function getFileSizeMB(filePath) {
|
||||
const stats = fs.statSync(filePath)
|
||||
return (stats.size / 1024 / 1024).toFixed(2)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get directory size in MB
|
||||
*/
|
||||
function getDirectorySizeMB(dirPath) {
|
||||
let totalSize = 0
|
||||
const files = fs.readdirSync(dirPath)
|
||||
|
||||
for (const file of files) {
|
||||
const filePath = path.join(dirPath, file)
|
||||
const stats = fs.statSync(filePath)
|
||||
if (stats.isFile()) {
|
||||
totalSize += stats.size
|
||||
}
|
||||
}
|
||||
|
||||
return (totalSize / 1024 / 1024).toFixed(2)
|
||||
}
|
||||
|
||||
/**
|
||||
* Compress model weights by reducing precision
|
||||
*/
|
||||
async function compressModelWeights(modelPath, outputPath, precision = 'float16') {
|
||||
try {
|
||||
console.log(`🔄 Loading model from: ${modelPath}`)
|
||||
const model = await tf.loadGraphModel(`file://${modelPath}`)
|
||||
|
||||
console.log(`🗜️ Compressing weights to ${precision} precision...`)
|
||||
|
||||
// Get model artifacts
|
||||
const artifacts = await model.serialize()
|
||||
|
||||
// Compress weight data
|
||||
if (artifacts.weightData) {
|
||||
const originalWeights = new Float32Array(artifacts.weightData)
|
||||
let compressedWeights
|
||||
|
||||
if (precision === 'float16') {
|
||||
// Simulate float16 by reducing precision
|
||||
compressedWeights = new Float32Array(originalWeights.length)
|
||||
for (let i = 0; i < originalWeights.length; i++) {
|
||||
// Round to reduce precision (simulating float16)
|
||||
compressedWeights[i] = Math.round(originalWeights[i] * 1000) / 1000
|
||||
}
|
||||
} else if (precision === 'int8') {
|
||||
// Quantize to int8 range
|
||||
const min = Math.min(...originalWeights)
|
||||
const max = Math.max(...originalWeights)
|
||||
const scale = (max - min) / 255
|
||||
|
||||
compressedWeights = new Float32Array(originalWeights.length)
|
||||
for (let i = 0; i < originalWeights.length; i++) {
|
||||
const quantized = Math.round((originalWeights[i] - min) / scale)
|
||||
compressedWeights[i] = (quantized * scale) + min
|
||||
}
|
||||
}
|
||||
|
||||
artifacts.weightData = compressedWeights.buffer
|
||||
}
|
||||
|
||||
// Update metadata to indicate compression
|
||||
if (artifacts.userDefinedMetadata) {
|
||||
artifacts.userDefinedMetadata.compressed = true
|
||||
artifacts.userDefinedMetadata.compressionType = precision
|
||||
artifacts.userDefinedMetadata.compressionDate = new Date().toISOString()
|
||||
}
|
||||
|
||||
// Save compressed model
|
||||
await tf.io.fileSystem(outputPath).save(artifacts)
|
||||
|
||||
console.log(`✅ Compressed model saved to: ${outputPath}`)
|
||||
|
||||
model.dispose()
|
||||
|
||||
return true
|
||||
} catch (error) {
|
||||
console.error(`❌ Error compressing model: ${error.message}`)
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create optimized model variants
|
||||
*/
|
||||
async function createOptimizedVariants() {
|
||||
try {
|
||||
const originalModelPath = path.join(USE_MODEL_DIR, 'model.json')
|
||||
|
||||
if (!fs.existsSync(originalModelPath)) {
|
||||
console.error('❌ Original model not found. Please run "npm run download-models" first.')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
console.log('📊 Original model size:', getDirectorySizeMB(USE_MODEL_DIR), 'MB')
|
||||
|
||||
// Create float16 compressed version
|
||||
const float16Path = path.join(COMPRESSED_DIR, 'float16')
|
||||
if (!fs.existsSync(float16Path)) {
|
||||
fs.mkdirSync(float16Path, { recursive: true })
|
||||
}
|
||||
|
||||
console.log('\n🗜️ Creating float16 compressed version...')
|
||||
const float16Success = await compressModelWeights(
|
||||
originalModelPath,
|
||||
path.join(float16Path, 'model.json'),
|
||||
'float16'
|
||||
)
|
||||
|
||||
if (float16Success) {
|
||||
console.log('📊 Float16 model size:', getDirectorySizeMB(float16Path), 'MB')
|
||||
}
|
||||
|
||||
// Create int8 quantized version
|
||||
const int8Path = path.join(COMPRESSED_DIR, 'int8')
|
||||
if (!fs.existsSync(int8Path)) {
|
||||
fs.mkdirSync(int8Path, { recursive: true })
|
||||
}
|
||||
|
||||
console.log('\n🗜️ Creating int8 quantized version...')
|
||||
const int8Success = await compressModelWeights(
|
||||
originalModelPath,
|
||||
path.join(int8Path, 'model.json'),
|
||||
'int8'
|
||||
)
|
||||
|
||||
if (int8Success) {
|
||||
console.log('📊 Int8 model size:', getDirectorySizeMB(int8Path), 'MB')
|
||||
}
|
||||
|
||||
// Create compression summary
|
||||
const compressionSummary = {
|
||||
originalSize: getDirectorySizeMB(USE_MODEL_DIR),
|
||||
variants: {
|
||||
float16: {
|
||||
available: float16Success,
|
||||
size: float16Success ? getDirectorySizeMB(float16Path) : null,
|
||||
compressionRatio: float16Success ?
|
||||
(parseFloat(getDirectorySizeMB(USE_MODEL_DIR)) / parseFloat(getDirectorySizeMB(float16Path))).toFixed(2) : null
|
||||
},
|
||||
int8: {
|
||||
available: int8Success,
|
||||
size: int8Success ? getDirectorySizeMB(int8Path) : null,
|
||||
compressionRatio: int8Success ?
|
||||
(parseFloat(getDirectorySizeMB(USE_MODEL_DIR)) / parseFloat(getDirectorySizeMB(int8Path))).toFixed(2) : null
|
||||
}
|
||||
},
|
||||
createdAt: new Date().toISOString()
|
||||
}
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(COMPRESSED_DIR, 'compression-summary.json'),
|
||||
JSON.stringify(compressionSummary, null, 2)
|
||||
)
|
||||
|
||||
console.log('\n📋 Compression Summary:')
|
||||
console.log(`Original: ${compressionSummary.originalSize} MB`)
|
||||
if (float16Success) {
|
||||
console.log(`Float16: ${compressionSummary.variants.float16.size} MB (${compressionSummary.variants.float16.compressionRatio}x smaller)`)
|
||||
}
|
||||
if (int8Success) {
|
||||
console.log(`Int8: ${compressionSummary.variants.int8.size} MB (${compressionSummary.variants.int8.compressionRatio}x smaller)`)
|
||||
}
|
||||
|
||||
console.log('\n✨ Model compression completed successfully!')
|
||||
console.log('Compressed models are available for applications requiring smaller file sizes.')
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Error during compression:', error)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Optimize model for specific use cases
|
||||
*/
|
||||
async function optimizeForUseCase(useCase = 'general') {
|
||||
console.log(`\n🎯 Optimizing model for use case: ${useCase}`)
|
||||
|
||||
const optimizations = {
|
||||
general: {
|
||||
description: 'Balanced performance and size',
|
||||
precision: 'float16',
|
||||
batchSize: 32
|
||||
},
|
||||
'low-memory': {
|
||||
description: 'Minimal memory footprint',
|
||||
precision: 'int8',
|
||||
batchSize: 1
|
||||
},
|
||||
'high-performance': {
|
||||
description: 'Maximum inference speed',
|
||||
precision: 'float32',
|
||||
batchSize: 64
|
||||
}
|
||||
}
|
||||
|
||||
const config = optimizations[useCase] || optimizations.general
|
||||
|
||||
console.log(`📝 Optimization config: ${config.description}`)
|
||||
console.log(` Precision: ${config.precision}`)
|
||||
console.log(` Batch size: ${config.batchSize}`)
|
||||
|
||||
// Create optimization metadata
|
||||
const optimizationMetadata = {
|
||||
useCase,
|
||||
config,
|
||||
createdAt: new Date().toISOString(),
|
||||
recommendations: {
|
||||
'low-memory': 'Use int8 quantized model for memory-constrained environments',
|
||||
'high-performance': 'Use original float32 model with larger batch sizes',
|
||||
'general': 'Use float16 model for balanced performance'
|
||||
}
|
||||
}
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(COMPRESSED_DIR, `optimization-${useCase}.json`),
|
||||
JSON.stringify(optimizationMetadata, null, 2)
|
||||
)
|
||||
|
||||
console.log(`✅ Optimization profile created for ${useCase}`)
|
||||
}
|
||||
|
||||
// Main execution
|
||||
async function main() {
|
||||
try {
|
||||
await createOptimizedVariants()
|
||||
await optimizeForUseCase('general')
|
||||
await optimizeForUseCase('low-memory')
|
||||
await optimizeForUseCase('high-performance')
|
||||
|
||||
console.log('\n🎉 All optimizations completed successfully!')
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Compression failed:', error)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(console.error)
|
||||
|
|
@ -1,107 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Create GitHub Release Script for @soulcraft/brainy-models-package
|
||||
*
|
||||
* This script creates a GitHub release with auto-generated release notes
|
||||
* for the current version of the brainy-models-package.
|
||||
*
|
||||
* It uses the GitHub CLI (gh) to create the release, so the gh CLI must be installed
|
||||
* and authenticated with appropriate permissions.
|
||||
*
|
||||
* The script:
|
||||
* 1. Gets the current version from package.json
|
||||
* 2. Creates a GitHub release for that version with models-package prefix
|
||||
* 3. Auto-generates release notes based on commits since the last release
|
||||
*
|
||||
* This ensures that each npm release has a corresponding GitHub release with notes.
|
||||
*/
|
||||
|
||||
/* global process, console */
|
||||
|
||||
import { execSync } from 'child_process'
|
||||
import fs from 'fs'
|
||||
import path from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
|
||||
// Get the directory of the current module
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = path.dirname(__filename)
|
||||
|
||||
// Path to the brainy-models-package directory
|
||||
const packageDir = path.join(__dirname, '..')
|
||||
const rootDir = path.join(__dirname, '..', '..')
|
||||
|
||||
// Path to package.json
|
||||
const packageJsonPath = path.join(packageDir, 'package.json')
|
||||
|
||||
// Read package.json
|
||||
const packageJson = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8'))
|
||||
const version = packageJson.version
|
||||
const tagName = `models-package-v${version}`
|
||||
|
||||
// Check if GitHub CLI is installed
|
||||
try {
|
||||
execSync('gh --version', { stdio: 'ignore' })
|
||||
} catch (error) {
|
||||
console.error('Error: GitHub CLI (gh) is not installed or not in PATH')
|
||||
console.error('Please install it from https://cli.github.com/ and authenticate with `gh auth login`')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// Check if the tag exists locally
|
||||
let tagExistsLocally = false
|
||||
try {
|
||||
const tagOutput = execSync(`git tag -l ${tagName}`, { stdio: 'pipe', cwd: rootDir }).toString().trim()
|
||||
tagExistsLocally = tagOutput === tagName
|
||||
} catch (error) {
|
||||
console.log(`Error checking if tag exists: ${error.message}`)
|
||||
tagExistsLocally = false
|
||||
}
|
||||
|
||||
// Create and push the tag if it doesn't exist
|
||||
if (!tagExistsLocally) {
|
||||
try {
|
||||
console.log(`Creating tag ${tagName}...`)
|
||||
execSync(`git tag ${tagName}`, { stdio: 'inherit', cwd: rootDir })
|
||||
console.log(`Successfully created tag ${tagName}`)
|
||||
} catch (error) {
|
||||
console.error(`Error creating tag: ${error.message}`)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// Push the tag to remote
|
||||
try {
|
||||
console.log(`Pushing tag ${tagName} to remote...`)
|
||||
execSync(`git push origin ${tagName}`, { stdio: 'inherit', cwd: rootDir })
|
||||
console.log(`Successfully pushed tag ${tagName} to remote`)
|
||||
} catch (error) {
|
||||
console.error(`Error pushing tag to remote: ${error.message}`)
|
||||
// Continue with release creation even if tag push fails
|
||||
}
|
||||
|
||||
// Create the GitHub release
|
||||
try {
|
||||
console.log(`Creating GitHub release for @soulcraft/brainy-models-package v${version}...`)
|
||||
|
||||
// Create a release with auto-generated notes
|
||||
// The --generate-notes flag automatically generates release notes based on PRs and commits
|
||||
execSync(
|
||||
`gh release create ${tagName} --title "@soulcraft/brainy-models-package v${version}" --generate-notes --notes "Release of @soulcraft/brainy-models-package v${version} - Pre-bundled TensorFlow models for maximum reliability with Brainy vector database."`,
|
||||
{ stdio: 'inherit', cwd: rootDir }
|
||||
)
|
||||
|
||||
console.log(`GitHub release ${tagName} created successfully!`)
|
||||
console.log('GitHub release created with auto-generated notes')
|
||||
} catch (error) {
|
||||
// If the release already exists, this is not a fatal error
|
||||
if (error.message.includes('already exists')) {
|
||||
console.log(`GitHub release ${tagName} already exists, skipping creation.`)
|
||||
console.log('GitHub release already exists with auto-generated notes')
|
||||
} else {
|
||||
console.error('Error creating GitHub release:', error.message)
|
||||
// Don't exit with error to allow the npm publish to continue
|
||||
// process.exit(1)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,213 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/* eslint-env node */
|
||||
/* eslint-disable no-console */
|
||||
|
||||
/**
|
||||
* Download Full Models Script for @soulcraft/brainy-models
|
||||
*
|
||||
* This script downloads the complete Universal Sentence Encoder model
|
||||
* and saves it locally for offline use, providing maximum reliability.
|
||||
*/
|
||||
|
||||
import fs from 'fs'
|
||||
import path from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
import * as tf from '@tensorflow/tfjs-node'
|
||||
import * as use from '@tensorflow-models/universal-sentence-encoder'
|
||||
import https from 'https'
|
||||
import { promisify } from 'util'
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = path.dirname(__filename)
|
||||
|
||||
const MODELS_DIR = path.join(__dirname, '..', 'models')
|
||||
const USE_MODEL_DIR = path.join(MODELS_DIR, 'universal-sentence-encoder')
|
||||
|
||||
// Ensure directories exist
|
||||
if (!fs.existsSync(MODELS_DIR)) {
|
||||
fs.mkdirSync(MODELS_DIR, { recursive: true })
|
||||
}
|
||||
|
||||
if (!fs.existsSync(USE_MODEL_DIR)) {
|
||||
fs.mkdirSync(USE_MODEL_DIR, { recursive: true })
|
||||
}
|
||||
|
||||
console.log('🚀 Starting full model download for @soulcraft/brainy-models...')
|
||||
console.log('This will download the complete Universal Sentence Encoder model (~25MB)')
|
||||
console.log('for offline use and maximum reliability.\n')
|
||||
|
||||
/**
|
||||
* Download a file from URL to local path
|
||||
*/
|
||||
async function downloadFile(url, filePath, maxRedirects = 5) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const file = fs.createWriteStream(filePath)
|
||||
|
||||
const handleRequest = (requestUrl, redirectCount = 0) => {
|
||||
https.get(requestUrl, (response) => {
|
||||
// Handle redirects
|
||||
if (response.statusCode >= 300 && response.statusCode < 400) {
|
||||
if (redirectCount >= maxRedirects) {
|
||||
reject(new Error(`Too many redirects (${redirectCount}) for ${url}`))
|
||||
return
|
||||
}
|
||||
|
||||
const location = response.headers.location
|
||||
if (!location) {
|
||||
reject(new Error(`Redirect response without location header for ${url}`))
|
||||
return
|
||||
}
|
||||
|
||||
// Handle relative redirects
|
||||
const redirectUrl = location.startsWith('http') ? location : new URL(location, requestUrl).href
|
||||
console.log(`📍 Following redirect ${redirectCount + 1}: ${redirectUrl}`)
|
||||
|
||||
// Close the current file stream and start over with the redirect URL
|
||||
file.close()
|
||||
fs.unlink(filePath, () => {}) // Delete partial file
|
||||
|
||||
// Recursively handle the redirect
|
||||
return downloadFile(redirectUrl, filePath, maxRedirects).then(resolve).catch(reject)
|
||||
}
|
||||
|
||||
if (response.statusCode !== 200) {
|
||||
reject(new Error(`Failed to download ${url}: ${response.statusCode}`))
|
||||
return
|
||||
}
|
||||
|
||||
const totalSize = parseInt(response.headers['content-length'] || '0')
|
||||
let downloadedSize = 0
|
||||
|
||||
response.on('data', (chunk) => {
|
||||
downloadedSize += chunk.length
|
||||
if (totalSize > 0) {
|
||||
const progress = ((downloadedSize / totalSize) * 100).toFixed(1)
|
||||
process.stdout.write(`\r📥 Downloading: ${progress}% (${downloadedSize}/${totalSize} bytes)`)
|
||||
}
|
||||
})
|
||||
|
||||
response.pipe(file)
|
||||
|
||||
file.on('finish', () => {
|
||||
file.close()
|
||||
console.log(`\n✅ Downloaded: ${path.basename(filePath)}`)
|
||||
resolve()
|
||||
})
|
||||
|
||||
file.on('error', (err) => {
|
||||
fs.unlink(filePath, () => {}) // Delete partial file
|
||||
reject(err)
|
||||
})
|
||||
}).on('error', reject)
|
||||
}
|
||||
|
||||
handleRequest(url)
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Download the complete Universal Sentence Encoder model
|
||||
*/
|
||||
async function downloadFullModel() {
|
||||
try {
|
||||
console.log('🔍 Loading model to get download URLs...')
|
||||
|
||||
// Load the model to get access to its internal structure
|
||||
const model = await use.load()
|
||||
console.log('✅ Model loaded successfully')
|
||||
|
||||
// Test the model to ensure it works
|
||||
console.log('🧪 Testing model functionality...')
|
||||
const testEmbedding = await model.embed(['Hello world'])
|
||||
const testArray = await testEmbedding.array()
|
||||
console.log(`✅ Model test passed - embedding dimensions: ${testArray[0].length}`)
|
||||
testEmbedding.dispose()
|
||||
|
||||
// The Universal Sentence Encoder model URL (using Google Cloud Storage which still works)
|
||||
const modelBaseUrl = 'https://storage.googleapis.com/tfjs-models/savedmodel/universal_sentence_encoder'
|
||||
|
||||
console.log('📦 Downloading model files...')
|
||||
console.log('Using Google Cloud Storage URLs (TensorFlow Hub URLs are deprecated)...')
|
||||
|
||||
// Download model.json
|
||||
const modelJsonUrl = `${modelBaseUrl}/model.json`
|
||||
const modelJsonPath = path.join(USE_MODEL_DIR, 'model.json')
|
||||
await downloadFile(modelJsonUrl, modelJsonPath)
|
||||
|
||||
// Read the model.json to get the weights manifest
|
||||
const modelJson = JSON.parse(fs.readFileSync(modelJsonPath, 'utf8'))
|
||||
|
||||
// Add the required "format" field for TensorFlow.js compatibility
|
||||
if (!modelJson.format) {
|
||||
modelJson.format = 'tfjs-graph-model'
|
||||
fs.writeFileSync(modelJsonPath, JSON.stringify(modelJson, null, 2))
|
||||
console.log('✅ Added "format" field to model.json for TensorFlow.js compatibility')
|
||||
}
|
||||
|
||||
// Download all weight files
|
||||
if (modelJson.weightsManifest) {
|
||||
for (const manifest of modelJson.weightsManifest) {
|
||||
for (const weightFile of manifest.paths) {
|
||||
const weightUrl = `${modelBaseUrl}/${weightFile}`
|
||||
const weightPath = path.join(USE_MODEL_DIR, weightFile)
|
||||
await downloadFile(weightUrl, weightPath)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Create metadata for the bundled model
|
||||
const metadata = {
|
||||
name: 'universal-sentence-encoder',
|
||||
version: '1.0.0',
|
||||
description: 'Complete Universal Sentence Encoder model bundled for offline use',
|
||||
dimensions: 512,
|
||||
downloadDate: new Date().toISOString(),
|
||||
source: 'tensorflow-models/universal-sentence-encoder',
|
||||
approach: 'full-bundle',
|
||||
modelUrl: modelBaseUrl,
|
||||
bundledLocally: true,
|
||||
reliability: 'maximum'
|
||||
}
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(USE_MODEL_DIR, 'metadata.json'),
|
||||
JSON.stringify(metadata, null, 2)
|
||||
)
|
||||
|
||||
// Verify all files exist and calculate total size
|
||||
const modelFiles = fs.readdirSync(USE_MODEL_DIR)
|
||||
let totalSize = 0
|
||||
|
||||
console.log('\n📋 Downloaded files:')
|
||||
for (const file of modelFiles) {
|
||||
const filePath = path.join(USE_MODEL_DIR, file)
|
||||
const stats = fs.statSync(filePath)
|
||||
totalSize += stats.size
|
||||
console.log(` ✅ ${file} (${(stats.size / 1024 / 1024).toFixed(2)} MB)`)
|
||||
}
|
||||
|
||||
console.log(`\n🎉 Model download complete!`)
|
||||
console.log(`📊 Total size: ${(totalSize / 1024 / 1024).toFixed(2)} MB`)
|
||||
console.log(`📁 Location: ${USE_MODEL_DIR}`)
|
||||
console.log(`🔒 Reliability: Maximum (fully offline)`)
|
||||
|
||||
// Test loading the downloaded model
|
||||
console.log('\n🧪 Testing downloaded model...')
|
||||
const offlineModel = await tf.loadGraphModel(`file://${path.join(USE_MODEL_DIR, 'model.json')}`)
|
||||
console.log('✅ Offline model loads successfully')
|
||||
|
||||
// Clean up
|
||||
offlineModel.dispose()
|
||||
|
||||
console.log('\n✨ Full model bundling completed successfully!')
|
||||
console.log('The model is now available for offline use with maximum reliability.')
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Error downloading full model:', error)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// Run the download
|
||||
downloadFullModel().catch(console.error)
|
||||
|
|
@ -1,171 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Release Workflow Script for @soulcraft/brainy-models-package
|
||||
*
|
||||
* This script provides a comprehensive workflow for releasing a new version:
|
||||
* 1. Updates the version (major, minor, or patch)
|
||||
* 2. Automatically updates the CHANGELOG.md with commit messages since the last release
|
||||
* 3. Creates a GitHub release
|
||||
* 4. Deploys to NPM
|
||||
*
|
||||
* Usage:
|
||||
* node scripts/release-workflow.js [patch|minor|major]
|
||||
*
|
||||
* If no version type is specified, it defaults to "patch"
|
||||
*/
|
||||
|
||||
/* global process, console */
|
||||
|
||||
import { execSync } from 'child_process'
|
||||
import fs from 'fs'
|
||||
import path from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
import readline from 'readline'
|
||||
|
||||
// Get the directory of the current module
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = path.dirname(__filename)
|
||||
|
||||
// Path to the brainy-models-package directory
|
||||
const packageDir = path.join(__dirname, '..')
|
||||
const rootDir = path.join(__dirname, '..', '..')
|
||||
|
||||
// Get the version type from command line arguments
|
||||
const args = process.argv.slice(2)
|
||||
const versionType = args[0] || 'patch'
|
||||
|
||||
// Validate version type
|
||||
if (!['patch', 'minor', 'major'].includes(versionType)) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.error('Error: Version type must be one of: patch, minor, major')
|
||||
// eslint-disable-next-line no-process-exit
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// Function to execute a command and log its output
|
||||
function executeStep(command, description, cwd = packageDir) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`\n🚀 ${description}...\n`)
|
||||
try {
|
||||
execSync(command, { stdio: 'inherit', cwd })
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`✅ ${description} completed successfully!\n`)
|
||||
return true
|
||||
} catch (error) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.error(
|
||||
`❌ Error during ${description.toLowerCase()}: ${error.message}`
|
||||
)
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Main workflow
|
||||
async function runReleaseWorkflow() {
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(
|
||||
`\n=== Starting @soulcraft/brainy-models-package Release Workflow (${versionType}) ===\n`
|
||||
)
|
||||
|
||||
// Step 1: Build the project
|
||||
if (!executeStep('npm run build', 'Building brainy-models-package')) {
|
||||
// eslint-disable-next-line no-process-exit
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// Step 2: Run tests to ensure everything is working
|
||||
if (!executeStep('npm test', 'Running tests')) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(
|
||||
'⚠️ Tests failed. This might indicate issues with the release.'
|
||||
)
|
||||
|
||||
// Ask the user if they want to continue despite test failures
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(
|
||||
'\n⚠️ Do you want to continue with the release process despite test failures? (y/N)'
|
||||
)
|
||||
|
||||
const rl = readline.createInterface({
|
||||
input: process.stdin,
|
||||
output: process.stdout
|
||||
})
|
||||
|
||||
const response = await new Promise((resolve) => {
|
||||
rl.question('', (answer) => {
|
||||
rl.close()
|
||||
resolve(answer.toLowerCase())
|
||||
})
|
||||
})
|
||||
|
||||
if (response !== 'y' && response !== 'yes') {
|
||||
// eslint-disable-next-line no-console
|
||||
console.error('Release process aborted due to test failures.')
|
||||
// eslint-disable-next-line no-process-exit
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// eslint-disable-next-line no-console
|
||||
console.log('Continuing with release process despite test failures...')
|
||||
}
|
||||
|
||||
// Step 3: Update version and generate changelog
|
||||
if (
|
||||
!executeStep(
|
||||
`npm run release:${versionType}`,
|
||||
`Updating version (${versionType}) and generating changelog`
|
||||
)
|
||||
) {
|
||||
// eslint-disable-next-line no-process-exit
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// Step 4: Create GitHub release
|
||||
if (!executeStep('npm run _github-release', 'Creating GitHub release')) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(
|
||||
'Warning: GitHub release creation failed, but continuing with deployment...'
|
||||
)
|
||||
}
|
||||
|
||||
// Step 5: Publish to NPM
|
||||
if (!executeStep('npm publish', 'Publishing to NPM')) {
|
||||
// eslint-disable-next-line no-process-exit
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// Get the new version from package.json
|
||||
const packageJsonPath = path.join(packageDir, 'package.json')
|
||||
const packageJson = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8'))
|
||||
const newVersion = packageJson.version
|
||||
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(
|
||||
`\n🎉 @soulcraft/brainy-models-package v${newVersion} release completed successfully! 🎉\n`
|
||||
)
|
||||
// eslint-disable-next-line no-console
|
||||
console.log('Summary of actions:')
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`- Package built and tested`)
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`- Version bumped to v${newVersion} (${versionType})`)
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`- CHANGELOG.md updated with recent commits`)
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`- GitHub release created with auto-generated notes`)
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`- Package published to NPM as @soulcraft/brainy-models-package`)
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(
|
||||
'\nThank you for using the brainy-models-package release workflow!\n'
|
||||
)
|
||||
}
|
||||
|
||||
// Run the workflow
|
||||
runReleaseWorkflow().catch((error) => {
|
||||
// eslint-disable-next-line no-console
|
||||
console.error('Unexpected error during release workflow:', error)
|
||||
// eslint-disable-next-line no-process-exit
|
||||
process.exit(1)
|
||||
})
|
||||
|
|
@ -1,302 +0,0 @@
|
|||
/**
|
||||
* @soulcraft/brainy-models
|
||||
*
|
||||
* Pre-bundled TensorFlow models for maximum reliability with Brainy vector database.
|
||||
* This package provides offline access to the Universal Sentence Encoder model,
|
||||
* eliminating network dependencies and ensuring consistent performance.
|
||||
*/
|
||||
|
||||
import * as tf from '@tensorflow/tfjs'
|
||||
import { readFileSync, existsSync } from 'fs'
|
||||
import { join, dirname } from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
|
||||
/**
|
||||
* Helper function to safely extract error message from unknown error type
|
||||
*/
|
||||
function getErrorMessage(error: unknown): string {
|
||||
if (error instanceof Error) {
|
||||
return error.message
|
||||
}
|
||||
if (typeof error === 'string') {
|
||||
return error
|
||||
}
|
||||
return String(error)
|
||||
}
|
||||
|
||||
// Get the package directory
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = dirname(__filename)
|
||||
const PACKAGE_ROOT = join(__dirname, '..')
|
||||
const MODELS_DIR = join(PACKAGE_ROOT, 'models')
|
||||
|
||||
export interface ModelMetadata {
|
||||
name: string
|
||||
version: string
|
||||
description: string
|
||||
dimensions: number
|
||||
downloadDate: string
|
||||
source: string
|
||||
approach: string
|
||||
modelUrl: string
|
||||
bundledLocally: boolean
|
||||
reliability: string
|
||||
}
|
||||
|
||||
export interface BundledModelOptions {
|
||||
verbose?: boolean
|
||||
preferCompressed?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Bundled Universal Sentence Encoder for offline use
|
||||
*/
|
||||
export class BundledUniversalSentenceEncoder {
|
||||
private model: tf.GraphModel | null = null
|
||||
private metadata: ModelMetadata | null = null
|
||||
private options: BundledModelOptions
|
||||
|
||||
constructor(options: BundledModelOptions = {}) {
|
||||
this.options = {
|
||||
verbose: false,
|
||||
preferCompressed: false,
|
||||
...options
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Load the bundled model from local files
|
||||
*/
|
||||
async load(): Promise<void> {
|
||||
try {
|
||||
const modelDir = join(MODELS_DIR, 'universal-sentence-encoder')
|
||||
const modelPath = join(modelDir, 'model.json')
|
||||
const metadataPath = join(modelDir, 'metadata.json')
|
||||
|
||||
if (!existsSync(modelPath)) {
|
||||
throw new Error(
|
||||
`Bundled model not found at ${modelPath}. ` +
|
||||
'Please run "npm run download-models" to download the model files.'
|
||||
)
|
||||
}
|
||||
|
||||
if (this.options.verbose) {
|
||||
console.log('🔄 Loading bundled Universal Sentence Encoder model...')
|
||||
}
|
||||
|
||||
// Load metadata
|
||||
if (existsSync(metadataPath)) {
|
||||
const metadataContent = readFileSync(metadataPath, 'utf8')
|
||||
this.metadata = JSON.parse(metadataContent)
|
||||
|
||||
if (this.options.verbose) {
|
||||
console.log(`📋 Model metadata:`, this.metadata)
|
||||
}
|
||||
}
|
||||
|
||||
// Load the model
|
||||
this.model = await tf.loadGraphModel(`file://${modelPath}`)
|
||||
|
||||
if (this.options.verbose) {
|
||||
console.log('✅ Bundled model loaded successfully')
|
||||
console.log(`🔒 Reliability: Maximum (fully offline)`)
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to load bundled model: ${getErrorMessage(error)}`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate embeddings for the given texts
|
||||
*/
|
||||
async embed(texts: string[]): Promise<tf.Tensor2D> {
|
||||
if (!this.model) {
|
||||
throw new Error('Model not loaded. Call load() first.')
|
||||
}
|
||||
|
||||
try {
|
||||
// Convert texts to tensor
|
||||
const inputTensor = tf.tensor1d(texts, 'string')
|
||||
|
||||
// Run inference
|
||||
const embeddings = this.model.predict(inputTensor) as tf.Tensor2D
|
||||
|
||||
// Clean up input tensor
|
||||
inputTensor.dispose()
|
||||
|
||||
return embeddings
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to generate embeddings: ${getErrorMessage(error)}`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate embeddings and return as JavaScript arrays
|
||||
*/
|
||||
async embedToArrays(texts: string[]): Promise<number[][]> {
|
||||
const embeddings = await this.embed(texts)
|
||||
const arrays = await embeddings.array() as number[][]
|
||||
embeddings.dispose()
|
||||
return arrays
|
||||
}
|
||||
|
||||
/**
|
||||
* Get model metadata
|
||||
*/
|
||||
getMetadata(): ModelMetadata | null {
|
||||
return this.metadata
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if the model is loaded
|
||||
*/
|
||||
isLoaded(): boolean {
|
||||
return this.model !== null
|
||||
}
|
||||
|
||||
/**
|
||||
* Get model information
|
||||
*/
|
||||
getModelInfo(): { inputShape: number[], outputShape: number[] } | null {
|
||||
if (!this.model) {
|
||||
return null
|
||||
}
|
||||
|
||||
return {
|
||||
inputShape: this.model.inputs[0].shape || [],
|
||||
outputShape: this.model.outputs[0].shape || []
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Dispose of the model and free memory
|
||||
*/
|
||||
dispose(): void {
|
||||
if (this.model) {
|
||||
this.model.dispose()
|
||||
this.model = null
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Model compression utilities
|
||||
*/
|
||||
export class ModelCompressor {
|
||||
/**
|
||||
* Compress model weights using quantization
|
||||
* Note: TensorFlow.js doesn't currently support model quantization
|
||||
*/
|
||||
static async quantizeModel(
|
||||
modelPath: string,
|
||||
outputPath: string,
|
||||
options: { dtype?: 'int8' | 'int16' } = {}
|
||||
): Promise<void> {
|
||||
const { dtype = 'int8' } = options
|
||||
|
||||
try {
|
||||
console.log(`🔄 Loading model for quantization: ${modelPath}`)
|
||||
const model = await tf.loadGraphModel(`file://${modelPath}`)
|
||||
|
||||
console.log(`🗜️ Quantizing model to ${dtype}...`)
|
||||
|
||||
// TensorFlow.js doesn't have built-in quantization or model serialization APIs yet
|
||||
// This is a placeholder implementation that acknowledges the limitation
|
||||
console.warn('⚠️ Model quantization is not yet supported in TensorFlow.js')
|
||||
console.log(`📋 Model loaded successfully from: ${modelPath}`)
|
||||
console.log(`📋 Target output path: ${outputPath}`)
|
||||
console.log(`📋 Target dtype: ${dtype}`)
|
||||
|
||||
model.dispose()
|
||||
|
||||
throw new Error('Model quantization is not yet supported in TensorFlow.js. This feature requires server-side processing with TensorFlow Python.')
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to compress model: ${getErrorMessage(error)}`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get model size information by reading files from disk
|
||||
*/
|
||||
static async getModelSize(modelPath: string): Promise<{
|
||||
totalSize: number
|
||||
weightsSize: number
|
||||
modelJsonSize: number
|
||||
}> {
|
||||
try {
|
||||
// Load model to verify it's valid
|
||||
const model = await tf.loadGraphModel(`file://${modelPath}`)
|
||||
model.dispose()
|
||||
|
||||
// Get model.json size
|
||||
const modelJsonSize = existsSync(modelPath) ? readFileSync(modelPath).length : 0
|
||||
|
||||
// Calculate weights size by reading weight files
|
||||
let weightsSize = 0
|
||||
const modelDir = dirname(modelPath)
|
||||
|
||||
// Read model.json to get weight file names
|
||||
if (existsSync(modelPath)) {
|
||||
const modelJson = JSON.parse(readFileSync(modelPath, 'utf8'))
|
||||
if (modelJson.weightsManifest) {
|
||||
for (const manifest of modelJson.weightsManifest) {
|
||||
for (const path of manifest.paths) {
|
||||
const weightFilePath = join(modelDir, path)
|
||||
if (existsSync(weightFilePath)) {
|
||||
weightsSize += readFileSync(weightFilePath).length
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const totalSize = weightsSize + modelJsonSize
|
||||
|
||||
return {
|
||||
totalSize,
|
||||
weightsSize,
|
||||
modelJsonSize
|
||||
}
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to get model size: ${getErrorMessage(error)}`)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Utility functions
|
||||
*/
|
||||
export const utils = {
|
||||
/**
|
||||
* Check if bundled models are available
|
||||
*/
|
||||
checkModelsAvailable(): boolean {
|
||||
const modelPath = join(MODELS_DIR, 'universal-sentence-encoder', 'model.json')
|
||||
return existsSync(modelPath)
|
||||
},
|
||||
|
||||
/**
|
||||
* Get bundled models directory
|
||||
*/
|
||||
getModelsDirectory(): string {
|
||||
return MODELS_DIR
|
||||
},
|
||||
|
||||
/**
|
||||
* List available bundled models
|
||||
*/
|
||||
listAvailableModels(): string[] {
|
||||
const models: string[] = []
|
||||
const useModelPath = join(MODELS_DIR, 'universal-sentence-encoder', 'model.json')
|
||||
|
||||
if (existsSync(useModelPath)) {
|
||||
models.push('universal-sentence-encoder')
|
||||
}
|
||||
|
||||
return models
|
||||
}
|
||||
}
|
||||
|
||||
// Default export for convenience
|
||||
export default BundledUniversalSentenceEncoder
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "node",
|
||||
"lib": ["ES2022"],
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"declaration": true,
|
||||
"declarationMap": true,
|
||||
"sourceMap": true,
|
||||
"removeComments": false,
|
||||
"allowSyntheticDefaultImports": true,
|
||||
"resolveJsonModule": true
|
||||
},
|
||||
"include": [
|
||||
"src/**/*"
|
||||
],
|
||||
"exclude": [
|
||||
"node_modules",
|
||||
"dist",
|
||||
"test",
|
||||
"scripts"
|
||||
]
|
||||
}
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Script to reproduce the FileSystemStorage initialization error
|
||||
*/
|
||||
|
||||
import { createStorage } from './dist/unified.js'
|
||||
|
||||
async function reproduceError() {
|
||||
console.log('Attempting to reproduce FileSystemStorage error...')
|
||||
|
||||
try {
|
||||
// This should trigger the same error as in the test
|
||||
const storage = await createStorage({ forceFileSystemStorage: true })
|
||||
console.log('Storage created successfully:', storage.constructor.name)
|
||||
|
||||
// This should fail with the fs/path modules error
|
||||
await storage.init()
|
||||
console.log('Storage initialized successfully')
|
||||
|
||||
} catch (error) {
|
||||
console.error('Error reproduced:', error.message)
|
||||
console.error('Full error:', error)
|
||||
}
|
||||
}
|
||||
|
||||
reproduceError()
|
||||
|
|
@ -1,236 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Reproduction script for race condition issues in Brainy
|
||||
*
|
||||
* This script demonstrates two main issues:
|
||||
* 1. Race condition: Verbs arrive before their associated nouns are indexed
|
||||
* 2. Indexing delay: Newly inserted nouns can't be looked up immediately
|
||||
*/
|
||||
|
||||
const { BrainyData } = require('./dist/unified.js');
|
||||
|
||||
async function reproduceRaceCondition() {
|
||||
console.log('🧠 Starting Brainy Race Condition Reproduction Test');
|
||||
console.log('=' .repeat(60));
|
||||
|
||||
const brainy = new BrainyData({
|
||||
dimensions: 512, // Use correct dimensions for Universal Sentence Encoder
|
||||
maxConnections: 16,
|
||||
efConstruction: 200,
|
||||
storageType: 'memory' // Use memory storage for faster testing
|
||||
});
|
||||
|
||||
await brainy.init();
|
||||
|
||||
console.log('\n📊 Test 1: Race Condition - Verbs before Nouns');
|
||||
console.log('-'.repeat(50));
|
||||
|
||||
try {
|
||||
// Simulate streaming data where verbs arrive before nouns
|
||||
const sourceId = 'user-123';
|
||||
const targetId = 'post-456';
|
||||
|
||||
console.log(`Attempting to add verb between ${sourceId} and ${targetId} before nouns exist...`);
|
||||
|
||||
// This should fail because the nouns don't exist yet
|
||||
await brainy.addVerb(sourceId, targetId, null, {
|
||||
type: 'likes',
|
||||
metadata: { action: 'like', timestamp: Date.now() }
|
||||
});
|
||||
|
||||
console.log('❌ Expected failure did not occur - this indicates the race condition is not properly handled');
|
||||
|
||||
} catch (error) {
|
||||
console.log('✅ Expected error occurred:', error.message);
|
||||
}
|
||||
|
||||
console.log('\n📊 Test 2: Indexing Delay Issue');
|
||||
console.log('-'.repeat(50));
|
||||
|
||||
try {
|
||||
// Add a noun
|
||||
const nounId = 'rapid-noun-' + Date.now();
|
||||
console.log(`Adding noun with ID: ${nounId}`);
|
||||
|
||||
await brainy.add(Array.from({length: 512}, () => Math.random()), {
|
||||
type: 'user',
|
||||
name: 'Test User'
|
||||
}, { id: nounId });
|
||||
|
||||
console.log('✅ Noun added successfully');
|
||||
|
||||
// Immediately try to add a verb that references this noun
|
||||
const verbTargetId = 'target-' + Date.now();
|
||||
|
||||
// Add target noun
|
||||
await brainy.add(Array.from({length: 512}, () => Math.random()), {
|
||||
type: 'post',
|
||||
title: 'Test Post'
|
||||
}, { id: verbTargetId });
|
||||
|
||||
console.log('✅ Target noun added successfully');
|
||||
|
||||
// Now try to add verb immediately - this might fail due to indexing delay
|
||||
console.log(`Attempting to add verb between ${nounId} and ${verbTargetId} immediately after noun creation...`);
|
||||
|
||||
const verbId = await brainy.addVerb(nounId, verbTargetId, null, {
|
||||
type: 'created',
|
||||
metadata: { action: 'create', timestamp: Date.now() }
|
||||
});
|
||||
|
||||
console.log('✅ Verb added successfully with ID:', verbId);
|
||||
|
||||
} catch (error) {
|
||||
console.log('❌ Indexing delay error occurred:', error.message);
|
||||
}
|
||||
|
||||
console.log('\n📊 Test 3: Rapid Streaming Simulation');
|
||||
console.log('-'.repeat(50));
|
||||
|
||||
const errors = [];
|
||||
const successes = [];
|
||||
|
||||
// Simulate rapid streaming of mixed noun and verb data
|
||||
const operations = [];
|
||||
|
||||
for (let i = 0; i < 50; i++) {
|
||||
const userId = `user-${i}`;
|
||||
const postId = `post-${i}`;
|
||||
|
||||
// Randomly order noun and verb operations to simulate streaming
|
||||
if (Math.random() > 0.5) {
|
||||
// Add verb first (should fail without autoCreateMissingNouns)
|
||||
operations.push({
|
||||
type: 'verb',
|
||||
sourceId: userId,
|
||||
targetId: postId,
|
||||
verbType: 'likes',
|
||||
id: i
|
||||
});
|
||||
|
||||
// Then add nouns
|
||||
operations.push({
|
||||
type: 'noun',
|
||||
id: userId,
|
||||
vector: Array.from({length: 512}, () => Math.random()),
|
||||
metadata: { type: 'user', name: `User ${i}` }
|
||||
});
|
||||
|
||||
operations.push({
|
||||
type: 'noun',
|
||||
id: postId,
|
||||
vector: Array.from({length: 512}, () => Math.random()),
|
||||
metadata: { type: 'post', title: `Post ${i}` }
|
||||
});
|
||||
} else {
|
||||
// Add nouns first
|
||||
operations.push({
|
||||
type: 'noun',
|
||||
id: userId,
|
||||
vector: Array.from({length: 384}, () => Math.random()),
|
||||
metadata: { type: 'user', name: `User ${i}` }
|
||||
});
|
||||
|
||||
operations.push({
|
||||
type: 'noun',
|
||||
id: postId,
|
||||
vector: Array.from({length: 384}, () => Math.random()),
|
||||
metadata: { type: 'post', title: `Post ${i}` }
|
||||
});
|
||||
|
||||
// Then add verb
|
||||
operations.push({
|
||||
type: 'verb',
|
||||
sourceId: userId,
|
||||
targetId: postId,
|
||||
verbType: 'likes',
|
||||
id: i
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Executing ${operations.length} operations in streaming order...`);
|
||||
|
||||
for (const op of operations) {
|
||||
try {
|
||||
if (op.type === 'noun') {
|
||||
await brainy.add(op.vector, op.metadata, { id: op.id });
|
||||
successes.push(`Added noun ${op.id}`);
|
||||
} else if (op.type === 'verb') {
|
||||
await brainy.addVerb(op.sourceId, op.targetId, null, {
|
||||
type: op.verbType,
|
||||
metadata: { streamingTest: true, operationId: op.id }
|
||||
});
|
||||
successes.push(`Added verb ${op.sourceId} -> ${op.targetId}`);
|
||||
}
|
||||
} catch (error) {
|
||||
errors.push(`${op.type} operation failed: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log('\n📈 Results Summary:');
|
||||
console.log(`✅ Successful operations: ${successes.length}`);
|
||||
console.log(`❌ Failed operations: ${errors.length}`);
|
||||
|
||||
if (errors.length > 0) {
|
||||
console.log('\n❌ Error Details:');
|
||||
errors.slice(0, 10).forEach(error => console.log(` - ${error}`));
|
||||
if (errors.length > 10) {
|
||||
console.log(` ... and ${errors.length - 10} more errors`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log('\n📊 Test 4: Auto-Create Missing Nouns Feature');
|
||||
console.log('-'.repeat(50));
|
||||
|
||||
try {
|
||||
const autoSourceId = 'auto-user-' + Date.now();
|
||||
const autoTargetId = 'auto-post-' + Date.now();
|
||||
|
||||
console.log(`Testing autoCreateMissingNouns feature with ${autoSourceId} -> ${autoTargetId}`);
|
||||
|
||||
const autoVerbId = await brainy.addVerb(autoSourceId, autoTargetId, null, {
|
||||
type: 'follows',
|
||||
autoCreateMissingNouns: true,
|
||||
missingNounMetadata: { autoCreated: true, testCase: 'reproduction' },
|
||||
metadata: { testFeature: 'autoCreate' }
|
||||
});
|
||||
|
||||
console.log('✅ Auto-create feature worked! Verb ID:', autoVerbId);
|
||||
|
||||
// Verify the auto-created nouns exist
|
||||
const autoSourceNoun = await brainy.get(autoSourceId);
|
||||
const autoTargetNoun = await brainy.get(autoTargetId);
|
||||
|
||||
console.log('✅ Auto-created source noun exists:', !!autoSourceNoun);
|
||||
console.log('✅ Auto-created target noun exists:', !!autoTargetNoun);
|
||||
|
||||
} catch (error) {
|
||||
console.log('❌ Auto-create feature failed:', error.message);
|
||||
}
|
||||
|
||||
await brainy.shutDown();
|
||||
|
||||
console.log('\n🏁 Race Condition Reproduction Test Complete');
|
||||
console.log('=' .repeat(60));
|
||||
|
||||
if (errors.length > 0) {
|
||||
console.log('\n💡 Recommendations:');
|
||||
console.log('1. Implement fallback storage lookup when index lookup fails');
|
||||
console.log('2. Add deferred resolution queue for missing noun references');
|
||||
console.log('3. Implement write-only mode that bypasses index checks');
|
||||
console.log('4. Add proper index synchronization mechanisms');
|
||||
|
||||
process.exit(1);
|
||||
} else {
|
||||
console.log('\n✅ All tests passed - race conditions may already be handled');
|
||||
process.exit(0);
|
||||
}
|
||||
}
|
||||
|
||||
// Run the reproduction test
|
||||
reproduceRaceCondition().catch(error => {
|
||||
console.error('💥 Reproduction script failed:', error);
|
||||
process.exit(1);
|
||||
});
|
||||
|
|
@ -1,127 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Script to reproduce the write-only mode issues described in the GitHub issue
|
||||
*/
|
||||
|
||||
import { BrainyData } from './dist/unified.js'
|
||||
|
||||
async function reproduceWriteOnlyIssues() {
|
||||
console.log('🧠 Reproducing write-only mode issues...\n')
|
||||
|
||||
try {
|
||||
// Create a BrainyData instance
|
||||
const brainy = new BrainyData({
|
||||
dimensions: 512,
|
||||
storage: {
|
||||
type: 'memory'
|
||||
}
|
||||
})
|
||||
|
||||
// Initialize the database
|
||||
await brainy.init()
|
||||
console.log('✅ BrainyData initialized')
|
||||
|
||||
// Set to write-only mode
|
||||
brainy.setWriteOnly(true)
|
||||
console.log('✅ Set to write-only mode')
|
||||
|
||||
// Try to add some data - this should work
|
||||
console.log('\n📝 Testing add operations in write-only mode...')
|
||||
const id1 = await brainy.add('This is test data 1', { type: 'test' })
|
||||
console.log(`✅ Added item with ID: ${id1}`)
|
||||
|
||||
const id2 = await brainy.add('This is test data 2', { type: 'test' })
|
||||
console.log(`✅ Added item with ID: ${id2}`)
|
||||
|
||||
// Try to search - this should fail with current implementation
|
||||
console.log('\n🔍 Testing search operations in write-only mode...')
|
||||
try {
|
||||
const results = await brainy.search('test query', 5)
|
||||
console.log('❌ UNEXPECTED: Search succeeded in write-only mode')
|
||||
console.log('Results:', results)
|
||||
} catch (error) {
|
||||
console.log('✅ EXPECTED: Search failed in write-only mode')
|
||||
console.log('Error:', error.message)
|
||||
}
|
||||
|
||||
// Try existence check via get() - this should now work in write-only mode
|
||||
console.log('\n🔍 Testing existence checks in write-only mode...')
|
||||
try {
|
||||
const item = await brainy.get(id1)
|
||||
console.log('✅ EXPECTED: Get operation succeeded in write-only mode (existence check)')
|
||||
console.log('Item found:', item ? 'Yes' : 'No')
|
||||
if (item) {
|
||||
console.log('Item ID:', item.id)
|
||||
console.log('Has metadata:', !!item.metadata)
|
||||
}
|
||||
} catch (error) {
|
||||
console.log('❌ UNEXPECTED: Get operation failed in write-only mode')
|
||||
console.log('Error:', error.message)
|
||||
}
|
||||
|
||||
// Test adding with existing ID to verify existence check
|
||||
console.log('\n🔄 Testing existence check during add operation...')
|
||||
try {
|
||||
const duplicateId = await brainy.add('This is duplicate data', { type: 'duplicate' }, { id: id1 })
|
||||
console.log('✅ Successfully handled duplicate ID:', duplicateId)
|
||||
} catch (error) {
|
||||
console.log('❌ Failed to handle duplicate ID:', error.message)
|
||||
}
|
||||
|
||||
// Test addVerb with writeOnlyMode to see placeholder noun behavior
|
||||
console.log('\n🔗 Testing addVerb with writeOnlyMode (placeholder nouns)...')
|
||||
try {
|
||||
const verbId = await brainy.addVerb('noun1', 'noun2', undefined, {
|
||||
type: 'relates_to',
|
||||
writeOnlyMode: true,
|
||||
metadata: { test: 'verb' }
|
||||
})
|
||||
console.log(`✅ Added verb with placeholder nouns, ID: ${verbId}`)
|
||||
} catch (error) {
|
||||
console.log('❌ Failed to add verb with writeOnlyMode:', error.message)
|
||||
}
|
||||
|
||||
// Switch back to normal mode to test search
|
||||
console.log('\n🔄 Switching back to normal mode...')
|
||||
brainy.setWriteOnly(false)
|
||||
|
||||
try {
|
||||
const results = await brainy.search('test', 5)
|
||||
console.log(`✅ Search succeeded in normal mode, found ${results.length} results`)
|
||||
|
||||
// Check if any results are placeholder nouns
|
||||
const placeholderResults = results.filter(r =>
|
||||
r.metadata &&
|
||||
typeof r.metadata === 'object' &&
|
||||
'writeOnlyMode' in r.metadata
|
||||
)
|
||||
|
||||
if (placeholderResults.length > 0) {
|
||||
console.log('⚠️ WARNING: Found placeholder nouns in search results:')
|
||||
placeholderResults.forEach(r => {
|
||||
console.log(` - ID: ${r.id}, metadata:`, r.metadata)
|
||||
})
|
||||
} else {
|
||||
console.log('✅ No placeholder nouns found in search results')
|
||||
}
|
||||
} catch (error) {
|
||||
console.log('❌ Search failed in normal mode:', error.message)
|
||||
}
|
||||
|
||||
console.log('\n📊 Summary of Implementation Status:')
|
||||
console.log('1. ✅ Search operations properly blocked in write-only mode with helpful error message')
|
||||
console.log('2. ✅ Existence checks (get operations) now work in write-only mode via storage')
|
||||
console.log('3. ✅ Add operations can check for existing data in write-only mode')
|
||||
console.log('4. ✅ Placeholder nouns are filtered out of search results')
|
||||
console.log('5. ✅ Mechanism implemented to update placeholder nouns when real data is found')
|
||||
console.log('6. ✅ Auto-configuration: Brainy detects write-only mode and skips index loading')
|
||||
console.log('\n🎉 All write-only mode issues have been resolved!')
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Error during reproduction:', error)
|
||||
}
|
||||
}
|
||||
|
||||
// Run the reproduction script
|
||||
reproduceWriteOnlyIssues().catch(console.error)
|
||||
2510
test-consumer/package-lock.json
generated
2510
test-consumer/package-lock.json
generated
File diff suppressed because it is too large
Load diff
|
|
@ -1,16 +0,0 @@
|
|||
{
|
||||
"name": "test-consumer",
|
||||
"version": "1.0.0",
|
||||
"description": "",
|
||||
"main": "index.js",
|
||||
"scripts": {
|
||||
"test": "echo \"Error: no test specified\" && exit 1"
|
||||
},
|
||||
"keywords": [],
|
||||
"author": "",
|
||||
"license": "ISC",
|
||||
"type": "commonjs",
|
||||
"dependencies": {
|
||||
"@soulcraft/brainy": "file:soulcraft-brainy-0.41.0.tgz"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
import { BrainyData } from '@soulcraft/brainy';
|
||||
|
||||
async function testBrainy() {
|
||||
try {
|
||||
console.log('Creating BrainyData instance...');
|
||||
const brainy = new BrainyData({
|
||||
storageType: 'memory',
|
||||
defaultEmbeddingOptions: { verbose: false }
|
||||
});
|
||||
|
||||
console.log('Initializing...');
|
||||
await brainy.init();
|
||||
|
||||
console.log('Adding data...');
|
||||
await brainy.add({ name: 'test', data: 'Test document' });
|
||||
|
||||
console.log('Searching...');
|
||||
const results = await brainy.search({ query: 'test' });
|
||||
console.log('Search results:', results);
|
||||
|
||||
console.log('Test completed successfully!');
|
||||
process.exit(0);
|
||||
} catch (error) {
|
||||
console.error('Test failed:', error);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
testBrainy();
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -1,125 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Simple test script to verify race condition fixes
|
||||
*/
|
||||
|
||||
const { BrainyData } = require('./dist/unified.js');
|
||||
|
||||
async function testRaceConditionFixes() {
|
||||
console.log('🧠 Testing Race Condition Fixes');
|
||||
console.log('=' .repeat(50));
|
||||
|
||||
const brainy = new BrainyData({
|
||||
dimensions: 512,
|
||||
maxConnections: 16,
|
||||
efConstruction: 200,
|
||||
storageType: 'memory'
|
||||
});
|
||||
|
||||
await brainy.init();
|
||||
|
||||
console.log('\n📊 Test 1: Write-Only Mode');
|
||||
console.log('-'.repeat(30));
|
||||
|
||||
try {
|
||||
// Test writeOnlyMode - should succeed even without existing nouns
|
||||
const verbId = await brainy.addVerb('user-writeonly-1', 'post-writeonly-1', null, {
|
||||
type: 'likes',
|
||||
writeOnlyMode: true,
|
||||
metadata: { test: 'writeOnlyMode' }
|
||||
});
|
||||
|
||||
console.log('✅ Write-only mode verb added successfully:', verbId);
|
||||
|
||||
// Verify the verb was created
|
||||
const verb = await brainy.getVerb(verbId);
|
||||
console.log('✅ Verb retrieved successfully:', !!verb);
|
||||
|
||||
} catch (error) {
|
||||
console.log('❌ Write-only mode test failed:', error.message);
|
||||
}
|
||||
|
||||
console.log('\n📊 Test 2: Auto-Create Missing Nouns');
|
||||
console.log('-'.repeat(30));
|
||||
|
||||
try {
|
||||
// Test autoCreateMissingNouns
|
||||
const verbId2 = await brainy.addVerb('user-auto-1', 'post-auto-1', null, {
|
||||
type: 'follows',
|
||||
autoCreateMissingNouns: true,
|
||||
metadata: { test: 'autoCreate' }
|
||||
});
|
||||
|
||||
console.log('✅ Auto-create verb added successfully:', verbId2);
|
||||
|
||||
// Verify the auto-created nouns exist
|
||||
const sourceNoun = await brainy.get('user-auto-1');
|
||||
const targetNoun = await brainy.get('post-auto-1');
|
||||
|
||||
console.log('✅ Auto-created source noun exists:', !!sourceNoun);
|
||||
console.log('✅ Auto-created target noun exists:', !!targetNoun);
|
||||
|
||||
} catch (error) {
|
||||
console.log('❌ Auto-create test failed:', error.message);
|
||||
}
|
||||
|
||||
console.log('\n📊 Test 3: Normal Mode (Should Fail)');
|
||||
console.log('-'.repeat(30));
|
||||
|
||||
try {
|
||||
// Test normal mode without existing nouns - should fail
|
||||
await brainy.addVerb('user-normal-1', 'post-normal-1', null, {
|
||||
type: 'mentions',
|
||||
metadata: { test: 'normalMode' }
|
||||
});
|
||||
|
||||
console.log('❌ Normal mode should have failed but succeeded');
|
||||
|
||||
} catch (error) {
|
||||
console.log('✅ Normal mode correctly failed:', error.message);
|
||||
}
|
||||
|
||||
console.log('\n📊 Test 4: Fallback Storage Lookup');
|
||||
console.log('-'.repeat(30));
|
||||
|
||||
try {
|
||||
// First add a noun normally
|
||||
const nounId = await brainy.add(Array.from({length: 512}, () => Math.random()), {
|
||||
type: 'user',
|
||||
name: 'Test User for Fallback'
|
||||
}, { id: 'fallback-test-user' });
|
||||
|
||||
console.log('✅ Noun added for fallback test:', nounId);
|
||||
|
||||
// Add another noun
|
||||
const targetId = await brainy.add(Array.from({length: 512}, () => Math.random()), {
|
||||
type: 'post',
|
||||
title: 'Test Post for Fallback'
|
||||
}, { id: 'fallback-test-post' });
|
||||
|
||||
console.log('✅ Target noun added for fallback test:', targetId);
|
||||
|
||||
// Now try to add a verb - this should work with fallback storage lookup
|
||||
const verbId3 = await brainy.addVerb('fallback-test-user', 'fallback-test-post', null, {
|
||||
type: 'created',
|
||||
metadata: { test: 'fallbackLookup' }
|
||||
});
|
||||
|
||||
console.log('✅ Fallback storage lookup verb added successfully:', verbId3);
|
||||
|
||||
} catch (error) {
|
||||
console.log('❌ Fallback storage lookup test failed:', error.message);
|
||||
}
|
||||
|
||||
await brainy.shutDown();
|
||||
|
||||
console.log('\n🏁 Race Condition Fixes Test Complete');
|
||||
console.log('=' .repeat(50));
|
||||
}
|
||||
|
||||
// Run the test
|
||||
testRaceConditionFixes().catch(error => {
|
||||
console.error('💥 Test failed:', error);
|
||||
process.exit(1);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue