- Introduced new documentation files under `docs/`:
- `model-bundling-analysis.md`: Provides detailed analysis of current, bundled, hybrid, and dynamic model loading approaches, including pros, cons, and recommendations.
- `model-management.md`: Explains how Brainy manages Universal Sentence Encoder models, including setup, usage, and troubleshooting.
- `optional-model-bundling.md`: Details the `@soulcraft/brainy-models` package for offline reliability with pre-bundled models.
- Added `src/utils/robustModelLoader.ts`:
- Implements enhanced model loading with retry mechanisms, timeout handling, fallback URLs, and optional local model bundling.
- Supports Node.js and browser environments with exponential backoff logic.
- Key Updates:
- **Hybrid Loading Strategy**: Recommended for balancing reliability and flexibility via hybrid online/offline mechanisms.
- **Enhanced Fallback Scenarios**: Robust loader improves network-dependent reliability for embedding workflows.
- **Offline Reliability Support**: Optional model bundling eliminates dependency on external services, supporting air-gapped and edge environments.
**Purpose**: Introduce a hybrid model loading approach with robust options for
- Added `cache-configuration.md` under `docs/guides`:
- Detailed multi-level cache system (hot, warm, cold) overview.
- Explained new adaptive tuning capabilities:
- Memory-based adjustments across Node.js, Browser, and Worker environments.
- Dynamic sizing for read-heavy/write-heavy workloads.
- Environment-specific configurations for optimal caching.
- Included best practices for large datasets, memory-constrained and read-only environments.
- Added monitoring and advanced manual tuning instructions.
- Modified `cacheManager.ts`:
- Introduced `environmentConfig` for tailored per-environment cache settings.
- Enhanced auto-tuning with support for dynamic memory detection and cache hit/miss ratio.
- Added fine-grained tuning for eviction thresholds, TTLs, and batch sizes based on workload characteristics.
- Improved adaptive tuning with async memory detection and detailed cache statistics tracking.
**Purpose**: Provide developers with detailed guidance and dynamic tools for optimizing Brainy's cache system, ensuring better performance across diverse environments and workloads.
- Created `service-identification.md` in `docs/guides`:
- Detailed guidelines on how services should identify themselves within Brainy.
- Documented two identification methods: default service initialization and operation-specific service naming.
- Included service name conventions and common examples (`github`, `reddit`, `default`).
- Described benefits of proper service identification:
- Enhanced statistics tracking and JSON field discovery by service.
- Provided best practices for consistent and descriptive service naming.
- Explained internal implementation details, such as `getServiceName` usage and statistic tracking.
**Purpose**: Help users properly identify services to enable statistics tracking, field discovery, and improved data management in Brainy.
- Added support for field-specific and prioritized searches in `brainyData`:
- Introduced `searchField` option to enable targeted field-level searches.
- Implemented `priorityFields` option for weighted vectorization and query relevance.
- Developed utilities in `jsonProcessing.ts` and `fieldNameTracking.ts`:
- `extractTextFromJson` for text extraction with customizable depth and field prioritization.
- `extractFieldFromJson` to target specific fields in JSON documents.
- `prepareJsonForVectorization` for optimized JSON vectorization.
- Enhanced management of field names and mappings:
- Integrated `trackFieldNames` to associate fields with their services.
- Supported cross-service consistency through `standardFieldMappings`.
- Updated documentation:
- Added detailed guides for JSON search enhancements and HNSW limitations.
- Extended usage examples in `README.md` and `json-search-test.js`.
- Verified improvements with comprehensive tests:
- Created unit and integration tests demonstrating search behavior improvements.
- Addressed previous TypeScript errors related to search parameters.
**Purpose**: Improve search accuracy and usability when working with complex JSON documents by enabling field-specific searches and enhancing contextual relevance.
- Deleted `CHANGES.md`, `CHANGES_SUMMARY.md`, `CONCURRENCY_ANALYSIS.md`, `CONCURRENCY_IMPLEMENTATION_SUMMARY.md`, and related developer documentation files.
- Removed redundant or legacy content no longer aligned with the current codebase and workflows.
- Updated repository to reflect streamlined documentation approach, reducing clutter and improving maintainability.
**Purpose**: Simplify and declutter repository by removing obsolete documentation files, ensuring it remains focused and relevant.
- Introduced `test-fallback-function.js` and `test-fallback-simple.js` to validate `executeInThread` fallback functionality with both named and anonymous compute-intensive functions.
- Added `test-tensorflow-textencoder.js` for TensorFlow.js and TextEncoder tests in a Node.js environment.
- Created `test-tensorflow-textencoder.html` for browser-based TensorFlow.js and TextEncoder tests.
- Implemented cross-environment test support in `cli-package/src/test-tensorflow-textencoder.ts` for CLI functionality.
- Enhanced `src/utils/embedding.ts`, `textEncoding.ts`, and `brainy-wrapper.js` to include updated global `TextEncoder` and `TextDecoder` utilities for compatibility and worker improvements.
- Standardized and expanded utility methods in `PlatformNode` for broader support, including `isFloat32Array` and `isTypedArray` checks.
- Updated Node.js requirement to `>= 24.4.0` across documentation and configuration files for compatibility improvements.
This update introduces comprehensive testing for fallback mechanisms, TensorFlow.js, and TextEncoder across multiple environments, ensuring robustness and compatibility.
- Updated version to `0.10.0` in `README.md`, `cli-package/package.json`, and `src/utils/version.ts`.
- Removed `CHANGELOG.md` from the root and `cli-package`, as GitHub auto-generates release notes.
- Refactored `brainy-wrapper.js` to simplify utility methods (`isFloat32Array` and `isTypedArray`) by moving them to instance-level definitions.
- Standardized Node.js version requirement to `>= 24.3.0` across documentation and configuration files.
- Improved `textEncoding.ts` by introducing a unified `Platform` class for cross-environment compatibility (Node.js and browser).
- Restructured scripts:
- Replaced outdated commands (e.g., `test-cli` -> `test:cli` and `test-all` -> `test:all`) for consistency.
- Removed changelog updates from `scripts/create-github-release.js` as part of simplification.
- Enhanced `patch-textencoder.js` to also patch `worker.js` for both `TextEncoder` and `TextDecoder`.
This release improves compatibility with Node.js 24, simplifies the deployment/changelog process, and refines text encoding and worker functionalities.
- Updated version to `0.9.30` in `package.json`, `cli-package/package.json`, `src/utils/version.ts`, and `README.md`.
- Synced `@soulcraft/brainy` dependency version to `0.9.30` for CLI package consistency.
- Modified npm version badge in `README.md` to reflect the new version.
- Added detailed publishing instructions in `docs/publishing-cli.md` to guide users on publishing both, main, or CLI packages.
- Introduced new npm scripts `publish:cli` and `build:cli` for streamlined package-specific operations.
This update ensures cohesive versioning, provides clearer publishing guidelines, and improves maintainability with new build/publish scripts.
- Added a new `@soulcraft/brainy-cli` package to provide a dedicated CLI interface for the Brainy database.
- Implemented a CLI wrapper script (`cli-wrapper.js`) to manage global package execution and version handling.
- Introduced `scripts/publish-cli.js` to automate synchronized publishing of the main package and CLI, ensuring version consistency.
- Created documentation for CLI usage and publishing workflow (`publishing-cli.md`).
- Updated `package.json` to include CLI-related build and deploy scripts.
- Adjusted `README.md` to document the CLI package, usage instructions, and build separation.
This feature introduces a standalone CLI package, improves deployment automation, and enhances documentation to align with the updated architecture.
