- Added model shard files (`group1-shard1of7` to `group1-shard7of7`) to support the Universal Sentence Encoder (USE).
- Organized shard files under `src/models/universal-sentence-encoder/` to ensure structured storage and scalability for embedding operations.
**Purpose**: Include necessary model shards for the Universal Sentence Encoder to enable reliable and efficient embedding generation.
- Added new documentation files:
- `COMPATIBILITY.md` detailing environment-specific compatibility and behavior (Node.js, Browser, Worker).
- `TESTING.md` providing instructions for verifying cache detection across environments.
- Created browser (`test-browser-cache-detection.html`) and worker (`test-worker-cache-detection.html`) test scripts to validate cache mechanisms.
- Removed fallback mechanisms for embedding:
- Updated `embedding.ts` to enforce strict usage of Universal Sentence Encoder (USE).
- Fallback methods (`generateFallbackVector`) and related logic have been removed.
- Errors are thrown when USE initialization or embedding fails, ensuring stricter reliability.
- Improved error handling:
- Standardized error throwing for all USE-related failures across single and batch embeddings.
- Logging updated to reflect critical embedding issues without allowing degraded operations.
**Purpose**: Improve documentation for environment compatibility and testing while enforcing consistent use of Universal Sentence Encoder for deterministic embeddings, removing unreliable fallback mechanisms.
- Created `service-identification.md` in `docs/guides`:
- Detailed guidelines on how services should identify themselves within Brainy.
- Documented two identification methods: default service initialization and operation-specific service naming.
- Included service name conventions and common examples (`github`, `reddit`, `default`).
- Described benefits of proper service identification:
- Enhanced statistics tracking and JSON field discovery by service.
- Provided best practices for consistent and descriptive service naming.
- Explained internal implementation details, such as `getServiceName` usage and statistic tracking.
**Purpose**: Help users properly identify services to enable statistics tracking, field discovery, and improved data management in Brainy.
- Enhanced `CacheManager` for better handling of large datasets, especially in `S3` or remote storage:
- Added `REMOTE_API` as a supported storage type.
- Improved cache sizing and batch tuning:
- Optimized memory usage based on environment (Browser, Node.js, Worker).
- Increased cache aggressiveness in read-only mode and for large datasets.
- Adjusted cache parameters dynamically for S3 or remote storage.
- Introduced `isReadOnly` and `isRemoteStorage` checks to refine tuning logic.
- Added new `cacheConfig` options:
- `autoTune`, `autoTuneInterval`, and mode-specific settings for read-only optimizations.
- Batch sizes, eviction thresholds, and TTLs tailored for operating modes.
- Enhanced documentation:
- Detailed performance-tuning guides and S3 examples in `README.md`.
- Included new configuration examples for large datasets in cloud storage.
- Improved extensibility:
- Unified cache and batch logic under storage type and mode-aware rules.
- Updated interfaces (`BrainyData`, `StorageFactory`) to include new cache settings.
- Verified enhancements with rigorous testing across multiple configurations.
**Purpose**: Improve caching strategy and query performance in complex cloud and on-premise environments with flexible, dynamic tuning.
- Added support for field-specific and prioritized searches in `brainyData`:
- Introduced `searchField` option to enable targeted field-level searches.
- Implemented `priorityFields` option for weighted vectorization and query relevance.
- Developed utilities in `jsonProcessing.ts` and `fieldNameTracking.ts`:
- `extractTextFromJson` for text extraction with customizable depth and field prioritization.
- `extractFieldFromJson` to target specific fields in JSON documents.
- `prepareJsonForVectorization` for optimized JSON vectorization.
- Enhanced management of field names and mappings:
- Integrated `trackFieldNames` to associate fields with their services.
- Supported cross-service consistency through `standardFieldMappings`.
- Updated documentation:
- Added detailed guides for JSON search enhancements and HNSW limitations.
- Extended usage examples in `README.md` and `json-search-test.js`.
- Verified improvements with comprehensive tests:
- Created unit and integration tests demonstrating search behavior improvements.
- Addressed previous TypeScript errors related to search parameters.
**Purpose**: Improve search accuracy and usability when working with complex JSON documents by enabling field-specific searches and enhancing contextual relevance.
- Added a new `CacheManager` class in `cacheManager.ts` to support three-level caching strategy:
- **Level 1**: Hot cache in RAM for most accessed nodes.
- **Level 2**: Warm cache using OPFS, Filesystem, or S3, depending on the environment.
- **Level 3**: Cold storage for longer-term data storage.
- Integrated features for dynamic tuning:
- Auto-detection of environment (Browser, Node.js, Worker) and memory availability.
- Parameter tuning for cache size, eviction thresholds, and TTL based on usage patterns.
- Enhanced support for:
- LRU-based eviction in hot cache.
- Batch-based operations with configurable batch sizes.
- Comprehensive logging and debug outputs for cache operations.
- Ensured robust fallback handling to manage storage in constrained environments.
- Improved extensibility for storage adapters (warm and cold storage detection and initialization).
**Purpose**: Optimize data access and storage across multiple environments with seamless scalability and dynamic parameter adjustments.
- Introduced `getNounTypes`, `getVerbTypes`, `getNounTypeMap`, and `getVerbTypeMap` utilities for managing noun and verb types at runtime.
- Added comprehensive unit tests (`type-utils.test.ts`) to ensure correctness of type utility functions.
- Created new example files (`type-utils-example.js`, `type-utils-example.ts`) to demonstrate the use of type utilities in JavaScript and TypeScript environments.
- Updated `README.md` with detailed documentation and usage examples for the new type utilities.
- Enhanced `index.ts` to export the new utility functions, making them accessible throughout the library.
**Purpose**: Facilitate easy access, validation, and manipulation of noun and verb types in client applications, providing better runtime type management.
- Updated `storage-adapter-coverage.test` to handle different adapter behaviors:
- Memory adapter: size remains 0 after restoration.
- FileSystem adapter: size matches restored items.
- Enhanced `delete` method in `brainyData.ts`:
- Added handling for content text passed instead of ID.
- Improved logging for better traceability during deletions.
- Modified restoration logic to skip index rebuilding during test scenarios:
- Clears index explicitly in test environments when performing a backup restoration for storage tests.
- Refined logic in `specialized-scenarios.test`:
- Validated database size changes after adding and deleting items.
- Enhanced clarity and debugging
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
- Updated error handling in `S3CompatibleStorage` to include checks for `NotFound` errors alongside `NoSuchKey` during lock operations.
- Ensured robustness in determining lock existence and managing exceptions related to missing keys.
**Purpose**: Improve resilience of the S3-compatible storage adapter by addressing additional error scenarios, ensuring accurate lock detection and stable operation.
- Introduced `writeOnly` mode in `BrainyData` allowing optimized data ingestion by skipping index loading and disabling search operations.
- Enhanced `BrainyDataConfig` to include `writeOnly` support and validate compatibility with `readOnly` mode.
- Implemented error handling for search attempts during `writeOnly` mode.
- Updated the README with detailed usage examples for database modes, including `writeOnly` and `readOnly`.
- Added `examples/write-only-mode.js` to demonstrate practical applications of the `writeOnly` mode.
**Purpose**: Optimize memory usage and startup time for data ingestion scenarios by enabling write-only mode with comprehensive documentation and examples.
- Reformatted export statements across `index.ts` and related modules for consistent style, improving code readability and maintainability.
- Updated graph types in `graphTypes.ts` to include additional standardized noun and verb categories, enhancing the flexibility of the type system for graph modeling.
- Replaced `Place` with `Location` and merged similar types (e.g., `Group` into `Collection`) to eliminate redundancy in entity definitions.
- Expanded verb types in `VerbType` to cover more comprehensive use cases, including social, temporal, and ownership relationships.
**Purpose**: Streamline code structure with consistent export formatting, simplify type definitions, and enhance the type system for broader modeling capabilities.
- Added `BrainyError` class to classify and handle errors with types like `TIMEOUT`, `NETWORK`, `STORAGE`, `NOT_FOUND`, and `RETRY_EXHAUSTED`. Includes static helper methods for error creation and retry determination.
- Introduced `operationUtils` with utility functions for timeout, retry logic, and exponential backoff. Implements features like `withTimeout`, `withRetry`, and a combined `withTimeoutAndRetry`.
- Updated `S3CompatibleStorage` to leverage new operation utilities for timeout and retry handling, including `StorageOperationExecutors` for clean operation execution.
- Enhanced `storageFactory` to pass `OperationConfig` for configurable timeout and retry behavior.
- Extended `BrainyData` to include timeout and retry policy configuration at initialization.
**Purpose**: Improve storage reliability by introducing configurable and reusable error-handling and operation utilities, reducing code duplication and enhancing maintainability.
- Modified `storageFactory` to ensure `FileSystemStorage` gracefully degrades to `MemoryStorage` in browser environments, with proper warnings added.
- Enhanced `opfsStorage` adapter to support recursive directory removal with the `recursive` option.
- Removed `test-fix.js` script, as it is no longer relevant with recent storage fixes and updates.
**Purpose**: Streamline and ensure cross-environment compatibility for `FileSystemStorage`, while removing outdated test artifacts for better maintainability.
- Introduced `CONCURRENCY_ANALYSIS.md` to outline identified concurrency issues, including statistics handling, index synchronization, and storage contention.
- Added `CONCURRENCY_IMPLEMENTATION_SUMMARY.md` to summarize concurrency improvements, such as distributed locking and change log mechanisms.
- Created `STORAGE_CONCURRENCY_ANALYSIS.md` to evaluate concurrency risks and applied solutions for different storage adapters (`S3CompatibleStorage`, `FileSystemStorage`, `OPFSStorage`, and `MemoryStorage`).
- Updated codebase with changes related to concurrency, including distributed locking, atomic updates, event-driven synchronization, and change log support.
- Refactored tests to verify behavior of new concurrency mechanisms, including robust error handling and cleanup functions.
**Purpose**: Provides comprehensive documentation and implementation details to ensure robust concurrency handling in multi-instance, high-throughput environments.
- Updated `test-results.json` to reflect the latest test outcomes.
- Ensures accuracy of recorded test results after recent changes.
**Purpose**: Keep test result records up-to-date for reliable tracking and reference.
- **Removed Files**:
- Deleted outdated statistics documentation files (`statistics.md`, `statistics-flush-solution.md`, `statistics-summary.md`) to clean up the repository and avoid confusion.
- **Added Standards**:
- Introduced `DOCUMENTATION_STANDARDS.md` to outline naming conventions and troubleshooting practices for more consistent and maintainable project documentation.
- **Tests**:
- Added a new test file `edge-cases.test.ts` to verify handling of edge cases, ensuring robust behavior against boundary values and invalid inputs.
**Purpose**: Cleans up deprecated documentation while introducing concrete standards for maintaining and updating documentation. Enhances test coverage for unusual or boundary inputs, improving overall system resilience.
- **Integration Tests**:
- Introduced `api-integration.test.ts` to validate API functionality:
- Verifies text insertion, vector embedding generation, and search operations.
- Confirms HNSW index correctness for vector similarity search.
- Ensures no dimensional mismatches in embeddings.
- **Test Server**:
- Added test server utilizing Express for endpoint simulation (`/insert` and `/search/text`).
- **Dependencies**:
- Introduced `express` and `node-fetch` as new dependencies for testing purposes.
- **Vitest Fix**:
- Updated `vitest.config.ts` to resolve the `process.memoryUsage` error by setting `logHeapUsage: false`.
- **Package Updates**:
- Modified `package-lock.json` to include newly added dependencies and updates.
**Purpose**: Guarantees the stability of core API endpoints and vector-related functionality, ensuring reliable behavior for end-to-end scenarios.
- **Core**:
- Added `check-database.js` to verify database status and validate search functionality.
- Created `fix-dimension-mismatch.js` to handle re-embedding of existing data to resolve dimension mismatch from 3 to 512.
- Improved test cases by updating vector operations to support 512 dimensions, replacing previously hardcoded dimensions.
- **Migration**:
- Developed `DIMENSION_MISMATCH_SUMMARY.md`, detailing the root cause, solution, and preventive strategies for dimension mismatch issues.
- Added `production-migration-guide.md` for structured production migration with detailed steps on re-embedding strategies, batching, and error handling.
- **Tests**:
- Enhanced test coverage with 512-dimensional vector validation.
- Introduced helper functions for consistent vector testing behavior and streamlined search test cases.
- **Documentation**:
- Updated project documentation to highlight the resolution process for dimension mismatches, emphasizing preventive mechanisms such as auto-migration and version tracking.
**Purpose**: Address critical dimension mismatch issues caused by embedding changes, restore functionality, and provide a roadmap for robust prevention strategies and migration processes.
- **Storage**: Replaced all instances of `HNSWNoun_internal` and `Verb` type aliases with their original equivalents (`HNSWNoun` and `GraphVerb`) in `MemoryStorage` adapter. Simplified method definitions and internal logic by directly using the original types.
- **Code Cleanup**:
- Removed unused type alias declarations to reduce code clutter and improve readability.
- Adjusted method parameters and return types accordingly to maintain consistency.
**Purpose**: Simplify the codebase by removing redundant type aliasing, ensuring consistency and better readability across storage adapter logic. Reduces potential confusion and streamlines type usage.
- **Examples**: Added a new `flush-statistics-example.js` script to demonstrate the usage of the `flushStatistics` method for ensuring updated statistics after data insertion.
- **Core**:
- Implemented `flushStatistics` in the `BrainyData` class to allow immediate flushing of statistics to storage.
- Updated `BaseStorageAdapter` with `flushStatisticsToStorage` to support flushing cached statistics.
- Modified the `shutDown` method to ensure statistics are flushed before database shutdown.
- **Documentation**: Added `statistics-flush-solution.md` to explain the batch update mechanism, the issue with delayed statistics updates, and how to manually flush statistics in storage.
**Purpose**: Provide users with the ability to manually flush statistics for real-time accuracy, particularly useful for systems relying on immediate updates. Improved documentation and examples to guide developers in implementing and using this functionality effectively.
- **Core**: Improved verb creation logic by adding `createdAt`, `updatedAt`, and `createdBy` attributes. These fields include timestamped metadata (`seconds`, `nanoseconds`) and source augmentation/service information for better tracking.
- **Storage**: Refactored `BaseStorage` methods to utilize internal variants (e.g., `saveVerb_internal`, `getNoun_internal`). Added support for new verb attributes while maintaining backward compatibility with existing data structures.
- **Tests**:
- Updated `s3-storage.test.ts` and `opfs-storage.test.ts` to validate changes in verb attributes such as timestamps and augmentation metadata.
- Added assertions for `createdAt`, `updatedAt`, and `createdBy` fields in test cases.
- **Cleanup**: Replaced ambiguous type aliases like `Edge` and `HNSWNode` with clearer equivalents (`Verb` and `HNSWNoun_internal`) for consistency across storage adapters.
**Purpose**: Enhance metadata tracking and standardize attribute handling across storage and core modules to ensure accurate and consistent data throughout the system.
- **Code Cleanup**: Removed unnecessary trailing whitespaces across `src/brainyData.ts`. Adjusted formatting for inline object spreads to maintain a consistent coding style.
- **Purpose**: Enhance code readability and ensure adherence to formatting standards without altering functionality.
- **Tests**: Added new `statistics-storage.test.ts` to validate statistics storage functionality across scenarios including saving, retrieving, time-based partitioning, and backward compatibility. Ensured tests dynamically handle missing environment variables by skipping S3-related tests when credentials are unavailable.
- **Docs**: Enhanced `statistics.md` with detailed explanations of scalability improvements, including adaptive flush timing, batched updates, and time-based partitioning. Improved readability and structure.
- **Storage**: Updated all storage adapters to integrate time-based partitioning and maintain backward compatibility with legacy statistics storage formats.
- **Dependencies**: Added `dotenv` to support environmental variable management for storage adapter tests.
**Purpose**: Strengthen system reliability by adding comprehensive test coverage for statistics storage, improve scalability documentation, and ensure consistency across storage adapters with robust implementations.
- **Core**: Introduced a `getCurrentAugmentation` method for detecting active augmentation names. Updated metadata handling to include `createdBy`, `createdAt`, and `updatedAt` attributes for improved tracking.
- **Storage**: Added support for service-based statistics tracking with new methods such as `incrementStatistic`, `decrementStatistic`, and `updateHnswIndexSize`. Implemented persistence for statistics in storage adapters.
- **Statistics**: Enhanced `getStatistics` functionality to provide service-specific breakdowns and support filtering by services. Improved noun, verb, and metadata tracking mechanisms.
- **Search**: Added `service` option to filter results during searches for nouns, verbs, and metadata, ensuring accurate service-based query results.
- **Refactor**: Simplified search logic by integrating HNSW index filtering for better performance when retrieving service-specific results.
- **Tests**: Added comprehensive test coverage for service-level statistics and filtering by service.
**Purpose**: Improve service-level data tracking and analytics while enhancing functionality for filtering and maintaining metadata accuracy to support detailed insights for diverse use cases.
- **Removal**: Deleted `FileSystemStorage` and `OPFSStorage` adapters from the `/storage` directory.
- **Refactor**: Simplifies the codebase by eliminating unused, redundant, or outdated storage implementations.
- **Impact**: Ensures maintainability by focusing on actively supported storage solutions.
**Purpose**: Streamline the project by removing deprecated or unused storage adapters, reducing complexity and maintenance overhead.
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
- **New Method**:
- Introduced a new `getStatistics` method in `brainyData.ts` to retrieve key database metrics:
- Counts for nouns, verbs, metadata entries, and HNSW index size.
- **Error Handling**:
- Added try-catch blocks to ensure robust error management when fetching metadata or logging failures.
- **Purpose**:
- Enhances observability of the database state, providing valuable insights for diagnostics and monitoring.
- Added `autoCreateMissingNouns` and `missingNounMetadata` options to the `addVerb` method in `brainyData.ts`, enabling automatic creation of missing source or target nouns.
- Improved error handling and logging for auto-creation failures, ensuring better feedback during runtime.
- Reformatted existing code for storage options validation to improve readability and maintain consistency.
**Purpose**: Simplify the addition of relationships by automating the process for non-existing nouns and enhance developer experience with better error handling and logging.
- **Removal of `generate-version.js`**:
- Deleted `scripts/generate-version.js` as it is no longer required for version handling.
- Removed associated output files, including `src/utils/version.ts`.
- **Prebuild and Version Process Updates**:
- Updated `prebuild` and `version` scripts in `package.json` to reflect the new workflow:
- Replaced `generate-version.js` with simplified echo commands for version updates and build steps.
- Adjusted deployment scripts (`deploy:cli` and `deploy:web-service`) to align with the removal of the script.
- **Documentation and Config Updates**:
- Cleaned up `README.md` by removing unused version badge references.
- Updated `cli-package/package.json` to specify a version range (`^0.15.0`) for `@soulcraft/brainy`.
**Purpose**: This refactor simplifies the version handling process by removing outdated scripts, reducing maintenance overhead, and aligning workflows with project conventions. Improves clarity and consistency in both configuration and documentation.
- Updated `VERSION` in `version.ts` to `0.15.0`.
- Updated `npm` badge in `README.md` to reflect version `0.15.0`.
- Updated `version` field and dependency reference in `cli-package/package.json` to `0.15.0`.
**Purpose**: Prepare for release by synchronizing version across source files, documentation, and package dependencies.
- Updated all imports and exports in `src/index.ts` and `src/storage/storageFactory.ts` to use a consistent formatting style.
- Enhanced readability by standardizing spacing, alignment, and indentation across the codebase.
- Applied consistent import/export styling for structured comments and multi-line objects.
**Purpose**: Improve code maintainability and readability by enforcing uniform formatting, aligning imports and exports for better clarity.
- Introduced comprehensive mock for the Origin Private File System (OPFS) in `tests/mocks/opfs-mock.ts`, simulating environment for detailed storage testing.
- Added `tests/opfs-storage.test.ts`, containing extensive test cases for `OPFSStorage` operations including metadata, nouns, verbs, and storage status.
- Improved S3 mock implementation in `tests/mocks/s3-mock.ts` with better object persistence, validation, and logging to emulate real S3 behavior.
- Resolved issues related to metadata, nouns, verbs, and storage usage inconsistencies in mock storage adapters.
- Enhanced logging and error handling to aid in debugging and test reliability.
Purpose: Improve test completeness and reliability by introducing detailed mocks and extended test cases for S3 and OPFS storage systems.
- Added `getFile` method to `FileSystemHandle` interface for enhanced TypeScript compatibility with File System Access API.
- Improved maintainability by reformatting and aligning imports in `brainyData.ts`.
- Standardized spacing and indentation across structured comments and interface fields.
Purpose: Improve TypeScript support for file system operations and enhance code readability with consistent formatting and import management.
- Added `BaseStorage` abstract class to provide common functionality for storage adapters.
- Implemented `FileSystemStorage` adapter for Node.js environments, supporting persistent file-based storage.
- Added directory structure management methods to `FileSystemStorage` (`ensureDirectoryExists`, `clear`, etc.).
- Implemented `MemoryStorage` adapter for in-memory storage suitable for non-persistent environments.
- Supported CRUD operations for nouns, verbs, and metadata in both `FileSystemStorage` and `MemoryStorage`.
- Included utility methods like `mapToObject` for serialization and deserialization of complex data structures.
Purpose: Provide extensible, reusable storage solutions with both persistent and non-persistent options for diverse application needs.
- Updated `VERSION` in `version.ts` to `0.14.0`.
- Updated `npm` badge in `README.md` to reflect version `0.14.0`.
- Updated `version` field and dependency reference in `cli-package/package.json` to `0.14.0`.
Purpose: Prepare for release by synchronizing version across source files and documentation.
- Added a check to avoid appending `ROOT_DIR` if the `rootDir` already ends with it.
- Ensures proper directory paths setup without redundant path segments.
- Refactor embedding functions to support customizable verbosity through new helper methods (`getDefaultEmbeddingFunction`, `getDefaultBatchEmbeddingFunction`).
- Ensure metadata initialization with default values when null during search results.
- **Logging Cleanup**:
- Removed redundant `console.log` statements in `brainyData.ts` and `embedding.ts` related to Universal Sentence Encoder initialization and load function detection.
- Replaced detailed logs with concise comments to streamline debugging and reduce noisy outputs.
- **Purpose**:
- Improves code readability, reduces runtime logging noise, and aligns logging verbosity with the project's streamlined debugging practices.
- **Core Improvements**:
- Refactored logging functions into a unified `logger` method for consistent output across the library.
- Enabled the `forceMemoryStorage` option in `BrainyData` initialization for improved storage flexibility in tests and specific use cases.
- **TensorFlow.js and Environment Updates**:
- Clarified the dependency structure in `README.md` to emphasize bundled dependencies and remove legacy peer dependency instructions.
- Simplified and reformatted environment detection logic for better maintainability and readability.
- **Testing Enhancements**:
- Added `tests/package-size-limit.test.ts` to monitor and validate npm package size against defined thresholds.
- Updated `tests/environment.node.test.ts` and core tests to leverage `forceMemoryStorage` for better test setup standardization.
- Improved test isolation with expanded `globalThis` utility definitions and cleanup logic.
- **Documentation**:
- Added detailed best practices for debugging and organizing tests in `DEVELOPERS.md`.
- Removed outdated installation hints from `package.json` and streamlined scripts by including `test:size` for package size validation.
**Purpose**: These changes unify core logging mechanisms, expand configurability of storage options, and improve testing reliability and coverage. Documentation and clarity are enhanced to align with updated functionality and best practices.
- Updated the version in `README.md`, `package.json`, and `version.ts` to `0.12.0` to reflect the new release.
- Adjusted dependency reference for `@soulcraft/brainy` in `cli-package/package.json` to use version `0.12.0`.
**Purpose**: Prepares for the 0.12.0 release, ensuring all version references are consistent across documentation and configuration files.
- **Vector Handling Updates**:
- Added a `dimensions` property to `BrainyDataConfig` for specifying vector dimensions.
- Introduced validation for vector dimensions during database creation and insertion to ensure consistency.
- Enhanced error handling and logging for dimension mismatches.
- **Model Loading Improvements**:
- Implemented retry logic for Universal Sentence Encoder model loading to handle network instability and JSON parsing errors gracefully.
- Improved logging and debugging support for failures during model initialization and embedding operations.
- **Compatibility Enhancements**:
- Updated polyfills to support TensorFlow.js compatibility across diverse server environments (Node.js, serverless, etc.).
- Introduced and refactored global `TextEncoder`/`TextDecoder` definitions for seamless operation in non-browser environments.
- Simplified TensorFlow.js backend setup with streamlined imports and logging for GPU/WebGL fallback.
- **Purpose**:
- These updates improve BrainyData's robustness, enforce correct vector usage, and extend compatibility with varied runtime environments. The changes enhance the usability, reliability, and cross-platform readiness of core functionalities.
- Simplified explanation of server functionality (`WebSocket` and `REST`) to emphasize its exclusion from the main Brainy package for maintaining lightweight browser bundles.
- Removed detailed Node.js-specific setup examples, configuration options, and cloud-wrapper deployment instructions.
- Retained relevant core functionality usage details for broader accessibility.
This update aligns the documentation with recent repository changes, focusing on clarity, maintainability, and providing guidance only on actively supported features.
