- **Storage**: Removed redundant storage implementation files from `/storage` directory while retaining necessary `baseStorage.ts`. Verified no impact on functionality with passing tests.
- **Documentation**: Consolidated multiple overlapping `.md` files into a single `statistics.md` file. Added structured sections for easier reference. Applied markdown naming conventions for clarity.
- **Project**: Introduced `CHANGES_SUMMARY.md` and `MARKDOWN_CONVENTIONS.md` to document changes and naming guidelines, improving project organization. Updated `demo.md` file naming for consistency.
**Purpose**: Simplify the codebase by removing duplicates, enhance maintainability through consolidated and structured documentation, and improve developer experience with clear guidelines and better organization.
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
- **New Method**:
- Introduced a new `getStatistics` method in `brainyData.ts` to retrieve key database metrics:
- Counts for nouns, verbs, metadata entries, and HNSW index size.
- **Error Handling**:
- Added try-catch blocks to ensure robust error management when fetching metadata or logging failures.
- **Purpose**:
- Enhances observability of the database state, providing valuable insights for diagnostics and monitoring.
- Updated version to `0.18.0` in `cli-package/package.json` and `package-lock.json`.
- Upgraded `@soulcraft/brainy` dependency to `^0.17.0` for compatibility with recent changes.
- Synced package dependencies and resolved lockfile integrity for deployment.
**Purpose**: Prepare for release by synchronizing versioning across the CLI package and updating dependencies for compatibility.
- Added `autoCreateMissingNouns` and `missingNounMetadata` options to the `addVerb` method in `brainyData.ts`, enabling automatic creation of missing source or target nouns.
- Improved error handling and logging for auto-creation failures, ensuring better feedback during runtime.
- Reformatted existing code for storage options validation to improve readability and maintain consistency.
**Purpose**: Simplify the addition of relationships by automating the process for non-existing nouns and enhance developer experience with better error handling and logging.
- **Version Updates**:
- Updated `web-service-package/package.json` and `package-lock.json` from `0.15.0` to `0.16.0`.
- Updated `cli-package/package.json` and `package-lock.json` from `0.16.0` to `0.17.0`.
- **Purpose**:
- Prepares both packages for their respective new releases, ensuring version consistency across project dependencies.
- **Changes**:
- Eliminated the redundant `@ts-expect-error` comment for the `omelette` import in `cli.ts`.
- **Purpose**:
- Simplifies the code by removing an outdated comment that is no longer needed, ensuring cleaner and more readable source files.
- **Version Updates**:
- Updated `cli-package/package.json` and `package-lock.json` from `0.15.0` to `0.16.0`.
- **NPM Script Enhancements**:
- Added version-related scripts (`version`, `version:patch`, `version:minor`, `version:major`) to `web-service-package` and `cli-package`.
- Introduced `deploy` and `dry-run` scripts to streamline build and deployment workflows.
- **Script Cleanup**:
- Removed redundant `build:cli`, `deploy:cli`, and `deploy:web-service` scripts from the root `package.json`.
**Purpose**: Prepares for release 0.16.0 while introducing reusable and optimized build/release scripts, improving the development and deployment experience.
- **Removal of `generate-version.js`**:
- Deleted `scripts/generate-version.js` as it is no longer required for version handling.
- Removed associated output files, including `src/utils/version.ts`.
- **Prebuild and Version Process Updates**:
- Updated `prebuild` and `version` scripts in `package.json` to reflect the new workflow:
- Replaced `generate-version.js` with simplified echo commands for version updates and build steps.
- Adjusted deployment scripts (`deploy:cli` and `deploy:web-service`) to align with the removal of the script.
- **Documentation and Config Updates**:
- Cleaned up `README.md` by removing unused version badge references.
- Updated `cli-package/package.json` to specify a version range (`^0.15.0`) for `@soulcraft/brainy`.
**Purpose**: This refactor simplifies the version handling process by removing outdated scripts, reducing maintenance overhead, and aligning workflows with project conventions. Improves clarity and consistency in both configuration and documentation.
- **Cloud Storage Integration Testing**:
- Added `cloud-storage.test.ts` to validate cloud storage configurations (AWS S3, Cloudflare R2, Google Cloud Storage) and local storage fallback behavior:
- Includes tests for environment variable priority and configuration detection.
- Ensures proper override with `FORCE_LOCAL_STORAGE` when specified.
- **Package Settings Updates**:
- Introduced `web-service-package/package.json` with configurations for building, running, testing, and deployment:
- Added NPM scripts for building (`npm run build`), development (`npm run dev`), and testing (`npm run test:cloud`).
- Configured dependencies and devDependencies for web service functionality and cloud integration.
- **Purpose**:
- This update ensures comprehensive cloud storage testing and provides structured project configurations for seamless development and deployment workflows.
- **New Web Service Package**:
- Introduced `@soulcraft/brainy-web-service`, a REST API wrapper for the Brainy vector graph database.
- Added documentation and features to support secure, read-only search and retrieval operations.
- **Deployment Support**:
- Included comprehensive deployment instructions in the `web-service-package/README.md`:
- Options for Docker, serverless platforms, and cloud providers (AWS, GCP, Azure, Cloudflare).
- Example configurations for systemd, Nginx, and Docker Compose.
- **Scripts and Documentation Updates**:
- Added `deploy:web-service` script to `package.json` for streamlined build and publishing.
- Enhanced `README.md` to reflect the introduction of the web service package and its capabilities.
- **Purpose**:
- This update extends Brainy’s functionality by providing a production-ready, easy-to-deploy REST API for search operations. It ensures flexibility for diverse deployment scenarios while maintaining security and high performance.
- Updated `VERSION` in `version.ts` to `0.15.0`.
- Updated `npm` badge in `README.md` to reflect version `0.15.0`.
- Updated `version` field and dependency reference in `cli-package/package.json` to `0.15.0`.
**Purpose**: Prepare for release by synchronizing version across source files, documentation, and package dependencies.
- Updated all imports and exports in `src/index.ts` and `src/storage/storageFactory.ts` to use a consistent formatting style.
- Enhanced readability by standardizing spacing, alignment, and indentation across the codebase.
- Applied consistent import/export styling for structured comments and multi-line objects.
**Purpose**: Improve code maintainability and readability by enforcing uniform formatting, aligning imports and exports for better clarity.
- Introduced comprehensive mock for the Origin Private File System (OPFS) in `tests/mocks/opfs-mock.ts`, simulating environment for detailed storage testing.
- Added `tests/opfs-storage.test.ts`, containing extensive test cases for `OPFSStorage` operations including metadata, nouns, verbs, and storage status.
- Improved S3 mock implementation in `tests/mocks/s3-mock.ts` with better object persistence, validation, and logging to emulate real S3 behavior.
- Resolved issues related to metadata, nouns, verbs, and storage usage inconsistencies in mock storage adapters.
- Enhanced logging and error handling to aid in debugging and test reliability.
Purpose: Improve test completeness and reliability by introducing detailed mocks and extended test cases for S3 and OPFS storage systems.
- Added `getFile` method to `FileSystemHandle` interface for enhanced TypeScript compatibility with File System Access API.
- Improved maintainability by reformatting and aligning imports in `brainyData.ts`.
- Standardized spacing and indentation across structured comments and interface fields.
Purpose: Improve TypeScript support for file system operations and enhance code readability with consistent formatting and import management.
- Documented the use of a simplified, unified storage structure for all noun types.
- Enhanced storage system explanation to reflect recent architectural improvements.
Purpose: Improve developer understanding of the storage system by updating documentation with the latest features and design changes.
- Removed outdated test files: `test-tensorflow-import.cjs`, `test-tensorflow-import.js`, and `verify-package-size.js`.
- Introduced `tests/package-size-breakdown.test.ts` to analyze included npm package files and validate their sizes.
- Added comprehensive tests for storage adapters (`MemoryStorage`, `FileSystemStorage`, `OPFSStorage`, etc.) in `tests/storage-adapters.test.ts`.
- Improved OPFS storage mocking and test coverage for browser and Node.js environments.
- Introduced environment detection tests to ensure correct storage adapter selection.
- Created `STORAGE_TESTING.md` to document storage architecture, test coverage, and guidelines for further improvements.
Purpose: Modernize testing infrastructure and enhance storage system reliability through better test coverage and documentation.
- Added `BaseStorage` abstract class to provide common functionality for storage adapters.
- Implemented `FileSystemStorage` adapter for Node.js environments, supporting persistent file-based storage.
- Added directory structure management methods to `FileSystemStorage` (`ensureDirectoryExists`, `clear`, etc.).
- Implemented `MemoryStorage` adapter for in-memory storage suitable for non-persistent environments.
- Supported CRUD operations for nouns, verbs, and metadata in both `FileSystemStorage` and `MemoryStorage`.
- Included utility methods like `mapToObject` for serialization and deserialization of complex data structures.
Purpose: Provide extensible, reusable storage solutions with both persistent and non-persistent options for diverse application needs.
- Documented various WebSocket augmentation types (`IWebSocketSupport`, `IWebSocketSenseAugmentation`, etc.) to guide developers in extending augmentations with WebSocket capabilities.
- Added code example showcasing the creation of a WebSocket-enabled sense augmentation (`mySenseAug`) using `addWebSocketSupport`.
- Explained the purpose of combining base augmentation interfaces with `IWebSocketSupport` for type safety and autocompletion.
Purpose: Enhance developer documentation by providing clear guidance and examples for integrating WebSocket capabilities into augmentations.
- Updated `VERSION` in `version.ts` to `0.14.0`.
- Updated `npm` badge in `README.md` to reflect version `0.14.0`.
- Updated `version` field and dependency reference in `cli-package/package.json` to `0.14.0`.
Purpose: Prepare for release by synchronizing version across source files and documentation.
- Introduced `SearchResult` interface to standardize search result handling and improve code readability.
- Added TypeScript type annotations for command arguments and options in CLI methods to enforce stricter type safety.
- Updated `.version()` for CLI to align with best practices by removing redundant description parameter.
- Enhanced search logic with default parameter handling for limits and improved output formatting for results.
- Adjusted `tsconfig.json` to disable `noImplicitAny` while retaining strict typing practices.
- Streamlined CLI processes by parsing command-line arguments explicitly with `program.parse(process.argv)`.
Purpose: Improve developer productivity with better type safety, clearer interfaces, and refined CLI functionality.
- Added a check to avoid appending `ROOT_DIR` if the `rootDir` already ends with it.
- Ensures proper directory paths setup without redundant path segments.
- Refactor embedding functions to support customizable verbosity through new helper methods (`getDefaultEmbeddingFunction`, `getDefaultBatchEmbeddingFunction`).
- Ensure metadata initialization with default values when null during search results.
- **Logging Cleanup**:
- Removed redundant `console.log` statements in `brainyData.ts` and `embedding.ts` related to Universal Sentence Encoder initialization and load function detection.
- Replaced detailed logs with concise comments to streamline debugging and reduce noisy outputs.
- **Purpose**:
- Improves code readability, reduces runtime logging noise, and aligns logging verbosity with the project's streamlined debugging practices.
- **Core Improvements**:
- Refactored logging functions into a unified `logger` method for consistent output across the library.
- Enabled the `forceMemoryStorage` option in `BrainyData` initialization for improved storage flexibility in tests and specific use cases.
- **TensorFlow.js and Environment Updates**:
- Clarified the dependency structure in `README.md` to emphasize bundled dependencies and remove legacy peer dependency instructions.
- Simplified and reformatted environment detection logic for better maintainability and readability.
- **Testing Enhancements**:
- Added `tests/package-size-limit.test.ts` to monitor and validate npm package size against defined thresholds.
- Updated `tests/environment.node.test.ts` and core tests to leverage `forceMemoryStorage` for better test setup standardization.
- Improved test isolation with expanded `globalThis` utility definitions and cleanup logic.
- **Documentation**:
- Added detailed best practices for debugging and organizing tests in `DEVELOPERS.md`.
- Removed outdated installation hints from `package.json` and streamlined scripts by including `test:size` for package size validation.
**Purpose**: These changes unify core logging mechanisms, expand configurability of storage options, and improve testing reliability and coverage. Documentation and clarity are enhanced to align with updated functionality and best practices.
- **CLI Documentation Updates**:
- Updated `README.md` to reflect the separation of CLI functionality into a new package: `@soulcraft/brainy-cli`.
- Replaced inline CLI imports with instructions for installing and using `@soulcraft/brainy-cli`.
- Clarified notes on keeping the main package lightweight by decoupling CLI functionality.
- **Script and Command Adjustments**:
- Modified build instructions in `README.demo.md` to use `npm run build && npm run build:browser`.
- Updated demo scripts in `package.json` to point to `/demo/index.html` instead of `/index.html`.
- Revised style-check script in `CONTRIBUTING.md` to use `npm run check:style`.
- **Issues and Contribution Guidelines**:
- Improved issue reporting section in `DEVELOPERS.md` with clearer guidelines and examples for bug and feature requests.
**Purpose**: These changes enhance documentation accuracy, streamline developer guidance, and ensure commands/scripts are up-to-date with current usage patterns.
- Updated `version` of `@soulcraft/brainy` from `0.11.0` to `0.12.0` in `cli-package/package-lock.json`.
- Adjusted resolved package metadata (e.g., `integrity` values) to match the new release version.
**Purpose**: Ensures `package-lock.json` reflects the updated dependency version for consistency with the 0.12.0 release.
- Updated the version in `README.md`, `package.json`, and `version.ts` to `0.12.0` to reflect the new release.
- Adjusted dependency reference for `@soulcraft/brainy` in `cli-package/package.json` to use version `0.12.0`.
**Purpose**: Prepares for the 0.12.0 release, ensuring all version references are consistent across documentation and configuration files.
- **Vector Handling Updates**:
- Added a `dimensions` property to `BrainyDataConfig` for specifying vector dimensions.
- Introduced validation for vector dimensions during database creation and insertion to ensure consistency.
- Enhanced error handling and logging for dimension mismatches.
- **Model Loading Improvements**:
- Implemented retry logic for Universal Sentence Encoder model loading to handle network instability and JSON parsing errors gracefully.
- Improved logging and debugging support for failures during model initialization and embedding operations.
- **Compatibility Enhancements**:
- Updated polyfills to support TensorFlow.js compatibility across diverse server environments (Node.js, serverless, etc.).
- Introduced and refactored global `TextEncoder`/`TextDecoder` definitions for seamless operation in non-browser environments.
- Simplified TensorFlow.js backend setup with streamlined imports and logging for GPU/WebGL fallback.
- **Purpose**:
- These updates improve BrainyData's robustness, enforce correct vector usage, and extend compatibility with varied runtime environments. The changes enhance the usability, reliability, and cross-platform readiness of core functionalities.
- **Test Enhancements**:
- Refactored test setup in `core.test.ts` for better isolation and clarity:
- Added explicit `.clear()` calls to ensure clean state between tests.
- Replaced old vector addition logic with simplified data insertion methods.
- Updated search operations to reflect current functionality and debug-friendly output.
- Introduced the `@vitest-environment jsdom` annotation in `environment.browser.test.ts` for accurate browser environment emulation.
- **Configuration Updates**:
- Enhanced `vitest.config.ts`:
- Introduced custom `reporters` for cleaner and focused test result presentation.
- Expanded console log filtering with additional patterns for reducing noise from TensorFlow.js and setup processes.
- **Purpose**:
- These updates improve test clarity, consistency, and robustness while streamlining the configuration to minimize distractions in test outputs.
- **Configuration Updates**:
- Set default test environment to `node` for consistency.
- Reduced noise in test outputs by configuring options such as `silent`, `bail`, `hideSkippedTests`, `printConsoleTrace`, and `onConsoleLog`.
- Disabled test coverage reports by default for reduced verbosity.
- Added functionality to filter out noisy console logs more aggressively.
- **Purpose**:
- Improves the clarity and readability of test output, reduces unnecessary verbosity, and ensures consistent behavior in a `node` environment. Encourages focus on debugging failed tests by hiding irrelevant details.
- **Added Dependencies**:
- Introduced `@vitest/coverage-v8` for enhanced test coverage reporting.
- Added various dev dependencies such as `@ampproject/remapping`, `@babel/parser`, `@babel/types`, and `ast-v8-to-istanbul` to support improved source mapping and code coverage analysis.
- Included modules such as `html-escaper`, `path-scurry`, and `istanbul-lib-coverage` to align with updated testing and utility requirements.
- **Purpose**:
- This update ensures compatibility with modern testing tools, facilitates advanced debugging, and supports enhanced test coverage capabilities. It also updates metadata for package resolutions.
- **Test Improvements**:
- Introduced data-clearing steps (`.clear()`) across critical test cases for ensuring better test isolation and preventing state leakage.
- Extended support for overriding global utilities (`testUtils`) and added fallback behaviors for test vector creation.
- **Configuration Updates**:
- Added support for `distanceFunction` as an alternative to `metric` in vector operations for consistency.
- Adjusted and unified asynchronous `timeout` handling across test suites for predictability.
- **Purpose**:
- These updates improve reliability, maintainability, and clarity in test cases while ensuring compatibility across diverse test environments.
- **New Tests Added**:
- Introduced multiple test suites covering core functionalities (`core.test.ts`), vector operations (`vector-operations.test.ts`), Node.js environment (`environment.node.test.ts`), browser setup (`environment.browser.test.ts`), and TensorFlow.js-specific behaviors (`tensorflow-patch.test.ts`).
- Added performance, scalability, and error-handling tests to ensure robust validation of vector addition, search, and text embedding functionalities.
- Introduced setup utilities (`tests/setup.ts`) and standardized test utilities for creating predictable test cases.
- **Configuration**:
- Created `vitest.config.ts` for custom test configurations, including support for modern test environments (`jsdom`, `happy-dom`) and extended timeouts for asynchronous operations.
- **Validation**:
- Includes compatibility checks for TensorFlow.js imports and ensures proper handling of `TextEncoder`/`TextDecoder` in Node.js environments.
This commit significantly enhances the testing coverage and structure, ensuring Brainy functionality is robust, cross-platform, and aligned with evolving reliability standards.
- Removed extraneous semicolons across `cli-wrapper-example.js` for cleaner, consistent code style.
- Adjusted indentation and spacing for improved readability.
- Updated console statements to be consistent, ensuring uniform logging behavior across the example.
This change aligns the example file with the project's coding standards, improving code clarity and maintainability.
- **Documentation Updates**:
- Simplified server functionality details in `README.md`, removing cloud-specific deployment sections and focusing on core features for broader usability.
- **Shim Enhancements**:
- Improved `util` shim in `rollup.config.js` to include `TextEncoder`/`TextDecoder` support and enhanced `types` checks (e.g., `isFloat32Array`, `isUint8Array`).
- **Dependency Updates**:
- Added new dev dependencies (`@vitest/ui`, `jsdom`, `happy-dom`).
- Updated `package-lock.json` with additional dependency resolutions and metadata for better compatibility and maintainability.
This commit aligns documentation with current project focus, enhances cross-platform support via refined shims, and ensures updated dependencies for development and testing environments.
- Removed the `test-tensorflow-textencoder` command from `cli.ts` as part of deprecating redundant tests and unused CLI functionality.
- Simplified `TextEncoder` and `TextDecoder` definitions in `brainy-wrapper.js` by replacing direct constructor usage with shorthand properties within a utility object.
This refactor reduces unnecessary complexity in CLI commands and improves maintainability by aligning with current project goals and streamlined test structure.
