### Changes:
- Disabled the `@typescript-eslint/no-explicit-any` rule in `cloud-wrapper/package.json`.
- Reformatted ESLint rule configurations for `semi` and `@typescript-eslint/semi`.
- Reorganized dependencies in `package.json` for better readability:
- Moved `buffer`, `commander`, `omelette`, and `uuid` to the end of the list to match alphabetical order.
### Purpose:
Improved code clarity and maintainability by updating ESLint configurations and reordering dependencies in `package.json`. These adjustments ensure consistent formatting and easier navigation of dependencies.
### Changes:
- Introduced `global.d.ts` to define global type declarations for the `globalThis` interface.
- Added `__ENV__` object with the following properties:
- `isBrowser: boolean`
- `isNode: string | false`
- `isServerless: boolean`
- Included an export statement to ensure the file is treated as a module.
### Purpose:
Defined global environment properties to standardize and enhance environment detection across browser, Node.js, and serverless environments. This improves type safety and consistency in the codebase.
### Changes:
- Added `cloud-wrapper/dist` to ignore built output.
- Added `cloud-wrapper/node_modules` to ignore dependency files.
### Purpose:
Updated `.gitignore` to exclude build artifacts and dependency files specific to the `cloud-wrapper`, ensuring a cleaner repository and preventing unintentional commits of these files.
### Changes:
- **Core MCP Components**:
- Introduced `BrainyMCPAdapter`, `BrainyMCPService`, and `MCPAugmentationToolset` for handling data access, tool execution, system information, and authentication via MCP.
- Added asynchronous handlers to process requests for Brainy data, augmentations, and relationships.
- Built pipelines to expose Brainy augmentation capabilities as tools.
- **Server Implementations**:
- Added WebSocket and REST interfaces in `initializeMCPService` for external MCP requests.
- Included rate limiting, authentication, and CORS support for REST API.
- **Type Definitions**:
- Defined MCP-related types such as `MCPRequestType`, `MCPResponse`, and `MCPToolExecutionRequest` in `src/types/mcpTypes.ts`.
- Incorporated augmentation-type-specific methods from `augmentationPipeline`.
- **Exports**:
- Exposed MCP modules (`BrainyMCPAdapter`, `BrainyMCPService`, `MCPAugmentationToolset`) via `src/mcp/index.ts`.
### Purpose:
Introduced a Model Control Protocol (MCP) framework to enable seamless integration of Brainy data and augmentation tools with external models. The implementation provides structured, scalable access to data and tools through both WebSocket and REST APIs.
### Changes:
- Renamed the `publish` script to `deploy` in `package.json` for clarity and consistency with deployment terminology.
- Incremented version number from `0.8.11` to `0.9.0` in `src/utils/version.ts`.
### Purpose:
Aligned the naming of the deployment script with standard practices to improve readability and maintainability. Updated the version to `0.9.0` to reflect recent changes and enhancements in the project.
### Changes:
- Added `rollup.config.js` with custom build logic for unified and browser builds:
- Included TypeScript support with separate configurations (`tsconfig.unified.json` and `tsconfig.browser.json`).
- Consolidated environment detection logic with `globalThis.__ENV__` for browser, Node.js, and serverless environments.
- Provided global Buffer polyfill for browser environments.
- Enabled `inlineDynamicImports` for module compatibility.
- Introduced custom Rollup plugins:
- Shims for Node.js built-in modules to ensure compatibility in browsers.
- Fixes for `this` references in TensorFlow.js layers causing errors in browser environments.
- Enabled environment-specific replacements for `process.env.NODE_ENV` during builds.
- Configured outputs for both non-minified and minified builds (`.js` and `.min.js` files).
- Excluded specific dependencies from the bundle to optimize build size.
### Purpose:
Added a versatile Rollup configuration to streamline the build process for both browser and Node.js environments. This setup enhances cross-environment compatibility, reduces redundancy, and ensures better maintainability with unified build logic.
### Changes:
- Added explicit `.js` extensions to imports in `cloud-wrapper/src/index.ts` for Node.js compatibility.
- Refactored `setupRoutes` usage by introducing an `apiRouter` variable for clarity and consistency in middleware.
- Simplified environment detection in `src/unified.ts` by delegating it to build-time logic:
- Removed inline environment detection (`isBrowser`, `isNode`, `isServerless`).
- Updated `environment` export with build-generated properties.
### Purpose:
Enhanced compatibility with Node.js module resolution, improved code clarity in route handling, and streamlined environment detection by consolidating it into the build process. These changes reduce redundancy, improve maintainability, and align with modern JavaScript practices.
### Changes:
- Added new TensorFlow.js dependencies:
- `@tensorflow/tfjs-backend-cpu`
- `@tensorflow/tfjs-converter`
- `@tensorflow/tfjs-core`
- `@tensorflow/tfjs-layers`
- Updated `package-lock.json` to reflect dependency updates and changes in peer dependencies.
- Refactored build scripts for clarity and consistency:
- Simplified build commands using `BUILD_TYPE` environment variable for unified and browser builds.
- Updated `demo` and `publish` scripts to align with new build structure.
- Introduced specific deployment scripts for AWS, GCP, and Cloudflare in `cloud-wrapper`:
- `deploy☁️aws`
- `deploy☁️gcp`
- `deploy☁️cloudflare`
- Added a user-friendly `deploy:cloud` script for deployment guidance.
- Removed redundant script configurations and legacy-based Rollup files.
### Purpose:
Enhanced dependency management and standardized build/deployment workflows by improving script clarity, removing redundancy, and aligning configurations with TensorFlow.js changes. This improves maintainability and user experience in development and production environments.
### Changes:
- Introduced `asyncHandler` utility to simplify error handling in asynchronous route handlers.
- Updated all route definitions in `cloud-wrapper/src/routes.ts` to use the new `asyncHandler` wrapper:
- Standardized error management across CRUD operations for nouns, verbs, and search functionality.
- Replaced redundant `try-catch` blocks with the `asyncHandler` wrapper to streamline code.
- Added `NextFunction` and adjusted TypeScript typings for improved clarity and error handling.
### Purpose:
Refactored route handlers to enhance maintainability, readability, and consistency by centralizing asynchronous error handling with a reusable `asyncHandler` utility. This reduces redundancy and simplifies debugging for API routes.
### Changes:
- Expanded cloud deployment section in `README.md`:
- Added detailed npm scripts for deploying to AWS, Google Cloud, and Cloudflare.
- Described script usage for each cloud platform.
- Included instructions for configuring environment variables in `cloud-wrapper/.env`.
- Linked to the Cloud Wrapper README for additional setup and API documentation.
### Purpose:
Improved clarity and usability of cloud deployment instructions by providing specific npm script commands and configuration guidance. This helps streamline the deployment process for users across multiple cloud platforms.
### Changes:
- Removed two documentation files:
- `docs/build-time-augmentations.md`
- `docs/llm-augmentation.md`
- Deleted `rollup.unified.js` and `rollup_config.js` configurations to streamline build setup.
- Introduced custom plugins for Node.js module shims and "this" reference fixes directly into the remaining Rollup setup.
- Updated Rollup configuration entry and build logic for increased maintainability:
- Enabled `inlineDynamicImports` for unified builds.
- Enhanced browser compatibility with global Buffer polyfill and environment detection (`isBrowser`, `isNode`, `isServerless`).
- Adjusted error logging for module fetch failures in browser environments.
### Purpose:
Cleaned outdated and redundant documentation to align with the system's current capabilities. Consolidated and optimized Rollup build setup, ensuring a modern and maintainable configuration while improving cross-environment compatibility and error handling.
### Changes:
- Introduced `package-lock.json` file for the `cloud-wrapper` package.
- Includes configuration details for dependencies:
- Runtime dependencies: `@soulcraft/brainy`, `cors`, `dotenv`, `express`, `helmet`, `morgan`, `uuid`, `ws`.
- Development dependencies: TypeScript, nodemon, and type definitions for key libraries.
- Specifies supported Node.js version (`>=23.11.0`).
### Purpose:
Committed the `package-lock.json` to ensure consistency in dependency resolution during installation. This facilitates improved reliability across development and production environments.
### Changes:
- Introduced `ExtendedBrainyDataConfig` interface to include `rootDirectory` in the storage configuration.
- Enhanced S3 storage setup:
- Added support for `customS3Storage` when `S3_ENDPOINT` is provided.
- Retained standard `s3Storage` configuration for AWS S3 without custom endpoints.
- Updated fallback logic for default storage types (`filesystem`, `memory`) with consistent handling for `rootDirectory`.
- Refined `initializeBrainy` function to improve readability and configuration extensibility.
- Updated TypeScript typings for improved clarity and maintainability.
### Purpose:
Improved the `initializeBrainy` function by extending storage configuration capabilities, allowing seamless integration of custom S3 endpoints alongside standard AWS S3 settings. Enhanced overall code readability and adaptability for diverse storage configurations.
### Changes:
- Introduced environment-independent WebStreams API compatibility:
- Added `initializeStreamClasses` for dynamic detection of WebStreams API support in browsers and Node.js.
- Implemented fallback mechanism with detailed error messages for unsupported environments.
- Ensured asynchronous initialization of stream classes without blocking module execution.
- Updated `SequentialPipeline`:
- Added `ensureStreamClassesInitialized` method to streamline WebStreams setup across operations.
- Enhanced `initialize` to parallelize stream classes and `BrainyData` initialization.
- Refactored WebSocket handlers (`createWebSocketHandler` and `createWebSocketStreams`) to support async stream initialization.
- Improved TypeScript typings for streams (`TransformStreamDefaultController`, `PipelineResult<unknown>`).
- Adjusted `ServerSearchConduitAugmentation`:
- Introduced dependency on `BrainyDataInterface` instead of direct `BrainyData` usage.
- Ensured `localDb` is set prior to initialization with an appropriate error message.
- Updated methods (`setLocalDb`, `getLocalDb`, and vector operations) to rely on `BrainyDataInterface`.
- Enhanced error handling and logging throughout streaming and augmentation processes.
### Purpose:
Improved cross-environment compatibility of WebStreams API, ensuring seamless usage in both browsers and Node.js. Refactored `ServerSearchConduitAugmentation` to decouple `BrainyData` dependency,
### Changes:
- Renamed `nodes` to `nouns` and `edges` to `verbs` across `opfsStorage.ts`, improving code semantics:
- Directory constants (e.g., `NODES_DIR` → `NOUNS_DIR`).
- Class properties, method names, and variables updated accordingly.
- Introduced global type augmentation for `FileSystemDirectoryHandle` to support `entries()` method, improving compatibility.
- Updated directory management logic:
- Adjusted initialization, recreation, and metadata retrieval methods to reflect new nomenclature.
- Enhanced handling for directory entries in both `nouns` and `verbs` contexts.
- Refined comments, formatting, and TypeScript annotations for consistency.
### Purpose:
Improved the code's readability and semantic clarity by aligning directory and variable names with their conceptual purpose (`nouns` for entities, `verbs` for relationships). Enhanced maintainability and developer understanding of the storage implementation.
### Changes:
- Removed conditional configuration for `USE_SIMPLE_EMBEDDING` as Universal Sentence Encoder is now the sole embedding option.
- Updated comments to reflect the exclusive use of TensorFlow.js for embedding functionality.
- Cleaned up unnecessary blank lines and whitespace for improved code readability.
### Purpose:
Simplified the `initializeBrainy` function by consolidating embedding configuration and eliminating outdated options. This refinement aligns with the project's transition to TensorFlow-based embeddings and improves code clarity.
### Changes:
- Removed `SimpleEmbedding` class, including its character-based embedding functionality and associated utility methods.
- Refined the Universal Sentence Encoder (USE) implementation:
- Added support for handling edge cases such as empty strings and arrays by returning zero vectors.
- Filtered out invalid input, improving robustness.
- Addressed `EPSILON` flag initialization and TensorFlow.js environment setup to prevent runtime errors.
- Updated error handling for embedding failures to provide more detailed traceability.
- Simplified `createEmbeddingFunction` and `defaultEmbeddingFunction` to focus solely on TensorFlow-based embeddings.
### Purpose:
Streamlined the embedding API by removing the outdated `SimpleEmbedding` class and fully transitioning to TensorFlow's Universal Sentence Encoder. These changes improve input validation, error resilience, and compatibility with TensorFlow.js, ensuring a more consistent and reliable embedding experience.
### Changes:
- Enhanced `BrainyData` class to support `HNSWIndexOptimized` alongside the standard HNSW index.
- Added a new configuration parameter `hnswOptimized` for enabling optimized indexing.
- Created utility methods for handling optimized index-specific functionalities such as memory usage tracking and disk-based index management.
- Updated vector addition, retrieval, and management methods to accommodate both standard and optimized indices.
- Introduced support for using pre-defined IDs during vector and relationship creation.
- Standardized metadata handling for error resilience and embedding integrations.
- Implemented a `getAllNouns` method to retrieve all indexed data.
- Added validation for vector dimensions during relationship creation to ensure consistency.
- Refactored method signatures to improve readability and alignment.
### Purpose:
Improved the BrainyData API to support optimized indexing functionality powered by `HNSWIndexOptimized`, enhancing scalability for large datasets while maintaining backward compatibility. Elevated overall code structure and usability.
### Changes:
- Added `HNSWIndexOptimized` class in `src/hnsw/hnswIndexOptimized.ts`:
- Supports large datasets with product quantization for dimensionality reduction.
- Enables disk-based index storage via a configurable storage adapter.
- Introduces `HNSWOptimizedConfig` interface for advanced configuration:
- Includes memory thresholds, product quantization (subvectors, centroids), and disk-based indexing options.
- Implements methods for adding, searching, and managing indexed vectors with enhanced scalability.
- Introduced `ProductQuantizer` utility:
- Handles vector quantization with training and reconstruction capabilities.
- Reduces vector dimensions, improving memory and performance efficiency.
- Added accessor methods in `src/hnsw/hnswIndex.ts` for configuration and index metadata (e.g., dimension, max level, entry point).
### Purpose:
Enhanced the HNSW implementation to support optimized indexing and search for large datasets, addressing scalability challenges via product quantization and optional disk-based storage. This update elevates the library's performance and applicability for memory-constrained environments.
### Changes:
- Updated `VERSION` constant in `src/utils/version.ts` from `0.8.0` to `0.8.11`.
### Purpose:
Incremented the project version to reflect new changes, updates, or bug fixes, ensuring proper version tracking and release management.
### Changes:
- Added `backup` command to export database content into a JSON file.
- Includes noun and verb data with metadata.
- Allows specifying a custom output filename (`brainy-backup.json` by default).
- Added `restore` command to import data from a JSON file into the database.
- Supports clearing existing data with `--clear` option before restoration.
- Displays success information for imported nouns and verbs.
- Integrated help text examples for the new commands.
- Enhanced CLI interface with additional indentation for better readability.
- Adjusted `completion-setup` to include autocomplete for backup and restore operations.
### Purpose:
Introduced backup and restore functionality to improve database management and data portability, enabling users to save and load datasets seamlessly for better flexibility in usage and recovery.
### Changes:
- Created `examples/browser_compatible_exports.ts`:
- Introduced browser-friendly exports with polyfill for `Buffer` in browser environments.
- Implemented shims for Node.js modules (`fs`, `util`, and `path`) to avoid compatibility issues.
- Exported browser-compatible types and utilities: `BrainyData`, `NounType`, `VerbType`, graph types, core types, distance functions, embedding utilities, and storage adapters.
- Explicitly excluded Node.js/CLI-specific parts to focus on browser compatibility.
- Added `src/types/brainyDataInterface.ts`:
- Defined `BrainyDataInterface` to break the circular dependency between `brainyData.ts` and `serverSearchAugmentations.ts`.
- Provided methods for managing nodes, relationships, initialization, search, and adding vector data with metadata.
### Purpose:
- Enhanced the codebase to support browser environments by introducing browser-specific exports and avoiding Node.js dependencies.
- Established clearer separation of concerns and modularity by introducing the `BrainyDataInterface` for dependency management, improving maintainability and flexibility.
### Changes:
- Removed the `src/augmentations/llmAugmentations.ts` file:
- Contained cognition augmentations for creating, testing, exporting, and deploying LLM models.
- Included TensorFlow.js-based model configuration, training, testing, and deployment functionalities.
- Implemented various presets (`tiny`, `small`, `medium`, `large`) for simplified LLM model creation.
### Purpose:
This change removes legacy LLM-related augmentations and functionalities no longer relevant to the project's current focus, in alignment with the recent transition to TensorFlow-based embedding workflows. Cleaning up eliminates unused code and maintains a streamlined, up-to-date codebase.
### Changes:
- Updated README.md:
- Refined the project description to highlight its lightweight and powerful nature.
- Enhanced feature list, replacing outdated entries with modern updates (e.g., TensorFlow integration replacing LLM creation).
- Included detailed installation and usage instructions for TensorFlow.js support and embedding workflows.
- Added "Usage Options" section to define different import methods and supported environments.
- Expanded "Backup and Restore" details, showcasing CLI and scripting examples.
- Introduced optimized HNSW indexing for large datasets.
- Replaced examples with a unified and simplified demonstration approach.
- Embedded advanced build and environment-specific configurations for better clarity.
- Removed outdated content related to language model generation, including LLM-related commands and examples.
- Added README.demo.md with full instructions for running the interactive demo.
### Purpose:
This update modernizes the documentation and focuses on Brainy’s current capabilities, simplifying onboarding for new users while removing deprecated features. It also streamlines project understanding, enabling developers to better utilize its features in diverse environments.
### Changes:
- Added `rollup.unified.js` for building a unified bundle with environment detection (Node.js, Browser, Serverless).
- Added `rollup_config.js` for browser-specific bundle configurations.
- Created `tsconfig.browser.json` for browser builds and `tsconfig.unified.json` for unified builds.
- Implemented custom Rollup plugins:
- **`fixThisReferences`**: Resolves `this` reference issues in TensorFlow files.
- **`nodeModuleShims`**: Provides empty shims for Node.js built-in modules in browser environments.
- Updated dependencies in `package-lock.json` to include Rollup plugins and updated `buffer` dependency for polyfills.
- Created `src/unified.ts` as the unified library entry point with dynamic environment detection.
### Purpose:
- Introduced a unified and browser-compatible build pipeline to ensure `Brainy` can seamlessly operate across multiple environments (Node.js, Browser, Serverless).
- Resolved compatibility issues with Node.js modules and specific library builds.
- Enhanced flexibility and usability for developers working in diverse runtime environments.
### Changes:
- Deleted the following outdated example files from the `examples` directory:
- `buildTimeRegistration.js`
- `conduitAugmentationExample.js`
- `configurationTest.js`
- `dataInspectionExample.js`
- `import-graphTypes.js`
- `browser-server-search/index.html`
### Purpose:
These example files were removed as they were no longer relevant or up-to-date with the current state of the project. This cleanup ensures that users referencing examples have access to accurate and relevant documentation and code, reducing potential confusion.
### Changes:
- Adjusted the main project description in `README.md` for a more engaging tone by adding "A fun" to emphasize the platform's appeal.
### Purpose:
This update aims to make the project description more approachable and engaging for potential users while maintaining clarity about its capabilities.
Introduced `encode-image.js` to convert images to Base64 and create corresponding HTML `<img>` tags. Outputs a sample encoded result to `encoded-image.html`.
