### Changes:
- Introduced two new relationship types in `VerbType`:
- `Follows`: Indicates a following relationship.
- `Likes`: Indicates a liking relationship.
### Purpose:
Enhanced the `VerbType` to support additional relationship semantics, expanding its applicability for modeling complex graph structures and fostering richer data representation.
### Changes:
- Introduced a `findSimilar` method in `src/brainyData.ts`:
- Enables searching for entities similar to a given entity ID.
- Supports options such as result limit, noun type filtering, verb inclusion, and search mode (local, remote, or combined).
- Filters out the original entity in results and returns limited output based on the specified options.
- Updated `brainyDataInterface` in `src/types/brainyDataInterface.ts` to include the new `findSimilar` method definition.
### Purpose:
Added the `findSimilar` method to enhance entity retrieval capabilities by enabling similarity-based searches. This feature improves data exploration and provides a flexible API for building advanced search and recommendation functionalities.
### Changes:
- Updated `src/index.ts` exports to streamline pipeline and augmentation factory usage:
- Consolidated augmentation and streamlined pipeline exports under unified `Pipeline` module (e.g., `pipeline`, `createPipeline`, `executeStreamlined`).
- Added new augmentation factory exports:
- `createSenseAugmentation`, `addWebSocketSupport`, `executeAugmentation`, `loadAugmentationModule`.
- Enhanced backward compatibility with separate sequential pipeline exports.
- Introduced streamlined type exports, including `StreamlinedExecutionMode`, `StreamlinedPipelineOptions`, and `StreamlinedPipelineResult`.
### Purpose:
Standardized pipeline and augmentation factory exports for better modularity and backward compatibility. These changes improve developer experience by simplifying imports and aligning with the unified augmentation pipeline system.
### Changes:
- Added `Pipeline` class in `src/pipeline.ts` to unify primary and streamlined augmentation pipelines:
- Supports multiple execution modes: sequential, parallel, threaded, first success, and first result.
- Provides methods for registering, unregistering, and managing augmentations by type.
- Implements flexible execution options, including error handling, timeouts, and threading control.
- Introduced comprehensive augmentation registry for managing sense, dialog, memory, cognition, and WebSocket-based augmentations.
- Added global pipeline instance with legacy method support for backward compatibility.
- Enhanced augmentation execution capabilities:
- Support for static and streaming data processing.
- Pipeline creation for reusable augmentation workflows.
- Threaded execution for improved performance when supported.
- Added default pipeline integration with augmentation registry to resolve circular dependencies.
### Purpose:
Implemented a unified augmentation pipeline to streamline augmentation workflows, enhance flexibility, and simplify execution management. The changes improve performance, reduce complexity, and provide robust support for legacy methods.
### Changes:
- Introduced the `import-sparse` command in `src/cli.ts`:
- Allows importing sparse data from a JSON file into the database.
- Includes an option (`--clear`) to clear existing data before import.
- Displays counts of imported nouns and verbs upon successful completion.
- Updated CLI help and autocompletion configuration to support the new command.
### Purpose:
Enhanced the CLI by adding `import-sparse`, providing users with the ability to efficiently import sparse data into the database. This addition aligns with the application's functionality for managing sparse data operations.
### Changes:
- Introduced the `importSparseData` method in `src/brainyData.ts` to allow importing sparse data into the database.
- Supports embedding generation for nouns if vectors are missing.
- Includes options to clear existing data during import.
- Returns counts of imported nouns and verbs for tracking.
### Purpose:
Added a new method to manage sparse data imports, improving functionality and ensuring alignment with updated terminology across the application. This addition enhances flexibility in handling database operations.
### Changes:
- Introduced `initializeMCPService` in `cloud-wrapper/src/index.ts` to support initializing Model Control Protocol (MCP) services.
- Enabled configuration for WebSocket and REST APIs with customizable options like ports, authentication, rate limiting, and CORS.
- Exported MCP modules, types, and services (`BrainyMCPAdapter`, `BrainyMCPService`, etc.) in `src/index.ts`.
- Enhanced `unified.ts` to include global environment detection by populating `globalThis.__ENV__`.
- Added environment-specific logging for browser, Node.js, and serverless scenarios.
- Improved the structure of `environment` object and its global accessibility.
### Purpose:
Implemented MCP service integration to expand Brainy's external compatibility via WebSocket and REST interfaces. Enhanced environment detection provides better global access and debugging insights, ensuring robust operation across various runtime environments.
### Changes:
- Enhanced environment detection logic with a structured `environment` object (`isBrowser`, `isNode`, `isServerless`).
- Implemented fallback options for various environments:
- Added priority handling for storage types (e.g., MemoryStorage, OPFSStorage, FileSystemStorage).
- Improved dynamic imports for `s3CompatibleStorage.js` and `fileSystemStorage.js`.
- Unified and expanded cloud storage support by merging environment variables with provided options:
- Refined logic for R2, S3, and GCS storage configurations.
- Added detailed logging during storage type selection (`console.log` for selected storage).
- Introduced better error handling with warnings and defaults for unsupported environments.
- Removed redundant checks and reorganized storage selection logic for clarity.
### Purpose:
Streamlined environment detection and adaptive storage logic to ensure seamless operation across different platforms (browser, Node.js, serverless). These changes improve maintainability, enhance debugging, and simplify storage configuration while providing robust fallback mechanisms.
### Changes:
- Introduced `global.d.ts` to define global type declarations for the `globalThis` interface.
- Added `__ENV__` object with the following properties:
- `isBrowser: boolean`
- `isNode: string | false`
- `isServerless: boolean`
- Included an export statement to ensure the file is treated as a module.
### Purpose:
Defined global environment properties to standardize and enhance environment detection across browser, Node.js, and serverless environments. This improves type safety and consistency in the codebase.
### Changes:
- **Core MCP Components**:
- Introduced `BrainyMCPAdapter`, `BrainyMCPService`, and `MCPAugmentationToolset` for handling data access, tool execution, system information, and authentication via MCP.
- Added asynchronous handlers to process requests for Brainy data, augmentations, and relationships.
- Built pipelines to expose Brainy augmentation capabilities as tools.
- **Server Implementations**:
- Added WebSocket and REST interfaces in `initializeMCPService` for external MCP requests.
- Included rate limiting, authentication, and CORS support for REST API.
- **Type Definitions**:
- Defined MCP-related types such as `MCPRequestType`, `MCPResponse`, and `MCPToolExecutionRequest` in `src/types/mcpTypes.ts`.
- Incorporated augmentation-type-specific methods from `augmentationPipeline`.
- **Exports**:
- Exposed MCP modules (`BrainyMCPAdapter`, `BrainyMCPService`, `MCPAugmentationToolset`) via `src/mcp/index.ts`.
### Purpose:
Introduced a Model Control Protocol (MCP) framework to enable seamless integration of Brainy data and augmentation tools with external models. The implementation provides structured, scalable access to data and tools through both WebSocket and REST APIs.
### Changes:
- Renamed the `publish` script to `deploy` in `package.json` for clarity and consistency with deployment terminology.
- Incremented version number from `0.8.11` to `0.9.0` in `src/utils/version.ts`.
### Purpose:
Aligned the naming of the deployment script with standard practices to improve readability and maintainability. Updated the version to `0.9.0` to reflect recent changes and enhancements in the project.
### Changes:
- Added explicit `.js` extensions to imports in `cloud-wrapper/src/index.ts` for Node.js compatibility.
- Refactored `setupRoutes` usage by introducing an `apiRouter` variable for clarity and consistency in middleware.
- Simplified environment detection in `src/unified.ts` by delegating it to build-time logic:
- Removed inline environment detection (`isBrowser`, `isNode`, `isServerless`).
- Updated `environment` export with build-generated properties.
### Purpose:
Enhanced compatibility with Node.js module resolution, improved code clarity in route handling, and streamlined environment detection by consolidating it into the build process. These changes reduce redundancy, improve maintainability, and align with modern JavaScript practices.
### Changes:
- Introduced environment-independent WebStreams API compatibility:
- Added `initializeStreamClasses` for dynamic detection of WebStreams API support in browsers and Node.js.
- Implemented fallback mechanism with detailed error messages for unsupported environments.
- Ensured asynchronous initialization of stream classes without blocking module execution.
- Updated `SequentialPipeline`:
- Added `ensureStreamClassesInitialized` method to streamline WebStreams setup across operations.
- Enhanced `initialize` to parallelize stream classes and `BrainyData` initialization.
- Refactored WebSocket handlers (`createWebSocketHandler` and `createWebSocketStreams`) to support async stream initialization.
- Improved TypeScript typings for streams (`TransformStreamDefaultController`, `PipelineResult<unknown>`).
- Adjusted `ServerSearchConduitAugmentation`:
- Introduced dependency on `BrainyDataInterface` instead of direct `BrainyData` usage.
- Ensured `localDb` is set prior to initialization with an appropriate error message.
- Updated methods (`setLocalDb`, `getLocalDb`, and vector operations) to rely on `BrainyDataInterface`.
- Enhanced error handling and logging throughout streaming and augmentation processes.
### Purpose:
Improved cross-environment compatibility of WebStreams API, ensuring seamless usage in both browsers and Node.js. Refactored `ServerSearchConduitAugmentation` to decouple `BrainyData` dependency,
### Changes:
- Renamed `nodes` to `nouns` and `edges` to `verbs` across `opfsStorage.ts`, improving code semantics:
- Directory constants (e.g., `NODES_DIR` → `NOUNS_DIR`).
- Class properties, method names, and variables updated accordingly.
- Introduced global type augmentation for `FileSystemDirectoryHandle` to support `entries()` method, improving compatibility.
- Updated directory management logic:
- Adjusted initialization, recreation, and metadata retrieval methods to reflect new nomenclature.
- Enhanced handling for directory entries in both `nouns` and `verbs` contexts.
- Refined comments, formatting, and TypeScript annotations for consistency.
### Purpose:
Improved the code's readability and semantic clarity by aligning directory and variable names with their conceptual purpose (`nouns` for entities, `verbs` for relationships). Enhanced maintainability and developer understanding of the storage implementation.
### Changes:
- Removed `SimpleEmbedding` class, including its character-based embedding functionality and associated utility methods.
- Refined the Universal Sentence Encoder (USE) implementation:
- Added support for handling edge cases such as empty strings and arrays by returning zero vectors.
- Filtered out invalid input, improving robustness.
- Addressed `EPSILON` flag initialization and TensorFlow.js environment setup to prevent runtime errors.
- Updated error handling for embedding failures to provide more detailed traceability.
- Simplified `createEmbeddingFunction` and `defaultEmbeddingFunction` to focus solely on TensorFlow-based embeddings.
### Purpose:
Streamlined the embedding API by removing the outdated `SimpleEmbedding` class and fully transitioning to TensorFlow's Universal Sentence Encoder. These changes improve input validation, error resilience, and compatibility with TensorFlow.js, ensuring a more consistent and reliable embedding experience.
### Changes:
- Enhanced `BrainyData` class to support `HNSWIndexOptimized` alongside the standard HNSW index.
- Added a new configuration parameter `hnswOptimized` for enabling optimized indexing.
- Created utility methods for handling optimized index-specific functionalities such as memory usage tracking and disk-based index management.
- Updated vector addition, retrieval, and management methods to accommodate both standard and optimized indices.
- Introduced support for using pre-defined IDs during vector and relationship creation.
- Standardized metadata handling for error resilience and embedding integrations.
- Implemented a `getAllNouns` method to retrieve all indexed data.
- Added validation for vector dimensions during relationship creation to ensure consistency.
- Refactored method signatures to improve readability and alignment.
### Purpose:
Improved the BrainyData API to support optimized indexing functionality powered by `HNSWIndexOptimized`, enhancing scalability for large datasets while maintaining backward compatibility. Elevated overall code structure and usability.
### Changes:
- Added `HNSWIndexOptimized` class in `src/hnsw/hnswIndexOptimized.ts`:
- Supports large datasets with product quantization for dimensionality reduction.
- Enables disk-based index storage via a configurable storage adapter.
- Introduces `HNSWOptimizedConfig` interface for advanced configuration:
- Includes memory thresholds, product quantization (subvectors, centroids), and disk-based indexing options.
- Implements methods for adding, searching, and managing indexed vectors with enhanced scalability.
- Introduced `ProductQuantizer` utility:
- Handles vector quantization with training and reconstruction capabilities.
- Reduces vector dimensions, improving memory and performance efficiency.
- Added accessor methods in `src/hnsw/hnswIndex.ts` for configuration and index metadata (e.g., dimension, max level, entry point).
### Purpose:
Enhanced the HNSW implementation to support optimized indexing and search for large datasets, addressing scalability challenges via product quantization and optional disk-based storage. This update elevates the library's performance and applicability for memory-constrained environments.
### Changes:
- Updated `VERSION` constant in `src/utils/version.ts` from `0.8.0` to `0.8.11`.
### Purpose:
Incremented the project version to reflect new changes, updates, or bug fixes, ensuring proper version tracking and release management.
### Changes:
- Added `backup` command to export database content into a JSON file.
- Includes noun and verb data with metadata.
- Allows specifying a custom output filename (`brainy-backup.json` by default).
- Added `restore` command to import data from a JSON file into the database.
- Supports clearing existing data with `--clear` option before restoration.
- Displays success information for imported nouns and verbs.
- Integrated help text examples for the new commands.
- Enhanced CLI interface with additional indentation for better readability.
- Adjusted `completion-setup` to include autocomplete for backup and restore operations.
### Purpose:
Introduced backup and restore functionality to improve database management and data portability, enabling users to save and load datasets seamlessly for better flexibility in usage and recovery.
### Changes:
- Created `examples/browser_compatible_exports.ts`:
- Introduced browser-friendly exports with polyfill for `Buffer` in browser environments.
- Implemented shims for Node.js modules (`fs`, `util`, and `path`) to avoid compatibility issues.
- Exported browser-compatible types and utilities: `BrainyData`, `NounType`, `VerbType`, graph types, core types, distance functions, embedding utilities, and storage adapters.
- Explicitly excluded Node.js/CLI-specific parts to focus on browser compatibility.
- Added `src/types/brainyDataInterface.ts`:
- Defined `BrainyDataInterface` to break the circular dependency between `brainyData.ts` and `serverSearchAugmentations.ts`.
- Provided methods for managing nodes, relationships, initialization, search, and adding vector data with metadata.
### Purpose:
- Enhanced the codebase to support browser environments by introducing browser-specific exports and avoiding Node.js dependencies.
- Established clearer separation of concerns and modularity by introducing the `BrainyDataInterface` for dependency management, improving maintainability and flexibility.
### Changes:
- Removed the `src/augmentations/llmAugmentations.ts` file:
- Contained cognition augmentations for creating, testing, exporting, and deploying LLM models.
- Included TensorFlow.js-based model configuration, training, testing, and deployment functionalities.
- Implemented various presets (`tiny`, `small`, `medium`, `large`) for simplified LLM model creation.
### Purpose:
This change removes legacy LLM-related augmentations and functionalities no longer relevant to the project's current focus, in alignment with the recent transition to TensorFlow-based embedding workflows. Cleaning up eliminates unused code and maintains a streamlined, up-to-date codebase.
### Changes:
- Added `rollup.unified.js` for building a unified bundle with environment detection (Node.js, Browser, Serverless).
- Added `rollup_config.js` for browser-specific bundle configurations.
- Created `tsconfig.browser.json` for browser builds and `tsconfig.unified.json` for unified builds.
- Implemented custom Rollup plugins:
- **`fixThisReferences`**: Resolves `this` reference issues in TensorFlow files.
- **`nodeModuleShims`**: Provides empty shims for Node.js built-in modules in browser environments.
- Updated dependencies in `package-lock.json` to include Rollup plugins and updated `buffer` dependency for polyfills.
- Created `src/unified.ts` as the unified library entry point with dynamic environment detection.
### Purpose:
- Introduced a unified and browser-compatible build pipeline to ensure `Brainy` can seamlessly operate across multiple environments (Node.js, Browser, Serverless).
- Resolved compatibility issues with Node.js modules and specific library builds.
- Enhanced flexibility and usability for developers working in diverse runtime environments.
Standardized documentation by adding a centered Brainy logo across README files, examples, and guides. Adjusted text formatting for consistency, improved alignment, and readability of feature descriptions and examples.
Implemented new simplified commands (`create-simple`, `train-simple`, `generate-simple`) for easier LLM model management. Added pre-configured model presets (`tiny`, `small`, `medium`, `large`) and enhanced options to streamline usage for beginners. Updated CLI and API documentation with detailed examples. Incremented version to 0.8.0.
Introduced `llm-augmentation.md`, detailing the creation, training, testing, exporting, and deployment of language models using Brainy's graph database. Added examples showcasing practical usage and pipeline integration. Included class implementations in `llmAugmentations.ts`.
Implemented a WebSocket API for real-time communication and REST API routes for CRUD operations on nouns and verbs. Added a server initialization script with WebSocket server integration and improved application structure with modularized routing and services.
Implemented WebSocketConduitAugmentation and WebRTCConduitAugmentation for syncing Brainy instances. Added comprehensive usage examples to showcase data synchronization via WebSocket and WebRTC.
Incremented version to 0.7.5 across `version.ts` and `README.md`. Introduced WebSocket and WebRTC conduit augmentations in codebase and documented usage examples thoroughly in `README.md`.
Upgraded minimum Node.js version from 18.0.0 to 23.11.0 across `README.md`, `package.json`, and `version.ts`. Optimized distance utilities and pipelines to leverage Node.js 23.11+ native performance improvements (e.g., `array.reduce`, WebStreams API). Incremented version to 0.7.4 for consistency.
Updated `README.md` to reflect version `0.7.3` with improved feature descriptions and scaling highlights. Enhanced the versioning script to automatically update the version badge in `README.md` for consistency. Incremented `version.ts` to `0.7.3`.
Enhanced the `README.md` with more engaging and user-friendly descriptions, highlighting Brainy's adaptability and intelligence. Introduced new sections detailing adaptive intelligence, data pipeline enhancements, and use case expansion. Removed outdated dependencies from `package-lock.json`. Updated `version.ts` to `0.7.2`.
Introduced a script to generate a `version.ts` file exporting the package version from `package.json`. This simplifies accessing version data in the CLI, especially for global installations. Included auto-generated `version.ts` in the project structure.
Introduced `generateRandomGraph` method to create randomized graphs with configurable entity counts and types, supporting automated testing and experimentation. Enhanced metadata vectorization logic to handle various metadata formats reliably, ensuring compatibility and fallback mechanisms during embedding. Updated `README.md` with detailed feature descriptions and usage guides.
Enhanced the installation section with guidance on resolving TensorFlow.js dependency conflicts using `--legacy-peer-deps`. Added CLI autocomplete setup instructions to improve user experience. Suppressed specific TensorFlow.js Node.js backend warnings in the embedding utils. Updated dependencies to the latest versions.
Added a comprehensive command-line interface (CLI) for interacting with the Brainy vector database. The CLI supports various operations, including database initialization, adding/searching nouns, managing relationships, and querying database status. Enhanced type validation logic for nouns and verbs to ensure consistency and enforce default types for invalid inputs. Updated test scripts to verify type validation and edge cases.
Updated search methods to support `includeVerbs` for retrieving associated verbs in results. Enhanced edge creation to allow metadata embedding when no vector is provided. Improved query handling and vectorization logic for consistent processing.
Added extensive configuration support for storage adapters, including S3-compatible storage (R2, Amazon S3, Google Cloud Storage). Introduced environment variable fallback for seamless cloud integration. Updated createStorage logic for improved compatibility. Implemented read-only mode to restrict write operations and provide a safer database configuration.
Added the `SequentialPipeline` class for structured augmentation execution. Integrated threading capabilities via the `THREADED` execution mode in `augmentationPipeline`. Enhanced WebSocket support for real-time data processing through pipelines.
Introduced new utilities for detecting execution environments (browser, Node.js, Web Worker) and checking threading availability. Added `workerUtils` for executing functions in Web Workers or Worker Threads seamlessly based on the environment.
Added `FileSystemStorage` and exports for sequential pipelines, including `SequentialPipeline` and related types (`PipelineResult`, `SequentialPipelineOptions`).
Deleted all Firestore augmentation, storage, sync implementations, and related type definitions. Removed the FirestoreSync example file as part of the cleanup for disabled Firestore functionality.
Commented out all Firestore-related code across the project, including sync, storage, and vector search features. Removed Firebase dependencies (`firebase`, `firebase-admin`) from `package.json`. Updated `README.md` to reflect the temporary unavailability of Firestore features.
Introduced Firestore vector search integration using `@firebase/firestore-vector-search`. Added support for restricting searches by specific noun types to optimize performance and relevance. Updated `README.md` to document the new functionality and usage examples.
Refactored memory augmentation methods to return `Promise<AugmentationResponse>` for async handling. Introduced a new `search` method for vector similarity searches within the memory system.
Applied consistent indentation and line-breaking rules to code and text sections in `README.md`, ensuring better formatting and alignment for readability. No functional changes made.
Deleted `examples/externalPlugins.js` and `pluginLoader.ts` as they are no longer used. Updated all `.ts` imports to `.js` across the codebase. Added `MemberOf` to `VerbType` and filtered disabled augmentations in `executeAugmentationPipeline`. Updated documentation to reflect new augmentation registration process. Removed deprecated `allowImportingTsExtensions` from `tsconfig.json`.
