### Changes:
- Introduced `global.d.ts` to define global type declarations for the `globalThis` interface.
- Added `__ENV__` object with the following properties:
- `isBrowser: boolean`
- `isNode: string | false`
- `isServerless: boolean`
- Included an export statement to ensure the file is treated as a module.
### Purpose:
Defined global environment properties to standardize and enhance environment detection across browser, Node.js, and serverless environments. This improves type safety and consistency in the codebase.
### Changes:
- **Core MCP Components**:
- Introduced `BrainyMCPAdapter`, `BrainyMCPService`, and `MCPAugmentationToolset` for handling data access, tool execution, system information, and authentication via MCP.
- Added asynchronous handlers to process requests for Brainy data, augmentations, and relationships.
- Built pipelines to expose Brainy augmentation capabilities as tools.
- **Server Implementations**:
- Added WebSocket and REST interfaces in `initializeMCPService` for external MCP requests.
- Included rate limiting, authentication, and CORS support for REST API.
- **Type Definitions**:
- Defined MCP-related types such as `MCPRequestType`, `MCPResponse`, and `MCPToolExecutionRequest` in `src/types/mcpTypes.ts`.
- Incorporated augmentation-type-specific methods from `augmentationPipeline`.
- **Exports**:
- Exposed MCP modules (`BrainyMCPAdapter`, `BrainyMCPService`, `MCPAugmentationToolset`) via `src/mcp/index.ts`.
### Purpose:
Introduced a Model Control Protocol (MCP) framework to enable seamless integration of Brainy data and augmentation tools with external models. The implementation provides structured, scalable access to data and tools through both WebSocket and REST APIs.
### Changes:
- Renamed the `publish` script to `deploy` in `package.json` for clarity and consistency with deployment terminology.
- Incremented version number from `0.8.11` to `0.9.0` in `src/utils/version.ts`.
### Purpose:
Aligned the naming of the deployment script with standard practices to improve readability and maintainability. Updated the version to `0.9.0` to reflect recent changes and enhancements in the project.
### Changes:
- Added explicit `.js` extensions to imports in `cloud-wrapper/src/index.ts` for Node.js compatibility.
- Refactored `setupRoutes` usage by introducing an `apiRouter` variable for clarity and consistency in middleware.
- Simplified environment detection in `src/unified.ts` by delegating it to build-time logic:
- Removed inline environment detection (`isBrowser`, `isNode`, `isServerless`).
- Updated `environment` export with build-generated properties.
### Purpose:
Enhanced compatibility with Node.js module resolution, improved code clarity in route handling, and streamlined environment detection by consolidating it into the build process. These changes reduce redundancy, improve maintainability, and align with modern JavaScript practices.
### Changes:
- Introduced environment-independent WebStreams API compatibility:
- Added `initializeStreamClasses` for dynamic detection of WebStreams API support in browsers and Node.js.
- Implemented fallback mechanism with detailed error messages for unsupported environments.
- Ensured asynchronous initialization of stream classes without blocking module execution.
- Updated `SequentialPipeline`:
- Added `ensureStreamClassesInitialized` method to streamline WebStreams setup across operations.
- Enhanced `initialize` to parallelize stream classes and `BrainyData` initialization.
- Refactored WebSocket handlers (`createWebSocketHandler` and `createWebSocketStreams`) to support async stream initialization.
- Improved TypeScript typings for streams (`TransformStreamDefaultController`, `PipelineResult<unknown>`).
- Adjusted `ServerSearchConduitAugmentation`:
- Introduced dependency on `BrainyDataInterface` instead of direct `BrainyData` usage.
- Ensured `localDb` is set prior to initialization with an appropriate error message.
- Updated methods (`setLocalDb`, `getLocalDb`, and vector operations) to rely on `BrainyDataInterface`.
- Enhanced error handling and logging throughout streaming and augmentation processes.
### Purpose:
Improved cross-environment compatibility of WebStreams API, ensuring seamless usage in both browsers and Node.js. Refactored `ServerSearchConduitAugmentation` to decouple `BrainyData` dependency,
### Changes:
- Renamed `nodes` to `nouns` and `edges` to `verbs` across `opfsStorage.ts`, improving code semantics:
- Directory constants (e.g., `NODES_DIR` → `NOUNS_DIR`).
- Class properties, method names, and variables updated accordingly.
- Introduced global type augmentation for `FileSystemDirectoryHandle` to support `entries()` method, improving compatibility.
- Updated directory management logic:
- Adjusted initialization, recreation, and metadata retrieval methods to reflect new nomenclature.
- Enhanced handling for directory entries in both `nouns` and `verbs` contexts.
- Refined comments, formatting, and TypeScript annotations for consistency.
### Purpose:
Improved the code's readability and semantic clarity by aligning directory and variable names with their conceptual purpose (`nouns` for entities, `verbs` for relationships). Enhanced maintainability and developer understanding of the storage implementation.
### Changes:
- Removed `SimpleEmbedding` class, including its character-based embedding functionality and associated utility methods.
- Refined the Universal Sentence Encoder (USE) implementation:
- Added support for handling edge cases such as empty strings and arrays by returning zero vectors.
- Filtered out invalid input, improving robustness.
- Addressed `EPSILON` flag initialization and TensorFlow.js environment setup to prevent runtime errors.
- Updated error handling for embedding failures to provide more detailed traceability.
- Simplified `createEmbeddingFunction` and `defaultEmbeddingFunction` to focus solely on TensorFlow-based embeddings.
### Purpose:
Streamlined the embedding API by removing the outdated `SimpleEmbedding` class and fully transitioning to TensorFlow's Universal Sentence Encoder. These changes improve input validation, error resilience, and compatibility with TensorFlow.js, ensuring a more consistent and reliable embedding experience.
### Changes:
- Enhanced `BrainyData` class to support `HNSWIndexOptimized` alongside the standard HNSW index.
- Added a new configuration parameter `hnswOptimized` for enabling optimized indexing.
- Created utility methods for handling optimized index-specific functionalities such as memory usage tracking and disk-based index management.
- Updated vector addition, retrieval, and management methods to accommodate both standard and optimized indices.
- Introduced support for using pre-defined IDs during vector and relationship creation.
- Standardized metadata handling for error resilience and embedding integrations.
- Implemented a `getAllNouns` method to retrieve all indexed data.
- Added validation for vector dimensions during relationship creation to ensure consistency.
- Refactored method signatures to improve readability and alignment.
### Purpose:
Improved the BrainyData API to support optimized indexing functionality powered by `HNSWIndexOptimized`, enhancing scalability for large datasets while maintaining backward compatibility. Elevated overall code structure and usability.
### Changes:
- Added `HNSWIndexOptimized` class in `src/hnsw/hnswIndexOptimized.ts`:
- Supports large datasets with product quantization for dimensionality reduction.
- Enables disk-based index storage via a configurable storage adapter.
- Introduces `HNSWOptimizedConfig` interface for advanced configuration:
- Includes memory thresholds, product quantization (subvectors, centroids), and disk-based indexing options.
- Implements methods for adding, searching, and managing indexed vectors with enhanced scalability.
- Introduced `ProductQuantizer` utility:
- Handles vector quantization with training and reconstruction capabilities.
- Reduces vector dimensions, improving memory and performance efficiency.
- Added accessor methods in `src/hnsw/hnswIndex.ts` for configuration and index metadata (e.g., dimension, max level, entry point).
### Purpose:
Enhanced the HNSW implementation to support optimized indexing and search for large datasets, addressing scalability challenges via product quantization and optional disk-based storage. This update elevates the library's performance and applicability for memory-constrained environments.
### Changes:
- Updated `VERSION` constant in `src/utils/version.ts` from `0.8.0` to `0.8.11`.
### Purpose:
Incremented the project version to reflect new changes, updates, or bug fixes, ensuring proper version tracking and release management.
### Changes:
- Added `backup` command to export database content into a JSON file.
- Includes noun and verb data with metadata.
- Allows specifying a custom output filename (`brainy-backup.json` by default).
- Added `restore` command to import data from a JSON file into the database.
- Supports clearing existing data with `--clear` option before restoration.
- Displays success information for imported nouns and verbs.
- Integrated help text examples for the new commands.
- Enhanced CLI interface with additional indentation for better readability.
- Adjusted `completion-setup` to include autocomplete for backup and restore operations.
### Purpose:
Introduced backup and restore functionality to improve database management and data portability, enabling users to save and load datasets seamlessly for better flexibility in usage and recovery.
### Changes:
- Created `examples/browser_compatible_exports.ts`:
- Introduced browser-friendly exports with polyfill for `Buffer` in browser environments.
- Implemented shims for Node.js modules (`fs`, `util`, and `path`) to avoid compatibility issues.
- Exported browser-compatible types and utilities: `BrainyData`, `NounType`, `VerbType`, graph types, core types, distance functions, embedding utilities, and storage adapters.
- Explicitly excluded Node.js/CLI-specific parts to focus on browser compatibility.
- Added `src/types/brainyDataInterface.ts`:
- Defined `BrainyDataInterface` to break the circular dependency between `brainyData.ts` and `serverSearchAugmentations.ts`.
- Provided methods for managing nodes, relationships, initialization, search, and adding vector data with metadata.
### Purpose:
- Enhanced the codebase to support browser environments by introducing browser-specific exports and avoiding Node.js dependencies.
- Established clearer separation of concerns and modularity by introducing the `BrainyDataInterface` for dependency management, improving maintainability and flexibility.
### Changes:
- Removed the `src/augmentations/llmAugmentations.ts` file:
- Contained cognition augmentations for creating, testing, exporting, and deploying LLM models.
- Included TensorFlow.js-based model configuration, training, testing, and deployment functionalities.
- Implemented various presets (`tiny`, `small`, `medium`, `large`) for simplified LLM model creation.
### Purpose:
This change removes legacy LLM-related augmentations and functionalities no longer relevant to the project's current focus, in alignment with the recent transition to TensorFlow-based embedding workflows. Cleaning up eliminates unused code and maintains a streamlined, up-to-date codebase.
### Changes:
- Added `rollup.unified.js` for building a unified bundle with environment detection (Node.js, Browser, Serverless).
- Added `rollup_config.js` for browser-specific bundle configurations.
- Created `tsconfig.browser.json` for browser builds and `tsconfig.unified.json` for unified builds.
- Implemented custom Rollup plugins:
- **`fixThisReferences`**: Resolves `this` reference issues in TensorFlow files.
- **`nodeModuleShims`**: Provides empty shims for Node.js built-in modules in browser environments.
- Updated dependencies in `package-lock.json` to include Rollup plugins and updated `buffer` dependency for polyfills.
- Created `src/unified.ts` as the unified library entry point with dynamic environment detection.
### Purpose:
- Introduced a unified and browser-compatible build pipeline to ensure `Brainy` can seamlessly operate across multiple environments (Node.js, Browser, Serverless).
- Resolved compatibility issues with Node.js modules and specific library builds.
- Enhanced flexibility and usability for developers working in diverse runtime environments.
Standardized documentation by adding a centered Brainy logo across README files, examples, and guides. Adjusted text formatting for consistency, improved alignment, and readability of feature descriptions and examples.
Implemented new simplified commands (`create-simple`, `train-simple`, `generate-simple`) for easier LLM model management. Added pre-configured model presets (`tiny`, `small`, `medium`, `large`) and enhanced options to streamline usage for beginners. Updated CLI and API documentation with detailed examples. Incremented version to 0.8.0.
Introduced `llm-augmentation.md`, detailing the creation, training, testing, exporting, and deployment of language models using Brainy's graph database. Added examples showcasing practical usage and pipeline integration. Included class implementations in `llmAugmentations.ts`.
Implemented a WebSocket API for real-time communication and REST API routes for CRUD operations on nouns and verbs. Added a server initialization script with WebSocket server integration and improved application structure with modularized routing and services.
Implemented WebSocketConduitAugmentation and WebRTCConduitAugmentation for syncing Brainy instances. Added comprehensive usage examples to showcase data synchronization via WebSocket and WebRTC.
Incremented version to 0.7.5 across `version.ts` and `README.md`. Introduced WebSocket and WebRTC conduit augmentations in codebase and documented usage examples thoroughly in `README.md`.
Upgraded minimum Node.js version from 18.0.0 to 23.11.0 across `README.md`, `package.json`, and `version.ts`. Optimized distance utilities and pipelines to leverage Node.js 23.11+ native performance improvements (e.g., `array.reduce`, WebStreams API). Incremented version to 0.7.4 for consistency.
Updated `README.md` to reflect version `0.7.3` with improved feature descriptions and scaling highlights. Enhanced the versioning script to automatically update the version badge in `README.md` for consistency. Incremented `version.ts` to `0.7.3`.
Enhanced the `README.md` with more engaging and user-friendly descriptions, highlighting Brainy's adaptability and intelligence. Introduced new sections detailing adaptive intelligence, data pipeline enhancements, and use case expansion. Removed outdated dependencies from `package-lock.json`. Updated `version.ts` to `0.7.2`.
Introduced a script to generate a `version.ts` file exporting the package version from `package.json`. This simplifies accessing version data in the CLI, especially for global installations. Included auto-generated `version.ts` in the project structure.
Introduced `generateRandomGraph` method to create randomized graphs with configurable entity counts and types, supporting automated testing and experimentation. Enhanced metadata vectorization logic to handle various metadata formats reliably, ensuring compatibility and fallback mechanisms during embedding. Updated `README.md` with detailed feature descriptions and usage guides.
Enhanced the installation section with guidance on resolving TensorFlow.js dependency conflicts using `--legacy-peer-deps`. Added CLI autocomplete setup instructions to improve user experience. Suppressed specific TensorFlow.js Node.js backend warnings in the embedding utils. Updated dependencies to the latest versions.
Added a comprehensive command-line interface (CLI) for interacting with the Brainy vector database. The CLI supports various operations, including database initialization, adding/searching nouns, managing relationships, and querying database status. Enhanced type validation logic for nouns and verbs to ensure consistency and enforce default types for invalid inputs. Updated test scripts to verify type validation and edge cases.
Updated search methods to support `includeVerbs` for retrieving associated verbs in results. Enhanced edge creation to allow metadata embedding when no vector is provided. Improved query handling and vectorization logic for consistent processing.
Added extensive configuration support for storage adapters, including S3-compatible storage (R2, Amazon S3, Google Cloud Storage). Introduced environment variable fallback for seamless cloud integration. Updated createStorage logic for improved compatibility. Implemented read-only mode to restrict write operations and provide a safer database configuration.
Added the `SequentialPipeline` class for structured augmentation execution. Integrated threading capabilities via the `THREADED` execution mode in `augmentationPipeline`. Enhanced WebSocket support for real-time data processing through pipelines.
Introduced new utilities for detecting execution environments (browser, Node.js, Web Worker) and checking threading availability. Added `workerUtils` for executing functions in Web Workers or Worker Threads seamlessly based on the environment.
Added `FileSystemStorage` and exports for sequential pipelines, including `SequentialPipeline` and related types (`PipelineResult`, `SequentialPipelineOptions`).
Deleted all Firestore augmentation, storage, sync implementations, and related type definitions. Removed the FirestoreSync example file as part of the cleanup for disabled Firestore functionality.
Commented out all Firestore-related code across the project, including sync, storage, and vector search features. Removed Firebase dependencies (`firebase`, `firebase-admin`) from `package.json`. Updated `README.md` to reflect the temporary unavailability of Firestore features.
Introduced Firestore vector search integration using `@firebase/firestore-vector-search`. Added support for restricting searches by specific noun types to optimize performance and relevance. Updated `README.md` to document the new functionality and usage examples.
Refactored memory augmentation methods to return `Promise<AugmentationResponse>` for async handling. Introduced a new `search` method for vector similarity searches within the memory system.
Applied consistent indentation and line-breaking rules to code and text sections in `README.md`, ensuring better formatting and alignment for readability. No functional changes made.
Deleted `examples/externalPlugins.js` and `pluginLoader.ts` as they are no longer used. Updated all `.ts` imports to `.js` across the codebase. Added `MemberOf` to `VerbType` and filtered disabled augmentations in `executeAugmentationPipeline`. Updated documentation to reflect new augmentation registration process. Removed deprecated `allowImportingTsExtensions` from `tsconfig.json`.
Improved the fallback sequence for storage initialization by adding a secondary attempt to use FileSystemStorage in browser environments before defaulting to in-memory storage. Enhanced error handling and logging for unavailable storage options.
Updated error messages for initialization failures to use "BrainyData" for clarity. Enhanced `getStorageStatus` to handle storage adapters without the `getStorageStatus` method, ensuring proper reporting of storage type, usage, and adapter details.
Introduced `pluginLoader.ts` to enable loading and configuring augmentation plugins from external packages. Added `examples/externalPlugins.js` to demonstrate usage. Enhanced storage status reporting in `opfsStorage` and `BrainyData` with detailed storage capacity and usage methods. Updated README with an external plugin usage guide.
Replaced type aliases with interface extensions to enhance type compatibility and readability. Reorganized exports in `index.ts` to streamline imports.
Introduced an optional `requestPersistentStorage` flag to enable persistent storage permission requests in the browser. Updated `OPFSStorage` to include methods for requesting and checking persistent storage status, and modified `createStorage` to handle the new flag.
Introduced direct exports for `GraphNoun`, `GraphVerb`, `EmbeddedGraphVerb`, and related graph types (`Person`, `Place`, `Thing`, etc.) to streamline imports and improve type accessibility.
Introduced direct exports for several augmentation interfaces (e.g., ISenseAugmentation, ICognitionAugmentation) to simplify imports and enhance type accessibility. Updated type imports and exports in relevant files.
Introduced a new README section explaining how to import graph types individually to enable tree shaking. Updated `package.json` exports to support direct graph type imports. Enhanced `GraphNoun` structure by adding optional `label` and `embeddedVerbs` properties while simplifying the `CreatorMetadata` interface.
