### Changes:
- Enhanced `BrainyData` class to support `HNSWIndexOptimized` alongside the standard HNSW index.
- Added a new configuration parameter `hnswOptimized` for enabling optimized indexing.
- Created utility methods for handling optimized index-specific functionalities such as memory usage tracking and disk-based index management.
- Updated vector addition, retrieval, and management methods to accommodate both standard and optimized indices.
- Introduced support for using pre-defined IDs during vector and relationship creation.
- Standardized metadata handling for error resilience and embedding integrations.
- Implemented a `getAllNouns` method to retrieve all indexed data.
- Added validation for vector dimensions during relationship creation to ensure consistency.
- Refactored method signatures to improve readability and alignment.
### Purpose:
Improved the BrainyData API to support optimized indexing functionality powered by `HNSWIndexOptimized`, enhancing scalability for large datasets while maintaining backward compatibility. Elevated overall code structure and usability.
### Changes:
- Added `HNSWIndexOptimized` class in `src/hnsw/hnswIndexOptimized.ts`:
- Supports large datasets with product quantization for dimensionality reduction.
- Enables disk-based index storage via a configurable storage adapter.
- Introduces `HNSWOptimizedConfig` interface for advanced configuration:
- Includes memory thresholds, product quantization (subvectors, centroids), and disk-based indexing options.
- Implements methods for adding, searching, and managing indexed vectors with enhanced scalability.
- Introduced `ProductQuantizer` utility:
- Handles vector quantization with training and reconstruction capabilities.
- Reduces vector dimensions, improving memory and performance efficiency.
- Added accessor methods in `src/hnsw/hnswIndex.ts` for configuration and index metadata (e.g., dimension, max level, entry point).
### Purpose:
Enhanced the HNSW implementation to support optimized indexing and search for large datasets, addressing scalability challenges via product quantization and optional disk-based storage. This update elevates the library's performance and applicability for memory-constrained environments.
### Changes:
- Updated `VERSION` constant in `src/utils/version.ts` from `0.8.0` to `0.8.11`.
### Purpose:
Incremented the project version to reflect new changes, updates, or bug fixes, ensuring proper version tracking and release management.
### Changes:
- Added `backup` command to export database content into a JSON file.
- Includes noun and verb data with metadata.
- Allows specifying a custom output filename (`brainy-backup.json` by default).
- Added `restore` command to import data from a JSON file into the database.
- Supports clearing existing data with `--clear` option before restoration.
- Displays success information for imported nouns and verbs.
- Integrated help text examples for the new commands.
- Enhanced CLI interface with additional indentation for better readability.
- Adjusted `completion-setup` to include autocomplete for backup and restore operations.
### Purpose:
Introduced backup and restore functionality to improve database management and data portability, enabling users to save and load datasets seamlessly for better flexibility in usage and recovery.
### Changes:
- Created `examples/browser_compatible_exports.ts`:
- Introduced browser-friendly exports with polyfill for `Buffer` in browser environments.
- Implemented shims for Node.js modules (`fs`, `util`, and `path`) to avoid compatibility issues.
- Exported browser-compatible types and utilities: `BrainyData`, `NounType`, `VerbType`, graph types, core types, distance functions, embedding utilities, and storage adapters.
- Explicitly excluded Node.js/CLI-specific parts to focus on browser compatibility.
- Added `src/types/brainyDataInterface.ts`:
- Defined `BrainyDataInterface` to break the circular dependency between `brainyData.ts` and `serverSearchAugmentations.ts`.
- Provided methods for managing nodes, relationships, initialization, search, and adding vector data with metadata.
### Purpose:
- Enhanced the codebase to support browser environments by introducing browser-specific exports and avoiding Node.js dependencies.
- Established clearer separation of concerns and modularity by introducing the `BrainyDataInterface` for dependency management, improving maintainability and flexibility.
### Changes:
- Removed the `src/augmentations/llmAugmentations.ts` file:
- Contained cognition augmentations for creating, testing, exporting, and deploying LLM models.
- Included TensorFlow.js-based model configuration, training, testing, and deployment functionalities.
- Implemented various presets (`tiny`, `small`, `medium`, `large`) for simplified LLM model creation.
### Purpose:
This change removes legacy LLM-related augmentations and functionalities no longer relevant to the project's current focus, in alignment with the recent transition to TensorFlow-based embedding workflows. Cleaning up eliminates unused code and maintains a streamlined, up-to-date codebase.
### Changes:
- Updated README.md:
- Refined the project description to highlight its lightweight and powerful nature.
- Enhanced feature list, replacing outdated entries with modern updates (e.g., TensorFlow integration replacing LLM creation).
- Included detailed installation and usage instructions for TensorFlow.js support and embedding workflows.
- Added "Usage Options" section to define different import methods and supported environments.
- Expanded "Backup and Restore" details, showcasing CLI and scripting examples.
- Introduced optimized HNSW indexing for large datasets.
- Replaced examples with a unified and simplified demonstration approach.
- Embedded advanced build and environment-specific configurations for better clarity.
- Removed outdated content related to language model generation, including LLM-related commands and examples.
- Added README.demo.md with full instructions for running the interactive demo.
### Purpose:
This update modernizes the documentation and focuses on Brainy’s current capabilities, simplifying onboarding for new users while removing deprecated features. It also streamlines project understanding, enabling developers to better utilize its features in diverse environments.
### Changes:
- Added `rollup.unified.js` for building a unified bundle with environment detection (Node.js, Browser, Serverless).
- Added `rollup_config.js` for browser-specific bundle configurations.
- Created `tsconfig.browser.json` for browser builds and `tsconfig.unified.json` for unified builds.
- Implemented custom Rollup plugins:
- **`fixThisReferences`**: Resolves `this` reference issues in TensorFlow files.
- **`nodeModuleShims`**: Provides empty shims for Node.js built-in modules in browser environments.
- Updated dependencies in `package-lock.json` to include Rollup plugins and updated `buffer` dependency for polyfills.
- Created `src/unified.ts` as the unified library entry point with dynamic environment detection.
### Purpose:
- Introduced a unified and browser-compatible build pipeline to ensure `Brainy` can seamlessly operate across multiple environments (Node.js, Browser, Serverless).
- Resolved compatibility issues with Node.js modules and specific library builds.
- Enhanced flexibility and usability for developers working in diverse runtime environments.
### Changes:
- Deleted the following outdated example files from the `examples` directory:
- `buildTimeRegistration.js`
- `conduitAugmentationExample.js`
- `configurationTest.js`
- `dataInspectionExample.js`
- `import-graphTypes.js`
- `browser-server-search/index.html`
### Purpose:
These example files were removed as they were no longer relevant or up-to-date with the current state of the project. This cleanup ensures that users referencing examples have access to accurate and relevant documentation and code, reducing potential confusion.
### Changes:
- Adjusted the main project description in `README.md` for a more engaging tone by adding "A fun" to emphasize the platform's appeal.
### Purpose:
This update aims to make the project description more approachable and engaging for potential users while maintaining clarity about its capabilities.
Introduced `encode-image.js` to convert images to Base64 and create corresponding HTML `<img>` tags. Outputs a sample encoded result to `encoded-image.html`.
Standardized documentation by adding a centered Brainy logo across README files, examples, and guides. Adjusted text formatting for consistency, improved alignment, and readability of feature descriptions and examples.
Implemented new simplified commands (`create-simple`, `train-simple`, `generate-simple`) for easier LLM model management. Added pre-configured model presets (`tiny`, `small`, `medium`, `large`) and enhanced options to streamline usage for beginners. Updated CLI and API documentation with detailed examples. Incremented version to 0.8.0.
Introduced `llm-augmentation.md`, detailing the creation, training, testing, exporting, and deployment of language models using Brainy's graph database. Added examples showcasing practical usage and pipeline integration. Included class implementations in `llmAugmentations.ts`.
Added several badges for TypeScript, TensorFlow, and Cartographer. Expanded on iConduit augmentations with WebSocket and WebRTC details, revised feature descriptions, and introduced a related projects section. Updated examples for improved clarity.
Implemented a WebSocket API for real-time communication and REST API routes for CRUD operations on nouns and verbs. Added a server initialization script with WebSocket server integration and improved application structure with modularized routing and services.
Introduced a cloud deployment wrapper for Brainy supporting AWS Lambda, Google Cloud Run, and Cloudflare Workers. Added configuration options, deployment scripts, and API reference. Updated `README.md` with contribution guidelines and cloud deployment documentation.
Implemented WebSocketConduitAugmentation and WebRTCConduitAugmentation for syncing Brainy instances. Added comprehensive usage examples to showcase data synchronization via WebSocket and WebRTC.
