- Updated error handling in `S3CompatibleStorage` to include checks for `NotFound` errors alongside `NoSuchKey` during lock operations.
- Ensured robustness in determining lock existence and managing exceptions related to missing keys.
**Purpose**: Improve resilience of the S3-compatible storage adapter by addressing additional error scenarios, ensuring accurate lock detection and stable operation.
Remove `@modelcontextprotocol/server-sequential-thinking` and its nested dependencies from `devDependencies` in `package-lock.json` as they are no longer required for the project. This reduces the overall package size and improves maintenance.
- Introduced `writeOnly` mode in `BrainyData` allowing optimized data ingestion by skipping index loading and disabling search operations.
- Enhanced `BrainyDataConfig` to include `writeOnly` support and validate compatibility with `readOnly` mode.
- Implemented error handling for search attempts during `writeOnly` mode.
- Updated the README with detailed usage examples for database modes, including `writeOnly` and `readOnly`.
- Added `examples/write-only-mode.js` to demonstrate practical applications of the `writeOnly` mode.
**Purpose**: Optimize memory usage and startup time for data ingestion scenarios by enabling write-only mode with comprehensive documentation and examples.
- Reformatted export statements across `index.ts` and related modules for consistent style, improving code readability and maintainability.
- Updated graph types in `graphTypes.ts` to include additional standardized noun and verb categories, enhancing the flexibility of the type system for graph modeling.
- Replaced `Place` with `Location` and merged similar types (e.g., `Group` into `Collection`) to eliminate redundancy in entity definitions.
- Expanded verb types in `VerbType` to cover more comprehensive use cases, including social, temporal, and ownership relationships.
**Purpose**: Streamline code structure with consistent export formatting, simplify type definitions, and enhance the type system for broader modeling capabilities.
- Deleted `CHANGES.md`, `CHANGES_SUMMARY.md`, `CONCURRENCY_ANALYSIS.md`, `CONCURRENCY_IMPLEMENTATION_SUMMARY.md`, and related developer documentation files.
- Removed redundant or legacy content no longer aligned with the current codebase and workflows.
- Updated repository to reflect streamlined documentation approach, reducing clutter and improving maintainability.
**Purpose**: Simplify and declutter repository by removing obsolete documentation files, ensuring it remains focused and relevant.
- Added `BrainyError` class to classify and handle errors with types like `TIMEOUT`, `NETWORK`, `STORAGE`, `NOT_FOUND`, and `RETRY_EXHAUSTED`. Includes static helper methods for error creation and retry determination.
- Introduced `operationUtils` with utility functions for timeout, retry logic, and exponential backoff. Implements features like `withTimeout`, `withRetry`, and a combined `withTimeoutAndRetry`.
- Updated `S3CompatibleStorage` to leverage new operation utilities for timeout and retry handling, including `StorageOperationExecutors` for clean operation execution.
- Enhanced `storageFactory` to pass `OperationConfig` for configurable timeout and retry behavior.
- Extended `BrainyData` to include timeout and retry policy configuration at initialization.
**Purpose**: Improve storage reliability by introducing configurable and reusable error-handling and operation utilities, reducing code duplication and enhancing maintainability.
- Modified `storageFactory` to ensure `FileSystemStorage` gracefully degrades to `MemoryStorage` in browser environments, with proper warnings added.
- Enhanced `opfsStorage` adapter to support recursive directory removal with the `recursive` option.
- Removed `test-fix.js` script, as it is no longer relevant with recent storage fixes and updates.
**Purpose**: Streamline and ensure cross-environment compatibility for `FileSystemStorage`, while removing outdated test artifacts for better maintainability.
- Introduced `CONCURRENCY_ANALYSIS.md` to outline identified concurrency issues, including statistics handling, index synchronization, and storage contention.
- Added `CONCURRENCY_IMPLEMENTATION_SUMMARY.md` to summarize concurrency improvements, such as distributed locking and change log mechanisms.
- Created `STORAGE_CONCURRENCY_ANALYSIS.md` to evaluate concurrency risks and applied solutions for different storage adapters (`S3CompatibleStorage`, `FileSystemStorage`, `OPFSStorage`, and `MemoryStorage`).
- Updated codebase with changes related to concurrency, including distributed locking, atomic updates, event-driven synchronization, and change log support.
- Refactored tests to verify behavior of new concurrency mechanisms, including robust error handling and cleanup functions.
**Purpose**: Provides comprehensive documentation and implementation details to ensure robust concurrency handling in multi-instance, high-throughput environments.
- Added patterns for temporary test files (`test*.js`, `test*.ts`, `temp-test*.js`, `temp-test*.ts`, etc.) to `.gitignore`.
- Included directories `/temp/` and `/temp-tests/` to prevent accidental commits of temporary or debug-related files.
**Purpose**: Ensure cleaner version control by excluding unnecessary temporary and debug files created during development.
- Updated test results output path in `vitest.config.ts` to `./tests/results/test-results.json` for better organization.
- Modified `.gitignore` to include `/tests/results/` and its `.json` files to avoid committing test artifacts.
**Purpose**: Improves test results file structure and ensures consistent exclusion of test artifact files from version control.
- Updated `test-results.json` to reflect the latest test outcomes.
- Ensures accuracy of recorded test results after recent changes.
**Purpose**: Keep test result records up-to-date for reliable tracking and reference.
- **Removed Files**:
- Deleted outdated statistics documentation files (`statistics.md`, `statistics-flush-solution.md`, `statistics-summary.md`) to clean up the repository and avoid confusion.
- **Added Standards**:
- Introduced `DOCUMENTATION_STANDARDS.md` to outline naming conventions and troubleshooting practices for more consistent and maintainable project documentation.
- **Tests**:
- Added a new test file `edge-cases.test.ts` to verify handling of edge cases, ensuring robust behavior against boundary values and invalid inputs.
**Purpose**: Cleans up deprecated documentation while introducing concrete standards for maintaining and updating documentation. Enhances test coverage for unusual or boundary inputs, improving overall system resilience.
- **Integration Tests**:
- Introduced `api-integration.test.ts` to validate API functionality:
- Verifies text insertion, vector embedding generation, and search operations.
- Confirms HNSW index correctness for vector similarity search.
- Ensures no dimensional mismatches in embeddings.
- **Test Server**:
- Added test server utilizing Express for endpoint simulation (`/insert` and `/search/text`).
- **Dependencies**:
- Introduced `express` and `node-fetch` as new dependencies for testing purposes.
- **Vitest Fix**:
- Updated `vitest.config.ts` to resolve the `process.memoryUsage` error by setting `logHeapUsage: false`.
- **Package Updates**:
- Modified `package-lock.json` to include newly added dependencies and updates.
**Purpose**: Guarantees the stability of core API endpoints and vector-related functionality, ensuring reliable behavior for end-to-end scenarios.
- **Core**:
- Removed the `dimensions` property from `BrainyData` initialization in `server.ts` as it was unused and redundant.
- Simplified code structure and reduced potential confusion by avoiding unnecessary arguments in object initialization.
**Purpose**: Improve code clarity and maintainability by removing unused properties from the `BrainyData` setup, aligning with standardized dimension handling approaches.
- **Tests**:
- Introduced `database-operations.test.ts` to validate core database functionalities, including initialization, CRUD operations, statistics retrieval, and search capabilities.
- Added `dimension-standardization.test.ts` to ensure vector dimension consistency (fixed at 512) throughout operations like embedding, configuration, and validation.
- Enhanced test cases to include scenarios for adding, retrieving, and handling errors for incorrect vector dimensions.
- **Documentation**:
- Created `VECTOR_DIMENSION_STANDARDIZATION.md` to detail the transition to standardizing vectors to 512 dimensions, rationale for the change, potential impacts, and migration steps.
- Includes best practices for handling vectors and utilizing the built-in embedding functions.
**Purpose**: Improve system robustness with comprehensive test coverage focusing on critical database and vector operations while providing clear documentation for developers to adapt to the standardized vector dimensions.
- **Core**:
- Added `check-database.js` to verify database status and validate search functionality.
- Created `fix-dimension-mismatch.js` to handle re-embedding of existing data to resolve dimension mismatch from 3 to 512.
- Improved test cases by updating vector operations to support 512 dimensions, replacing previously hardcoded dimensions.
- **Migration**:
- Developed `DIMENSION_MISMATCH_SUMMARY.md`, detailing the root cause, solution, and preventive strategies for dimension mismatch issues.
- Added `production-migration-guide.md` for structured production migration with detailed steps on re-embedding strategies, batching, and error handling.
- **Tests**:
- Enhanced test coverage with 512-dimensional vector validation.
- Introduced helper functions for consistent vector testing behavior and streamlined search test cases.
- **Documentation**:
- Updated project documentation to highlight the resolution process for dimension mismatches, emphasizing preventive mechanisms such as auto-migration and version tracking.
**Purpose**: Address critical dimension mismatch issues caused by embedding changes, restore functionality, and provide a roadmap for robust prevention strategies and migration processes.
- **Storage**: Replaced all instances of `HNSWNoun_internal` and `Verb` type aliases with their original equivalents (`HNSWNoun` and `GraphVerb`) in `MemoryStorage` adapter. Simplified method definitions and internal logic by directly using the original types.
- **Code Cleanup**:
- Removed unused type alias declarations to reduce code clutter and improve readability.
- Adjusted method parameters and return types accordingly to maintain consistency.
**Purpose**: Simplify the codebase by removing redundant type aliasing, ensuring consistency and better readability across storage adapter logic. Reduces potential confusion and streamlines type usage.
- **Examples**: Added a new `flush-statistics-example.js` script to demonstrate the usage of the `flushStatistics` method for ensuring updated statistics after data insertion.
- **Core**:
- Implemented `flushStatistics` in the `BrainyData` class to allow immediate flushing of statistics to storage.
- Updated `BaseStorageAdapter` with `flushStatisticsToStorage` to support flushing cached statistics.
- Modified the `shutDown` method to ensure statistics are flushed before database shutdown.
- **Documentation**: Added `statistics-flush-solution.md` to explain the batch update mechanism, the issue with delayed statistics updates, and how to manually flush statistics in storage.
**Purpose**: Provide users with the ability to manually flush statistics for real-time accuracy, particularly useful for systems relying on immediate updates. Improved documentation and examples to guide developers in implementing and using this functionality effectively.
- **Documentation**: Added a new `statistics-summary.md` file detailing how statistics are tracked, updated, and stored within the system. Includes examples of API usage, implementation details, and consumer use cases.
- **Purpose**: Provide clear guidance for developers and stakeholders on the statistics system's functionality, including real-time updates, service-specific breakdowns, and persistent storage mechanisms. Aids in understanding and leveraging the system effectively.
- **Core**: Improved verb creation logic by adding `createdAt`, `updatedAt`, and `createdBy` attributes. These fields include timestamped metadata (`seconds`, `nanoseconds`) and source augmentation/service information for better tracking.
- **Storage**: Refactored `BaseStorage` methods to utilize internal variants (e.g., `saveVerb_internal`, `getNoun_internal`). Added support for new verb attributes while maintaining backward compatibility with existing data structures.
- **Tests**:
- Updated `s3-storage.test.ts` and `opfs-storage.test.ts` to validate changes in verb attributes such as timestamps and augmentation metadata.
- Added assertions for `createdAt`, `updatedAt`, and `createdBy` fields in test cases.
- **Cleanup**: Replaced ambiguous type aliases like `Edge` and `HNSWNode` with clearer equivalents (`Verb` and `HNSWNoun_internal`) for consistency across storage adapters.
**Purpose**: Enhance metadata tracking and standardize attribute handling across storage and core modules to ensure accurate and consistent data throughout the system.
- **Code Cleanup**: Removed unnecessary trailing whitespaces across `src/brainyData.ts`. Adjusted formatting for inline object spreads to maintain a consistent coding style.
- **Purpose**: Enhance code readability and ensure adherence to formatting standards without altering functionality.
- **Tests**: Added new `statistics-storage.test.ts` to validate statistics storage functionality across scenarios including saving, retrieving, time-based partitioning, and backward compatibility. Ensured tests dynamically handle missing environment variables by skipping S3-related tests when credentials are unavailable.
- **Docs**: Enhanced `statistics.md` with detailed explanations of scalability improvements, including adaptive flush timing, batched updates, and time-based partitioning. Improved readability and structure.
- **Storage**: Updated all storage adapters to integrate time-based partitioning and maintain backward compatibility with legacy statistics storage formats.
- **Dependencies**: Added `dotenv` to support environmental variable management for storage adapter tests.
**Purpose**: Strengthen system reliability by adding comprehensive test coverage for statistics storage, improve scalability documentation, and ensure consistency across storage adapters with robust implementations.
- **Core**: Introduced a `getCurrentAugmentation` method for detecting active augmentation names. Updated metadata handling to include `createdBy`, `createdAt`, and `updatedAt` attributes for improved tracking.
- **Storage**: Added support for service-based statistics tracking with new methods such as `incrementStatistic`, `decrementStatistic`, and `updateHnswIndexSize`. Implemented persistence for statistics in storage adapters.
- **Statistics**: Enhanced `getStatistics` functionality to provide service-specific breakdowns and support filtering by services. Improved noun, verb, and metadata tracking mechanisms.
- **Search**: Added `service` option to filter results during searches for nouns, verbs, and metadata, ensuring accurate service-based query results.
- **Refactor**: Simplified search logic by integrating HNSW index filtering for better performance when retrieving service-specific results.
- **Tests**: Added comprehensive test coverage for service-level statistics and filtering by service.
**Purpose**: Improve service-level data tracking and analytics while enhancing functionality for filtering and maintaining metadata accuracy to support detailed insights for diverse use cases.
- **Removal**: Deleted `FileSystemStorage` and `OPFSStorage` adapters from the `/storage` directory.
- **Refactor**: Simplifies the codebase by eliminating unused, redundant, or outdated storage implementations.
- **Impact**: Ensures maintainability by focusing on actively supported storage solutions.
**Purpose**: Streamline the project by removing deprecated or unused storage adapters, reducing complexity and maintenance overhead.
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
- Updated `tsconfig.unified.json`, `tsconfig.browser.json`, and `tsconfig.json` to include `"DOM.Asynciterable"` in the `lib` option.
- Ensured compatibility with asynchronous iteration patterns in DOM APIs.
**Purpose**: Enable support for DOM asynchronous iterable utilities, improving compatibility with modern TypeScript and DOM usage scenarios.
- **Storage**: Removed redundant storage implementation files from `/storage` directory while retaining necessary `baseStorage.ts`. Verified no impact on functionality with passing tests.
- **Documentation**: Consolidated multiple overlapping `.md` files into a single `statistics.md` file. Added structured sections for easier reference. Applied markdown naming conventions for clarity.
- **Project**: Introduced `CHANGES_SUMMARY.md` and `MARKDOWN_CONVENTIONS.md` to document changes and naming guidelines, improving project organization. Updated `demo.md` file naming for consistency.
**Purpose**: Simplify the codebase by removing duplicates, enhance maintainability through consolidated and structured documentation, and improve developer experience with clear guidelines and better organization.
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
- **New Method**:
- Introduced a new `getStatistics` method in `brainyData.ts` to retrieve key database metrics:
- Counts for nouns, verbs, metadata entries, and HNSW index size.
- **Error Handling**:
- Added try-catch blocks to ensure robust error management when fetching metadata or logging failures.
- **Purpose**:
- Enhances observability of the database state, providing valuable insights for diagnostics and monitoring.
- Updated version to `0.18.0` in `cli-package/package.json` and `package-lock.json`.
- Upgraded `@soulcraft/brainy` dependency to `^0.17.0` for compatibility with recent changes.
- Synced package dependencies and resolved lockfile integrity for deployment.
**Purpose**: Prepare for release by synchronizing versioning across the CLI package and updating dependencies for compatibility.
