- **Documentation Additions**:
- Introduced `SEARCH_AND_METADATA_GUIDE.md` to provide an in-depth guide on Brainy's search and metadata retrieval system:
- Detailed explanation of search workflows, metadata structures (`GraphNoun`, `GraphVerb`), and core components like `SearchResult`.
- Usage examples showcasing search queries, filtering by noun/verb types, and advanced features like multi-modal search.
- Included performance tips on caching, HNSW indexing, lazy loading, and augmentation pipeline.
- **Storage System Updates**:
- Enhanced memory and file storage adapters to support dedicated noun and verb metadata handling:
- Added methods `saveN
- **Compatibility Enhancements**:
- Added support to detect and inject missing `"format"` field in `model.json` files for TensorFlow.js compatibility.
- Modified model loading logic to handle both `tfjs-graph-model` and `tfjs-layers-model` formats.
- **New Features**:
- Introduced additional fallback paths for locating models to increase reliability in varying environments.
- Added support for mock implementations of the Universal Sentence Encoder in test environments.
- **Bug Fixes**:
- Fixed module loading resolution in `FileSystemStorage` with improved initialization and error handling for Node.js environments.
- Resolved issues with test assertions to improve validation logic in core tests.
**Purpose**: Improve model loading reliability, expand compatibility with TensorFlow.js models, and enhance test environment support.
- Introduced `CONCURRENCY_ANALYSIS.md` to outline identified concurrency issues, including statistics handling, index synchronization, and storage contention.
- Added `CONCURRENCY_IMPLEMENTATION_SUMMARY.md` to summarize concurrency improvements, such as distributed locking and change log mechanisms.
- Created `STORAGE_CONCURRENCY_ANALYSIS.md` to evaluate concurrency risks and applied solutions for different storage adapters (`S3CompatibleStorage`, `FileSystemStorage`, `OPFSStorage`, and `MemoryStorage`).
- Updated codebase with changes related to concurrency, including distributed locking, atomic updates, event-driven synchronization, and change log support.
- Refactored tests to verify behavior of new concurrency mechanisms, including robust error handling and cleanup functions.
**Purpose**: Provides comprehensive documentation and implementation details to ensure robust concurrency handling in multi-instance, high-throughput environments.
- Updated `test-results.json` to reflect the latest test outcomes.
- Ensures accuracy of recorded test results after recent changes.
**Purpose**: Keep test result records up-to-date for reliable tracking and reference.
- **Removed Files**:
- Deleted outdated statistics documentation files (`statistics.md`, `statistics-flush-solution.md`, `statistics-summary.md`) to clean up the repository and avoid confusion.
- **Added Standards**:
- Introduced `DOCUMENTATION_STANDARDS.md` to outline naming conventions and troubleshooting practices for more consistent and maintainable project documentation.
- **Tests**:
- Added a new test file `edge-cases.test.ts` to verify handling of edge cases, ensuring robust behavior against boundary values and invalid inputs.
**Purpose**: Cleans up deprecated documentation while introducing concrete standards for maintaining and updating documentation. Enhances test coverage for unusual or boundary inputs, improving overall system resilience.
- **Integration Tests**:
- Introduced `api-integration.test.ts` to validate API functionality:
- Verifies text insertion, vector embedding generation, and search operations.
- Confirms HNSW index correctness for vector similarity search.
- Ensures no dimensional mismatches in embeddings.
- **Test Server**:
- Added test server utilizing Express for endpoint simulation (`/insert` and `/search/text`).
- **Dependencies**:
- Introduced `express` and `node-fetch` as new dependencies for testing purposes.
- **Vitest Fix**:
- Updated `vitest.config.ts` to resolve the `process.memoryUsage` error by setting `logHeapUsage: false`.
- **Package Updates**:
- Modified `package-lock.json` to include newly added dependencies and updates.
**Purpose**: Guarantees the stability of core API endpoints and vector-related functionality, ensuring reliable behavior for end-to-end scenarios.
- Added `BaseStorage` abstract class to provide common functionality for storage adapters.
- Implemented `FileSystemStorage` adapter for Node.js environments, supporting persistent file-based storage.
- Added directory structure management methods to `FileSystemStorage` (`ensureDirectoryExists`, `clear`, etc.).
- Implemented `MemoryStorage` adapter for in-memory storage suitable for non-persistent environments.
- Supported CRUD operations for nouns, verbs, and metadata in both `FileSystemStorage` and `MemoryStorage`.
- Included utility methods like `mapToObject` for serialization and deserialization of complex data structures.
Purpose: Provide extensible, reusable storage solutions with both persistent and non-persistent options for diverse application needs.
