David Snelling
3d80df1726
- Fix soft delete functionality by filtering out deleted items in search results - Add support for encrypted configuration storage and retrieval - Copy test suite and vitest config to ensure all functionality works - Maintain all core brainy functionality during repository cleanup
127 lines
4 KiB
Markdown
127 lines
4 KiB
Markdown
# Brainy Test Matrix
|
|
|
|
This document outlines a comprehensive testing strategy for the Brainy vector database, ensuring all functionality works correctly across different environments and configurations.
|
|
|
|
## Test Dimensions
|
|
|
|
The test matrix covers the following dimensions:
|
|
|
|
1. **Public Methods**: All public methods of the BrainyData class
|
|
2. **Storage Adapters**: All supported storage types
|
|
3. **Environments**: All supported runtime environments
|
|
4. **Test Types**: Happy path, error handling, edge cases, performance
|
|
|
|
## Storage Adapters
|
|
|
|
- Memory Storage
|
|
- File System Storage
|
|
- OPFS (Origin Private File System) Storage
|
|
- S3-Compatible Storage (including R2)
|
|
|
|
## Environments
|
|
|
|
- Node.js
|
|
- Browser
|
|
- Web Worker
|
|
- Worker Threads
|
|
|
|
## Test Types
|
|
|
|
- **Happy Path**: Tests with valid inputs and expected behavior
|
|
- **Error Handling**: Tests with invalid inputs, error conditions
|
|
- **Edge Cases**: Tests with boundary values, empty inputs, etc.
|
|
- **Performance**: Tests measuring execution time with various dataset sizes
|
|
|
|
## Core Method Test Matrix
|
|
|
|
| Method | Memory | FileSystem | OPFS | S3 | Error Handling | Edge Cases | Performance |
|
|
|--------|--------|------------|------|----|--------------------|------------|-------------|
|
|
| init() | ✅ | ✅ | ⚠️ | ⚠️ | ⚠️ | ⚠️ | ❌ |
|
|
| add() | ✅ | ✅ | ⚠️ | ❌ | ⚠️ | ⚠️ | ❌ |
|
|
| addBatch() | ✅ | ✅ | ❌ | ❌ | ⚠️ | ❌ | ❌ |
|
|
| search() | ✅ | ✅ | ⚠️ | ❌ | ⚠️ | ⚠️ | ❌ |
|
|
| searchText() | ✅ | ✅ | ❌ | ❌ | ⚠️ | ❌ | ❌ |
|
|
| get() | ✅ | ✅ | ❌ | ❌ | ⚠️ | ❌ | ❌ |
|
|
| delete() | ✅ | ✅ | ❌ | ❌ | ⚠️ | ❌ | ❌ |
|
|
| updateMetadata() | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
|
| relate() | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
|
| findSimilar() | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
|
| clear() | ✅ | ✅ | ⚠️ | ❌ | ❌ | ❌ | ❌ |
|
|
| isReadOnly()/setReadOnly() | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
|
| getStatistics() | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
|
| backup()/restore() | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
|
|
|
Legend:
|
|
- ✅ Well tested
|
|
- ⚠️ Partially tested
|
|
- ❌ Not tested
|
|
|
|
## Environment Test Matrix
|
|
|
|
| Environment | Memory | FileSystem | OPFS | S3 |
|
|
|-------------|--------|------------|------|-----|
|
|
| Node.js | ✅ | ✅ | N/A | ⚠️ |
|
|
| Browser | ⚠️ | N/A | ⚠️ | ❌ |
|
|
| Web Worker | ❌ | N/A | ❌ | ❌ |
|
|
| Worker Threads | ❌ | ⚠️ | N/A | ❌ |
|
|
|
|
## Testing Gaps to Address
|
|
|
|
1. **Error handling scenarios** for each method
|
|
- Invalid inputs
|
|
- Network failures
|
|
- Storage failures
|
|
- Concurrent operation conflicts
|
|
|
|
2. **Edge cases**
|
|
- Empty queries
|
|
- Invalid IDs
|
|
- Maximum size datasets
|
|
- Zero-length vectors
|
|
- Dimension mismatches
|
|
|
|
3. **Different storage adapters**
|
|
- Complete OPFS testing
|
|
- Complete S3 testing
|
|
- Test adapter switching/fallback
|
|
|
|
4. **Multi-environment behavior**
|
|
- Browser-specific tests
|
|
- Web Worker tests
|
|
- Worker Threads tests
|
|
|
|
5. **Read-only mode enforcement**
|
|
- Test all write operations in read-only mode
|
|
|
|
6. **Relationship operations**
|
|
- Complete testing for relate()
|
|
- Complete testing for findSimilar()
|
|
|
|
7. **Metadata handling**
|
|
- Test metadata in add/relate operations
|
|
- Test updateMetadata edge cases
|
|
|
|
8. **Large dataset operations**
|
|
- Performance with 10k+ vectors
|
|
- Memory usage optimization
|
|
|
|
9. **Concurrent operations**
|
|
- Thread safety
|
|
- Race condition handling
|
|
|
|
10. **Statistics and monitoring**
|
|
- Accuracy of statistics
|
|
- Performance impact of statistics tracking
|
|
|
|
## Implementation Plan
|
|
|
|
1. Create error handling tests for core methods
|
|
2. Create edge case tests for core methods
|
|
3. Complete storage adapter tests for OPFS and S3
|
|
4. Create environment-specific test suites
|
|
5. Implement read-only mode tests
|
|
6. Complete relationship operation tests
|
|
7. Create metadata handling tests
|
|
8. Implement performance tests with various dataset sizes
|
|
9. Create concurrent operation tests
|
|
10. Complete statistics and monitoring tests
|