**chore(archive): remove outdated documentation and summaries**
- Deleted the following obsolete files: - `CHANGES.md`, `changes-summary.md`, `CHANGES_SUMMARY.md`: Contained redundant or outdated change logs and implementation summaries. - `COMPATIBILITY.md`: Detailed compatibility behavior no longer relevant after environment detection updates. - `fix-documentation.md`: Addressed a resolved issue regarding `process.memoryUsage` errors in testing. - `DIMENSION_MISMATCH_SUMMARY.md`: Provided a legacy summary of resolved embedding dimension mismatch issues. - `demo.md`: Documented an outdated demo process for testing Brainy features. - `CONCURRENCY_IMPLEMENTATION_SUMMARY.md`: Summarized already-documented concurrency features. - `IMPLEMENTATION_SUMMARY.md`: Detailed an obsolete implementation of optional model bundling. - Purpose: - Streamline and declutter archive by removing redundant or outdated documentation. - Align repository with current feature set and documentation standards.
This commit is contained in:
parent
082a055e75
commit
63bc50d5ad
22 changed files with 234 additions and 1122 deletions
168
docs/COMPATIBILITY.md
Normal file
168
docs/COMPATIBILITY.md
Normal file
|
|
@ -0,0 +1,168 @@
|
|||
# Brainy Compatibility Across Environments
|
||||
|
||||
This document outlines Brainy's compatibility across different JavaScript environments and how it adapts to each environment.
|
||||
|
||||
## Environment Detection
|
||||
|
||||
Brainy automatically detects the environment it's running in:
|
||||
|
||||
```javascript
|
||||
// Method to detect the current environment
|
||||
function detectEnvironment() {
|
||||
if (typeof window !== 'undefined' && typeof document !== 'undefined') {
|
||||
return 'BROWSER';
|
||||
} else if (typeof self !== 'undefined' && typeof window === 'undefined') {
|
||||
// In a worker environment, self is defined but window is not
|
||||
return 'WORKER';
|
||||
} else {
|
||||
return 'NODE';
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Cache Size Detection
|
||||
|
||||
Brainy's cache manager adapts its cache size based on the detected environment:
|
||||
|
||||
### Node.js Environment
|
||||
|
||||
In Node.js, Brainy uses fixed default memory values to ensure compatibility with ES modules:
|
||||
|
||||
```javascript
|
||||
// Use conservative defaults that don't require OS module
|
||||
// These values are reasonable for most systems
|
||||
const estimatedTotalMemory = 8 * 1024 * 1024 * 1024; // Assume 8GB total
|
||||
const estimatedFreeMemory = 4 * 1024 * 1024 * 1024; // Assume 4GB free
|
||||
```
|
||||
|
||||
This approach ensures compatibility with both CommonJS and ES modules without requiring dynamic imports or the `os` module.
|
||||
|
||||
### Browser Environment
|
||||
|
||||
In browsers, Brainy uses the `navigator.deviceMemory` API when available:
|
||||
|
||||
```javascript
|
||||
if (environment === 'BROWSER' && navigator.deviceMemory) {
|
||||
// Base entries per GB
|
||||
let entriesPerGB = 500;
|
||||
|
||||
// Adjust based on operating mode and dataset size
|
||||
if (isReadOnly) {
|
||||
entriesPerGB = 800; // More aggressive caching in read-only mode
|
||||
|
||||
if (isLargeDataset) {
|
||||
entriesPerGB = 1000; // Even more aggressive for large datasets
|
||||
}
|
||||
} else if (isLargeDataset) {
|
||||
entriesPerGB = 600; // Slightly more aggressive for large datasets
|
||||
}
|
||||
|
||||
// Calculate based on device memory
|
||||
const browserCacheSize = Math.max(navigator.deviceMemory * entriesPerGB, 1000);
|
||||
|
||||
// If we know the total dataset size, cap at a reasonable percentage
|
||||
if (totalItems > 0) {
|
||||
// In read-only mode, we can cache a larger percentage
|
||||
const maxPercentage = isReadOnly ? 0.4 : 0.25;
|
||||
const maxItems = Math.ceil(totalItems * maxPercentage);
|
||||
|
||||
// Return the smaller of the two to avoid excessive memory usage
|
||||
return Math.min(browserCacheSize, maxItems);
|
||||
}
|
||||
|
||||
return browserCacheSize;
|
||||
}
|
||||
```
|
||||
|
||||
If `navigator.deviceMemory` is not available, it falls back to conservative defaults.
|
||||
|
||||
### Worker Environment
|
||||
|
||||
For Web Workers, Brainy uses a more conservative approach:
|
||||
|
||||
```javascript
|
||||
if (environment === 'WORKER') {
|
||||
// Workers typically have limited memory, be conservative
|
||||
return isReadOnly ? 2000 : 1000;
|
||||
}
|
||||
```
|
||||
|
||||
## Storage Type Detection
|
||||
|
||||
Brainy also adapts its storage strategy based on the environment:
|
||||
|
||||
### Warm Storage
|
||||
|
||||
```javascript
|
||||
// Method to detect the appropriate warm storage type
|
||||
function detectWarmStorageType() {
|
||||
if (environment === 'BROWSER') {
|
||||
// Use OPFS if available, otherwise use memory
|
||||
if ('storage' in navigator && 'getDirectory' in navigator.storage) {
|
||||
return 'OPFS';
|
||||
}
|
||||
return 'MEMORY';
|
||||
} else if (environment === 'WORKER') {
|
||||
// Use OPFS if available, otherwise use memory
|
||||
if ('storage' in self && 'getDirectory' in self.storage) {
|
||||
return 'OPFS';
|
||||
}
|
||||
return 'MEMORY';
|
||||
} else {
|
||||
// In Node.js, use filesystem
|
||||
return 'FILESYSTEM';
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cold Storage
|
||||
|
||||
```javascript
|
||||
// Method to detect the appropriate cold storage type
|
||||
function detectColdStorageType() {
|
||||
if (environment === 'BROWSER') {
|
||||
// Use OPFS if available, otherwise use memory
|
||||
if ('storage' in navigator && 'getDirectory' in navigator.storage) {
|
||||
return 'OPFS';
|
||||
}
|
||||
return 'MEMORY';
|
||||
} else if (environment === 'WORKER') {
|
||||
// Use OPFS if available, otherwise use memory
|
||||
if ('storage' in self && 'getDirectory' in self.storage) {
|
||||
return 'OPFS';
|
||||
}
|
||||
return 'MEMORY';
|
||||
} else {
|
||||
// In Node.js, use S3 if configured, otherwise filesystem
|
||||
return 'S3';
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Compatibility Summary
|
||||
|
||||
| Feature | Node.js | Browser | Web Worker |
|
||||
|---------|---------|---------|------------|
|
||||
| Environment Detection | ✅ | ✅ | ✅ |
|
||||
| Cache Size Detection | ✅ (Fixed defaults) | ✅ (deviceMemory API) | ✅ (Conservative) |
|
||||
| Warm Storage | Filesystem | OPFS/Memory | OPFS/Memory |
|
||||
| Cold Storage | S3/Filesystem | OPFS/Memory | OPFS/Memory |
|
||||
| ES Module Support | ✅ | ✅ | ✅ |
|
||||
|
||||
## Recommendations
|
||||
|
||||
1. **Node.js Applications**:
|
||||
- No special configuration needed
|
||||
- Works with both CommonJS and ES modules
|
||||
|
||||
2. **Browser Applications**:
|
||||
- For optimal performance, use in browsers that support the `navigator.deviceMemory` API
|
||||
- Falls back gracefully in older browsers
|
||||
|
||||
3. **Worker Applications**:
|
||||
- Works in both dedicated and shared workers
|
||||
- Uses conservative cache sizes to avoid memory issues
|
||||
|
||||
4. **Memory-Constrained Environments**:
|
||||
- Consider setting a smaller `hotCacheMaxSize` in the options
|
||||
- Example: `new BrainyData({ hotCacheMaxSize: 500 })`
|
||||
|
|
@ -18,39 +18,36 @@ The documentation has been reorganized from 29 scattered markdown files at the r
|
|||
```
|
||||
docs/
|
||||
├── technical/ # Technical documentation and analysis
|
||||
│ ├── concurrency-analysis.md
|
||||
│ ├── storage-concurrency-analysis.md
|
||||
│ ├── threading.md
|
||||
│ ├── statistics.md
|
||||
│ ├── testing.md
|
||||
│ ├── vitest-improvements.md
|
||||
│ ├── realtime-updates.md
|
||||
│ ├── metadata-handling.md
|
||||
│ ├── vector-dimension-standardization.md
|
||||
│ ├── use-model-loading-explanation.md
|
||||
│ ├── dimension-mismatch-summary.md
|
||||
│ ├── storage-testing.md
|
||||
│ ├── technical-guides.md
|
||||
│ └── concurrency-implementation-summary.md
|
||||
│ ├── CONCURRENCY_ANALYSIS.md
|
||||
│ ├── STORAGE_CONCURRENCY_ANALYSIS.md
|
||||
│ ├── THREADING.md
|
||||
│ ├── STATISTICS.md
|
||||
│ ├── TESTING.md
|
||||
│ ├── VITEST_IMPROVEMENTS.md
|
||||
│ ├── REALTIME_UPDATES.md
|
||||
│ ├── METADATA_HANDLING.md
|
||||
│ ├── VECTOR_DIMENSION_STANDARDIZATION.md
|
||||
│ ├── USE_MODEL_LOADING_EXPLANATION.md
|
||||
│ ├── STORAGE_TESTING.md
|
||||
│ ├── TECHNICAL_GUIDES.md
|
||||
│ ├── ENVIRONMENT_TESTING.md
|
||||
│ └── SCALING_STRATEGY.md
|
||||
├── development/ # Development and contributor documentation
|
||||
│ ├── developers.md
|
||||
│ ├── documentation-standards.md
|
||||
│ ├── markdown-conventions.md
|
||||
│ ├── expected-test-messages.md
|
||||
│ └── pretty-test-reporter.md
|
||||
│ ├── DEVELOPERS.md
|
||||
│ ├── DOCUMENTATION_STANDARDS.md
|
||||
│ ├── MARKDOWN_CONVENTIONS.md
|
||||
│ ├── EXPECTED_TEST_MESSAGES.md
|
||||
│ └── PRETTY_TEST_REPORTER.md
|
||||
└── guides/ # User guides and migration documentation
|
||||
└── production-migration-guide.md
|
||||
├── cache-configuration.md
|
||||
├── hnsw-field-search.md
|
||||
├── json-document-search.md
|
||||
├── model-management.md
|
||||
├── optional-model-bundling.md
|
||||
├── production-migration-guide.md
|
||||
└── service-identification.md
|
||||
```
|
||||
|
||||
### Archive Directory
|
||||
```
|
||||
archive/ # Archived and temporary files
|
||||
├── changes.md # Old detailed changelog
|
||||
├── changes-summary.md # Old changelog summary
|
||||
├── demo.md # Demo documentation
|
||||
├── fix-documentation.md # Temporary fix notes
|
||||
└── test-issue-summary.md # Test issue summary
|
||||
```
|
||||
|
||||
## CHANGELOG.md Management
|
||||
|
||||
|
|
@ -86,12 +83,12 @@ The CHANGELOG.md is automatically used for:
|
|||
|
||||
## Benefits of New Structure
|
||||
|
||||
1. **Clean Root Directory**: Reduced from 29 to 4 essential markdown files
|
||||
2. **Better Organization**: Logical categorization of documentation
|
||||
1. **Clean Root Directory**: Only 4 essential markdown files at root level
|
||||
2. **Better Organization**: Logical categorization of documentation in docs/ subdirectories
|
||||
3. **GitHub Compliance**: Follows GitHub and NPM best practices
|
||||
4. **Automated Maintenance**: Changelog updates are automated
|
||||
5. **Easy Navigation**: Clear directory structure for different doc types
|
||||
6. **Historical Preservation**: Old documentation archived, not lost
|
||||
6. **Reduced Clutter**: Temporary and outdated files have been removed
|
||||
|
||||
## Workflow for Contributors
|
||||
|
||||
|
|
@ -99,7 +96,7 @@ The CHANGELOG.md is automatically used for:
|
|||
1. **Technical docs** → `docs/technical/`
|
||||
2. **Development docs** → `docs/development/`
|
||||
3. **User guides** → `docs/guides/`
|
||||
4. **Temporary files** → `archive/` (if needed)
|
||||
4. **Temporary files** → Should be avoided; use issues or PRs for temporary documentation
|
||||
|
||||
### Making Changes
|
||||
1. Add changes to `[Unreleased]` section in `CHANGELOG.md`
|
||||
|
|
@ -115,9 +112,10 @@ The CHANGELOG.md is automatically used for:
|
|||
|
||||
## Migration Notes
|
||||
|
||||
- All technical documentation moved to `docs/technical/`
|
||||
- Development documentation moved to `docs/development/`
|
||||
- Old changelog files archived in `archive/`
|
||||
- All technical documentation organized in `docs/technical/`
|
||||
- Development documentation organized in `docs/development/`
|
||||
- User guides organized in `docs/guides/`
|
||||
- Temporary summary files and archived content have been cleaned up
|
||||
- Links in existing documentation may need updates
|
||||
- New automation ensures changelog stays current
|
||||
|
||||
|
|
|
|||
195
docs/MARKDOWN_FILE_MANAGEMENT.md
Normal file
195
docs/MARKDOWN_FILE_MANAGEMENT.md
Normal file
|
|
@ -0,0 +1,195 @@
|
|||
# Markdown File Management Guidelines
|
||||
|
||||
## Overview
|
||||
|
||||
This document establishes standards for managing .md files in the Brainy project to maintain a clean, organized, and professional documentation structure.
|
||||
|
||||
## Root Directory Standards
|
||||
|
||||
The project root should contain **only** these essential .md files:
|
||||
|
||||
### Required Files (GitHub/NPM Standards)
|
||||
- `README.md` - Main project documentation and entry point
|
||||
- `CHANGELOG.md` - Version history and release notes
|
||||
- `CODE_OF_CONDUCT.md` - Community guidelines
|
||||
- `CONTRIBUTING.md` - Contribution guidelines
|
||||
- `LICENSE` - Legal license file (not .md but related)
|
||||
|
||||
### Prohibited in Root
|
||||
❌ **Never place these in root:**
|
||||
- Temporary summary files (e.g., `IMPLEMENTATION_SUMMARY.md`)
|
||||
- Fix-related documentation (e.g., `RELIABILITY_IMPROVEMENTS_SUMMARY.md`)
|
||||
- Organizational notes (e.g., `SCRIPT_ORGANIZATION_SOLUTION.md`)
|
||||
- Update logs (e.g., `README_updates.md`, `changes-summary.md`)
|
||||
- Environment-specific guides (should go in docs/)
|
||||
|
||||
## Documentation Organization Structure
|
||||
|
||||
```
|
||||
docs/
|
||||
├── COMPATIBILITY.md # Cross-platform compatibility info
|
||||
├── DOCUMENTATION_ORGANIZATION.md # This file's organization guide
|
||||
├── development/ # Developer-focused documentation
|
||||
│ ├── DEVELOPERS.md
|
||||
│ ├── DOCUMENTATION_STANDARDS.md
|
||||
│ └── MARKDOWN_CONVENTIONS.md
|
||||
├── guides/ # User guides and tutorials
|
||||
│ ├── cache-configuration.md
|
||||
│ ├── model-management.md
|
||||
│ └── production-migration-guide.md
|
||||
└── technical/ # Technical implementation details
|
||||
├── TESTING.md # Comprehensive testing guide
|
||||
├── ENVIRONMENT_TESTING.md # Environment-specific testing
|
||||
├── CONCURRENCY_ANALYSIS.md
|
||||
└── STORAGE_TESTING.md
|
||||
```
|
||||
|
||||
## File Naming Conventions
|
||||
|
||||
### Use UPPERCASE for Major Documents
|
||||
- `README.md`, `CHANGELOG.md`, `CONTRIBUTING.md`
|
||||
- `TESTING.md`, `COMPATIBILITY.md`
|
||||
|
||||
### Use lowercase-with-hyphens for Specific Guides
|
||||
- `cache-configuration.md`
|
||||
- `model-management.md`
|
||||
- `production-migration-guide.md`
|
||||
|
||||
### Use Descriptive Names
|
||||
✅ **Good:**
|
||||
- `ENVIRONMENT_TESTING.md` (specific purpose)
|
||||
- `cache-configuration.md` (clear topic)
|
||||
- `production-migration-guide.md` (clear audience and purpose)
|
||||
|
||||
❌ **Bad:**
|
||||
- `IMPLEMENTATION_SUMMARY.md` (temporary)
|
||||
- `changes-summary.md` (temporary)
|
||||
- `notes.md` (vague)
|
||||
|
||||
## File Lifecycle Management
|
||||
|
||||
### Temporary Files
|
||||
**Rule: Temporary files should be deleted immediately after their purpose is fulfilled.**
|
||||
|
||||
Examples of temporary files that should be deleted:
|
||||
- Implementation summaries after feature completion
|
||||
- Fix documentation after issues are resolved
|
||||
- Organizational notes after reorganization is complete
|
||||
- Update logs after updates are integrated
|
||||
|
||||
### Permanent Documentation
|
||||
Files that should be maintained long-term:
|
||||
- User guides and tutorials
|
||||
- Technical reference documentation
|
||||
- API documentation
|
||||
- Testing guides
|
||||
- Development standards
|
||||
|
||||
## Where to Place Different Types of Documentation
|
||||
|
||||
### Root Directory
|
||||
- Only essential project files (README, CHANGELOG, etc.)
|
||||
|
||||
### docs/development/
|
||||
- Developer setup guides
|
||||
- Build instructions
|
||||
- Code standards
|
||||
- Documentation standards
|
||||
|
||||
### docs/guides/
|
||||
- User tutorials
|
||||
- Configuration guides
|
||||
- Migration guides
|
||||
- How-to documentation
|
||||
|
||||
### docs/technical/
|
||||
- Technical implementation details
|
||||
- Architecture documentation
|
||||
- Testing documentation
|
||||
- Performance analysis
|
||||
|
||||
### Package-Specific
|
||||
- Each package (cli-package/, web-service-package/, etc.) should have its own README.md
|
||||
- Package-specific documentation stays with the package
|
||||
|
||||
## Review Process
|
||||
|
||||
### Before Adding New .md Files
|
||||
1. **Determine if it's temporary or permanent**
|
||||
- Temporary: Consider using issues, PRs, or comments instead
|
||||
- Permanent: Proceed with proper placement
|
||||
|
||||
2. **Choose the correct location**
|
||||
- Root: Only for essential project files
|
||||
- docs/: For all other documentation
|
||||
|
||||
3. **Use proper naming conventions**
|
||||
- Descriptive names that indicate purpose
|
||||
- Consistent with existing patterns
|
||||
|
||||
### Regular Cleanup
|
||||
- Review .md files quarterly
|
||||
- Delete temporary files that have served their purpose
|
||||
- Consolidate duplicate or overlapping documentation
|
||||
- Update links when files are moved
|
||||
|
||||
## Migration Guidelines
|
||||
|
||||
When reorganizing existing documentation:
|
||||
|
||||
1. **Categorize existing files**
|
||||
- Essential (keep in root)
|
||||
- Useful (move to docs/)
|
||||
- Temporary (delete)
|
||||
|
||||
2. **Update references**
|
||||
- Search for links to moved files
|
||||
- Update README.md and other documentation
|
||||
- Test that all links work
|
||||
|
||||
3. **Maintain backward compatibility when possible**
|
||||
- Consider redirects for important moved files
|
||||
- Update package.json scripts if they reference moved files
|
||||
|
||||
## Enforcement
|
||||
|
||||
### Code Review Checklist
|
||||
- [ ] New .md files are in appropriate locations
|
||||
- [ ] Temporary files are not being committed
|
||||
- [ ] Links to documentation are correct
|
||||
- [ ] File names follow conventions
|
||||
|
||||
### Automated Checks (Future)
|
||||
Consider implementing:
|
||||
- Linting rules for .md file placement
|
||||
- Link checking in CI/CD
|
||||
- Automated cleanup of temporary files
|
||||
|
||||
## Examples
|
||||
|
||||
### ✅ Good Documentation Structure
|
||||
```
|
||||
README.md # Main project docs
|
||||
CHANGELOG.md # Version history
|
||||
docs/guides/setup.md # User guide
|
||||
docs/technical/api.md # Technical reference
|
||||
```
|
||||
|
||||
### ❌ Bad Documentation Structure
|
||||
```
|
||||
README.md
|
||||
IMPLEMENTATION_SUMMARY.md # Temporary - should be deleted
|
||||
FIX_NOTES.md # Temporary - should be deleted
|
||||
setup.md # Should be in docs/guides/
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
Following these guidelines ensures:
|
||||
- Clean, professional project structure
|
||||
- Easy navigation for users and contributors
|
||||
- Reduced maintenance overhead
|
||||
- Consistent documentation organization
|
||||
- Better discoverability of information
|
||||
|
||||
**Remember: When in doubt, ask "Is this temporary or permanent?" and "Who is the audience?" to determine the right approach.**
|
||||
|
|
@ -28,7 +28,7 @@ Use lowercase filenames for technical documentation and implementation details:
|
|||
- Architecture documentation
|
||||
- Specific feature documentation
|
||||
|
||||
Examples: `scalingStrategy.md`, `statistics.md`
|
||||
Examples: `SCALING_STRATEGY.md`, `STATISTICS.md`
|
||||
|
||||
## Rationale
|
||||
|
||||
|
|
@ -59,9 +59,9 @@ This convention makes it easy to distinguish between:
|
|||
|
||||
### Technical Documentation (Lowercase)
|
||||
|
||||
- scalingStrategy.md
|
||||
- SCALING_STRATEGY.md
|
||||
- statistics.md
|
||||
- architecture.md
|
||||
- implementation-details.md
|
||||
|
||||
By following these conventions, we maintain consistency and make it easier for contributors to find the right documentation.
|
||||
By following these conventions, we maintain consistency and make it easier for contributors to find the right documentation.
|
||||
|
|
|
|||
|
|
@ -1,115 +0,0 @@
|
|||
# Concurrency Implementation Summary
|
||||
|
||||
## Overview
|
||||
This document summarizes all the concurrency improvements that have been implemented based on the recommendations in CONCURRENCY_ANALYSIS.md.
|
||||
|
||||
## ✅ Completed High Priority Implementations
|
||||
|
||||
### 1. Distributed Locking for Statistics Updates
|
||||
**Location**: `S3CompatibleStorage.flushStatistics()`
|
||||
**Implementation**:
|
||||
- Added `acquireLock()` and `releaseLock()` methods using S3 objects as locks
|
||||
- Implemented lock timeout (15 seconds) and automatic cleanup
|
||||
- Statistics updates now use distributed locking to prevent race conditions
|
||||
- Graceful handling when another instance is updating statistics
|
||||
|
||||
### 2. Change Log Mechanism for Efficient Index Synchronization
|
||||
**Location**: `S3CompatibleStorage` and `BrainyData.checkForUpdates()`
|
||||
**Implementation**:
|
||||
- Added `ChangeLogEntry` interface for tracking data modifications
|
||||
- Implemented `appendToChangeLog()` method that logs all CRUD operations
|
||||
- Added `getChangesSince()` method for retrieving changes since a timestamp
|
||||
- Updated `BrainyData.checkForUpdates()` to use change log instead of expensive full scans
|
||||
- Fallback mechanism for storage adapters that don't support change logs
|
||||
- Automatic cleanup of old change log entries
|
||||
|
||||
### 3. Thread-Safe Memory Usage Tracking
|
||||
**Location**: `HNSWIndexOptimized`
|
||||
**Implementation**:
|
||||
- Added `memoryUpdateLock` using Promise chaining for thread safety
|
||||
- Implemented `updateMemoryUsage()` and `getMemoryUsage()` methods
|
||||
- Updated `addItem()`, `removeItem()`, and `clear()` methods to use thread-safe updates
|
||||
- Prevents race conditions in memory usage calculations
|
||||
|
||||
### 4. Atomic Statistics Updates with Merge Strategy
|
||||
**Location**: `S3CompatibleStorage.flushStatistics()`
|
||||
**Implementation**:
|
||||
- Read current statistics from storage before updating
|
||||
- Merge local changes with storage statistics to prevent data loss
|
||||
- Use distributed locking to ensure atomic updates
|
||||
- Proper error handling and lock cleanup in finally blocks
|
||||
|
||||
### 5. Comprehensive Change Log Integration
|
||||
**Location**: All CRUD operations in `S3CompatibleStorage`
|
||||
**Implementation**:
|
||||
- `saveNode()`: Logs 'add' operations for nouns
|
||||
- `saveEdge()`: Logs 'add' operations for verbs
|
||||
- `deleteNode()`: Logs 'delete' operations for nouns
|
||||
- `deleteEdge()`: Logs 'delete' operations for verbs
|
||||
- `saveMetadata()`: Logs metadata changes
|
||||
- All operations include timestamp, operation type, entity type, and relevant data
|
||||
|
||||
## ✅ Performance Improvements Achieved
|
||||
|
||||
Based on the original analysis expectations:
|
||||
|
||||
1. **Statistics Updates**: 90% reduction in conflicts achieved through distributed locking
|
||||
2. **Index Synchronization**: 95% reduction in data transfer achieved through change log mechanism
|
||||
3. **Memory Usage Tracking**: Race conditions eliminated through thread-safe updates
|
||||
4. **Search Performance**: Improved through better cache consistency and reduced contention
|
||||
|
||||
## 📊 Storage Adapter Analysis
|
||||
|
||||
### S3CompatibleStorage ✅ FULLY IMPLEMENTED
|
||||
- **Risk Level**: HIGH (multi-instance distributed deployment)
|
||||
- **Status**: All concurrency improvements implemented and tested
|
||||
- **Features**: Distributed locking, change logs, atomic updates, lock cleanup
|
||||
|
||||
### FileSystemStorage 📋 ANALYSIS COMPLETE
|
||||
- **Risk Level**: MEDIUM (multi-process scenarios)
|
||||
- **Status**: Analysis complete, improvements optional for typical use cases
|
||||
- **Recommendation**: File-based locking for multi-process scenarios (not critical)
|
||||
|
||||
### OPFSStorage 📋 ANALYSIS COMPLETE
|
||||
- **Risk Level**: LOW-MEDIUM (multi-tab browser scenarios)
|
||||
- **Status**: Analysis complete, improvements optional
|
||||
- **Recommendation**: Browser-based locking for multi-tab scenarios (not critical)
|
||||
|
||||
### MemoryStorage 📋 ANALYSIS COMPLETE
|
||||
- **Risk Level**: VERY LOW (single-process in-memory)
|
||||
- **Status**: No changes needed
|
||||
- **Recommendation**: No improvements required for typical use cases
|
||||
|
||||
## 🧪 Testing Results
|
||||
|
||||
All implementations have been tested and verified:
|
||||
- **Test Files**: 20 passed | 1 skipped (21)
|
||||
- **Tests**: 178 passed | 18 skipped (196)
|
||||
- **Duration**: 22.18s
|
||||
- **Status**: ✅ All tests passing
|
||||
|
||||
## 📈 Impact Assessment
|
||||
|
||||
### Before Implementation
|
||||
- Race conditions in statistics updates causing data corruption
|
||||
- Inefficient full scans on every index update check
|
||||
- Memory usage tracking race conditions
|
||||
- No coordination between multiple service instances
|
||||
|
||||
### After Implementation
|
||||
- Distributed coordination prevents data corruption
|
||||
- Change log mechanism provides 95% reduction in data transfer
|
||||
- Thread-safe memory tracking eliminates race conditions
|
||||
- Robust multi-instance deployment support
|
||||
|
||||
## 🎯 Conclusion
|
||||
|
||||
All high-priority concurrency improvements from CONCURRENCY_ANALYSIS.md have been successfully implemented and tested. The system now provides:
|
||||
|
||||
1. **Robust Multi-Instance Support**: Multiple web services can safely share S3 storage
|
||||
2. **Efficient Synchronization**: Change log mechanism eliminates expensive full scans
|
||||
3. **Data Integrity**: Distributed locking prevents race conditions and data corruption
|
||||
4. **Performance Optimization**: Significant improvements in high-throughput scenarios
|
||||
5. **Backward Compatibility**: Fallback mechanisms ensure compatibility with all storage types
|
||||
|
||||
The implementation addresses all identified concurrency issues while maintaining system stability and performance.
|
||||
|
|
@ -1,103 +0,0 @@
|
|||
# Dimension Mismatch Issue: Summary and Recommendations
|
||||
|
||||
## What Happened
|
||||
|
||||
The search functionality in Brainy stopped working because of a dimension mismatch between stored vectors and the expected dimensions in the current version of the codebase:
|
||||
|
||||
1. **Previous State**: The system was using vectors with 3 dimensions.
|
||||
2. **Current State**: The system now expects 512-dimensional vectors from the Universal Sentence Encoder.
|
||||
3. **Code Change**: Recent updates (around July 16, 2025) introduced dimension validation during initialization, which skips vectors with mismatched dimensions.
|
||||
4. **Result**: During initialization, vectors with 3 dimensions were skipped, resulting in an empty search index and no search results.
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
The root cause was identified by examining the codebase:
|
||||
|
||||
1. In `brainyData.ts`, the `init()` method checks if vector dimensions match the expected dimensions (line 400):
|
||||
```javascript
|
||||
if (noun.vector.length !== this._dimensions) {
|
||||
console.warn(
|
||||
`Skipping noun ${noun.id} due to dimension mismatch: expected ${this._dimensions}, got ${noun.vector.length}`
|
||||
)
|
||||
// Skip this noun and continue with the next one
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
2. The default dimension is set to 512 in the constructor (line 200):
|
||||
```javascript
|
||||
this._dimensions = config.dimensions || 512
|
||||
```
|
||||
|
||||
3. The `UniversalSentenceEncoder` class in `embedding.ts` produces 512-dimensional vectors (lines 358-359):
|
||||
```javascript
|
||||
// Return a zero vector of appropriate dimension (512 is the default for USE)
|
||||
return new Array(512).fill(0)
|
||||
```
|
||||
|
||||
4. Git history shows that on July 16, 2025, a commit was made that added dimension validation:
|
||||
```
|
||||
Added a `dimensions` property to `BrainyDataConfig` for specifying vector dimensions.
|
||||
Introduced validation for vector dimensions during database creation and insertion to ensure consistency.
|
||||
Enhanced error handling and logging for dimension mismatches.
|
||||
```
|
||||
|
||||
This indicates that the system previously used 3-dimensional vectors, but after the update, it expects 512-dimensional vectors. The existing data was not migrated, causing the search functionality to break.
|
||||
|
||||
## Solution Implemented
|
||||
|
||||
We created and tested a fix script (`fix-dimension-mismatch.js`) that:
|
||||
|
||||
1. Creates a backup of the existing data
|
||||
2. Reads all noun files directly from the filesystem
|
||||
3. For each noun:
|
||||
- Extracts text from metadata
|
||||
- Deletes the existing noun
|
||||
- Re-adds the noun with the same ID but using the current embedding function
|
||||
4. Recreates all verb relationships between the re-embedded nouns
|
||||
5. Verifies that search works by performing a test search
|
||||
|
||||
The script successfully fixed the issue by re-embedding all data with the correct dimensions, and search functionality was restored.
|
||||
|
||||
## Production Recommendations
|
||||
|
||||
For production environments, we recommend:
|
||||
|
||||
### 1. Use the Enhanced Migration Script
|
||||
|
||||
We've created a comprehensive production migration guide (`production-migration-guide.md`) that includes:
|
||||
|
||||
- Enhanced backup strategies with metadata
|
||||
- Batching for large datasets
|
||||
- Robust error handling and recovery
|
||||
- Progress monitoring and reporting
|
||||
- A parallel database approach for mission-critical systems
|
||||
|
||||
### 2. Implement Preventive Measures
|
||||
|
||||
To prevent similar issues in the future:
|
||||
|
||||
- **Version Tracking**: Add version information to stored vectors
|
||||
- **Auto-Migration**: Enhance initialization to automatically re-embed mismatched vectors
|
||||
- **Regular Validation**: Implement a database validation process
|
||||
- **Documentation**: Document embedding changes in release notes
|
||||
|
||||
### 3. Scheduling and Communication
|
||||
|
||||
- Schedule the migration during a maintenance window
|
||||
- Communicate the change to all stakeholders
|
||||
- Have a rollback plan in case of issues
|
||||
- Monitor the system after the migration
|
||||
|
||||
## Conclusion
|
||||
|
||||
The dimension mismatch issue was caused by a change in the embedding function that increased vector dimensions from 3 to 512. The solution is to re-embed all existing data using the current embedding function, which can be done using the provided `fix-dimension-mismatch.js` script with the enhancements suggested for production environments.
|
||||
|
||||
By implementing the preventive measures outlined in the production migration guide, you can avoid similar issues in the future and ensure smoother transitions when embedding functions or vector dimensions change.
|
||||
|
||||
## Files Created
|
||||
|
||||
1. `check-database.js` - Script to verify database status and search functionality
|
||||
2. `fix-dimension-mismatch.js` - Script to fix the dimension mismatch issue
|
||||
3. `production-migration-guide.md` - Comprehensive guide for production migration
|
||||
4. `DIMENSION_MISMATCH_SUMMARY.md` - This summary document
|
||||
97
docs/technical/ENVIRONMENT_TESTING.md
Normal file
97
docs/technical/ENVIRONMENT_TESTING.md
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
# Testing Brainy Across Different Environments
|
||||
|
||||
This document provides instructions for testing Brainy's cache detection functionality across different environments.
|
||||
|
||||
## Testing in Node.js Environment
|
||||
|
||||
To test Brainy in a Node.js environment:
|
||||
|
||||
1. Build the project:
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
2. Run the Node.js test script:
|
||||
```bash
|
||||
node test-cache-detection.js
|
||||
```
|
||||
|
||||
3. Expected output:
|
||||
```
|
||||
Brainy: Successfully patched TensorFlow.js PlatformNode at module load time
|
||||
Applied TensorFlow.js patch via ES modules in setup.ts
|
||||
Brainy running in Node.js environment
|
||||
Creating BrainyData instance...
|
||||
BrainyData instance created successfully!
|
||||
Test completed successfully!
|
||||
```
|
||||
|
||||
## Testing in Browser Environment
|
||||
|
||||
To test Brainy in a browser environment:
|
||||
|
||||
1. Build the project:
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
2. Start a local web server:
|
||||
```bash
|
||||
npx http-server -p 8080
|
||||
```
|
||||
|
||||
3. Open the browser test page:
|
||||
```
|
||||
http://localhost:8080/test-browser-cache-detection.html
|
||||
```
|
||||
|
||||
4. Click the "Run Test" button on the page.
|
||||
|
||||
5. Expected results:
|
||||
- The page should display success messages
|
||||
- No errors should appear in the browser console
|
||||
- You should see "BrainyData instance created successfully!" and "Test completed successfully!"
|
||||
|
||||
## Testing in Web Worker Environment
|
||||
|
||||
To test Brainy in a Web Worker environment:
|
||||
|
||||
1. Build the project:
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
2. Start a local web server:
|
||||
```bash
|
||||
npx http-server -p 8080
|
||||
```
|
||||
|
||||
3. Open the worker test page:
|
||||
```
|
||||
http://localhost:8080/test-worker-cache-detection.html
|
||||
```
|
||||
|
||||
4. Click the "Run Test" button on the page.
|
||||
|
||||
5. Expected results:
|
||||
- The page should display success messages from the worker
|
||||
- No errors should appear in the browser console
|
||||
- You should see "BrainyData instance created successfully!" and "Test completed successfully!"
|
||||
|
||||
## Compatibility Notes
|
||||
|
||||
Brainy's cache detection has been designed to work across all environments:
|
||||
|
||||
1. **Node.js Environment**:
|
||||
- Uses fixed default memory values (8GB total, 4GB free) for cache size calculation
|
||||
- This approach ensures compatibility with ES modules
|
||||
|
||||
2. **Browser Environment**:
|
||||
- Uses navigator.deviceMemory API when available
|
||||
- Falls back to conservative defaults when the API is not available
|
||||
|
||||
3. **Worker Environment**:
|
||||
- Uses a more conservative approach to cache sizing
|
||||
- Automatically detects the worker environment and adjusts accordingly
|
||||
|
||||
The cache manager automatically detects the environment and adjusts its behavior to ensure optimal performance in each context.
|
||||
|
|
@ -56,4 +56,4 @@ This script:
|
|||
|
||||
The Universal Sentence Encoder model produces 512-dimensional vectors by default. This is now the standard dimension for all vectors in Brainy, ensuring consistency across all operations.
|
||||
|
||||
For more information about the dimension mismatch issue and its resolution, see `DIMENSION_MISMATCH_SUMMARY.md`.
|
||||
This standardization resolves dimension mismatch issues that could previously cause search functionality to break.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue