David Snelling
7e75221ae7
- Enhanced `CacheManager` for better handling of large datasets, especially in `S3` or remote storage:
- Added `REMOTE_API` as a supported storage type.
- Improved cache sizing and batch tuning:
- Optimized memory usage based on environment (Browser, Node.js, Worker).
- Increased cache aggressiveness in read-only mode and for large datasets.
- Adjusted cache parameters dynamically for S3 or remote storage.
- Introduced `isReadOnly` and `isRemoteStorage` checks to refine tuning logic.
- Added new `cacheConfig` options:
- `autoTune`, `autoTuneInterval`, and mode-specific settings for read-only optimizations.
- Batch sizes, eviction thresholds, and TTLs tailored for operating modes.
- Enhanced documentation:
- Detailed performance-tuning guides and S3 examples in `README.md`.
- Included new configuration examples for large datasets in cloud storage.
- Improved extensibility:
- Unified cache and batch logic under storage type and mode-aware rules.
- Updated interfaces (`BrainyData`, `StorageFactory`) to include new cache settings.
- Verified enhancements with rigorous testing across multiple configurations.
**Purpose**: Improve caching strategy and query performance in complex cloud and on-premise environments with flexible, dynamic tuning.
|
||
|---|---|---|
| .. | ||
| augmentations | ||
| errors | ||
| examples | ||
| hnsw | ||
| mcp | ||
| storage | ||
| testing | ||
| types | ||
| utils | ||
| augmentationFactory.ts | ||
| augmentationPipeline.ts | ||
| augmentationRegistry.ts | ||
| augmentationRegistryLoader.ts | ||
| brainyData.ts | ||
| coreTypes.ts | ||
| index.ts | ||
| patched-platform-node.ts | ||
| pipeline.ts | ||
| sequentialPipeline.ts | ||
| setup.ts | ||
| unified.ts | ||
| worker.js | ||
| worker.ts | ||