**feat(storage): implement multi-level cache manager with dynamic tuning**

- Added a new `CacheManager` class in `cacheManager.ts` to support three-level caching strategy:
  - **Level 1**: Hot cache in RAM for most accessed nodes.
  - **Level 2**: Warm cache using OPFS, Filesystem, or S3, depending on the environment.
  - **Level 3**: Cold storage for longer-term data storage.
- Integrated features for dynamic tuning:
  - Auto-detection of environment (Browser, Node.js, Worker) and memory availability.
  - Parameter tuning for cache size, eviction thresholds, and TTL based on usage patterns.
- Enhanced support for:
  - LRU-based eviction in hot cache.
  - Batch-based operations with configurable batch sizes.
  - Comprehensive logging and debug outputs for cache operations.
- Ensured robust fallback handling to manage storage in constrained environments.
- Improved extensibility for storage adapters (warm and cold storage detection and initialization).

**Purpose**: Optimize data access and storage across multiple environments with seamless scalability and dynamic parameter adjustments.

README.md

@@ -1379,6 +1379,106 @@ terabyte-scale data that can't fit entirely in memory, we provide several approa
 
 For detailed information on how to scale Brainy for large datasets, vector dimension standardization, threading implementation, storage testing, and other technical topics, see our comprehensive [Technical Guides](TECHNICAL_GUIDES.md).
 
+## Recent Changes and Performance Improvements
+
+### Enhanced Memory Management and Scalability
+
+Brainy has been significantly improved to handle larger datasets more efficiently:
+
+- **Pagination Support**: All data retrieval methods now support pagination to avoid loading entire datasets into memory at once. The deprecated `getAllNouns()` and `getAllVerbs()` methods have been replaced with `getNouns()` and `getVerbs()` methods that support pagination, filtering, and cursor-based navigation.
+
+- **Multi-level Caching**: A sophisticated three-level caching strategy has been implemented:
+  - **Level 1**: Hot cache (most accessed nodes) - RAM (automatically detecting and adjusting in each environment)
+  - **Level 2**: Warm cache (recent nodes) - OPFS, Filesystem or S3 depending on environment
+  - **Level 3**: Cold storage (all nodes) - OPFS, Filesystem or S3 depending on environment
+
+- **Adaptive Memory Usage**: The system automatically detects available memory and adjusts cache sizes accordingly:
+  - In Node.js: Uses 10% of free memory (minimum 1000 entries)
+  - In browsers: Scales based on device memory (500 entries per GB, minimum 1000)
+
+- **Intelligent Cache Eviction**: Implements a Least Recently Used (LRU) policy that evicts the oldest 20% of items when the cache reaches the configured threshold.
+
+- **Prefetching Strategy**: Implements batch prefetching to improve performance while avoiding overwhelming system resources.
+
+### S3-Compatible Storage Improvements
+
+- **Enhanced Cloud Storage**: Improved support for S3-compatible storage services including AWS S3, Cloudflare R2, and others.
+
+- **Optimized Data Access**: Batch operations and error handling for efficient cloud storage access.
+
+- **Change Log Management**: Efficient synchronization through change logs to track updates.
+
+### Data Compatibility
+
+Yes, you can use existing data indexed from an old version. Brainy includes robust data migration capabilities:
+
+- **Vector Regeneration**: If vectors are missing in imported data, they will be automatically created using the embedding function.
+
+- **HNSW Index Reconstruction**: The system can reconstruct the HNSW index from backup data, ensuring compatibility with previous versions.
+
+- **Sparse Data Import**: Support for importing sparse data (without vectors) through the `importSparseData()` method.
+
+### System Requirements
+
+#### Default Mode
+
+- **Memory**: 
+  - Minimum: 512MB RAM
+  - Recommended: 2GB+ RAM for medium datasets, 8GB+ for large datasets
+  
+- **CPU**: 
+  - Minimum: 2 cores
+  - Recommended: 4+ cores for better performance with parallel operations
+
+- **Storage**:
+  - Minimum: 1GB available storage
+  - Recommended: Storage space at least 3x the size of your dataset
+
+#### Read-Only Mode
+
+Read-only mode prevents all write operations (add, update, delete) and is optimized for search operations.
+
+- **Memory**: 
+  - Minimum: 256MB RAM
+  - Recommended: 1GB+ RAM
+  
+- **CPU**: 
+  - Minimum: 1 core
+  - Recommended: 2+ cores
+
+- **Storage**:
+  - Minimum: Storage space equal to the size of your dataset
+  - Recommended: 2x the size of your dataset for caching
+
+- **New Feature**: Lazy loading support in read-only mode for improved performance with large datasets.
+
+#### Write-Only Mode
+
+Write-only mode prevents all search operations and is optimized for initial data loading or when you want to optimize for write performance.
+
+- **Memory**: 
+  - Minimum: 512MB RAM
+  - Recommended: 2GB+ RAM
+  
+- **CPU**: 
+  - Minimum: 2 cores
+  - Recommended: 4+ cores for faster data ingestion
+
+- **Storage**:
+  - Minimum: Storage space at least 2x the size of your dataset
+  - Recommended: 4x the size of your dataset for optimal performance
+
+### Performance Tuning Parameters
+
+Brainy offers several configuration options for performance tuning:
+
+- **Hot Cache Size**: Control the maximum number of items to keep in memory.
+- **Eviction Threshold**: Set the threshold at which cache eviction begins (default: 0.8 or 80% of max size).
+- **Warm Cache TTL**: Set the time-to-live for items in the warm cache (default: 24 hours).
+- **Batch Size**: Control the number of items to process in a single batch for operations like prefetching (default: 10).
+
+These improvements make Brainy more efficient, scalable, and adaptable to different environments and usage patterns.
+
 ## Testing
 
 Brainy uses Vitest for testing. For detailed information about testing in Brainy, including test configuration, scripts, reporting tools, and best practices, see our [Testing Guide](TESTING.md).