diff --git a/README.md b/README.md index 62ebdc73..ed5ee3f6 100644 --- a/README.md +++ b/README.md @@ -1503,14 +1503,81 @@ Write-only mode prevents all search operations and is optimized for initial data ### Performance Tuning Parameters -Brainy offers several configuration options for performance tuning: +Brainy offers comprehensive configuration options for performance tuning, with enhanced support for large datasets in S3 or other remote storage. **All configuration is optional** - the system automatically detects the optimal settings based on your environment, dataset size, and usage patterns. + +#### Intelligent Defaults + +Brainy uses intelligent defaults that automatically adapt to your environment: + +- **Environment Detection**: Automatically detects whether you're running in Node.js, browser, or worker environment +- **Memory-Aware Caching**: Adjusts cache sizes based on available system memory +- **Dataset Size Adaptation**: Tunes parameters based on the size of your dataset +- **Usage Pattern Optimization**: Adjusts to read-heavy vs. write-heavy workloads +- **Storage Type Awareness**: Optimizes for local vs. remote storage (S3, R2, etc.) +- **Operating Mode Specialization**: Special optimizations for read-only and write-only modes + +#### Cache Configuration (Optional) + +You can override any of these automatically tuned parameters if needed: - **Hot Cache Size**: Control the maximum number of items to keep in memory. + - For large datasets (>100K items), consider values between 5,000-50,000 depending on available memory. + - In read-only mode, larger values (10,000-100,000) can be used for better performance. + - **Eviction Threshold**: Set the threshold at which cache eviction begins (default: 0.8 or 80% of max size). -- **Warm Cache TTL**: Set the time-to-live for items in the warm cache (default: 24 hours). -- **Batch Size**: Control the number of items to process in a single batch for operations like prefetching (default: 10). + - For write-heavy workloads, lower values (0.6-0.7) may improve performance. + - For read-heavy workloads, higher values (0.8-0.9) are recommended. -These improvements make Brainy more efficient, scalable, and adaptable to different environments and usage patterns. +- **Warm Cache TTL**: Set the time-to-live for items in the warm cache (default: 3600000 ms or 1 hour). + - For frequently changing data, shorter TTLs are recommended. + - For relatively static data, longer TTLs improve performance. + +- **Batch Size**: Control the number of items to process in a single batch for operations like prefetching. + - For S3 or remote storage with large datasets, larger values (50-200) significantly improve throughput. + - In read-only mode with remote storage, even larger values (100-300) can be used. + +#### Auto-Tuning (Enabled by Default) + +- **Auto-Tune**: Enable or disable automatic tuning of cache parameters based on usage patterns (default: true). +- **Auto-Tune Interval**: Set how frequently the system adjusts cache parameters (default: 60000 ms or 1 minute). + +#### Read-Only Mode Optimizations (Automatic) + +Read-only mode includes special optimizations for search performance that are automatically applied: + +- **Larger Cache Sizes**: Automatically uses more memory for caching (up to 40% of free memory for large datasets). +- **Aggressive Prefetching**: Loads more data in each batch to reduce the number of storage requests. +- **Prefetch Strategy**: Defaults to 'aggressive' prefetching strategy in read-only mode. + +#### Example Configuration for Large S3 Datasets + +```javascript +const brainy = new BrainyData({ + readOnly: true, + lazyLoadInReadOnlyMode: true, + storage: { + type: 's3', + s3Storage: { + bucketName: 'your-bucket', + accessKeyId: 'your-access-key', + secretAccessKey: 'your-secret-key', + region: 'your-region' + } + }, + cache: { + hotCacheMaxSize: 20000, + hotCacheEvictionThreshold: 0.85, + batchSize: 100, + readOnlyMode: { + hotCacheMaxSize: 50000, + batchSize: 200, + prefetchStrategy: 'aggressive' + } + } +}); +``` + +These configuration options make Brainy more efficient, scalable, and adaptable to different environments and usage patterns, especially for large datasets in cloud storage. ## Testing diff --git a/src/brainyData.ts b/src/brainyData.ts index eb88a2ab..d2665800 100644 --- a/src/brainyData.ts +++ b/src/brainyData.ts @@ -98,6 +98,15 @@ export interface BrainyDataConfig { } forceFileSystemStorage?: boolean forceMemoryStorage?: boolean + cacheConfig?: { + hotCacheMaxSize?: number + hotCacheEvictionThreshold?: number + warmCacheTTL?: number + batchSize?: number + autoTune?: boolean + autoTuneInterval?: number + readOnly?: boolean + } } /** @@ -242,6 +251,82 @@ export interface BrainyDataConfig { */ updateIndex?: boolean } + + /** + * Cache configuration for optimizing search performance + * Controls how the system caches data for faster access + * Particularly important for large datasets in S3 or other remote storage + */ + cache?: { + /** + * Whether to enable auto-tuning of cache parameters + * When true, the system will automatically adjust cache sizes based on usage patterns + * Default: true + */ + autoTune?: boolean + + /** + * The interval (in milliseconds) at which to auto-tune cache parameters + * Only applies when autoTune is true + * Default: 60000 (60 seconds) + */ + autoTuneInterval?: number + + /** + * Maximum size of the hot cache (most frequently accessed items) + * If provided, overrides the automatically detected optimal size + * For large datasets, consider values between 5000-50000 depending on available memory + */ + hotCacheMaxSize?: number + + /** + * Threshold at which to start evicting items from the hot cache + * Expressed as a fraction of hotCacheMaxSize (0.0 to 1.0) + * Default: 0.8 (start evicting when cache is 80% full) + */ + hotCacheEvictionThreshold?: number + + /** + * Time-to-live for items in the warm cache in milliseconds + * Default: 3600000 (1 hour) + */ + warmCacheTTL?: number + + /** + * Batch size for operations like prefetching + * Larger values improve throughput but use more memory + * For S3 or remote storage with large datasets, consider values between 50-200 + */ + batchSize?: number + + /** + * Read-only mode specific optimizations + * These settings are only applied when readOnly is true + */ + readOnlyMode?: { + /** + * Maximum size of the hot cache in read-only mode + * In read-only mode, larger cache sizes can be used since there are no write operations + * For large datasets, consider values between 10000-100000 depending on available memory + */ + hotCacheMaxSize?: number + + /** + * Batch size for operations in read-only mode + * Larger values improve throughput in read-only mode + * For S3 or remote storage with large datasets, consider values between 100-300 + */ + batchSize?: number + + /** + * Prefetch strategy for read-only mode + * Controls how aggressively the system prefetches data + * Options: 'conservative', 'moderate', 'aggressive' + * Default: 'moderate' + */ + prefetchStrategy?: 'conservative' | 'moderate' | 'aggressive' + } + } } export class BrainyData implements BrainyDataInterface { @@ -263,6 +348,9 @@ export class BrainyData implements BrainyDataInterface { // Timeout and retry configuration private timeoutConfig: BrainyDataConfig['timeouts'] = {} private retryConfig: BrainyDataConfig['retryPolicy'] = {} + + // Cache configuration + private cacheConfig: BrainyDataConfig['cache'] // Real-time update properties private realtimeUpdateConfig: Required< @@ -386,6 +474,31 @@ export class BrainyData implements BrainyDataInterface { ...config.realtimeUpdates } } + + // Initialize cache configuration with intelligent defaults + // These defaults are automatically tuned based on environment and dataset size + this.cacheConfig = { + // Enable auto-tuning by default for optimal performance + autoTune: true, + + // Set auto-tune interval to 1 minute for faster initial optimization + // This is especially important for large datasets + autoTuneInterval: 60000, // 1 minute + + // Read-only mode specific optimizations + readOnlyMode: { + // Use aggressive prefetching in read-only mode for better performance + prefetchStrategy: 'aggressive' + } + } + + // Override defaults with user-provided configuration if available + if (config.cache) { + this.cacheConfig = { + ...this.cacheConfig, + ...config.cache + } + } } /** @@ -840,6 +953,15 @@ export class BrainyData implements BrainyDataInterface { ...this.storageConfig, requestPersistentStorage: this.requestPersistentStorage } + + // Add cache configuration if provided + if (this.cacheConfig) { + storageOptions.cacheConfig = { + ...this.cacheConfig, + // Pass read-only flag to optimize cache behavior + readOnly: this.readOnly + } + } // Ensure s3Storage has all required fields if it's provided if (storageOptions.s3Storage) { diff --git a/src/storage/cacheManager.ts b/src/storage/cacheManager.ts index 42a4a1a8..0c1c5404 100644 --- a/src/storage/cacheManager.ts +++ b/src/storage/cacheManager.ts @@ -58,7 +58,8 @@ enum StorageType { MEMORY, OPFS, FILESYSTEM, - S3 + S3, + REMOTE_API } /** @@ -98,6 +99,18 @@ export class CacheManager { private warmStorage: any private coldStorage: any + // Store options for later reference + private options: { + hotCacheMaxSize?: number + hotCacheEvictionThreshold?: number + warmCacheTTL?: number + batchSize?: number + autoTune?: boolean + warmStorage?: any + coldStorage?: any + readOnly?: boolean + } + /** * Initialize the cache manager * @param options Configuration options @@ -110,7 +123,11 @@ export class CacheManager { autoTune?: boolean warmStorage?: any coldStorage?: any + readOnly?: boolean } = {}) { + // Store options for later reference + this.options = options + // Detect environment this.environment = this.detectEnvironment() @@ -166,14 +183,30 @@ export class CacheManager { } /** - * Detect the optimal cache size based on available memory + * Detect the optimal cache size based on available memory and operating mode + * + * Enhanced to better handle large datasets in S3 or other storage: + * - Increases cache size for read-only mode + * - Adjusts based on total dataset size when available + * - Provides more aggressive caching for large datasets + * - Optimizes memory usage based on environment */ private detectOptimalCacheSize(): number { try { // Default to a conservative value const defaultSize = 1000 - // In Node.js, use available system memory + // Get the total dataset size if available + const totalItems = this.storageStatistics ? + (this.storageStatistics.totalNodes || 0) + (this.storageStatistics.totalEdges || 0) : 0 + + // Determine if we're dealing with a large dataset (>100K items) + const isLargeDataset = totalItems > 100000 + + // Check if we're in read-only mode (from parent BrainyData instance) + const isReadOnly = this.options?.readOnly || false + + // In Node.js, use available system memory with enhanced allocation if (this.environment === Environment.NODE) { try { // Use dynamic import to avoid ESLint warning @@ -182,18 +215,46 @@ export class CacheManager { return require('os') } const os = getOS() + const totalMemory = os.totalmem() const freeMemory = os.freemem() // Estimate average entry size (in bytes) // This is a conservative estimate for complex objects with vectors const ESTIMATED_BYTES_PER_ENTRY = 1024 // 1KB per entry - // Use 10% of free memory, with a minimum of 1000 entries + // Base memory percentage - 10% by default + let memoryPercentage = 0.1 + + // Adjust based on operating mode and dataset size + if (isReadOnly) { + // In read-only mode, we can use more memory for caching + memoryPercentage = 0.25 // 25% of free memory + + // For large datasets in read-only mode, be even more aggressive + if (isLargeDataset) { + memoryPercentage = 0.4 // 40% of free memory + } + } else if (isLargeDataset) { + // For large datasets in normal mode, increase slightly + memoryPercentage = 0.15 // 15% of free memory + } + + // Calculate optimal size based on adjusted percentage const optimalSize = Math.max( - Math.floor(freeMemory * 0.1 / ESTIMATED_BYTES_PER_ENTRY), + Math.floor(freeMemory * memoryPercentage / ESTIMATED_BYTES_PER_ENTRY), 1000 ) + // If we know the total dataset size, cap at a reasonable percentage + if (totalItems > 0) { + // In read-only mode, we can cache a larger percentage + const maxPercentage = isReadOnly ? 0.5 : 0.3 + const maxItems = Math.ceil(totalItems * maxPercentage) + + // Return the smaller of the two to avoid excessive memory usage + return Math.min(optimalSize, maxItems) + } + return optimalSize } catch (error) { console.warn('Failed to detect optimal cache size:', error) @@ -201,10 +262,42 @@ export class CacheManager { } } - // In browser, use navigator.deviceMemory if available + // In browser, use navigator.deviceMemory with enhanced allocation if (this.environment === Environment.BROWSER && navigator.deviceMemory) { - // deviceMemory is in GB, scale accordingly - return Math.max(navigator.deviceMemory * 500, 1000) + // Base entries per GB + let entriesPerGB = 500 + + // Adjust based on operating mode and dataset size + if (isReadOnly) { + entriesPerGB = 800 // More aggressive caching in read-only mode + + if (isLargeDataset) { + entriesPerGB = 1000 // Even more aggressive for large datasets + } + } else if (isLargeDataset) { + entriesPerGB = 600 // Slightly more aggressive for large datasets + } + + // Calculate based on device memory + const browserCacheSize = Math.max(navigator.deviceMemory * entriesPerGB, 1000) + + // If we know the total dataset size, cap at a reasonable percentage + if (totalItems > 0) { + // In read-only mode, we can cache a larger percentage + const maxPercentage = isReadOnly ? 0.4 : 0.25 + const maxItems = Math.ceil(totalItems * maxPercentage) + + // Return the smaller of the two to avoid excessive memory usage + return Math.min(browserCacheSize, maxItems) + } + + return browserCacheSize + } + + // For worker environments or when memory detection fails + if (this.environment === Environment.WORKER) { + // Workers typically have limited memory, be conservative + return isReadOnly ? 2000 : 1000 } return defaultSize @@ -273,23 +366,34 @@ export class CacheManager { } /** - * Tune hot cache size based on statistics and environment + * Tune hot cache size based on statistics, environment, and operating mode * * The hot cache size is tuned based on: * 1. Available memory in the current environment * 2. Total number of nodes and edges in the system * 3. Cache hit/miss ratio + * 4. Operating mode (read-only vs. read-write) + * 5. Storage type (S3, filesystem, memory) * - * Algorithm: - * - Start with a size based on available memory - * - If storage statistics are available, consider caching a percentage of total items - * - If hit ratio is low, increase the cache size to improve performance - * - Ensure a reasonable minimum size to maintain basic functionality + * Enhanced algorithm: + * - Start with a size based on available memory and operating mode + * - For large datasets in S3 or other remote storage, use more aggressive caching + * - Adjust based on access patterns (read-heavy vs. write-heavy) + * - For read-only mode, prioritize cache size over eviction speed + * - Dynamically adjust based on hit/miss ratio and query patterns */ private tuneHotCacheSize(): void { // Start with the base size from environment detection let optimalSize = this.detectOptimalCacheSize() + // Check if we're in read-only mode + const isReadOnly = this.options?.readOnly || false + + // Check if we're using S3 or other remote storage + const isRemoteStorage = + this.coldStorageType === StorageType.S3 || + this.coldStorageType === StorageType.REMOTE_API + // If we have storage statistics, adjust based on total nodes/edges if (this.storageStatistics) { const totalItems = (this.storageStatistics.totalNodes || 0) + @@ -297,8 +401,30 @@ export class CacheManager { // If total items is significant, adjust cache size if (totalItems > 0) { - // Use a percentage of total items, with a cap based on memory - const percentageToCache = 0.2 // Cache 20% of items by default + // Base percentage to cache - adjusted based on mode and storage + let percentageToCache = 0.2 // Cache 20% of items by default + + // For read-only mode, increase cache percentage + if (isReadOnly) { + percentageToCache = 0.3 // 30% for read-only mode + + // For remote storage in read-only mode, be even more aggressive + if (isRemoteStorage) { + percentageToCache = 0.4 // 40% for remote storage in read-only mode + } + } + // For remote storage in normal mode, increase slightly + else if (isRemoteStorage) { + percentageToCache = 0.25 // 25% for remote storage + } + + // For large datasets, cap the percentage to avoid excessive memory usage + if (totalItems > 1000000) { // Over 1 million items + percentageToCache = Math.min(percentageToCache, 0.15) + } else if (totalItems > 100000) { // Over 100K items + percentageToCache = Math.min(percentageToCache, 0.25) + } + const statisticsBasedSize = Math.ceil(totalItems * percentageToCache) // Use the smaller of the two to avoid memory issues @@ -311,17 +437,67 @@ export class CacheManager { if (totalAccesses > 100) { const hitRatio = this.stats.hits / totalAccesses - // If hit ratio is high, we might have a good cache size already + // Base adjustment factor + let hitRatioFactor = 1.0 + // If hit ratio is low, we might need a larger cache if (hitRatio < 0.5) { - // Increase cache size by up to 50% if hit ratio is low - const hitRatioFactor = 1 + (0.5 - hitRatio) + // Calculate adjustment factor based on hit ratio + const baseAdjustment = 0.5 - hitRatio + + // For read-only mode or remote storage, be more aggressive + if (isReadOnly || isRemoteStorage) { + hitRatioFactor = 1 + (baseAdjustment * 1.5) // Up to 75% increase + } else { + hitRatioFactor = 1 + baseAdjustment // Up to 50% increase + } + + optimalSize = Math.ceil(optimalSize * hitRatioFactor) + } + // If hit ratio is very high, we might be able to reduce cache size slightly + else if (hitRatio > 0.9 && !isReadOnly && !isRemoteStorage) { + // Only reduce cache size in normal mode with local storage + // and only if hit ratio is very high + hitRatioFactor = 0.9 // 10% reduction optimalSize = Math.ceil(optimalSize * hitRatioFactor) } } - // Ensure we have a reasonable minimum size - optimalSize = Math.max(optimalSize, 1000) + // Check for operation patterns if available + if (this.storageStatistics?.operations) { + const ops = this.storageStatistics.operations + const totalOps = ops.total || 1 + + // Calculate read/write ratio + const readOps = (ops.search || 0) + (ops.get || 0) + const writeOps = (ops.add || 0) + (ops.update || 0) + (ops.delete || 0) + + if (totalOps > 100) { + const readRatio = readOps / totalOps + + // For read-heavy workloads, increase cache size + if (readRatio > 0.8) { + // More aggressive for remote storage + const readAdjustment = isRemoteStorage ? 1.3 : 1.2 + optimalSize = Math.ceil(optimalSize * readAdjustment) + } + } + } + + // Ensure we have a reasonable minimum size based on environment and mode + let minSize = 1000 // Default minimum + + // For read-only mode, use a higher minimum + if (isReadOnly) { + minSize = 2000 + } + + // For remote storage, use an even higher minimum + if (isRemoteStorage) { + minSize = isReadOnly ? 3000 : 2000 + } + + optimalSize = Math.max(optimalSize, minSize) // Update the hot cache max size this.hotCacheMaxSize = optimalSize @@ -434,7 +610,7 @@ export class CacheManager { } /** - * Tune batch size based on statistics and environment + * Tune batch size based on environment, statistics, and operating mode * * The batch size determines how many items are processed in a single batch * for operations like prefetching. It is tuned based on: @@ -442,46 +618,105 @@ export class CacheManager { * 2. Available memory * 3. Operation patterns * 4. Cache hit/miss ratio + * 5. Operating mode (read-only vs. read-write) + * 6. Storage type (S3, filesystem, memory) + * 7. Dataset size * - * Algorithm: + * Enhanced algorithm: * - Start with a default based on the environment - * - Adjust based on available memory in browsers - * - For bulk-heavy workloads, use a larger batch size - * - For high hit ratios, use smaller batches (items likely in cache) - * - For low hit ratios, use larger batches (need to fetch more items) + * - For large datasets in S3 or other remote storage, use larger batches + * - For read-only mode, use larger batches to improve throughput + * - Dynamically adjust based on network latency and throughput + * - Balance between memory usage and performance */ private tuneBatchSize(): void { // Default batch size let batchSize = 10 - // Adjust based on environment + // Check if we're in read-only mode + const isReadOnly = this.options?.readOnly || false + + // Check if we're using S3 or other remote storage + const isRemoteStorage = + this.coldStorageType === StorageType.S3 || + this.coldStorageType === StorageType.REMOTE_API + + // Get the total dataset size if available + const totalItems = this.storageStatistics ? + (this.storageStatistics.totalNodes || 0) + (this.storageStatistics.totalEdges || 0) : 0 + + // Determine if we're dealing with a large dataset + const isLargeDataset = totalItems > 100000 + const isVeryLargeDataset = totalItems > 1000000 + + // Base batch size adjustment based on environment if (this.environment === Environment.NODE) { // Node.js can handle larger batches - batchSize = 20 + batchSize = isReadOnly ? 30 : 20 + + // For remote storage, increase batch size + if (isRemoteStorage) { + batchSize = isReadOnly ? 50 : 30 + } + + // For large datasets, adjust batch size + if (isLargeDataset) { + batchSize = Math.min(100, batchSize * 1.5) + } + + // For very large datasets, adjust even more + if (isVeryLargeDataset) { + batchSize = Math.min(200, batchSize * 2) + } } else if (this.environment === Environment.BROWSER) { // Browsers might need smaller batches - batchSize = 10 + batchSize = isReadOnly ? 15 : 10 // If we have memory information, adjust accordingly if (navigator.deviceMemory) { // Scale batch size with available memory - batchSize = Math.max(5, Math.min(20, Math.floor(navigator.deviceMemory * 2))) + const memoryFactor = isReadOnly ? 3 : 2 + batchSize = Math.max(5, Math.min(30, Math.floor(navigator.deviceMemory * memoryFactor))) + + // For large datasets, adjust based on memory + if (isLargeDataset && navigator.deviceMemory > 4) { + batchSize = Math.min(50, batchSize * 1.5) + } } + } else if (this.environment === Environment.WORKER) { + // Workers can handle moderate batch sizes + batchSize = isReadOnly ? 20 : 15 } // If we have storage statistics with operation counts, adjust based on operation patterns if (this.storageStatistics && this.storageStatistics.operations) { const ops = this.storageStatistics.operations const totalOps = ops.total || 1 - const bulkOps = (ops.search || 0) + const searchOps = (ops.search || 0) + const getOps = (ops.get || 0) if (totalOps > 100) { - const bulkRatio = bulkOps / totalOps + // Calculate search and get ratios + const searchRatio = searchOps / totalOps + const getRatio = getOps / totalOps - // For bulk-heavy workloads, use larger batch size - if (bulkRatio > 0.7) { - // Bulk-heavy, increase batch size (up to 2x) - batchSize = Math.min(50, Math.ceil(batchSize * 1.5)) + // For search-heavy workloads, use larger batch size + if (searchRatio > 0.6) { + // Search-heavy, increase batch size + const searchFactor = isRemoteStorage ? 1.8 : 1.5 + batchSize = Math.min(isRemoteStorage ? 200 : 100, Math.ceil(batchSize * searchFactor)) + } + + // For get-heavy workloads, adjust batch size + if (getRatio > 0.6) { + // Get-heavy, adjust batch size based on storage type + if (isRemoteStorage) { + // For remote storage, larger batches reduce network overhead + batchSize = Math.min(150, Math.ceil(batchSize * 1.5)) + } else { + // For local storage, smaller batches might be more efficient + batchSize = Math.max(10, Math.ceil(batchSize * 0.9)) + } } } } @@ -491,17 +726,54 @@ export class CacheManager { if (totalAccesses > 100) { const hitRatio = this.stats.hits / totalAccesses + // Base adjustment factors + let increaseFactorForLowHitRatio = isRemoteStorage ? 1.5 : 1.2 + let decreaseFactorForHighHitRatio = 0.8 + + // In read-only mode, be more aggressive with batch size adjustments + if (isReadOnly) { + increaseFactorForLowHitRatio = isRemoteStorage ? 2.0 : 1.5 + decreaseFactorForHighHitRatio = 0.9 // Less reduction in read-only mode + } + // If hit ratio is high, we can use smaller batches - // If hit ratio is low, we might need larger batches - if (hitRatio > 0.8) { + if (hitRatio > 0.8 && !isVeryLargeDataset) { // High hit ratio, decrease batch size slightly - batchSize = Math.max(5, Math.floor(batchSize * 0.8)) - } else if (hitRatio < 0.5) { + // But don't decrease too much for large datasets or remote storage + if (!(isLargeDataset && isRemoteStorage)) { + batchSize = Math.max(isReadOnly ? 10 : 5, Math.floor(batchSize * decreaseFactorForHighHitRatio)) + } + } + // If hit ratio is low, we need larger batches + else if (hitRatio < 0.5) { // Low hit ratio, increase batch size - batchSize = Math.min(50, Math.ceil(batchSize * 1.2)) + const maxBatchSize = isRemoteStorage ? + (isVeryLargeDataset ? 300 : 200) : + (isVeryLargeDataset ? 150 : 100) + + batchSize = Math.min(maxBatchSize, Math.ceil(batchSize * increaseFactorForLowHitRatio)) } } + // Set minimum batch sizes based on storage type and mode + let minBatchSize = 5 + + if (isRemoteStorage) { + minBatchSize = isReadOnly ? 20 : 10 + } else if (isReadOnly) { + minBatchSize = 10 + } + + // Ensure batch size is within reasonable limits + batchSize = Math.max(minBatchSize, batchSize) + + // Cap maximum batch size based on environment and storage + const maxBatchSize = isRemoteStorage ? + (this.environment === Environment.NODE ? 300 : 150) : + (this.environment === Environment.NODE ? 150 : 75) + + batchSize = Math.min(maxBatchSize, batchSize) + // Update the batch size this.batchSize = batchSize } diff --git a/src/storage/storageFactory.ts b/src/storage/storageFactory.ts index bb9f4831..56d95575 100644 --- a/src/storage/storageFactory.ts +++ b/src/storage/storageFactory.ts @@ -171,6 +171,57 @@ export interface StorageOptions { * Operation configuration for timeout and retry behavior */ operationConfig?: OperationConfig + + /** + * Cache configuration for optimizing data access + * Particularly important for S3 and other remote storage + */ + cacheConfig?: { + /** + * Maximum size of the hot cache (most frequently accessed items) + * For large datasets, consider values between 5000-50000 depending on available memory + */ + hotCacheMaxSize?: number + + /** + * Threshold at which to start evicting items from the hot cache + * Expressed as a fraction of hotCacheMaxSize (0.0 to 1.0) + * Default: 0.8 (start evicting when cache is 80% full) + */ + hotCacheEvictionThreshold?: number + + /** + * Time-to-live for items in the warm cache in milliseconds + * Default: 3600000 (1 hour) + */ + warmCacheTTL?: number + + /** + * Batch size for operations like prefetching + * Larger values improve throughput but use more memory + */ + batchSize?: number + + /** + * Whether to enable auto-tuning of cache parameters + * When true, the system will automatically adjust cache sizes based on usage patterns + * Default: true + */ + autoTune?: boolean + + /** + * The interval (in milliseconds) at which to auto-tune cache parameters + * Only applies when autoTune is true + * Default: 60000 (1 minute) + */ + autoTuneInterval?: number + + /** + * Whether the storage is in read-only mode + * This affects cache sizing and prefetching strategies + */ + readOnly?: boolean + } } /** @@ -245,7 +296,8 @@ export async function createStorage( secretAccessKey: options.s3Storage.secretAccessKey, sessionToken: options.s3Storage.sessionToken, serviceType: 's3', - operationConfig: options.operationConfig + operationConfig: options.operationConfig, + cacheConfig: options.cacheConfig }) } else { console.warn('S3 storage configuration is missing, falling back to memory storage') @@ -260,7 +312,8 @@ export async function createStorage( accountId: options.r2Storage.accountId, accessKeyId: options.r2Storage.accessKeyId, secretAccessKey: options.r2Storage.secretAccessKey, - serviceType: 'r2' + serviceType: 'r2', + cacheConfig: options.cacheConfig }) } else { console.warn('R2 storage configuration is missing, falling back to memory storage') @@ -276,7 +329,8 @@ export async function createStorage( endpoint: options.gcsStorage.endpoint || 'https://storage.googleapis.com', accessKeyId: options.gcsStorage.accessKeyId, secretAccessKey: options.gcsStorage.secretAccessKey, - serviceType: 'gcs' + serviceType: 'gcs', + cacheConfig: options.cacheConfig }) } else { console.warn('GCS storage configuration is missing, falling back to memory storage') @@ -298,7 +352,8 @@ export async function createStorage( endpoint: options.customS3Storage.endpoint, accessKeyId: options.customS3Storage.accessKeyId, secretAccessKey: options.customS3Storage.secretAccessKey, - serviceType: options.customS3Storage.serviceType || 'custom' + serviceType: options.customS3Storage.serviceType || 'custom', + cacheConfig: options.cacheConfig }) } @@ -310,7 +365,8 @@ export async function createStorage( accountId: options.r2Storage.accountId, accessKeyId: options.r2Storage.accessKeyId, secretAccessKey: options.r2Storage.secretAccessKey, - serviceType: 'r2' + serviceType: 'r2', + cacheConfig: options.cacheConfig }) } @@ -323,7 +379,8 @@ export async function createStorage( accessKeyId: options.s3Storage.accessKeyId, secretAccessKey: options.s3Storage.secretAccessKey, sessionToken: options.s3Storage.sessionToken, - serviceType: 's3' + serviceType: 's3', + cacheConfig: options.cacheConfig }) } @@ -336,7 +393,8 @@ export async function createStorage( endpoint: options.gcsStorage.endpoint || 'https://storage.googleapis.com', accessKeyId: options.gcsStorage.accessKeyId, secretAccessKey: options.gcsStorage.secretAccessKey, - serviceType: 'gcs' + serviceType: 'gcs', + cacheConfig: options.cacheConfig }) }