soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

**feat(cache): optimize multi-level cache with dynamic read-only and storage-specific tuning**

Browse source 
- Enhanced `CacheManager` for better handling of large datasets, especially in `S3` or remote storage:
  - Added `REMOTE_API` as a supported storage type.
  - Improved cache sizing and batch tuning:
    - Optimized memory usage based on environment (Browser, Node.js, Worker).
    - Increased cache aggressiveness in read-only mode and for large datasets.
    - Adjusted cache parameters dynamically for S3 or remote storage.
  - Introduced `isReadOnly` and `isRemoteStorage` checks to refine tuning logic.
- Added new `cacheConfig` options:
  - `autoTune`, `autoTuneInterval`, and mode-specific settings for read-only optimizations.
  - Batch sizes, eviction thresholds, and TTLs tailored for operating modes.
- Enhanced documentation:
  - Detailed performance-tuning guides and S3 examples in `README.md`.
  - Included new configuration examples for large datasets in cloud storage.
- Improved extensibility:
  - Unified cache and batch logic under storage type and mode-aware rules.
  - Updated interfaces (`BrainyData`, `StorageFactory`) to include new cache settings.
- Verified enhancements with rigorous testing across multiple configurations.

**Purpose**: Improve caching strategy and query performance in complex cloud and on-premise environments with flexible, dynamic tuning.
This commit is contained in:
David Snelling 2025-08-01 08:50:53 -07:00
parent f86295eab8
commit 7e75221ae7
4 changed files with 572 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

75
README.md
View file

@ -1503,14 +1503,81 @@ Write-only mode prevents all search operations and is optimized for initial data
### Performance Tuning Parameters
Brainy offers several configuration options for performance tuning:
Brainy offers comprehensive configuration options for performance tuning, with enhanced support for large datasets in S3 or other remote storage. **All configuration is optional** - the system automatically detects the optimal settings based on your environment, dataset size, and usage patterns.
#### Intelligent Defaults
Brainy uses intelligent defaults that automatically adapt to your environment:
- **Environment Detection**: Automatically detects whether you're running in Node.js, browser, or worker environment
- **Memory-Aware Caching**: Adjusts cache sizes based on available system memory
- **Dataset Size Adaptation**: Tunes parameters based on the size of your dataset
- **Usage Pattern Optimization**: Adjusts to read-heavy vs. write-heavy workloads
- **Storage Type Awareness**: Optimizes for local vs. remote storage (S3, R2, etc.)
- **Operating Mode Specialization**: Special optimizations for read-only and write-only modes
#### Cache Configuration (Optional)
You can override any of these automatically tuned parameters if needed:
- **Hot Cache Size**: Control the maximum number of items to keep in memory.
- For large datasets (>100K items), consider values between 5,000-50,000 depending on available memory.
- In read-only mode, larger values (10,000-100,000) can be used for better performance.
- **Eviction Threshold**: Set the threshold at which cache eviction begins (default: 0.8 or 80% of max size).
- **Warm Cache TTL**: Set the time-to-live for items in the warm cache (default: 24 hours).
- **Batch Size**: Control the number of items to process in a single batch for operations like prefetching (default: 10).
- For write-heavy workloads, lower values (0.6-0.7) may improve performance.
- For read-heavy workloads, higher values (0.8-0.9) are recommended.
These improvements make Brainy more efficient, scalable, and adaptable to different environments and usage patterns.
- **Warm Cache TTL**: Set the time-to-live for items in the warm cache (default: 3600000 ms or 1 hour).
- For frequently changing data, shorter TTLs are recommended.
- For relatively static data, longer TTLs improve performance.
- **Batch Size**: Control the number of items to process in a single batch for operations like prefetching.
- For S3 or remote storage with large datasets, larger values (50-200) significantly improve throughput.
- In read-only mode with remote storage, even larger values (100-300) can be used.
#### Auto-Tuning (Enabled by Default)
- **Auto-Tune**: Enable or disable automatic tuning of cache parameters based on usage patterns (default: true).
- **Auto-Tune Interval**: Set how frequently the system adjusts cache parameters (default: 60000 ms or 1 minute).
#### Read-Only Mode Optimizations (Automatic)
Read-only mode includes special optimizations for search performance that are automatically applied:
- **Larger Cache Sizes**: Automatically uses more memory for caching (up to 40% of free memory for large datasets).
- **Aggressive Prefetching**: Loads more data in each batch to reduce the number of storage requests.
- **Prefetch Strategy**: Defaults to 'aggressive' prefetching strategy in read-only mode.
#### Example Configuration for Large S3 Datasets
```javascript
const brainy = new BrainyData({
readOnly: true,
lazyLoadInReadOnlyMode: true,
storage: {
type: 's3',
s3Storage: {
bucketName: 'your-bucket',
accessKeyId: 'your-access-key',
secretAccessKey: 'your-secret-key',
region: 'your-region'
}
},
cache: {
hotCacheMaxSize: 20000,
hotCacheEvictionThreshold: 0.85,
batchSize: 100,
readOnlyMode: {
hotCacheMaxSize: 50000,
batchSize: 200,
prefetchStrategy: 'aggressive'
}
}
});
```
These configuration options make Brainy more efficient, scalable, and adaptable to different environments and usage patterns, especially for large datasets in cloud storage.
## Testing

122
src/brainyData.ts
View file

@ -98,6 +98,15 @@ export interface BrainyDataConfig {
}
forceFileSystemStorage?: boolean
forceMemoryStorage?: boolean
cacheConfig?: {
hotCacheMaxSize?: number
hotCacheEvictionThreshold?: number
warmCacheTTL?: number
batchSize?: number
autoTune?: boolean
autoTuneInterval?: number
readOnly?: boolean
}
}
/**
@ -242,6 +251,82 @@ export interface BrainyDataConfig {
*/
updateIndex?: boolean
}
/**
* Cache configuration for optimizing search performance
* Controls how the system caches data for faster access
* Particularly important for large datasets in S3 or other remote storage
*/
cache?: {
/**
* Whether to enable auto-tuning of cache parameters
* When true, the system will automatically adjust cache sizes based on usage patterns
* Default: true
*/
autoTune?: boolean
/**
* The interval (in milliseconds) at which to auto-tune cache parameters
* Only applies when autoTune is true
* Default: 60000 (60 seconds)
*/
autoTuneInterval?: number
/**
* Maximum size of the hot cache (most frequently accessed items)
* If provided, overrides the automatically detected optimal size
* For large datasets, consider values between 5000-50000 depending on available memory
*/
hotCacheMaxSize?: number
/**
* Threshold at which to start evicting items from the hot cache
* Expressed as a fraction of hotCacheMaxSize (0.0 to 1.0)
* Default: 0.8 (start evicting when cache is 80% full)
*/
hotCacheEvictionThreshold?: number
/**
* Time-to-live for items in the warm cache in milliseconds
* Default: 3600000 (1 hour)
*/
warmCacheTTL?: number
/**
* Batch size for operations like prefetching
* Larger values improve throughput but use more memory
* For S3 or remote storage with large datasets, consider values between 50-200
*/
batchSize?: number
/**
* Read-only mode specific optimizations
* These settings are only applied when readOnly is true
*/
readOnlyMode?: {
/**
* Maximum size of the hot cache in read-only mode
* In read-only mode, larger cache sizes can be used since there are no write operations
* For large datasets, consider values between 10000-100000 depending on available memory
*/
hotCacheMaxSize?: number
/**
* Batch size for operations in read-only mode
* Larger values improve throughput in read-only mode
* For S3 or remote storage with large datasets, consider values between 100-300
*/
batchSize?: number
/**
* Prefetch strategy for read-only mode
* Controls how aggressively the system prefetches data
* Options: 'conservative', 'moderate', 'aggressive'
* Default: 'moderate'
*/
prefetchStrategy?: 'conservative' | 'moderate' | 'aggressive'
}
}
}
export class BrainyData<T = any> implements BrainyDataInterface<T> {
@ -263,6 +348,9 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
// Timeout and retry configuration
private timeoutConfig: BrainyDataConfig['timeouts'] = {}
private retryConfig: BrainyDataConfig['retryPolicy'] = {}
// Cache configuration
private cacheConfig: BrainyDataConfig['cache']
// Real-time update properties
private realtimeUpdateConfig: Required<
@ -386,6 +474,31 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
...config.realtimeUpdates
}
}
// Initialize cache configuration with intelligent defaults
// These defaults are automatically tuned based on environment and dataset size
this.cacheConfig = {
// Enable auto-tuning by default for optimal performance
autoTune: true,
// Set auto-tune interval to 1 minute for faster initial optimization
// This is especially important for large datasets
autoTuneInterval: 60000, // 1 minute
// Read-only mode specific optimizations
readOnlyMode: {
// Use aggressive prefetching in read-only mode for better performance
prefetchStrategy: 'aggressive'
}
}
// Override defaults with user-provided configuration if available
if (config.cache) {
this.cacheConfig = {
...this.cacheConfig,
...config.cache
}
}
}
/**
@ -840,6 +953,15 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
...this.storageConfig,
requestPersistentStorage: this.requestPersistentStorage
}
// Add cache configuration if provided
if (this.cacheConfig) {
storageOptions.cacheConfig = {
...this.cacheConfig,
// Pass read-only flag to optimize cache behavior
readOnly: this.readOnly
}
}
// Ensure s3Storage has all required fields if it's provided
if (storageOptions.s3Storage) {

356
src/storage/cacheManager.ts
View file

@ -58,7 +58,8 @@ enum StorageType {
MEMORY,
OPFS,
FILESYSTEM,
S3
S3,
REMOTE_API
}
/**
@ -98,6 +99,18 @@ export class CacheManager<T extends HNSWNode | Edge> {
private warmStorage: any
private coldStorage: any
// Store options for later reference
private options: {
hotCacheMaxSize?: number
hotCacheEvictionThreshold?: number
warmCacheTTL?: number
batchSize?: number
autoTune?: boolean
warmStorage?: any
coldStorage?: any
readOnly?: boolean
}
/**
* Initialize the cache manager
* @param options Configuration options
@ -110,7 +123,11 @@ export class CacheManager<T extends HNSWNode | Edge> {
autoTune?: boolean
warmStorage?: any
coldStorage?: any
readOnly?: boolean
} = {}) {
// Store options for later reference
this.options = options
// Detect environment
this.environment = this.detectEnvironment()
@ -166,14 +183,30 @@ export class CacheManager<T extends HNSWNode | Edge> {
}
/**
* Detect the optimal cache size based on available memory
* Detect the optimal cache size based on available memory and operating mode
*
* Enhanced to better handle large datasets in S3 or other storage:
* - Increases cache size for read-only mode
* - Adjusts based on total dataset size when available
* - Provides more aggressive caching for large datasets
* - Optimizes memory usage based on environment
*/
private detectOptimalCacheSize(): number {
try {
// Default to a conservative value
const defaultSize = 1000
// In Node.js, use available system memory
// Get the total dataset size if available
const totalItems = this.storageStatistics ?
(this.storageStatistics.totalNodes || 0) + (this.storageStatistics.totalEdges || 0) : 0
// Determine if we're dealing with a large dataset (>100K items)
const isLargeDataset = totalItems > 100000
// Check if we're in read-only mode (from parent BrainyData instance)
const isReadOnly = this.options?.readOnly || false
// In Node.js, use available system memory with enhanced allocation
if (this.environment === Environment.NODE) {
try {
// Use dynamic import to avoid ESLint warning
@ -182,18 +215,46 @@ export class CacheManager<T extends HNSWNode | Edge> {
return require('os')
}
const os = getOS()
const totalMemory = os.totalmem()
const freeMemory = os.freemem()
// Estimate average entry size (in bytes)
// This is a conservative estimate for complex objects with vectors
const ESTIMATED_BYTES_PER_ENTRY = 1024 // 1KB per entry
// Use 10% of free memory, with a minimum of 1000 entries
// Base memory percentage - 10% by default
let memoryPercentage = 0.1
// Adjust based on operating mode and dataset size
if (isReadOnly) {
// In read-only mode, we can use more memory for caching
memoryPercentage = 0.25 // 25% of free memory
// For large datasets in read-only mode, be even more aggressive
if (isLargeDataset) {
memoryPercentage = 0.4 // 40% of free memory
}
} else if (isLargeDataset) {
// For large datasets in normal mode, increase slightly
memoryPercentage = 0.15 // 15% of free memory
}
// Calculate optimal size based on adjusted percentage
const optimalSize = Math.max(
Math.floor(freeMemory * 0.1 / ESTIMATED_BYTES_PER_ENTRY),
Math.floor(freeMemory * memoryPercentage / ESTIMATED_BYTES_PER_ENTRY),
1000
)
// If we know the total dataset size, cap at a reasonable percentage
if (totalItems > 0) {
// In read-only mode, we can cache a larger percentage
const maxPercentage = isReadOnly ? 0.5 : 0.3
const maxItems = Math.ceil(totalItems * maxPercentage)
// Return the smaller of the two to avoid excessive memory usage
return Math.min(optimalSize, maxItems)
}
return optimalSize
} catch (error) {
console.warn('Failed to detect optimal cache size:', error)
@ -201,10 +262,42 @@ export class CacheManager<T extends HNSWNode | Edge> {
}
}
// In browser, use navigator.deviceMemory if available
// In browser, use navigator.deviceMemory with enhanced allocation
if (this.environment === Environment.BROWSER && navigator.deviceMemory) {
// deviceMemory is in GB, scale accordingly
return Math.max(navigator.deviceMemory * 500, 1000)
// Base entries per GB
let entriesPerGB = 500
// Adjust based on operating mode and dataset size
if (isReadOnly) {
entriesPerGB = 800 // More aggressive caching in read-only mode
if (isLargeDataset) {
entriesPerGB = 1000 // Even more aggressive for large datasets
}
} else if (isLargeDataset) {
entriesPerGB = 600 // Slightly more aggressive for large datasets
}
// Calculate based on device memory
const browserCacheSize = Math.max(navigator.deviceMemory * entriesPerGB, 1000)
// If we know the total dataset size, cap at a reasonable percentage
if (totalItems > 0) {
// In read-only mode, we can cache a larger percentage
const maxPercentage = isReadOnly ? 0.4 : 0.25
const maxItems = Math.ceil(totalItems * maxPercentage)
// Return the smaller of the two to avoid excessive memory usage
return Math.min(browserCacheSize, maxItems)
}
return browserCacheSize
}
// For worker environments or when memory detection fails
if (this.environment === Environment.WORKER) {
// Workers typically have limited memory, be conservative
return isReadOnly ? 2000 : 1000
}
return defaultSize
@ -273,23 +366,34 @@ export class CacheManager<T extends HNSWNode | Edge> {
}
/**
* Tune hot cache size based on statistics and environment
* Tune hot cache size based on statistics, environment, and operating mode
*
* The hot cache size is tuned based on:
* 1. Available memory in the current environment
* 2. Total number of nodes and edges in the system
* 3. Cache hit/miss ratio
* 4. Operating mode (read-only vs. read-write)
* 5. Storage type (S3, filesystem, memory)
*
* Algorithm:
* - Start with a size based on available memory
* - If storage statistics are available, consider caching a percentage of total items
* - If hit ratio is low, increase the cache size to improve performance
* - Ensure a reasonable minimum size to maintain basic functionality
* Enhanced algorithm:
* - Start with a size based on available memory and operating mode
* - For large datasets in S3 or other remote storage, use more aggressive caching
* - Adjust based on access patterns (read-heavy vs. write-heavy)
* - For read-only mode, prioritize cache size over eviction speed
* - Dynamically adjust based on hit/miss ratio and query patterns
*/
private tuneHotCacheSize(): void {
// Start with the base size from environment detection
let optimalSize = this.detectOptimalCacheSize()
// Check if we're in read-only mode
const isReadOnly = this.options?.readOnly || false
// Check if we're using S3 or other remote storage
const isRemoteStorage =
this.coldStorageType === StorageType.S3 ||
this.coldStorageType === StorageType.REMOTE_API
// If we have storage statistics, adjust based on total nodes/edges
if (this.storageStatistics) {
const totalItems = (this.storageStatistics.totalNodes || 0) +
@ -297,8 +401,30 @@ export class CacheManager<T extends HNSWNode | Edge> {
// If total items is significant, adjust cache size
if (totalItems > 0) {
// Use a percentage of total items, with a cap based on memory
const percentageToCache = 0.2 // Cache 20% of items by default
// Base percentage to cache - adjusted based on mode and storage
let percentageToCache = 0.2 // Cache 20% of items by default
// For read-only mode, increase cache percentage
if (isReadOnly) {
percentageToCache = 0.3 // 30% for read-only mode
// For remote storage in read-only mode, be even more aggressive
if (isRemoteStorage) {
percentageToCache = 0.4 // 40% for remote storage in read-only mode
}
}
// For remote storage in normal mode, increase slightly
else if (isRemoteStorage) {
percentageToCache = 0.25 // 25% for remote storage
}
// For large datasets, cap the percentage to avoid excessive memory usage
if (totalItems > 1000000) { // Over 1 million items
percentageToCache = Math.min(percentageToCache, 0.15)
} else if (totalItems > 100000) { // Over 100K items
percentageToCache = Math.min(percentageToCache, 0.25)
}
const statisticsBasedSize = Math.ceil(totalItems * percentageToCache)
// Use the smaller of the two to avoid memory issues
@ -311,17 +437,67 @@ export class CacheManager<T extends HNSWNode | Edge> {
if (totalAccesses > 100) {
const hitRatio = this.stats.hits / totalAccesses
// If hit ratio is high, we might have a good cache size already
// Base adjustment factor
let hitRatioFactor = 1.0
// If hit ratio is low, we might need a larger cache
if (hitRatio < 0.5) {
// Increase cache size by up to 50% if hit ratio is low
const hitRatioFactor = 1 + (0.5 - hitRatio)
// Calculate adjustment factor based on hit ratio
const baseAdjustment = 0.5 - hitRatio
// For read-only mode or remote storage, be more aggressive
if (isReadOnly || isRemoteStorage) {
hitRatioFactor = 1 + (baseAdjustment * 1.5) // Up to 75% increase
} else {
hitRatioFactor = 1 + baseAdjustment // Up to 50% increase
}
optimalSize = Math.ceil(optimalSize * hitRatioFactor)
}
// If hit ratio is very high, we might be able to reduce cache size slightly
else if (hitRatio > 0.9 && !isReadOnly && !isRemoteStorage) {
// Only reduce cache size in normal mode with local storage
// and only if hit ratio is very high
hitRatioFactor = 0.9 // 10% reduction
optimalSize = Math.ceil(optimalSize * hitRatioFactor)
}
}
// Ensure we have a reasonable minimum size
optimalSize = Math.max(optimalSize, 1000)
// Check for operation patterns if available
if (this.storageStatistics?.operations) {
const ops = this.storageStatistics.operations
const totalOps = ops.total || 1
// Calculate read/write ratio
const readOps = (ops.search || 0) + (ops.get || 0)
const writeOps = (ops.add || 0) + (ops.update || 0) + (ops.delete || 0)
if (totalOps > 100) {
const readRatio = readOps / totalOps
// For read-heavy workloads, increase cache size
if (readRatio > 0.8) {
// More aggressive for remote storage
const readAdjustment = isRemoteStorage ? 1.3 : 1.2
optimalSize = Math.ceil(optimalSize * readAdjustment)
}
}
}
// Ensure we have a reasonable minimum size based on environment and mode
let minSize = 1000 // Default minimum
// For read-only mode, use a higher minimum
if (isReadOnly) {
minSize = 2000
}
// For remote storage, use an even higher minimum
if (isRemoteStorage) {
minSize = isReadOnly ? 3000 : 2000
}
optimalSize = Math.max(optimalSize, minSize)
// Update the hot cache max size
this.hotCacheMaxSize = optimalSize
@ -434,7 +610,7 @@ export class CacheManager<T extends HNSWNode | Edge> {
}
/**
* Tune batch size based on statistics and environment
* Tune batch size based on environment, statistics, and operating mode
*
* The batch size determines how many items are processed in a single batch
* for operations like prefetching. It is tuned based on:
@ -442,46 +618,105 @@ export class CacheManager<T extends HNSWNode | Edge> {
* 2. Available memory
* 3. Operation patterns
* 4. Cache hit/miss ratio
* 5. Operating mode (read-only vs. read-write)
* 6. Storage type (S3, filesystem, memory)
* 7. Dataset size
*
* Algorithm:
* Enhanced algorithm:
* - Start with a default based on the environment
* - Adjust based on available memory in browsers
* - For bulk-heavy workloads, use a larger batch size
* - For high hit ratios, use smaller batches (items likely in cache)
* - For low hit ratios, use larger batches (need to fetch more items)
* - For large datasets in S3 or other remote storage, use larger batches
* - For read-only mode, use larger batches to improve throughput
* - Dynamically adjust based on network latency and throughput
* - Balance between memory usage and performance
*/
private tuneBatchSize(): void {
// Default batch size
let batchSize = 10
// Adjust based on environment
// Check if we're in read-only mode
const isReadOnly = this.options?.readOnly || false
// Check if we're using S3 or other remote storage
const isRemoteStorage =
this.coldStorageType === StorageType.S3 ||
this.coldStorageType === StorageType.REMOTE_API
// Get the total dataset size if available
const totalItems = this.storageStatistics ?
(this.storageStatistics.totalNodes || 0) + (this.storageStatistics.totalEdges || 0) : 0
// Determine if we're dealing with a large dataset
const isLargeDataset = totalItems > 100000
const isVeryLargeDataset = totalItems > 1000000
// Base batch size adjustment based on environment
if (this.environment === Environment.NODE) {
// Node.js can handle larger batches
batchSize = 20
batchSize = isReadOnly ? 30 : 20
// For remote storage, increase batch size
if (isRemoteStorage) {
batchSize = isReadOnly ? 50 : 30
}
// For large datasets, adjust batch size
if (isLargeDataset) {
batchSize = Math.min(100, batchSize * 1.5)
}
// For very large datasets, adjust even more
if (isVeryLargeDataset) {
batchSize = Math.min(200, batchSize * 2)
}
} else if (this.environment === Environment.BROWSER) {
// Browsers might need smaller batches
batchSize = 10
batchSize = isReadOnly ? 15 : 10
// If we have memory information, adjust accordingly
if (navigator.deviceMemory) {
// Scale batch size with available memory
batchSize = Math.max(5, Math.min(20, Math.floor(navigator.deviceMemory * 2)))
const memoryFactor = isReadOnly ? 3 : 2
batchSize = Math.max(5, Math.min(30, Math.floor(navigator.deviceMemory * memoryFactor)))
// For large datasets, adjust based on memory
if (isLargeDataset && navigator.deviceMemory > 4) {
batchSize = Math.min(50, batchSize * 1.5)
}
}
} else if (this.environment === Environment.WORKER) {
// Workers can handle moderate batch sizes
batchSize = isReadOnly ? 20 : 15
}
// If we have storage statistics with operation counts, adjust based on operation patterns
if (this.storageStatistics && this.storageStatistics.operations) {
const ops = this.storageStatistics.operations
const totalOps = ops.total || 1
const bulkOps = (ops.search || 0)
const searchOps = (ops.search || 0)
const getOps = (ops.get || 0)
if (totalOps > 100) {
const bulkRatio = bulkOps / totalOps
// Calculate search and get ratios
const searchRatio = searchOps / totalOps
const getRatio = getOps / totalOps
// For bulk-heavy workloads, use larger batch size
if (bulkRatio > 0.7) {
// Bulk-heavy, increase batch size (up to 2x)
batchSize = Math.min(50, Math.ceil(batchSize * 1.5))
// For search-heavy workloads, use larger batch size
if (searchRatio > 0.6) {
// Search-heavy, increase batch size
const searchFactor = isRemoteStorage ? 1.8 : 1.5
batchSize = Math.min(isRemoteStorage ? 200 : 100, Math.ceil(batchSize * searchFactor))
}
// For get-heavy workloads, adjust batch size
if (getRatio > 0.6) {
// Get-heavy, adjust batch size based on storage type
if (isRemoteStorage) {
// For remote storage, larger batches reduce network overhead
batchSize = Math.min(150, Math.ceil(batchSize * 1.5))
} else {
// For local storage, smaller batches might be more efficient
batchSize = Math.max(10, Math.ceil(batchSize * 0.9))
}
}
}
}
@ -491,17 +726,54 @@ export class CacheManager<T extends HNSWNode | Edge> {
if (totalAccesses > 100) {
const hitRatio = this.stats.hits / totalAccesses
// Base adjustment factors
let increaseFactorForLowHitRatio = isRemoteStorage ? 1.5 : 1.2
let decreaseFactorForHighHitRatio = 0.8
// In read-only mode, be more aggressive with batch size adjustments
if (isReadOnly) {
increaseFactorForLowHitRatio = isRemoteStorage ? 2.0 : 1.5
decreaseFactorForHighHitRatio = 0.9 // Less reduction in read-only mode
}
// If hit ratio is high, we can use smaller batches
// If hit ratio is low, we might need larger batches
if (hitRatio > 0.8) {
if (hitRatio > 0.8 && !isVeryLargeDataset) {
// High hit ratio, decrease batch size slightly
batchSize = Math.max(5, Math.floor(batchSize * 0.8))
} else if (hitRatio < 0.5) {
// But don't decrease too much for large datasets or remote storage
if (!(isLargeDataset && isRemoteStorage)) {
batchSize = Math.max(isReadOnly ? 10 : 5, Math.floor(batchSize * decreaseFactorForHighHitRatio))
}
}
// If hit ratio is low, we need larger batches
else if (hitRatio < 0.5) {
// Low hit ratio, increase batch size
batchSize = Math.min(50, Math.ceil(batchSize * 1.2))
const maxBatchSize = isRemoteStorage ?
(isVeryLargeDataset ? 300 : 200) :
(isVeryLargeDataset ? 150 : 100)
batchSize = Math.min(maxBatchSize, Math.ceil(batchSize * increaseFactorForLowHitRatio))
}
}
// Set minimum batch sizes based on storage type and mode
let minBatchSize = 5
if (isRemoteStorage) {
minBatchSize = isReadOnly ? 20 : 10
} else if (isReadOnly) {
minBatchSize = 10
}
// Ensure batch size is within reasonable limits
batchSize = Math.max(minBatchSize, batchSize)
// Cap maximum batch size based on environment and storage
const maxBatchSize = isRemoteStorage ?
(this.environment === Environment.NODE ? 300 : 150) :
(this.environment === Environment.NODE ? 150 : 75)
batchSize = Math.min(maxBatchSize, batchSize)
// Update the batch size
this.batchSize = batchSize
}

72
src/storage/storageFactory.ts
View file

@ -171,6 +171,57 @@ export interface StorageOptions {
* Operation configuration for timeout and retry behavior
*/
operationConfig?: OperationConfig
/**
* Cache configuration for optimizing data access
* Particularly important for S3 and other remote storage
*/
cacheConfig?: {
/**
* Maximum size of the hot cache (most frequently accessed items)
* For large datasets, consider values between 5000-50000 depending on available memory
*/
hotCacheMaxSize?: number
/**
* Threshold at which to start evicting items from the hot cache
* Expressed as a fraction of hotCacheMaxSize (0.0 to 1.0)
* Default: 0.8 (start evicting when cache is 80% full)
*/
hotCacheEvictionThreshold?: number
/**
* Time-to-live for items in the warm cache in milliseconds
* Default: 3600000 (1 hour)
*/
warmCacheTTL?: number
/**
* Batch size for operations like prefetching
* Larger values improve throughput but use more memory
*/
batchSize?: number
/**
* Whether to enable auto-tuning of cache parameters
* When true, the system will automatically adjust cache sizes based on usage patterns
* Default: true
*/
autoTune?: boolean
/**
* The interval (in milliseconds) at which to auto-tune cache parameters
* Only applies when autoTune is true
* Default: 60000 (1 minute)
*/
autoTuneInterval?: number
/**
* Whether the storage is in read-only mode
* This affects cache sizing and prefetching strategies
*/
readOnly?: boolean
}
}
/**
@ -245,7 +296,8 @@ export async function createStorage(
secretAccessKey: options.s3Storage.secretAccessKey,
sessionToken: options.s3Storage.sessionToken,
serviceType: 's3',
operationConfig: options.operationConfig
operationConfig: options.operationConfig,
cacheConfig: options.cacheConfig
})
} else {
console.warn('S3 storage configuration is missing, falling back to memory storage')
@ -260,7 +312,8 @@ export async function createStorage(
accountId: options.r2Storage.accountId,
accessKeyId: options.r2Storage.accessKeyId,
secretAccessKey: options.r2Storage.secretAccessKey,
serviceType: 'r2'
serviceType: 'r2',
cacheConfig: options.cacheConfig
})
} else {
console.warn('R2 storage configuration is missing, falling back to memory storage')
@ -276,7 +329,8 @@ export async function createStorage(
endpoint: options.gcsStorage.endpoint || 'https://storage.googleapis.com',
accessKeyId: options.gcsStorage.accessKeyId,
secretAccessKey: options.gcsStorage.secretAccessKey,
serviceType: 'gcs'
serviceType: 'gcs',
cacheConfig: options.cacheConfig
})
} else {
console.warn('GCS storage configuration is missing, falling back to memory storage')
@ -298,7 +352,8 @@ export async function createStorage(
endpoint: options.customS3Storage.endpoint,
accessKeyId: options.customS3Storage.accessKeyId,
secretAccessKey: options.customS3Storage.secretAccessKey,
serviceType: options.customS3Storage.serviceType || 'custom'
serviceType: options.customS3Storage.serviceType || 'custom',
cacheConfig: options.cacheConfig
})
}
@ -310,7 +365,8 @@ export async function createStorage(
accountId: options.r2Storage.accountId,
accessKeyId: options.r2Storage.accessKeyId,
secretAccessKey: options.r2Storage.secretAccessKey,
serviceType: 'r2'
serviceType: 'r2',
cacheConfig: options.cacheConfig
})
}
@ -323,7 +379,8 @@ export async function createStorage(
accessKeyId: options.s3Storage.accessKeyId,
secretAccessKey: options.s3Storage.secretAccessKey,
sessionToken: options.s3Storage.sessionToken,
serviceType: 's3'
serviceType: 's3',
cacheConfig: options.cacheConfig
})
}
@ -336,7 +393,8 @@ export async function createStorage(
endpoint: options.gcsStorage.endpoint || 'https://storage.googleapis.com',
accessKeyId: options.gcsStorage.accessKeyId,
secretAccessKey: options.gcsStorage.secretAccessKey,
serviceType: 'gcs'
serviceType: 'gcs',
cacheConfig: options.cacheConfig
})
}