fix(storage): add universal batch metadata reading to ALL storage adapters

CRITICAL FIX: getMetadataBatch was only implemented in S3CompatibleStorage,
causing other adapters to fall back to individual calls = socket exhaustion\!

Universal Implementation:
- Add getMetadataBatch() to MemoryStorage (in-memory batch processing)
- Add getMetadataBatch() to FileSystemStorage (10 concurrent file reads)
- Add getMetadataBatch() to OPFSStorage (10 concurrent OPFS operations)
- Enhanced S3CompatibleStorage with adaptive delays and timeout handling

Enhanced Debugging:
- Log storage adapter type and batch availability
- Clear fallback warnings if batch processing unavailable
- Progress reporting with success rates
- Better timeout and error handling

This ensures socket exhaustion prevention works regardless of storage adapter.
Services using Memory/FileSystem/OPFS storage will now use batch processing
instead of 1400+ individual getMetadata() calls during initialization.

Expected result: Services initialize successfully across ALL storage types
This commit is contained in:
David Snelling 2025-08-07 12:14:23 -07:00
parent 019b578d7d
commit 49236f449d
5 changed files with 137 additions and 8 deletions

View file

@ -492,6 +492,45 @@ export class FileSystemStorage extends BaseStorage {
}
}
/**
* Get multiple metadata objects in batches (CRITICAL: Prevents socket exhaustion)
* FileSystem implementation uses controlled concurrency to prevent too many file reads
*/
public async getMetadataBatch(ids: string[]): Promise<Map<string, any>> {
await this.ensureInitialized()
const results = new Map<string, any>()
const batchSize = 10 // Process 10 files at a time
// Process in batches to avoid overwhelming the filesystem
for (let i = 0; i < ids.length; i += batchSize) {
const batch = ids.slice(i, i + batchSize)
const batchPromises = batch.map(async (id) => {
try {
const metadata = await this.getMetadata(id)
return { id, metadata }
} catch (error) {
console.debug(`Failed to read metadata for ${id}:`, error)
return { id, metadata: null }
}
})
const batchResults = await Promise.all(batchPromises)
for (const { id, metadata } of batchResults) {
if (metadata !== null) {
results.set(id, metadata)
}
}
// Small yield between batches
await new Promise(resolve => setImmediate(resolve))
}
return results
}
/**
* Save noun metadata to storage
*/