fix(storage): add universal batch metadata reading to ALL storage adapters
CRITICAL FIX: getMetadataBatch was only implemented in S3CompatibleStorage, causing other adapters to fall back to individual calls = socket exhaustion\! Universal Implementation: - Add getMetadataBatch() to MemoryStorage (in-memory batch processing) - Add getMetadataBatch() to FileSystemStorage (10 concurrent file reads) - Add getMetadataBatch() to OPFSStorage (10 concurrent OPFS operations) - Enhanced S3CompatibleStorage with adaptive delays and timeout handling Enhanced Debugging: - Log storage adapter type and batch availability - Clear fallback warnings if batch processing unavailable - Progress reporting with success rates - Better timeout and error handling This ensures socket exhaustion prevention works regardless of storage adapter. Services using Memory/FileSystem/OPFS storage will now use batch processing instead of 1400+ individual getMetadata() calls during initialization. Expected result: Services initialize successfully across ALL storage types
This commit is contained in:
parent
019b578d7d
commit
49236f449d
5 changed files with 137 additions and 8 deletions
|
|
@ -492,6 +492,45 @@ export class FileSystemStorage extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get multiple metadata objects in batches (CRITICAL: Prevents socket exhaustion)
|
||||
* FileSystem implementation uses controlled concurrency to prevent too many file reads
|
||||
*/
|
||||
public async getMetadataBatch(ids: string[]): Promise<Map<string, any>> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
const results = new Map<string, any>()
|
||||
const batchSize = 10 // Process 10 files at a time
|
||||
|
||||
// Process in batches to avoid overwhelming the filesystem
|
||||
for (let i = 0; i < ids.length; i += batchSize) {
|
||||
const batch = ids.slice(i, i + batchSize)
|
||||
|
||||
const batchPromises = batch.map(async (id) => {
|
||||
try {
|
||||
const metadata = await this.getMetadata(id)
|
||||
return { id, metadata }
|
||||
} catch (error) {
|
||||
console.debug(`Failed to read metadata for ${id}:`, error)
|
||||
return { id, metadata: null }
|
||||
}
|
||||
})
|
||||
|
||||
const batchResults = await Promise.all(batchPromises)
|
||||
|
||||
for (const { id, metadata } of batchResults) {
|
||||
if (metadata !== null) {
|
||||
results.set(id, metadata)
|
||||
}
|
||||
}
|
||||
|
||||
// Small yield between batches
|
||||
await new Promise(resolve => setImmediate(resolve))
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
/**
|
||||
* Save noun metadata to storage
|
||||
*/
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue