feat: Remove dangerous getAllNouns/getAllVerbs methods, add safe pagination

BREAKING CHANGE: Removed getAllNouns() and getAllVerbs() from StorageAdapter interface
These methods could cause expensive full scans on cloud storage (S3/R2) leading to
high costs and performance issues. Replaced with safe paginated methods.

Changes:
- Remove getAllNouns/getAllVerbs from StorageAdapter interface and implementations
- Add internal optimization methods for intelligent preloading when safe
- Fix OPFS storage file naming consistency (.json extension)
- Fix S3 high-volume mode detection thresholds (was too aggressive)
- Fix TypeScript compilation errors with async methods
- Update all tests to use paginated methods

Performance:
- Add smart dataset size detection for automatic optimization
- Maintain all internal performance optimizations through safe preloading
- Only preload data in read-only mode or when dataset is small (<10k entities)

Fixes:
- Fix intelligent verb scoring tests metadata structure
- Fix S3 storage getVerbsBySource/Target/Type methods
- Fix memory usage in search operations using pagination

Docs:
- Add comprehensive storage architecture documentation
- Document known bash redirection issue
- Update README with architecture doc link

All affected tests passing

src/augmentations/memoryAugmentations.ts

@@ -174,42 +174,54 @@ abstract class BaseMemoryAugmentation implements IMemoryAugmentation {
                 }
             }
 
-            // Get all nodes from storage
-            const nodes = await this.storage.getAllNouns()
-
-            // Calculate distances and prepare results
-            const results: Array<{
+            // Process nodes in batches to avoid loading everything into memory
+            const allResults: Array<{
                 id: string;
                 score: number;
                 data: unknown;
             }> = []
 
-            for (const node of nodes) {
-                // Skip nodes that don't have a vector
-                if (!node.vector || !Array.isArray(node.vector)) {
-                    continue
+            let hasMore = true
+            let cursor: string | undefined
+
+            while (hasMore) {
+                // Get a batch of nodes
+                const batchResult = await this.storage.getNouns({
+                    pagination: { limit: 100, cursor }
+                })
+
+                // Process this batch
+                for (const noun of batchResult.items) {
+                    // Skip nodes that don't have a vector
+                    if (!noun.vector || !Array.isArray(noun.vector)) {
+                        continue
+                    }
+
+                    // Get metadata for the node
+                    const metadata = await this.storage.getMetadata(noun.id)
+
+                    // Calculate distance between query vector and node vector
+                    const distance = cosineDistance(queryVector, noun.vector)
+
+                    // Convert distance to similarity score (1 - distance for cosine)
+                    // This way higher scores are better (more similar)
+                    const score = 1 - distance
+
+                    allResults.push({
+                        id: noun.id,
+                        score,
+                        data: metadata
+                    })
                 }
 
-                // Get metadata for the node
-                const metadata = await this.storage.getMetadata(node.id)
-
-                // Calculate distance between query vector and node vector
-                const distance = cosineDistance(queryVector, node.vector)
-
-                // Convert distance to similarity score (1 - distance for cosine)
-                // This way higher scores are better (more similar)
-                const score = 1 - distance
-
-                results.push({
-                    id: node.id,
-                    score,
-                    data: metadata
-                })
+                // Update pagination state
+                hasMore = batchResult.hasMore
+                cursor = batchResult.nextCursor
             }
 
             // Sort results by score (descending) and take top k
-            results.sort((a, b) => b.score - a.score)
-            const topResults = results.slice(0, k)
+            allResults.sort((a, b) => b.score - a.score)
+            const topResults = allResults.slice(0, k)
 
             return {
                 success: true,