feat: Remove dangerous getAllNouns/getAllVerbs methods, add safe pagination

BREAKING CHANGE: Removed getAllNouns() and getAllVerbs() from StorageAdapter interface
These methods could cause expensive full scans on cloud storage (S3/R2) leading to
high costs and performance issues. Replaced with safe paginated methods.

Changes:
- Remove getAllNouns/getAllVerbs from StorageAdapter interface and implementations
- Add internal optimization methods for intelligent preloading when safe
- Fix OPFS storage file naming consistency (.json extension)
- Fix S3 high-volume mode detection thresholds (was too aggressive)
- Fix TypeScript compilation errors with async methods
- Update all tests to use paginated methods

Performance:
- Add smart dataset size detection for automatic optimization
- Maintain all internal performance optimizations through safe preloading
- Only preload data in read-only mode or when dataset is small (<10k entities)

Fixes:
- Fix intelligent verb scoring tests metadata structure
- Fix S3 storage getVerbsBySource/Target/Type methods
- Fix memory usage in search operations using pagination

Docs:
- Add comprehensive storage architecture documentation
- Document known bash redirection issue
- Update README with architecture doc link

All affected tests passing

tests/s3-storage.test.ts

@@ -187,10 +187,10 @@ describe('S3CompatibleStorage', () => {
     expect(retrievedNoun?.connections.get(0)?.has('test-noun-2')).toBe(true)
     expect(retrievedNoun?.connections.get(0)?.has('test-noun-3')).toBe(true)
     
-    // Test getAllNouns
-    const allNouns = await s3Storage.getAllNouns()
-    expect(allNouns.length).toBe(1)
-    expect(allNouns[0].id).toBe('test-noun-1')
+    // Test getNouns with pagination
+    const nounsResult = await s3Storage.getNouns({ pagination: { limit: 100 } })
+    expect(nounsResult.items.length).toBe(1)
+    expect(nounsResult.items[0].id).toBe('test-noun-1')
     
     // Test deleteNoun
     await s3Storage.deleteNoun('test-noun-1')
@@ -259,10 +259,10 @@ describe('S3CompatibleStorage', () => {
     expect(retrievedVerb?.weight).toBe(0.75)
     expect(retrievedVerb?.metadata).toEqual({ description: 'Test relation' })
     
-    // Test getAllVerbs
-    const allVerbs = await s3Storage.getAllVerbs()
-    expect(allVerbs.length).toBe(1)
-    expect(allVerbs[0].id).toBe('test-verb-1')
+    // Test getVerbs with pagination
+    const verbsResult = await s3Storage.getVerbs({ pagination: { limit: 100 } })
+    expect(verbsResult.items.length).toBe(1)
+    expect(verbsResult.items[0].id).toBe('test-verb-1')
     
     // Test getVerbsBySource
     const verbsBySource = await s3Storage.getVerbsBySource('source-noun-1')
@@ -366,9 +366,9 @@ describe('S3CompatibleStorage', () => {
       await s3Storage.saveNoun(testNoun)
     }
     
-    // Test getAllNouns
-    const allNouns = await s3Storage.getAllNouns()
-    expect(allNouns.length).toBe(nounCount)
+    // Test getNouns with pagination
+    const nounsResult = await s3Storage.getNouns({ pagination: { limit: 100 } })
+    expect(nounsResult.items.length).toBe(nounCount)
     
     // Clean up
     await s3Storage.clear()