feat: Remove dangerous getAllNouns/getAllVerbs methods, add safe pagination

Browse source

BREAKING CHANGE: Removed getAllNouns() and getAllVerbs() from StorageAdapter interface
These methods could cause expensive full scans on cloud storage (S3/R2) leading to
high costs and performance issues. Replaced with safe paginated methods.

Changes:
- Remove getAllNouns/getAllVerbs from StorageAdapter interface and implementations
- Add internal optimization methods for intelligent preloading when safe
- Fix OPFS storage file naming consistency (.json extension)
- Fix S3 high-volume mode detection thresholds (was too aggressive)
- Fix TypeScript compilation errors with async methods
- Update all tests to use paginated methods

Performance:
- Add smart dataset size detection for automatic optimization
- Maintain all internal performance optimizations through safe preloading
- Only preload data in read-only mode or when dataset is small (<10k entities)

Fixes:
- Fix intelligent verb scoring tests metadata structure
- Fix S3 storage getVerbsBySource/Target/Type methods
- Fix memory usage in search operations using pagination

Docs:
- Add comprehensive storage architecture documentation
- Document known bash redirection issue
- Update README with architecture doc link

All affected tests passing

...

This commit is contained in:

David Snelling 2025-08-10 16:25:12 -07:00

parent 30fe350d54

commit abc17397b1

17 changed files with 998 additions and 390 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

0

~/.npm/_update-notifier-last-checked Normal file

Unescape Escape View file