feat: Remove dangerous getAllNouns/getAllVerbs methods, add safe pagination
BREAKING CHANGE: Removed getAllNouns() and getAllVerbs() from StorageAdapter interface These methods could cause expensive full scans on cloud storage (S3/R2) leading to high costs and performance issues. Replaced with safe paginated methods. Changes: - Remove getAllNouns/getAllVerbs from StorageAdapter interface and implementations - Add internal optimization methods for intelligent preloading when safe - Fix OPFS storage file naming consistency (.json extension) - Fix S3 high-volume mode detection thresholds (was too aggressive) - Fix TypeScript compilation errors with async methods - Update all tests to use paginated methods Performance: - Add smart dataset size detection for automatic optimization - Maintain all internal performance optimizations through safe preloading - Only preload data in read-only mode or when dataset is small (<10k entities) Fixes: - Fix intelligent verb scoring tests metadata structure - Fix S3 storage getVerbsBySource/Target/Type methods - Fix memory usage in search operations using pagination Docs: - Add comprehensive storage architecture documentation - Document known bash redirection issue - Update README with architecture doc link All affected tests passing
This commit is contained in:
parent
d4ef17d1f1
commit
fb80808f44
25 changed files with 1071 additions and 390 deletions
3
test-single.sh
Executable file
3
test-single.sh
Executable file
|
|
@ -0,0 +1,3 @@
|
|||
#!/bin/bash
|
||||
# Run a single test file without the "2" issue
|
||||
./node_modules/.bin/vitest run tests/intelligent-verb-scoring.test.ts -t "should provide learning statistics"
|
||||
Loading…
Add table
Add a link
Reference in a new issue