soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat: Remove dangerous getAllNouns/getAllVerbs methods, add safe pagination

Browse source 
BREAKING CHANGE: Removed getAllNouns() and getAllVerbs() from StorageAdapter interface
These methods could cause expensive full scans on cloud storage (S3/R2) leading to
high costs and performance issues. Replaced with safe paginated methods.

Changes:
- Remove getAllNouns/getAllVerbs from StorageAdapter interface and implementations
- Add internal optimization methods for intelligent preloading when safe
- Fix OPFS storage file naming consistency (.json extension)
- Fix S3 high-volume mode detection thresholds (was too aggressive)
- Fix TypeScript compilation errors with async methods
- Update all tests to use paginated methods

Performance:
- Add smart dataset size detection for automatic optimization
- Maintain all internal performance optimizations through safe preloading
- Only preload data in read-only mode or when dataset is small (<10k entities)

Fixes:
- Fix intelligent verb scoring tests metadata structure
- Fix S3 storage getVerbsBySource/Target/Type methods
- Fix memory usage in search operations using pagination

Docs:
- Add comprehensive storage architecture documentation
- Document known bash redirection issue
- Update README with architecture doc link

All affected tests passing
This commit is contained in:
David Snelling 2025-08-10 16:25:12 -07:00
parent 30fe350d54
commit abc17397b1
17 changed files with 998 additions and 390 deletions
Download patch file Download diff file Expand all files Collapse all files

31
~/.npm/_logs/2025-08-10T19_24_37_677Z-debug-0.log Normal file
View file

@ -0,0 +1,31 @@
0 verbose cli /home/dpsifr/.nvm/versions/node/v24.4.1/bin/node /home/dpsifr/.nvm/versions/node/v24.4.1/bin/npm
1 info using npm@11.4.2
2 info using node@v24.4.1
3 silly config load:file:/home/dpsifr/.nvm/versions/node/v24.4.1/lib/node_modules/npm/npmrc
4 silly config load:file:/home/dpsifr/Projects/brainy/.npmrc
5 silly config load:file:/home/dpsifr/Projects/brainy/~/.npmrc
6 silly config load:file:/home/dpsifr/.nvm/versions/node/v24.4.1/etc/npmrc
7 verbose title npm test 2
8 verbose argv "test" "2"
9 verbose logfile logs-max:10 dir:/home/dpsifr/Projects/brainy/~/.npm/_logs/2025-08-10T19_24_37_677Z-
10 verbose logfile /home/dpsifr/Projects/brainy/~/.npm/_logs/2025-08-10T19_24_37_677Z-debug-0.log
11 silly logfile done cleaning log files
12 verbose stack Error: spawn sh ENOENT
12 verbose stack at ChildProcess._handle.onexit (node:internal/child_process:286:19)
12 verbose stack at onErrorNT (node:internal/child_process:484:16)
12 verbose stack at process.processTicksAndRejections (node:internal/process/task_queues:90:21)
13 verbose pkgid @soulcraft/brainy@0.59.2
14 error code ENOENT
15 error syscall spawn sh
16 error path /home/dpsifr/Projects/brainy
17 error errno -2
18 error enoent spawn sh ENOENT
19 error enoent This is related to npm not being able to find a file.
19 error enoent
20 verbose cwd /home/dpsifr/Projects/brainy
21 verbose os Linux 6.14.0-27-generic
22 verbose node v24.4.1
23 verbose npm v11.4.2
24 verbose exit -2
25 verbose code -2
26 error A complete log of this run can be found in: /home/dpsifr/Projects/brainy/~/.npm/_logs/2025-08-10T19_24_37_677Z-debug-0.log

0
~/.npm/_update-notifier-last-checked Normal file
View file