feat: Remove dangerous getAllNouns/getAllVerbs methods, add safe pagination

BREAKING CHANGE: Removed getAllNouns() and getAllVerbs() from StorageAdapter interface
These methods could cause expensive full scans on cloud storage (S3/R2) leading to
high costs and performance issues. Replaced with safe paginated methods.

Changes:
- Remove getAllNouns/getAllVerbs from StorageAdapter interface and implementations
- Add internal optimization methods for intelligent preloading when safe
- Fix OPFS storage file naming consistency (.json extension)
- Fix S3 high-volume mode detection thresholds (was too aggressive)
- Fix TypeScript compilation errors with async methods
- Update all tests to use paginated methods

Performance:
- Add smart dataset size detection for automatic optimization
- Maintain all internal performance optimizations through safe preloading
- Only preload data in read-only mode or when dataset is small (<10k entities)

Fixes:
- Fix intelligent verb scoring tests metadata structure
- Fix S3 storage getVerbsBySource/Target/Type methods
- Fix memory usage in search operations using pagination

Docs:
- Add comprehensive storage architecture documentation
- Document known bash redirection issue
- Update README with architecture doc link

All affected tests passing
This commit is contained in:
David Snelling 2025-08-10 16:25:12 -07:00
parent d4ef17d1f1
commit fb80808f44
25 changed files with 1071 additions and 390 deletions

View file

@ -7,9 +7,10 @@
[![Node.js](https://img.shields.io/badge/node-%3E%3D24.4.1-brightgreen.svg)](https://nodejs.org/)
[![TypeScript](https://img.shields.io/badge/TypeScript-5.4.5-blue.svg)](https://www.typescriptlang.org/)
# BRAINY: The Brain in a Jar Database™
# BRAINY: Multi-Dimensional AI Database™
**The world's only Vector + Graph + AI database and realtime data platform**
**The world's first Multi-Dimensional AI Database**
*Vector similarity • Graph relationships • Metadata facets • AI context*
*Zero-to-Smart™ technology that thinks so you don't have to*
@ -48,13 +49,13 @@ const results = await brainy.search("AI language models", 5, {
Pinecone ($$$) + Neo4j ($$$) + Elasticsearch ($$$) + Sync Hell = 😱
```
### ✅ The Brainy Way: One Smart Brain
### ✅ The Brainy Way: One Multi-Dimensional Brain
```
Vector Search + Graph Relations + Metadata Filtering + AI Intelligence = 🧠✨
Multi-Dimensional AI Database = Vector + Graph + Facets + AI = 🧠✨
```
**Your data gets a brain upgrade. No assembly required.**
**Your data gets a multi-dimensional brain upgrade. No assembly required.**
## ⚡ QUICK & EASY: From Zero to Smart in 60 Seconds
@ -458,6 +459,7 @@ brainy augment trial notion # Start 14-day free trial
### Advanced Topics
- [**🏗️ Storage & Retrieval Architecture**](docs/technical/STORAGE_AND_RETRIEVAL_ARCHITECTURE.md) - Multi-dimensional database internals
- [**Brainy CLI**](docs/brainy-cli.md) - Command-line superpowers
- [**Brainy Chat**](BRAINY-CHAT.md) - Conversational AI interface
- [**Cortex AI**](CORTEX.md) - Intelligence augmentation