feat: v0.49 - Filter discovery API, remove deprecated methods, improve performance

BREAKING CHANGES:
- Removed deprecated getAllNouns() and getAllVerbs() methods
- All internal usage migrated to pagination-based methods

New Features:
- Filter Discovery API:
  - getFilterValues(field): Get all available values for a field
  - getFilterFields(): Get all filterable fields
  - Enables dynamic filter UI generation with O(1) field discovery
- Hybrid metadata indexing with field-level indexes
- Adaptive auto-flush for optimal performance
- LRU caching for metadata indexes

Improvements:
- Fixed ENAMETOOLONG errors from vector-based filenames
- Safe filename generation using hash-based approach
- Scalable chunked value storage for millions of entries
- Performance optimization with adaptive flush thresholds
- Added support for $includes operator in metadata filters

Technical:
- Replaced vector-based filenames with safe hash approach
- Implemented MetadataIndexCache with existing SearchCache pattern
- Field indexes enable O(1) filter discovery
- Adaptive flush based on performance metrics (20-200 entries)
- All tests passing with improved metadata filtering
This commit is contained in:
David Snelling 2025-08-06 14:39:33 -07:00
parent ac5b3183e3
commit 2dc909909a
17 changed files with 942 additions and 486 deletions

View file

@ -11,7 +11,26 @@
</div>
## 🔥 MAJOR UPDATES: What's New in v0.46+ & v0.48+
## 🔥 MAJOR UPDATES: What's New in v0.49, v0.48 & v0.46+
### 🎯 **v0.49: Filter Discovery & Performance Improvements**
**Discover available filters and scale to millions of items!**
```javascript
// Discover what filters are available
const categories = await brainy.getFilterValues('category')
// Returns: ['electronics', 'books', 'clothing', ...]
const fields = await brainy.getFilterFields()
// Returns: ['category', 'price', 'brand', 'rating', ...]
```
- ✅ **Filter Discovery API**: See what values are available for filtering
- ✅ **Improved Performance**: Removed deprecated methods, now uses pagination everywhere
- ✅ **Better Scalability**: Hybrid indexing approach handles millions of items
- ✅ **Smart Caching**: LRU cache for frequently accessed filters
- ✅ **Zero Configuration**: Everything auto-optimizes based on usage patterns
### 🚀 **v0.48: MongoDB-Style Metadata Filtering**