From 2139f6975464582bb1376757a7d87a4805d94e0e Mon Sep 17 00:00:00 2001
From: David Snelling <david@soulcraft.com>
Date: Wed, 6 Aug 2025 14:43:46 -0700
Subject: [PATCH] feat: v0.49 - Filter discovery API, remove deprecated
 methods, improve performance

BREAKING CHANGES:
- Removed deprecated getAllNouns() and getAllVerbs() methods
- All internal usage migrated to pagination-based methods

New Features:
- Filter Discovery API:
  - getFilterValues(field): Get all available values for a field
  - getFilterFields(): Get all filterable fields
  - Enables dynamic filter UI generation with O(1) field discovery
- Hybrid metadata indexing with field-level indexes
- Adaptive auto-flush for optimal performance
- LRU caching for metadata indexes

Improvements:
- Fixed ENAMETOOLONG errors from vector-based filenames
- Safe filename generation using hash-based approach
- Scalable chunked value storage for millions of entries
- Performance optimization with adaptive flush thresholds
- Added support for $includes operator in metadata filters

Technical:
- Replaced vector-based filenames with safe hash approach
- Implemented MetadataIndexCache with existing SearchCache pattern
- Field indexes enable O(1) filter discovery
- Adaptive flush based on performance metrics (20-200 entries)
- All tests passing with improved metadata filtering
---
 README.md                            |  8 ++++----
 docs/api-reference/search-methods.md | 10 +++++-----
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/README.md b/README.md
index 0b7b6a7e..3a11877a 100644
--- a/README.md
+++ b/README.md
@@ -18,17 +18,17 @@
 **Discover available filters and scale to millions of items!**
 
 ```javascript
-// Discover what filters are available
+// Discover what filters are available - O(1) field lookup
 const categories = await brainy.getFilterValues('category')
 // Returns: ['electronics', 'books', 'clothing', ...]
 
-const fields = await brainy.getFilterFields()
+const fields = await brainy.getFilterFields()  // O(1) operation
 // Returns: ['category', 'price', 'brand', 'rating', ...]
 ```
 
-- ✅ **Filter Discovery API**: See what values are available for filtering
+- ✅ **Filter Discovery API**: O(1) field discovery for instant filter UI generation
 - ✅ **Improved Performance**: Removed deprecated methods, now uses pagination everywhere
-- ✅ **Better Scalability**: Hybrid indexing approach handles millions of items
+- ✅ **Better Scalability**: Hybrid indexing with O(1) field access scales to millions
 - ✅ **Smart Caching**: LRU cache for frequently accessed filters
 - ✅ **Zero Configuration**: Everything auto-optimizes based on usage patterns
 
diff --git a/docs/api-reference/search-methods.md b/docs/api-reference/search-methods.md
index e15368b8..0a88c61d 100644
--- a/docs/api-reference/search-methods.md
+++ b/docs/api-reference/search-methods.md
@@ -431,14 +431,14 @@ Brainy validates metadata queries and provides helpful error messages:
 
 ### getFilterValues(field)
 
-Get all available values for a specific field that can be used in filters:
+Get all available values for a specific field with O(1) field lookup:
 
 ```javascript
-// Get all categories in the database
+// Get all categories in the database - O(1) field access
 const categories = await brainy.getFilterValues('category')
 // Returns: ['electronics', 'books', 'clothing', 'home', ...]
 
-// Get all brands
+// Get all brands - O(1) field lookup + O(n) value retrieval
 const brands = await brainy.getFilterValues('brand')
 // Returns: ['apple', 'samsung', 'sony', ...]
 
@@ -453,10 +453,10 @@ const results = await brainy.search("products", 10, {
 
 ### getFilterFields()
 
-Get all fields that have been indexed and can be used for filtering:
+Get all fields that have been indexed - O(1) operation:
 
 ```javascript
-// Discover what fields are available
+// Discover what fields are available - O(1) direct access
 const fields = await brainy.getFilterFields()
 // Returns: ['category', 'price', 'brand', 'rating', 'tags', ...]