feat: v0.49 - Filter discovery API, remove deprecated methods, improve performance

BREAKING CHANGES:
- Removed deprecated getAllNouns() and getAllVerbs() methods
- All internal usage migrated to pagination-based methods

New Features:
- Filter Discovery API:
  - getFilterValues(field): Get all available values for a field
  - getFilterFields(): Get all filterable fields
  - Enables dynamic filter UI generation with O(1) field discovery
- Hybrid metadata indexing with field-level indexes
- Adaptive auto-flush for optimal performance
- LRU caching for metadata indexes

Improvements:
- Fixed ENAMETOOLONG errors from vector-based filenames
- Safe filename generation using hash-based approach
- Scalable chunked value storage for millions of entries
- Performance optimization with adaptive flush thresholds
- Added support for $includes operator in metadata filters

Technical:
- Replaced vector-based filenames with safe hash approach
- Implemented MetadataIndexCache with existing SearchCache pattern
- Field indexes enable O(1) filter discovery
- Adaptive flush based on performance metrics (20-200 entries)
- All tests passing with improved metadata filtering

docs/api-reference/search-methods.md

@@ -427,6 +427,73 @@ Brainy validates metadata queries and provides helpful error messages:
 
 ---
 
+## Filter Discovery API (v0.49+)
+
+### getFilterValues(field)
+
+Get all available values for a specific field that can be used in filters:
+
+```javascript
+// Get all categories in the database
+const categories = await brainy.getFilterValues('category')
+// Returns: ['electronics', 'books', 'clothing', 'home', ...]
+
+// Get all brands
+const brands = await brainy.getFilterValues('brand')
+// Returns: ['apple', 'samsung', 'sony', ...]
+
+// Use discovered values in filters
+const results = await brainy.search("products", 10, {
+  metadata: {
+    category: { $in: categories.slice(0, 3) },  // First 3 categories
+    brand: brands[0]                            // Specific brand
+  }
+})
+```
+
+### getFilterFields()
+
+Get all fields that have been indexed and can be used for filtering:
+
+```javascript
+// Discover what fields are available
+const fields = await brainy.getFilterFields()
+// Returns: ['category', 'price', 'brand', 'rating', 'tags', ...]
+
+// Build dynamic filters based on available fields
+const filter = {}
+if (fields.includes('category')) {
+  filter.category = 'electronics'
+}
+if (fields.includes('price')) {
+  filter.price = { $lte: 1000 }
+}
+
+const results = await brainy.search("query", 10, { metadata: filter })
+```
+
+### Use Cases
+
+1. **Dynamic UI Generation**: Build filter dropdowns from actual data
+2. **Data Exploration**: Understand what metadata exists
+3. **Validation**: Check if a field exists before filtering
+4. **Analytics**: See distribution of values
+
+```javascript
+// Build a filter UI dynamically
+async function buildFilterUI() {
+  const fields = await brainy.getFilterFields()
+  
+  for (const field of fields) {
+    const values = await brainy.getFilterValues(field)
+    console.log(`${field}: ${values.length} unique values`)
+    
+    // Create dropdown/checkbox for each field
+    createFilterControl(field, values)
+  }
+}
+```
+
 ## Migration Guide
 
 ### From Simple Filtering