feat: v0.49 - Filter discovery API, remove deprecated methods, improve performance

BREAKING CHANGES:
- Removed deprecated getAllNouns() and getAllVerbs() methods
- All internal usage migrated to pagination-based methods

New Features:
- Filter Discovery API:
  - getFilterValues(field): Get all available values for a field
  - getFilterFields(): Get all filterable fields
  - Enables dynamic filter UI generation with O(1) field discovery
- Hybrid metadata indexing with field-level indexes
- Adaptive auto-flush for optimal performance
- LRU caching for metadata indexes

Improvements:
- Fixed ENAMETOOLONG errors from vector-based filenames
- Safe filename generation using hash-based approach
- Scalable chunked value storage for millions of entries
- Performance optimization with adaptive flush thresholds
- Added support for $includes operator in metadata filters

Technical:
- Replaced vector-based filenames with safe hash approach
- Implemented MetadataIndexCache with existing SearchCache pattern
- Field indexes enable O(1) filter discovery
- Adaptive flush based on performance metrics (20-200 entries)
- All tests passing with improved metadata filtering

docs/guides/metadata-filtering.md

@@ -372,10 +372,33 @@ const results = await brainy.search("laptop", 10, {
    { specs: { display: "4K" } }  // ❌ Won't work as expected
    ```
 
+## 🔍 Filter Discovery API (v0.49+)
+
+Discover what filters are available in your data:
+
+```javascript
+// Get all available values for a field
+const categories = await brainy.getFilterValues('category')
+console.log('Available categories:', categories)
+// Output: ['electronics', 'books', 'clothing', ...]
+
+// Get all filterable fields
+const fields = await brainy.getFilterFields()
+console.log('Filterable fields:', fields)
+// Output: ['category', 'price', 'brand', 'rating', ...]
+
+// Build dynamic filter UI
+for (const field of fields) {
+  const values = await brainy.getFilterValues(field)
+  createDropdown(field, values)
+}
+```
+
 ## 🎉 What's Next?
 
 This powerful filtering system opens up possibilities for:
 - **Advanced search UIs** with multiple filter controls
+- **Dynamic filter discovery** to build UIs from actual data
 - **Personalized recommendations** based on user preferences  
 - **Complex business logic** in search applications
 - **Multi-tenant filtering** by organization or user

docs/guides/production-migration-guide.md

@@ -250,12 +250,24 @@ async function validateDatabase() {
   
   console.log('Validating database...');
   
-  // Get all nouns
-  const nouns = await db.getAllNouns();
-  console.log(`Found ${nouns.length} nouns.`);
+  // Get nouns using pagination (v0.49+)
+  let allNouns = [];
+  let offset = 0;
+  const limit = 100;
+  let hasMore = true;
   
-  // Check dimensions
-  const dimensionMismatches = nouns.filter(noun => noun.vector.length !== 512);
+  while (hasMore) {
+    const result = await db.getNouns({ 
+      pagination: { offset, limit } 
+    });
+    allNouns = allNouns.concat(result.items);
+    hasMore = result.hasMore;
+    offset += limit;
+  }
+  console.log(`Found ${allNouns.length} nouns.`);
+  
+  // Check dimensions (now 384 for Transformers.js, was 512 for TensorFlow)
+  const dimensionMismatches = allNouns.filter(noun => noun.vector.length !== 384);
   
   if (dimensionMismatches.length > 0) {
     console.warn(`Found ${dimensionMismatches.length} nouns with dimension mismatches.`);