feat: v0.49 - Filter discovery API, remove deprecated methods, improve performance

BREAKING CHANGES:
- Removed deprecated getAllNouns() and getAllVerbs() methods
- All internal usage migrated to pagination-based methods

New Features:
- Filter Discovery API:
  - getFilterValues(field): Get all available values for a field
  - getFilterFields(): Get all filterable fields
  - Enables dynamic filter UI generation with O(1) field discovery
- Hybrid metadata indexing with field-level indexes
- Adaptive auto-flush for optimal performance
- LRU caching for metadata indexes

Improvements:
- Fixed ENAMETOOLONG errors from vector-based filenames
- Safe filename generation using hash-based approach
- Scalable chunked value storage for millions of entries
- Performance optimization with adaptive flush thresholds
- Added support for $includes operator in metadata filters

Technical:
- Replaced vector-based filenames with safe hash approach
- Implemented MetadataIndexCache with existing SearchCache pattern
- Field indexes enable O(1) filter discovery
- Adaptive flush based on performance metrics (20-200 entries)
- All tests passing with improved metadata filtering

docs/api-reference/search-methods.md

@@ -427,6 +427,73 @@ Brainy validates metadata queries and provides helpful error messages:
 
 ---
 
+## Filter Discovery API (v0.49+)
+
+### getFilterValues(field)
+
+Get all available values for a specific field that can be used in filters:
+
+```javascript
+// Get all categories in the database
+const categories = await brainy.getFilterValues('category')
+// Returns: ['electronics', 'books', 'clothing', 'home', ...]
+
+// Get all brands
+const brands = await brainy.getFilterValues('brand')
+// Returns: ['apple', 'samsung', 'sony', ...]
+
+// Use discovered values in filters
+const results = await brainy.search("products", 10, {
+  metadata: {
+    category: { $in: categories.slice(0, 3) },  // First 3 categories
+    brand: brands[0]                            // Specific brand
+  }
+})
+```
+
+### getFilterFields()
+
+Get all fields that have been indexed and can be used for filtering:
+
+```javascript
+// Discover what fields are available
+const fields = await brainy.getFilterFields()
+// Returns: ['category', 'price', 'brand', 'rating', 'tags', ...]
+
+// Build dynamic filters based on available fields
+const filter = {}
+if (fields.includes('category')) {
+  filter.category = 'electronics'
+}
+if (fields.includes('price')) {
+  filter.price = { $lte: 1000 }
+}
+
+const results = await brainy.search("query", 10, { metadata: filter })
+```
+
+### Use Cases
+
+1. **Dynamic UI Generation**: Build filter dropdowns from actual data
+2. **Data Exploration**: Understand what metadata exists
+3. **Validation**: Check if a field exists before filtering
+4. **Analytics**: See distribution of values
+
+```javascript
+// Build a filter UI dynamically
+async function buildFilterUI() {
+  const fields = await brainy.getFilterFields()
+  
+  for (const field of fields) {
+    const values = await brainy.getFilterValues(field)
+    console.log(`${field}: ${values.length} unique values`)
+    
+    // Create dropdown/checkbox for each field
+    createFilterControl(field, values)
+  }
+}
+```
+
 ## Migration Guide
 
 ### From Simple Filtering

docs/guides/metadata-filtering.md

@@ -372,10 +372,33 @@ const results = await brainy.search("laptop", 10, {
    { specs: { display: "4K" } }  // ❌ Won't work as expected
    ```
 
+## 🔍 Filter Discovery API (v0.49+)
+
+Discover what filters are available in your data:
+
+```javascript
+// Get all available values for a field
+const categories = await brainy.getFilterValues('category')
+console.log('Available categories:', categories)
+// Output: ['electronics', 'books', 'clothing', ...]
+
+// Get all filterable fields
+const fields = await brainy.getFilterFields()
+console.log('Filterable fields:', fields)
+// Output: ['category', 'price', 'brand', 'rating', ...]
+
+// Build dynamic filter UI
+for (const field of fields) {
+  const values = await brainy.getFilterValues(field)
+  createDropdown(field, values)
+}
+```
+
 ## 🎉 What's Next?
 
 This powerful filtering system opens up possibilities for:
 - **Advanced search UIs** with multiple filter controls
+- **Dynamic filter discovery** to build UIs from actual data
 - **Personalized recommendations** based on user preferences  
 - **Complex business logic** in search applications
 - **Multi-tenant filtering** by organization or user

docs/guides/production-migration-guide.md

@@ -250,12 +250,24 @@ async function validateDatabase() {
   
   console.log('Validating database...');
   
-  // Get all nouns
-  const nouns = await db.getAllNouns();
-  console.log(`Found ${nouns.length} nouns.`);
+  // Get nouns using pagination (v0.49+)
+  let allNouns = [];
+  let offset = 0;
+  const limit = 100;
+  let hasMore = true;
   
-  // Check dimensions
-  const dimensionMismatches = nouns.filter(noun => noun.vector.length !== 512);
+  while (hasMore) {
+    const result = await db.getNouns({ 
+      pagination: { offset, limit } 
+    });
+    allNouns = allNouns.concat(result.items);
+    hasMore = result.hasMore;
+    offset += limit;
+  }
+  console.log(`Found ${allNouns.length} nouns.`);
+  
+  // Check dimensions (now 384 for Transformers.js, was 512 for TensorFlow)
+  const dimensionMismatches = allNouns.filter(noun => noun.vector.length !== 384);
   
   if (dimensionMismatches.length > 0) {
     console.warn(`Found ${dimensionMismatches.length} nouns with dimension mismatches.`);