feat: v0.49 - Filter discovery API, remove deprecated methods, improve performance

BREAKING CHANGES:
- Removed deprecated getAllNouns() and getAllVerbs() methods
- All internal usage migrated to pagination-based methods

New Features:
- Filter Discovery API:
  - getFilterValues(field): Get all available values for a field
  - getFilterFields(): Get all filterable fields
  - Enables dynamic filter UI generation with O(1) field discovery
- Hybrid metadata indexing with field-level indexes
- Adaptive auto-flush for optimal performance
- LRU caching for metadata indexes

Improvements:
- Fixed ENAMETOOLONG errors from vector-based filenames
- Safe filename generation using hash-based approach
- Scalable chunked value storage for millions of entries
- Performance optimization with adaptive flush thresholds
- Added support for $includes operator in metadata filters

Technical:
- Replaced vector-based filenames with safe hash approach
- Implemented MetadataIndexCache with existing SearchCache pattern
- Field indexes enable O(1) filter discovery
- Adaptive flush based on performance metrics (20-200 entries)
- All tests passing with improved metadata filtering

docs/guides/production-migration-guide.md

@@ -250,12 +250,24 @@ async function validateDatabase() {
   
   console.log('Validating database...');
   
-  // Get all nouns
-  const nouns = await db.getAllNouns();
-  console.log(`Found ${nouns.length} nouns.`);
+  // Get nouns using pagination (v0.49+)
+  let allNouns = [];
+  let offset = 0;
+  const limit = 100;
+  let hasMore = true;
   
-  // Check dimensions
-  const dimensionMismatches = nouns.filter(noun => noun.vector.length !== 512);
+  while (hasMore) {
+    const result = await db.getNouns({ 
+      pagination: { offset, limit } 
+    });
+    allNouns = allNouns.concat(result.items);
+    hasMore = result.hasMore;
+    offset += limit;
+  }
+  console.log(`Found ${allNouns.length} nouns.`);
+  
+  // Check dimensions (now 384 for Transformers.js, was 512 for TensorFlow)
+  const dimensionMismatches = allNouns.filter(noun => noun.vector.length !== 384);
   
   if (dimensionMismatches.length > 0) {
     console.warn(`Found ${dimensionMismatches.length} nouns with dimension mismatches.`);