feat: v0.49 - Filter discovery API, remove deprecated methods, improve performance

BREAKING CHANGES:
- Removed deprecated getAllNouns() and getAllVerbs() methods
- All internal usage migrated to pagination-based methods

New Features:
- Filter Discovery API:
  - getFilterValues(field): Get all available values for a field
  - getFilterFields(): Get all filterable fields
  - Enables dynamic filter UI generation with O(1) field discovery
- Hybrid metadata indexing with field-level indexes
- Adaptive auto-flush for optimal performance
- LRU caching for metadata indexes

Improvements:
- Fixed ENAMETOOLONG errors from vector-based filenames
- Safe filename generation using hash-based approach
- Scalable chunked value storage for millions of entries
- Performance optimization with adaptive flush thresholds
- Added support for $includes operator in metadata filters

Technical:
- Replaced vector-based filenames with safe hash approach
- Implemented MetadataIndexCache with existing SearchCache pattern
- Field indexes enable O(1) filter discovery
- Adaptive flush based on performance metrics (20-200 entries)
- All tests passing with improved metadata filtering
This commit is contained in:
David Snelling 2025-08-06 14:39:33 -07:00
parent cc1c6fb718
commit 2231f90913
18 changed files with 958 additions and 486 deletions

View file

@ -752,12 +752,6 @@ export class FileSystemStorage extends BaseStorage {
return this.getNode(id)
}
/**
* Get all nouns from storage
*/
protected async getAllNouns_internal(): Promise<HNSWNoun[]> {
return this.getAllNodes()
}
/**
* Get nouns by noun type
@ -789,12 +783,6 @@ export class FileSystemStorage extends BaseStorage {
return this.getEdge(id)
}
/**
* Get all verbs from storage
*/
protected async getAllVerbs_internal(): Promise<HNSWVerb[]> {
return this.getAllEdges()
}
/**
* Get verbs by source