Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:
1. validateConsistency() to falsely detect corruption on every startup,
triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
and report inflated totalEntries/totalIds stats
Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
- Add textMatches, textScore, semanticScore, matchSource to search results
- Add highlight() method for zero-config text + semantic highlighting
- Increase word indexing limit to 5000 (handles articles/chapters)
- Optimize findMatchingWords() with O(1) fast path for semantic-only results
- Add production safety limits (500 chunks for highlight)
- Add comprehensive tests for new features (35 tests)
- Update docs with match visibility and highlight() API
- Remove includeVFS parameter and broken isVFS filtering logic
- Add excludeVFS parameter for optional VFS entity filtering
- VFS entities now part of knowledge graph by default
- Enable O(1) graph adjacency optimizations for VFS operations
- Update all VFS projections and PathResolver
- Add comprehensive VFS visibility documentation
This fixes the bug where VFS operations returned empty results due to
operator object mismatch in storage adapters. VFS relationships now use
proper graph traversal without metadata filtering.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
## New Documentation:
### docs/FIND_SYSTEM.md (Complete Find Guide):
- Triple Intelligence architecture (vector + metadata + graph)
- All query types: NLP, structured, proximity, graph traversal
- Detailed index usage: HNSW, HashMap, Sorted arrays, Adjacency maps
- Type-aware NLP processing with dynamic field discovery
- Query execution flow with parallel search and fusion scoring
- Performance characteristics and scalability metrics
- Real-world query examples with execution plans
### docs/PERFORMANCE.md (Updated):
- Added type-aware NLP performance metrics
- Updated metadata index to show incremental sorted indices
- Added type embeddings and field affinity memory usage
- Corrected sorted index behavior (no more lazy loading)
- New performance table with type detection and field matching
## Key Features Documented:
✅ Zero hardcoded fields (only 30+ noun, 40+ verb types)
✅ Dynamic field discovery from real data patterns
✅ Type-field affinity tracking and optimization
✅ Semantic field matching: 'by' → 'author' (87% confidence)
✅ Field-type validation with intelligent suggestions
✅ O(1) graph queries, O(log n) range queries, O(1) exact matches
✅ Sub-millisecond performance at scale with measured benchmarks
This documents the most advanced query system in any vector database.
