docs: update architecture docs and README for hybrid search

This commit is contained in:
David Snelling 2026-01-26 17:16:49 -08:00
parent 4adba1b254
commit a78f3bb0d2
3 changed files with 267 additions and 0 deletions

View file

@ -25,6 +25,7 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
- **FieldTypeInference** - DuckDB-inspired value-based field type detection
- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count)
- **Sorted Indexes** - Support orderBy queries (automatically maintained)
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes (v7.7.0)
**GraphAdjacencyIndex contains:**
- **lsmTreeSource** - Source → Targets (outgoing edges)
@ -228,6 +229,37 @@ interface ZoneMap {
**Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field
#### Word Index (`__words__`) - v7.7.0
```typescript
// Special field for text/keyword search
// Entity text content is tokenized and indexed as word hashes
// Tokenization:
// "David Smith is a software engineer" → ["david", "smith", "is", "software", "engineer"]
// Word Hashing (FNV-1a):
// "david" → hashWord("david") → 1234567 (int32)
// "smith" → hashWord("smith") → 9876543 (int32)
// Index structure (same as other fields):
// __words__ → 1234567 → RoaringBitmap{entity1, entity5, ...}
// __words__ → 9876543 → RoaringBitmap{entity1, entity3, ...}
```
**Design Decisions**:
- **Max 50 words per entity**: Prevents index bloat for large documents
- **FNV-1a hashing**: Fast, low collision rate, int32 output
- **Min word length 2 chars**: Filters out noise words
- **Lowercase normalization**: Case-insensitive matching
- **Automatic integration**: Words extracted via `extractIndexableFields()`
**Hybrid Search** (v7.7.0): Text results combined with vector results using Reciprocal Rank Fusion (RRF):
```typescript
// RRF formula: score(d) = sum(1 / (k + rank(d)))
// where k = 60 (standard constant)
// alpha = weight for semantic (0 = text only, 1 = semantic only)
```
### Query Algorithm (v3.42.0)
**Exact Match Query**: