feat: add match visibility and semantic highlighting to hybrid search
- Add textMatches, textScore, semanticScore, matchSource to search results - Add highlight() method for zero-config text + semantic highlighting - Increase word indexing limit to 5000 (handles articles/chapters) - Optimize findMatchingWords() with O(1) fast path for semantic-only results - Add production safety limits (500 chunks for highlight) - Add comprehensive tests for new features (35 tests) - Update docs with match visibility and highlight() API
This commit is contained in:
parent
b0ea2f3810
commit
4adba1b254
7 changed files with 1727 additions and 31 deletions
|
|
@ -4,7 +4,7 @@
|
|||
|
||||
Brainy's `find()` method is the most advanced query system in any vector database, combining **Triple Intelligence** (vector + metadata + graph) with **Type-Aware NLP** for natural language understanding.
|
||||
|
||||
## Architecture: Three Intelligence Systems
|
||||
## Architecture: Four Intelligence Systems
|
||||
|
||||
### 1. Vector Intelligence (HNSW Index)
|
||||
- **Purpose**: Semantic similarity search using embeddings
|
||||
|
|
@ -13,14 +13,22 @@ Brainy's `find()` method is the most advanced query system in any vector databas
|
|||
- **Data Structure**: Multi-layer graph with 16 connections per node
|
||||
- **Use Cases**: "Find similar documents", "Content like this"
|
||||
|
||||
### 2. Metadata Intelligence (Incremental Indices)
|
||||
### 2. Text Intelligence (Word Index) - v7.7.0
|
||||
- **Purpose**: Keyword/exact text matching
|
||||
- **Algorithm**: Inverted word index with FNV-1a hashing
|
||||
- **Performance**: O(log C) where C = chunks (~50 values each)
|
||||
- **Data Structure**: `__words__ → hash → Roaring Bitmap of entity IDs`
|
||||
- **Use Cases**: "Find exact name", "Keyword search"
|
||||
- **Integration**: Automatically combined with Vector via RRF fusion
|
||||
|
||||
### 3. Metadata Intelligence (Incremental Indices)
|
||||
- **Purpose**: Fast filtering on structured data
|
||||
- **Algorithm**: HashMap for exact matches, Sorted arrays for ranges
|
||||
- **Performance**: O(1) exact, O(log n) ranges, <1ms typical
|
||||
- **Data Structure**: `Map<field:value, Set<id>>` + sorted value arrays
|
||||
- **Use Cases**: "Documents from 2023", "Status equals active"
|
||||
|
||||
### 3. Graph Intelligence (Adjacency Maps)
|
||||
### 4. Graph Intelligence (Adjacency Maps)
|
||||
- **Purpose**: Relationship traversal and connection analysis
|
||||
- **Algorithm**: Pure O(1) neighbor lookups via Map operations
|
||||
- **Performance**: O(1) per hop, ~0.1ms typical
|
||||
|
|
@ -84,6 +92,84 @@ await brain.find({
|
|||
})
|
||||
```
|
||||
|
||||
### 4. Hybrid Search (v7.7.0+)
|
||||
```typescript
|
||||
// Zero-config hybrid: automatically combines text + semantic search
|
||||
await brain.find({ query: "David Smith" })
|
||||
// Uses Reciprocal Rank Fusion (RRF) to combine results
|
||||
|
||||
// Force text-only search
|
||||
await brain.find({ query: "exact keyword", searchMode: 'text' })
|
||||
|
||||
// Force semantic-only search
|
||||
await brain.find({ query: "AI concepts", searchMode: 'semantic' })
|
||||
|
||||
// Custom hybrid weighting (0 = text only, 1 = semantic only)
|
||||
await brain.find({ query: "search term", hybridAlpha: 0.3 })
|
||||
```
|
||||
|
||||
**How Auto-Alpha Works:**
|
||||
- Short queries (1-2 words): alpha = 0.3 (favor text matching)
|
||||
- Medium queries (3-4 words): alpha = 0.5 (balanced)
|
||||
- Long queries (5+ words): alpha = 0.7 (favor semantic matching)
|
||||
|
||||
### 5. Match Visibility (v7.8.0)
|
||||
|
||||
Search results include match details showing what matched:
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({ query: 'david the warrior' })
|
||||
|
||||
// Each result has:
|
||||
results[0].textMatches // ["david", "warrior"] - exact words found
|
||||
results[0].textScore // 0.25 - text match quality (0-1)
|
||||
results[0].semanticScore // 0.87 - semantic similarity (0-1)
|
||||
results[0].matchSource // 'both' | 'text' | 'semantic'
|
||||
```
|
||||
|
||||
Use this for:
|
||||
- **Highlighting** exact matches in UI (textMatches)
|
||||
- **Explaining** why a result was found (matchSource)
|
||||
- **Debugging** search behavior (separate scores)
|
||||
|
||||
### 6. Semantic Highlighting (v7.8.0)
|
||||
|
||||
Highlight which concepts/words in text matched your query:
|
||||
|
||||
```typescript
|
||||
// Find semantically similar words + exact matches
|
||||
const highlights = await brain.highlight({
|
||||
query: "david the warrior",
|
||||
text: "David Smith is a brave fighter who battles dragons"
|
||||
})
|
||||
|
||||
// Returns:
|
||||
// [
|
||||
// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' },
|
||||
// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' },
|
||||
// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
|
||||
// ]
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- `matchType: 'text'` - Exact word match (score = 1.0)
|
||||
- `matchType: 'semantic'` - Concept match (score varies)
|
||||
- `position` - [start, end] for precise highlighting
|
||||
- `granularity` - 'word' (default), 'phrase', or 'sentence'
|
||||
- `threshold` - Minimum semantic score (default: 0.5)
|
||||
|
||||
**UI Usage Pattern:**
|
||||
```typescript
|
||||
// Highlight search results with different styles
|
||||
function highlightResult(text: string, highlights: Highlight[]) {
|
||||
return highlights.map(h => ({
|
||||
text: h.text,
|
||||
position: h.position,
|
||||
style: h.matchType === 'text' ? 'strong' : 'emphasis' // Different UI styles
|
||||
}))
|
||||
}
|
||||
```
|
||||
|
||||
## Index Usage in Detail
|
||||
|
||||
### Metadata Index Operations
|
||||
|
|
|
|||
|
|
@ -202,11 +202,115 @@ const results = await brain.find({
|
|||
- `depth?`: `number` - Traversal depth
|
||||
- `limit?`: `number` - Max results (default: 10)
|
||||
- `offset?`: `number` - Skip results
|
||||
- `searchMode?`: `'auto' | 'text' | 'semantic' | 'hybrid'` - Search strategy (v7.7.0):
|
||||
- `'auto'` (default): Zero-config hybrid combining text + semantic search
|
||||
- `'text'`: Pure keyword/text matching
|
||||
- `'semantic'`: Pure vector similarity
|
||||
- `'hybrid'`: Explicit hybrid mode
|
||||
- `hybridAlpha?`: `number` - Balance between text (0.0) and semantic (1.0) search. Auto-detected by query length if not specified.
|
||||
|
||||
**Returns:** `Promise<Result[]>` - Matching entities with scores
|
||||
|
||||
---
|
||||
|
||||
### Hybrid Search (v7.7.0)
|
||||
|
||||
Brainy automatically combines text (keyword) and semantic (vector) search for optimal results. No configuration needed.
|
||||
|
||||
```typescript
|
||||
// Zero-config hybrid search (just works)
|
||||
const results = await brain.find({
|
||||
query: 'David Smith' // Finds both exact text matches AND semantically similar
|
||||
})
|
||||
|
||||
// Force text-only search (exact keyword matching)
|
||||
const textResults = await brain.find({
|
||||
query: 'exact keyword',
|
||||
searchMode: 'text'
|
||||
})
|
||||
|
||||
// Force semantic-only search (vector similarity)
|
||||
const semanticResults = await brain.find({
|
||||
query: 'artificial intelligence concepts',
|
||||
searchMode: 'semantic'
|
||||
})
|
||||
|
||||
// Custom hybrid weighting (0 = text only, 1 = semantic only)
|
||||
const customResults = await brain.find({
|
||||
query: 'David Smith',
|
||||
hybridAlpha: 0.3 // Favor text matching
|
||||
})
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
- Short queries (1-2 words) automatically favor text matching
|
||||
- Long queries (5+ words) automatically favor semantic search
|
||||
- Results are combined using Reciprocal Rank Fusion (RRF)
|
||||
|
||||
---
|
||||
|
||||
### Match Visibility (v7.8.0)
|
||||
|
||||
Search results include detailed match information:
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({ query: 'david the warrior' })
|
||||
|
||||
// Each result now includes:
|
||||
results[0].textMatches // ["david", "warrior"] - exact query words found
|
||||
results[0].textScore // 0.25 - text match quality (0-1)
|
||||
results[0].semanticScore // 0.87 - semantic similarity (0-1)
|
||||
results[0].matchSource // 'both' | 'text' | 'semantic'
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
- Highlight exact matches in UI (textMatches)
|
||||
- Explain why a result ranked high (matchSource)
|
||||
- Debug search behavior (separate scores)
|
||||
|
||||
---
|
||||
|
||||
### `highlight(params)` → `Promise<Highlight[]>` ✨ *New v7.8.0*
|
||||
|
||||
Zero-config highlighting for both exact matches AND semantic concepts.
|
||||
|
||||
```typescript
|
||||
const highlights = await brain.highlight({
|
||||
query: "david the warrior",
|
||||
text: "David Smith is a brave fighter who battles dragons"
|
||||
})
|
||||
|
||||
// Returns both text matches AND semantic matches:
|
||||
// [
|
||||
// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' },
|
||||
// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' },
|
||||
// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
|
||||
// ]
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `query`: `string` - The search query
|
||||
- `text`: `string` - Text to highlight (e.g., entity.data)
|
||||
- `granularity?`: `'word' | 'phrase' | 'sentence'` - Highlight unit (default: 'word')
|
||||
- `threshold?`: `number` - Min similarity for semantic matches (default: 0.5)
|
||||
|
||||
**Returns:** `Promise<Highlight[]>`
|
||||
- `text` - The matched text
|
||||
- `score` - Match score (1.0 for text matches, varies for semantic)
|
||||
- `position` - [start, end] indices in original text
|
||||
- `matchType` - `'text'` (exact) or `'semantic'` (concept)
|
||||
|
||||
**UI Pattern:**
|
||||
```typescript
|
||||
// Style differently based on match type
|
||||
highlights.forEach(h => {
|
||||
const style = h.matchType === 'text' ? 'font-weight: bold' : 'background: yellow'
|
||||
// Apply style from h.position[0] to h.position[1]
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Query Operators
|
||||
|
||||
Brainy uses clean, readable operators:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue