feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
This commit is contained in:
parent
2b901974ed
commit
cca1cd8ce2
12 changed files with 1341 additions and 37 deletions
74
CHANGELOG.md
74
CHANGELOG.md
|
|
@ -2,6 +2,80 @@
|
|||
|
||||
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
||||
|
||||
## [7.8.0](https://github.com/soulcraftlabs/brainy/compare/v7.7.0...v7.8.0) (2026-01-27)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
**highlight() hangs on structured text input**
|
||||
|
||||
Three root causes fixed:
|
||||
|
||||
1. **embedBatch() now uses native WASM batch API** — Previously called `embed()` individually N times via `Promise.all`, each creating a separate forward pass. Now delegates to `engine.embedBatch()` for a single WASM forward pass. Applies globally to all `embedBatch()` callers.
|
||||
|
||||
2. **Smart content extraction for structured text** — `highlight()` now auto-detects content type (plain text, rich-text JSON, HTML, Markdown) and extracts meaningful text segments instead of splitting raw JSON/HTML into garbage chunks like `{"type":`. Supports TipTap, Slate.js, Lexical, Draft.js, and Quill Delta formats out of the box.
|
||||
|
||||
3. **Timeout protection** — Semantic matching phase now has a 10-second timeout. On timeout or error, `highlight()` returns Phase 1 text-only matches (always fast) instead of hanging indefinitely.
|
||||
|
||||
**extractTextContent() skips arrays of objects** — Changed from length-based skip (`data.length > 10`) to type-based check (`typeof data[0] === 'number'`). Arrays of objects (e.g., team members, items) are now properly indexed for text search instead of being silently skipped.
|
||||
|
||||
### Features
|
||||
|
||||
**Structured Content Highlighting**
|
||||
|
||||
`highlight()` now handles structured text formats automatically:
|
||||
|
||||
```typescript
|
||||
// Rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill)
|
||||
const highlights = await brain.highlight({
|
||||
query: "warrior",
|
||||
text: JSON.stringify(tiptapDocument)
|
||||
})
|
||||
// Each highlight includes contentCategory: 'heading' | 'prose' | 'code' | 'label'
|
||||
|
||||
// HTML
|
||||
await brain.highlight({ query: "warrior", text: "<h1>Warriors</h1><p>Brave fighters.</p>" })
|
||||
|
||||
// Markdown
|
||||
await brain.highlight({ query: "warrior", text: "# Warriors\n\nBrave fighters." })
|
||||
```
|
||||
|
||||
**Content Category Annotations**
|
||||
|
||||
Each `Highlight` now includes `contentCategory` when input is structured:
|
||||
- `'heading'` — from `<h1>`-`<h6>`, `# Heading`, or heading nodes
|
||||
- `'code'` — from `<code>`/`<pre>`, fenced/indented code blocks, or code nodes
|
||||
- `'prose'` — regular paragraph text
|
||||
- `'label'` — labels, captions, metadata-like text
|
||||
|
||||
**Custom Content Extractors**
|
||||
|
||||
New `contentExtractor` parameter lets developers plug in custom parsers:
|
||||
|
||||
```typescript
|
||||
const highlights = await brain.highlight({
|
||||
query: "function",
|
||||
text: sourceCode,
|
||||
contentExtractor: (text) => treeSitterParse(text) // Custom parser
|
||||
})
|
||||
```
|
||||
|
||||
**Content Type Hints**
|
||||
|
||||
New `contentType` parameter to skip auto-detection:
|
||||
|
||||
```typescript
|
||||
await brain.highlight({ query: "test", text: input, contentType: 'html' })
|
||||
```
|
||||
|
||||
### New Types
|
||||
|
||||
- `ContentType`: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`
|
||||
- `ContentCategory`: `'prose' | 'heading' | 'code' | 'label'`
|
||||
- `ExtractedSegment`: `{ text: string, contentCategory: ContentCategory }`
|
||||
- `HighlightParams.contentType?` — optional content type hint
|
||||
- `HighlightParams.contentExtractor?` — optional custom parser callback
|
||||
- `Highlight.contentCategory?` — content role annotation
|
||||
|
||||
## [7.7.0](https://github.com/soulcraftlabs/brainy/compare/v7.6.1...v7.7.0) (2026-01-26)
|
||||
|
||||
### Features
|
||||
|
|
|
|||
|
|
@ -412,6 +412,13 @@ const results = await brain.find({ query: 'David Smith' })
|
|||
await brain.find({ query: 'exact match', searchMode: 'text' }) // Text only
|
||||
await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // Semantic only
|
||||
await brain.find({ query: 'hybrid', hybridAlpha: 0.3 }) // Custom weighting
|
||||
|
||||
// Highlight structured content (v7.8.0) — plain text, JSON, HTML, Markdown
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior',
|
||||
text: entity.data // Auto-detects TipTap, Slate, Lexical, HTML, Markdown
|
||||
})
|
||||
// Each highlight has: text, score, position, matchType, contentCategory
|
||||
```
|
||||
|
||||
**→ [Hybrid Search Documentation](docs/api/README.md#hybrid-search-v770)**
|
||||
|
|
|
|||
|
|
@ -507,14 +507,16 @@ console.log(vector.length) // 384
|
|||
|
||||
---
|
||||
|
||||
### `async embedBatch(texts: string[]): Promise<number[][]>` ✨ *v7.1.0*
|
||||
Batch embed multiple texts efficiently.
|
||||
### `async embedBatch(texts: string[]): Promise<number[][]>` ✨ *v7.1.0, Optimized v7.9.0*
|
||||
Batch embed multiple texts using native WASM batch API (single forward pass).
|
||||
|
||||
**Parameters:**
|
||||
- `texts` - Array of strings to embed
|
||||
|
||||
**Returns:** Array of 384-dimensional vectors
|
||||
|
||||
> **v7.9.0**: Uses the engine's native `embed_batch()` for a single model forward pass instead of N individual `embed()` calls.
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
const embeddings = await brain.embedBatch([
|
||||
|
|
|
|||
|
|
@ -270,41 +270,85 @@ results[0].matchSource // 'both' | 'text' | 'semantic'
|
|||
|
||||
---
|
||||
|
||||
### `highlight(params)` → `Promise<Highlight[]>` ✨ *New v7.8.0*
|
||||
### `highlight(params)` → `Promise<Highlight[]>` ✨ *New v7.8.0, Enhanced v7.9.0*
|
||||
|
||||
Zero-config highlighting for both exact matches AND semantic concepts.
|
||||
Handles plain text, rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill), HTML, and Markdown automatically.
|
||||
|
||||
```typescript
|
||||
// Plain text (works as before)
|
||||
const highlights = await brain.highlight({
|
||||
query: "david the warrior",
|
||||
text: "David Smith is a brave fighter who battles dragons"
|
||||
})
|
||||
|
||||
// Returns both text matches AND semantic matches:
|
||||
// [
|
||||
// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' },
|
||||
// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' },
|
||||
// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
|
||||
// ]
|
||||
|
||||
// Rich-text JSON (auto-detected, v7.9.0)
|
||||
const highlights = await brain.highlight({
|
||||
query: "david the warrior",
|
||||
text: JSON.stringify(tiptapDocument) // TipTap, Slate, Lexical, Draft.js, Quill
|
||||
})
|
||||
// Extracts text from nodes, annotates with contentCategory:
|
||||
// [
|
||||
// { text: "David", score: 1.0, matchType: 'text', contentCategory: 'heading' },
|
||||
// { text: "fighter", score: 0.78, matchType: 'semantic', contentCategory: 'prose' }
|
||||
// ]
|
||||
|
||||
// HTML input (auto-detected, v7.9.0)
|
||||
const highlights = await brain.highlight({
|
||||
query: "warrior",
|
||||
text: "<h1>David the Warrior</h1><p>A brave fighter.</p>"
|
||||
})
|
||||
|
||||
// Custom extractor for proprietary formats (v7.9.0)
|
||||
const highlights = await brain.highlight({
|
||||
query: "function",
|
||||
text: sourceCode,
|
||||
contentExtractor: (text) => treeSitterParse(text) // Your custom parser
|
||||
})
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `query`: `string` - The search query
|
||||
- `text`: `string` - Text to highlight (e.g., entity.data)
|
||||
- `text`: `string` - Text to highlight (plain text, JSON, HTML, or Markdown)
|
||||
- `granularity?`: `'word' | 'phrase' | 'sentence'` - Highlight unit (default: 'word')
|
||||
- `threshold?`: `number` - Min similarity for semantic matches (default: 0.5)
|
||||
- `contentType?`: `ContentType` - Optional hint: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`. Skips auto-detection when provided. *(v7.9.0)*
|
||||
- `contentExtractor?`: `(text: string) => ExtractedSegment[]` - Custom parser. Bypasses built-in detection entirely. *(v7.9.0)*
|
||||
|
||||
**Returns:** `Promise<Highlight[]>`
|
||||
- `text` - The matched text
|
||||
- `score` - Match score (1.0 for text matches, varies for semantic)
|
||||
- `position` - [start, end] indices in original text
|
||||
- `position` - [start, end] indices in extracted text
|
||||
- `matchType` - `'text'` (exact) or `'semantic'` (concept)
|
||||
- `contentCategory?` - `'prose' | 'heading' | 'code' | 'label'` — Role of the source text in the document. Present when input is structured. *(v7.9.0)*
|
||||
|
||||
**Supported Rich-Text Formats (v7.9.0):**
|
||||
|
||||
| Format | Detection | Text nodes |
|
||||
|--------|-----------|------------|
|
||||
| TipTap / ProseMirror | `{ type: 'doc', content: [...] }` | `{ type: 'text', text }` |
|
||||
| Slate.js | `[{ type, children }]` | `{ text }` |
|
||||
| Lexical | `{ root: { children } }` | `{ type: 'text', text }` |
|
||||
| Draft.js | `{ blocks: [{ text }] }` | `{ text }` in block |
|
||||
| Quill Delta | `{ ops: [{ insert }] }` | `{ insert }` |
|
||||
| HTML | Tags like `<h1>`, `<p>`, `<code>` | Visible text content |
|
||||
| Markdown | `#` headings, ` ``` ` code blocks | Stripped markup |
|
||||
|
||||
**Timeout Protection (v7.9.0):**
|
||||
Semantic matching has a 10-second timeout. If embedding takes too long (e.g., WASM stall), `highlight()` returns text-only matches instead of hanging.
|
||||
|
||||
**UI Pattern:**
|
||||
```typescript
|
||||
// Style differently based on match type
|
||||
// Style differently based on match type and content category
|
||||
highlights.forEach(h => {
|
||||
const style = h.matchType === 'text' ? 'font-weight: bold' : 'background: yellow'
|
||||
if (h.contentCategory === 'heading') { /* render as heading highlight */ }
|
||||
if (h.contentCategory === 'code') { /* render with code styling */ }
|
||||
// Apply style from h.position[0] to h.position[1]
|
||||
})
|
||||
```
|
||||
|
|
@ -1677,9 +1721,9 @@ console.log(vector.length) // 384
|
|||
|
||||
---
|
||||
|
||||
### `embedBatch(texts)` → `Promise<number[][]>` ✨ *New v7.1.0*
|
||||
### `embedBatch(texts)` → `Promise<number[][]>` ✨ *New v7.1.0, Optimized v7.9.0*
|
||||
|
||||
Batch embed multiple texts efficiently.
|
||||
Batch embed multiple texts using native WASM batch API (single forward pass).
|
||||
|
||||
```typescript
|
||||
const embeddings = await brain.embedBatch([
|
||||
|
|
@ -1691,6 +1735,8 @@ console.log(embeddings.length) // 3
|
|||
console.log(embeddings[0].length) // 384
|
||||
```
|
||||
|
||||
> **v7.9.0**: Now uses the WASM engine's native `embed_batch()` for a single model forward pass instead of N individual calls. This is the same batch API used internally by `highlight()`.
|
||||
|
||||
---
|
||||
|
||||
### `similarity(textA, textB)` → `Promise<number>` ✨ *New v7.1.0*
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ import { TripleIntelligenceSystem } from './triple/TripleIntelligenceSystem.js'
|
|||
import { VirtualFileSystem } from './vfs/VirtualFileSystem.js'
|
||||
import { VersioningAPI } from './versioning/VersioningAPI.js'
|
||||
import { MetadataIndexManager } from './utils/metadataIndex.js'
|
||||
import { detectContentType, extractForHighlighting } from './utils/contentExtractor.js'
|
||||
import { GraphAdjacencyIndex } from './graph/graphAdjacencyIndex.js'
|
||||
import { CommitBuilder } from './storage/cow/CommitObject.js'
|
||||
import { BlobStorage } from './storage/cow/BlobStorage.js'
|
||||
|
|
@ -4663,13 +4664,8 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
return []
|
||||
}
|
||||
|
||||
// Use the underlying embedder for each text
|
||||
// The WASM engine handles batching internally
|
||||
const embeddings = await Promise.all(
|
||||
texts.map(text => this.embedder(text))
|
||||
)
|
||||
|
||||
return embeddings
|
||||
// Use native WASM batch API: single forward pass instead of N individual calls
|
||||
return await embeddingManager.embedBatch(texts)
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -4734,14 +4730,38 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
async highlight(params: import('./types/brainy.types.js').HighlightParams): Promise<import('./types/brainy.types.js').Highlight[]> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
const { query, text, granularity = 'word', threshold = 0.5 } = params
|
||||
const { query, text, granularity = 'word', threshold = 0.5, contentType, contentExtractor } = params
|
||||
|
||||
if (!query || !text) {
|
||||
return []
|
||||
}
|
||||
|
||||
// Split text into chunks based on granularity
|
||||
const allChunks = this.splitForHighlighting(text, granularity)
|
||||
// v7.9.0: Extract text from structured content (JSON, HTML, Markdown)
|
||||
// Custom extractor takes priority, then built-in detection
|
||||
type ChunkWithCategory = { text: string, position: [number, number], contentCategory?: import('./types/brainy.types.js').ContentCategory }
|
||||
|
||||
let segments: import('./types/brainy.types.js').ExtractedSegment[]
|
||||
if (contentExtractor) {
|
||||
segments = contentExtractor(text)
|
||||
} else {
|
||||
segments = extractForHighlighting(text, contentType)
|
||||
}
|
||||
|
||||
// Build concatenated text from segments for position tracking
|
||||
// and split each segment into chunks based on granularity
|
||||
const allChunks: ChunkWithCategory[] = []
|
||||
let offset = 0
|
||||
for (const segment of segments) {
|
||||
const segmentChunks = this.splitForHighlighting(segment.text, granularity)
|
||||
for (const chunk of segmentChunks) {
|
||||
allChunks.push({
|
||||
text: chunk.text,
|
||||
position: [chunk.position[0] + offset, chunk.position[1] + offset],
|
||||
contentCategory: segment.contentCategory
|
||||
})
|
||||
}
|
||||
offset += segment.text.length + 1 // +1 for space between segments
|
||||
}
|
||||
|
||||
if (allChunks.length === 0) {
|
||||
return []
|
||||
|
|
@ -4749,7 +4769,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
|
||||
// v7.8.0 Production safety: Limit chunks to prevent memory explosion
|
||||
// At 500 words × 384 dimensions × 4 bytes = 768KB temp memory (acceptable)
|
||||
// At 10,000 words = 15MB temp memory (too much for concurrent requests)
|
||||
const MAX_HIGHLIGHT_CHUNKS = 500
|
||||
const chunks = allChunks.slice(0, MAX_HIGHLIGHT_CHUNKS)
|
||||
|
||||
|
|
@ -4768,16 +4787,50 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
text: chunk.text,
|
||||
score: 1.0,
|
||||
position: chunk.position,
|
||||
matchType: 'text'
|
||||
matchType: 'text',
|
||||
contentCategory: chunk.contentCategory
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// === PHASE 2: Find semantic matches (score varies, matchType = 'semantic') ===
|
||||
// === PHASE 2: Find semantic matches with timeout fallback ===
|
||||
const SEMANTIC_TIMEOUT_MS = 10_000
|
||||
try {
|
||||
const semanticResult = await Promise.race([
|
||||
this.highlightSemanticPhase(query, chunks, threshold, highlightMap),
|
||||
new Promise<'timeout'>((resolve) => setTimeout(() => resolve('timeout'), SEMANTIC_TIMEOUT_MS))
|
||||
])
|
||||
|
||||
if (semanticResult === 'timeout') {
|
||||
// Return Phase 1 text-only results on timeout
|
||||
const textHighlights = Array.from(highlightMap.values())
|
||||
return textHighlights.sort((a, b) => b.score - a.score)
|
||||
}
|
||||
} catch {
|
||||
// On any error in semantic phase, return Phase 1 results
|
||||
const textHighlights = Array.from(highlightMap.values())
|
||||
return textHighlights.sort((a, b) => b.score - a.score)
|
||||
}
|
||||
|
||||
// Sort by score descending (text matches will be first with score=1.0)
|
||||
const highlights = Array.from(highlightMap.values())
|
||||
return highlights.sort((a, b) => b.score - a.score)
|
||||
}
|
||||
|
||||
/**
|
||||
* Phase 2 of highlight(): semantic matching with batch embedding
|
||||
* @internal
|
||||
*/
|
||||
private async highlightSemanticPhase(
|
||||
query: string,
|
||||
chunks: Array<{ text: string, position: [number, number], contentCategory?: import('./types/brainy.types.js').ContentCategory }>,
|
||||
threshold: number,
|
||||
highlightMap: Map<string, import('./types/brainy.types.js').Highlight>
|
||||
): Promise<void> {
|
||||
// Get query embedding
|
||||
const queryVector = await this.embed(query)
|
||||
|
||||
// Batch embed all chunks (efficient!)
|
||||
// Batch embed all chunks using native WASM batch API
|
||||
const chunkTexts = chunks.map(c => c.text)
|
||||
const chunkVectors = await this.embedBatch(chunkTexts)
|
||||
|
||||
|
|
@ -4796,14 +4849,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
text: chunks[i].text,
|
||||
score: similarity,
|
||||
position: chunks[i].position,
|
||||
matchType: 'semantic'
|
||||
matchType: 'semantic',
|
||||
contentCategory: chunks[i].contentCategory
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Sort by score descending (text matches will be first with score=1.0)
|
||||
const highlights = Array.from(highlightMap.values())
|
||||
return highlights.sort((a, b) => b.score - a.score)
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -206,6 +206,35 @@ export class EmbeddingManager {
|
|||
return vector
|
||||
}
|
||||
|
||||
/**
|
||||
* Batch embed multiple texts using native WASM batch API
|
||||
*
|
||||
* Uses the engine's embedBatch() for a single WASM forward pass
|
||||
* instead of N individual embed() calls.
|
||||
*
|
||||
* @param texts Array of strings to embed
|
||||
* @returns Array of embedding vectors (384 dimensions each)
|
||||
*/
|
||||
async embedBatch(texts: string[]): Promise<number[][]> {
|
||||
if (texts.length === 0) return []
|
||||
|
||||
const isTestMode =
|
||||
process.env.BRAINY_UNIT_TEST === 'true' ||
|
||||
(globalThis as any).__BRAINY_UNIT_TEST__
|
||||
|
||||
if (isTestMode) {
|
||||
if (process.env.NODE_ENV === 'production') {
|
||||
throw new Error('CRITICAL: Mock embeddings in production!')
|
||||
}
|
||||
return texts.map(t => this.getMockEmbedding(t))
|
||||
}
|
||||
|
||||
await this.init()
|
||||
const results = await this.engine.embedBatch(texts)
|
||||
this.embedCount += texts.length
|
||||
return results
|
||||
}
|
||||
|
||||
/**
|
||||
* Get embedding function for compatibility
|
||||
*/
|
||||
|
|
|
|||
|
|
@ -789,25 +789,71 @@ export interface NeuralAnomalyParams {
|
|||
returnScores?: boolean // Return anomaly scores
|
||||
}
|
||||
|
||||
// ============= Content Extraction Types (v7.9.0) =============
|
||||
|
||||
/**
|
||||
* Detected content type for smart text extraction
|
||||
*
|
||||
* @since v7.9.0
|
||||
*/
|
||||
export type ContentType = 'plaintext' | 'richtext-json' | 'html' | 'markdown'
|
||||
|
||||
/**
|
||||
* Content category for extracted segments
|
||||
*
|
||||
* Allows UI to apply different styling based on content role:
|
||||
* - 'prose': Regular text content (paragraphs, list items)
|
||||
* - 'heading': Heading/title text (h1-h6)
|
||||
* - 'code': Code blocks or inline code
|
||||
* - 'label': Labels, captions, or metadata-like text
|
||||
*
|
||||
* @since v7.9.0
|
||||
*/
|
||||
export type ContentCategory = 'prose' | 'heading' | 'code' | 'label'
|
||||
|
||||
/**
|
||||
* A segment of extracted text with its content category
|
||||
*
|
||||
* @since v7.9.0
|
||||
*/
|
||||
export interface ExtractedSegment {
|
||||
/** The extracted text content */
|
||||
text: string
|
||||
/** What role this text plays in the document */
|
||||
contentCategory: ContentCategory
|
||||
}
|
||||
|
||||
// ============= Semantic Highlighting Types (v7.8.0) =============
|
||||
|
||||
/**
|
||||
* Parameters for hybrid highlighting (v7.8.0)
|
||||
* Parameters for hybrid highlighting (v7.8.0, enhanced v7.9.0)
|
||||
*
|
||||
* Zero-config highlighting that returns both text (exact) and semantic (concept) matches.
|
||||
* Perfect for UI highlighting at different levels.
|
||||
*
|
||||
* v7.9.0: Added contentType hint and contentExtractor callback for structured text
|
||||
* (rich-text JSON, HTML, Markdown). Auto-detects format when not specified.
|
||||
*
|
||||
* @example
|
||||
* ```typescript
|
||||
* // Plain text (unchanged from v7.8.0)
|
||||
* const highlights = await brain.highlight({
|
||||
* query: "david the warrior",
|
||||
* text: "David Smith is a brave fighter who battles dragons"
|
||||
* })
|
||||
* // Returns: [
|
||||
* // { text: "David", score: 1.0, position: [0, 5], matchType: 'text' }, // Exact match
|
||||
* // { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' }, // Concept match
|
||||
* // { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' } // Concept match
|
||||
* // ]
|
||||
*
|
||||
* // Rich-text JSON (auto-detected)
|
||||
* const highlights = await brain.highlight({
|
||||
* query: "david the warrior",
|
||||
* text: JSON.stringify(tiptapDocument)
|
||||
* })
|
||||
*
|
||||
* // Custom extractor (for proprietary formats)
|
||||
* const highlights = await brain.highlight({
|
||||
* query: "function",
|
||||
* text: sourceCode,
|
||||
* contentExtractor: (text) => treeSitterParse(text)
|
||||
* })
|
||||
* ```
|
||||
*/
|
||||
export interface HighlightParams {
|
||||
|
|
@ -822,6 +868,21 @@ export interface HighlightParams {
|
|||
|
||||
/** Minimum semantic similarity score for semantic matches (default: 0.5) */
|
||||
threshold?: number
|
||||
|
||||
/**
|
||||
* Optional content type hint to skip auto-detection.
|
||||
* When omitted, the content type is detected from the text content.
|
||||
* @since v7.9.0
|
||||
*/
|
||||
contentType?: ContentType
|
||||
|
||||
/**
|
||||
* Optional custom content extractor function.
|
||||
* When provided, bypasses built-in detection and extraction entirely.
|
||||
* Use this to plug in custom parsers (tree-sitter, Monaco, proprietary formats).
|
||||
* @since v7.9.0
|
||||
*/
|
||||
contentExtractor?: (text: string) => ExtractedSegment[]
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -843,6 +904,13 @@ export interface Highlight {
|
|||
|
||||
/** Match type: 'text' (exact word match) or 'semantic' (concept match) */
|
||||
matchType: 'text' | 'semantic'
|
||||
|
||||
/**
|
||||
* Content category of the source segment (heading, code, prose, label).
|
||||
* Present when the input text was structured (JSON, HTML, Markdown).
|
||||
* @since v7.9.0
|
||||
*/
|
||||
contentCategory?: ContentCategory
|
||||
}
|
||||
|
||||
// ============= Export all types =============
|
||||
|
|
|
|||
403
src/utils/contentExtractor.ts
Normal file
403
src/utils/contentExtractor.ts
Normal file
|
|
@ -0,0 +1,403 @@
|
|||
/**
|
||||
* Content Extractor (v7.9.0)
|
||||
*
|
||||
* Detects content type (plaintext, rich-text JSON, HTML, Markdown) and extracts
|
||||
* text segments with content categories (prose, heading, code, label).
|
||||
*
|
||||
* Supports common rich-text editor formats:
|
||||
* - TipTap / ProseMirror: { type: 'doc', content: [...] }
|
||||
* - Slate.js: [{ type, children: [{ text }] }]
|
||||
* - Lexical: { root: { children: [...] } }
|
||||
* - Draft.js: { blocks: [{ text }] }
|
||||
* - Quill Delta: { ops: [{ insert }] }
|
||||
*
|
||||
* Falls back gracefully: structured text that doesn't match known patterns
|
||||
* is extracted as plain prose via recursive text collection.
|
||||
*/
|
||||
|
||||
import type { ContentType, ContentCategory, ExtractedSegment } from '../types/brainy.types.js'
|
||||
|
||||
/**
|
||||
* Detect content type from text content (no filename needed)
|
||||
*/
|
||||
export function detectContentType(text: string): ContentType {
|
||||
const trimmed = text.trimStart()
|
||||
|
||||
// JSON detection (covers TipTap, Slate, Lexical, Draft.js, Quill, etc.)
|
||||
if (trimmed.startsWith('{') || trimmed.startsWith('[')) {
|
||||
try {
|
||||
JSON.parse(trimmed)
|
||||
return 'richtext-json'
|
||||
} catch {
|
||||
// Not valid JSON, fall through
|
||||
}
|
||||
}
|
||||
|
||||
// HTML detection (tag at start of content)
|
||||
if (/<[a-z!][a-z0-9]*[\s>/]/i.test(trimmed.substring(0, 200))) {
|
||||
return 'html'
|
||||
}
|
||||
|
||||
// Markdown detection (heading or code fence at start)
|
||||
if (/^#{1,6}\s/m.test(trimmed.substring(0, 500)) ||
|
||||
/^```/m.test(trimmed.substring(0, 500))) {
|
||||
return 'markdown'
|
||||
}
|
||||
|
||||
return 'plaintext'
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract text segments from content, auto-detecting type if not provided
|
||||
*/
|
||||
export function extractForHighlighting(
|
||||
text: string,
|
||||
contentType?: ContentType
|
||||
): ExtractedSegment[] {
|
||||
const type = contentType || detectContentType(text)
|
||||
|
||||
switch (type) {
|
||||
case 'richtext-json':
|
||||
return extractFromJson(text)
|
||||
case 'html':
|
||||
return extractFromHtml(text)
|
||||
case 'markdown':
|
||||
return extractFromMarkdown(text)
|
||||
case 'plaintext':
|
||||
default:
|
||||
return [{ text, contentCategory: 'prose' }]
|
||||
}
|
||||
}
|
||||
|
||||
// ============= Rich-Text JSON Extraction =============
|
||||
|
||||
/**
|
||||
* Extract text from any JSON rich-text format.
|
||||
* Walks the node tree looking for text leaves across all common editors.
|
||||
*/
|
||||
function extractFromJson(text: string): ExtractedSegment[] {
|
||||
let parsed: any
|
||||
try {
|
||||
parsed = JSON.parse(text.trim())
|
||||
} catch {
|
||||
// Invalid JSON — treat as plain text
|
||||
return [{ text, contentCategory: 'prose' }]
|
||||
}
|
||||
|
||||
const segments = walkRichTextNodes(parsed)
|
||||
|
||||
// If the walker found real text segments, return them
|
||||
if (segments.length > 0 && segments.some(s => s.text.trim().length > 0)) {
|
||||
return segments.filter(s => s.text.trim().length > 0)
|
||||
}
|
||||
|
||||
// Fallback: collect all string values from the JSON
|
||||
const fallbackText = extractTextFromJsonValue(parsed)
|
||||
if (fallbackText.trim()) {
|
||||
return [{ text: fallbackText, contentCategory: 'prose' }]
|
||||
}
|
||||
|
||||
return []
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk rich-text nodes recursively.
|
||||
* Handles TipTap/ProseMirror, Slate, Lexical, Draft.js, and Quill Delta.
|
||||
*/
|
||||
function walkRichTextNodes(node: any): ExtractedSegment[] {
|
||||
if (node === null || node === undefined) return []
|
||||
|
||||
const segments: ExtractedSegment[] = []
|
||||
|
||||
// Handle arrays (Slate root is an array, Draft.js blocks, Quill ops)
|
||||
if (Array.isArray(node)) {
|
||||
for (const child of node) {
|
||||
segments.push(...walkRichTextNodes(child))
|
||||
}
|
||||
return segments
|
||||
}
|
||||
|
||||
if (typeof node !== 'object') return []
|
||||
|
||||
// Leaf: text content (TipTap/ProseMirror/Lexical/Slate)
|
||||
if (typeof node.text === 'string' && node.text.length > 0) {
|
||||
segments.push({ text: node.text, contentCategory: 'prose' })
|
||||
return segments
|
||||
}
|
||||
|
||||
// Leaf: Quill Delta insert
|
||||
if (typeof node.insert === 'string' && node.insert.length > 0) {
|
||||
segments.push({ text: node.insert, contentCategory: 'prose' })
|
||||
return segments
|
||||
}
|
||||
|
||||
// Draft.js block with text
|
||||
if (typeof node.text === 'string' && node.type !== undefined && node.text.length > 0) {
|
||||
const category = categorizeNodeType(node.type)
|
||||
segments.push({ text: node.text, contentCategory: category })
|
||||
return segments
|
||||
}
|
||||
|
||||
// Categorize by node type
|
||||
const category = categorizeNodeType(node.type)
|
||||
|
||||
// Walk children arrays: content (TipTap), children (Slate/Lexical), blocks (Draft.js), ops (Quill)
|
||||
const children = node.content || node.children || node.blocks || node.ops
|
||||
if (Array.isArray(children)) {
|
||||
for (const child of children) {
|
||||
const childSegments = walkRichTextNodes(child)
|
||||
// Apply parent's category if it's more specific than 'prose'
|
||||
if (category !== 'prose') {
|
||||
childSegments.forEach(s => { s.contentCategory = category })
|
||||
}
|
||||
segments.push(...childSegments)
|
||||
}
|
||||
}
|
||||
|
||||
// Lexical root wrapper
|
||||
if (node.root && typeof node.root === 'object') {
|
||||
segments.push(...walkRichTextNodes(node.root))
|
||||
}
|
||||
|
||||
return segments
|
||||
}
|
||||
|
||||
/**
|
||||
* Categorize a node type string into a ContentCategory
|
||||
*/
|
||||
function categorizeNodeType(type?: string): ContentCategory {
|
||||
if (!type) return 'prose'
|
||||
const t = type.toLowerCase()
|
||||
if (t === 'heading' || /^h[1-6]$/.test(t)) return 'heading'
|
||||
if (t === 'code' || t === 'codeblock' || t === 'code_block') return 'code'
|
||||
return 'prose'
|
||||
}
|
||||
|
||||
/**
|
||||
* Fallback: recursively extract all string values from JSON
|
||||
*/
|
||||
function extractTextFromJsonValue(value: any): string {
|
||||
if (typeof value === 'string') return value
|
||||
if (Array.isArray(value)) {
|
||||
return value.map(extractTextFromJsonValue).filter(Boolean).join(' ')
|
||||
}
|
||||
if (typeof value === 'object' && value !== null) {
|
||||
const texts: string[] = []
|
||||
for (const v of Object.values(value)) {
|
||||
const text = extractTextFromJsonValue(v)
|
||||
if (text) texts.push(text)
|
||||
}
|
||||
return texts.join(' ')
|
||||
}
|
||||
return ''
|
||||
}
|
||||
|
||||
// ============= HTML Extraction =============
|
||||
|
||||
/**
|
||||
* Extract text from HTML using a simple state-machine tag parser.
|
||||
* No external dependencies.
|
||||
*/
|
||||
function extractFromHtml(html: string): ExtractedSegment[] {
|
||||
const segments: ExtractedSegment[] = []
|
||||
let current = ''
|
||||
let currentCategory: ContentCategory = 'prose'
|
||||
let i = 0
|
||||
let insideTag = false
|
||||
let tagName = ''
|
||||
let isClosingTag = false
|
||||
let skipContent = false
|
||||
|
||||
// Tag stack to track nesting
|
||||
const tagStack: string[] = []
|
||||
|
||||
while (i < html.length) {
|
||||
if (html[i] === '<') {
|
||||
// Flush current text if we have any
|
||||
if (current.trim() && !skipContent) {
|
||||
segments.push({ text: current.trim(), contentCategory: currentCategory })
|
||||
}
|
||||
current = ''
|
||||
|
||||
insideTag = true
|
||||
tagName = ''
|
||||
isClosingTag = false
|
||||
i++
|
||||
|
||||
// Check for closing tag
|
||||
if (i < html.length && html[i] === '/') {
|
||||
isClosingTag = true
|
||||
i++
|
||||
}
|
||||
|
||||
// Read tag name
|
||||
while (i < html.length && html[i] !== '>' && html[i] !== ' ' && html[i] !== '/') {
|
||||
tagName += html[i]
|
||||
i++
|
||||
}
|
||||
|
||||
// Skip to end of tag
|
||||
while (i < html.length && html[i] !== '>') {
|
||||
i++
|
||||
}
|
||||
if (i < html.length) i++ // skip '>'
|
||||
|
||||
const tagLower = tagName.toLowerCase()
|
||||
|
||||
if (isClosingTag) {
|
||||
// Pop tag stack
|
||||
if (tagStack.length > 0 && tagStack[tagStack.length - 1] === tagLower) {
|
||||
tagStack.pop()
|
||||
}
|
||||
if (tagLower === 'script' || tagLower === 'style') {
|
||||
skipContent = false
|
||||
}
|
||||
// Reset category based on remaining stack
|
||||
currentCategory = getCategoryFromTagStack(tagStack)
|
||||
} else {
|
||||
// Opening tag
|
||||
if (tagLower === 'script' || tagLower === 'style') {
|
||||
skipContent = true
|
||||
}
|
||||
|
||||
// Self-closing tags don't affect stack
|
||||
const selfClosing = html[i - 2] === '/' ||
|
||||
['br', 'hr', 'img', 'input', 'meta', 'link'].includes(tagLower)
|
||||
|
||||
if (!selfClosing) {
|
||||
tagStack.push(tagLower)
|
||||
currentCategory = getCategoryFromTagStack(tagStack)
|
||||
}
|
||||
}
|
||||
|
||||
insideTag = false
|
||||
continue
|
||||
}
|
||||
|
||||
if (!insideTag) {
|
||||
// Decode common HTML entities
|
||||
if (html[i] === '&') {
|
||||
const entityEnd = html.indexOf(';', i)
|
||||
if (entityEnd !== -1 && entityEnd - i < 10) {
|
||||
const entity = html.substring(i, entityEnd + 1)
|
||||
current += decodeHtmlEntity(entity)
|
||||
i = entityEnd + 1
|
||||
continue
|
||||
}
|
||||
}
|
||||
current += html[i]
|
||||
}
|
||||
i++
|
||||
}
|
||||
|
||||
// Flush remaining text
|
||||
if (current.trim() && !skipContent) {
|
||||
segments.push({ text: current.trim(), contentCategory: currentCategory })
|
||||
}
|
||||
|
||||
return segments.filter(s => s.text.length > 0)
|
||||
}
|
||||
|
||||
/**
|
||||
* Determine content category from the current tag stack
|
||||
*/
|
||||
function getCategoryFromTagStack(stack: string[]): ContentCategory {
|
||||
for (let i = stack.length - 1; i >= 0; i--) {
|
||||
const tag = stack[i]
|
||||
if (/^h[1-6]$/.test(tag)) return 'heading'
|
||||
if (tag === 'code' || tag === 'pre') return 'code'
|
||||
}
|
||||
return 'prose'
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode common HTML entities
|
||||
*/
|
||||
function decodeHtmlEntity(entity: string): string {
|
||||
const entities: Record<string, string> = {
|
||||
'&': '&',
|
||||
'<': '<',
|
||||
'>': '>',
|
||||
'"': '"',
|
||||
''': "'",
|
||||
''': "'",
|
||||
' ': ' ',
|
||||
}
|
||||
return entities[entity] || entity
|
||||
}
|
||||
|
||||
// ============= Markdown Extraction =============
|
||||
|
||||
/**
|
||||
* Extract text from Markdown with category detection
|
||||
*/
|
||||
function extractFromMarkdown(text: string): ExtractedSegment[] {
|
||||
const segments: ExtractedSegment[] = []
|
||||
const lines = text.split('\n')
|
||||
let inCodeBlock = false
|
||||
let codeContent = ''
|
||||
let i = 0
|
||||
|
||||
while (i < lines.length) {
|
||||
const line = lines[i]
|
||||
|
||||
// Fenced code block toggle
|
||||
if (/^```/.test(line.trimStart())) {
|
||||
if (inCodeBlock) {
|
||||
// End of code block
|
||||
if (codeContent.trim()) {
|
||||
segments.push({ text: codeContent.trim(), contentCategory: 'code' })
|
||||
}
|
||||
codeContent = ''
|
||||
inCodeBlock = false
|
||||
} else {
|
||||
// Start of code block
|
||||
inCodeBlock = true
|
||||
}
|
||||
i++
|
||||
continue
|
||||
}
|
||||
|
||||
if (inCodeBlock) {
|
||||
codeContent += (codeContent ? '\n' : '') + line
|
||||
i++
|
||||
continue
|
||||
}
|
||||
|
||||
// Indented code block (4 spaces or 1 tab)
|
||||
if (/^(?: |\t)/.test(line) && line.trim().length > 0) {
|
||||
let codeLines = line.replace(/^(?: |\t)/, '')
|
||||
i++
|
||||
while (i < lines.length && (/^(?: |\t)/.test(lines[i]) || lines[i].trim() === '')) {
|
||||
codeLines += '\n' + lines[i].replace(/^(?: |\t)/, '')
|
||||
i++
|
||||
}
|
||||
if (codeLines.trim()) {
|
||||
segments.push({ text: codeLines.trim(), contentCategory: 'code' })
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// Heading (# style)
|
||||
const headingMatch = line.match(/^#{1,6}\s+(.+)$/)
|
||||
if (headingMatch) {
|
||||
segments.push({ text: headingMatch[1].trim(), contentCategory: 'heading' })
|
||||
i++
|
||||
continue
|
||||
}
|
||||
|
||||
// Regular prose line
|
||||
if (line.trim().length > 0) {
|
||||
segments.push({ text: line.trim(), contentCategory: 'prose' })
|
||||
}
|
||||
|
||||
i++
|
||||
}
|
||||
|
||||
// Flush unclosed code block
|
||||
if (inCodeBlock && codeContent.trim()) {
|
||||
segments.push({ text: codeContent.trim(), contentCategory: 'code' })
|
||||
}
|
||||
|
||||
return segments.filter(s => s.text.length > 0)
|
||||
}
|
||||
|
|
@ -1273,8 +1273,8 @@ export class MetadataIndexManager {
|
|||
if (typeof data === 'string') return data
|
||||
if (typeof data === 'number' || typeof data === 'boolean') return String(data)
|
||||
if (Array.isArray(data)) {
|
||||
// Skip large arrays (likely vectors)
|
||||
if (data.length > 10) return ''
|
||||
// Skip numeric arrays (vectors/embeddings), allow object/string arrays
|
||||
if (data.length > 0 && typeof data[0] === 'number') return ''
|
||||
return data.map(d => this.extractTextContent(d)).filter(Boolean).join(' ')
|
||||
}
|
||||
if (typeof data === 'object') {
|
||||
|
|
|
|||
|
|
@ -738,4 +738,242 @@ describe('Hybrid Search (v7.7.0)', () => {
|
|||
expect(programmingMatch?.score).toBe(1.0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Structured Content Extraction (v7.9.0)', () => {
|
||||
it('should extract text from TipTap/ProseMirror JSON', async () => {
|
||||
const tiptapDoc = JSON.stringify({
|
||||
type: 'doc',
|
||||
content: [
|
||||
{
|
||||
type: 'heading',
|
||||
attrs: { level: 1 },
|
||||
content: [{ type: 'text', text: 'David the Warrior' }]
|
||||
},
|
||||
{
|
||||
type: 'paragraph',
|
||||
content: [{ type: 'text', text: 'He is a brave fighter who battles dragons' }]
|
||||
}
|
||||
]
|
||||
})
|
||||
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior fighter',
|
||||
text: tiptapDoc,
|
||||
granularity: 'word',
|
||||
threshold: 0.3
|
||||
})
|
||||
|
||||
expect(highlights.length).toBeGreaterThan(0)
|
||||
// Should find "Warrior" as text match from heading
|
||||
const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
|
||||
expect(warriorMatch).toBeDefined()
|
||||
expect(warriorMatch?.matchType).toBe('text')
|
||||
// Should annotate heading content
|
||||
expect(warriorMatch?.contentCategory).toBe('heading')
|
||||
})
|
||||
|
||||
it('should extract text from Slate.js JSON', async () => {
|
||||
const slateDoc = JSON.stringify([
|
||||
{
|
||||
type: 'heading',
|
||||
children: [{ text: 'Introduction' }]
|
||||
},
|
||||
{
|
||||
type: 'paragraph',
|
||||
children: [
|
||||
{ text: 'This is ' },
|
||||
{ text: 'bold text', bold: true },
|
||||
{ text: ' in a paragraph' }
|
||||
]
|
||||
}
|
||||
])
|
||||
|
||||
const highlights = await brain.highlight({
|
||||
query: 'Introduction bold',
|
||||
text: slateDoc,
|
||||
granularity: 'word',
|
||||
threshold: 0.3
|
||||
})
|
||||
|
||||
expect(highlights.length).toBeGreaterThan(0)
|
||||
const introMatch = highlights.find(h => h.text === 'Introduction')
|
||||
expect(introMatch).toBeDefined()
|
||||
expect(introMatch?.matchType).toBe('text')
|
||||
expect(introMatch?.contentCategory).toBe('heading')
|
||||
})
|
||||
|
||||
it('should extract text from Quill Delta JSON', async () => {
|
||||
const quilDoc = JSON.stringify({
|
||||
ops: [
|
||||
{ insert: 'Hello World\n', attributes: { header: 1 } },
|
||||
{ insert: 'This is a paragraph with warrior content\n' }
|
||||
]
|
||||
})
|
||||
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior',
|
||||
text: quilDoc,
|
||||
granularity: 'word',
|
||||
threshold: 0.3
|
||||
})
|
||||
|
||||
expect(highlights.length).toBeGreaterThan(0)
|
||||
const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
|
||||
expect(warriorMatch).toBeDefined()
|
||||
expect(warriorMatch?.matchType).toBe('text')
|
||||
})
|
||||
|
||||
it('should fall back to text extraction for generic JSON', async () => {
|
||||
const apiResponse = JSON.stringify({
|
||||
name: 'warrior',
|
||||
description: 'A brave fighter',
|
||||
stats: { strength: 10, agility: 8 }
|
||||
})
|
||||
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior',
|
||||
text: apiResponse,
|
||||
granularity: 'word',
|
||||
threshold: 0.3
|
||||
})
|
||||
|
||||
expect(highlights.length).toBeGreaterThan(0)
|
||||
const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
|
||||
expect(warriorMatch).toBeDefined()
|
||||
})
|
||||
|
||||
it('should extract text from HTML with headings and code', async () => {
|
||||
const html = '<h1>Warrior Guide</h1><p>A brave fighter battles enemies.</p><code>const warrior = new Fighter()</code>'
|
||||
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior fighter',
|
||||
text: html,
|
||||
granularity: 'word',
|
||||
threshold: 0.3
|
||||
})
|
||||
|
||||
expect(highlights.length).toBeGreaterThan(0)
|
||||
// Should find "Warrior" from heading
|
||||
const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
|
||||
expect(warriorMatch).toBeDefined()
|
||||
expect(warriorMatch?.contentCategory).toBe('heading')
|
||||
|
||||
// Should find "fighter" from paragraph
|
||||
const fighterMatch = highlights.find(h => h.text.toLowerCase() === 'fighter')
|
||||
expect(fighterMatch).toBeDefined()
|
||||
expect(fighterMatch?.contentCategory).toBe('prose')
|
||||
})
|
||||
|
||||
it('should extract text from Markdown with headings and code', async () => {
|
||||
const markdown = '# Warrior Guide\n\nA brave fighter battles enemies.\n\n```\nconst warrior = new Fighter()\n```'
|
||||
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior fighter',
|
||||
text: markdown,
|
||||
granularity: 'word',
|
||||
threshold: 0.3
|
||||
})
|
||||
|
||||
expect(highlights.length).toBeGreaterThan(0)
|
||||
// Should find text from heading
|
||||
const headingMatch = highlights.find(h => h.text === 'Guide' && h.contentCategory === 'heading')
|
||||
|| highlights.find(h => h.text === 'Warrior' && h.contentCategory === 'heading')
|
||||
expect(headingMatch).toBeDefined()
|
||||
})
|
||||
|
||||
it('should preserve plain text behavior (regression)', async () => {
|
||||
const plainText = 'David Smith is a brave warrior who battles dragons'
|
||||
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior',
|
||||
text: plainText,
|
||||
granularity: 'word',
|
||||
threshold: 0.3
|
||||
})
|
||||
|
||||
expect(highlights.length).toBeGreaterThan(0)
|
||||
const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
|
||||
expect(warriorMatch).toBeDefined()
|
||||
expect(warriorMatch?.matchType).toBe('text')
|
||||
expect(warriorMatch?.score).toBe(1.0)
|
||||
// Plain text gets 'prose' category
|
||||
expect(warriorMatch?.contentCategory).toBe('prose')
|
||||
})
|
||||
|
||||
it('should use contentType hint to skip auto-detection', async () => {
|
||||
// Pass HTML as plain text via content type hint
|
||||
const htmlAsPlain = '<h1>test</h1>'
|
||||
|
||||
const highlights = await brain.highlight({
|
||||
query: 'test',
|
||||
text: htmlAsPlain,
|
||||
granularity: 'word',
|
||||
threshold: 0.3,
|
||||
contentType: 'plaintext' // Force plain text treatment
|
||||
})
|
||||
|
||||
// When treated as plain text, the raw HTML tags become part of chunks
|
||||
// The content should not be parsed as HTML
|
||||
expect(highlights).toBeDefined()
|
||||
})
|
||||
|
||||
it('should use custom contentExtractor when provided', async () => {
|
||||
const customExtractor = (text: string) => {
|
||||
return [
|
||||
{ text: 'custom extracted warrior content', contentCategory: 'prose' as const },
|
||||
{ text: 'Code Section', contentCategory: 'code' as const }
|
||||
]
|
||||
}
|
||||
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior',
|
||||
text: 'ignored because custom extractor is used',
|
||||
granularity: 'word',
|
||||
threshold: 0.3,
|
||||
contentExtractor: customExtractor
|
||||
})
|
||||
|
||||
expect(highlights.length).toBeGreaterThan(0)
|
||||
const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
|
||||
expect(warriorMatch).toBeDefined()
|
||||
expect(warriorMatch?.matchType).toBe('text')
|
||||
expect(warriorMatch?.contentCategory).toBe('prose')
|
||||
})
|
||||
})
|
||||
|
||||
describe('Timeout Protection (v7.9.0)', () => {
|
||||
it('should return text-only results if semantic phase times out', async () => {
|
||||
// We test the timeout path by ensuring the highlight method doesn't hang.
|
||||
// The mock embeddings are fast so timeout won't trigger in normal test,
|
||||
// but we verify the method completes successfully.
|
||||
const highlights = await brain.highlight({
|
||||
query: 'warrior',
|
||||
text: 'David is a brave warrior who battles dragons',
|
||||
granularity: 'word',
|
||||
threshold: 0.3
|
||||
})
|
||||
|
||||
// Should complete without hanging
|
||||
expect(highlights).toBeDefined()
|
||||
expect(highlights.length).toBeGreaterThan(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('embedBatch uses native batch API (v7.9.0)', () => {
|
||||
it('should embed multiple texts in a single call', async () => {
|
||||
const texts = ['hello world', 'foo bar', 'test content']
|
||||
const embeddings = await brain.embedBatch(texts)
|
||||
|
||||
expect(embeddings.length).toBe(3)
|
||||
// Each embedding should be 384 dimensions
|
||||
for (const emb of embeddings) {
|
||||
expect(emb.length).toBe(384)
|
||||
}
|
||||
})
|
||||
|
||||
it('should return empty array for empty input', async () => {
|
||||
const embeddings = await brain.embedBatch([])
|
||||
expect(embeddings).toEqual([])
|
||||
})
|
||||
})
|
||||
})
|
||||
|
|
|
|||
321
tests/unit/utils/contentExtractor.test.ts
Normal file
321
tests/unit/utils/contentExtractor.test.ts
Normal file
|
|
@ -0,0 +1,321 @@
|
|||
/**
|
||||
* Content Extractor Tests (v7.9.0)
|
||||
*
|
||||
* Tests for content type detection and text extraction from
|
||||
* structured formats (rich-text JSON, HTML, Markdown).
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import {
|
||||
detectContentType,
|
||||
extractForHighlighting
|
||||
} from '../../../src/utils/contentExtractor'
|
||||
|
||||
describe('Content Extractor (v7.9.0)', () => {
|
||||
describe('detectContentType()', () => {
|
||||
it('should detect plaintext', () => {
|
||||
expect(detectContentType('Hello world')).toBe('plaintext')
|
||||
expect(detectContentType('Just some regular text.')).toBe('plaintext')
|
||||
})
|
||||
|
||||
it('should detect rich-text JSON (object)', () => {
|
||||
const tiptap = JSON.stringify({ type: 'doc', content: [] })
|
||||
expect(detectContentType(tiptap)).toBe('richtext-json')
|
||||
})
|
||||
|
||||
it('should detect rich-text JSON (array)', () => {
|
||||
const slate = JSON.stringify([{ type: 'paragraph', children: [{ text: 'hi' }] }])
|
||||
expect(detectContentType(slate)).toBe('richtext-json')
|
||||
})
|
||||
|
||||
it('should detect HTML', () => {
|
||||
expect(detectContentType('<h1>Title</h1>')).toBe('html')
|
||||
expect(detectContentType('<div>content</div>')).toBe('html')
|
||||
expect(detectContentType('<!DOCTYPE html><html>')).toBe('html')
|
||||
})
|
||||
|
||||
it('should detect Markdown', () => {
|
||||
expect(detectContentType('# Heading\n\nSome text')).toBe('markdown')
|
||||
expect(detectContentType('```\ncode\n```')).toBe('markdown')
|
||||
})
|
||||
|
||||
it('should handle invalid JSON as plaintext', () => {
|
||||
expect(detectContentType('{ broken json')).toBe('plaintext')
|
||||
expect(detectContentType('[not valid')).toBe('plaintext')
|
||||
})
|
||||
|
||||
it('should handle whitespace-prefixed JSON', () => {
|
||||
const json = ' { "type": "doc" }'
|
||||
expect(detectContentType(json)).toBe('richtext-json')
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractForHighlighting() — TipTap/ProseMirror', () => {
|
||||
it('should extract text nodes with content categories', () => {
|
||||
const doc = JSON.stringify({
|
||||
type: 'doc',
|
||||
content: [
|
||||
{
|
||||
type: 'heading',
|
||||
attrs: { level: 1 },
|
||||
content: [{ type: 'text', text: 'My Heading' }]
|
||||
},
|
||||
{
|
||||
type: 'paragraph',
|
||||
content: [{ type: 'text', text: 'Regular paragraph text' }]
|
||||
},
|
||||
{
|
||||
type: 'codeBlock',
|
||||
content: [{ type: 'text', text: 'const x = 1' }]
|
||||
}
|
||||
]
|
||||
})
|
||||
|
||||
const segments = extractForHighlighting(doc)
|
||||
expect(segments.length).toBe(3)
|
||||
|
||||
expect(segments[0].text).toBe('My Heading')
|
||||
expect(segments[0].contentCategory).toBe('heading')
|
||||
|
||||
expect(segments[1].text).toBe('Regular paragraph text')
|
||||
expect(segments[1].contentCategory).toBe('prose')
|
||||
|
||||
expect(segments[2].text).toBe('const x = 1')
|
||||
expect(segments[2].contentCategory).toBe('code')
|
||||
})
|
||||
|
||||
it('should handle nested content with marks', () => {
|
||||
const doc = JSON.stringify({
|
||||
type: 'doc',
|
||||
content: [
|
||||
{
|
||||
type: 'paragraph',
|
||||
content: [
|
||||
{ type: 'text', text: 'Hello ' },
|
||||
{ type: 'text', text: 'bold world', marks: [{ type: 'bold' }] }
|
||||
]
|
||||
}
|
||||
]
|
||||
})
|
||||
|
||||
const segments = extractForHighlighting(doc)
|
||||
expect(segments.length).toBe(2)
|
||||
expect(segments[0].text).toBe('Hello ')
|
||||
expect(segments[1].text).toBe('bold world')
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractForHighlighting() — Slate.js', () => {
|
||||
it('should extract text from Slate children nodes', () => {
|
||||
const doc = JSON.stringify([
|
||||
{
|
||||
type: 'heading',
|
||||
children: [{ text: 'Slate Heading' }]
|
||||
},
|
||||
{
|
||||
type: 'paragraph',
|
||||
children: [
|
||||
{ text: 'First part ' },
|
||||
{ text: 'second part', bold: true }
|
||||
]
|
||||
}
|
||||
])
|
||||
|
||||
const segments = extractForHighlighting(doc)
|
||||
expect(segments.length).toBe(3)
|
||||
expect(segments[0].text).toBe('Slate Heading')
|
||||
expect(segments[0].contentCategory).toBe('heading')
|
||||
expect(segments[1].text).toBe('First part ')
|
||||
expect(segments[1].contentCategory).toBe('prose')
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractForHighlighting() — Lexical', () => {
|
||||
it('should extract text from Lexical root structure', () => {
|
||||
const doc = JSON.stringify({
|
||||
root: {
|
||||
children: [
|
||||
{
|
||||
type: 'heading',
|
||||
children: [{ type: 'text', text: 'Lexical Heading' }]
|
||||
},
|
||||
{
|
||||
type: 'paragraph',
|
||||
children: [{ type: 'text', text: 'Lexical body text' }]
|
||||
}
|
||||
]
|
||||
}
|
||||
})
|
||||
|
||||
const segments = extractForHighlighting(doc)
|
||||
expect(segments.length).toBe(2)
|
||||
expect(segments[0].text).toBe('Lexical Heading')
|
||||
expect(segments[0].contentCategory).toBe('heading')
|
||||
expect(segments[1].text).toBe('Lexical body text')
|
||||
expect(segments[1].contentCategory).toBe('prose')
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractForHighlighting() — Draft.js', () => {
|
||||
it('should extract text from Draft.js blocks', () => {
|
||||
const doc = JSON.stringify({
|
||||
blocks: [
|
||||
{ type: 'header-one', text: 'Draft Heading' },
|
||||
{ type: 'unstyled', text: 'Draft body text' }
|
||||
]
|
||||
})
|
||||
|
||||
const segments = extractForHighlighting(doc)
|
||||
expect(segments.length).toBeGreaterThanOrEqual(2)
|
||||
// Draft.js blocks have text property directly with type
|
||||
const headingSegment = segments.find(s => s.text === 'Draft Heading')
|
||||
const bodySegment = segments.find(s => s.text === 'Draft body text')
|
||||
expect(headingSegment).toBeDefined()
|
||||
expect(bodySegment).toBeDefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractForHighlighting() — Quill Delta', () => {
|
||||
it('should extract text from Quill Delta ops', () => {
|
||||
const doc = JSON.stringify({
|
||||
ops: [
|
||||
{ insert: 'Hello World\n' },
|
||||
{ insert: 'Second line\n' }
|
||||
]
|
||||
})
|
||||
|
||||
const segments = extractForHighlighting(doc)
|
||||
expect(segments.length).toBe(2)
|
||||
expect(segments[0].text).toContain('Hello World')
|
||||
expect(segments[1].text).toContain('Second line')
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractForHighlighting() — Generic JSON fallback', () => {
|
||||
it('should extract all string values from unrecognized JSON', () => {
|
||||
const generic = JSON.stringify({
|
||||
name: 'David',
|
||||
description: 'A brave warrior',
|
||||
stats: { strength: 10 }
|
||||
})
|
||||
|
||||
const segments = extractForHighlighting(generic)
|
||||
expect(segments.length).toBeGreaterThan(0)
|
||||
const allText = segments.map(s => s.text).join(' ')
|
||||
expect(allText).toContain('David')
|
||||
expect(allText).toContain('brave warrior')
|
||||
})
|
||||
|
||||
it('should return empty for JSON with no text', () => {
|
||||
const noText = JSON.stringify({ count: 42, active: true })
|
||||
const segments = extractForHighlighting(noText)
|
||||
// No string values to extract
|
||||
expect(segments).toBeDefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractForHighlighting() — HTML', () => {
|
||||
it('should extract headings, paragraphs, and code', () => {
|
||||
const html = '<h1>Title</h1><p>Body text here.</p><pre><code>let x = 1</code></pre>'
|
||||
|
||||
const segments = extractForHighlighting(html)
|
||||
expect(segments.length).toBeGreaterThanOrEqual(3)
|
||||
|
||||
const headingSegment = segments.find(s => s.text === 'Title')
|
||||
expect(headingSegment).toBeDefined()
|
||||
expect(headingSegment?.contentCategory).toBe('heading')
|
||||
|
||||
const bodySegment = segments.find(s => s.text === 'Body text here.')
|
||||
expect(bodySegment).toBeDefined()
|
||||
expect(bodySegment?.contentCategory).toBe('prose')
|
||||
|
||||
const codeSegment = segments.find(s => s.text === 'let x = 1')
|
||||
expect(codeSegment).toBeDefined()
|
||||
expect(codeSegment?.contentCategory).toBe('code')
|
||||
})
|
||||
|
||||
it('should skip script and style content', () => {
|
||||
const html = '<p>Visible</p><script>alert("hidden")</script><style>.foo{}</style><p>Also visible</p>'
|
||||
|
||||
const segments = extractForHighlighting(html)
|
||||
const allText = segments.map(s => s.text).join(' ')
|
||||
expect(allText).toContain('Visible')
|
||||
expect(allText).toContain('Also visible')
|
||||
expect(allText).not.toContain('alert')
|
||||
expect(allText).not.toContain('.foo')
|
||||
})
|
||||
|
||||
it('should decode common HTML entities', () => {
|
||||
const html = '<p>Tom & Jerry <3</p>'
|
||||
const segments = extractForHighlighting(html)
|
||||
expect(segments.length).toBeGreaterThan(0)
|
||||
expect(segments[0].text).toContain('Tom & Jerry <3')
|
||||
})
|
||||
|
||||
it('should handle nested heading tags', () => {
|
||||
const html = '<h2><span>Nested Heading</span></h2>'
|
||||
const segments = extractForHighlighting(html)
|
||||
const heading = segments.find(s => s.text === 'Nested Heading')
|
||||
expect(heading).toBeDefined()
|
||||
expect(heading?.contentCategory).toBe('heading')
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractForHighlighting() — Markdown', () => {
|
||||
it('should extract headings', () => {
|
||||
const md = '# Main Title\n\nSome body text\n\n## Subtitle'
|
||||
|
||||
const segments = extractForHighlighting(md)
|
||||
const heading1 = segments.find(s => s.text === 'Main Title')
|
||||
const heading2 = segments.find(s => s.text === 'Subtitle')
|
||||
const body = segments.find(s => s.text === 'Some body text')
|
||||
|
||||
expect(heading1).toBeDefined()
|
||||
expect(heading1?.contentCategory).toBe('heading')
|
||||
expect(heading2).toBeDefined()
|
||||
expect(heading2?.contentCategory).toBe('heading')
|
||||
expect(body).toBeDefined()
|
||||
expect(body?.contentCategory).toBe('prose')
|
||||
})
|
||||
|
||||
it('should extract fenced code blocks', () => {
|
||||
const md = '# Title\n\n```typescript\nconst x = 1\nconst y = 2\n```\n\nMore text'
|
||||
|
||||
const segments = extractForHighlighting(md)
|
||||
const code = segments.find(s => s.contentCategory === 'code')
|
||||
expect(code).toBeDefined()
|
||||
expect(code?.text).toContain('const x = 1')
|
||||
})
|
||||
|
||||
it('should extract indented code blocks', () => {
|
||||
// Indented code alone doesn't trigger markdown detection, so pass explicit type
|
||||
const md = 'Normal text\n\n code line 1\n code line 2\n\nMore text'
|
||||
|
||||
const segments = extractForHighlighting(md, 'markdown')
|
||||
const code = segments.find(s => s.contentCategory === 'code')
|
||||
expect(code).toBeDefined()
|
||||
expect(code?.text).toContain('code line 1')
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractForHighlighting() — Plaintext', () => {
|
||||
it('should return single prose segment for plain text', () => {
|
||||
const text = 'Just some plain text content'
|
||||
const segments = extractForHighlighting(text)
|
||||
|
||||
expect(segments.length).toBe(1)
|
||||
expect(segments[0].text).toBe(text)
|
||||
expect(segments[0].contentCategory).toBe('prose')
|
||||
})
|
||||
|
||||
it('should respect contentType override', () => {
|
||||
// Even if content looks like HTML, plaintext hint should skip detection
|
||||
const html = '<h1>Title</h1>'
|
||||
const segments = extractForHighlighting(html, 'plaintext')
|
||||
|
||||
expect(segments.length).toBe(1)
|
||||
expect(segments[0].text).toBe(html) // Raw HTML, not extracted
|
||||
expect(segments[0].contentCategory).toBe('prose')
|
||||
})
|
||||
})
|
||||
})
|
||||
|
|
@ -215,4 +215,70 @@ describe('MetadataIndexManager Text Indexing (v7.7.0)', () => {
|
|||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractTextContent() array handling (v7.9.0 fix)', () => {
|
||||
let manager: MetadataIndexManager
|
||||
|
||||
beforeEach(() => {
|
||||
const mockStorage = {
|
||||
getMetadata: async () => null,
|
||||
saveMetadata: async () => {},
|
||||
getNoun: async () => null,
|
||||
getNouns: async () => ({ items: [], total: 0, hasMore: false }),
|
||||
getVerbsFromSource: async () => ({ items: [], total: 0, hasMore: false }),
|
||||
} as any
|
||||
|
||||
manager = new MetadataIndexManager(mockStorage, {})
|
||||
})
|
||||
|
||||
it('should skip numeric arrays (vectors/embeddings)', () => {
|
||||
const result = manager.extractTextContent([0.1, 0.2, 0.3, 0.4, 0.5])
|
||||
expect(result).toBe('')
|
||||
})
|
||||
|
||||
it('should extract text from string arrays', () => {
|
||||
const result = manager.extractTextContent(['hello', 'world', 'test'])
|
||||
expect(result).toContain('hello')
|
||||
expect(result).toContain('world')
|
||||
expect(result).toContain('test')
|
||||
})
|
||||
|
||||
it('should extract text from arrays of objects', () => {
|
||||
const result = manager.extractTextContent([
|
||||
{ name: 'Alice', role: 'engineer' },
|
||||
{ name: 'Bob', role: 'designer' },
|
||||
{ name: 'Carol', role: 'manager' },
|
||||
{ name: 'Dave', role: 'analyst' },
|
||||
{ name: 'Eve', role: 'scientist' },
|
||||
{ name: 'Frank', role: 'developer' },
|
||||
{ name: 'Grace', role: 'architect' },
|
||||
{ name: 'Heidi', role: 'lead' },
|
||||
{ name: 'Ivan', role: 'intern' },
|
||||
{ name: 'Judy', role: 'director' },
|
||||
{ name: 'Karl', role: 'founder' },
|
||||
])
|
||||
// Should NOT skip arrays of objects even with >10 elements
|
||||
expect(result).toContain('Alice')
|
||||
expect(result).toContain('Karl')
|
||||
expect(result).toContain('engineer')
|
||||
expect(result).toContain('founder')
|
||||
})
|
||||
|
||||
it('should handle large numeric arrays (vectors)', () => {
|
||||
const vector = Array.from({ length: 384 }, (_, i) => Math.sin(i))
|
||||
const result = manager.extractTextContent(vector)
|
||||
expect(result).toBe('')
|
||||
})
|
||||
|
||||
it('should handle empty arrays', () => {
|
||||
const result = manager.extractTextContent([])
|
||||
expect(result).toBe('')
|
||||
})
|
||||
|
||||
it('should handle mixed arrays starting with string', () => {
|
||||
const result = manager.extractTextContent(['text', 42, true])
|
||||
expect(result).toContain('text')
|
||||
expect(result).toContain('42')
|
||||
})
|
||||
})
|
||||
})
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue