Brave fighters.
" }) + +// Markdown +await brain.highlight({ query: "warrior", text: "# Warriors\n\nBrave fighters." }) +``` + +**Content Category Annotations** + +Each `Highlight` now includes `contentCategory` when input is structured: +- `'heading'` — from ``/``, fenced/indented code blocks, or code nodes
+- `'prose'` — regular paragraph text
+- `'label'` — labels, captions, metadata-like text
+
+**Custom Content Extractors**
+
+New `contentExtractor` parameter lets developers plug in custom parsers:
+
+```typescript
+const highlights = await brain.highlight({
+ query: "function",
+ text: sourceCode,
+ contentExtractor: (text) => treeSitterParse(text) // Custom parser
+})
+```
+
+**Content Type Hints**
+
+New `contentType` parameter to skip auto-detection:
+
+```typescript
+await brain.highlight({ query: "test", text: input, contentType: 'html' })
+```
+
+### New Types
+
+- `ContentType`: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`
+- `ContentCategory`: `'prose' | 'heading' | 'code' | 'label'`
+- `ExtractedSegment`: `{ text: string, contentCategory: ContentCategory }`
+- `HighlightParams.contentType?` — optional content type hint
+- `HighlightParams.contentExtractor?` — optional custom parser callback
+- `Highlight.contentCategory?` — content role annotation
+
## [7.7.0](https://github.com/soulcraftlabs/brainy/compare/v7.6.1...v7.7.0) (2026-01-26)
### Features
diff --git a/README.md b/README.md
index edfc0846..1c85d92f 100644
--- a/README.md
+++ b/README.md
@@ -412,6 +412,13 @@ const results = await brain.find({ query: 'David Smith' })
await brain.find({ query: 'exact match', searchMode: 'text' }) // Text only
await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // Semantic only
await brain.find({ query: 'hybrid', hybridAlpha: 0.3 }) // Custom weighting
+
+// Highlight structured content (v7.8.0) — plain text, JSON, HTML, Markdown
+const highlights = await brain.highlight({
+ query: 'warrior',
+ text: entity.data // Auto-detects TipTap, Slate, Lexical, HTML, Markdown
+})
+// Each highlight has: text, score, position, matchType, contentCategory
```
**→ [Hybrid Search Documentation](docs/api/README.md#hybrid-search-v770)**
diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md
index e9b6711b..2ca57833 100644
--- a/docs/API_REFERENCE.md
+++ b/docs/API_REFERENCE.md
@@ -507,14 +507,16 @@ console.log(vector.length) // 384
---
-### `async embedBatch(texts: string[]): Promise` ✨ *v7.1.0*
-Batch embed multiple texts efficiently.
+### `async embedBatch(texts: string[]): Promise` ✨ *v7.1.0, Optimized v7.9.0*
+Batch embed multiple texts using native WASM batch API (single forward pass).
**Parameters:**
- `texts` - Array of strings to embed
**Returns:** Array of 384-dimensional vectors
+> **v7.9.0**: Uses the engine's native `embed_batch()` for a single model forward pass instead of N individual `embed()` calls.
+
**Example:**
```typescript
const embeddings = await brain.embedBatch([
diff --git a/docs/api/README.md b/docs/api/README.md
index e62f4f49..95da2014 100644
--- a/docs/api/README.md
+++ b/docs/api/README.md
@@ -270,41 +270,85 @@ results[0].matchSource // 'both' | 'text' | 'semantic'
---
-### `highlight(params)` → `Promise` ✨ *New v7.8.0*
+### `highlight(params)` → `Promise` ✨ *New v7.8.0, Enhanced v7.9.0*
Zero-config highlighting for both exact matches AND semantic concepts.
+Handles plain text, rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill), HTML, and Markdown automatically.
```typescript
+// Plain text (works as before)
const highlights = await brain.highlight({
query: "david the warrior",
text: "David Smith is a brave fighter who battles dragons"
})
-
-// Returns both text matches AND semantic matches:
// [
// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' },
// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' },
// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
// ]
+
+// Rich-text JSON (auto-detected, v7.9.0)
+const highlights = await brain.highlight({
+ query: "david the warrior",
+ text: JSON.stringify(tiptapDocument) // TipTap, Slate, Lexical, Draft.js, Quill
+})
+// Extracts text from nodes, annotates with contentCategory:
+// [
+// { text: "David", score: 1.0, matchType: 'text', contentCategory: 'heading' },
+// { text: "fighter", score: 0.78, matchType: 'semantic', contentCategory: 'prose' }
+// ]
+
+// HTML input (auto-detected, v7.9.0)
+const highlights = await brain.highlight({
+ query: "warrior",
+ text: "David the Warrior
A brave fighter.
"
+})
+
+// Custom extractor for proprietary formats (v7.9.0)
+const highlights = await brain.highlight({
+ query: "function",
+ text: sourceCode,
+ contentExtractor: (text) => treeSitterParse(text) // Your custom parser
+})
```
**Parameters:**
- `query`: `string` - The search query
-- `text`: `string` - Text to highlight (e.g., entity.data)
+- `text`: `string` - Text to highlight (plain text, JSON, HTML, or Markdown)
- `granularity?`: `'word' | 'phrase' | 'sentence'` - Highlight unit (default: 'word')
- `threshold?`: `number` - Min similarity for semantic matches (default: 0.5)
+- `contentType?`: `ContentType` - Optional hint: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`. Skips auto-detection when provided. *(v7.9.0)*
+- `contentExtractor?`: `(text: string) => ExtractedSegment[]` - Custom parser. Bypasses built-in detection entirely. *(v7.9.0)*
**Returns:** `Promise`
- `text` - The matched text
- `score` - Match score (1.0 for text matches, varies for semantic)
-- `position` - [start, end] indices in original text
+- `position` - [start, end] indices in extracted text
- `matchType` - `'text'` (exact) or `'semantic'` (concept)
+- `contentCategory?` - `'prose' | 'heading' | 'code' | 'label'` — Role of the source text in the document. Present when input is structured. *(v7.9.0)*
+
+**Supported Rich-Text Formats (v7.9.0):**
+
+| Format | Detection | Text nodes |
+|--------|-----------|------------|
+| TipTap / ProseMirror | `{ type: 'doc', content: [...] }` | `{ type: 'text', text }` |
+| Slate.js | `[{ type, children }]` | `{ text }` |
+| Lexical | `{ root: { children } }` | `{ type: 'text', text }` |
+| Draft.js | `{ blocks: [{ text }] }` | `{ text }` in block |
+| Quill Delta | `{ ops: [{ insert }] }` | `{ insert }` |
+| HTML | Tags like ``, `
`, `` | Visible text content |
+| Markdown | `#` headings, ` ``` ` code blocks | Stripped markup |
+
+**Timeout Protection (v7.9.0):**
+Semantic matching has a 10-second timeout. If embedding takes too long (e.g., WASM stall), `highlight()` returns text-only matches instead of hanging.
**UI Pattern:**
```typescript
-// Style differently based on match type
+// Style differently based on match type and content category
highlights.forEach(h => {
const style = h.matchType === 'text' ? 'font-weight: bold' : 'background: yellow'
+ if (h.contentCategory === 'heading') { /* render as heading highlight */ }
+ if (h.contentCategory === 'code') { /* render with code styling */ }
// Apply style from h.position[0] to h.position[1]
})
```
@@ -1677,9 +1721,9 @@ console.log(vector.length) // 384
---
-### `embedBatch(texts)` → `Promise` ✨ *New v7.1.0*
+### `embedBatch(texts)` → `Promise` ✨ *New v7.1.0, Optimized v7.9.0*
-Batch embed multiple texts efficiently.
+Batch embed multiple texts using native WASM batch API (single forward pass).
```typescript
const embeddings = await brain.embedBatch([
@@ -1691,6 +1735,8 @@ console.log(embeddings.length) // 3
console.log(embeddings[0].length) // 384
```
+> **v7.9.0**: Now uses the WASM engine's native `embed_batch()` for a single model forward pass instead of N individual calls. This is the same batch API used internally by `highlight()`.
+
---
### `similarity(textA, textB)` → `Promise` ✨ *New v7.1.0*
diff --git a/src/brainy.ts b/src/brainy.ts
index d7bfec7a..e7372ac0 100644
--- a/src/brainy.ts
+++ b/src/brainy.ts
@@ -26,6 +26,7 @@ import { TripleIntelligenceSystem } from './triple/TripleIntelligenceSystem.js'
import { VirtualFileSystem } from './vfs/VirtualFileSystem.js'
import { VersioningAPI } from './versioning/VersioningAPI.js'
import { MetadataIndexManager } from './utils/metadataIndex.js'
+import { detectContentType, extractForHighlighting } from './utils/contentExtractor.js'
import { GraphAdjacencyIndex } from './graph/graphAdjacencyIndex.js'
import { CommitBuilder } from './storage/cow/CommitObject.js'
import { BlobStorage } from './storage/cow/BlobStorage.js'
@@ -4663,13 +4664,8 @@ export class Brainy implements BrainyInterface {
return []
}
- // Use the underlying embedder for each text
- // The WASM engine handles batching internally
- const embeddings = await Promise.all(
- texts.map(text => this.embedder(text))
- )
-
- return embeddings
+ // Use native WASM batch API: single forward pass instead of N individual calls
+ return await embeddingManager.embedBatch(texts)
}
/**
@@ -4734,14 +4730,38 @@ export class Brainy implements BrainyInterface {
async highlight(params: import('./types/brainy.types.js').HighlightParams): Promise {
await this.ensureInitialized()
- const { query, text, granularity = 'word', threshold = 0.5 } = params
+ const { query, text, granularity = 'word', threshold = 0.5, contentType, contentExtractor } = params
if (!query || !text) {
return []
}
- // Split text into chunks based on granularity
- const allChunks = this.splitForHighlighting(text, granularity)
+ // v7.9.0: Extract text from structured content (JSON, HTML, Markdown)
+ // Custom extractor takes priority, then built-in detection
+ type ChunkWithCategory = { text: string, position: [number, number], contentCategory?: import('./types/brainy.types.js').ContentCategory }
+
+ let segments: import('./types/brainy.types.js').ExtractedSegment[]
+ if (contentExtractor) {
+ segments = contentExtractor(text)
+ } else {
+ segments = extractForHighlighting(text, contentType)
+ }
+
+ // Build concatenated text from segments for position tracking
+ // and split each segment into chunks based on granularity
+ const allChunks: ChunkWithCategory[] = []
+ let offset = 0
+ for (const segment of segments) {
+ const segmentChunks = this.splitForHighlighting(segment.text, granularity)
+ for (const chunk of segmentChunks) {
+ allChunks.push({
+ text: chunk.text,
+ position: [chunk.position[0] + offset, chunk.position[1] + offset],
+ contentCategory: segment.contentCategory
+ })
+ }
+ offset += segment.text.length + 1 // +1 for space between segments
+ }
if (allChunks.length === 0) {
return []
@@ -4749,7 +4769,6 @@ export class Brainy implements BrainyInterface {
// v7.8.0 Production safety: Limit chunks to prevent memory explosion
// At 500 words × 384 dimensions × 4 bytes = 768KB temp memory (acceptable)
- // At 10,000 words = 15MB temp memory (too much for concurrent requests)
const MAX_HIGHLIGHT_CHUNKS = 500
const chunks = allChunks.slice(0, MAX_HIGHLIGHT_CHUNKS)
@@ -4768,16 +4787,50 @@ export class Brainy implements BrainyInterface {
text: chunk.text,
score: 1.0,
position: chunk.position,
- matchType: 'text'
+ matchType: 'text',
+ contentCategory: chunk.contentCategory
})
}
}
- // === PHASE 2: Find semantic matches (score varies, matchType = 'semantic') ===
+ // === PHASE 2: Find semantic matches with timeout fallback ===
+ const SEMANTIC_TIMEOUT_MS = 10_000
+ try {
+ const semanticResult = await Promise.race([
+ this.highlightSemanticPhase(query, chunks, threshold, highlightMap),
+ new Promise<'timeout'>((resolve) => setTimeout(() => resolve('timeout'), SEMANTIC_TIMEOUT_MS))
+ ])
+
+ if (semanticResult === 'timeout') {
+ // Return Phase 1 text-only results on timeout
+ const textHighlights = Array.from(highlightMap.values())
+ return textHighlights.sort((a, b) => b.score - a.score)
+ }
+ } catch {
+ // On any error in semantic phase, return Phase 1 results
+ const textHighlights = Array.from(highlightMap.values())
+ return textHighlights.sort((a, b) => b.score - a.score)
+ }
+
+ // Sort by score descending (text matches will be first with score=1.0)
+ const highlights = Array.from(highlightMap.values())
+ return highlights.sort((a, b) => b.score - a.score)
+ }
+
+ /**
+ * Phase 2 of highlight(): semantic matching with batch embedding
+ * @internal
+ */
+ private async highlightSemanticPhase(
+ query: string,
+ chunks: Array<{ text: string, position: [number, number], contentCategory?: import('./types/brainy.types.js').ContentCategory }>,
+ threshold: number,
+ highlightMap: Map
+ ): Promise {
// Get query embedding
const queryVector = await this.embed(query)
- // Batch embed all chunks (efficient!)
+ // Batch embed all chunks using native WASM batch API
const chunkTexts = chunks.map(c => c.text)
const chunkVectors = await this.embedBatch(chunkTexts)
@@ -4796,14 +4849,11 @@ export class Brainy implements BrainyInterface {
text: chunks[i].text,
score: similarity,
position: chunks[i].position,
- matchType: 'semantic'
+ matchType: 'semantic',
+ contentCategory: chunks[i].contentCategory
})
}
}
-
- // Sort by score descending (text matches will be first with score=1.0)
- const highlights = Array.from(highlightMap.values())
- return highlights.sort((a, b) => b.score - a.score)
}
/**
diff --git a/src/embeddings/EmbeddingManager.ts b/src/embeddings/EmbeddingManager.ts
index ea88ffd7..1f9bfe6b 100644
--- a/src/embeddings/EmbeddingManager.ts
+++ b/src/embeddings/EmbeddingManager.ts
@@ -206,6 +206,35 @@ export class EmbeddingManager {
return vector
}
+ /**
+ * Batch embed multiple texts using native WASM batch API
+ *
+ * Uses the engine's embedBatch() for a single WASM forward pass
+ * instead of N individual embed() calls.
+ *
+ * @param texts Array of strings to embed
+ * @returns Array of embedding vectors (384 dimensions each)
+ */
+ async embedBatch(texts: string[]): Promise {
+ if (texts.length === 0) return []
+
+ const isTestMode =
+ process.env.BRAINY_UNIT_TEST === 'true' ||
+ (globalThis as any).__BRAINY_UNIT_TEST__
+
+ if (isTestMode) {
+ if (process.env.NODE_ENV === 'production') {
+ throw new Error('CRITICAL: Mock embeddings in production!')
+ }
+ return texts.map(t => this.getMockEmbedding(t))
+ }
+
+ await this.init()
+ const results = await this.engine.embedBatch(texts)
+ this.embedCount += texts.length
+ return results
+ }
+
/**
* Get embedding function for compatibility
*/
diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts
index 53ac4fd4..bb754946 100644
--- a/src/types/brainy.types.ts
+++ b/src/types/brainy.types.ts
@@ -789,25 +789,71 @@ export interface NeuralAnomalyParams {
returnScores?: boolean // Return anomaly scores
}
+// ============= Content Extraction Types (v7.9.0) =============
+
+/**
+ * Detected content type for smart text extraction
+ *
+ * @since v7.9.0
+ */
+export type ContentType = 'plaintext' | 'richtext-json' | 'html' | 'markdown'
+
+/**
+ * Content category for extracted segments
+ *
+ * Allows UI to apply different styling based on content role:
+ * - 'prose': Regular text content (paragraphs, list items)
+ * - 'heading': Heading/title text (h1-h6)
+ * - 'code': Code blocks or inline code
+ * - 'label': Labels, captions, or metadata-like text
+ *
+ * @since v7.9.0
+ */
+export type ContentCategory = 'prose' | 'heading' | 'code' | 'label'
+
+/**
+ * A segment of extracted text with its content category
+ *
+ * @since v7.9.0
+ */
+export interface ExtractedSegment {
+ /** The extracted text content */
+ text: string
+ /** What role this text plays in the document */
+ contentCategory: ContentCategory
+}
+
// ============= Semantic Highlighting Types (v7.8.0) =============
/**
- * Parameters for hybrid highlighting (v7.8.0)
+ * Parameters for hybrid highlighting (v7.8.0, enhanced v7.9.0)
*
* Zero-config highlighting that returns both text (exact) and semantic (concept) matches.
* Perfect for UI highlighting at different levels.
*
+ * v7.9.0: Added contentType hint and contentExtractor callback for structured text
+ * (rich-text JSON, HTML, Markdown). Auto-detects format when not specified.
+ *
* @example
* ```typescript
+ * // Plain text (unchanged from v7.8.0)
* const highlights = await brain.highlight({
* query: "david the warrior",
* text: "David Smith is a brave fighter who battles dragons"
* })
- * // Returns: [
- * // { text: "David", score: 1.0, position: [0, 5], matchType: 'text' }, // Exact match
- * // { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' }, // Concept match
- * // { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' } // Concept match
- * // ]
+ *
+ * // Rich-text JSON (auto-detected)
+ * const highlights = await brain.highlight({
+ * query: "david the warrior",
+ * text: JSON.stringify(tiptapDocument)
+ * })
+ *
+ * // Custom extractor (for proprietary formats)
+ * const highlights = await brain.highlight({
+ * query: "function",
+ * text: sourceCode,
+ * contentExtractor: (text) => treeSitterParse(text)
+ * })
* ```
*/
export interface HighlightParams {
@@ -822,6 +868,21 @@ export interface HighlightParams {
/** Minimum semantic similarity score for semantic matches (default: 0.5) */
threshold?: number
+
+ /**
+ * Optional content type hint to skip auto-detection.
+ * When omitted, the content type is detected from the text content.
+ * @since v7.9.0
+ */
+ contentType?: ContentType
+
+ /**
+ * Optional custom content extractor function.
+ * When provided, bypasses built-in detection and extraction entirely.
+ * Use this to plug in custom parsers (tree-sitter, Monaco, proprietary formats).
+ * @since v7.9.0
+ */
+ contentExtractor?: (text: string) => ExtractedSegment[]
}
/**
@@ -843,6 +904,13 @@ export interface Highlight {
/** Match type: 'text' (exact word match) or 'semantic' (concept match) */
matchType: 'text' | 'semantic'
+
+ /**
+ * Content category of the source segment (heading, code, prose, label).
+ * Present when the input text was structured (JSON, HTML, Markdown).
+ * @since v7.9.0
+ */
+ contentCategory?: ContentCategory
}
// ============= Export all types =============
diff --git a/src/utils/contentExtractor.ts b/src/utils/contentExtractor.ts
new file mode 100644
index 00000000..f0d19b08
--- /dev/null
+++ b/src/utils/contentExtractor.ts
@@ -0,0 +1,403 @@
+/**
+ * Content Extractor (v7.9.0)
+ *
+ * Detects content type (plaintext, rich-text JSON, HTML, Markdown) and extracts
+ * text segments with content categories (prose, heading, code, label).
+ *
+ * Supports common rich-text editor formats:
+ * - TipTap / ProseMirror: { type: 'doc', content: [...] }
+ * - Slate.js: [{ type, children: [{ text }] }]
+ * - Lexical: { root: { children: [...] } }
+ * - Draft.js: { blocks: [{ text }] }
+ * - Quill Delta: { ops: [{ insert }] }
+ *
+ * Falls back gracefully: structured text that doesn't match known patterns
+ * is extracted as plain prose via recursive text collection.
+ */
+
+import type { ContentType, ContentCategory, ExtractedSegment } from '../types/brainy.types.js'
+
+/**
+ * Detect content type from text content (no filename needed)
+ */
+export function detectContentType(text: string): ContentType {
+ const trimmed = text.trimStart()
+
+ // JSON detection (covers TipTap, Slate, Lexical, Draft.js, Quill, etc.)
+ if (trimmed.startsWith('{') || trimmed.startsWith('[')) {
+ try {
+ JSON.parse(trimmed)
+ return 'richtext-json'
+ } catch {
+ // Not valid JSON, fall through
+ }
+ }
+
+ // HTML detection (tag at start of content)
+ if (/<[a-z!][a-z0-9]*[\s>/]/i.test(trimmed.substring(0, 200))) {
+ return 'html'
+ }
+
+ // Markdown detection (heading or code fence at start)
+ if (/^#{1,6}\s/m.test(trimmed.substring(0, 500)) ||
+ /^```/m.test(trimmed.substring(0, 500))) {
+ return 'markdown'
+ }
+
+ return 'plaintext'
+}
+
+/**
+ * Extract text segments from content, auto-detecting type if not provided
+ */
+export function extractForHighlighting(
+ text: string,
+ contentType?: ContentType
+): ExtractedSegment[] {
+ const type = contentType || detectContentType(text)
+
+ switch (type) {
+ case 'richtext-json':
+ return extractFromJson(text)
+ case 'html':
+ return extractFromHtml(text)
+ case 'markdown':
+ return extractFromMarkdown(text)
+ case 'plaintext':
+ default:
+ return [{ text, contentCategory: 'prose' }]
+ }
+}
+
+// ============= Rich-Text JSON Extraction =============
+
+/**
+ * Extract text from any JSON rich-text format.
+ * Walks the node tree looking for text leaves across all common editors.
+ */
+function extractFromJson(text: string): ExtractedSegment[] {
+ let parsed: any
+ try {
+ parsed = JSON.parse(text.trim())
+ } catch {
+ // Invalid JSON — treat as plain text
+ return [{ text, contentCategory: 'prose' }]
+ }
+
+ const segments = walkRichTextNodes(parsed)
+
+ // If the walker found real text segments, return them
+ if (segments.length > 0 && segments.some(s => s.text.trim().length > 0)) {
+ return segments.filter(s => s.text.trim().length > 0)
+ }
+
+ // Fallback: collect all string values from the JSON
+ const fallbackText = extractTextFromJsonValue(parsed)
+ if (fallbackText.trim()) {
+ return [{ text: fallbackText, contentCategory: 'prose' }]
+ }
+
+ return []
+}
+
+/**
+ * Walk rich-text nodes recursively.
+ * Handles TipTap/ProseMirror, Slate, Lexical, Draft.js, and Quill Delta.
+ */
+function walkRichTextNodes(node: any): ExtractedSegment[] {
+ if (node === null || node === undefined) return []
+
+ const segments: ExtractedSegment[] = []
+
+ // Handle arrays (Slate root is an array, Draft.js blocks, Quill ops)
+ if (Array.isArray(node)) {
+ for (const child of node) {
+ segments.push(...walkRichTextNodes(child))
+ }
+ return segments
+ }
+
+ if (typeof node !== 'object') return []
+
+ // Leaf: text content (TipTap/ProseMirror/Lexical/Slate)
+ if (typeof node.text === 'string' && node.text.length > 0) {
+ segments.push({ text: node.text, contentCategory: 'prose' })
+ return segments
+ }
+
+ // Leaf: Quill Delta insert
+ if (typeof node.insert === 'string' && node.insert.length > 0) {
+ segments.push({ text: node.insert, contentCategory: 'prose' })
+ return segments
+ }
+
+ // Draft.js block with text
+ if (typeof node.text === 'string' && node.type !== undefined && node.text.length > 0) {
+ const category = categorizeNodeType(node.type)
+ segments.push({ text: node.text, contentCategory: category })
+ return segments
+ }
+
+ // Categorize by node type
+ const category = categorizeNodeType(node.type)
+
+ // Walk children arrays: content (TipTap), children (Slate/Lexical), blocks (Draft.js), ops (Quill)
+ const children = node.content || node.children || node.blocks || node.ops
+ if (Array.isArray(children)) {
+ for (const child of children) {
+ const childSegments = walkRichTextNodes(child)
+ // Apply parent's category if it's more specific than 'prose'
+ if (category !== 'prose') {
+ childSegments.forEach(s => { s.contentCategory = category })
+ }
+ segments.push(...childSegments)
+ }
+ }
+
+ // Lexical root wrapper
+ if (node.root && typeof node.root === 'object') {
+ segments.push(...walkRichTextNodes(node.root))
+ }
+
+ return segments
+}
+
+/**
+ * Categorize a node type string into a ContentCategory
+ */
+function categorizeNodeType(type?: string): ContentCategory {
+ if (!type) return 'prose'
+ const t = type.toLowerCase()
+ if (t === 'heading' || /^h[1-6]$/.test(t)) return 'heading'
+ if (t === 'code' || t === 'codeblock' || t === 'code_block') return 'code'
+ return 'prose'
+}
+
+/**
+ * Fallback: recursively extract all string values from JSON
+ */
+function extractTextFromJsonValue(value: any): string {
+ if (typeof value === 'string') return value
+ if (Array.isArray(value)) {
+ return value.map(extractTextFromJsonValue).filter(Boolean).join(' ')
+ }
+ if (typeof value === 'object' && value !== null) {
+ const texts: string[] = []
+ for (const v of Object.values(value)) {
+ const text = extractTextFromJsonValue(v)
+ if (text) texts.push(text)
+ }
+ return texts.join(' ')
+ }
+ return ''
+}
+
+// ============= HTML Extraction =============
+
+/**
+ * Extract text from HTML using a simple state-machine tag parser.
+ * No external dependencies.
+ */
+function extractFromHtml(html: string): ExtractedSegment[] {
+ const segments: ExtractedSegment[] = []
+ let current = ''
+ let currentCategory: ContentCategory = 'prose'
+ let i = 0
+ let insideTag = false
+ let tagName = ''
+ let isClosingTag = false
+ let skipContent = false
+
+ // Tag stack to track nesting
+ const tagStack: string[] = []
+
+ while (i < html.length) {
+ if (html[i] === '<') {
+ // Flush current text if we have any
+ if (current.trim() && !skipContent) {
+ segments.push({ text: current.trim(), contentCategory: currentCategory })
+ }
+ current = ''
+
+ insideTag = true
+ tagName = ''
+ isClosingTag = false
+ i++
+
+ // Check for closing tag
+ if (i < html.length && html[i] === '/') {
+ isClosingTag = true
+ i++
+ }
+
+ // Read tag name
+ while (i < html.length && html[i] !== '>' && html[i] !== ' ' && html[i] !== '/') {
+ tagName += html[i]
+ i++
+ }
+
+ // Skip to end of tag
+ while (i < html.length && html[i] !== '>') {
+ i++
+ }
+ if (i < html.length) i++ // skip '>'
+
+ const tagLower = tagName.toLowerCase()
+
+ if (isClosingTag) {
+ // Pop tag stack
+ if (tagStack.length > 0 && tagStack[tagStack.length - 1] === tagLower) {
+ tagStack.pop()
+ }
+ if (tagLower === 'script' || tagLower === 'style') {
+ skipContent = false
+ }
+ // Reset category based on remaining stack
+ currentCategory = getCategoryFromTagStack(tagStack)
+ } else {
+ // Opening tag
+ if (tagLower === 'script' || tagLower === 'style') {
+ skipContent = true
+ }
+
+ // Self-closing tags don't affect stack
+ const selfClosing = html[i - 2] === '/' ||
+ ['br', 'hr', 'img', 'input', 'meta', 'link'].includes(tagLower)
+
+ if (!selfClosing) {
+ tagStack.push(tagLower)
+ currentCategory = getCategoryFromTagStack(tagStack)
+ }
+ }
+
+ insideTag = false
+ continue
+ }
+
+ if (!insideTag) {
+ // Decode common HTML entities
+ if (html[i] === '&') {
+ const entityEnd = html.indexOf(';', i)
+ if (entityEnd !== -1 && entityEnd - i < 10) {
+ const entity = html.substring(i, entityEnd + 1)
+ current += decodeHtmlEntity(entity)
+ i = entityEnd + 1
+ continue
+ }
+ }
+ current += html[i]
+ }
+ i++
+ }
+
+ // Flush remaining text
+ if (current.trim() && !skipContent) {
+ segments.push({ text: current.trim(), contentCategory: currentCategory })
+ }
+
+ return segments.filter(s => s.text.length > 0)
+}
+
+/**
+ * Determine content category from the current tag stack
+ */
+function getCategoryFromTagStack(stack: string[]): ContentCategory {
+ for (let i = stack.length - 1; i >= 0; i--) {
+ const tag = stack[i]
+ if (/^h[1-6]$/.test(tag)) return 'heading'
+ if (tag === 'code' || tag === 'pre') return 'code'
+ }
+ return 'prose'
+}
+
+/**
+ * Decode common HTML entities
+ */
+function decodeHtmlEntity(entity: string): string {
+ const entities: Record = {
+ '&': '&',
+ '<': '<',
+ '>': '>',
+ '"': '"',
+ ''': "'",
+ ''': "'",
+ ' ': ' ',
+ }
+ return entities[entity] || entity
+}
+
+// ============= Markdown Extraction =============
+
+/**
+ * Extract text from Markdown with category detection
+ */
+function extractFromMarkdown(text: string): ExtractedSegment[] {
+ const segments: ExtractedSegment[] = []
+ const lines = text.split('\n')
+ let inCodeBlock = false
+ let codeContent = ''
+ let i = 0
+
+ while (i < lines.length) {
+ const line = lines[i]
+
+ // Fenced code block toggle
+ if (/^```/.test(line.trimStart())) {
+ if (inCodeBlock) {
+ // End of code block
+ if (codeContent.trim()) {
+ segments.push({ text: codeContent.trim(), contentCategory: 'code' })
+ }
+ codeContent = ''
+ inCodeBlock = false
+ } else {
+ // Start of code block
+ inCodeBlock = true
+ }
+ i++
+ continue
+ }
+
+ if (inCodeBlock) {
+ codeContent += (codeContent ? '\n' : '') + line
+ i++
+ continue
+ }
+
+ // Indented code block (4 spaces or 1 tab)
+ if (/^(?: |\t)/.test(line) && line.trim().length > 0) {
+ let codeLines = line.replace(/^(?: |\t)/, '')
+ i++
+ while (i < lines.length && (/^(?: |\t)/.test(lines[i]) || lines[i].trim() === '')) {
+ codeLines += '\n' + lines[i].replace(/^(?: |\t)/, '')
+ i++
+ }
+ if (codeLines.trim()) {
+ segments.push({ text: codeLines.trim(), contentCategory: 'code' })
+ }
+ continue
+ }
+
+ // Heading (# style)
+ const headingMatch = line.match(/^#{1,6}\s+(.+)$/)
+ if (headingMatch) {
+ segments.push({ text: headingMatch[1].trim(), contentCategory: 'heading' })
+ i++
+ continue
+ }
+
+ // Regular prose line
+ if (line.trim().length > 0) {
+ segments.push({ text: line.trim(), contentCategory: 'prose' })
+ }
+
+ i++
+ }
+
+ // Flush unclosed code block
+ if (inCodeBlock && codeContent.trim()) {
+ segments.push({ text: codeContent.trim(), contentCategory: 'code' })
+ }
+
+ return segments.filter(s => s.text.length > 0)
+}
diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts
index 2942f8dc..fa0c3903 100644
--- a/src/utils/metadataIndex.ts
+++ b/src/utils/metadataIndex.ts
@@ -1273,8 +1273,8 @@ export class MetadataIndexManager {
if (typeof data === 'string') return data
if (typeof data === 'number' || typeof data === 'boolean') return String(data)
if (Array.isArray(data)) {
- // Skip large arrays (likely vectors)
- if (data.length > 10) return ''
+ // Skip numeric arrays (vectors/embeddings), allow object/string arrays
+ if (data.length > 0 && typeof data[0] === 'number') return ''
return data.map(d => this.extractTextContent(d)).filter(Boolean).join(' ')
}
if (typeof data === 'object') {
diff --git a/tests/unit/brainy/hybrid-search.test.ts b/tests/unit/brainy/hybrid-search.test.ts
index 96fdbd2c..2aa9cc57 100644
--- a/tests/unit/brainy/hybrid-search.test.ts
+++ b/tests/unit/brainy/hybrid-search.test.ts
@@ -738,4 +738,242 @@ describe('Hybrid Search (v7.7.0)', () => {
expect(programmingMatch?.score).toBe(1.0)
})
})
+
+ describe('Structured Content Extraction (v7.9.0)', () => {
+ it('should extract text from TipTap/ProseMirror JSON', async () => {
+ const tiptapDoc = JSON.stringify({
+ type: 'doc',
+ content: [
+ {
+ type: 'heading',
+ attrs: { level: 1 },
+ content: [{ type: 'text', text: 'David the Warrior' }]
+ },
+ {
+ type: 'paragraph',
+ content: [{ type: 'text', text: 'He is a brave fighter who battles dragons' }]
+ }
+ ]
+ })
+
+ const highlights = await brain.highlight({
+ query: 'warrior fighter',
+ text: tiptapDoc,
+ granularity: 'word',
+ threshold: 0.3
+ })
+
+ expect(highlights.length).toBeGreaterThan(0)
+ // Should find "Warrior" as text match from heading
+ const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
+ expect(warriorMatch).toBeDefined()
+ expect(warriorMatch?.matchType).toBe('text')
+ // Should annotate heading content
+ expect(warriorMatch?.contentCategory).toBe('heading')
+ })
+
+ it('should extract text from Slate.js JSON', async () => {
+ const slateDoc = JSON.stringify([
+ {
+ type: 'heading',
+ children: [{ text: 'Introduction' }]
+ },
+ {
+ type: 'paragraph',
+ children: [
+ { text: 'This is ' },
+ { text: 'bold text', bold: true },
+ { text: ' in a paragraph' }
+ ]
+ }
+ ])
+
+ const highlights = await brain.highlight({
+ query: 'Introduction bold',
+ text: slateDoc,
+ granularity: 'word',
+ threshold: 0.3
+ })
+
+ expect(highlights.length).toBeGreaterThan(0)
+ const introMatch = highlights.find(h => h.text === 'Introduction')
+ expect(introMatch).toBeDefined()
+ expect(introMatch?.matchType).toBe('text')
+ expect(introMatch?.contentCategory).toBe('heading')
+ })
+
+ it('should extract text from Quill Delta JSON', async () => {
+ const quilDoc = JSON.stringify({
+ ops: [
+ { insert: 'Hello World\n', attributes: { header: 1 } },
+ { insert: 'This is a paragraph with warrior content\n' }
+ ]
+ })
+
+ const highlights = await brain.highlight({
+ query: 'warrior',
+ text: quilDoc,
+ granularity: 'word',
+ threshold: 0.3
+ })
+
+ expect(highlights.length).toBeGreaterThan(0)
+ const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
+ expect(warriorMatch).toBeDefined()
+ expect(warriorMatch?.matchType).toBe('text')
+ })
+
+ it('should fall back to text extraction for generic JSON', async () => {
+ const apiResponse = JSON.stringify({
+ name: 'warrior',
+ description: 'A brave fighter',
+ stats: { strength: 10, agility: 8 }
+ })
+
+ const highlights = await brain.highlight({
+ query: 'warrior',
+ text: apiResponse,
+ granularity: 'word',
+ threshold: 0.3
+ })
+
+ expect(highlights.length).toBeGreaterThan(0)
+ const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
+ expect(warriorMatch).toBeDefined()
+ })
+
+ it('should extract text from HTML with headings and code', async () => {
+ const html = 'Warrior Guide
A brave fighter battles enemies.
const warrior = new Fighter()'
+
+ const highlights = await brain.highlight({
+ query: 'warrior fighter',
+ text: html,
+ granularity: 'word',
+ threshold: 0.3
+ })
+
+ expect(highlights.length).toBeGreaterThan(0)
+ // Should find "Warrior" from heading
+ const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
+ expect(warriorMatch).toBeDefined()
+ expect(warriorMatch?.contentCategory).toBe('heading')
+
+ // Should find "fighter" from paragraph
+ const fighterMatch = highlights.find(h => h.text.toLowerCase() === 'fighter')
+ expect(fighterMatch).toBeDefined()
+ expect(fighterMatch?.contentCategory).toBe('prose')
+ })
+
+ it('should extract text from Markdown with headings and code', async () => {
+ const markdown = '# Warrior Guide\n\nA brave fighter battles enemies.\n\n```\nconst warrior = new Fighter()\n```'
+
+ const highlights = await brain.highlight({
+ query: 'warrior fighter',
+ text: markdown,
+ granularity: 'word',
+ threshold: 0.3
+ })
+
+ expect(highlights.length).toBeGreaterThan(0)
+ // Should find text from heading
+ const headingMatch = highlights.find(h => h.text === 'Guide' && h.contentCategory === 'heading')
+ || highlights.find(h => h.text === 'Warrior' && h.contentCategory === 'heading')
+ expect(headingMatch).toBeDefined()
+ })
+
+ it('should preserve plain text behavior (regression)', async () => {
+ const plainText = 'David Smith is a brave warrior who battles dragons'
+
+ const highlights = await brain.highlight({
+ query: 'warrior',
+ text: plainText,
+ granularity: 'word',
+ threshold: 0.3
+ })
+
+ expect(highlights.length).toBeGreaterThan(0)
+ const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
+ expect(warriorMatch).toBeDefined()
+ expect(warriorMatch?.matchType).toBe('text')
+ expect(warriorMatch?.score).toBe(1.0)
+ // Plain text gets 'prose' category
+ expect(warriorMatch?.contentCategory).toBe('prose')
+ })
+
+ it('should use contentType hint to skip auto-detection', async () => {
+ // Pass HTML as plain text via content type hint
+ const htmlAsPlain = 'test
'
+
+ const highlights = await brain.highlight({
+ query: 'test',
+ text: htmlAsPlain,
+ granularity: 'word',
+ threshold: 0.3,
+ contentType: 'plaintext' // Force plain text treatment
+ })
+
+ // When treated as plain text, the raw HTML tags become part of chunks
+ // The content should not be parsed as HTML
+ expect(highlights).toBeDefined()
+ })
+
+ it('should use custom contentExtractor when provided', async () => {
+ const customExtractor = (text: string) => {
+ return [
+ { text: 'custom extracted warrior content', contentCategory: 'prose' as const },
+ { text: 'Code Section', contentCategory: 'code' as const }
+ ]
+ }
+
+ const highlights = await brain.highlight({
+ query: 'warrior',
+ text: 'ignored because custom extractor is used',
+ granularity: 'word',
+ threshold: 0.3,
+ contentExtractor: customExtractor
+ })
+
+ expect(highlights.length).toBeGreaterThan(0)
+ const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
+ expect(warriorMatch).toBeDefined()
+ expect(warriorMatch?.matchType).toBe('text')
+ expect(warriorMatch?.contentCategory).toBe('prose')
+ })
+ })
+
+ describe('Timeout Protection (v7.9.0)', () => {
+ it('should return text-only results if semantic phase times out', async () => {
+ // We test the timeout path by ensuring the highlight method doesn't hang.
+ // The mock embeddings are fast so timeout won't trigger in normal test,
+ // but we verify the method completes successfully.
+ const highlights = await brain.highlight({
+ query: 'warrior',
+ text: 'David is a brave warrior who battles dragons',
+ granularity: 'word',
+ threshold: 0.3
+ })
+
+ // Should complete without hanging
+ expect(highlights).toBeDefined()
+ expect(highlights.length).toBeGreaterThan(0)
+ })
+ })
+
+ describe('embedBatch uses native batch API (v7.9.0)', () => {
+ it('should embed multiple texts in a single call', async () => {
+ const texts = ['hello world', 'foo bar', 'test content']
+ const embeddings = await brain.embedBatch(texts)
+
+ expect(embeddings.length).toBe(3)
+ // Each embedding should be 384 dimensions
+ for (const emb of embeddings) {
+ expect(emb.length).toBe(384)
+ }
+ })
+
+ it('should return empty array for empty input', async () => {
+ const embeddings = await brain.embedBatch([])
+ expect(embeddings).toEqual([])
+ })
+ })
})
diff --git a/tests/unit/utils/contentExtractor.test.ts b/tests/unit/utils/contentExtractor.test.ts
new file mode 100644
index 00000000..e1222129
--- /dev/null
+++ b/tests/unit/utils/contentExtractor.test.ts
@@ -0,0 +1,321 @@
+/**
+ * Content Extractor Tests (v7.9.0)
+ *
+ * Tests for content type detection and text extraction from
+ * structured formats (rich-text JSON, HTML, Markdown).
+ */
+
+import { describe, it, expect } from 'vitest'
+import {
+ detectContentType,
+ extractForHighlighting
+} from '../../../src/utils/contentExtractor'
+
+describe('Content Extractor (v7.9.0)', () => {
+ describe('detectContentType()', () => {
+ it('should detect plaintext', () => {
+ expect(detectContentType('Hello world')).toBe('plaintext')
+ expect(detectContentType('Just some regular text.')).toBe('plaintext')
+ })
+
+ it('should detect rich-text JSON (object)', () => {
+ const tiptap = JSON.stringify({ type: 'doc', content: [] })
+ expect(detectContentType(tiptap)).toBe('richtext-json')
+ })
+
+ it('should detect rich-text JSON (array)', () => {
+ const slate = JSON.stringify([{ type: 'paragraph', children: [{ text: 'hi' }] }])
+ expect(detectContentType(slate)).toBe('richtext-json')
+ })
+
+ it('should detect HTML', () => {
+ expect(detectContentType('Title
')).toBe('html')
+ expect(detectContentType('content')).toBe('html')
+ expect(detectContentType('')).toBe('html')
+ })
+
+ it('should detect Markdown', () => {
+ expect(detectContentType('# Heading\n\nSome text')).toBe('markdown')
+ expect(detectContentType('```\ncode\n```')).toBe('markdown')
+ })
+
+ it('should handle invalid JSON as plaintext', () => {
+ expect(detectContentType('{ broken json')).toBe('plaintext')
+ expect(detectContentType('[not valid')).toBe('plaintext')
+ })
+
+ it('should handle whitespace-prefixed JSON', () => {
+ const json = ' { "type": "doc" }'
+ expect(detectContentType(json)).toBe('richtext-json')
+ })
+ })
+
+ describe('extractForHighlighting() — TipTap/ProseMirror', () => {
+ it('should extract text nodes with content categories', () => {
+ const doc = JSON.stringify({
+ type: 'doc',
+ content: [
+ {
+ type: 'heading',
+ attrs: { level: 1 },
+ content: [{ type: 'text', text: 'My Heading' }]
+ },
+ {
+ type: 'paragraph',
+ content: [{ type: 'text', text: 'Regular paragraph text' }]
+ },
+ {
+ type: 'codeBlock',
+ content: [{ type: 'text', text: 'const x = 1' }]
+ }
+ ]
+ })
+
+ const segments = extractForHighlighting(doc)
+ expect(segments.length).toBe(3)
+
+ expect(segments[0].text).toBe('My Heading')
+ expect(segments[0].contentCategory).toBe('heading')
+
+ expect(segments[1].text).toBe('Regular paragraph text')
+ expect(segments[1].contentCategory).toBe('prose')
+
+ expect(segments[2].text).toBe('const x = 1')
+ expect(segments[2].contentCategory).toBe('code')
+ })
+
+ it('should handle nested content with marks', () => {
+ const doc = JSON.stringify({
+ type: 'doc',
+ content: [
+ {
+ type: 'paragraph',
+ content: [
+ { type: 'text', text: 'Hello ' },
+ { type: 'text', text: 'bold world', marks: [{ type: 'bold' }] }
+ ]
+ }
+ ]
+ })
+
+ const segments = extractForHighlighting(doc)
+ expect(segments.length).toBe(2)
+ expect(segments[0].text).toBe('Hello ')
+ expect(segments[1].text).toBe('bold world')
+ })
+ })
+
+ describe('extractForHighlighting() — Slate.js', () => {
+ it('should extract text from Slate children nodes', () => {
+ const doc = JSON.stringify([
+ {
+ type: 'heading',
+ children: [{ text: 'Slate Heading' }]
+ },
+ {
+ type: 'paragraph',
+ children: [
+ { text: 'First part ' },
+ { text: 'second part', bold: true }
+ ]
+ }
+ ])
+
+ const segments = extractForHighlighting(doc)
+ expect(segments.length).toBe(3)
+ expect(segments[0].text).toBe('Slate Heading')
+ expect(segments[0].contentCategory).toBe('heading')
+ expect(segments[1].text).toBe('First part ')
+ expect(segments[1].contentCategory).toBe('prose')
+ })
+ })
+
+ describe('extractForHighlighting() — Lexical', () => {
+ it('should extract text from Lexical root structure', () => {
+ const doc = JSON.stringify({
+ root: {
+ children: [
+ {
+ type: 'heading',
+ children: [{ type: 'text', text: 'Lexical Heading' }]
+ },
+ {
+ type: 'paragraph',
+ children: [{ type: 'text', text: 'Lexical body text' }]
+ }
+ ]
+ }
+ })
+
+ const segments = extractForHighlighting(doc)
+ expect(segments.length).toBe(2)
+ expect(segments[0].text).toBe('Lexical Heading')
+ expect(segments[0].contentCategory).toBe('heading')
+ expect(segments[1].text).toBe('Lexical body text')
+ expect(segments[1].contentCategory).toBe('prose')
+ })
+ })
+
+ describe('extractForHighlighting() — Draft.js', () => {
+ it('should extract text from Draft.js blocks', () => {
+ const doc = JSON.stringify({
+ blocks: [
+ { type: 'header-one', text: 'Draft Heading' },
+ { type: 'unstyled', text: 'Draft body text' }
+ ]
+ })
+
+ const segments = extractForHighlighting(doc)
+ expect(segments.length).toBeGreaterThanOrEqual(2)
+ // Draft.js blocks have text property directly with type
+ const headingSegment = segments.find(s => s.text === 'Draft Heading')
+ const bodySegment = segments.find(s => s.text === 'Draft body text')
+ expect(headingSegment).toBeDefined()
+ expect(bodySegment).toBeDefined()
+ })
+ })
+
+ describe('extractForHighlighting() — Quill Delta', () => {
+ it('should extract text from Quill Delta ops', () => {
+ const doc = JSON.stringify({
+ ops: [
+ { insert: 'Hello World\n' },
+ { insert: 'Second line\n' }
+ ]
+ })
+
+ const segments = extractForHighlighting(doc)
+ expect(segments.length).toBe(2)
+ expect(segments[0].text).toContain('Hello World')
+ expect(segments[1].text).toContain('Second line')
+ })
+ })
+
+ describe('extractForHighlighting() — Generic JSON fallback', () => {
+ it('should extract all string values from unrecognized JSON', () => {
+ const generic = JSON.stringify({
+ name: 'David',
+ description: 'A brave warrior',
+ stats: { strength: 10 }
+ })
+
+ const segments = extractForHighlighting(generic)
+ expect(segments.length).toBeGreaterThan(0)
+ const allText = segments.map(s => s.text).join(' ')
+ expect(allText).toContain('David')
+ expect(allText).toContain('brave warrior')
+ })
+
+ it('should return empty for JSON with no text', () => {
+ const noText = JSON.stringify({ count: 42, active: true })
+ const segments = extractForHighlighting(noText)
+ // No string values to extract
+ expect(segments).toBeDefined()
+ })
+ })
+
+ describe('extractForHighlighting() — HTML', () => {
+ it('should extract headings, paragraphs, and code', () => {
+ const html = 'Title
Body text here.
let x = 1
'
+
+ const segments = extractForHighlighting(html)
+ expect(segments.length).toBeGreaterThanOrEqual(3)
+
+ const headingSegment = segments.find(s => s.text === 'Title')
+ expect(headingSegment).toBeDefined()
+ expect(headingSegment?.contentCategory).toBe('heading')
+
+ const bodySegment = segments.find(s => s.text === 'Body text here.')
+ expect(bodySegment).toBeDefined()
+ expect(bodySegment?.contentCategory).toBe('prose')
+
+ const codeSegment = segments.find(s => s.text === 'let x = 1')
+ expect(codeSegment).toBeDefined()
+ expect(codeSegment?.contentCategory).toBe('code')
+ })
+
+ it('should skip script and style content', () => {
+ const html = 'Visible
Also visible
'
+
+ const segments = extractForHighlighting(html)
+ const allText = segments.map(s => s.text).join(' ')
+ expect(allText).toContain('Visible')
+ expect(allText).toContain('Also visible')
+ expect(allText).not.toContain('alert')
+ expect(allText).not.toContain('.foo')
+ })
+
+ it('should decode common HTML entities', () => {
+ const html = 'Tom & Jerry <3
'
+ const segments = extractForHighlighting(html)
+ expect(segments.length).toBeGreaterThan(0)
+ expect(segments[0].text).toContain('Tom & Jerry <3')
+ })
+
+ it('should handle nested heading tags', () => {
+ const html = 'Nested Heading
'
+ const segments = extractForHighlighting(html)
+ const heading = segments.find(s => s.text === 'Nested Heading')
+ expect(heading).toBeDefined()
+ expect(heading?.contentCategory).toBe('heading')
+ })
+ })
+
+ describe('extractForHighlighting() — Markdown', () => {
+ it('should extract headings', () => {
+ const md = '# Main Title\n\nSome body text\n\n## Subtitle'
+
+ const segments = extractForHighlighting(md)
+ const heading1 = segments.find(s => s.text === 'Main Title')
+ const heading2 = segments.find(s => s.text === 'Subtitle')
+ const body = segments.find(s => s.text === 'Some body text')
+
+ expect(heading1).toBeDefined()
+ expect(heading1?.contentCategory).toBe('heading')
+ expect(heading2).toBeDefined()
+ expect(heading2?.contentCategory).toBe('heading')
+ expect(body).toBeDefined()
+ expect(body?.contentCategory).toBe('prose')
+ })
+
+ it('should extract fenced code blocks', () => {
+ const md = '# Title\n\n```typescript\nconst x = 1\nconst y = 2\n```\n\nMore text'
+
+ const segments = extractForHighlighting(md)
+ const code = segments.find(s => s.contentCategory === 'code')
+ expect(code).toBeDefined()
+ expect(code?.text).toContain('const x = 1')
+ })
+
+ it('should extract indented code blocks', () => {
+ // Indented code alone doesn't trigger markdown detection, so pass explicit type
+ const md = 'Normal text\n\n code line 1\n code line 2\n\nMore text'
+
+ const segments = extractForHighlighting(md, 'markdown')
+ const code = segments.find(s => s.contentCategory === 'code')
+ expect(code).toBeDefined()
+ expect(code?.text).toContain('code line 1')
+ })
+ })
+
+ describe('extractForHighlighting() — Plaintext', () => {
+ it('should return single prose segment for plain text', () => {
+ const text = 'Just some plain text content'
+ const segments = extractForHighlighting(text)
+
+ expect(segments.length).toBe(1)
+ expect(segments[0].text).toBe(text)
+ expect(segments[0].contentCategory).toBe('prose')
+ })
+
+ it('should respect contentType override', () => {
+ // Even if content looks like HTML, plaintext hint should skip detection
+ const html = 'Title
'
+ const segments = extractForHighlighting(html, 'plaintext')
+
+ expect(segments.length).toBe(1)
+ expect(segments[0].text).toBe(html) // Raw HTML, not extracted
+ expect(segments[0].contentCategory).toBe('prose')
+ })
+ })
+})
diff --git a/tests/unit/utils/metadataIndex-text-indexing.test.ts b/tests/unit/utils/metadataIndex-text-indexing.test.ts
index d3b34244..47c158d4 100644
--- a/tests/unit/utils/metadataIndex-text-indexing.test.ts
+++ b/tests/unit/utils/metadataIndex-text-indexing.test.ts
@@ -215,4 +215,70 @@ describe('MetadataIndexManager Text Indexing (v7.7.0)', () => {
})
})
})
+
+ describe('extractTextContent() array handling (v7.9.0 fix)', () => {
+ let manager: MetadataIndexManager
+
+ beforeEach(() => {
+ const mockStorage = {
+ getMetadata: async () => null,
+ saveMetadata: async () => {},
+ getNoun: async () => null,
+ getNouns: async () => ({ items: [], total: 0, hasMore: false }),
+ getVerbsFromSource: async () => ({ items: [], total: 0, hasMore: false }),
+ } as any
+
+ manager = new MetadataIndexManager(mockStorage, {})
+ })
+
+ it('should skip numeric arrays (vectors/embeddings)', () => {
+ const result = manager.extractTextContent([0.1, 0.2, 0.3, 0.4, 0.5])
+ expect(result).toBe('')
+ })
+
+ it('should extract text from string arrays', () => {
+ const result = manager.extractTextContent(['hello', 'world', 'test'])
+ expect(result).toContain('hello')
+ expect(result).toContain('world')
+ expect(result).toContain('test')
+ })
+
+ it('should extract text from arrays of objects', () => {
+ const result = manager.extractTextContent([
+ { name: 'Alice', role: 'engineer' },
+ { name: 'Bob', role: 'designer' },
+ { name: 'Carol', role: 'manager' },
+ { name: 'Dave', role: 'analyst' },
+ { name: 'Eve', role: 'scientist' },
+ { name: 'Frank', role: 'developer' },
+ { name: 'Grace', role: 'architect' },
+ { name: 'Heidi', role: 'lead' },
+ { name: 'Ivan', role: 'intern' },
+ { name: 'Judy', role: 'director' },
+ { name: 'Karl', role: 'founder' },
+ ])
+ // Should NOT skip arrays of objects even with >10 elements
+ expect(result).toContain('Alice')
+ expect(result).toContain('Karl')
+ expect(result).toContain('engineer')
+ expect(result).toContain('founder')
+ })
+
+ it('should handle large numeric arrays (vectors)', () => {
+ const vector = Array.from({ length: 384 }, (_, i) => Math.sin(i))
+ const result = manager.extractTextContent(vector)
+ expect(result).toBe('')
+ })
+
+ it('should handle empty arrays', () => {
+ const result = manager.extractTextContent([])
+ expect(result).toBe('')
+ })
+
+ it('should handle mixed arrays starting with string', () => {
+ const result = manager.extractTextContent(['text', 42, true])
+ expect(result).toContain('text')
+ expect(result).toContain('42')
+ })
+ })
})