soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat: expand ContentCategory to universal 6-category set for highlight()

Browse source 
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:

- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation

Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).

Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
This commit is contained in:
David Snelling 2026-01-27 13:44:58 -08:00
parent 6c27f9f7fd
commit ff80b87a0a
6 changed files with 699 additions and 651 deletions
Download patch file Download diff file Expand all files Collapse all files

122
docs/api/README.md
View file

@ -1,9 +1,9 @@
# 🧠 Brainy v7.1.0 API Reference # 🧠 Brainy API Reference
> **Complete API documentation for Brainy v7.1.0** > **Complete API documentation for Brainy**
> Zero Configuration • Triple Intelligence • Git-Style Branching • Entity Versioning • Candle WASM Embeddings > Zero Configuration • Triple Intelligence • Git-Style Branching • Entity Versioning • Candle WASM Embeddings
**Updated:** 2026-01-06 for v7.1.0 **Updated:** 2026-01-06
**All APIs verified against actual code** **All APIs verified against actual code**
--- ---
@ -14,7 +14,7 @@
import { Brainy, NounType, VerbType } from '@soulcraft/brainy' import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
const brain = new Brainy() // Zero config! const brain = new Brainy() // Zero config!
await brain.init() // VFS auto-initialized in v5.1.0! await brain.init() // VFS auto-initialized!
// Add data (text auto-embeds!) // Add data (text auto-embeds!)
const id = await brain.add({ const id = await brain.add({
@ -30,12 +30,12 @@ const results = await brain.find({
connected: { from: id, depth: 2 } connected: { from: id, depth: 2 }
}) })
// Fork for safe experimentation (v5.0.0+) // Fork for safe experimentation
const experiment = await brain.fork('test-feature') const experiment = await brain.fork('test-feature')
await experiment.add({ data: 'test', type: NounType.Document }) await experiment.add({ data: 'test', type: NounType.Document })
await experiment.commit({ message: 'Add test data' }) await experiment.commit({ message: 'Add test data' })
// Entity versioning (v5.3.0+) // Entity versioning
await brain.versions.save(id, { tag: 'v1.0', description: 'Initial version' }) await brain.versions.save(id, { tag: 'v1.0', description: 'Initial version' })
await brain.update(id, { category: 'AI' }) await brain.update(id, { category: 'AI' })
await brain.versions.save(id, { tag: 'v2.0' }) await brain.versions.save(id, { tag: 'v2.0' })
@ -54,10 +54,10 @@ Typed connections between entities - building knowledge graphs.
### 🧠 Triple Intelligence ### 🧠 Triple Intelligence
Vector search + Graph traversal + Metadata filtering in one unified query. Vector search + Graph traversal + Metadata filtering in one unified query.
### 🌳 Git-Style Branching (v5.0.0+) ### 🌳 Git-Style Branching
Fork, experiment, and commit - Snowflake-style copy-on-write isolation. Fork, experiment, and commit - Snowflake-style copy-on-write isolation.
### 📜 Entity Versioning (v5.3.0+) ### 📜 Entity Versioning
Time-travel and history tracking for individual entities - Git-like version control with content-addressable storage. Time-travel and history tracking for individual entities - Git-like version control with content-addressable storage.
--- ---
@ -68,15 +68,15 @@ Time-travel and history tracking for individual entities - Git-like version cont
- [Search & Query](#search--query) - [Search & Query](#search--query)
- [Relationships](#relationships) - [Relationships](#relationships)
- [Batch Operations](#batch-operations) - [Batch Operations](#batch-operations)
- [Branch Management (v5.0+)](#branch-management-v50) - [Branch Management](#branch-management)
- [Entity Versioning (v5.3.0+)](#entity-versioning-v530) - [Entity Versioning](#entity-versioning)
- [Virtual Filesystem (VFS)](#virtual-filesystem-vfs) - [Virtual Filesystem (VFS)](#virtual-filesystem-vfs)
- [Neural API](#neural-api) - [Neural API](#neural-api)
- [Import & Export](#import--export) - [Import & Export](#import--export)
- [Configuration](#configuration) - [Configuration](#configuration)
- [Storage Adapters](#storage-adapters) - [Storage Adapters](#storage-adapters)
- [Utility Methods](#utility-methods) - [Utility Methods](#utility-methods)
- [Embedding & Analysis APIs (v7.1.0)](#embedding--analysis-apis-v710) - [Embedding & Analysis APIs](#embedding--analysis-apis)
- [Type System Reference](#type-system-reference) - [Type System Reference](#type-system-reference)
--- ---
@ -202,7 +202,7 @@ const results = await brain.find({
- `depth?`: `number` - Traversal depth - `depth?`: `number` - Traversal depth
- `limit?`: `number` - Max results (default: 10) - `limit?`: `number` - Max results (default: 10)
- `offset?`: `number` - Skip results - `offset?`: `number` - Skip results
- `searchMode?`: `'auto' | 'text' | 'semantic' | 'hybrid'` - Search strategy (v7.7.0): - `searchMode?`: `'auto' | 'text' | 'semantic' | 'hybrid'` - Search strategy:
- `'auto'` (default): Zero-config hybrid combining text + semantic search - `'auto'` (default): Zero-config hybrid combining text + semantic search
- `'text'`: Pure keyword/text matching - `'text'`: Pure keyword/text matching
- `'semantic'`: Pure vector similarity - `'semantic'`: Pure vector similarity
@ -213,7 +213,7 @@ const results = await brain.find({
--- ---
### Hybrid Search (v7.7.0) ### Hybrid Search
Brainy automatically combines text (keyword) and semantic (vector) search for optimal results. No configuration needed. Brainy automatically combines text (keyword) and semantic (vector) search for optimal results. No configuration needed.
@ -249,7 +249,7 @@ const customResults = await brain.find({
--- ---
### Match Visibility (v7.8.0) ### Match Visibility
Search results include detailed match information: Search results include detailed match information:
@ -270,7 +270,7 @@ results[0].matchSource // 'both' | 'text' | 'semantic'
--- ---
### `highlight(params)` → `Promise<Highlight[]>` ✨ *New v7.8.0, Enhanced v7.9.0* ### `highlight(params)` → `Promise<Highlight[]>` ✨
Zero-config highlighting for both exact matches AND semantic concepts. Zero-config highlighting for both exact matches AND semantic concepts.
Handles plain text, rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill), HTML, and Markdown automatically. Handles plain text, rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill), HTML, and Markdown automatically.
@ -287,24 +287,24 @@ const highlights = await brain.highlight({
// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' } // { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
// ] // ]
// Rich-text JSON (auto-detected, v7.9.0) // Rich-text JSON (auto-detected)
const highlights = await brain.highlight({ const highlights = await brain.highlight({
query: "david the warrior", query: "david the warrior",
text: JSON.stringify(tiptapDocument) // TipTap, Slate, Lexical, Draft.js, Quill text: JSON.stringify(tiptapDocument) // TipTap, Slate, Lexical, Draft.js, Quill
}) })
// Extracts text from nodes, annotates with contentCategory: // Extracts text from nodes, annotates with contentCategory:
// [ // [
// { text: "David", score: 1.0, matchType: 'text', contentCategory: 'heading' }, // { text: "David", score: 1.0, matchType: 'text', contentCategory: 'title' },
// { text: "fighter", score: 0.78, matchType: 'semantic', contentCategory: 'prose' } // { text: "fighter", score: 0.78, matchType: 'semantic', contentCategory: 'content' }
// ] // ]
// HTML input (auto-detected, v7.9.0) // HTML input (auto-detected)
const highlights = await brain.highlight({ const highlights = await brain.highlight({
query: "warrior", query: "warrior",
text: "<h1>David the Warrior</h1><p>A brave fighter.</p>" text: "<h1>David the Warrior</h1><p>A brave fighter.</p>"
}) })
// Custom extractor for proprietary formats (v7.9.0) // Custom extractor for proprietary formats
const highlights = await brain.highlight({ const highlights = await brain.highlight({
query: "function", query: "function",
text: sourceCode, text: sourceCode,
@ -317,17 +317,17 @@ const highlights = await brain.highlight({
- `text`: `string` - Text to highlight (plain text, JSON, HTML, or Markdown) - `text`: `string` - Text to highlight (plain text, JSON, HTML, or Markdown)
- `granularity?`: `'word' | 'phrase' | 'sentence'` - Highlight unit (default: 'word') - `granularity?`: `'word' | 'phrase' | 'sentence'` - Highlight unit (default: 'word')
- `threshold?`: `number` - Min similarity for semantic matches (default: 0.5) - `threshold?`: `number` - Min similarity for semantic matches (default: 0.5)
- `contentType?`: `ContentType` - Optional hint: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`. Skips auto-detection when provided. *(v7.9.0)* - `contentType?`: `ContentType` - Optional hint: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`. Skips auto-detection when provided.
- `contentExtractor?`: `(text: string) => ExtractedSegment[]` - Custom parser. Bypasses built-in detection entirely. *(v7.9.0)* - `contentExtractor?`: `(text: string) => ExtractedSegment[]` - Custom parser. Bypasses built-in detection entirely.
**Returns:** `Promise<Highlight[]>` **Returns:** `Promise<Highlight[]>`
- `text` - The matched text - `text` - The matched text
- `score` - Match score (1.0 for text matches, varies for semantic) - `score` - Match score (1.0 for text matches, varies for semantic)
- `position` - [start, end] indices in extracted text - `position` - [start, end] indices in extracted text
- `matchType` - `'text'` (exact) or `'semantic'` (concept) - `matchType` - `'text'` (exact) or `'semantic'` (concept)
- `contentCategory?` - `'prose' | 'heading' | 'code' | 'label'` — Role of the source text in the document. Present when input is structured. *(v7.9.0)* - `contentCategory?` - `'title' | 'annotation' | 'content' | 'value' | 'code' | 'structural'` — Role of the source text. Built-in extractors produce `'title'`, `'content'`, `'code'`. All 6 categories are available for custom parsers.
**Supported Rich-Text Formats (v7.9.0):** **Supported Rich-Text Formats:**
| Format | Detection | Text nodes | | Format | Detection | Text nodes |
|--------|-----------|------------| |--------|-----------|------------|
@ -339,7 +339,7 @@ const highlights = await brain.highlight({
| HTML | Tags like `<h1>`, `<p>`, `<code>` | Visible text content | | HTML | Tags like `<h1>`, `<p>`, `<code>` | Visible text content |
| Markdown | `#` headings, ` ``` ` code blocks | Stripped markup | | Markdown | `#` headings, ` ``` ` code blocks | Stripped markup |
**Timeout Protection (v7.9.0):** **Timeout Protection:**
Semantic matching has a 10-second timeout. If embedding takes too long (e.g., WASM stall), `highlight()` returns text-only matches instead of hanging. Semantic matching has a 10-second timeout. If embedding takes too long (e.g., WASM stall), `highlight()` returns text-only matches instead of hanging.
**UI Pattern:** **UI Pattern:**
@ -347,8 +347,9 @@ Semantic matching has a 10-second timeout. If embedding takes too long (e.g., WA
// Style differently based on match type and content category // Style differently based on match type and content category
highlights.forEach(h => { highlights.forEach(h => {
const style = h.matchType === 'text' ? 'font-weight: bold' : 'background: yellow' const style = h.matchType === 'text' ? 'font-weight: bold' : 'background: yellow'
if (h.contentCategory === 'heading') { /* render as heading highlight */ } if (h.contentCategory === 'title') { /* render as heading highlight */ }
if (h.contentCategory === 'code') { /* render with code styling */ } if (h.contentCategory === 'code') { /* render with code styling */ }
if (h.contentCategory === 'annotation') { /* render as comment/caption */ }
// Apply style from h.position[0] to h.position[1] // Apply style from h.position[0] to h.position[1]
}) })
``` ```
@ -495,9 +496,9 @@ const ids = await brain.relateMany({
--- ---
## Branch Management (v5.0+) ## Branch Management
**NEW in v5.0.0:** Git-style branching with Snowflake-style copy-on-write. Git-style branching with Snowflake-style copy-on-write.
### `fork(branch?, options?)` → `Promise<Brainy>` ### `fork(branch?, options?)` → `Promise<Brainy>`
@ -651,9 +652,9 @@ await snapshot.close()
--- ---
## Entity Versioning (v5.3.0+) ## Entity Versioning
**NEW in v5.3.0:** Git-style versioning for individual entities with content-addressable storage. Git-style versioning for individual entities with content-addressable storage.
### Overview ### Overview
@ -664,7 +665,7 @@ Entity Versioning provides time-travel and history tracking for individual entit
- **Branch-Isolated** - Versions isolated per branch - **Branch-Isolated** - Versions isolated per branch
- **Selective Auto-Versioning** - Optional augmentation for automatic version creation - **Selective Auto-Versioning** - Optional augmentation for automatic version creation
- **Production-Scale** - Designed for billions of entities - **Production-Scale** - Designed for billions of entities
- **VFS File Support (v6.3.2+)** - Full versioning for VFS files with actual blob content - **VFS File Support** - Full versioning for VFS files with actual blob content
--- ---
@ -1031,7 +1032,7 @@ versions.forEach(v => {
}) })
``` ```
#### VFS File Versioning (v6.3.2+) #### VFS File Versioning
```typescript ```typescript
// VFS files can be versioned with actual blob content // VFS files can be versioned with actual blob content
@ -1049,7 +1050,7 @@ await brain.vfs.writeFile('/docs/readme.md', 'Version 2 - updated content')
// Save version 2 // Save version 2
await brain.versions.save(stat.entityId, { tag: 'v2', description: 'Updated docs' }) await brain.versions.save(stat.entityId, { tag: 'v2', description: 'Updated docs' })
// Compare versions - content is DIFFERENT (fixed in v6.3.2) // Compare versions - content is DIFFERENT
const v1 = await brain.versions.getContent(stat.entityId, 1) const v1 = await brain.versions.getContent(stat.entityId, 1)
const v2 = await brain.versions.getContent(stat.entityId, 2) const v2 = await brain.versions.getContent(stat.entityId, 2)
console.log(v1.data !== v2.data) // true console.log(v1.data !== v2.data) // true
@ -1070,11 +1071,11 @@ console.log(content.toString()) // 'Version 1 content'
## Virtual Filesystem (VFS) ## Virtual Filesystem (VFS)
**Auto-initialized in v5.1.0!** Access via `brain.vfs` (property, not method). Access via `brain.vfs` (property, not method). Auto-initialized during `brain.init()`.
### Filtering VFS Entities ### Filtering VFS Entities
**NEW in v5.3.0:** All VFS entities (files/folders) have `metadata.isVFSEntity: true` set automatically. All VFS entities (files/folders) have `metadata.isVFSEntity: true` set automatically.
Use this to filter VFS entities from semantic search results: Use this to filter VFS entities from semantic search results:
@ -1473,7 +1474,7 @@ await brain.import('https://api.example.com/data.json')
**Returns:** `Promise<ImportResult>` - Import statistics **Returns:** `Promise<ImportResult>` - Import statistics
**Note:** Import always uses the current branch (v5.1.0 verified). **Note:** Import always uses the current branch.
**[📖 Complete Import Guide →](../guides/import-anything.md)** **[📖 Complete Import Guide →](../guides/import-anything.md)**
@ -1519,7 +1520,7 @@ const brain = new Brainy({
M: 16, // Connections per layer M: 16, // Connections per layer
efConstruction: 200, // Construction quality efConstruction: 200, // Construction quality
efSearch: 100, // Search quality efSearch: 100, // Search quality
typeAware: true // Enable type-aware indexing (v4.0+) typeAware: true // Enable type-aware indexing
}, },
// Model configuration (embedded in WASM - zero config needed) // Model configuration (embedded in WASM - zero config needed)
@ -1534,14 +1535,14 @@ const brain = new Brainy({
} }
}) })
await brain.init() // Required! VFS auto-initialized in v5.1.0 await brain.init() // Required! VFS auto-initialized
``` ```
--- ---
## Storage Adapters ## Storage Adapters
All 7 storage adapters support **copy-on-write branching** (v5.0+). All 7 storage adapters support **copy-on-write branching**.
### Memory (Default) ### Memory (Default)
@ -1709,7 +1710,7 @@ const count = await brain.getVerbCount()
--- ---
### `embed(data)` → `Promise<number[]>` ✨ *Enhanced v7.1.0* ### `embed(data)` → `Promise<number[]>` ✨
Generate embedding vector from text or data. Generate embedding vector from text or data.
@ -1721,7 +1722,7 @@ console.log(vector.length) // 384
--- ---
### `embedBatch(texts)` → `Promise<number[][]>` ✨ *New v7.1.0, Optimized v7.9.0* ### `embedBatch(texts)` → `Promise<number[][]>` ✨
Batch embed multiple texts using native WASM batch API (single forward pass). Batch embed multiple texts using native WASM batch API (single forward pass).
@ -1735,11 +1736,11 @@ console.log(embeddings.length) // 3
console.log(embeddings[0].length) // 384 console.log(embeddings[0].length) // 384
``` ```
> **v7.9.0**: Now uses the WASM engine's native `embed_batch()` for a single model forward pass instead of N individual calls. This is the same batch API used internally by `highlight()`. > Uses the WASM engine's native `embed_batch()` for a single model forward pass instead of N individual calls. This is the same batch API used internally by `highlight()`.
--- ---
### `similarity(textA, textB)` → `Promise<number>` ✨ *New v7.1.0* ### `similarity(textA, textB)` → `Promise<number>` ✨
Calculate semantic similarity between two texts. Calculate semantic similarity between two texts.
@ -1755,7 +1756,7 @@ console.log(score) // ~0.85 (high semantic similarity)
--- ---
### `neighbors(entityId, options?)` → `Promise<string[]>` ✨ *New v7.1.0* ### `neighbors(entityId, options?)` → `Promise<string[]>` ✨
Get graph neighbors of an entity. Get graph neighbors of an entity.
@ -1784,7 +1785,7 @@ const extended = await brain.neighbors(entityId, {
--- ---
### `findDuplicates(options?)` → `Promise<DuplicateResult[]>` ✨ *New v7.1.0* ### `findDuplicates(options?)` → `Promise<DuplicateResult[]>` ✨
Find semantic duplicates in the database. Find semantic duplicates in the database.
@ -1814,7 +1815,7 @@ const personDupes = await brain.findDuplicates({
--- ---
### `indexStats()` → `Promise<IndexStats>` ✨ *New v7.1.0* ### `indexStats()` → `Promise<IndexStats>` ✨
Get comprehensive index statistics. Get comprehensive index statistics.
@ -1839,7 +1840,7 @@ console.log(`Fields: ${stats.metadataFields.join(', ')}`)
--- ---
### `cluster(options?)` → `Promise<ClusterResult[]>` ✨ *New v7.1.0* ### `cluster(options?)` → `Promise<ClusterResult[]>` ✨
Cluster entities by semantic similarity. Cluster entities by semantic similarity.
@ -1893,10 +1894,10 @@ console.log(stats.cacheHitRate)
```typescript ```typescript
const brain = new Brainy(config) const brain = new Brainy(config)
await brain.init() // Required! VFS auto-initialized here (v5.1.0) await brain.init() // Required! VFS auto-initialized here
``` ```
**v5.1.0 Change:** VFS is now auto-initialized during `brain.init()` - no separate `vfs.init()` needed! VFS is auto-initialized during `brain.init()` - no separate `vfs.init()` needed!
--- ---
@ -1984,7 +1985,7 @@ const results = await brain.find({
--- ---
### Git-Style Workflow (v5.0+) ### Git-Style Workflow
```typescript ```typescript
// Fork for experimentation // Fork for experimentation
@ -2033,7 +2034,7 @@ const tree = await brain.vfs.getTreeStructure('/projects', {
## Type System Reference ## Type System Reference
**NEW in v5.5.0:** Stage 3 CANONICAL taxonomy with 169 types (42 nouns + 127 verbs) Stage 3 CANONICAL taxonomy with 169 types (42 nouns + 127 verbs)
### Noun Types (42) ### Noun Types (42)
@ -2107,8 +2108,7 @@ For the full taxonomy with all 169 types and their descriptions, see:
- **[Stage 3 CANONICAL Taxonomy](../STAGE3-CANONICAL-TAXONOMY.md)** - Complete list with categories - **[Stage 3 CANONICAL Taxonomy](../STAGE3-CANONICAL-TAXONOMY.md)** - Complete list with categories
- **[Noun-Verb Taxonomy Architecture](../architecture/noun-verb-taxonomy.md)** - Design rationale - **[Noun-Verb Taxonomy Architecture](../architecture/noun-verb-taxonomy.md)** - Design rationale
### Migration from v5.4.0 ### Migration from
**Breaking Changes:** **Breaking Changes:**
- `NounType.Content` removed → Use `Document`, `Message`, or `InformationContent` - `NounType.Content` removed → Use `Document`, `Message`, or `InformationContent`
- `NounType.User` removed → Use `Person` or `Agent` - `NounType.User` removed → Use `Person` or `Agent`
@ -2120,34 +2120,24 @@ For the full taxonomy with all 169 types and their descriptions, see:
--- ---
## What's New in v5.0 ## Key Features
### v5.3.0 (Latest)
- ✅ **Entity Versioning** - Git-style versioning for individual entities - ✅ **Entity Versioning** - Git-style versioning for individual entities
- ✅ **Content-Addressable Storage** - SHA-256 deduplication for versions - ✅ **Content-Addressable Storage** - SHA-256 deduplication for versions
- ✅ **Auto-Versioning Augmentation** - Automatic version creation on updates - ✅ **Auto-Versioning Augmentation** - Automatic version creation on updates
- ✅ **Branch-Isolated Versions** - Versions isolated per branch - ✅ **Branch-Isolated Versions** - Versions isolated per branch
- ✅ **VFS Entity Filtering** - All VFS entities now have `isVFSEntity: true` flag - ✅ **VFS Entity Filtering** - All VFS entities now have `isVFSEntity: true` flag
- ✅ **CRITICAL FIX:** commit() now updates branch refs correctly (brainy.ts:2385)
- ✅ **CRITICAL FIX:** VFS entities now properly flagged for filtering
### v5.1.0
- ✅ **VFS Auto-Initialization** - No more separate `vfs.init()` calls - ✅ **VFS Auto-Initialization** - No more separate `vfs.init()` calls
- ✅ **VFS Property Access** - Use `brain.vfs.method()` instead of `brain.vfs().method()` - ✅ **VFS Property Access** - Use `brain.vfs.method()` instead of `brain.vfs().method()`
- ✅ **Complete COW Support** - All 20 TypeAware methods use COW helpers - ✅ **Complete COW Support** - All 20 TypeAware methods use COW helpers
- ✅ **Verified Import/Export** - Work correctly with current branch - ✅ **Verified Import/Export** - Work correctly with current branch
### v5.0.0
- ✅ **Instant Fork** - Snowflake-style copy-on-write (<100ms fork time) - ✅ **Instant Fork** - Snowflake-style copy-on-write (<100ms fork time)
- ✅ **Git-Style Branching** - fork, commit, checkout, listBranches - ✅ **Git-Style Branching** - fork, commit, checkout, listBranches
- ✅ **Full Branch Isolation** - Parent and fork fully isolated - ✅ **Full Branch Isolation** - Parent and fork fully isolated
- ✅ **Read-Through Inheritance** - Forks see parent + own data - ✅ **Read-Through Inheritance** - Forks see parent + own data
- ✅ **Universal Storage Support** - All 7 adapters support branching - ✅ **Universal Storage Support** - All 7 adapters support branching
**[📖 Complete v5.0 Changes →](../../.strategy/v5.1.0-CHANGES.md)** **[📖 Complete Changes →](../../.strategy/v5.1.0-CHANGES.md)**
--- ---
@ -2175,5 +2165,5 @@ For the full taxonomy with all 169 types and their descriptions, see:
--- ---
*Brainy v5.0+ - The Knowledge Operating System* *Brainy - The Knowledge Operating System*
*From prototype to planet-scale • Zero configuration • Triple Intelligence™ • Git-Style Branching* *From prototype to planet-scale • Zero configuration • Triple Intelligence™ • Git-Style Branching*

354
src/brainy.ts
View file

File diff suppressed because it is too large Load diff

69
src/types/brainy.types.ts
View file

@ -83,11 +83,12 @@ export interface Result<T = any> {
// Score transparency // Score transparency
explanation?: ScoreExplanation explanation?: ScoreExplanation
// Match visibility (v7.8.0) - shows what matched in hybrid search // Match visibility - shows what matched in hybrid search
textMatches?: string[] // Query words found in entity (e.g., ["david", "warrior"]) textMatches?: string[] // Query words found in entity (e.g., ["david", "warrior"])
textScore?: number // Normalized text match score (0-1) textScore?: number // Normalized text match score (0-1)
semanticScore?: number // Semantic similarity score (0-1) semanticScore?: number // Semantic similarity score (0-1)
matchSource?: 'text' | 'semantic' | 'both' // Where this result came from matchSource?: 'text' | 'semantic' | 'both' // Where this result came from
rrfScore?: number // Raw RRF fusion score (for advanced ranking analysis)
} }
/** /**
@ -188,7 +189,7 @@ export interface FindParams<T = any> {
offset?: number // Skip N results offset?: number // Skip N results
cursor?: string // Cursor-based pagination cursor?: string // Cursor-based pagination
// Sorting (v4.5.4) // Sorting
orderBy?: string // Field to sort by (e.g., 'createdAt', 'title', 'metadata.priority') orderBy?: string // Field to sort by (e.g., 'createdAt', 'title', 'metadata.priority')
order?: 'asc' | 'desc' // Sort direction: 'asc' (default) or 'desc' order?: 'asc' | 'desc' // Sort direction: 'asc' (default) or 'desc'
@ -196,10 +197,10 @@ export interface FindParams<T = any> {
mode?: SearchMode // Search strategy mode?: SearchMode // Search strategy
explain?: boolean // Return scoring explanation explain?: boolean // Return scoring explanation
includeRelations?: boolean // Include entity relationships includeRelations?: boolean // Include entity relationships
excludeVFS?: boolean // v4.7.0: Exclude VFS entities from results (default: false - VFS included) excludeVFS?: boolean // Exclude VFS entities from results (default: false - VFS included)
service?: string // Multi-tenancy filter service?: string // Multi-tenancy filter
// Hybrid search options (v7.7.0) // Hybrid search options
searchMode?: 'auto' | 'text' | 'semantic' | 'vector' | 'hybrid' // Search strategy: auto (default), text-only, semantic/vector-only, or explicit hybrid searchMode?: 'auto' | 'text' | 'semantic' | 'vector' | 'hybrid' // Search strategy: auto (default), text-only, semantic/vector-only, or explicit hybrid
hybridAlpha?: number // Weight between text (0.0) and semantic (1.0) search, default: auto-detected by query length hybridAlpha?: number // Weight between text (0.0) and semantic (1.0) search, default: auto-detected by query length
@ -252,7 +253,7 @@ export interface SimilarParams<T = any> {
type?: NounType | NounType[] // Restrict to types type?: NounType | NounType[] // Restrict to types
where?: Partial<T> // Additional filters where?: Partial<T> // Additional filters
service?: string // Multi-tenancy service?: string // Multi-tenancy
excludeVFS?: boolean // v4.7.0: Exclude VFS entities (default: false - VFS included) excludeVFS?: boolean // Exclude VFS entities (default: false - VFS included)
} }
/** /**
@ -289,8 +290,8 @@ export interface SimilarParams<T = any> {
* }) * })
* ``` * ```
* *
* @since v4.1.3 - Fixed bug where calling without parameters returned empty array * Fixed bug where calling without parameters returned empty array
* @since v4.1.3 - Added string ID shorthand syntax * Added string ID shorthand syntax
*/ */
export interface GetRelationsParams { export interface GetRelationsParams {
/** /**
@ -376,7 +377,7 @@ export interface DeleteManyParams {
where?: any // Delete by metadata where?: any // Delete by metadata
limit?: number // Max to delete (safety) limit?: number // Max to delete (safety)
onProgress?: (done: number, total: number) => void onProgress?: (done: number, total: number) => void
continueOnError?: boolean // v6.2.0: Continue processing if a delete fails continueOnError?: boolean // Continue processing if a delete fails
} }
/** /**
@ -403,7 +404,7 @@ export interface BatchResult<T = any> {
duration: number // Time taken in ms duration: number // Time taken in ms
} }
// ============= Import Progress (v4.5.0) ============= // ============= Import Progress =============
/** /**
* Import stage enumeration * Import stage enumeration
@ -435,7 +436,6 @@ export type ImportStatus =
* - Time estimates * - Time estimates
* - Performance metrics * - Performance metrics
* *
* @since v4.5.0
*/ */
export interface ImportProgress { export interface ImportProgress {
// Overall Progress // Overall Progress
@ -540,7 +540,7 @@ export interface ImportResult {
/** /**
* Options for brain.get() entity retrieval * Options for brain.get() entity retrieval
* *
* **Performance Optimization (v5.11.1)**: * **Performance Optimization**:
* By default, brain.get() loads ONLY metadata (not vectors), resulting in: * By default, brain.get() loads ONLY metadata (not vectors), resulting in:
* - **76-81% faster** reads (10ms vs 43ms for metadata-only) * - **76-81% faster** reads (10ms vs 43ms for metadata-only)
* - **95% less bandwidth** (300 bytes vs 6KB per entity) * - **95% less bandwidth** (300 bytes vs 6KB per entity)
@ -573,7 +573,6 @@ export interface ImportResult {
* await vfs.readFile('/file.txt') // 53ms → 10ms (81% faster) * await vfs.readFile('/file.txt') // 53ms → 10ms (81% faster)
* ``` * ```
* *
* @since v5.11.1
*/ */
export interface GetOptions { export interface GetOptions {
/** /**
@ -635,7 +634,7 @@ export type AggregateMetric =
// ============= Configuration ============= // ============= Configuration =============
/** /**
* Integration Hub configuration (v7.4.0) * Integration Hub configuration
* *
* Enables external tool integrations: Excel, Power BI, Google Sheets, etc. * Enables external tool integrations: Excel, Power BI, Google Sheets, etc.
* Works in all environments with zero external dependencies. * Works in all environments with zero external dependencies.
@ -722,18 +721,18 @@ export interface BrainyConfig {
batchWrites?: boolean // Enable write batching for better performance batchWrites?: boolean // Enable write batching for better performance
maxConcurrentOperations?: number // Limit concurrent file operations maxConcurrentOperations?: number // Limit concurrent file operations
// HNSW persistence mode (v6.2.8) // HNSW persistence mode
// Controls when HNSW graph connections are persisted to storage // Controls when HNSW graph connections are persisted to storage
// - 'immediate': Persist on every add (slow but durable, default for filesystem) // - 'immediate': Persist on every add (slow but durable, default for filesystem)
// - 'deferred': Persist only on flush/close (fast, default for cloud storage) // - 'deferred': Persist only on flush/close (fast, default for cloud storage)
// Cloud storage (GCS/S3/R2/Azure) should use 'deferred' for 30-50× faster adds // Cloud storage (GCS/S3/R2/Azure) should use 'deferred' for 30-50× faster adds
hnswPersistMode?: 'immediate' | 'deferred' hnswPersistMode?: 'immediate' | 'deferred'
// Memory management options (v5.11.0) // Memory management options
maxQueryLimit?: number // Override auto-detected query result limit (max: 100000) maxQueryLimit?: number // Override auto-detected query result limit (max: 100000)
reservedQueryMemory?: number // Memory reserved for queries in bytes (e.g., 1073741824 = 1GB) reservedQueryMemory?: number // Memory reserved for queries in bytes (e.g., 1073741824 = 1GB)
// Embedding initialization (v7.1.2) // Embedding initialization
// Controls when the WASM embedding engine is initialized // Controls when the WASM embedding engine is initialized
// - false (default): Lazy initialization on first embed() call // - false (default): Lazy initialization on first embed() call
// - true: Eager initialization during brain.init() // - true: Eager initialization during brain.init()
@ -745,7 +744,7 @@ export interface BrainyConfig {
verbose?: boolean // Enable verbose logging verbose?: boolean // Enable verbose logging
silent?: boolean // Suppress all logging output silent?: boolean // Suppress all logging output
// Integration Hub (v7.4.0) // Integration Hub
// Enable external tool integrations: Excel, Power BI, Google Sheets, etc. // Enable external tool integrations: Excel, Power BI, Google Sheets, etc.
// - true: Enable all integrations with default paths // - true: Enable all integrations with default paths
// - false/undefined: Disable integrations (default) // - false/undefined: Disable integrations (default)
@ -789,54 +788,55 @@ export interface NeuralAnomalyParams {
returnScores?: boolean // Return anomaly scores returnScores?: boolean // Return anomaly scores
} }
// ============= Content Extraction Types (v7.9.0) ============= // ============= Content Extraction Types =============
/** /**
* Detected content type for smart text extraction * Detected content type for smart text extraction
* *
* @since v7.9.0
*/ */
export type ContentType = 'plaintext' | 'richtext-json' | 'html' | 'markdown' export type ContentType = 'plaintext' | 'richtext-json' | 'html' | 'markdown'
/** /**
* Content category for extracted segments * Content category for extracted segments
* *
* Allows UI to apply different styling based on content role: * Universal categories that work across documents, code, and UI:
* - 'prose': Regular text content (paragraphs, list items) * - 'title': Names the subject — headings, identifiers, labels, JSON keys
* - 'heading': Heading/title text (h1-h6) * - 'annotation': Human explanation — comments, docstrings, captions, alt text
* - 'code': Code blocks or inline code * - 'content': Body substance — paragraphs, list items, flowing text
* - 'label': Labels, captions, or metadata-like text * - 'value': Data literals — strings, numbers, form values, error messages
* - 'code': Unparsed code blocks (custom parsers decompose into above)
* - 'structural': Boilerplate — keywords, operators, punctuation, formatting
* *
* @since v7.9.0 * Built-in extractors produce: 'title', 'content', 'code'.
* All 6 categories are available for custom parsers.
*/ */
export type ContentCategory = 'prose' | 'heading' | 'code' | 'label' export type ContentCategory = 'title' | 'annotation' | 'content' | 'value' | 'code' | 'structural'
/** /**
* A segment of extracted text with its content category * A segment of extracted text with its content category
* *
* @since v7.9.0
*/ */
export interface ExtractedSegment { export interface ExtractedSegment {
/** The extracted text content */ /** The extracted text content */
text: string text: string
/** What role this text plays in the document */ /** What role this text plays: 'title' | 'annotation' | 'content' | 'value' | 'code' | 'structural' */
contentCategory: ContentCategory contentCategory: ContentCategory
} }
// ============= Semantic Highlighting Types (v7.8.0) ============= // ============= Semantic Highlighting Types =============
/** /**
* Parameters for hybrid highlighting (v7.8.0, enhanced v7.9.0) * Parameters for hybrid highlighting
* *
* Zero-config highlighting that returns both text (exact) and semantic (concept) matches. * Zero-config highlighting that returns both text (exact) and semantic (concept) matches.
* Perfect for UI highlighting at different levels. * Perfect for UI highlighting at different levels.
* *
* v7.9.0: Added contentType hint and contentExtractor callback for structured text * Added contentType hint and contentExtractor callback for structured text
* (rich-text JSON, HTML, Markdown). Auto-detects format when not specified. * (rich-text JSON, HTML, Markdown). Auto-detects format when not specified.
* *
* @example * @example
* ```typescript * ```typescript
* // Plain text (unchanged from v7.8.0) * // Plain text
* const highlights = await brain.highlight({ * const highlights = await brain.highlight({
* query: "david the warrior", * query: "david the warrior",
* text: "David Smith is a brave fighter who battles dragons" * text: "David Smith is a brave fighter who battles dragons"
@ -872,7 +872,6 @@ export interface HighlightParams {
/** /**
* Optional content type hint to skip auto-detection. * Optional content type hint to skip auto-detection.
* When omitted, the content type is detected from the text content. * When omitted, the content type is detected from the text content.
* @since v7.9.0
*/ */
contentType?: ContentType contentType?: ContentType
@ -880,7 +879,6 @@ export interface HighlightParams {
* Optional custom content extractor function. * Optional custom content extractor function.
* When provided, bypasses built-in detection and extraction entirely. * When provided, bypasses built-in detection and extraction entirely.
* Use this to plug in custom parsers (tree-sitter, Monaco, proprietary formats). * Use this to plug in custom parsers (tree-sitter, Monaco, proprietary formats).
* @since v7.9.0
*/ */
contentExtractor?: (text: string) => ExtractedSegment[] contentExtractor?: (text: string) => ExtractedSegment[]
} }
@ -906,9 +904,8 @@ export interface Highlight {
matchType: 'text' | 'semantic' matchType: 'text' | 'semantic'
/** /**
* Content category of the source segment (heading, code, prose, label). * Content category of the source segment: 'title' | 'annotation' | 'content' | 'value' | 'code' | 'structural'.
* Present when the input text was structured (JSON, HTML, Markdown). * Present when the input text was structured (JSON, HTML, Markdown).
* @since v7.9.0
*/ */
contentCategory?: ContentCategory contentCategory?: ContentCategory
} }

73
src/utils/contentExtractor.ts
View file

@ -1,8 +1,8 @@
/** /**
* Content Extractor (v7.9.0) * Content Extractor
* *
* Detects content type (plaintext, rich-text JSON, HTML, Markdown) and extracts * Detects content type (plaintext, rich-text JSON, HTML, Markdown) and extracts
* text segments with content categories (prose, heading, code, label). * text segments with content categories (title, content, code, etc.).
* *
* Supports common rich-text editor formats: * Supports common rich-text editor formats:
* - TipTap / ProseMirror: { type: 'doc', content: [...] } * - TipTap / ProseMirror: { type: 'doc', content: [...] }
@ -12,7 +12,7 @@
* - Quill Delta: { ops: [{ insert }] } * - Quill Delta: { ops: [{ insert }] }
* *
* Falls back gracefully: structured text that doesn't match known patterns * Falls back gracefully: structured text that doesn't match known patterns
* is extracted as plain prose via recursive text collection. * is extracted as plain content via recursive text collection.
*/ */
import type { ContentType, ContentCategory, ExtractedSegment } from '../types/brainy.types.js' import type { ContentType, ContentCategory, ExtractedSegment } from '../types/brainy.types.js'
@ -65,7 +65,7 @@ export function extractForHighlighting(
return extractFromMarkdown(text) return extractFromMarkdown(text)
case 'plaintext': case 'plaintext':
default: default:
return [{ text, contentCategory: 'prose' }] return [{ text, contentCategory: 'content' }]
} }
} }
@ -81,7 +81,7 @@ function extractFromJson(text: string): ExtractedSegment[] {
parsed = JSON.parse(text.trim()) parsed = JSON.parse(text.trim())
} catch { } catch {
// Invalid JSON — treat as plain text // Invalid JSON — treat as plain text
return [{ text, contentCategory: 'prose' }] return [{ text, contentCategory: 'content' }]
} }
const segments = walkRichTextNodes(parsed) const segments = walkRichTextNodes(parsed)
@ -94,7 +94,7 @@ function extractFromJson(text: string): ExtractedSegment[] {
// Fallback: collect all string values from the JSON // Fallback: collect all string values from the JSON
const fallbackText = extractTextFromJsonValue(parsed) const fallbackText = extractTextFromJsonValue(parsed)
if (fallbackText.trim()) { if (fallbackText.trim()) {
return [{ text: fallbackText, contentCategory: 'prose' }] return [{ text: fallbackText, contentCategory: 'content' }]
} }
return [] return []
@ -121,13 +121,13 @@ function walkRichTextNodes(node: any): ExtractedSegment[] {
// Leaf: text content (TipTap/ProseMirror/Lexical/Slate) // Leaf: text content (TipTap/ProseMirror/Lexical/Slate)
if (typeof node.text === 'string' && node.text.length > 0) { if (typeof node.text === 'string' && node.text.length > 0) {
segments.push({ text: node.text, contentCategory: 'prose' }) segments.push({ text: node.text, contentCategory: 'content' })
return segments return segments
} }
// Leaf: Quill Delta insert // Leaf: Quill Delta insert
if (typeof node.insert === 'string' && node.insert.length > 0) { if (typeof node.insert === 'string' && node.insert.length > 0) {
segments.push({ text: node.insert, contentCategory: 'prose' }) segments.push({ text: node.insert, contentCategory: 'content' })
return segments return segments
} }
@ -146,8 +146,8 @@ function walkRichTextNodes(node: any): ExtractedSegment[] {
if (Array.isArray(children)) { if (Array.isArray(children)) {
for (const child of children) { for (const child of children) {
const childSegments = walkRichTextNodes(child) const childSegments = walkRichTextNodes(child)
// Apply parent's category if it's more specific than 'prose' // Apply parent's category if it's more specific than 'content'
if (category !== 'prose') { if (category !== 'content') {
childSegments.forEach(s => { s.contentCategory = category }) childSegments.forEach(s => { s.contentCategory = category })
} }
segments.push(...childSegments) segments.push(...childSegments)
@ -166,11 +166,11 @@ function walkRichTextNodes(node: any): ExtractedSegment[] {
* Categorize a node type string into a ContentCategory * Categorize a node type string into a ContentCategory
*/ */
function categorizeNodeType(type?: string): ContentCategory { function categorizeNodeType(type?: string): ContentCategory {
if (!type) return 'prose' if (!type) return 'content'
const t = type.toLowerCase() const t = type.toLowerCase()
if (t === 'heading' || /^h[1-6]$/.test(t)) return 'heading' if (t === 'heading' || /^h[1-6]$/.test(t)) return 'title'
if (t === 'code' || t === 'codeblock' || t === 'code_block') return 'code' if (t === 'code' || t === 'codeblock' || t === 'code_block') return 'code'
return 'prose' return 'content'
} }
/** /**
@ -201,7 +201,7 @@ function extractTextFromJsonValue(value: any): string {
function extractFromHtml(html: string): ExtractedSegment[] { function extractFromHtml(html: string): ExtractedSegment[] {
const segments: ExtractedSegment[] = [] const segments: ExtractedSegment[] = []
let current = '' let current = ''
let currentCategory: ContentCategory = 'prose' let currentCategory: ContentCategory = 'content'
let i = 0 let i = 0
let insideTag = false let insideTag = false
let tagName = '' let tagName = ''
@ -304,10 +304,10 @@ function extractFromHtml(html: string): ExtractedSegment[] {
function getCategoryFromTagStack(stack: string[]): ContentCategory { function getCategoryFromTagStack(stack: string[]): ContentCategory {
for (let i = stack.length - 1; i >= 0; i--) { for (let i = stack.length - 1; i >= 0; i--) {
const tag = stack[i] const tag = stack[i]
if (/^h[1-6]$/.test(tag)) return 'heading' if (/^h[1-6]$/.test(tag)) return 'title'
if (tag === 'code' || tag === 'pre') return 'code' if (tag === 'code' || tag === 'pre') return 'code'
} }
return 'prose' return 'content'
} }
/** /**
@ -328,6 +328,41 @@ function decodeHtmlEntity(entity: string): string {
// ============= Markdown Extraction ============= // ============= Markdown Extraction =============
/**
* Split a prose line into alternating content/code segments based on
* backtick-delimited inline code spans. Lines without backticks return
* a single 'content' segment.
*/
function splitInlineCode(line: string): ExtractedSegment[] {
const segments: ExtractedSegment[] = []
const pattern = /`([^`]+)`/g
let lastIndex = 0
let match: RegExpExecArray | null
while ((match = pattern.exec(line)) !== null) {
// Text before the backtick span
if (match.index > lastIndex) {
const before = line.substring(lastIndex, match.index).trim()
if (before.length > 0) {
segments.push({ text: before, contentCategory: 'content' })
}
}
// The code span (without backticks)
segments.push({ text: match[1], contentCategory: 'code' })
lastIndex = match.index + match[0].length
}
// Remaining text after last backtick span (or entire line if no backticks)
if (lastIndex < line.length) {
const remaining = line.substring(lastIndex).trim()
if (remaining.length > 0) {
segments.push({ text: remaining, contentCategory: 'content' })
}
}
return segments
}
/** /**
* Extract text from Markdown with category detection * Extract text from Markdown with category detection
*/ */
@ -381,14 +416,14 @@ function extractFromMarkdown(text: string): ExtractedSegment[] {
// Heading (# style) // Heading (# style)
const headingMatch = line.match(/^#{1,6}\s+(.+)$/) const headingMatch = line.match(/^#{1,6}\s+(.+)$/)
if (headingMatch) { if (headingMatch) {
segments.push({ text: headingMatch[1].trim(), contentCategory: 'heading' }) segments.push({ text: headingMatch[1].trim(), contentCategory: 'title' })
i++ i++
continue continue
} }
// Regular prose line // Regular content line — split out inline code spans
if (line.trim().length > 0) { if (line.trim().length > 0) {
segments.push({ text: line.trim(), contentCategory: 'prose' }) segments.push(...splitInlineCode(line.trim()))
} }
i++ i++

21
tests/unit/brainy/hybrid-search.test.ts
View file

@ -769,7 +769,7 @@ describe('Hybrid Search (v7.7.0)', () => {
expect(warriorMatch).toBeDefined() expect(warriorMatch).toBeDefined()
expect(warriorMatch?.matchType).toBe('text') expect(warriorMatch?.matchType).toBe('text')
// Should annotate heading content // Should annotate heading content
expect(warriorMatch?.contentCategory).toBe('heading') expect(warriorMatch?.contentCategory).toBe('title')
}) })
it('should extract text from Slate.js JSON', async () => { it('should extract text from Slate.js JSON', async () => {
@ -799,7 +799,7 @@ describe('Hybrid Search (v7.7.0)', () => {
const introMatch = highlights.find(h => h.text === 'Introduction') const introMatch = highlights.find(h => h.text === 'Introduction')
expect(introMatch).toBeDefined() expect(introMatch).toBeDefined()
expect(introMatch?.matchType).toBe('text') expect(introMatch?.matchType).toBe('text')
expect(introMatch?.contentCategory).toBe('heading') expect(introMatch?.contentCategory).toBe('title')
}) })
it('should extract text from Quill Delta JSON', async () => { it('should extract text from Quill Delta JSON', async () => {
@ -856,12 +856,12 @@ describe('Hybrid Search (v7.7.0)', () => {
// Should find "Warrior" from heading // Should find "Warrior" from heading
const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior') const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
expect(warriorMatch).toBeDefined() expect(warriorMatch).toBeDefined()
expect(warriorMatch?.contentCategory).toBe('heading') expect(warriorMatch?.contentCategory).toBe('title')
// Should find "fighter" from paragraph // Should find "fighter" from paragraph
const fighterMatch = highlights.find(h => h.text.toLowerCase() === 'fighter') const fighterMatch = highlights.find(h => h.text.toLowerCase() === 'fighter')
expect(fighterMatch).toBeDefined() expect(fighterMatch).toBeDefined()
expect(fighterMatch?.contentCategory).toBe('prose') expect(fighterMatch?.contentCategory).toBe('content')
}) })
it('should extract text from Markdown with headings and code', async () => { it('should extract text from Markdown with headings and code', async () => {
@ -876,8 +876,8 @@ describe('Hybrid Search (v7.7.0)', () => {
expect(highlights.length).toBeGreaterThan(0) expect(highlights.length).toBeGreaterThan(0)
// Should find text from heading // Should find text from heading
const headingMatch = highlights.find(h => h.text === 'Guide' && h.contentCategory === 'heading') const headingMatch = highlights.find(h => h.text === 'Guide' && h.contentCategory === 'title')
|| highlights.find(h => h.text === 'Warrior' && h.contentCategory === 'heading') || highlights.find(h => h.text === 'Warrior' && h.contentCategory === 'title')
expect(headingMatch).toBeDefined() expect(headingMatch).toBeDefined()
}) })
@ -896,8 +896,8 @@ describe('Hybrid Search (v7.7.0)', () => {
expect(warriorMatch).toBeDefined() expect(warriorMatch).toBeDefined()
expect(warriorMatch?.matchType).toBe('text') expect(warriorMatch?.matchType).toBe('text')
expect(warriorMatch?.score).toBe(1.0) expect(warriorMatch?.score).toBe(1.0)
// Plain text gets 'prose' category // Plain text gets 'content' category
expect(warriorMatch?.contentCategory).toBe('prose') expect(warriorMatch?.contentCategory).toBe('content')
}) })
it('should use contentType hint to skip auto-detection', async () => { it('should use contentType hint to skip auto-detection', async () => {
@ -920,7 +920,7 @@ describe('Hybrid Search (v7.7.0)', () => {
it('should use custom contentExtractor when provided', async () => { it('should use custom contentExtractor when provided', async () => {
const customExtractor = (text: string) => { const customExtractor = (text: string) => {
return [ return [
{ text: 'custom extracted warrior content', contentCategory: 'prose' as const }, { text: 'custom extracted warrior content', contentCategory: 'content' as const },
{ text: 'Code Section', contentCategory: 'code' as const } { text: 'Code Section', contentCategory: 'code' as const }
] ]
} }
@ -937,8 +937,9 @@ describe('Hybrid Search (v7.7.0)', () => {
const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior') const warriorMatch = highlights.find(h => h.text.toLowerCase() === 'warrior')
expect(warriorMatch).toBeDefined() expect(warriorMatch).toBeDefined()
expect(warriorMatch?.matchType).toBe('text') expect(warriorMatch?.matchType).toBe('text')
expect(warriorMatch?.contentCategory).toBe('prose') expect(warriorMatch?.contentCategory).toBe('content')
}) })
}) })
describe('Timeout Protection (v7.9.0)', () => { describe('Timeout Protection (v7.9.0)', () => {

65
tests/unit/utils/contentExtractor.test.ts
View file

@ -75,10 +75,10 @@ describe('Content Extractor (v7.9.0)', () => {
expect(segments.length).toBe(3) expect(segments.length).toBe(3)
expect(segments[0].text).toBe('My Heading') expect(segments[0].text).toBe('My Heading')
expect(segments[0].contentCategory).toBe('heading') expect(segments[0].contentCategory).toBe('title')
expect(segments[1].text).toBe('Regular paragraph text') expect(segments[1].text).toBe('Regular paragraph text')
expect(segments[1].contentCategory).toBe('prose') expect(segments[1].contentCategory).toBe('content')
expect(segments[2].text).toBe('const x = 1') expect(segments[2].text).toBe('const x = 1')
expect(segments[2].contentCategory).toBe('code') expect(segments[2].contentCategory).toBe('code')
@ -124,9 +124,9 @@ describe('Content Extractor (v7.9.0)', () => {
const segments = extractForHighlighting(doc) const segments = extractForHighlighting(doc)
expect(segments.length).toBe(3) expect(segments.length).toBe(3)
expect(segments[0].text).toBe('Slate Heading') expect(segments[0].text).toBe('Slate Heading')
expect(segments[0].contentCategory).toBe('heading') expect(segments[0].contentCategory).toBe('title')
expect(segments[1].text).toBe('First part ') expect(segments[1].text).toBe('First part ')
expect(segments[1].contentCategory).toBe('prose') expect(segments[1].contentCategory).toBe('content')
}) })
}) })
@ -150,9 +150,9 @@ describe('Content Extractor (v7.9.0)', () => {
const segments = extractForHighlighting(doc) const segments = extractForHighlighting(doc)
expect(segments.length).toBe(2) expect(segments.length).toBe(2)
expect(segments[0].text).toBe('Lexical Heading') expect(segments[0].text).toBe('Lexical Heading')
expect(segments[0].contentCategory).toBe('heading') expect(segments[0].contentCategory).toBe('title')
expect(segments[1].text).toBe('Lexical body text') expect(segments[1].text).toBe('Lexical body text')
expect(segments[1].contentCategory).toBe('prose') expect(segments[1].contentCategory).toBe('content')
}) })
}) })
@ -223,11 +223,11 @@ describe('Content Extractor (v7.9.0)', () => {
const headingSegment = segments.find(s => s.text === 'Title') const headingSegment = segments.find(s => s.text === 'Title')
expect(headingSegment).toBeDefined() expect(headingSegment).toBeDefined()
expect(headingSegment?.contentCategory).toBe('heading') expect(headingSegment?.contentCategory).toBe('title')
const bodySegment = segments.find(s => s.text === 'Body text here.') const bodySegment = segments.find(s => s.text === 'Body text here.')
expect(bodySegment).toBeDefined() expect(bodySegment).toBeDefined()
expect(bodySegment?.contentCategory).toBe('prose') expect(bodySegment?.contentCategory).toBe('content')
const codeSegment = segments.find(s => s.text === 'let x = 1') const codeSegment = segments.find(s => s.text === 'let x = 1')
expect(codeSegment).toBeDefined() expect(codeSegment).toBeDefined()
@ -257,7 +257,7 @@ describe('Content Extractor (v7.9.0)', () => {
const segments = extractForHighlighting(html) const segments = extractForHighlighting(html)
const heading = segments.find(s => s.text === 'Nested Heading') const heading = segments.find(s => s.text === 'Nested Heading')
expect(heading).toBeDefined() expect(heading).toBeDefined()
expect(heading?.contentCategory).toBe('heading') expect(heading?.contentCategory).toBe('title')
}) })
}) })
@ -271,11 +271,11 @@ describe('Content Extractor (v7.9.0)', () => {
const body = segments.find(s => s.text === 'Some body text') const body = segments.find(s => s.text === 'Some body text')
expect(heading1).toBeDefined() expect(heading1).toBeDefined()
expect(heading1?.contentCategory).toBe('heading') expect(heading1?.contentCategory).toBe('title')
expect(heading2).toBeDefined() expect(heading2).toBeDefined()
expect(heading2?.contentCategory).toBe('heading') expect(heading2?.contentCategory).toBe('title')
expect(body).toBeDefined() expect(body).toBeDefined()
expect(body?.contentCategory).toBe('prose') expect(body?.contentCategory).toBe('content')
}) })
it('should extract fenced code blocks', () => { it('should extract fenced code blocks', () => {
@ -296,16 +296,51 @@ describe('Content Extractor (v7.9.0)', () => {
expect(code).toBeDefined() expect(code).toBeDefined()
expect(code?.text).toContain('code line 1') expect(code?.text).toContain('code line 1')
}) })
it('should split inline code spans into separate segments', () => {
const md = '# Title\n\nUse the `authenticate()` function to verify users'
const segments = extractForHighlighting(md)
// Should have: title segment, content "Use the", code "authenticate()", content "function to verify users"
const codeSegment = segments.find(s => s.text === 'authenticate()' && s.contentCategory === 'code')
expect(codeSegment).toBeDefined()
const beforeCode = segments.find(s => s.text === 'Use the' && s.contentCategory === 'content')
expect(beforeCode).toBeDefined()
const afterCode = segments.find(s => s.text === 'function to verify users' && s.contentCategory === 'content')
expect(afterCode).toBeDefined()
})
it('should handle multiple inline code spans in a single line', () => {
const md = '# API\n\nCall `foo()` then `bar()` for results'
const segments = extractForHighlighting(md, 'markdown')
const fooSegment = segments.find(s => s.text === 'foo()' && s.contentCategory === 'code')
const barSegment = segments.find(s => s.text === 'bar()' && s.contentCategory === 'code')
expect(fooSegment).toBeDefined()
expect(barSegment).toBeDefined()
})
it('should not split backticks inside fenced code blocks', () => {
const md = '# Title\n\n```\nconst x = `template`\n```'
const segments = extractForHighlighting(md)
const code = segments.find(s => s.contentCategory === 'code')
expect(code).toBeDefined()
// The fenced block content should be intact, not split by backticks
expect(code?.text).toContain('const x = `template`')
})
}) })
describe('extractForHighlighting() — Plaintext', () => { describe('extractForHighlighting() — Plaintext', () => {
it('should return single prose segment for plain text', () => { it('should return single content segment for plain text', () => {
const text = 'Just some plain text content' const text = 'Just some plain text content'
const segments = extractForHighlighting(text) const segments = extractForHighlighting(text)
expect(segments.length).toBe(1) expect(segments.length).toBe(1)
expect(segments[0].text).toBe(text) expect(segments[0].text).toBe(text)
expect(segments[0].contentCategory).toBe('prose') expect(segments[0].contentCategory).toBe('content')
}) })
it('should respect contentType override', () => { it('should respect contentType override', () => {
@ -315,7 +350,7 @@ describe('Content Extractor (v7.9.0)', () => {
expect(segments.length).toBe(1) expect(segments.length).toBe(1)
expect(segments[0].text).toBe(html) // Raw HTML, not extracted expect(segments[0].text).toBe(html) // Raw HTML, not extracted
expect(segments[0].contentCategory).toBe('prose') expect(segments[0].contentCategory).toBe('content')
}) })
}) })
}) })