brainy/docs/neural-extraction.md

Neural Entity Extraction Guide

Version: 5.7.6+ Status: Production-Ready Performance: ~15-20ms per extraction

---

Overview

Brainy's neural extraction system uses a 4-signal ensemble architecture to classify entities and relationships with high accuracy. The system is production-tested and handles 7 different document formats with format-specific intelligence.

Key Components

1. brain.extractEntities() - Simplest API, use for 95% of cases
2. SmartExtractor - Direct entity type classifier (advanced)
3. SmartRelationshipExtractor - Relationship type classifier
4. NeuralEntityExtractor - Full extraction orchestrator

---

Quick Start

Method 1: Brain Instance (Recommended)

import { Brainy, NounType } from '@soulcraft/brainy'

const brain = new Brainy()
await brain.init()

// Extract all entities
const entities = await brain.extractEntities('John Smith founded Acme Corp in New York')
// Returns:
// [
//   { text: 'John Smith', type: NounType.Person, confidence: 0.95 },
//   { text: 'Acme Corp', type: NounType.Organization, confidence: 0.92 },
//   { text: 'New York', type: NounType.Location, confidence: 0.88 }
// ]

// Extract with filters
const people = await brain.extractEntities('...', {
  types: [NounType.Person],
  confidence: 0.8,
  neuralMatching: true
})

Method 2: Direct Import (Advanced)

import {
  SmartExtractor,
  SmartRelationshipExtractor
} from '@soulcraft/brainy'
// Or use subpath imports:
import { SmartExtractor } from '@soulcraft/brainy/neural/SmartExtractor'

const brain = new Brainy()
await brain.init()

const extractor = new SmartExtractor(brain, { minConfidence: 0.7 })
const result = await extractor.extract('CEO')
// { type: NounType.Role, confidence: 0.89, source: 'ensemble', evidence: '...' }

---

Architecture

4-Signal Ensemble

Both SmartExtractor and SmartRelationshipExtractor use parallel signal execution:

Signal	Weight	Description	Speed
ExactMatch	40%	Dictionary lookups, aliases	~1ms
Embedding	35%	Semantic similarity (384-dim vectors)	~8ms
Pattern	20%	Regex patterns, format-aware	~2ms
Context	5%	Surrounding text hints	~4ms

Total Execution Time: ~15-20ms (parallel)

Format Intelligence

The system adapts to 7 document formats:

await extractor.extract('CEO', {
  formatContext: {
    format: 'excel',
    columnHeader: 'Job Title',  // Boosts Role type
    columnIndex: 3,
    adjacentHeaders: ['Name', 'Department']
  }
})

Supported Formats:

• Excel - Column headers, position intelligence
• CSV - Same as Excel
• PDF - Page structure, section detection
• YAML - Key paths, nesting levels
• DOCX - Styles, headings, lists
• JSON - Key paths, schema hints
• Markdown - Headers, lists, links

---

API Reference

brain.extractEntities()

Primary extraction method. Handles candidate detection + classification automatically.

async brain.extractEntities(
  text: string,
  options?: {
    types?: NounType[]          // Filter by types
    confidence?: number          // Min confidence (0-1)
    includeVectors?: boolean     // Add embeddings to results
    neuralMatching?: boolean     // Enable ensemble scoring
  }
): Promise<ExtractedEntity[]>

Returns:

interface ExtractedEntity {
  text: string                   // Original text
  type: NounType                 // Classified type
  confidence: number             // Score (0-1)
  start?: number                 // Character offset
  end?: number                   // Character offset
  vector?: number[]              // 384-dim embedding (if requested)
}

Examples:

// Extract all entities
const all = await brain.extractEntities(markdown)

// Extract only people
const people = await brain.extractEntities(text, {
  types: [NounType.Person]
})

// High-confidence only
const highConf = await brain.extractEntities(text, {
  confidence: 0.9
})

// Include vectors for similarity
const withVectors = await brain.extractEntities(text, {
  includeVectors: true
})

---

SmartExtractor

Direct entity type classifier. Use when you have pre-detected candidates or need custom configuration.

import { SmartExtractor, FormatContext } from '@soulcraft/brainy'

const extractor = new SmartExtractor(brain, {
  minConfidence: 0.7,              // Threshold
  enableEnsemble: true,             // Use all signals
  enableExactMatch: true,           // Dictionary lookups
  enableEmbedding: true,            // Semantic similarity
  enablePattern: true,              // Regex patterns
  enableContext: true,              // Context hints
  weights: {                        // Custom signal weights
    exactMatch: 0.5,                // 50%
    embedding: 0.3,                 // 30%
    pattern: 0.15,                  // 15%
    context: 0.05                   // 5%
  }
})

// Extract entity type
const result = await extractor.extract(
  'CEO',                            // Candidate text
  {
    formatContext: {                // Optional format hints
      format: 'excel',
      columnHeader: 'Title'
    },
    contextWindow: 'John is the CEO'  // Optional context
  }
)

Returns:

interface ExtractionResult {
  type: NounType                   // Classified type
  confidence: number               // Score (0-1)
  source: string                   // 'exact-match' | 'embedding' | 'ensemble'
  evidence: string                 // Human-readable explanation
  signalScores?: {                 // Individual signal scores
    exactMatch?: number
    embedding?: number
    pattern?: number
    context?: number
  }
}

---

SmartRelationshipExtractor

Relationship type classifier. Determines verb/relationship types between entities.

import { SmartRelationshipExtractor } from '@soulcraft/brainy'

const relExtractor = new SmartRelationshipExtractor(brain, {
  minConfidence: 0.6,
  enableEnsemble: true
})

// Infer relationship type
const relationship = await relExtractor.infer(
  'Alice',                         // Subject
  'UCSF',                          // Object
  'Alice works as a researcher at UCSF',  // Context
  {
    subjectType: NounType.Person,  // Optional type hints
    objectType: NounType.Organization
  }
)

Returns:

interface RelationshipExtractionResult {
  type: VerbType                   // Classified relationship
  confidence: number               // Score (0-1)
  source: string                   // Signal source
  evidence: string                 // Explanation
  signalScores?: {                 // Individual scores
    exactMatch?: number
    embedding?: number
    pattern?: number
    context?: number
  }
}

Example:

const rel = await relExtractor.infer(
  'John',
  'Acme Corp',
  'John Smith is the CEO of Acme Corp'
)
// {
//   type: VerbType.WorksFor,
//   confidence: 0.87,
//   source: 'ensemble',
//   evidence: 'Ensemble: exact-match (CEO pattern) + embedding (0.89 similarity)'
// }

---

NeuralEntityExtractor

Full extraction orchestrator. Handles candidate detection, classification, and deduplication.

import { NeuralEntityExtractor } from '@soulcraft/brainy'

const extractor = new NeuralEntityExtractor(brain)

// Full pipeline
const entities = await extractor.extract(text, {
  types: [NounType.Person, NounType.Organization],
  confidence: 0.7,
  neuralMatching: true
})

When to use:

• Need automatic candidate detection
• Want deduplication ("John Smith" and "Smith" → same entity)
• Building custom extraction pipelines
• Advanced configuration requirements

Typically accessed via brain.extractEntities() instead.

---

Import Preview Mode

Extract entities without persisting them to the database:

// Method 1: Using import() with preview mode
const result = await brain.import(markdownContent, {
  format: 'markdown',
  enableNeuralExtraction: true,  // Enable extraction
  enableConceptExtraction: true, // Enable concepts
  createEntities: false,          // DON'T persist to database
  vfsPath: null,                  // DON'T create VFS structure
  returnExtracted: true           // Return extracted data
})

// Access extracted entities
const entities = result.extractedEntities
// [
//   { text: 'John Smith', type: NounType.Person, confidence: 0.95 },
//   ...
// ]

// Method 2: Direct extraction (simpler)
const entities = await brain.extractEntities(markdownContent, {
  confidence: 0.7
})

Preview Mode Options:

Option	Effect
createEntities: false	Don't add to database
vfsPath: null	Don't create VFS files/folders
returnExtracted: true	Include extraction results
enableNeuralExtraction: true	Run entity extraction
enableConceptExtraction: true	Extract concepts/tags

---

Confidence Scoring

How Confidence is Calculated

Ensemble Mode (default):

confidence = (
  exactMatch × 0.40 +
  embedding × 0.35 +
  pattern × 0.20 +
  context × 0.05
)

Signal Scores:

• ExactMatch: 0.0 (no match) or 1.0 (exact match)
• Embedding: Cosine similarity (0.0-1.0)
• Pattern: Pattern match confidence (0.5-1.0)
• Context: Context relevance (0.0-1.0)

Example Calculation:

Text: "CEO"
ExactMatch: 1.0 (in dictionary)
Embedding: 0.89 (similar to "Role")
Pattern: 0.8 (job title pattern)
Context: 0.3 (mentioned near name)

Final: 1.0×0.40 + 0.89×0.35 + 0.8×0.20 + 0.3×0.05
     = 0.40 + 0.3115 + 0.16 + 0.015
     = 0.8865 (87% confidence)

Recommended Thresholds

Use Case	Threshold	Precision	Recall
High precision	0.9	95%	70%
Balanced	0.7	85%	85%
High recall	0.5	75%	95%

---

NounType Detection

42 Universal Types

Brainy supports 42 noun types covering 95% of all domains:

Core 7:

• Person - Human individuals
• Organization - Companies, institutions
• Location - Places, addresses
• Thing - Physical objects
• Concept - Abstract ideas
• Event - Occurrences, meetings
• Agent - Software, bots, AI

Extended 35:

• Document, Media, File - Content types
• Message, Collection, Dataset - Data structures
• Product, Service - Commercial
• Task, Project, Process - Work
• State, Role, Language - Properties
• Currency, Measurement - Quantitative
• Hypothesis, Experiment - Scientific
• Contract, Regulation - Legal
• ... [see types/graphTypes.ts for complete list]

Type Detection Methods

1. ExactMatch Signal (40% weight)

• Dictionary: 10,000+ aliases per type
• Examples: "CEO" → Role, "USD" → Currency
• Speed: ~1ms

2. Embedding Signal (35% weight)

• Semantic similarity to type embeddings
• 384-dimensional vectors
• Examples: "Chief Executive" → Role (cosine: 0.92)
• Speed: ~8ms

3. Pattern Signal (20% weight)

• Regex patterns for each type
• Format-aware (email → Message, URL → Document)
• Examples: \d{4}-\d{2}-\d{2} → Event
• Speed: ~2ms

4. Context Signal (5% weight)

• Surrounding word patterns
• Examples: "works at [X]" → X is Organization
• Speed: ~4ms

---

Performance Optimization

Caching

All extractors use LRU caching:

const extractor = new SmartExtractor(brain, {
  cache: {
    maxSize: 10000,              // Max cached items
    ttl: 3600000                 // 1 hour TTL
  }
})

// Cache stats
const stats = extractor.getCacheStats()
// { hits: 8432, misses: 1568, hitRate: 0.843 }

Batch Processing

// Process multiple candidates in parallel
const candidates = ['CEO', 'Alice', 'Acme Corp', 'New York']

const results = await Promise.all(
  candidates.map(text => extractor.extract(text))
)

Format Context Reuse

const formatContext = {
  format: 'excel' as const,
  columnHeader: 'Title'
}

// Reuse context for entire column
for (const cell of column) {
  await extractor.extract(cell, { formatContext })
}

---

Advanced: Custom Signal Weights

Adjust weights for domain-specific extraction:

// Medical domain: Boost pattern matching
const medicalExtractor = new SmartExtractor(brain, {
  weights: {
    exactMatch: 0.3,
    embedding: 0.2,
    pattern: 0.45,      // High for medical codes
    context: 0.05
  }
})

// Legal domain: Boost exact matching
const legalExtractor = new SmartExtractor(brain, {
  weights: {
    exactMatch: 0.6,    // High for legal terms
    embedding: 0.25,
    pattern: 0.10,
    context: 0.05
  }
})

---

Troubleshooting

Low Confidence Scores

Problem: Entities extracted with confidence <0.5

Solutions:

1. Add format context hints
2. Provide more surrounding context
3. Lower confidence threshold
4. Add domain-specific aliases

// Before: Generic extraction
const result = await extractor.extract('PM')
// { confidence: 0.45 }

// After: With context
const result = await extractor.extract('PM', {
  formatContext: {
    format: 'excel',
    columnHeader: 'Job Title'
  },
  contextWindow: 'The PM leads the project team'
})
// { confidence: 0.87 }

Type Misclassification

Problem: "John Smith" classified as Organization instead of Person

Solutions:

1. Provide type hints
2. Add more context
3. Check for name patterns

// Force type filtering
const people = await brain.extractEntities(text, {
  types: [NounType.Person]  // Only consider Person type
})

Slow Extraction

Problem: Extraction taking >100ms

Solutions:

1. Enable caching
2. Reduce context window
3. Disable unused signals
4. Use batch processing

const fastExtractor = new SmartExtractor(brain, {
  enableContext: false,        // Disable slowest signal
  cache: { maxSize: 50000 }    // Large cache
})

---

Examples

Example 1: PDF Resume Extraction

const resume = `
John Smith
Senior Software Engineer
Acme Corp (2020-2024)
Skills: Python, TypeScript, React
Location: San Francisco, CA
`

const entities = await brain.extractEntities(resume, {
  types: [NounType.Person, NounType.Organization, NounType.Location, NounType.Role],
  confidence: 0.7
})

// Filter by type
const person = entities.find(e => e.type === NounType.Person)
const companies = entities.filter(e => e.type === NounType.Organization)
const locations = entities.filter(e => e.type === NounType.Location)

Example 2: Excel Data Classification

import { SmartExtractor } from '@soulcraft/brainy'

const extractor = new SmartExtractor(brain)

// Process Excel column
const results = []
for (let i = 0; i < cells.length; i++) {
  const result = await extractor.extract(cells[i], {
    formatContext: {
      format: 'excel',
      columnHeader: headers[columnIndex],
      columnIndex,
      rowIndex: i
    }
  })
  results.push(result)
}

Example 3: Relationship Extraction

import { SmartRelationshipExtractor } from '@soulcraft/brainy'

const relExtractor = new SmartRelationshipExtractor(brain)

const text = 'Alice works as a researcher at UCSF'

// Extract relationship
const rel = await relExtractor.infer('Alice', 'UCSF', text, {
  subjectType: NounType.Person,
  objectType: NounType.Organization
})

// Create relationship in brain
if (rel.confidence > 0.7) {
  await brain.relate({
    from: aliceId,
    to: ucsfId,
    type: rel.type,
    metadata: { confidence: rel.confidence }
  })
}

---

Best Practices

1. Use brain.extractEntities() for 95% of cases

   • Handles everything automatically
   • Optimal for general use

2. Use direct extractors for:

   • Custom signal weights
   • Format-specific extraction
   • Batch processing optimization

3. Always provide context when possible

   • Improves confidence by 10-20%
   • Especially important for ambiguous terms

4. Enable caching for production

   • 80-90% cache hit rate typical
   • 10x speedup for repeated extractions

5. Filter by types when you know the domain

   • Reduces false positives
   • Improves performance

6. Monitor confidence distributions

   • Adjust thresholds per use case
   • Balance precision vs recall

---

See Also

• API Reference
• Type System
• Import System
• VFS System

---

Questions or Issues? https://github.com/soulcraftlabs/brainy/issues