brainy/docs/vfs/NEURAL_EXTRACTION.md

Neural Extraction API - AI-Powered Concept and Entity Detection

Overview

Brainy's Neural Extraction system uses embeddings and a sophisticated NounType taxonomy to extract meaningful entities and concepts from text. Unlike simple regex-based extraction, neural extraction understands semantic meaning and context.

Architecture

┌─────────────────────────────────────┐
│     brain.extractConcepts()         │
│   (High-level concept wrapper)      │
└──────────────┬──────────────────────┘
               │
               ▼
┌─────────────────────────────────────┐
│       brain.extract()               │
│   (Full entity extraction)          │
└──────────────┬──────────────────────┘
               │
               ▼
┌─────────────────────────────────────┐
│     NeuralEntityExtractor           │
│  (394-line production impl)         │
└──────────────┬──────────────────────┘
               │
        ┌──────┴──────┐
        ▼             ▼
  ┌─────────┐   ┌──────────┐
  │ Pattern │   │ Embeddings│
  │ Matching│   │ + NounType│
  │         │   │ Taxonomy  │
  └─────────┘   └──────────┘

NounType Taxonomy

Brainy uses a 30+ type taxonomy for entity classification:

Core Types

• Person - Individual humans
• Organization - Companies, institutions, groups
• Location - Places, cities, countries
• Event - Occurrences, happenings
• Product - Goods, services, items
• Concept - Abstract ideas, theories
• Topic - Subject areas, domains

Technical Types

• API - Application programming interfaces
• Service - Software services
• Component - System components
• Function - Code functions
• Class - Object-oriented classes
• Module - Software modules

Creative Types

• Character - Fictional characters
• Setting - Story locations
• Plot - Story arcs
• Theme - Narrative themes

Business Types

• Customer - Clients, users
• Project - Business projects
• Process - Business processes
• KPI - Key performance indicators

And more...

API Reference

brain.extract(text, options)

Extracts entities from text with full configuration options.

Parameters:

brain.extract(
  text: string,
  options?: {
    types?: NounType[]        // Filter to specific types
    confidence?: number       // Min confidence (0-1, default: 0.6)
    includeVectors?: boolean  // Include embeddings
    neuralMatching?: boolean  // Use neural classification (default: true)
  }
): Promise<ExtractedEntity[]>

Returns:

interface ExtractedEntity {
  text: string              // Extracted text
  type: NounType           // Classified type
  position: number         // Position in text
  confidence: number       // Confidence score (0-1)
  vector?: number[]        // Optional embedding
}

Example:

const brain = new Brainy()
await brain.init()

const text = `
  The UserService API provides authentication and authorization.
  It integrates with the Database component for user storage.
`

// Extract all entities
const entities = await brain.extract(text)
console.log(entities)
// [
//   { text: 'UserService', type: 'Service', confidence: 0.87 },
//   { text: 'API', type: 'API', confidence: 0.92 },
//   { text: 'authentication', type: 'Concept', confidence: 0.79 },
//   { text: 'authorization', type: 'Concept', confidence: 0.81 },
//   { text: 'Database', type: 'Component', confidence: 0.85 }
// ]

// Extract only technical entities
const technical = await brain.extract(text, {
  types: [NounType.Service, NounType.API, NounType.Component],
  confidence: 0.8
})
console.log(technical)
// [
//   { text: 'UserService', type: 'Service', confidence: 0.87 },
//   { text: 'API', type: 'API', confidence: 0.92 },
//   { text: 'Database', type: 'Component', confidence: 0.85 }
// ]

brain.extractConcepts(text, options)

Simplified API specifically for concept extraction.

Parameters:

brain.extractConcepts(
  text: string,
  options?: {
    confidence?: number    // Min confidence (default: 0.7)
    limit?: number        // Max concepts to return
  }
): Promise<string[]>

Returns:

string[]  // Array of concept names (deduplicated, lowercase)

Example:

const text = `
  Our authentication system uses JWT tokens for security.
  The authorization layer checks user permissions and roles.
`

const concepts = await brain.extractConcepts(text, {
  confidence: 0.7,
  limit: 10
})
console.log(concepts)
// ['authentication', 'security', 'authorization', 'permissions']

How It Works

1. Pattern-Based Candidate Detection

First, NeuralEntityExtractor scans text for potential entities using:

• Capitalized words and phrases
• Technical patterns (camelCase, PascalCase, UPPER_CASE)
• Quoted strings
• Common entity patterns

2. Embedding Generation

Each candidate is converted to a semantic embedding vector:

const candidateVector = await brain.getEmbedding("UserService")
// [0.234, -0.123, 0.567, ...] (1536 dimensions)

3. NounType Classification

The embedding is compared against pre-computed embeddings for each NounType using cosine similarity:

const serviceVector = typeEmbeddings.get(NounType.Service)
const similarity = cosineSimilarity(candidateVector, serviceVector)
// similarity = 0.87 (87% match)

4. Context-Based Boosting

Confidence is adjusted based on surrounding context:

// "The UserService API" gets boosted for Service type
// "CEO of Company" gets boosted for Person type
// "located in Paris" gets boosted for Location type

5. Deduplication

Similar or overlapping entities are merged to avoid duplicates.

VFS Integration

VFS automatically uses neural extraction when writing files:

const vfs = brain.vfs()
await vfs.init()

// Automatic concept extraction (if enabled)
await vfs.writeFile('/docs/api.md', `
  # User Authentication API

  The UserService provides secure authentication using JWT tokens.
  Integrates with the Database for user storage.
`, {
  intelligence: {
    autoConcepts: true  // Enable concept extraction (default)
  }
})

// Concepts automatically extracted and indexed:
// ['authentication', 'security', 'database', 'user']

// Now searchable by concept
const authFiles = await vfs.readdir('/by-concept/authentication')
// Includes /docs/api.md

Performance

• Candidate extraction: O(n) where n = text length
• Embedding generation: ~10-50ms per candidate (cached)
• Type classification: O(t) where t = number of types to check
• Total time: Typically 100-500ms for a document

Caching:

• Embeddings are cached (UnifiedCache, 2GB default)
• TypeEmbeddings precomputed once at initialization
• Results cached for identical text

Configuration

VFS Auto-Concept Extraction

const vfs = brain.vfs()
await vfs.init({
  intelligence: {
    enabled: true,           // Enable AI features (default: true)
    autoConcepts: true,      // Auto-extract concepts (default: true)
    autoExtract: true,       // Auto-extract entities (default: true)
  }
})

Custom Confidence Thresholds

// Strict extraction (fewer, higher confidence)
const strict = await brain.extract(text, { confidence: 0.9 })

// Permissive extraction (more entities, lower confidence)
const permissive = await brain.extract(text, { confidence: 0.5 })

Type-Specific Extraction

// Extract only people and organizations
const entities = await brain.extract(text, {
  types: [NounType.Person, NounType.Organization]
})

// Extract only technical entities
const technical = await brain.extract(text, {
  types: [
    NounType.API,
    NounType.Service,
    NounType.Component,
    NounType.Function
  ]
})

Examples

Example 1: Technical Documentation

const technicalDoc = `
  The PaymentService API integrates with Stripe for processing transactions.
  The OrderManager component coordinates between UserService and InventoryService.
`

const entities = await brain.extract(technicalDoc, {
  types: [NounType.Service, NounType.API, NounType.Component]
})

console.log(entities)
// [
//   { text: 'PaymentService', type: 'Service', confidence: 0.89 },
//   { text: 'API', type: 'API', confidence: 0.93 },
//   { text: 'Stripe', type: 'Service', confidence: 0.76 },
//   { text: 'OrderManager', type: 'Component', confidence: 0.84 },
//   { text: 'UserService', type: 'Service', confidence: 0.91 },
//   { text: 'InventoryService', type: 'Service', confidence: 0.88 }
// ]

Example 2: Creative Writing

const story = `
  Detective Sarah Chen arrived at the scene. The abandoned warehouse
  held secrets about the mysterious organization known as Shadow Corp.
`

const entities = await brain.extract(story, {
  types: [NounType.Person, NounType.Location, NounType.Organization]
})

console.log(entities)
// [
//   { text: 'Detective Sarah Chen', type: 'Person', confidence: 0.94 },
//   { text: 'warehouse', type: 'Location', confidence: 0.71 },
//   { text: 'Shadow Corp', type: 'Organization', confidence: 0.82 }
// ]

Example 3: Business Documents

const businessDoc = `
  Q3 revenue exceeded targets by 15%. The marketing team's new campaign
  generated 50,000 leads. Customer satisfaction remains our top KPI.
`

const concepts = await brain.extractConcepts(businessDoc, {
  confidence: 0.65
})

console.log(concepts)
// ['revenue', 'marketing', 'campaign', 'leads', 'customer', 'satisfaction']

const entities = await brain.extract(businessDoc, {
  types: [NounType.KPI, NounType.Process, NounType.Customer]
})
// [
//   { text: 'revenue', type: 'KPI', confidence: 0.81 },
//   { text: 'Customer satisfaction', type: 'KPI', confidence: 0.87 }
// ]

Advanced Usage

With Vector Embeddings

const entities = await brain.extract(text, {
  includeVectors: true
})

// Use embeddings for custom similarity search
for (const entity of entities) {
  const similar = await brain.similar(entity.vector, { limit: 5 })
  console.log(`Similar to ${entity.text}:`, similar)
}

Custom Entity Processing

const entities = await brain.extract(text)

// Group by type
const byType = entities.reduce((acc, entity) => {
  if (!acc[entity.type]) acc[entity.type] = []
  acc[entity.type].push(entity)
  return acc
}, {})

console.log('Services:', byType[NounType.Service])
console.log('APIs:', byType[NounType.API])
console.log('Concepts:', byType[NounType.Concept])

Batch Processing

const documents = [doc1, doc2, doc3, /* ... */]

// Extract concepts from all documents
const allConcepts = await Promise.all(
  documents.map(doc => brain.extractConcepts(doc))
)

// Combine and deduplicate
const uniqueConcepts = [...new Set(allConcepts.flat())]
console.log('All concepts:', uniqueConcepts)

Zero-Config Design

Neural extraction works out-of-the-box:

• ✅ No configuration required
• ✅ No training needed
• ✅ No API keys needed
• ✅ Works with all storage adapters
• ✅ Automatic caching and optimization
• ✅ Sensible defaults for all parameters

See Also

• VFS Core Documentation - Complete filesystem API
• Semantic VFS - Multi-dimensional file access
• Triple Intelligence™ - Underlying architecture
• NounType Taxonomy - Complete type reference