brainy/docs/guides/migrating-to-v4.md

Migrating from Brainy v3.x to v4.x

Brainy v4.0.0 introduces breaking changes to the import API for improved clarity, better defaults, and more powerful features.

This guide will help you migrate your code quickly and painlessly.

---

🎯 Quick Migration Checklist

If you just want to fix your code fast, here's what to do:

• Replace extractRelationships with enableRelationshipInference
• Remove autoDetect (auto-detection is now always enabled)
• Replace createFileStructure: true with vfsPath: '/your/path'
• Remove excelSheets (all sheets are now processed automatically)
• Remove pdfExtractTables (table extraction is now automatic)
• Add enableNeuralExtraction: true to enable AI entity extraction
• Add preserveSource: true if you want to keep the original file

---

📋 Option Name Changes

Complete Mapping Table

v3.x Option	v4.x Option	Action Required
extractRelationships	enableRelationshipInference	Rename option
autoDetect	(removed)	Delete option (always enabled)
createFileStructure	vfsPath	Replace with VFS directory path
excelSheets	(removed)	Delete option (all sheets processed)
pdfExtractTables	(removed)	Delete option (always enabled)
-	enableNeuralExtraction	Add option (new in v4.x)
-	enableConceptExtraction	Add option (new in v4.x)
-	preserveSource	Add option (new in v4.x)

---

🔄 Migration Examples

Example 1: Basic Excel Import

Before (v3.x):

const result = await brain.import('./glossary.xlsx', {
  extractRelationships: true,
  createFileStructure: true,
  groupBy: 'type'
})

After (v4.x):

const result = await brain.import('./glossary.xlsx', {
  enableRelationshipInference: true,  // ✅ Renamed
  vfsPath: '/imports/glossary',       // ✅ Replaced createFileStructure
  groupBy: 'type'                     // ✅ No change
})

---

Example 2: Full-Featured Import

Before (v3.x):

const result = await brain.import('./data.xlsx', {
  extractRelationships: true,
  autoDetect: true,
  createFileStructure: true,
  groupBy: 'type',
  enableDeduplication: true
})

After (v4.x):

const result = await brain.import('./data.xlsx', {
  // AI features
  enableNeuralExtraction: true,      // ✅ NEW - Extract entity names
  enableRelationshipInference: true, // ✅ Renamed from extractRelationships
  enableConceptExtraction: true,     // ✅ NEW - Extract entity types

  // VFS features
  vfsPath: '/imports/data',          // ✅ Replaced createFileStructure
  groupBy: 'type',                   // ✅ No change
  preserveSource: true,              // ✅ NEW - Save original file

  // Performance
  enableDeduplication: true          // ✅ No change
})

---

Example 3: Simple Import (Defaults)

Before (v3.x):

const result = await brain.import('./data.csv', {
  autoDetect: true,
  extractRelationships: true
})

After (v4.x):

// Auto-detection is always enabled now
// Just enable the features you want
const result = await brain.import('./data.csv', {
  enableRelationshipInference: true
})

// Or use all defaults (AI features enabled)
const result = await brain.import('./data.csv')

---

Example 4: PDF Import

Before (v3.x):

const result = await brain.import('./document.pdf', {
  pdfExtractTables: true,
  extractRelationships: true,
  createFileStructure: true
})

After (v4.x):

const result = await brain.import('./document.pdf', {
  // pdfExtractTables removed - always enabled
  enableRelationshipInference: true,
  vfsPath: '/imports/documents'
})

---

💡 Why These Changes?

Clearer Option Names

v3.x naming was ambiguous:

• extractRelationships → Could mean "create relationships" or "infer relationships"
• createFileStructure → Doesn't explain what structure or where

v4.x naming is explicit:

• enableRelationshipInference → Clearly means "use AI to infer semantic relationships"
• vfsPath → Explicitly sets the virtual filesystem directory path
• enableNeuralExtraction → Clearly indicates AI-powered entity extraction

Separation of Concerns

v4.x separates import features into clear categories:

1. Neural/AI Features:

   • enableNeuralExtraction - Extract entity names and metadata
   • enableRelationshipInference - Infer semantic relationships
   • enableConceptExtraction - Extract entity types and concepts

2. VFS Features:

   • vfsPath - Virtual filesystem directory
   • groupBy - Grouping strategy
   • preserveSource - Keep original file

3. Performance Features:

   • enableDeduplication - Merge similar entities
   • confidenceThreshold - AI confidence threshold
   • onProgress - Progress callbacks

Better Defaults

v3.x required explicit enabling:

// Had to enable everything manually
await brain.import(file, {
  autoDetect: true,
  extractRelationships: true,
  createFileStructure: true
})

v4.x has smart defaults:

// Auto-detection and AI features enabled by default
await brain.import(file)

// Or customize specific features
await brain.import(file, {
  vfsPath: '/my/data',
  confidenceThreshold: 0.8
})

---

🆕 New Features in v4.x

Neural Entity Extraction

Extract entity names, types, and metadata using AI:

const result = await brain.import('./glossary.xlsx', {
  enableNeuralExtraction: true,      // Extract entity names from "Term" column
  enableConceptExtraction: true,     // Detect entity types (Place, Person, etc.)
  confidenceThreshold: 0.7           // Minimum AI confidence (0-1)
})

// Result includes rich entity metadata
result.entities.forEach(entity => {
  console.log(`${entity.name} (${entity.type})`)
  console.log(`Confidence: ${entity.confidence}`)
})

VFS Integration

Imported data is organized in a virtual filesystem:

const result = await brain.import('./data.xlsx', {
  vfsPath: '/projects/myproject/data',
  groupBy: 'type',        // Group by entity type
  preserveSource: true    // Save original .xlsx file
})

// Access via VFS
const vfs = brain.vfs()
const files = await vfs.readdir('/projects/myproject/data')
// ['Places/', 'Characters/', 'Concepts/', '_source.xlsx', '_metadata.json']

// Read entity file
const content = await vfs.readFile('/projects/myproject/data/Places/Talifar.json')

Semantic Relationship Inference

AI infers relationship types from context:

const result = await brain.import('./glossary.xlsx', {
  enableRelationshipInference: true
})

// Instead of generic "contains" relationships,
// you get semantic verbs like:
// - "capital_of"
// - "located_in"
// - "guards"
// - "part_of"
// - "related_to"

const relations = await brain.related({ limit: 100 })
const types = new Set(relations.map(r => r.label))
console.log(types)
// Set { 'capital_of', 'guards', 'located_in', 'related_to' }

---

🔍 What Breaks & How to Fix It

Error: "Invalid import options: 'extractRelationships'"

Cause: Using v3.x option name

Fix:

// Before
await brain.import(file, { extractRelationships: true })

// After
await brain.import(file, { enableRelationshipInference: true })

---

Error: "Invalid import options: 'autoDetect'"

Cause: Using v3.x option that's been removed

Fix:

// Before
await brain.import(file, { autoDetect: true })

// After - just remove it (auto-detection always enabled)
await brain.import(file)

---

Error: "Invalid import options: 'createFileStructure'"

Cause: Using v3.x option name

Fix:

// Before
await brain.import(file, { createFileStructure: true })

// After - specify VFS path explicitly
await brain.import(file, { vfsPath: '/imports/mydata' })

---

Issue: Import succeeds but entities have generic names like "Entity_144"

Cause: Neural extraction is disabled

Fix:

// Ensure AI features are enabled
await brain.import(file, {
  enableNeuralExtraction: true,      // ✅ Extract entity names
  enableRelationshipInference: true, // ✅ Infer relationships
  enableConceptExtraction: true      // ✅ Extract types
})

---

Issue: All relationships are type "contains"

Cause: Relationship inference is disabled

Fix:

// Enable relationship inference
await brain.import(file, {
  enableRelationshipInference: true  // ✅ Use AI to detect semantic relationships
})

---

Issue: VFS directory doesn't exist in filesystem

This is NORMAL! VFS is virtual - it uses Brainy entities, not physical files.

How to access VFS:

// DON'T do this:
// ls brainy-data/vfs/  ❌ Won't work

// DO this instead:
const vfs = brain.vfs()
await vfs.init()
const files = await vfs.readdir('/imports')  // ✅ Correct

---

📦 TypeScript Users

Compile-Time Errors

If you're using TypeScript, you'll get compile-time errors when using deprecated options:

// TypeScript will show error:
// "Type 'true' is not assignable to type 'never'"
await brain.import(file, {
  extractRelationships: true  // ❌ Type error
})

// Fix: Use correct option name
await brain.import(file, {
  enableRelationshipInference: true  // ✅ Type correct
})

IDE Autocomplete

Your IDE will show deprecation warnings and suggest the correct option names:

await brain.import(file, {
  extract...  // IDE suggests: enableNeuralExtraction, enableRelationshipInference
})

---

🎓 Best Practices for v4.x

1. Enable All AI Features by Default

// Good: Enable all intelligent features
await brain.import('./data.xlsx', {
  enableNeuralExtraction: true,
  enableRelationshipInference: true,
  enableConceptExtraction: true,
  vfsPath: '/imports/data'
})

2. Use VFS for Organization

// Good: Organize by project
await brain.import('./project-A.xlsx', {
  vfsPath: '/projects/project-a/data'
})

await brain.import('./project-B.csv', {
  vfsPath: '/projects/project-b/data'
})

3. Preserve Source Files

// Good: Keep original files for reference
await brain.import('./important-data.xlsx', {
  preserveSource: true,  // Saves original .xlsx in VFS
  vfsPath: '/archives/2025'
})

4. Tune Confidence Threshold

// For high-quality data: Lower threshold
await brain.import('./curated-glossary.xlsx', {
  confidenceThreshold: 0.5  // Extract more entities
})

// For noisy data: Higher threshold
await brain.import('./scraped-data.csv', {
  confidenceThreshold: 0.8  // Only high-confidence entities
})

5. Disable Deduplication for Large Imports

// For small imports: Keep deduplication
await brain.import('./small-data.xlsx', {
  enableDeduplication: true
})

// For large imports (>1000 rows): Disable for performance
await brain.import('./huge-database.csv', {
  enableDeduplication: false  // Much faster
})

---

🚀 Migration Automation (Future)

We're working on an automated migration tool:

# Coming soon
npx @soulcraft/brainy-migrate

# Will scan your code and automatically update:
# - Option names
# - TypeScript types
# - Import patterns

---

📚 Additional Resources

• API Documentation: https://brainy.dev/docs/api/import
• Examples: examples/import-excel/
• Changelog: CHANGELOG.md
• Support: GitHub Issues

---

💬 Need Help?

If you're stuck migrating:

1. Check the error message - it includes migration hints
2. Review the examples in this guide
3. Open an issue on GitHub with your use case
4. Join our Discord community for real-time help

---

Happy migrating! 🎉