feat: add import API validation and v4.x migration guide
Add comprehensive migration support for v4.x import API changes: - Runtime validation that rejects deprecated v3.x options with clear errors - Configurable error verbosity (detailed in dev, concise in prod) - Complete migration guide (docs/guides/migrating-to-v4.md) - Enhanced CHANGELOG with breaking changes documentation - TypeScript type safety using 'never' types for deprecated options - Comprehensive JSDoc with deprecation warnings and examples Deprecated v3.x options now throw helpful errors: - extractRelationships → enableRelationshipInference - createFileStructure → vfsPath - autoDetect → (removed - always enabled) - excelSheets → (removed - all sheets processed) - pdfExtractTables → (removed - always enabled) Resolves Workshop team import issues where incorrect option names disabled all intelligent features, causing generic entity creation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
1001af9a34
commit
a1a0576d04
4 changed files with 822 additions and 24 deletions
491
docs/guides/migrating-to-v4.md
Normal file
491
docs/guides/migrating-to-v4.md
Normal file
|
|
@ -0,0 +1,491 @@
|
|||
# Migrating from Brainy v3.x to v4.x
|
||||
|
||||
**Brainy v4.0.0** introduces breaking changes to the import API for improved clarity, better defaults, and more powerful features.
|
||||
|
||||
This guide will help you migrate your code quickly and painlessly.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Quick Migration Checklist
|
||||
|
||||
If you just want to fix your code fast, here's what to do:
|
||||
|
||||
- [ ] Replace `extractRelationships` with `enableRelationshipInference`
|
||||
- [ ] Remove `autoDetect` (auto-detection is now always enabled)
|
||||
- [ ] Replace `createFileStructure: true` with `vfsPath: '/your/path'`
|
||||
- [ ] Remove `excelSheets` (all sheets are now processed automatically)
|
||||
- [ ] Remove `pdfExtractTables` (table extraction is now automatic)
|
||||
- [ ] Add `enableNeuralExtraction: true` to enable AI entity extraction
|
||||
- [ ] Add `preserveSource: true` if you want to keep the original file
|
||||
|
||||
---
|
||||
|
||||
## 📋 Option Name Changes
|
||||
|
||||
### Complete Mapping Table
|
||||
|
||||
| v3.x Option | v4.x Option | Action Required |
|
||||
|-------------|-------------|-----------------|
|
||||
| `extractRelationships` | `enableRelationshipInference` | **Rename option** |
|
||||
| `autoDetect` | *(removed)* | **Delete option** (always enabled) |
|
||||
| `createFileStructure` | `vfsPath` | **Replace** with VFS directory path |
|
||||
| `excelSheets` | *(removed)* | **Delete option** (all sheets processed) |
|
||||
| `pdfExtractTables` | *(removed)* | **Delete option** (always enabled) |
|
||||
| - | `enableNeuralExtraction` | **Add option** (new in v4.x) |
|
||||
| - | `enableConceptExtraction` | **Add option** (new in v4.x) |
|
||||
| - | `preserveSource` | **Add option** (new in v4.x) |
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Migration Examples
|
||||
|
||||
### Example 1: Basic Excel Import
|
||||
|
||||
**Before (v3.x):**
|
||||
```typescript
|
||||
const result = await brain.import('./glossary.xlsx', {
|
||||
extractRelationships: true,
|
||||
createFileStructure: true,
|
||||
groupBy: 'type'
|
||||
})
|
||||
```
|
||||
|
||||
**After (v4.x):**
|
||||
```typescript
|
||||
const result = await brain.import('./glossary.xlsx', {
|
||||
enableRelationshipInference: true, // ✅ Renamed
|
||||
vfsPath: '/imports/glossary', // ✅ Replaced createFileStructure
|
||||
groupBy: 'type' // ✅ No change
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 2: Full-Featured Import
|
||||
|
||||
**Before (v3.x):**
|
||||
```typescript
|
||||
const result = await brain.import('./data.xlsx', {
|
||||
extractRelationships: true,
|
||||
autoDetect: true,
|
||||
createFileStructure: true,
|
||||
groupBy: 'type',
|
||||
enableDeduplication: true
|
||||
})
|
||||
```
|
||||
|
||||
**After (v4.x):**
|
||||
```typescript
|
||||
const result = await brain.import('./data.xlsx', {
|
||||
// AI features
|
||||
enableNeuralExtraction: true, // ✅ NEW - Extract entity names
|
||||
enableRelationshipInference: true, // ✅ Renamed from extractRelationships
|
||||
enableConceptExtraction: true, // ✅ NEW - Extract entity types
|
||||
|
||||
// VFS features
|
||||
vfsPath: '/imports/data', // ✅ Replaced createFileStructure
|
||||
groupBy: 'type', // ✅ No change
|
||||
preserveSource: true, // ✅ NEW - Save original file
|
||||
|
||||
// Performance
|
||||
enableDeduplication: true // ✅ No change
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 3: Simple Import (Defaults)
|
||||
|
||||
**Before (v3.x):**
|
||||
```typescript
|
||||
const result = await brain.import('./data.csv', {
|
||||
autoDetect: true,
|
||||
extractRelationships: true
|
||||
})
|
||||
```
|
||||
|
||||
**After (v4.x):**
|
||||
```typescript
|
||||
// Auto-detection is always enabled now
|
||||
// Just enable the features you want
|
||||
const result = await brain.import('./data.csv', {
|
||||
enableRelationshipInference: true
|
||||
})
|
||||
|
||||
// Or use all defaults (AI features enabled)
|
||||
const result = await brain.import('./data.csv')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 4: PDF Import
|
||||
|
||||
**Before (v3.x):**
|
||||
```typescript
|
||||
const result = await brain.import('./document.pdf', {
|
||||
pdfExtractTables: true,
|
||||
extractRelationships: true,
|
||||
createFileStructure: true
|
||||
})
|
||||
```
|
||||
|
||||
**After (v4.x):**
|
||||
```typescript
|
||||
const result = await brain.import('./document.pdf', {
|
||||
// pdfExtractTables removed - always enabled
|
||||
enableRelationshipInference: true,
|
||||
vfsPath: '/imports/documents'
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💡 Why These Changes?
|
||||
|
||||
### Clearer Option Names
|
||||
|
||||
**v3.x naming was ambiguous:**
|
||||
- `extractRelationships` → Could mean "create relationships" or "infer relationships"
|
||||
- `createFileStructure` → Doesn't explain what structure or where
|
||||
|
||||
**v4.x naming is explicit:**
|
||||
- `enableRelationshipInference` → Clearly means "use AI to infer semantic relationships"
|
||||
- `vfsPath` → Explicitly sets the virtual filesystem directory path
|
||||
- `enableNeuralExtraction` → Clearly indicates AI-powered entity extraction
|
||||
|
||||
### Separation of Concerns
|
||||
|
||||
**v4.x separates import features into clear categories:**
|
||||
|
||||
1. **Neural/AI Features:**
|
||||
- `enableNeuralExtraction` - Extract entity names and metadata
|
||||
- `enableRelationshipInference` - Infer semantic relationships
|
||||
- `enableConceptExtraction` - Extract entity types and concepts
|
||||
|
||||
2. **VFS Features:**
|
||||
- `vfsPath` - Virtual filesystem directory
|
||||
- `groupBy` - Grouping strategy
|
||||
- `preserveSource` - Keep original file
|
||||
|
||||
3. **Performance Features:**
|
||||
- `enableDeduplication` - Merge similar entities
|
||||
- `confidenceThreshold` - AI confidence threshold
|
||||
- `onProgress` - Progress callbacks
|
||||
|
||||
### Better Defaults
|
||||
|
||||
**v3.x required explicit enabling:**
|
||||
```typescript
|
||||
// Had to enable everything manually
|
||||
await brain.import(file, {
|
||||
autoDetect: true,
|
||||
extractRelationships: true,
|
||||
createFileStructure: true
|
||||
})
|
||||
```
|
||||
|
||||
**v4.x has smart defaults:**
|
||||
```typescript
|
||||
// Auto-detection and AI features enabled by default
|
||||
await brain.import(file)
|
||||
|
||||
// Or customize specific features
|
||||
await brain.import(file, {
|
||||
vfsPath: '/my/data',
|
||||
confidenceThreshold: 0.8
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🆕 New Features in v4.x
|
||||
|
||||
### Neural Entity Extraction
|
||||
Extract entity names, types, and metadata using AI:
|
||||
|
||||
```typescript
|
||||
const result = await brain.import('./glossary.xlsx', {
|
||||
enableNeuralExtraction: true, // Extract entity names from "Term" column
|
||||
enableConceptExtraction: true, // Detect entity types (Place, Person, etc.)
|
||||
confidenceThreshold: 0.7 // Minimum AI confidence (0-1)
|
||||
})
|
||||
|
||||
// Result includes rich entity metadata
|
||||
result.entities.forEach(entity => {
|
||||
console.log(`${entity.name} (${entity.type})`)
|
||||
console.log(`Confidence: ${entity.confidence}`)
|
||||
})
|
||||
```
|
||||
|
||||
### VFS Integration
|
||||
Imported data is organized in a virtual filesystem:
|
||||
|
||||
```typescript
|
||||
const result = await brain.import('./data.xlsx', {
|
||||
vfsPath: '/projects/myproject/data',
|
||||
groupBy: 'type', // Group by entity type
|
||||
preserveSource: true // Save original .xlsx file
|
||||
})
|
||||
|
||||
// Access via VFS
|
||||
const vfs = brain.vfs()
|
||||
const files = await vfs.readdir('/projects/myproject/data')
|
||||
// ['Places/', 'Characters/', 'Concepts/', '_source.xlsx', '_metadata.json']
|
||||
|
||||
// Read entity file
|
||||
const content = await vfs.readFile('/projects/myproject/data/Places/Talifar.json')
|
||||
```
|
||||
|
||||
### Semantic Relationship Inference
|
||||
AI infers relationship types from context:
|
||||
|
||||
```typescript
|
||||
const result = await brain.import('./glossary.xlsx', {
|
||||
enableRelationshipInference: true
|
||||
})
|
||||
|
||||
// Instead of generic "contains" relationships,
|
||||
// you get semantic verbs like:
|
||||
// - "capital_of"
|
||||
// - "located_in"
|
||||
// - "guards"
|
||||
// - "part_of"
|
||||
// - "related_to"
|
||||
|
||||
const relations = await brain.getRelations({ limit: 100 })
|
||||
const types = new Set(relations.map(r => r.label))
|
||||
console.log(types)
|
||||
// Set { 'capital_of', 'guards', 'located_in', 'related_to' }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 What Breaks & How to Fix It
|
||||
|
||||
### Error: "Invalid import options: 'extractRelationships'"
|
||||
|
||||
**Cause:** Using v3.x option name
|
||||
|
||||
**Fix:**
|
||||
```typescript
|
||||
// Before
|
||||
await brain.import(file, { extractRelationships: true })
|
||||
|
||||
// After
|
||||
await brain.import(file, { enableRelationshipInference: true })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Error: "Invalid import options: 'autoDetect'"
|
||||
|
||||
**Cause:** Using v3.x option that's been removed
|
||||
|
||||
**Fix:**
|
||||
```typescript
|
||||
// Before
|
||||
await brain.import(file, { autoDetect: true })
|
||||
|
||||
// After - just remove it (auto-detection always enabled)
|
||||
await brain.import(file)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Error: "Invalid import options: 'createFileStructure'"
|
||||
|
||||
**Cause:** Using v3.x option name
|
||||
|
||||
**Fix:**
|
||||
```typescript
|
||||
// Before
|
||||
await brain.import(file, { createFileStructure: true })
|
||||
|
||||
// After - specify VFS path explicitly
|
||||
await brain.import(file, { vfsPath: '/imports/mydata' })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Import succeeds but entities have generic names like "Entity_144"
|
||||
|
||||
**Cause:** Neural extraction is disabled
|
||||
|
||||
**Fix:**
|
||||
```typescript
|
||||
// Ensure AI features are enabled
|
||||
await brain.import(file, {
|
||||
enableNeuralExtraction: true, // ✅ Extract entity names
|
||||
enableRelationshipInference: true, // ✅ Infer relationships
|
||||
enableConceptExtraction: true // ✅ Extract types
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: All relationships are type "contains"
|
||||
|
||||
**Cause:** Relationship inference is disabled
|
||||
|
||||
**Fix:**
|
||||
```typescript
|
||||
// Enable relationship inference
|
||||
await brain.import(file, {
|
||||
enableRelationshipInference: true // ✅ Use AI to detect semantic relationships
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: VFS directory doesn't exist in filesystem
|
||||
|
||||
**This is NORMAL!** VFS is virtual - it uses Brainy entities, not physical files.
|
||||
|
||||
**How to access VFS:**
|
||||
```typescript
|
||||
// DON'T do this:
|
||||
// ls brainy-data/vfs/ ❌ Won't work
|
||||
|
||||
// DO this instead:
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
const files = await vfs.readdir('/imports') // ✅ Correct
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📦 TypeScript Users
|
||||
|
||||
### Compile-Time Errors
|
||||
|
||||
If you're using TypeScript, you'll get compile-time errors when using deprecated options:
|
||||
|
||||
```typescript
|
||||
// TypeScript will show error:
|
||||
// "Type 'true' is not assignable to type 'never'"
|
||||
await brain.import(file, {
|
||||
extractRelationships: true // ❌ Type error
|
||||
})
|
||||
|
||||
// Fix: Use correct option name
|
||||
await brain.import(file, {
|
||||
enableRelationshipInference: true // ✅ Type correct
|
||||
})
|
||||
```
|
||||
|
||||
### IDE Autocomplete
|
||||
|
||||
Your IDE will show deprecation warnings and suggest the correct option names:
|
||||
|
||||
```typescript
|
||||
await brain.import(file, {
|
||||
extract... // IDE suggests: enableNeuralExtraction, enableRelationshipInference
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Best Practices for v4.x
|
||||
|
||||
### 1. Enable All AI Features by Default
|
||||
|
||||
```typescript
|
||||
// Good: Enable all intelligent features
|
||||
await brain.import('./data.xlsx', {
|
||||
enableNeuralExtraction: true,
|
||||
enableRelationshipInference: true,
|
||||
enableConceptExtraction: true,
|
||||
vfsPath: '/imports/data'
|
||||
})
|
||||
```
|
||||
|
||||
### 2. Use VFS for Organization
|
||||
|
||||
```typescript
|
||||
// Good: Organize by project
|
||||
await brain.import('./project-A.xlsx', {
|
||||
vfsPath: '/projects/project-a/data'
|
||||
})
|
||||
|
||||
await brain.import('./project-B.csv', {
|
||||
vfsPath: '/projects/project-b/data'
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Preserve Source Files
|
||||
|
||||
```typescript
|
||||
// Good: Keep original files for reference
|
||||
await brain.import('./important-data.xlsx', {
|
||||
preserveSource: true, // Saves original .xlsx in VFS
|
||||
vfsPath: '/archives/2025'
|
||||
})
|
||||
```
|
||||
|
||||
### 4. Tune Confidence Threshold
|
||||
|
||||
```typescript
|
||||
// For high-quality data: Lower threshold
|
||||
await brain.import('./curated-glossary.xlsx', {
|
||||
confidenceThreshold: 0.5 // Extract more entities
|
||||
})
|
||||
|
||||
// For noisy data: Higher threshold
|
||||
await brain.import('./scraped-data.csv', {
|
||||
confidenceThreshold: 0.8 // Only high-confidence entities
|
||||
})
|
||||
```
|
||||
|
||||
### 5. Disable Deduplication for Large Imports
|
||||
|
||||
```typescript
|
||||
// For small imports: Keep deduplication
|
||||
await brain.import('./small-data.xlsx', {
|
||||
enableDeduplication: true
|
||||
})
|
||||
|
||||
// For large imports (>1000 rows): Disable for performance
|
||||
await brain.import('./huge-database.csv', {
|
||||
enableDeduplication: false // Much faster
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Migration Automation (Future)
|
||||
|
||||
We're working on an automated migration tool:
|
||||
|
||||
```bash
|
||||
# Coming soon
|
||||
npx @soulcraft/brainy-migrate
|
||||
|
||||
# Will scan your code and automatically update:
|
||||
# - Option names
|
||||
# - TypeScript types
|
||||
# - Import patterns
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- **API Documentation:** [https://brainy.dev/docs/api/import](https://brainy.dev/docs/api/import)
|
||||
- **Examples:** [examples/import-excel/](../../examples/import-excel/)
|
||||
- **Changelog:** [CHANGELOG.md](../../CHANGELOG.md)
|
||||
- **Support:** [GitHub Issues](https://github.com/soulcraft/brainy/issues)
|
||||
|
||||
---
|
||||
|
||||
## 💬 Need Help?
|
||||
|
||||
If you're stuck migrating:
|
||||
|
||||
1. Check the error message - it includes migration hints
|
||||
2. Review the examples in this guide
|
||||
3. Open an issue on GitHub with your use case
|
||||
4. Join our Discord community for real-time help
|
||||
|
||||
---
|
||||
|
||||
**Happy migrating! 🎉**
|
||||
Loading…
Add table
Add a link
Reference in a new issue