feat: add import API validation and v4.x migration guide
Add comprehensive migration support for v4.x import API changes: - Runtime validation that rejects deprecated v3.x options with clear errors - Configurable error verbosity (detailed in dev, concise in prod) - Complete migration guide (docs/guides/migrating-to-v4.md) - Enhanced CHANGELOG with breaking changes documentation - TypeScript type safety using 'never' types for deprecated options - Comprehensive JSDoc with deprecation warnings and examples Deprecated v3.x options now throw helpful errors: - extractRelationships → enableRelationshipInference - createFileStructure → vfsPath - autoDetect → (removed - always enabled) - excelSheets → (removed - all sheets processed) - pdfExtractTables → (removed - always enabled) Resolves Workshop team import issues where incorrect option names disabled all intelligent features, causing generic entity creation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
1001af9a34
commit
a1a0576d04
4 changed files with 822 additions and 24 deletions
|
|
@ -1863,33 +1863,91 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
|
||||
/**
|
||||
* Import files with auto-detection and dual storage (VFS + Knowledge Graph)
|
||||
* Import files with intelligent extraction and dual storage (VFS + Knowledge Graph)
|
||||
*
|
||||
* Unified import system that:
|
||||
* - Auto-detects format (Excel, PDF, CSV, JSON, Markdown)
|
||||
* - Extracts entities and relationships
|
||||
* - Extracts entities with AI-powered name/type detection
|
||||
* - Infers semantic relationships from context
|
||||
* - Stores in both VFS (organized files) and Knowledge Graph (connected entities)
|
||||
* - Links VFS files to graph entities
|
||||
*
|
||||
* @example
|
||||
* // Import from file path
|
||||
* const result = await brain.import('/path/to/file.xlsx')
|
||||
* @since 4.0.0
|
||||
*
|
||||
* @example
|
||||
* // Import from buffer
|
||||
* @example Quick Start (All AI features enabled by default)
|
||||
* ```typescript
|
||||
* const result = await brain.import('./glossary.xlsx')
|
||||
* // Auto-detects format, extracts entities, infers relationships
|
||||
* ```
|
||||
*
|
||||
* @example Full-Featured Import (v4.x)
|
||||
* ```typescript
|
||||
* const result = await brain.import('./data.xlsx', {
|
||||
* // AI features
|
||||
* enableNeuralExtraction: true, // Extract entity names/metadata
|
||||
* enableRelationshipInference: true, // Detect semantic relationships
|
||||
* enableConceptExtraction: true, // Extract types/concepts
|
||||
*
|
||||
* // VFS features
|
||||
* vfsPath: '/imports/my-data', // Store in VFS directory
|
||||
* groupBy: 'type', // Organize by entity type
|
||||
* preserveSource: true, // Keep original file
|
||||
*
|
||||
* // Progress tracking
|
||||
* onProgress: (p) => console.log(p.message)
|
||||
* })
|
||||
* ```
|
||||
*
|
||||
* @example Performance Tuning (Large Files)
|
||||
* ```typescript
|
||||
* const result = await brain.import('./huge-file.csv', {
|
||||
* enableDeduplication: false, // Skip dedup for speed
|
||||
* confidenceThreshold: 0.8, // Higher threshold = fewer entities
|
||||
* onProgress: (p) => console.log(`${p.processed}/${p.total}`)
|
||||
* })
|
||||
* ```
|
||||
*
|
||||
* @example Import from Buffer or Object
|
||||
* ```typescript
|
||||
* // From buffer
|
||||
* const result = await brain.import(buffer, { format: 'pdf' })
|
||||
*
|
||||
* @example
|
||||
* // Import JSON object
|
||||
* // From object
|
||||
* const result = await brain.import({ entities: [...] })
|
||||
* ```
|
||||
*
|
||||
* @example
|
||||
* // Custom VFS path and grouping
|
||||
* const result = await brain.import(buffer, {
|
||||
* vfsPath: '/my-imports/data',
|
||||
* groupBy: 'type',
|
||||
* onProgress: (progress) => console.log(progress.message)
|
||||
* })
|
||||
* @throws {Error} If invalid options are provided (v4.x breaking changes)
|
||||
*
|
||||
* @see {@link https://brainy.dev/docs/api/import API Documentation}
|
||||
* @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
|
||||
*
|
||||
* @remarks
|
||||
* **⚠️ Breaking Changes from v3.x:**
|
||||
*
|
||||
* The import API was redesigned in v4.0.0 for clarity and better feature control.
|
||||
* Old v3.x option names are **no longer recognized** and will throw errors.
|
||||
*
|
||||
* **Option Changes:**
|
||||
* - ❌ `extractRelationships` → ✅ `enableRelationshipInference`
|
||||
* - ❌ `createFileStructure` → ✅ `vfsPath: '/your/path'`
|
||||
* - ❌ `autoDetect` → ✅ *(removed - always enabled)*
|
||||
* - ❌ `excelSheets` → ✅ *(removed - all sheets processed)*
|
||||
* - ❌ `pdfExtractTables` → ✅ *(removed - always enabled)*
|
||||
*
|
||||
* **New Options:**
|
||||
* - ✅ `enableNeuralExtraction` - Extract entity names via AI
|
||||
* - ✅ `enableConceptExtraction` - Extract entity types via AI
|
||||
* - ✅ `preserveSource` - Save original file in VFS
|
||||
*
|
||||
* **If you get an error:**
|
||||
* The error message includes migration instructions and examples.
|
||||
* See the complete migration guide for all details.
|
||||
*
|
||||
* **Why these changes?**
|
||||
* - Clearer option names (explicitly describe what they do)
|
||||
* - Separation of concerns (neural, relationships, VFS are separate)
|
||||
* - Better defaults (AI features enabled by default)
|
||||
* - Reduced confusion (removed redundant options)
|
||||
*/
|
||||
async import(
|
||||
source: Buffer | string | object,
|
||||
|
|
|
|||
|
|
@ -36,7 +36,10 @@ export interface ImportSource {
|
|||
filename?: string
|
||||
}
|
||||
|
||||
export interface ImportOptions {
|
||||
/**
|
||||
* Valid import options for v4.x
|
||||
*/
|
||||
export interface ValidImportOptions {
|
||||
/** Force specific format (skip auto-detection) */
|
||||
format?: SupportedFormat
|
||||
|
||||
|
|
@ -86,6 +89,51 @@ export interface ImportOptions {
|
|||
onProgress?: (progress: ImportProgress) => void
|
||||
}
|
||||
|
||||
/**
|
||||
* Deprecated import options from v3.x
|
||||
* Using these will cause TypeScript compile errors
|
||||
*
|
||||
* @deprecated These options are no longer supported in v4.x
|
||||
* @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
|
||||
*/
|
||||
export interface DeprecatedImportOptions {
|
||||
/**
|
||||
* @deprecated Use `enableRelationshipInference` instead
|
||||
* @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
|
||||
*/
|
||||
extractRelationships?: never
|
||||
|
||||
/**
|
||||
* @deprecated Removed in v4.x - auto-detection is now always enabled
|
||||
* @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
|
||||
*/
|
||||
autoDetect?: never
|
||||
|
||||
/**
|
||||
* @deprecated Use `vfsPath` to specify the directory path instead
|
||||
* @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
|
||||
*/
|
||||
createFileStructure?: never
|
||||
|
||||
/**
|
||||
* @deprecated Removed in v4.x - all sheets are now processed automatically
|
||||
* @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
|
||||
*/
|
||||
excelSheets?: never
|
||||
|
||||
/**
|
||||
* @deprecated Removed in v4.x - table extraction is now automatic for PDF imports
|
||||
* @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
|
||||
*/
|
||||
pdfExtractTables?: never
|
||||
}
|
||||
|
||||
/**
|
||||
* Complete import options interface
|
||||
* Combines valid v4.x options with deprecated v3.x options (which cause TypeScript errors)
|
||||
*/
|
||||
export type ImportOptions = ValidImportOptions & DeprecatedImportOptions
|
||||
|
||||
export interface ImportProgress {
|
||||
stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'relationships' | 'complete'
|
||||
/** Phase of import - extraction or relationship building (v3.49.0) */
|
||||
|
|
@ -211,6 +259,9 @@ export class ImportCoordinator {
|
|||
const startTime = Date.now()
|
||||
const importId = uuidv4()
|
||||
|
||||
// Validate options (v4.0.0+: Reject deprecated v3.x options)
|
||||
this.validateOptions(options)
|
||||
|
||||
// Normalize source
|
||||
const normalizedSource = this.normalizeSource(source, options.format)
|
||||
|
||||
|
|
@ -800,4 +851,114 @@ export class ImportCoordinator {
|
|||
// Fallback: return as-is
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate options and reject deprecated v3.x options (v4.0.0+)
|
||||
* Throws clear errors with migration guidance
|
||||
*/
|
||||
private validateOptions(options: any): void {
|
||||
const invalidOptions: Array<{ old: string; new: string; message: string }> = []
|
||||
|
||||
// Check for v3.x deprecated options
|
||||
if ('extractRelationships' in options) {
|
||||
invalidOptions.push({
|
||||
old: 'extractRelationships',
|
||||
new: 'enableRelationshipInference',
|
||||
message: 'Option renamed for clarity in v4.x - explicitly indicates AI-powered relationship inference'
|
||||
})
|
||||
}
|
||||
|
||||
if ('autoDetect' in options) {
|
||||
invalidOptions.push({
|
||||
old: 'autoDetect',
|
||||
new: '(removed)',
|
||||
message: 'Auto-detection is now always enabled - no need to specify this option'
|
||||
})
|
||||
}
|
||||
|
||||
if ('createFileStructure' in options) {
|
||||
invalidOptions.push({
|
||||
old: 'createFileStructure',
|
||||
new: 'vfsPath',
|
||||
message: 'Use vfsPath to explicitly specify the virtual filesystem directory path'
|
||||
})
|
||||
}
|
||||
|
||||
if ('excelSheets' in options) {
|
||||
invalidOptions.push({
|
||||
old: 'excelSheets',
|
||||
new: '(removed)',
|
||||
message: 'All sheets are now processed automatically - no configuration needed'
|
||||
})
|
||||
}
|
||||
|
||||
if ('pdfExtractTables' in options) {
|
||||
invalidOptions.push({
|
||||
old: 'pdfExtractTables',
|
||||
new: '(removed)',
|
||||
message: 'Table extraction is now automatic for PDF imports'
|
||||
})
|
||||
}
|
||||
|
||||
// If invalid options found, throw error with detailed message
|
||||
if (invalidOptions.length > 0) {
|
||||
const errorMessage = this.buildValidationErrorMessage(invalidOptions)
|
||||
throw new Error(errorMessage)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build detailed error message for invalid options
|
||||
* Respects LOG_LEVEL for verbosity (detailed in dev, concise in prod)
|
||||
*/
|
||||
private buildValidationErrorMessage(
|
||||
invalidOptions: Array<{ old: string; new: string; message: string }>
|
||||
): string {
|
||||
// Check environment for verbosity level
|
||||
const verbose =
|
||||
process.env.LOG_LEVEL === 'debug' ||
|
||||
process.env.LOG_LEVEL === 'verbose' ||
|
||||
process.env.NODE_ENV === 'development' ||
|
||||
process.env.NODE_ENV === 'dev'
|
||||
|
||||
if (verbose) {
|
||||
// DETAILED mode (development)
|
||||
const optionDetails = invalidOptions
|
||||
.map(
|
||||
(opt) => `
|
||||
❌ ${opt.old}
|
||||
→ Use: ${opt.new}
|
||||
→ Why: ${opt.message}`
|
||||
)
|
||||
.join('\n')
|
||||
|
||||
return `
|
||||
❌ Invalid import options detected (Brainy v4.x breaking changes)
|
||||
|
||||
The following v3.x options are no longer supported:
|
||||
${optionDetails}
|
||||
|
||||
📖 Migration Guide: https://brainy.dev/docs/guides/migrating-to-v4
|
||||
💡 Quick Fix Examples:
|
||||
|
||||
Before (v3.x):
|
||||
await brain.import(file, {
|
||||
extractRelationships: true,
|
||||
createFileStructure: true
|
||||
})
|
||||
|
||||
After (v4.x):
|
||||
await brain.import(file, {
|
||||
enableRelationshipInference: true,
|
||||
vfsPath: '/imports/my-data'
|
||||
})
|
||||
|
||||
🔗 Full API docs: https://brainy.dev/docs/api/import
|
||||
`.trim()
|
||||
} else {
|
||||
// CONCISE mode (production)
|
||||
const optionsList = invalidOptions.map((o) => `'${o.old}'`).join(', ')
|
||||
return `Invalid import options: ${optionsList}. See https://brainy.dev/docs/guides/migrating-to-v4`
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue