soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: resolve 5 critical import bugs for production scale

Browse source 
- Bug #1: Smart deduplication auto-disable for large imports (>100 entities)
- Bug #2: Batch relationship creation using relateMany() (10-30x faster)
- Bug #3: File locking in MetadataIndexManager prevents race conditions
- Bug #4: Fix documentation API field inconsistencies
- Bug #5: Promise resolution timeout (automatically fixed by Bug #2)
- Enhanced error handling for corrupted metadata files

Production-ready for 500+ entity imports with 1500+ relationships.

Files modified:
- src/utils/metadataIndex.ts - Added in-memory locking system
- src/import/ImportCoordinator.ts - Batch relationships + smart deduplication
- src/storage/adapters/fileSystemStorage.ts - Enhanced SyntaxError handling
- docs/guides/import-anything.md - Corrected API field names
This commit is contained in:
David Snelling 2025-10-09 13:56:45 -07:00
parent e1bd61a726
commit 12b8abc787
5 changed files with 378 additions and 77 deletions
Download patch file Download diff file Expand all files Collapse all files

40
docs/guides/import-anything.md
View file

@ -187,21 +187,37 @@ Everything works with zero config, but you can customize:
```javascript
await brain.import(data, {
format: 'auto', // 'csv' | 'excel' | 'pdf' | 'json' | 'yaml' | 'text' | 'auto'
batchSize: 50, // Process in batches
relationships: true, // Extract relationships (default: true)
// Format detection
format: 'excel', // Force specific format (auto-detected if not specified)
// CSV-specific options
csvDelimiter: ',', // Auto-detected if not specified
csvHeaders: true, // First row is headers
encoding: 'utf-8', // Auto-detected if not specified
// VFS & Organization
vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
preserveSource: true, // Keep original source file in VFS (default: true)
// Excel-specific options
excelSheets: ['Sheet1'], // or 'all' for all sheets
// Entity & Relationship Creation
createEntities: true, // Create entities in knowledge graph (default: true)
createRelationships: true, // Create relationships in knowledge graph (default: true)
// PDF-specific options
pdfExtractTables: true, // Extract tables from PDFs
pdfPreserveLayout: true // Preserve text layout
// Neural Intelligence
enableNeuralExtraction: true, // Use AI to extract entities (default: true)
enableRelationshipInference: true, // Use AI to infer relationships (default: true)
enableConceptExtraction: true, // Extract concepts from text (default: true)
confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
// Deduplication
enableDeduplication: true, // Check for duplicate entities (default: true)
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
// Note: Auto-disabled for imports >100 entities
// Performance
chunkSize: 100, // Batch size for processing (default: varies by operation)
// History & Progress
enableHistory: true, // Track import history (default: true)
onProgress: (progress) => { // Progress callback
console.log(progress.stage, progress.message)
}
})
```