feat: add intelligent import for CSV, Excel, and PDF files

Add IntelligentImportAugmentation with support for:
- CSV: auto-detection of encoding, delimiters, and field types
- Excel: multi-sheet extraction with metadata preservation
- PDF: text extraction, table detection, and metadata extraction

Features:
- Automatic format detection from file extension or content
- Intelligent type inference (string, number, boolean, date)
- Seamless integration with neural entity extraction
- Production-ready with 69 comprehensive tests

Dependencies added:
- xlsx@^0.18.5 for Excel parsing
- pdfjs-dist@^4.0.379 for PDF parsing
- csv-parse@^6.1.0 for CSV parsing
- chardet@^2.0.0 for encoding detection

Documentation:
- Updated README with import examples
- Updated API_REFERENCE with comprehensive import() docs
- Updated import-anything.md guide
- Added working example: import-excel-pdf-csv.ts
- Updated augmentations README
This commit is contained in:
David Snelling 2025-10-01 16:51:03 -07:00
parent aaf8e0f411
commit 814cbb48ee
33 changed files with 4664 additions and 28 deletions

View file

@ -10,6 +10,47 @@ Brainy's functionality in various ways.
## Available Augmentations
### Core Augmentations
#### IntelligentImportAugmentation
Automatically detects and processes CSV, Excel, and PDF files with intelligent extraction. This augmentation is enabled by default and provides:
- **CSV Support**: Auto-detection of encoding, delimiters, and field types
- **Excel Support**: Multi-sheet extraction with metadata preservation
- **PDF Support**: Text extraction, table detection, and metadata extraction
- **Type Inference**: Automatically infers data types (string, number, boolean, date)
- **Neural Integration**: Seamlessly integrates with entity extraction and relationship detection
```javascript
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({
intelligentImport: {
enableCSV: true,
enableExcel: true,
enablePDF: true,
maxFileSize: 100 * 1024 * 1024 // 100MB
}
})
await brain.init()
// Import CSV with auto-detection
await brain.import('customers.csv')
// Import Excel with specific sheets
await brain.import('sales-data.xlsx', {
excelSheets: ['Q1', 'Q2']
})
// Import PDF with table extraction
await brain.import('report.pdf', {
pdfExtractTables: true
})
```
**See:** [Import Anything Guide](../../docs/guides/import-anything.md) | [Example](../../examples/import-excel-pdf-csv.ts)
### Conduit Augmentations
Conduit augmentations provide data synchronization between Brainy instances.