feat: add intelligent import for CSV, Excel, and PDF files

Add IntelligentImportAugmentation with support for:
- CSV: auto-detection of encoding, delimiters, and field types
- Excel: multi-sheet extraction with metadata preservation
- PDF: text extraction, table detection, and metadata extraction

Features:
- Automatic format detection from file extension or content
- Intelligent type inference (string, number, boolean, date)
- Seamless integration with neural entity extraction
- Production-ready with 69 comprehensive tests

Dependencies added:
- xlsx@^0.18.5 for Excel parsing
- pdfjs-dist@^4.0.379 for PDF parsing
- csv-parse@^6.1.0 for CSV parsing
- chardet@^2.0.0 for encoding detection

Documentation:
- Updated README with import examples
- Updated API_REFERENCE with comprehensive import() docs
- Updated import-anything.md guide
- Added working example: import-excel-pdf-csv.ts
- Updated augmentations README
This commit is contained in:
David Snelling 2025-10-01 16:51:03 -07:00
parent aaf8e0f411
commit 814cbb48ee
33 changed files with 4664 additions and 28 deletions

View file

@ -457,6 +457,41 @@ npm run download-models # Download Q8 model
npm run download-models:q8 # Download Q8 model
```
## 🚀 Import Anything - Files, Data, URLs
Brainy's universal import intelligently handles **any data format**:
```javascript
// Import CSV with auto-detection
await brain.import('customers.csv')
// ✨ Auto-detects: encoding, delimiter, types, creates entities!
// Import Excel workbooks with multi-sheet support
await brain.import('sales-data.xlsx', {
excelSheets: ['Q1', 'Q2'] // or 'all' for all sheets
})
// ✨ Processes all sheets, preserves structure, infers types!
// Import PDF documents with table extraction
await brain.import('research-paper.pdf', {
pdfExtractTables: true
})
// ✨ Extracts text, detects tables, preserves metadata!
// Import JSON/YAML data
await brain.import([
{ name: 'Alice', role: 'Engineer' },
{ name: 'Bob', role: 'Designer' }
])
// ✨ Automatically creates Person entities with relationships!
// Import from URLs (auto-fetched)
await brain.import('https://api.example.com/data.json')
// ✨ Auto-detects URL, fetches, parses, processes!
```
**📖 [Complete Import Guide →](docs/guides/import-anything.md)** | **[Live Example →](examples/import-excel-pdf-csv.ts)**
## 📚 Core API
### `search()` - Vector Similarity
@ -513,6 +548,11 @@ await brain.deleteVerb(verbId)
// Bulk operations
await brain.import(arrayOfData)
const exported = await brain.export({format: 'json'})
// Import from CSV, Excel, PDF files (auto-detected)
await brain.import('customers.csv') // CSV with encoding detection
await brain.import('sales-report.xlsx') // Excel with multi-sheet support
await brain.import('research.pdf') // PDF with table extraction
```
## 🌐 Distributed System (NEW!)