feat: add intelligent import for CSV, Excel, and PDF files
Add IntelligentImportAugmentation with support for: - CSV: auto-detection of encoding, delimiters, and field types - Excel: multi-sheet extraction with metadata preservation - PDF: text extraction, table detection, and metadata extraction Features: - Automatic format detection from file extension or content - Intelligent type inference (string, number, boolean, date) - Seamless integration with neural entity extraction - Production-ready with 69 comprehensive tests Dependencies added: - xlsx@^0.18.5 for Excel parsing - pdfjs-dist@^4.0.379 for PDF parsing - csv-parse@^6.1.0 for CSV parsing - chardet@^2.0.0 for encoding detection Documentation: - Updated README with import examples - Updated API_REFERENCE with comprehensive import() docs - Updated import-anything.md guide - Added working example: import-excel-pdf-csv.ts - Updated augmentations README
This commit is contained in:
parent
aaf8e0f411
commit
814cbb48ee
33 changed files with 4664 additions and 28 deletions
|
|
@ -683,26 +683,75 @@ await data.clear({
|
|||
---
|
||||
|
||||
### `async import(params): Promise<ImportResult>`
|
||||
Imports data from various formats.
|
||||
Imports data from various formats with automatic detection.
|
||||
|
||||
**Parameters:**
|
||||
- `data` (required) - Data to import
|
||||
- `format` (required) - 'json' | 'csv' | 'parquet'
|
||||
- `mapping` - Field mapping
|
||||
- `batchSize` - Import batch size
|
||||
- `validate` - Validate items
|
||||
- `data` (required) - Data to import (Buffer, string, array, or file path)
|
||||
- `format` - 'auto' | 'json' | 'csv' | 'excel' | 'pdf' | 'yaml' | 'text' (default: 'auto')
|
||||
- `mapping` - Field mapping configuration
|
||||
- `batchSize` - Import batch size (default: 50)
|
||||
- `validate` - Validate items before import
|
||||
- `relationships` - Extract relationships automatically (default: true)
|
||||
|
||||
**Example:**
|
||||
**CSV-specific options:**
|
||||
- `csvDelimiter` - Column delimiter (auto-detected if not specified)
|
||||
- `csvHeaders` - First row contains headers (default: true)
|
||||
- `encoding` - Character encoding (auto-detected if not specified)
|
||||
|
||||
**Excel-specific options:**
|
||||
- `excelSheets` - Sheet names array or 'all' for all sheets
|
||||
|
||||
**PDF-specific options:**
|
||||
- `pdfExtractTables` - Extract tables from PDFs (default: true)
|
||||
- `pdfPreserveLayout` - Preserve text layout (default: true)
|
||||
|
||||
**Examples:**
|
||||
```typescript
|
||||
const result = await data.import({
|
||||
data: csvData,
|
||||
format: 'csv',
|
||||
mapping: {
|
||||
'name': 'title',
|
||||
'desc': 'description'
|
||||
},
|
||||
batchSize: 500
|
||||
// Import CSV file with auto-detection
|
||||
const csvResult = await brain.import('customers.csv')
|
||||
// Auto-detects: format, encoding, delimiter, field types
|
||||
|
||||
// Import Excel workbook
|
||||
const excelResult = await brain.import('sales-data.xlsx', {
|
||||
excelSheets: ['Q1', 'Q2'] // Import specific sheets
|
||||
})
|
||||
|
||||
// Import PDF with table extraction
|
||||
const pdfResult = await brain.import('report.pdf', {
|
||||
pdfExtractTables: true
|
||||
})
|
||||
|
||||
// Import data array
|
||||
const dataResult = await brain.import([
|
||||
{ name: 'Alice', role: 'Engineer' },
|
||||
{ name: 'Bob', role: 'Designer' }
|
||||
], {
|
||||
batchSize: 100,
|
||||
relationships: true // Auto-extract relationships
|
||||
})
|
||||
|
||||
// Import with custom CSV delimiter
|
||||
const tsvResult = await brain.import('data.tsv', {
|
||||
format: 'csv',
|
||||
csvDelimiter: '\t'
|
||||
})
|
||||
```
|
||||
|
||||
**Returns:**
|
||||
```typescript
|
||||
{
|
||||
success: boolean
|
||||
imported: number // Number of items successfully imported
|
||||
failed: number // Number of items that failed
|
||||
entityIds: string[] // IDs of created entities
|
||||
metadata: {
|
||||
format: string // Detected format
|
||||
encoding?: string // Detected encoding (CSV)
|
||||
delimiter?: string // Detected delimiter (CSV)
|
||||
sheets?: string[] // Processed sheets (Excel)
|
||||
pageCount?: number // Number of pages (PDF)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ Brainy's import is **ONE magical method** that understands EVERYTHING:
|
|||
- 📊 Data (objects, arrays, strings)
|
||||
- 📁 Files (auto-detects by path)
|
||||
- 🌐 URLs (auto-fetches)
|
||||
- 📄 Formats (JSON, CSV, YAML, text - all auto-detected)
|
||||
- 📄 Formats (JSON, CSV, Excel, PDF, YAML, text - all auto-detected)
|
||||
|
||||
## The Ultimate Simplicity
|
||||
|
||||
|
|
@ -36,7 +36,7 @@ await brain.import(people)
|
|||
```javascript
|
||||
// From file? Just pass the path!
|
||||
await brain.import('customers.csv')
|
||||
// ✨ Auto-detects it's a file, parses CSV, creates entities!
|
||||
// ✨ Auto-detects encoding, delimiter, types - creates entities!
|
||||
|
||||
// Or pass CSV content directly
|
||||
const csv = `name,age,city
|
||||
|
|
@ -47,6 +47,32 @@ await brain.import(csv, { format: 'csv' })
|
|||
// ✨ Smart CSV parsing handles quotes, escapes, everything!
|
||||
```
|
||||
|
||||
### 📊 Import Excel - Multi-Sheet Support
|
||||
```javascript
|
||||
// Import entire Excel workbook
|
||||
await brain.import('sales-report.xlsx')
|
||||
// ✨ Processes all sheets, preserves structure, infers types!
|
||||
|
||||
// Or specific sheets only
|
||||
await brain.import('data.xlsx', {
|
||||
excelSheets: ['Customers', 'Orders']
|
||||
})
|
||||
// ✨ Multi-sheet data becomes interconnected entities!
|
||||
```
|
||||
|
||||
### 📑 Import PDF - Text & Tables
|
||||
```javascript
|
||||
// Import PDF documents
|
||||
await brain.import('research-paper.pdf')
|
||||
// ✨ Extracts text, detects tables, preserves metadata!
|
||||
|
||||
// With table extraction
|
||||
await brain.import('report.pdf', {
|
||||
pdfExtractTables: true
|
||||
})
|
||||
// ✨ Converts PDF tables to structured data automatically!
|
||||
```
|
||||
|
||||
### 📝 Import YAML - File or String
|
||||
```javascript
|
||||
// From file? Auto-detected!
|
||||
|
|
@ -92,11 +118,12 @@ await brain.import(article, { format: 'text' })
|
|||
|
||||
When you import data, Brainy:
|
||||
|
||||
1. **Auto-detects format** - JSON, CSV, YAML, text, or by file extension
|
||||
2. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
|
||||
3. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
|
||||
4. **Creates embeddings** - Makes everything semantically searchable
|
||||
5. **Indexes metadata** - Enables lightning-fast filtering
|
||||
1. **Auto-detects format** - CSV, Excel, PDF, JSON, YAML, text, or by file extension
|
||||
2. **Intelligent parsing** - CSV (encoding/delimiter detection), Excel (multi-sheet), PDF (text/tables)
|
||||
3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
|
||||
4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
|
||||
5. **Creates embeddings** - Makes everything semantically searchable
|
||||
6. **Indexes metadata** - Enables lightning-fast filtering
|
||||
|
||||
## Intelligent Type Detection
|
||||
|
||||
|
|
@ -160,9 +187,21 @@ Everything works with zero config, but you can customize:
|
|||
|
||||
```javascript
|
||||
await brain.import(data, {
|
||||
format: 'auto', // 'json' | 'csv' | 'yaml' | 'text' | 'auto'
|
||||
batchSize: 50, // Process in batches
|
||||
relationships: true // Extract relationships (default: true)
|
||||
format: 'auto', // 'csv' | 'excel' | 'pdf' | 'json' | 'yaml' | 'text' | 'auto'
|
||||
batchSize: 50, // Process in batches
|
||||
relationships: true, // Extract relationships (default: true)
|
||||
|
||||
// CSV-specific options
|
||||
csvDelimiter: ',', // Auto-detected if not specified
|
||||
csvHeaders: true, // First row is headers
|
||||
encoding: 'utf-8', // Auto-detected if not specified
|
||||
|
||||
// Excel-specific options
|
||||
excelSheets: ['Sheet1'], // or 'all' for all sheets
|
||||
|
||||
// PDF-specific options
|
||||
pdfExtractTables: true, // Extract tables from PDFs
|
||||
pdfPreserveLayout: true // Preserve text layout
|
||||
})
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue