brainy/docs/guides/import-progress-examples.md

# Import Progress - Usage Examples

**How to Use Progress Tracking in Your Applications**

Brainy provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).

> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation.

---

## 🚀 Quick Start

### Basic Progress Tracking

```typescript
import { Brainy } from '@soulcraft/brainy'
import * as fs from 'fs'

const brain = await Brainy.create()

// Import with progress tracking
const result = await brain.import(fs.readFileSync('large-file.xlsx'), {
 onProgress: (progress) => {
 console.log(`Progress: ${progress.stage}`)
 console.log(` Message: ${progress.message}`)
 console.log(` Entities: ${progress.entities || 0}`)
 console.log(` Relationships: ${progress.relationships || 0}`)
 }
})

console.log(`Import complete: ${result.entities.length} entities created`)
```

**Expected Output:**
```
Progress: detecting
 Message: Detecting format...
 Entities: 0
 Relationships: 0
Progress: extracting
 Message: Loading Excel workbook...
 Entities: 0
 Relationships: 0
Progress: extracting
 Message: Reading sheet: Sales (1/3)
 Entities: 0
 Relationships: 0
Progress: extracting
 Message: Parsing Excel (33%)
 Entities: 0
 Relationships: 0
Progress: extracting
 Message: Reading sheet: Products (2/3)
 Entities: 0
 Relationships: 0
... (more progress updates)
Progress: complete
 Message: Import complete
 Entities: 1523
 Relationships: 892
Import complete: 1523 entities created
```

---

## 🎯 Universal Progress Handler (Works for ALL Formats)

The examples below show format-specific messages, but **you don't need format-specific code**! The `ImportProgress` interface is the same for all formats:

```typescript
// ONE HANDLER FOR ALL FORMATS!
function universalProgressHandler(progress) {
 console.log(`[${progress.stage}] ${progress.message}`)

 if (progress.processed && progress.total) {
 console.log(` Progress: ${progress.processed}/${progress.total}`)
 }

 if (progress.entities || progress.relationships) {
 console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
 }

 if (progress.throughput && progress.eta) {
 console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
 }
}

// Use it for ANY format!
await brain.import(csvBuffer, { onProgress: universalProgressHandler })
await brain.import(pdfBuffer, { onProgress: universalProgressHandler })
await brain.import(excelBuffer, { onProgress: universalProgressHandler })
await brain.import(jsonBuffer, { onProgress: universalProgressHandler })
await brain.import(markdownString, { onProgress: universalProgressHandler })
await brain.import(yamlBuffer, { onProgress: universalProgressHandler })
await brain.import(docxBuffer, { onProgress: universalProgressHandler })
```

---

## 📊 What Different Formats Look Like (Same Handler!)

The examples below show **what messages look like** for different formats using the **same universal handler** above.

### CSV Import (Row-by-Row Progress)

```typescript
await brain.import(csvBuffer, {
 format: 'csv',
 onProgress: (progress) => {
 if (progress.stage === 'extracting') {
 // CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
 console.log(progress.message)
 }
 }
})
```

**CSV Progress Messages:**
- ✅ "Detecting CSV encoding and delimiter..."
- ✅ "Parsing CSV rows (delimiter: ",")"
- ✅ "Parsed 75%" (via bytes processed)
- ✅ "Extracted 1000 rows"
- ✅ "Converting types: 5000/10000 rows..."
- ✅ "CSV processing complete: 10000 rows"

---

### PDF Import (Page-by-Page Progress)

```typescript
await brain.import(pdfBuffer, {
 format: 'pdf',
 onProgress: (progress) => {
 // PDF reports exact page numbers
 console.log(progress.message)
 // Example: "Processing page 5 of 23"
 }
})
```

**PDF Progress Messages:**
- ✅ "Loading PDF document..."
- ✅ "Processing 23 pages..."
- ✅ "Processing page 5 of 23"
- ✅ "Parsed 22%" (via bytes processed)
- ✅ "Extracted 156 items from PDF"
- ✅ "PDF complete: 23 pages, 156 items extracted"

---

### Excel Import (Sheet-by-Sheet Progress)

```typescript
await brain.import(excelBuffer, {
 format: 'excel',
 onProgress: (progress) => {
 // Excel reports sheet names
 console.log(progress.message)
 // Example: "Reading sheet: Q2 Sales (2/5)"
 }
})
```

**Excel Progress Messages:**
- ✅ "Loading Excel workbook..."
- ✅ "Processing 3 sheets..."
- ✅ "Reading sheet: Sales (1/3)"
- ✅ "Parsing Excel (33%)" (via bytes processed)
- ✅ "Extracted 5234 rows from Excel"
- ✅ "Excel complete: 3 sheets, 5234 rows"

---

### JSON Import (Node Traversal)

```typescript
await brain.import(jsonBuffer, {
 format: 'json',
 onProgress: (progress) => {
 // JSON reports every 10 nodes
 console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
 }
})
```

---

### Markdown Import (Section-by-Section)

```typescript
await brain.import(markdownString, {
 format: 'markdown',
 onProgress: (progress) => {
 console.log(`Section ${progress.processed}/${progress.total}`)
 }
})
```

---

## 🎯 Building Progress UI Components

### React Progress Bar

```typescript
function ImportProgress({ file }: { file: File }) {
 const [progress, setProgress] = useState({
 stage: 'idle',
 message: '',
 percent: 0,
 entities: 0,
 relationships: 0
 })

 const handleImport = async () => {
 const buffer = await file.arrayBuffer()

 await brain.import(Buffer.from(buffer), {
 onProgress: (p) => {
 setProgress({
 stage: p.stage,
 message: p.message,
 // Estimate percentage from stage
 percent: {
 detecting: 10,
 extracting: 50,
 'storing-vfs': 80,
 'storing-graph': 90,
 complete: 100
 }[p.stage] || 0,
 entities: p.entities || 0,
 relationships: p.relationships || 0
 })
 }
 })
 }

 return (
 <div>
 <ProgressBar value={progress.percent} />
 <p>{progress.message}</p>
 <p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
 </div>
 )
}
```

---

### CLI Progress Spinner

```typescript
import ora from 'ora'

const spinner = ora('Starting import...').start()

await brain.import(buffer, {
 onProgress: (progress) => {
 spinner.text = progress.message

 if (progress.stage === 'complete') {
 spinner.succeed(`Import complete: ${progress.entities} entities`)
 }
 }
})
```

**CLI Output:**
```
⠋ Detecting format...
⠙ Loading Excel workbook...
⠹ Reading sheet: Sales (1/3)
⠸ Parsing Excel (33%)
⠼ Reading sheet: Products (2/3)
...
✔ Import complete: 1523 entities
```

---

### Progress Dashboard with ETA

```typescript
let startTime = Date.now()
let lastUpdate = startTime

await brain.import(buffer, {
 onProgress: (progress) => {
 const elapsed = Date.now() - startTime
 const rate = progress.entities / (elapsed / 1000) // entities/sec

 console.clear()
 console.log('Import Progress Dashboard')
 console.log('========================')
 console.log(`Stage: ${progress.stage}`)
 console.log(`Status: ${progress.message}`)
 console.log(`Entities: ${progress.entities}`)
 console.log(`Relationships: ${progress.relationships}`)
 console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
 console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
 }
})
```

---

## 🔧 Advanced: Format-Specific Optimization

### Detecting Format to Show Appropriate Progress

```typescript
const formatMessages = {
 csv: (p) => `CSV: ${p.message}`,
 pdf: (p) => `PDF: ${p.message}`,
 excel: (p) => `Excel: ${p.message}`,
 json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
 markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
 yaml: (p) => `YAML: ${p.processed} nodes`,
 docx: (p) => `DOCX: ${p.processed} paragraphs`
}

await brain.import(buffer, {
 onProgress: (progress) => {
 // Format is available in progress.stage metadata
 const message = formatMessages[detectedFormat]?.(progress) || progress.message
 console.log(message)
 }
})
```

---

## ⚡ Performance Tips

### Throttle UI Updates

```typescript
let lastUIUpdate = 0
const THROTTLE_MS = 100 // Update UI max once per 100ms

await brain.import(buffer, {
 onProgress: (progress) => {
 const now = Date.now()
 if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
 return // Skip this update
 }

 lastUIUpdate = now
 updateUI(progress) // Only update every 100ms
 }
})
```

**Note:** Brainy already throttles progress callbacks internally, but additional UI throttling can help with heavy rendering.

---

## 📝 Summary

✅ **All 7 formats** have consistent progress reporting
✅ **Real-time updates** during long imports (no more "0%" hangs)
✅ **Contextual messages** show exactly what's happening
✅ **Build reliable tools** with standardized progress callbacks
✅ **Problem SOLVED** - users see progress throughout import

**Files Modified:**
- 3 handlers: `csvHandler.ts`, `pdfHandler.ts`, `excelHandler.ts`
- 7 importers: `SmartCSVImporter.ts`, `SmartPDFImporter.ts`, `SmartExcelImporter.ts`, `SmartJSONImporter.ts`, `SmartMarkdownImporter.ts`, `SmartYAMLImporter.ts`, `SmartDOCXImporter.ts`

**Result:** Comprehensive, consistent progress tracking across ALL import formats!