brainy/docs/guides/import-progress-examples.md

Import Progress - Usage Examples

How to Use Progress Tracking in Your Applications

Brainy v4.5.0+ provides real-time progress tracking for all 7 supported file formats (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).

⚠️ KEY FEATURE: The progress API is 100% standardized. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See Standard Import Progress API for the complete interface documentation.

---

🚀 Quick Start

Basic Progress Tracking

import { Brainy } from '@soulcraft/brainy'
import * as fs from 'fs'

const brain = await Brainy.create()

// Import with progress tracking
const result = await brain.import(fs.readFileSync('large-file.xlsx'), {
  onProgress: (progress) => {
    console.log(`Progress: ${progress.stage}`)
    console.log(`  Message: ${progress.message}`)
    console.log(`  Entities: ${progress.entities || 0}`)
    console.log(`  Relationships: ${progress.relationships || 0}`)
  }
})

console.log(`Import complete: ${result.entities.length} entities created`)

Expected Output:

Progress: detecting
  Message: Detecting format...
  Entities: 0
  Relationships: 0
Progress: extracting
  Message: Loading Excel workbook...
  Entities: 0
  Relationships: 0
Progress: extracting
  Message: Reading sheet: Sales (1/3)
  Entities: 0
  Relationships: 0
Progress: extracting
  Message: Parsing Excel (33%)
  Entities: 0
  Relationships: 0
Progress: extracting
  Message: Reading sheet: Products (2/3)
  Entities: 0
  Relationships: 0
... (more progress updates)
Progress: complete
  Message: Import complete
  Entities: 1523
  Relationships: 892
Import complete: 1523 entities created

---

🎯 Universal Progress Handler (Works for ALL Formats)

The examples below show format-specific messages, but you don't need format-specific code! The ImportProgress interface is the same for all formats:

// ONE HANDLER FOR ALL FORMATS!
function universalProgressHandler(progress) {
  console.log(`[${progress.stage}] ${progress.message}`)

  if (progress.processed && progress.total) {
    console.log(`  Progress: ${progress.processed}/${progress.total}`)
  }

  if (progress.entities || progress.relationships) {
    console.log(`  Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
  }

  if (progress.throughput && progress.eta) {
    console.log(`  Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
  }
}

// Use it for ANY format!
await brain.import(csvBuffer, { onProgress: universalProgressHandler })
await brain.import(pdfBuffer, { onProgress: universalProgressHandler })
await brain.import(excelBuffer, { onProgress: universalProgressHandler })
await brain.import(jsonBuffer, { onProgress: universalProgressHandler })
await brain.import(markdownString, { onProgress: universalProgressHandler })
await brain.import(yamlBuffer, { onProgress: universalProgressHandler })
await brain.import(docxBuffer, { onProgress: universalProgressHandler })

---

📊 What Different Formats Look Like (Same Handler!)

The examples below show what messages look like for different formats using the same universal handler above.

CSV Import (Row-by-Row Progress)

await brain.import(csvBuffer, {
  format: 'csv',
  onProgress: (progress) => {
    if (progress.stage === 'extracting') {
      // CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
      console.log(progress.message)
    }
  }
})

CSV Progress Messages:

• ✅ "Detecting CSV encoding and delimiter..."
• ✅ "Parsing CSV rows (delimiter: ",")"
• ✅ "Parsed 75%" (via bytes processed)
• ✅ "Extracted 1000 rows"
• ✅ "Converting types: 5000/10000 rows..."
• ✅ "CSV processing complete: 10000 rows"

---

PDF Import (Page-by-Page Progress)

await brain.import(pdfBuffer, {
  format: 'pdf',
  onProgress: (progress) => {
    // PDF reports exact page numbers
    console.log(progress.message)
    // Example: "Processing page 5 of 23"
  }
})

PDF Progress Messages:

• ✅ "Loading PDF document..."
• ✅ "Processing 23 pages..."
• ✅ "Processing page 5 of 23"
• ✅ "Parsed 22%" (via bytes processed)
• ✅ "Extracted 156 items from PDF"
• ✅ "PDF complete: 23 pages, 156 items extracted"

---

Excel Import (Sheet-by-Sheet Progress)

await brain.import(excelBuffer, {
  format: 'excel',
  onProgress: (progress) => {
    // Excel reports sheet names
    console.log(progress.message)
    // Example: "Reading sheet: Q2 Sales (2/5)"
  }
})

Excel Progress Messages:

• ✅ "Loading Excel workbook..."
• ✅ "Processing 3 sheets..."
• ✅ "Reading sheet: Sales (1/3)"
• ✅ "Parsing Excel (33%)" (via bytes processed)
• ✅ "Extracted 5234 rows from Excel"
• ✅ "Excel complete: 3 sheets, 5234 rows"

---

JSON Import (Node Traversal)

await brain.import(jsonBuffer, {
  format: 'json',
  onProgress: (progress) => {
    // JSON reports every 10 nodes
    console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
  }
})

---

Markdown Import (Section-by-Section)

await brain.import(markdownString, {
  format: 'markdown',
  onProgress: (progress) => {
    console.log(`Section ${progress.processed}/${progress.total}`)
  }
})

---

🎯 Building Progress UI Components

React Progress Bar

function ImportProgress({ file }: { file: File }) {
  const [progress, setProgress] = useState({
    stage: 'idle',
    message: '',
    percent: 0,
    entities: 0,
    relationships: 0
  })

  const handleImport = async () => {
    const buffer = await file.arrayBuffer()

    await brain.import(Buffer.from(buffer), {
      onProgress: (p) => {
        setProgress({
          stage: p.stage,
          message: p.message,
          // Estimate percentage from stage
          percent: {
            detecting: 10,
            extracting: 50,
            'storing-vfs': 80,
            'storing-graph': 90,
            complete: 100
          }[p.stage] || 0,
          entities: p.entities || 0,
          relationships: p.relationships || 0
        })
      }
    })
  }

  return (
    <div>
      <ProgressBar value={progress.percent} />
      <p>{progress.message}</p>
      <p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
    </div>
  )
}

---

CLI Progress Spinner

import ora from 'ora'

const spinner = ora('Starting import...').start()

await brain.import(buffer, {
  onProgress: (progress) => {
    spinner.text = progress.message

    if (progress.stage === 'complete') {
      spinner.succeed(`Import complete: ${progress.entities} entities`)
    }
  }
})

CLI Output:

⠋ Detecting format...
⠙ Loading Excel workbook...
⠹ Reading sheet: Sales (1/3)
⠸ Parsing Excel (33%)
⠼ Reading sheet: Products (2/3)
...
✔ Import complete: 1523 entities

---

Progress Dashboard with ETA

let startTime = Date.now()
let lastUpdate = startTime

await brain.import(buffer, {
  onProgress: (progress) => {
    const elapsed = Date.now() - startTime
    const rate = progress.entities / (elapsed / 1000)  // entities/sec

    console.clear()
    console.log('Import Progress Dashboard')
    console.log('========================')
    console.log(`Stage: ${progress.stage}`)
    console.log(`Status: ${progress.message}`)
    console.log(`Entities: ${progress.entities}`)
    console.log(`Relationships: ${progress.relationships}`)
    console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
    console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
  }
})

---

🔧 Advanced: Format-Specific Optimization

Detecting Format to Show Appropriate Progress

const formatMessages = {
  csv: (p) => `CSV: ${p.message}`,
  pdf: (p) => `PDF: ${p.message}`,
  excel: (p) => `Excel: ${p.message}`,
  json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
  markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
  yaml: (p) => `YAML: ${p.processed} nodes`,
  docx: (p) => `DOCX: ${p.processed} paragraphs`
}

await brain.import(buffer, {
  onProgress: (progress) => {
    // Format is available in progress.stage metadata
    const message = formatMessages[detectedFormat]?.(progress) || progress.message
    console.log(message)
  }
})

---

⚡ Performance Tips

Throttle UI Updates

let lastUIUpdate = 0
const THROTTLE_MS = 100  // Update UI max once per 100ms

await brain.import(buffer, {
  onProgress: (progress) => {
    const now = Date.now()
    if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
      return  // Skip this update
    }

    lastUIUpdate = now
    updateUI(progress)  // Only update every 100ms
  }
})

Note: Brainy already throttles progress callbacks internally, but additional UI throttling can help with heavy rendering.

---

📝 Summary

✅ All 7 formats have consistent progress reporting ✅ Real-time updates during long imports (no more "0%" hangs) ✅ Contextual messages show exactly what's happening ✅ Build reliable tools with standardized progress callbacks ✅ Workshop team problem SOLVED - users see progress throughout import

Files Modified:

• 3 handlers: csvHandler.ts, pdfHandler.ts, excelHandler.ts
• 7 importers: SmartCSVImporter.ts, SmartPDFImporter.ts, SmartExcelImporter.ts, SmartJSONImporter.ts, SmartMarkdownImporter.ts, SmartYAMLImporter.ts, SmartDOCXImporter.ts

Result: Comprehensive, consistent progress tracking across ALL import formats!