fix: exclude __words__ keyword index from corruption detection and getStats()
The __words__ keyword index stores 50-5000 entries per entity (one per word), which inflated avg entries/entity well above the corruption threshold of 100. This caused: 1. validateConsistency() to falsely detect corruption on every startup, triggering unnecessary clearAllIndexData() + rebuild() cycles 2. getStats() to log false "Metadata index may be corrupted" warnings and report inflated totalEntries/totalIds stats Both methods now skip __words__ when counting, so stats and health checks reflect metadata fields only (noun, type, createdAt, etc.). Keyword search is unaffected since the __words__ field index itself is not modified.
This commit is contained in:
parent
32dbdcec61
commit
364360d447
128 changed files with 5637 additions and 5682 deletions
|
|
@ -24,8 +24,8 @@ await brain.import(anything)
|
|||
```javascript
|
||||
// Array of objects? No problem.
|
||||
const people = [
|
||||
{ name: 'Alice', role: 'Engineer', company: 'TechCorp' },
|
||||
{ name: 'Bob', role: 'Designer', company: 'TechCorp' }
|
||||
{ name: 'Alice', role: 'Engineer', company: 'TechCorp' },
|
||||
{ name: 'Bob', role: 'Designer', company: 'TechCorp' }
|
||||
]
|
||||
|
||||
await brain.import(people)
|
||||
|
|
@ -55,7 +55,7 @@ await brain.import('sales-report.xlsx')
|
|||
|
||||
// Or specific sheets only
|
||||
await brain.import('data.xlsx', {
|
||||
excelSheets: ['Customers', 'Orders']
|
||||
excelSheets: ['Customers', 'Orders']
|
||||
})
|
||||
// ✨ Multi-sheet data becomes interconnected entities!
|
||||
```
|
||||
|
|
@ -68,7 +68,7 @@ await brain.import('research-paper.pdf')
|
|||
|
||||
// With table extraction
|
||||
await brain.import('report.pdf', {
|
||||
pdfExtractTables: true
|
||||
pdfExtractTables: true
|
||||
})
|
||||
// ✨ Converts PDF tables to structured data automatically!
|
||||
```
|
||||
|
|
@ -83,16 +83,16 @@ await brain.import('config.yaml')
|
|||
const yaml = `
|
||||
project: AI Assistant
|
||||
team:
|
||||
- name: Alice
|
||||
role: Lead
|
||||
- name: Bob
|
||||
role: Dev
|
||||
- name: Alice
|
||||
role: Lead
|
||||
- name: Bob
|
||||
role: Dev
|
||||
`
|
||||
await brain.import(yaml, { format: 'yaml' })
|
||||
// ✨ Hierarchical data becomes a connected graph!
|
||||
```
|
||||
|
||||
### 📄 Import Word Documents (DOCX) - v4.2.0
|
||||
### 📄 Import Word Documents (DOCX) -
|
||||
```javascript
|
||||
// From file path
|
||||
await brain.import('research-paper.docx')
|
||||
|
|
@ -105,8 +105,8 @@ await brain.import(buffer, { format: 'docx' })
|
|||
|
||||
// With neural extraction
|
||||
await brain.import('report.docx', {
|
||||
enableNeuralExtraction: true,
|
||||
enableHierarchicalRelationships: true
|
||||
enableNeuralExtraction: true,
|
||||
enableHierarchicalRelationships: true
|
||||
})
|
||||
// ✨ Extracts entities from paragraphs and creates relationships within sections!
|
||||
```
|
||||
|
|
@ -121,25 +121,25 @@ await brain.import('https://api.example.com/data.json')
|
|||
await brain.import('https://data.gov/census.csv')
|
||||
// ✨ Fetches CSV from web, parses, imports!
|
||||
|
||||
// With authentication (v4.2.0)
|
||||
// With authentication
|
||||
await brain.import({
|
||||
type: 'url',
|
||||
data: 'https://api.example.com/private/data.xlsx',
|
||||
auth: {
|
||||
username: 'user',
|
||||
password: 'pass'
|
||||
}
|
||||
type: 'url',
|
||||
data: 'https://api.example.com/private/data.xlsx',
|
||||
auth: {
|
||||
username: 'user',
|
||||
password: 'pass'
|
||||
}
|
||||
})
|
||||
// ✨ Supports basic authentication for protected resources!
|
||||
|
||||
// With custom headers (v4.2.0)
|
||||
// With custom headers
|
||||
await brain.import({
|
||||
type: 'url',
|
||||
data: 'https://api.example.com/data.json',
|
||||
headers: {
|
||||
'Authorization': 'Bearer TOKEN',
|
||||
'X-API-Key': 'your-key'
|
||||
}
|
||||
type: 'url',
|
||||
data: 'https://api.example.com/data.json',
|
||||
headers: {
|
||||
'Authorization': 'Bearer TOKEN',
|
||||
'X-API-Key': 'your-key'
|
||||
}
|
||||
})
|
||||
// ✨ Full HTTP header customization support!
|
||||
```
|
||||
|
|
@ -163,9 +163,9 @@ When you import data, Brainy:
|
|||
2. **Intelligent parsing** - CSV (encoding/delimiter), Excel (multi-sheet), PDF (text/tables), DOCX (headings/paragraphs)
|
||||
3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
|
||||
4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
|
||||
5. **Scores confidence & weight** - Every entity and relationship gets quality metrics (v4.2.0)
|
||||
5. **Scores confidence & weight** - Every entity and relationship gets quality metrics
|
||||
6. **Creates embeddings** - Makes everything semantically searchable
|
||||
7. **Indexes metadata** - Enables lightning-fast filtering with range queries (v4.2.0)
|
||||
7. **Indexes metadata** - Enables lightning-fast filtering with range queries
|
||||
|
||||
## Intelligent Type Detection
|
||||
|
||||
|
|
@ -176,7 +176,7 @@ Brainy automatically detects what TYPE of data you're importing:
|
|||
{ name: 'John', email: 'john@example.com' }
|
||||
|
||||
// This becomes an Organization
|
||||
{ companyName: 'Acme', employees: 500 }
|
||||
{ companyName: 'Acme', employees: 500 }
|
||||
|
||||
// This becomes a Document
|
||||
{ title: 'Report', content: '...', author: 'Jane' }
|
||||
|
|
@ -193,9 +193,9 @@ Brainy finds connections in your data:
|
|||
|
||||
```javascript
|
||||
const data = [
|
||||
{ id: 'u1', name: 'Alice', managerId: 'u2' },
|
||||
{ id: 'u2', name: 'Bob', departmentId: 'd1' },
|
||||
{ id: 'd1', name: 'Engineering' }
|
||||
{ id: 'u1', name: 'Alice', managerId: 'u2' },
|
||||
{ id: 'u2', name: 'Bob', departmentId: 'd1' },
|
||||
{ id: 'd1', name: 'Engineering' }
|
||||
]
|
||||
|
||||
await brain.import(data)
|
||||
|
|
@ -204,28 +204,27 @@ await brain.import(data)
|
|||
// - Bob "memberOf" Engineering
|
||||
```
|
||||
|
||||
## Confidence & Weight Scoring - v4.2.0
|
||||
|
||||
## Confidence & Weight Scoring -
|
||||
Every entity and relationship gets confidence and weight scores:
|
||||
|
||||
```javascript
|
||||
// Import with confidence threshold
|
||||
await brain.import(data, {
|
||||
confidenceThreshold: 0.8 // Only extract entities with >80% confidence
|
||||
confidenceThreshold: 0.8 // Only extract entities with >80% confidence
|
||||
})
|
||||
|
||||
// Query high-confidence entities using range queries
|
||||
const highConfidence = await brain.find({
|
||||
where: {
|
||||
confidence: { gte: 0.8 } // Get entities with confidence >= 0.8
|
||||
}
|
||||
where: {
|
||||
confidence: { gte: 0.8 } // Get entities with confidence >= 0.8
|
||||
}
|
||||
})
|
||||
|
||||
// Range query operators: gt, gte, lt, lte, between
|
||||
const mediumConfidence = await brain.find({
|
||||
where: {
|
||||
confidence: { between: [0.6, 0.8] }
|
||||
}
|
||||
where: {
|
||||
confidence: { between: [0.6, 0.8] }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -236,24 +235,23 @@ const mediumConfidence = await brain.find({
|
|||
|
||||
**Weights** indicate importance/relevance within the document context.
|
||||
|
||||
## Per-Sheet Excel Extraction - v4.2.0
|
||||
|
||||
## Per-Sheet Excel Extraction -
|
||||
Excel files with multiple sheets can be organized by sheet:
|
||||
|
||||
```javascript
|
||||
// Group entities by sheet in VFS
|
||||
await brain.import('multi-sheet-data.xlsx', {
|
||||
groupBy: 'sheet' // Creates separate directories for each sheet
|
||||
groupBy: 'sheet' // Creates separate directories for each sheet
|
||||
})
|
||||
|
||||
// Result VFS structure:
|
||||
// /imports/data/
|
||||
// ├── Sheet1/
|
||||
// │ ├── entity1.json
|
||||
// │ └── entity2.json
|
||||
// └── Sheet2/
|
||||
// ├── entity3.json
|
||||
// └── entity4.json
|
||||
// ├── Sheet1/
|
||||
// │ ├── entity1.json
|
||||
// │ └── entity2.json
|
||||
// └── Sheet2/
|
||||
// ├── entity3.json
|
||||
// └── entity4.json
|
||||
|
||||
// Other groupBy options:
|
||||
// - 'type': Group by entity type (Person, Place, etc.)
|
||||
|
|
@ -274,9 +272,9 @@ const results = await brain.find('people in engineering who joined this year')
|
|||
|
||||
// Graph traversal + filters
|
||||
const connected = await brain.find({
|
||||
like: 'Alice',
|
||||
connected: { depth: 2 },
|
||||
where: { department: 'Engineering' }
|
||||
like: 'Alice',
|
||||
connected: { depth: 2 },
|
||||
where: { department: 'Engineering' }
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -286,37 +284,37 @@ Everything works with zero config, but you can customize:
|
|||
|
||||
```javascript
|
||||
await brain.import(data, {
|
||||
// Format detection
|
||||
format: 'excel', // Force specific format (auto-detected if not specified)
|
||||
// Format detection
|
||||
format: 'excel', // Force specific format (auto-detected if not specified)
|
||||
|
||||
// VFS & Organization
|
||||
vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
|
||||
groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
|
||||
preserveSource: true, // Keep original source file in VFS (default: true)
|
||||
// VFS & Organization
|
||||
vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
|
||||
groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
|
||||
preserveSource: true, // Keep original source file in VFS (default: true)
|
||||
|
||||
// Entity & Relationship Creation
|
||||
createEntities: true, // Create entities in knowledge graph (default: true)
|
||||
createRelationships: true, // Create relationships in knowledge graph (default: true)
|
||||
// Entity & Relationship Creation
|
||||
createEntities: true, // Create entities in knowledge graph (default: true)
|
||||
createRelationships: true, // Create relationships in knowledge graph (default: true)
|
||||
|
||||
// Neural Intelligence
|
||||
enableNeuralExtraction: true, // Use AI to extract entities (default: true)
|
||||
enableRelationshipInference: true, // Use AI to infer relationships (default: true)
|
||||
enableConceptExtraction: true, // Extract concepts from text (default: true)
|
||||
confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
|
||||
// Neural Intelligence
|
||||
enableNeuralExtraction: true, // Use AI to extract entities (default: true)
|
||||
enableRelationshipInference: true, // Use AI to infer relationships (default: true)
|
||||
enableConceptExtraction: true, // Extract concepts from text (default: true)
|
||||
confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
|
||||
|
||||
// Deduplication
|
||||
enableDeduplication: true, // Check for duplicate entities (default: true)
|
||||
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
|
||||
// Note: Auto-disabled for imports >100 entities
|
||||
// Deduplication
|
||||
enableDeduplication: true, // Check for duplicate entities (default: true)
|
||||
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
|
||||
// Note: Auto-disabled for imports >100 entities
|
||||
|
||||
// Performance
|
||||
chunkSize: 100, // Batch size for processing (default: varies by operation)
|
||||
// Performance
|
||||
chunkSize: 100, // Batch size for processing (default: varies by operation)
|
||||
|
||||
// History & Progress
|
||||
enableHistory: true, // Track import history (default: true)
|
||||
onProgress: (progress) => { // Progress callback
|
||||
console.log(progress.stage, progress.message)
|
||||
}
|
||||
// History & Progress
|
||||
enableHistory: true, // Track import history (default: true)
|
||||
onProgress: (progress) => { // Progress callback
|
||||
console.log(progress.stage, progress.message)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -343,9 +341,9 @@ const results = await brain.import(problematicData)
|
|||
### 🏢 Business Data
|
||||
```javascript
|
||||
// Import ANY source - ONE method!
|
||||
await brain.import('customers.csv') // File
|
||||
await brain.import('https://api.co/orders') // URL
|
||||
await brain.import(productsArray) // Data
|
||||
await brain.import('customers.csv') // File
|
||||
await brain.import('https://api.co/orders') // URL
|
||||
await brain.import(productsArray) // Data
|
||||
|
||||
// Now query across all of it!
|
||||
await brain.find('customers who bought products in Q4')
|
||||
|
|
@ -389,8 +387,8 @@ await brain.find('posts by users following Alice with >10 comments')
|
|||
|
||||
```javascript
|
||||
// ONE method that understands EVERYTHING:
|
||||
await brain.import(data) // Objects, arrays, strings
|
||||
await brain.import('file.csv') // Files (auto-detected)
|
||||
await brain.import(data) // Objects, arrays, strings
|
||||
await brain.import('file.csv') // Files (auto-detected)
|
||||
await brain.import('http://..') // URLs (auto-fetched)
|
||||
|
||||
// It ALWAYS knows what to do! ✨
|
||||
|
|
|
|||
|
|
@ -39,7 +39,7 @@ Each phase adds intelligence and structure to your raw data, transforming it int
|
|||
|
||||
---
|
||||
|
||||
## 🌊 Always-On Streaming Architecture (v4.2.0+)
|
||||
## 🌊 Always-On Streaming Architecture
|
||||
|
||||
All imports use streaming with **progressive flush intervals**:
|
||||
|
||||
|
|
@ -1062,7 +1062,7 @@ if (!fromEntity || !toEntity) {
|
|||
}
|
||||
```
|
||||
|
||||
#### 5.3b: Check for Duplicates (v3.43.2 Critical Fix)
|
||||
#### 5.3b: Check for Duplicates (Critical Fix)
|
||||
|
||||
**The Bug**: Without duplicate checking, re-importing would create:
|
||||
```
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
**How to Use Progress Tracking in Your Applications**
|
||||
|
||||
Brainy v4.5.0+ provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
|
||||
Brainy provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
|
||||
|
||||
> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation.
|
||||
|
||||
|
|
@ -20,12 +20,12 @@ const brain = await Brainy.create()
|
|||
|
||||
// Import with progress tracking
|
||||
const result = await brain.import(fs.readFileSync('large-file.xlsx'), {
|
||||
onProgress: (progress) => {
|
||||
console.log(`Progress: ${progress.stage}`)
|
||||
console.log(` Message: ${progress.message}`)
|
||||
console.log(` Entities: ${progress.entities || 0}`)
|
||||
console.log(` Relationships: ${progress.relationships || 0}`)
|
||||
}
|
||||
onProgress: (progress) => {
|
||||
console.log(`Progress: ${progress.stage}`)
|
||||
console.log(` Message: ${progress.message}`)
|
||||
console.log(` Entities: ${progress.entities || 0}`)
|
||||
console.log(` Relationships: ${progress.relationships || 0}`)
|
||||
}
|
||||
})
|
||||
|
||||
console.log(`Import complete: ${result.entities.length} entities created`)
|
||||
|
|
@ -34,30 +34,30 @@ console.log(`Import complete: ${result.entities.length} entities created`)
|
|||
**Expected Output:**
|
||||
```
|
||||
Progress: detecting
|
||||
Message: Detecting format...
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
Message: Detecting format...
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
Progress: extracting
|
||||
Message: Loading Excel workbook...
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
Message: Loading Excel workbook...
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
Progress: extracting
|
||||
Message: Reading sheet: Sales (1/3)
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
Message: Reading sheet: Sales (1/3)
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
Progress: extracting
|
||||
Message: Parsing Excel (33%)
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
Message: Parsing Excel (33%)
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
Progress: extracting
|
||||
Message: Reading sheet: Products (2/3)
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
Message: Reading sheet: Products (2/3)
|
||||
Entities: 0
|
||||
Relationships: 0
|
||||
... (more progress updates)
|
||||
Progress: complete
|
||||
Message: Import complete
|
||||
Entities: 1523
|
||||
Relationships: 892
|
||||
Message: Import complete
|
||||
Entities: 1523
|
||||
Relationships: 892
|
||||
Import complete: 1523 entities created
|
||||
```
|
||||
|
||||
|
|
@ -70,19 +70,19 @@ The examples below show format-specific messages, but **you don't need format-sp
|
|||
```typescript
|
||||
// ONE HANDLER FOR ALL FORMATS!
|
||||
function universalProgressHandler(progress) {
|
||||
console.log(`[${progress.stage}] ${progress.message}`)
|
||||
console.log(`[${progress.stage}] ${progress.message}`)
|
||||
|
||||
if (progress.processed && progress.total) {
|
||||
console.log(` Progress: ${progress.processed}/${progress.total}`)
|
||||
}
|
||||
if (progress.processed && progress.total) {
|
||||
console.log(` Progress: ${progress.processed}/${progress.total}`)
|
||||
}
|
||||
|
||||
if (progress.entities || progress.relationships) {
|
||||
console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
|
||||
}
|
||||
if (progress.entities || progress.relationships) {
|
||||
console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
|
||||
}
|
||||
|
||||
if (progress.throughput && progress.eta) {
|
||||
console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
|
||||
}
|
||||
if (progress.throughput && progress.eta) {
|
||||
console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
|
||||
}
|
||||
}
|
||||
|
||||
// Use it for ANY format!
|
||||
|
|
@ -105,20 +105,20 @@ The examples below show **what messages look like** for different formats using
|
|||
|
||||
```typescript
|
||||
await brain.import(csvBuffer, {
|
||||
format: 'csv',
|
||||
onProgress: (progress) => {
|
||||
if (progress.stage === 'extracting') {
|
||||
// CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
|
||||
console.log(progress.message)
|
||||
}
|
||||
}
|
||||
format: 'csv',
|
||||
onProgress: (progress) => {
|
||||
if (progress.stage === 'extracting') {
|
||||
// CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
|
||||
console.log(progress.message)
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**CSV Progress Messages:**
|
||||
- ✅ "Detecting CSV encoding and delimiter..."
|
||||
- ✅ "Parsing CSV rows (delimiter: ",")"
|
||||
- ✅ "Parsed 75%" (via bytes processed)
|
||||
- ✅ "Parsed 75%" (via bytes processed)
|
||||
- ✅ "Extracted 1000 rows"
|
||||
- ✅ "Converting types: 5000/10000 rows..."
|
||||
- ✅ "CSV processing complete: 10000 rows"
|
||||
|
|
@ -129,12 +129,12 @@ await brain.import(csvBuffer, {
|
|||
|
||||
```typescript
|
||||
await brain.import(pdfBuffer, {
|
||||
format: 'pdf',
|
||||
onProgress: (progress) => {
|
||||
// PDF reports exact page numbers
|
||||
console.log(progress.message)
|
||||
// Example: "Processing page 5 of 23"
|
||||
}
|
||||
format: 'pdf',
|
||||
onProgress: (progress) => {
|
||||
// PDF reports exact page numbers
|
||||
console.log(progress.message)
|
||||
// Example: "Processing page 5 of 23"
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -152,12 +152,12 @@ await brain.import(pdfBuffer, {
|
|||
|
||||
```typescript
|
||||
await brain.import(excelBuffer, {
|
||||
format: 'excel',
|
||||
onProgress: (progress) => {
|
||||
// Excel reports sheet names
|
||||
console.log(progress.message)
|
||||
// Example: "Reading sheet: Q2 Sales (2/5)"
|
||||
}
|
||||
format: 'excel',
|
||||
onProgress: (progress) => {
|
||||
// Excel reports sheet names
|
||||
console.log(progress.message)
|
||||
// Example: "Reading sheet: Q2 Sales (2/5)"
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -175,11 +175,11 @@ await brain.import(excelBuffer, {
|
|||
|
||||
```typescript
|
||||
await brain.import(jsonBuffer, {
|
||||
format: 'json',
|
||||
onProgress: (progress) => {
|
||||
// JSON reports every 10 nodes
|
||||
console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
|
||||
}
|
||||
format: 'json',
|
||||
onProgress: (progress) => {
|
||||
// JSON reports every 10 nodes
|
||||
console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -189,10 +189,10 @@ await brain.import(jsonBuffer, {
|
|||
|
||||
```typescript
|
||||
await brain.import(markdownString, {
|
||||
format: 'markdown',
|
||||
onProgress: (progress) => {
|
||||
console.log(`Section ${progress.processed}/${progress.total}`)
|
||||
}
|
||||
format: 'markdown',
|
||||
onProgress: (progress) => {
|
||||
console.log(`Section ${progress.processed}/${progress.total}`)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -204,44 +204,44 @@ await brain.import(markdownString, {
|
|||
|
||||
```typescript
|
||||
function ImportProgress({ file }: { file: File }) {
|
||||
const [progress, setProgress] = useState({
|
||||
stage: 'idle',
|
||||
message: '',
|
||||
percent: 0,
|
||||
entities: 0,
|
||||
relationships: 0
|
||||
})
|
||||
const [progress, setProgress] = useState({
|
||||
stage: 'idle',
|
||||
message: '',
|
||||
percent: 0,
|
||||
entities: 0,
|
||||
relationships: 0
|
||||
})
|
||||
|
||||
const handleImport = async () => {
|
||||
const buffer = await file.arrayBuffer()
|
||||
const handleImport = async () => {
|
||||
const buffer = await file.arrayBuffer()
|
||||
|
||||
await brain.import(Buffer.from(buffer), {
|
||||
onProgress: (p) => {
|
||||
setProgress({
|
||||
stage: p.stage,
|
||||
message: p.message,
|
||||
// Estimate percentage from stage
|
||||
percent: {
|
||||
detecting: 10,
|
||||
extracting: 50,
|
||||
'storing-vfs': 80,
|
||||
'storing-graph': 90,
|
||||
complete: 100
|
||||
}[p.stage] || 0,
|
||||
entities: p.entities || 0,
|
||||
relationships: p.relationships || 0
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
await brain.import(Buffer.from(buffer), {
|
||||
onProgress: (p) => {
|
||||
setProgress({
|
||||
stage: p.stage,
|
||||
message: p.message,
|
||||
// Estimate percentage from stage
|
||||
percent: {
|
||||
detecting: 10,
|
||||
extracting: 50,
|
||||
'storing-vfs': 80,
|
||||
'storing-graph': 90,
|
||||
complete: 100
|
||||
}[p.stage] || 0,
|
||||
entities: p.entities || 0,
|
||||
relationships: p.relationships || 0
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
return (
|
||||
<div>
|
||||
<ProgressBar value={progress.percent} />
|
||||
<p>{progress.message}</p>
|
||||
<p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
|
||||
</div>
|
||||
)
|
||||
return (
|
||||
<div>
|
||||
<ProgressBar value={progress.percent} />
|
||||
<p>{progress.message}</p>
|
||||
<p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -255,13 +255,13 @@ import ora from 'ora'
|
|||
const spinner = ora('Starting import...').start()
|
||||
|
||||
await brain.import(buffer, {
|
||||
onProgress: (progress) => {
|
||||
spinner.text = progress.message
|
||||
onProgress: (progress) => {
|
||||
spinner.text = progress.message
|
||||
|
||||
if (progress.stage === 'complete') {
|
||||
spinner.succeed(`Import complete: ${progress.entities} entities`)
|
||||
}
|
||||
}
|
||||
if (progress.stage === 'complete') {
|
||||
spinner.succeed(`Import complete: ${progress.entities} entities`)
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -285,20 +285,20 @@ let startTime = Date.now()
|
|||
let lastUpdate = startTime
|
||||
|
||||
await brain.import(buffer, {
|
||||
onProgress: (progress) => {
|
||||
const elapsed = Date.now() - startTime
|
||||
const rate = progress.entities / (elapsed / 1000) // entities/sec
|
||||
onProgress: (progress) => {
|
||||
const elapsed = Date.now() - startTime
|
||||
const rate = progress.entities / (elapsed / 1000) // entities/sec
|
||||
|
||||
console.clear()
|
||||
console.log('Import Progress Dashboard')
|
||||
console.log('========================')
|
||||
console.log(`Stage: ${progress.stage}`)
|
||||
console.log(`Status: ${progress.message}`)
|
||||
console.log(`Entities: ${progress.entities}`)
|
||||
console.log(`Relationships: ${progress.relationships}`)
|
||||
console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
|
||||
console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
|
||||
}
|
||||
console.clear()
|
||||
console.log('Import Progress Dashboard')
|
||||
console.log('========================')
|
||||
console.log(`Stage: ${progress.stage}`)
|
||||
console.log(`Status: ${progress.message}`)
|
||||
console.log(`Entities: ${progress.entities}`)
|
||||
console.log(`Relationships: ${progress.relationships}`)
|
||||
console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
|
||||
console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -310,21 +310,21 @@ await brain.import(buffer, {
|
|||
|
||||
```typescript
|
||||
const formatMessages = {
|
||||
csv: (p) => `CSV: ${p.message}`,
|
||||
pdf: (p) => `PDF: ${p.message}`,
|
||||
excel: (p) => `Excel: ${p.message}`,
|
||||
json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
|
||||
markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
|
||||
yaml: (p) => `YAML: ${p.processed} nodes`,
|
||||
docx: (p) => `DOCX: ${p.processed} paragraphs`
|
||||
csv: (p) => `CSV: ${p.message}`,
|
||||
pdf: (p) => `PDF: ${p.message}`,
|
||||
excel: (p) => `Excel: ${p.message}`,
|
||||
json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
|
||||
markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
|
||||
yaml: (p) => `YAML: ${p.processed} nodes`,
|
||||
docx: (p) => `DOCX: ${p.processed} paragraphs`
|
||||
}
|
||||
|
||||
await brain.import(buffer, {
|
||||
onProgress: (progress) => {
|
||||
// Format is available in progress.stage metadata
|
||||
const message = formatMessages[detectedFormat]?.(progress) || progress.message
|
||||
console.log(message)
|
||||
}
|
||||
onProgress: (progress) => {
|
||||
// Format is available in progress.stage metadata
|
||||
const message = formatMessages[detectedFormat]?.(progress) || progress.message
|
||||
console.log(message)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -336,18 +336,18 @@ await brain.import(buffer, {
|
|||
|
||||
```typescript
|
||||
let lastUIUpdate = 0
|
||||
const THROTTLE_MS = 100 // Update UI max once per 100ms
|
||||
const THROTTLE_MS = 100 // Update UI max once per 100ms
|
||||
|
||||
await brain.import(buffer, {
|
||||
onProgress: (progress) => {
|
||||
const now = Date.now()
|
||||
if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
|
||||
return // Skip this update
|
||||
}
|
||||
onProgress: (progress) => {
|
||||
const now = Date.now()
|
||||
if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
|
||||
return // Skip this update
|
||||
}
|
||||
|
||||
lastUIUpdate = now
|
||||
updateUI(progress) // Only update every 100ms
|
||||
}
|
||||
lastUIUpdate = now
|
||||
updateUI(progress) // Only update every 100ms
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -28,17 +28,17 @@
|
|||
```typescript
|
||||
// THE PUBLIC API - Same for ALL 7 formats!
|
||||
brain.import(buffer, {
|
||||
onProgress: (progress: ImportProgress) => {
|
||||
// These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
|
||||
progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
|
||||
progress.message // Human-readable status (varies by format, always readable)
|
||||
progress.processed // Items processed (optional)
|
||||
progress.total // Total items (optional)
|
||||
progress.entities // Entities extracted (optional)
|
||||
progress.relationships // Relationships inferred (optional)
|
||||
progress.throughput // Items/sec (optional, during extraction)
|
||||
progress.eta // Time remaining in ms (optional)
|
||||
}
|
||||
onProgress: (progress: ImportProgress) => {
|
||||
// These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
|
||||
progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
|
||||
progress.message // Human-readable status (varies by format, always readable)
|
||||
progress.processed // Items processed (optional)
|
||||
progress.total // Total items (optional)
|
||||
progress.entities // Entities extracted (optional)
|
||||
progress.relationships // Relationships inferred (optional)
|
||||
progress.throughput // Items/sec (optional, during extraction)
|
||||
progress.eta // Time remaining in ms (optional)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -49,14 +49,14 @@ The table below shows how formats implement progress *internally*. Normal develo
|
|||
```typescript
|
||||
// Internal: Binary formats use handler hooks (you added these!)
|
||||
interface FormatHandlerProgressHooks {
|
||||
onBytesProcessed?: (bytes: number) => void
|
||||
onCurrentItem?: (message: string) => void
|
||||
onDataExtracted?: (count: number, total?: number) => void
|
||||
onBytesProcessed?: (bytes: number) => void
|
||||
onCurrentItem?: (message: string) => void
|
||||
onDataExtracted?: (count: number, total?: number) => void
|
||||
}
|
||||
|
||||
// Internal: Text formats use importer callbacks
|
||||
interface ImporterProgressCallback {
|
||||
onProgress?: (stats: { processed, total, entities, relationships }) => void
|
||||
onProgress?: (stats: { processed, total, entities, relationships }) => void
|
||||
}
|
||||
|
||||
// Both are converted to ImportProgress by ImportCoordinator!
|
||||
|
|
@ -74,7 +74,7 @@ interface ImporterProgressCallback {
|
|||
|
||||
## 🎯 Overview
|
||||
|
||||
As of v4.5.0, Brainy supports comprehensive, multi-dimensional progress tracking for imports:
|
||||
Brainy supports comprehensive, multi-dimensional progress tracking for imports:
|
||||
- **Bytes processed** (always available, most deterministic)
|
||||
- **Entities extracted** (AI extraction phase)
|
||||
- **Stage-specific metrics** (parsing: MB/s, extraction: entities/s)
|
||||
|
|
@ -91,23 +91,23 @@ All handlers follow a simple, consistent pattern using **progress hooks**.
|
|||
|
||||
```typescript
|
||||
export interface FormatHandlerProgressHooks {
|
||||
/**
|
||||
* Report bytes processed
|
||||
* Call this as you read/parse the file
|
||||
*/
|
||||
onBytesProcessed?: (bytes: number) => void
|
||||
/**
|
||||
* Report bytes processed
|
||||
* Call this as you read/parse the file
|
||||
*/
|
||||
onBytesProcessed?: (bytes: number) => void
|
||||
|
||||
/**
|
||||
* Set current processing context
|
||||
* Examples: "Processing page 5", "Reading sheet: Q2 Sales"
|
||||
*/
|
||||
onCurrentItem?: (item: string) => void
|
||||
/**
|
||||
* Set current processing context
|
||||
* Examples: "Processing page 5", "Reading sheet: Q2 Sales"
|
||||
*/
|
||||
onCurrentItem?: (item: string) => void
|
||||
|
||||
/**
|
||||
* Report structured data extraction progress
|
||||
* Examples: "Extracted 100 rows", "Parsed 50 paragraphs"
|
||||
*/
|
||||
onDataExtracted?: (count: number, total?: number) => void
|
||||
/**
|
||||
* Report structured data extraction progress
|
||||
* Examples: "Extracted 100 rows", "Parsed 50 paragraphs"
|
||||
*/
|
||||
onDataExtracted?: (count: number, total?: number) => void
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -117,19 +117,19 @@ Progress hooks are automatically passed to your handler via `FormatHandlerOption
|
|||
|
||||
```typescript
|
||||
export interface FormatHandlerOptions {
|
||||
// ... existing options ...
|
||||
// ... existing options ...
|
||||
|
||||
/**
|
||||
* Progress hooks (v4.5.0)
|
||||
* Handlers call these to report progress during processing
|
||||
*/
|
||||
progressHooks?: FormatHandlerProgressHooks
|
||||
/**
|
||||
* Progress hooks
|
||||
* Handlers call these to report progress during processing
|
||||
*/
|
||||
progressHooks?: FormatHandlerProgressHooks
|
||||
|
||||
/**
|
||||
* Total file size in bytes (v4.5.0)
|
||||
* Used for progress percentage calculation
|
||||
*/
|
||||
totalBytes?: number
|
||||
/**
|
||||
* Total file size in bytes
|
||||
* Used for progress percentage calculation
|
||||
*/
|
||||
totalBytes?: number
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -141,36 +141,36 @@ Every handler follows these 5 steps:
|
|||
|
||||
```typescript
|
||||
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
|
||||
const progressHooks = options.progressHooks // Step 1: Get hooks
|
||||
const progressHooks = options.progressHooks // Step 1: Get hooks
|
||||
|
||||
// Step 2: Report initial progress
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem('Starting import...')
|
||||
}
|
||||
// Step 2: Report initial progress
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem('Starting import...')
|
||||
}
|
||||
|
||||
// Step 3: Report bytes as you process
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(0) // Start
|
||||
}
|
||||
// Step 3: Report bytes as you process
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(0) // Start
|
||||
}
|
||||
|
||||
// ... do parsing ...
|
||||
// ... do parsing ...
|
||||
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(buffer.length) // Complete
|
||||
}
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(buffer.length) // Complete
|
||||
}
|
||||
|
||||
// Step 4: Report data extraction
|
||||
if (progressHooks?.onDataExtracted) {
|
||||
progressHooks.onDataExtracted(data.length, data.length)
|
||||
}
|
||||
// Step 4: Report data extraction
|
||||
if (progressHooks?.onDataExtracted) {
|
||||
progressHooks.onDataExtracted(data.length, data.length)
|
||||
}
|
||||
|
||||
// Step 5: Report completion
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Complete: ${data.length} items processed`)
|
||||
}
|
||||
// Step 5: Report completion
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Complete: ${data.length} items processed`)
|
||||
}
|
||||
|
||||
return { format, data, metadata }
|
||||
return { format, data, metadata }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -182,76 +182,76 @@ Here's the **ACTUAL implementation** from CSV handler showing all the key progre
|
|||
|
||||
```typescript
|
||||
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
|
||||
const startTime = Date.now()
|
||||
const progressHooks = options.progressHooks // ✅ Step 1
|
||||
const startTime = Date.now()
|
||||
const progressHooks = options.progressHooks // ✅ Step 1
|
||||
|
||||
// Convert to buffer if string
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
|
||||
const totalBytes = buffer.length
|
||||
// Convert to buffer if string
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
|
||||
const totalBytes = buffer.length
|
||||
|
||||
// ✅ Step 2: Report start
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(0)
|
||||
}
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...')
|
||||
}
|
||||
// ✅ Step 2: Report start
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(0)
|
||||
}
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...')
|
||||
}
|
||||
|
||||
// Detect encoding
|
||||
const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
|
||||
const text = buffer.toString(detectedEncoding as BufferEncoding)
|
||||
// Detect encoding
|
||||
const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
|
||||
const text = buffer.toString(detectedEncoding as BufferEncoding)
|
||||
|
||||
// Detect delimiter
|
||||
const delimiter = options.csvDelimiter || this.detectDelimiter(text)
|
||||
// Detect delimiter
|
||||
const delimiter = options.csvDelimiter || this.detectDelimiter(text)
|
||||
|
||||
// ✅ Progress update: Parsing phase
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
|
||||
}
|
||||
// ✅ Progress update: Parsing phase
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
|
||||
}
|
||||
|
||||
// Parse CSV
|
||||
const records = parse(text, { /* options */ })
|
||||
// Parse CSV
|
||||
const records = parse(text, { /* options */ })
|
||||
|
||||
// ✅ Step 3: Report bytes processed (entire file parsed)
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(totalBytes)
|
||||
}
|
||||
// ✅ Step 3: Report bytes processed (entire file parsed)
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(totalBytes)
|
||||
}
|
||||
|
||||
const data = Array.isArray(records) ? records : [records]
|
||||
const data = Array.isArray(records) ? records : [records]
|
||||
|
||||
// ✅ Step 4: Report data extraction
|
||||
if (progressHooks?.onDataExtracted) {
|
||||
progressHooks.onDataExtracted(data.length, data.length)
|
||||
}
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`)
|
||||
}
|
||||
// ✅ Step 4: Report data extraction
|
||||
if (progressHooks?.onDataExtracted) {
|
||||
progressHooks.onDataExtracted(data.length, data.length)
|
||||
}
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`)
|
||||
}
|
||||
|
||||
// Type inference and conversion
|
||||
const fields = data.length > 0 ? Object.keys(data[0]) : []
|
||||
const types = this.inferFieldTypes(data)
|
||||
// Type inference and conversion
|
||||
const fields = data.length > 0 ? Object.keys(data[0]) : []
|
||||
const types = this.inferFieldTypes(data)
|
||||
|
||||
const convertedData = data.map((row, index) => {
|
||||
const converted = this.convertRow(row, types)
|
||||
const convertedData = data.map((row, index) => {
|
||||
const converted = this.convertRow(row, types)
|
||||
|
||||
// ✅ Progress update every 1000 rows (avoid spam)
|
||||
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
|
||||
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
|
||||
}
|
||||
// ✅ Progress update every 1000 rows (avoid spam)
|
||||
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
|
||||
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
|
||||
}
|
||||
|
||||
return converted
|
||||
})
|
||||
return converted
|
||||
})
|
||||
|
||||
// ✅ Step 5: Report completion
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
|
||||
}
|
||||
// ✅ Step 5: Report completion
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
|
||||
}
|
||||
|
||||
return {
|
||||
format: this.format,
|
||||
data: convertedData,
|
||||
metadata: { /* ... */ }
|
||||
}
|
||||
return {
|
||||
format: this.format,
|
||||
data: convertedData,
|
||||
metadata: { /* ... */ }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -292,51 +292,51 @@ Brainy supports **7 file formats** with full progress tracking:
|
|||
|
||||
```typescript
|
||||
async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
|
||||
const progressHooks = options.progressHooks
|
||||
const totalBytes = data.length
|
||||
const progressHooks = options.progressHooks
|
||||
const totalBytes = data.length
|
||||
|
||||
// Report start
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem('Loading PDF document...')
|
||||
}
|
||||
// Report start
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem('Loading PDF document...')
|
||||
}
|
||||
|
||||
const pdfDoc = await loadPDF(data)
|
||||
const totalPages = pdfDoc.numPages
|
||||
const pdfDoc = await loadPDF(data)
|
||||
const totalPages = pdfDoc.numPages
|
||||
|
||||
const extractedData: any[] = []
|
||||
let bytesProcessed = 0
|
||||
const extractedData: any[] = []
|
||||
let bytesProcessed = 0
|
||||
|
||||
for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
|
||||
// ✅ Report current page
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`)
|
||||
}
|
||||
for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
|
||||
// ✅ Report current page
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`)
|
||||
}
|
||||
|
||||
const page = await pdfDoc.getPage(pageNum)
|
||||
const text = await page.getTextContent()
|
||||
extractedData.push(this.processPageText(text))
|
||||
const page = await pdfDoc.getPage(pageNum)
|
||||
const text = await page.getTextContent()
|
||||
extractedData.push(this.processPageText(text))
|
||||
|
||||
// ✅ Estimate bytes processed (pages are sequential)
|
||||
bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes)
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(bytesProcessed)
|
||||
}
|
||||
// ✅ Estimate bytes processed (pages are sequential)
|
||||
bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes)
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(bytesProcessed)
|
||||
}
|
||||
|
||||
// ✅ Report extraction progress
|
||||
if (progressHooks?.onDataExtracted) {
|
||||
progressHooks.onDataExtracted(pageNum, totalPages)
|
||||
}
|
||||
}
|
||||
// ✅ Report extraction progress
|
||||
if (progressHooks?.onDataExtracted) {
|
||||
progressHooks.onDataExtracted(pageNum, totalPages)
|
||||
}
|
||||
}
|
||||
|
||||
// Final progress
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(totalBytes)
|
||||
}
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`)
|
||||
}
|
||||
// Final progress
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(totalBytes)
|
||||
}
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`)
|
||||
}
|
||||
|
||||
return { format: 'pdf', data: extractedData, metadata: { /* ... */ } }
|
||||
return { format: 'pdf', data: extractedData, metadata: { /* ... */ } }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -344,55 +344,55 @@ async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedDat
|
|||
|
||||
```typescript
|
||||
async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
|
||||
const progressHooks = options.progressHooks
|
||||
const totalBytes = data.length
|
||||
const progressHooks = options.progressHooks
|
||||
const totalBytes = data.length
|
||||
|
||||
// Load workbook
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem('Loading Excel workbook...')
|
||||
}
|
||||
// Load workbook
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem('Loading Excel workbook...')
|
||||
}
|
||||
|
||||
const workbook = XLSX.read(data)
|
||||
const sheetNames = options.excelSheets === 'all'
|
||||
? workbook.SheetNames
|
||||
: (options.excelSheets || [workbook.SheetNames[0]])
|
||||
const workbook = XLSX.read(data)
|
||||
const sheetNames = options.excelSheets === 'all'
|
||||
? workbook.SheetNames
|
||||
: (options.excelSheets || [workbook.SheetNames[0]])
|
||||
|
||||
const allData: any[] = []
|
||||
let bytesProcessed = 0
|
||||
const allData: any[] = []
|
||||
let bytesProcessed = 0
|
||||
|
||||
for (let i = 0; i < sheetNames.length; i++) {
|
||||
const sheetName = sheetNames[i]
|
||||
for (let i = 0; i < sheetNames.length; i++) {
|
||||
const sheetName = sheetNames[i]
|
||||
|
||||
// ✅ Report current sheet
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`)
|
||||
}
|
||||
// ✅ Report current sheet
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`)
|
||||
}
|
||||
|
||||
const sheet = workbook.Sheets[sheetName]
|
||||
const sheetData = XLSX.utils.sheet_to_json(sheet)
|
||||
allData.push(...sheetData)
|
||||
const sheet = workbook.Sheets[sheetName]
|
||||
const sheetData = XLSX.utils.sheet_to_json(sheet)
|
||||
allData.push(...sheetData)
|
||||
|
||||
// ✅ Estimate bytes processed (sheets processed sequentially)
|
||||
bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes)
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(bytesProcessed)
|
||||
}
|
||||
// ✅ Estimate bytes processed (sheets processed sequentially)
|
||||
bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes)
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(bytesProcessed)
|
||||
}
|
||||
|
||||
// ✅ Report data extraction
|
||||
if (progressHooks?.onDataExtracted) {
|
||||
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done
|
||||
}
|
||||
}
|
||||
// ✅ Report data extraction
|
||||
if (progressHooks?.onDataExtracted) {
|
||||
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done
|
||||
}
|
||||
}
|
||||
|
||||
// Final progress
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(totalBytes)
|
||||
}
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`)
|
||||
}
|
||||
// Final progress
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(totalBytes)
|
||||
}
|
||||
if (progressHooks?.onCurrentItem) {
|
||||
progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`)
|
||||
}
|
||||
|
||||
return { format: 'xlsx', data: allData, metadata: { /* ... */ } }
|
||||
return { format: 'xlsx', data: allData, metadata: { /* ... */ } }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -400,44 +400,44 @@ async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedDat
|
|||
|
||||
```typescript
|
||||
async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResult> {
|
||||
// ✅ Report parsing start
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
// ✅ Report parsing start
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
|
||||
// Parse JSON if string
|
||||
let jsonData = typeof data === 'string' ? JSON.parse(data) : data
|
||||
// Parse JSON if string
|
||||
let jsonData = typeof data === 'string' ? JSON.parse(data) : data
|
||||
|
||||
// ✅ Report parsing complete
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
// ✅ Report parsing complete
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
|
||||
// Traverse and extract (reports progress every 10 nodes)
|
||||
const entities: ExtractedJSONEntity[] = []
|
||||
const relationships: ExtractedJSONRelationship[] = []
|
||||
let nodesProcessed = 0
|
||||
// Traverse and extract (reports progress every 10 nodes)
|
||||
const entities: ExtractedJSONEntity[] = []
|
||||
const relationships: ExtractedJSONRelationship[] = []
|
||||
let nodesProcessed = 0
|
||||
|
||||
await this.traverseJSON(
|
||||
jsonData,
|
||||
entities,
|
||||
relationships,
|
||||
() => {
|
||||
nodesProcessed++
|
||||
if (nodesProcessed % 10 === 0) {
|
||||
options.onProgress?.({
|
||||
processed: nodesProcessed,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
}
|
||||
}
|
||||
)
|
||||
await this.traverseJSON(
|
||||
jsonData,
|
||||
entities,
|
||||
relationships,
|
||||
() => {
|
||||
nodesProcessed++
|
||||
if (nodesProcessed % 10 === 0) {
|
||||
options.onProgress?.({
|
||||
processed: nodesProcessed,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
// ✅ Report completion
|
||||
options.onProgress?.({
|
||||
processed: nodesProcessed,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
// ✅ Report completion
|
||||
options.onProgress?.({
|
||||
processed: nodesProcessed,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
|
||||
return { nodesProcessed, entitiesExtracted: entities.length, ... }
|
||||
return { nodesProcessed, entitiesExtracted: entities.length, ... }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -445,38 +445,38 @@ async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResul
|
|||
|
||||
```typescript
|
||||
async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<SmartMarkdownResult> {
|
||||
// ✅ Report parsing start
|
||||
options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 })
|
||||
// ✅ Report parsing start
|
||||
options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 })
|
||||
|
||||
// Parse markdown into sections
|
||||
const parsedSections = this.parseMarkdown(markdown, options)
|
||||
// Parse markdown into sections
|
||||
const parsedSections = this.parseMarkdown(markdown, options)
|
||||
|
||||
// ✅ Report parsing complete
|
||||
options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 })
|
||||
// ✅ Report parsing complete
|
||||
options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 })
|
||||
|
||||
// Process each section (reports progress after each section)
|
||||
const sections: MarkdownSection[] = []
|
||||
for (let i = 0; i < parsedSections.length; i++) {
|
||||
const section = await this.processSection(parsedSections[i], options)
|
||||
sections.push(section)
|
||||
// Process each section (reports progress after each section)
|
||||
const sections: MarkdownSection[] = []
|
||||
for (let i = 0; i < parsedSections.length; i++) {
|
||||
const section = await this.processSection(parsedSections[i], options)
|
||||
sections.push(section)
|
||||
|
||||
options.onProgress?.({
|
||||
processed: i + 1,
|
||||
total: parsedSections.length,
|
||||
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
|
||||
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
|
||||
})
|
||||
}
|
||||
options.onProgress?.({
|
||||
processed: i + 1,
|
||||
total: parsedSections.length,
|
||||
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
|
||||
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
|
||||
})
|
||||
}
|
||||
|
||||
// ✅ Report completion
|
||||
options.onProgress?.({
|
||||
processed: sections.length,
|
||||
total: sections.length,
|
||||
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
|
||||
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
|
||||
})
|
||||
// ✅ Report completion
|
||||
options.onProgress?.({
|
||||
processed: sections.length,
|
||||
total: sections.length,
|
||||
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
|
||||
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
|
||||
})
|
||||
|
||||
return { sectionsProcessed: sections.length, ... }
|
||||
return { sectionsProcessed: sections.length, ... }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -484,27 +484,27 @@ async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<Sma
|
|||
|
||||
```typescript
|
||||
async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Promise<SmartYAMLResult> {
|
||||
// ✅ Report parsing start
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
// ✅ Report parsing start
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
|
||||
// Parse YAML
|
||||
const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8')
|
||||
const data = yaml.load(yamlString)
|
||||
// Parse YAML
|
||||
const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8')
|
||||
const data = yaml.load(yamlString)
|
||||
|
||||
// ✅ Report parsing complete
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
// ✅ Report parsing complete
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
|
||||
// Traverse YAML structure (reports progress every 10 nodes)
|
||||
// ... similar to JSON traversal ...
|
||||
// Traverse YAML structure (reports progress every 10 nodes)
|
||||
// ... similar to JSON traversal ...
|
||||
|
||||
// ✅ Report completion (already implemented)
|
||||
options.onProgress?.({
|
||||
processed: nodesProcessed,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
// ✅ Report completion (already implemented)
|
||||
options.onProgress?.({
|
||||
processed: nodesProcessed,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
|
||||
return { nodesProcessed, entitiesExtracted: entities.length, ... }
|
||||
return { nodesProcessed, entitiesExtracted: entities.length, ... }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -512,39 +512,39 @@ async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Pro
|
|||
|
||||
```typescript
|
||||
async extract(buffer: Buffer, options: SmartDOCXOptions = {}): Promise<SmartDOCXResult> {
|
||||
// ✅ Report parsing start
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
// ✅ Report parsing start
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
|
||||
// Extract text and HTML using Mammoth
|
||||
const textResult = await mammoth.extractRawText({ buffer })
|
||||
const htmlResult = await mammoth.convertToHtml({ buffer })
|
||||
// Extract text and HTML using Mammoth
|
||||
const textResult = await mammoth.extractRawText({ buffer })
|
||||
const htmlResult = await mammoth.convertToHtml({ buffer })
|
||||
|
||||
// ✅ Report parsing complete
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
// ✅ Report parsing complete
|
||||
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
|
||||
|
||||
// Process paragraphs (reports progress every 10 paragraphs)
|
||||
const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength)
|
||||
// Process paragraphs (reports progress every 10 paragraphs)
|
||||
const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength)
|
||||
|
||||
for (let i = 0; i < paragraphs.length; i++) {
|
||||
await this.processParagraph(paragraphs[i])
|
||||
for (let i = 0; i < paragraphs.length; i++) {
|
||||
await this.processParagraph(paragraphs[i])
|
||||
|
||||
if (i % 10 === 0) {
|
||||
options.onProgress?.({
|
||||
processed: i + 1,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
}
|
||||
}
|
||||
if (i % 10 === 0) {
|
||||
options.onProgress?.({
|
||||
processed: i + 1,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ✅ Report completion (already implemented)
|
||||
options.onProgress?.({
|
||||
processed: paragraphs.length,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
// ✅ Report completion (already implemented)
|
||||
options.onProgress?.({
|
||||
processed: paragraphs.length,
|
||||
entities: entities.length,
|
||||
relationships: relationships.length
|
||||
})
|
||||
|
||||
return { paragraphsProcessed: paragraphs.length, ... }
|
||||
return { paragraphsProcessed: paragraphs.length, ... }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -559,20 +559,20 @@ Progress hooks are **optional**. Always check before calling:
|
|||
```typescript
|
||||
// ✅ Good - safe
|
||||
if (progressHooks?.onBytesProcessed) {
|
||||
progressHooks.onBytesProcessed(bytes)
|
||||
progressHooks.onBytesProcessed(bytes)
|
||||
}
|
||||
|
||||
// ❌ Bad - will crash if hooks undefined
|
||||
progressHooks.onBytesProcessed(bytes) // TypeError!
|
||||
progressHooks.onBytesProcessed(bytes) // TypeError!
|
||||
```
|
||||
|
||||
### 2. Report Bytes at Start and End
|
||||
|
||||
```typescript
|
||||
// ✅ Good - clear start and end
|
||||
progressHooks?.onBytesProcessed(0) // Start
|
||||
progressHooks?.onBytesProcessed(0) // Start
|
||||
// ... processing ...
|
||||
progressHooks?.onBytesProcessed(totalBytes) // End
|
||||
progressHooks?.onBytesProcessed(totalBytes) // End
|
||||
|
||||
// ❌ Bad - no clear boundaries
|
||||
// ... just start processing without reporting start
|
||||
|
|
@ -583,17 +583,17 @@ progressHooks?.onBytesProcessed(totalBytes) // End
|
|||
```typescript
|
||||
// ✅ Good - report every 1000 items
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
processItem(items[i])
|
||||
processItem(items[i])
|
||||
|
||||
if (i > 0 && i % 1000 === 0) {
|
||||
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`)
|
||||
}
|
||||
if (i > 0 && i % 1000 === 0) {
|
||||
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`)
|
||||
}
|
||||
}
|
||||
|
||||
// ❌ Bad - report EVERY item (spam!)
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
processItem(items[i])
|
||||
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks!
|
||||
processItem(items[i])
|
||||
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks!
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -614,13 +614,13 @@ progressHooks?.onCurrentItem('Working...')
|
|||
|
||||
```typescript
|
||||
// ✅ Good - total known
|
||||
progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows
|
||||
progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows
|
||||
|
||||
// ✅ Also good - total unknown (streaming)
|
||||
progressHooks?.onDataExtracted(100, undefined) // 100 rows so far
|
||||
progressHooks?.onDataExtracted(100, undefined) // 100 rows so far
|
||||
|
||||
// ✅ Also good - complete
|
||||
progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows
|
||||
progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -637,18 +637,18 @@ const handler = new CSVHandler()
|
|||
const data = fs.readFileSync('./test.csv')
|
||||
|
||||
const result = await handler.process(data, {
|
||||
filename: 'test.csv',
|
||||
progressHooks: {
|
||||
onBytesProcessed: (bytes) => {
|
||||
console.log(`Bytes: ${bytes}`)
|
||||
},
|
||||
onCurrentItem: (item) => {
|
||||
console.log(`Status: ${item}`)
|
||||
},
|
||||
onDataExtracted: (count, total) => {
|
||||
console.log(`Extracted: ${count}${total ? `/${total}` : ''}`)
|
||||
}
|
||||
}
|
||||
filename: 'test.csv',
|
||||
progressHooks: {
|
||||
onBytesProcessed: (bytes) => {
|
||||
console.log(`Bytes: ${bytes}`)
|
||||
},
|
||||
onCurrentItem: (item) => {
|
||||
console.log(`Status: ${item}`)
|
||||
},
|
||||
onDataExtracted: (count, total) => {
|
||||
console.log(`Extracted: ${count}${total ? `/${total}` : ''}`)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
console.log(`Complete: ${result.data.length} rows`)
|
||||
|
|
@ -673,24 +673,24 @@ Complete: 1000 rows
|
|||
|
||||
```
|
||||
User Imports File
|
||||
↓
|
||||
↓
|
||||
ImportManager
|
||||
↓
|
||||
↓
|
||||
Creates ProgressTracker
|
||||
↓
|
||||
↓
|
||||
Calls Handler.process() with progressHooks
|
||||
↓
|
||||
↓
|
||||
Handler Reports Progress:
|
||||
├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated
|
||||
├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated
|
||||
├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated
|
||||
├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated
|
||||
└─ onCurrentItem("Complete") → ProgressTracker → final progress
|
||||
↓
|
||||
├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated
|
||||
├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated
|
||||
├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated
|
||||
├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated
|
||||
└─ onCurrentItem("Complete") → ProgressTracker → final progress
|
||||
↓
|
||||
ProgressTracker emits to callback (throttled 100ms)
|
||||
↓
|
||||
↓
|
||||
User sees:
|
||||
"Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..."
|
||||
"Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..."
|
||||
```
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -91,7 +91,7 @@ await brain.import(file, {
|
|||
})
|
||||
```
|
||||
|
||||
### Import Tracking (v4.10.0+)
|
||||
### Import Tracking
|
||||
|
||||
Track and organize imports by project:
|
||||
|
||||
|
|
@ -149,7 +149,7 @@ await brain.import(file, {
|
|||
})
|
||||
```
|
||||
|
||||
### Always-On Streaming (v4.2.0+)
|
||||
### Always-On Streaming
|
||||
|
||||
All imports use streaming with adaptive flush intervals. Query data as it's imported:
|
||||
|
||||
|
|
|
|||
|
|
@ -65,7 +65,7 @@ interface ImportProgress {
|
|||
/** Estimated time remaining (milliseconds) */
|
||||
eta?: number
|
||||
|
||||
/** Whether data is queryable at this point (v4.2.0+) */
|
||||
/** Whether data is queryable at this point */
|
||||
queryable?: boolean
|
||||
}
|
||||
```
|
||||
|
|
|
|||
|
|
@ -300,7 +300,7 @@ interface ImportProgress {
|
|||
relationships?: number // Relationships inferred so far
|
||||
|
||||
/**
|
||||
* Whether data is queryable (v4.2.0+)
|
||||
* Whether data is queryable
|
||||
*
|
||||
* true = Indexes flushed, queries will be fast and complete
|
||||
* false/undefined = Data in storage but indexes not flushed yet
|
||||
|
|
@ -376,7 +376,7 @@ No changes required! Streaming is now always enabled with optimal defaults:
|
|||
// Before (v3.x, v4.0, v4.1): Works the same
|
||||
await brain.import(file)
|
||||
|
||||
// After (v4.2.0+): Streaming always on, zero config
|
||||
// After: Streaming always on, zero config
|
||||
await brain.import(file)
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue